Differences

This shows you the differences between two versions of the page.

--- cs501r_f2018:lab6 [2018/10/02 17:52]
carr
+++ cs501r_f2018:lab6 [2021/06/30 23:42] (current)
@@ Line 10: / Line 10: @@
 You will train it on a text corpus that you're interested in, and then
 show samples from the model.
+This lab is slightly different than previous labs in that we give you a large portion of the code, and you will just be filling in pieces of classes and functions. If you get suck, please get help from the TA's or your classmates.
 You should turn in your jupyter notebook
@@ Line 38: / Line 40: @@
 Your code/image will be graded on the following:
-  * 20% Correct use of sequence encoder
+  * 40% Correct implementation of the sequence to sequence class
-  * 20% Correct use of sequence decoder
+  * 20% Correct implementation of training and sampling
-  * 20% Correct implementation of GRU cell
+  * 5%  Correct implementation of GRU cell
-  * 20% Correct sharing of variables
+  * 20% Training and sampling on a novel text dataset (Must be your choice)
-  * 20% Training and sampling on a novel text dataset
+  * 15% Good coding style, readable output
 ----
@@ Line 66: / Line 68: @@
 ----
-**Part 0: Readings and data loading**
+**Part 0: Readings, data loading, and high level training**
 There is a tutorial here that will help build out scaffolding code, and get an understanding of using sequences in pytorch.
@@ Line 74: / Line 76: @@
 [[http://colah.github.io/posts/2015-08-Understanding-LSTMs/|Understanding LSTM Networks]]
+<code bash>
+! wget -O ./text_files.tar.gz 'https://piazza.com/redirect/s3?bucket=uploads&prefix=attach%2Fjlifkda6h0x5bk%2Fhzosotq4zil49m%2Fjn13x09arfeb%2Ftext_files.tar.gz'
+! tar -xzf text_files.tar.gz
+! pip install unidecode
+! pip install torch
+</code>
 <code python>
@@ Line 103: / Line 114: @@
 <code python>
+import torch
+from torch.autograd import Variable
 # Turn string into list of longs
 def char_tensor(string):
@@ Line 121: / Line 134: @@
 </code>
-----
+<code python>
-**Part 1: Sampling new sequences**
+import time
+n_epochs = 2000
+print_every = 100
+plot_every = 10
+hidden_size = 100
+n_layers = 1
+lr = 0.005
-Ok, great.  So we're training an RNN.  But how can we sample new text
+decoder = RNN(n_characters, hidden_size, n_characters, n_layers)
-from it?
+decoder_optimizer = torch.optim.Adam(decoder.parameters(), lr=lr)
+criterion = nn.CrossEntropyLoss()
-The basic idea is to carefully execute one step at a time.  To show
+start = time.time()
-you how to do this, the scaffold code has a "sampler" method in it
+all_losses = []
-that feeds characters one at a time into a "sampler graph", tracking
+loss_avg = 0
-the state and sampling from the resulting probabilities.
-You are welcome to modify this code however you want.
+for epoch in range(1, n_epochs + 1):
+    loss_ = train(*random_training_set())
+    loss_avg += loss_
-You need to create a separate computation graph for this sampling
+    if epoch % print_every == 0:
-step.  It will look identical to your main computation graph (**and
+        print('[%s (%d %d%%) %.4f]' % (time.time() - start, epoch, epoch / n_epochs * 100, loss_))
-should share all of the same parameters!**), but it will be for a
+        print(evaluate('Wh', 100), '\n')
-fixed "batch_size" of 1 and "sequence_length" of 1.
-If you name your variables appropriately, you should be able to just
+    if epoch % plot_every == 0:
-use my sampler function.
+        all_losses.append(loss_avg / plot_every)
+        loss_avg = 0
+</code>
 ----
-**Part 2: Creating your own GRU cell**
+**Part 1: Building a sequence to sequence model**
-The cell that you used in Part 0 was a pre-defined Tensorflow cell.
+Great! We have the data in a useable form. We can switch out which text file we are reading from, and trying to simulate.
-The Tensorflow RNN API allows you to define your own cell by
-subclassing the `RNNCell` class.  For this part, you should create a
+We now want to build out an RNN model, in this section, we will use all built in Pytorch pieces when building our RNN class.
-custom GRU cell by creating a new class by filling in the following
-method:
+Create an RNN class that extends from nn.Module.
 <code python>
+import torch
+import torch.nn as nn
+from torch.autograd import Variable
-from tensorflow.python.ops.rnn_cell import RNNCell
+class RNN(nn.Module):
+    def __init__(self, input_size, hidden_size, output_size, n_layers=1):
+        super(RNN, self).__init__()
+        self.input_size = input_size
+        self.hidden_size = hidden_size
+        self.output_size = output_size
+        self.n_layers = n_layers
+        # encode using embedding layer
+        # set up GRU passing in number of layers parameter (nn.GRU)
+        # decode output
+    def forward(self, input_char, hidden):
+        # by reviewing the documentation, construct a forward function that properly uses the output
+        # of the GRU
+        # return output and hidden
-class mygru( RNNCell ):
+    def init_hidden(self):
+        return Variable(torch.zeros(self.n_layers, 1, self.hidden_size))
+</code>
-    def __init__( self, ??? ):
+----
-    	pass
+**Part 2: Sample text and Training information**
-    @property
+We now want to be able to train our network, and sample text after training.
-    def state_size(self):
-    	pass
-    @property
+This function outlines how training a sequence style network goes. Fill in the pieces.
-    def output_size(self):
-    	pass
-    def __call__( self, inputs, state, scope=None ):
+<code python>
-    	pass
+def train(inp, target):
+    ## initialize hidden layers, set up gradient and loss
+      # your code here
+    ## /
+    loss = 0
+    for c in range(chunk_len):
+        output, hidden = # run the forward pass of your rnn with proper input
+        loss += criterion(output, target[c].unsqueeze(0))
+    ## calculate backwards loss and step the optimizer (globally)
+      # your code here
+    ## /
+    return loss.item() / chunk_len
 </code>
-To see how this really works, it is helpful to look at the definitions
+You can at this time, if you choose, also write out your train loop boilerplate that samples random sequences and trains your RNN. This will be helpful to have working before writing your own GRU class.
-of other cells, found (on my machine) in the file
-<code>
+If you are finished training, or during training, and you want to sample from the network you may consider using the following function. If your RNN model is instantiated as `decoder`then this will probabilistically sample a sequence of length `predict_len`
-/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn_cell.py
+<code python>
+def evaluate(prime_str='A', predict_len=100, temperature=0.8):
+    ## initialize hidden variable, initialize other useful variables
+      # your code here
+    ## /
+    prime_input = char_tensor(prime_str)
+    # Use priming string to "build up" hidden state
+    for p in range(len(prime_str) - 1):
+        _, hidden = decoder(prime_input[p], hidden)
+    inp = prime_input[-1]
+    for p in range(predict_len):
+        output, hidden = #run your RNN/decoder forward on the input
+        # Sample from the network as a multinomial distribution
+        output_dist = output.data.view(-1).div(temperature).exp()
+        top_i = torch.multinomial(output_dist, 1)[0]
+        ## get character from your list of all characters, add it to your output str sequence, set input
+        ## for the next pass through the model
+         # your code here
+        ## /
+    return predicted
 </code>
-**However, please try not to look at the GRU cell definition.**
+----
-That's right, the answer is right there, and in theory, you could
+**Part 3: Creating your own GRU cell**
+The cell that you used in Part 1 was a pre-defined Pytorch layer. Now, write your own GRU class using the same parameters as the built-in Pytorch class does.
+**Please try not to look at the GRU cell definition.**
+The answer is right there in the code, and in theory, you could
 just cut-and-paste it.  This bit is on your honor!
 ----
-**Part 3: Run it and generate your final text!**
+**Part 4: Run it and generate your final text!**
 Assuming everything has gone well, you should be able to run the main
-function in the scaffold code, using your custom GRU cell, and see
+function in the scaffold code, using either your custom GRU cell or the built in layer, and see
-output something like this.  I trained on the "alma.txt" dataset,
+output something like this.  I trained on the "lotr.txt" dataset,
-using sequence_length=50, batch_size=50, and state_dim=128.
+using chunk_length=200, hidden_size=100 for 2000 epochs gave.
 <code>
-0     4.2465
+[0m 9s (100 5%) 2.2169]
-And wdyunNutg weotete foe At
+Whaiss Mainde
- tharet mhg  tssaidatl atr
- eae to
-0     2.8634
-And utns, iugiin the cyel barthes hitaicen enthere ts sulod anta
-0     2.3854
-And gokse
- ne and thaseangang of the wesd unso he tait ewor besi
-0     2.2094
-And orto lhan I fome ho soy wort to come hyo deag copbeut.
+'
-, N
+he and the
-0     2.0746
-And kend Snowor to spae dil., hulth beemirss whar to wous thes m
-0     1.9668
-And ofolr yave preante for wa to.
- And thy it we be tho eprreno
-0     1.8655
-And an whis tretersJ of cil tony, now hat is snepin thee eos be
-0     1.7861
-And eomer shem; to thit tuscigh-whis porogw the reone uut whis i
-0     1.7170
-And bamonisery ald dad pasing their wich to ho ware great maviit
-0     1.6571
-And omen propre they wis pover upon of thy Lamanied agtoren moth
-0    1.6034
-And mantherefore, whiy well pofserse wo wo a ore thour paysien b
-0    1.5553
-And God of Nephe weat they welding oul hamy; and Amaming the Was
-0    1.5127
-And falling and is the my pithert to compiebion; thereforth, and
-0    1.4740
-And tothered of Nephites, and ristinvese accorangs was domlle ur
-0    1.4385
-And tire behinden to a sabe kean to past that chich me time: I a
-0    1.4072
-And to peepired farte; therefonch Chrathe in the kis iny? Moroni
-0    1.3782
-And wa righ, to ragking for it: I his brengmine med, and it your
-0    1.3519
-And and behold, and reptoregfor; who he kning:
- And recerited;
-0    1.3284
-And didnited upon the wam with is come words, that evong that no
-0    1.3051
-And ore their youndar oven the Lamanither hams, and low that ye
-0    1.2852
-And all; therefore ceme from to was ahore, in the city, and the
-0    1.2708
-And eforto Nephites, then; alanissuons or the, countom.
- And he
+'od and roulll and Are say the
-0    1.2497
+rere.
-And doth he pead, with did cities; and I to that it gath.
+'Wor
+'Iow anond wes ou
- And
+'Yi
-0    1.2330
-And keagition spanth which to man unto them. Wo and hear to that
-0    1.2121
-And to sent and having hopksect the awporit was waind of the lan
-0    1.1990
-And them a cit the sla; and behold is people that your land, tha
-0    1.1915
-And ans commack that they might upon their mand men by the naw a
-0    1.1767
-And mave the citer wat and surmed ox many of armon, the said thi
-0    1.1524
-And tremery believe Soviltent not their concitn of Teen to be pr
-0    1.1308
-And shim, the man unto it came to pass the rived to pose also; a
-0    1.1168
-And invaing the were have touslesssoon, and to our city of as te
-0    1.1086
-And bring.
- Now a quich, by fiel metry ming to the commandren t
+[0m 19s (200 10%) 2.0371]
-0    1.0965
+Whimbe.
-And from them in them, caused atgourness, my son, so, was war am
-0    1.0780
-And Staring: Moroni, and a say had bound from not have have foul
-0    1.0635
-And warp be not benong all thou hast dainthers, yea, which they
-0    1.0569
-And which I wore from the Spirits of our will come flact.
- Now
+'Thhe
-0    1.0491
+on not of they was thou hit of
-And werch ragets was etred which cast unto his armor of Amulek,
+sil ubat thith hy the seare
-0    1.0445
+as sower and of len beda
-And tought by owfs the hoig wirt lay hinds one their brethar, bu
-0    1.0294
-And word bring, and messer, now you, on the namb it come all one
-0    1.0074
-And against fantitions of Sidon, even to empt thises to sing man
-0    0.9927
-And wither the people which is fear ago not who thou hastinushs
-0    0.9840
-And not agam and armies for the words of Christ, inquer.
- For b
+[0m 29s (300 15%) 2.0051]
-0    0.9749
+Whis the cart. Whe courn!' 'Bu't of they aid dou giter of fintard of the not you ous,
-And wish spake tightiston slain who trought a privitions and sup
+'Thas orntie it
-0    0.9747
-And great as the eviek of the repturely from, and the people of
-0    0.9650
-And might they mickoning thick them rest exceedingly Lamoni land
-0    0.9476
-And is the hapthers?
- And ye was as my preacherce and of the li
+[0m 38s (400 20%) 1.8617]
-0    0.9373
+Wh win took be to the know the gost bing to kno wide dought, and he as of they thin.
-And might God, but now end their first, and according to to fait
-0    0.9230
-And that the Lamanites Lamanites for the words of Milshent up im
-0    0.9183
-And given to the king; then fore this givened that reternets to
-0    0.9082
-And thit had slaked all men what had gathere, which bringeded hi
-0    0.9186
-And wish power that ye shall pray, all this cursions a can to th
-0    0.9022
-And not giving with dimsed were trader, and place unto you.
- Be
+The Gonhis gura
-0    0.8966
-And never of the Lord; and he repentate of the sinces, whoughter
-0    0.8917
-And being the provored to the churchder.
- And that we should no
+[0m 48s (500 25%) 1.9821]
-0    0.8790
+When of they singly call the and thave thing
-And indachers more congen.
+they the nowly we'tly by ands, of less be grarmines of t
- And they were define among the peop
+[0m 58s (600 30%) 1.8170]
-0    0.8667
+Whinds to mass of I
-And him he might sayed the Lord from the cry to repent deliver,
+not ken we ting and dour
-0    0.8601
+and they.
-And might on the nint; and these chuncrusios of their hearts; ne
-0    0.8604
-And begirish--Zerahomnant the wide upon the Son of uster you, ha
-0    0.8481
-And to comeans, therefore they slougness for this great faith hi
-0    0.8438
-And who we had lid hath mansh, yea; and they twey swolling but f
-0    0.8352
-And again they did comvant of the people of pony city of Nephiha
-0    0.8333
-And who with piscorst.
- Now bon to take fallerant these wond as
-0    0.8206
-And poftor, they did take up rich unto him: I commencement food
-0    0.8202
-And to passed them.
- And now, do not a shange a string, ye lony
+'Wat res swe Ring set shat scmaid. The
-0    0.8148
+ha
-And befiel Requestion, but that therefore God Moroni, and their
-0    0.8107
-And their hapfenembly as done might not great, yea, and the land
-0    0.8024
-And with the king would no romp not and he wilterness of God, fo
-0    0.7942
-And for that it was not doot.
- And now it came to pass te his k
+[1m 7s (700 35%) 2.0367]
-0    0.7862
+Whad ded troud wanty agy. Ve tanle gour the gone veart on hear, as dent far of the Ridgees.'
-And buch against the church in the power of God; therefore I hav
-0    0.7872
-And were sins--the ether came pertays puran to wax liget.
- Cond
+'The Ri
-0    0.7935
-And frich mestireded to the Zoramites came of the crues had come
-0    0.8013
-And to pass, the Lord sit ad not blood; yea, this ye who will be
-0    0.7764
-And was given for the eart no men; and this was the word of Neph
-0    0.7634
-And free over the chief judge our implaties appointited from the
-0    0.7532
-And from himsmed, that the liveth by the mins and prisoner, and
-0    0.7604
-And hazrmen and phourth Aaron possession of the blessed against
-0    0.7584
-And buither many which and dilled that his people; whose the eas
-0    0.7512
-And whith him that he also smitting of the fled over working man
-0    0.7417
-And are not vost number of Care to the eard be ye think possessi
-0    0.7434
-And freich which that was not have aboume of our brethren, that
-0    0.7329
-And arw these after them in fone Ishlat in a happen up and take
-0    0.7394
-And with bring of his remainder and sent Curean by none wats, O
-0    0.7304
-And mind animed among them towarsh recommence of their coursend-
-0    0.7285
-And is them, and the wird Antipus sawe thy loft understands rete
-0    0.7160
-And mite thy are mottlo, yea, except they murnogeffers preparati
-0    0.7093
-And being had obear.
- Nevertheless, they were past into those c
+[1m 17s (800 40%) 1.9458]
-0    0.7045
+Whis is brouch Heared this lack and was weself, for on't
-And friont pervest should cut out God.
+abothom my and go staid it
+they curse arsh
- Now the consop gave the
+[1m 27s (900 45%) 1.7522]
-0    0.7014
+Whout bear the
-And with him; for know their people, become, and God the Lamanit
+Evening
-0    0.7094
+the pace spood, Arright the spaines beren the and Wish was was on the more yo
-And mind a. the things salver the command of their slave upon th
-0    0.6970
-And with his propheti, Ammon was a many thy fulfied after the Sp
-0    0.6904
-And havid; and thus beginn as plavellwary misenaged, saying, com
-0    0.6901
-And with their lives; and behold, jestinity. Now, let forth year
-0    0.6868
-And awau hard he swell these things pertaingd Middoni had set en
-0    0.6855
-And are care over the people of Nephi-Lehi, which he did jush as
-0    0.6819
-And freidded my son.
- And now, I say unto Zarahemla.
+[1m 37s (1000 50%) 1.6444]
+Whe Swarn. at colk. N(r)rce or they he
+wearing. And the on the he was are he said Pipin.
- Comfalal
+'Yes and i
-0    0.6808
-And pewer to place in Christ, but believe.
- Seet the last dain,
+[1m 47s (1100 55%) 1.8770]
-0    0.6670
+Whing at they and thins the Wil might
-And bite them they were men, that Lehon, as I say unto him: What
+happened you dlack rusting and thousting fy them, there lifted
-0    0.6642
-And him did that they had saw this children by Teancums and the
-0    0.6643
-And minam id ye beginged with the armies of the Lamanites was th
-0    0.6613
-And which had said that retain, ye begliving among all manner ov
-0   0.6599
-And age is very cones power in the sword.
- Therefore thou wilt
+[1m 57s (1200 60%) 1.9401]
-0   0.6740
+Wh the said Frodo eary him that the herremans!
-And praties of Moroni him about.
- And now these askengeh exerci
+'I the Lager into came and broveener he sanly
-0   0.6613
+for
-And are brought torded fordate werf and shedding unto repentance
+s
-0   0.6589
-And awe the works; and this man of plessed under the manst beste
-0   0.6489
-And from preserve them by the people to small their teother: I a
-0   0.6448
-And awai had great nardening the wickedness and the pride of the
-0   0.6427
-And bring when his bend of the flocks say unto his brethren may
-0   0.6378
-And firitious armide me stand give hast, saying: Telt redeem; be
-0   0.6317
-And are murders shall be angel because of given up just in their
-0   0.6341
-And about, which arthy forth Laman servants, brought up at a str
-0   0.6202
-And abority befold, this believer his laid cosumbly to executed
-0   0.6182
-And are.
- And now Nephi.
+[2m 7s (1300 65%) 1.8095]
+When lest
+- in sound fair, and
+the Did dark he in the gose cilling the stand I in the sight. Frodo y
- I would and men bar delighed, that w
+[2m 16s (1400 70%) 1.9229]
-0   0.6216
+Whing in a shade and Mowarse round and parse could pass not a have partainly. ' for as I come of I
-And freites, and be brought out of the land of Zarahemla. Ned th
+le
-0   0.6273
-And and Said hath not teot were cried--also be delivered by our
-0   0.6127
-And and time, their swords, and all the borders which laves amon
-0   0.6112
-And abehor is hearts of their long shord; yea, and also that it
-0   0.6051
-And awe this began to command him goreven; therefore there was I
-0   0.6054
-And into-tright, Zenamewhered abepting denis; and silver, they w
-0   0.6030
-And into the knewsed, and ablieve thing call, with him; for we w
-0   0.6028
-And and did this the judge Ammon, and his fathers year all our p
-0   0.6024
-And angifes; therefore they had fight and a sword, and there is
-0   0.5996
-And abie it supposed to deliver them.
- And it came to pass that
+[2m 26s (1500 75%) 1.8169]
-0   0.6005
+Whese one her of in a lief that,
-And bith a give as be saved--is to rememe far insteovert known u
+but. 'We repagessed,
-0   0.5872
+wandere in these fair of long one have here my
-And fapis of its; and also deny; their camp with him, and had de
-0   0.5854
-And into with his inuase men, on hercite watkennes of perime!
+[2m 36s (1600 80%) 1.6635]
-0   0.5817
+Where fread in thougraned in woohis, on the the green the
-And accord him suppose that there was ascomel not to hear; for t
+pohered alked tore becaming was seen what c
-0   0.5809
-And accien Siedgmentes of his brethren, which camp words his gre
-0   0.5765
-And awe this never his walls powers by the loss whereolah.
- Ant
+[2m 46s (1700 85%) 1.7868]
-0   0.5747
+Whil neat
-And eatek in there spiritted by mine say, because they had been
+came to
-0   0.5714
+is laked,
-And ir thirst; therefore a by your wives again; therefore he sai
+and fourst on him grey now they as pass away aren have in the border sw
-0   0.5662
-And enject his hand, they gathered tomsor; and he said unto men,
-0   0.5668
-And awain the coming of Noahor hath he said: Yea, they did not d
-0   0.5687
-And gid tike up their God.
- And it came to pass that the man? W
+[2m 56s (1800 90%) 1.6343]
-0   0.5634
+Wh magered.
-And aid the Nephites were spneed, and his brethren done of I hav
-0   0.5688
-And appeace the endemed with inhelvist destroy things know the l
-0   0.5628
-And afpiining this holy according to to learness before the Lama
-0   0.5638
-And accircures up their house to bring my buriaged which had int
-0   0.5534
-And freit. And it came to pass thas when Moroni had same pointen
-0   0.5485
-And firing to pass the Lord sixtt them.
- And now, behold, O the
+Then tell some tame had bear that
-0   0.5502
+came as it nome in
-And awau Ziesout had good were seniat in my righteok; and they a
+to houbbirnen and to heardy.
-0   0.5409
-And accien hampleate him, as I have some his dettrore to maintai
-0   0.5462
-And firen brethren, be relepor of king-men) sore wither every di
-0   0.5383
-And misins who were pleasated against the record rembant and the
-0   0.5419
-And iffinish to bahy tranch out of the land whatsoever no knowle
-0   0.5335
-And iffling their things, faith from the commencement rejoining
-0   0.5355
-And great pure good and to termy, in with their tenth in the pla
-0   0.5282
-And priesting him that might destroy their myselfings, insomperi
-0   0.5257
-And from with his ower of Pahoran, came over which they anger ag
-0   0.5308
-And friam timperisted against the man who had no power of whom t
-0   0.5230
-And acciritishe it corder, having fied against them by those men
-0   0.5254
-And cie tiand of joy, as the west to the power.
- And it came to
-0   0.5281
+'
-And awind him that my brethren speak unto him forward frighteent
-0   0.5270
+[3m 6s (1900 95%) 1.8191]
-And appeice when he led to back equace of for God, or by the Lam
+Who expey to must away be to the master felkly and for, what shours was alons? I had be the long to fo
-0   0.5190
-And afte it hat his preserving the plan eviltont of the hearts o
+[3m 16s (2000 100%) 1.8725]
-0   0.5106
+White, and his of his in before that for brown before can then took on the fainter smass about rifall
-And atininiful that there should death, and also me cometh, and
-0   0.5162
-And ifte the sword, who seon and disperibee unto me, shook feed
-0   0.5143
-And mitiquity mind, and according to the chief judge, so make at
-0   0.5192
-And ifte thin forgision forward thene over up to a fear not your
-0   0.5097
-And freitions, which is great God. Behold these are the loss sub
-0   0.5079
-And ache with the Lord hath bloes, which was done to the holy Gr
-0   0.5074
-And appeicis arm vinimonahites strong in name, to doth piseling
-0   0.5073
-And miniquithers these words, he commanded order not; neither sa
-0   0.5030
-And min for many would happine even to the earth, to said unto m
-0   0.5053
-And mie first be traditions? Behold, you, because it was a sound
-0   0.4915
-And from tike ended the Lamanites had administered, and I say bi
 </code>

BYU CS classes

User Tools

Site Tools

Differences

Page Tools