Differences

This shows you the differences between two versions of the page.

--- cs501r_f2016:lab10 [2016/10/25 02:26]
wingated
+++ cs501r_f2016:lab10 [2021/06/30 23:42]
@@ Line 1: / Line 1: @@
-====Objective:====
-To learn about recurrent neural networks, LSTMs, GRUs and
-Tensorflow's sequence-to-sequence capabilities.  To be able to read
-the core Tensorflow codebase.
-----
-====Deliverable:====
-For this lab, you will need to implement the `char-rnn` model of Karpathy.
-You will train it on a text corpus that you're interested in, and then
-show samples from the model.
-You should turn in your python code (not necessarily in a notebook, if you'd rather not), along
-with novel text samples showing that your code is working: first, after training on
-the provided "alma.txt" dataset, and second, samples from your
-network trained on a dataset of your choice.
-An example of my final samples are shown below (more detail in the
-final section of this writeup), after 150 passes through the data.
-Please generate about 15 samples for each dataset.
-<code>
-And ifte thin forgision forward thene over up to a fear not your
-And freitions, which is great God. Behold these are the loss sub
-And ache with the Lord hath bloes, which was done to the holy Gr
-And appeicis arm vinimonahites strong in name, to doth piseling
-And miniquithers these words, he commanded order not; neither sa
-And min for many would happine even to the earth, to said unto m
-And mie first be traditions? Behold, you, because it was a sound
-And from tike ended the Lamanites had administered, and I say bi
-</code>
-You are welcome to turn in your samples and your code separately.
-----
-====Grading standards:====
-Your code/image will be graded on the following:
-  * 20% Correct use of sequence encoder
-  * 20% Correct use of sequence decoder
-  * 20% Correct implementation of GRU cell
-  * 20% Correct sharing of variables
-  * 20% Training and sampling on a novel text dataset
-----
-====Description:====
-For this lab, you will code up the
-[[http://karpathy.github.io/2015/05/21/rnn-effectiveness/|char-rnn
-model of Karpathy]].  This is a recurrent neural network that is
-trained probabilistically on sequences of characters, and that can
-then be used to sample new sequences that are like the original.
-This lab will help you develop several new tensorflow skills, as well
-as understand some best practices needed for building large models.
-In addition, we'll be able to create networks that generate neat images!
-There are two parts of this lab: first, wiring up a basic
-sequence-to-sequence computation graph, and second, implementing your
-own GRU cell.
-To help you get started, you should download and look at the
-[[http://liftothers.org/byu/lab8_scaffold.tar.gz|Lab 8 scaffold code]]
-which contains a few extra goodies to help you: a TextLoader class,
-and some simple text files that you can use for debugging, and a
-little script called `switch.sh` that switches between text files.
-After unpacking the scaffold tarball, you should run switch.sh to select one of the provided training files:
-<code>
-./switch.sh alma.txt
-</code>
-----
-**Part 0: Creating a basic sequence-to-sequence model**
-Tensorflow has sequence modeling capabilities built in as a library.  There is a tutorial here
-[[https://www.tensorflow.org/versions/r0.11/tutorials/seq2seq/index.html|Tensorflow
-sequence-to-sequence tutorial]]
-However, I found the tutorial somewhat confusing.  Instead, I found it
-easier to read the relevant code; on my system, it was located at
-<code>
-/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/seq2seq.py
-</code>
-You should look at the `lab8_scaffold.py` code, where you see some
-helper code, outlining how to create your computation graph.  Most of
-the hard work of massaging inputs has already been done for you!
-For the first part of this lab, you should use the default LSTM cell
-included with Tensorflow.  Possibly like this:
-<code python>
-...
-from tensorflow.python.ops.rnn_cell import BasicLSTMCell
-cell = BasicLSTMCell( state_dim )
-...
-</code>
-Note that your computation graph will have a fixed `batch_size` --
-unfortunately, TF can't automatically scale that dimension for us.  :(
-----
-**Part 1: Sampling new sequences**
-Ok, great.  So we're training an RNN.  But how can we sample new text
-from it?
-The basic idea is to carefully execute one step at a time.  To show
-you how to do this, the scaffold code has a "sampler" method in it
-that feeds characters one at a time into a "sampler graph", tracking
-the state and sampling from the resulting probabilities.
-You are welcome to modify this code however you want.
-You need to create a separate computation graph for this sampling
-step.  It will look identical to your main computation graph (**and
-should share all of the same parameters!**), but it will be for a
-fixed "batch_size" of 1 and "sequence_length" of 1.
-If you name your variables appropriately, you should be able to just
-use my sampler function.
-----
-**Part 2: Creating your own GRU cell**
-The cell that you used in Part 0 was a pre-defined Tensorflow cell.
-The Tensorflow RNN API allows you to define your own cell by
-subclassing the `RNNCell` class.  For this part, you should create a
-custom GRU cell by creating a new class by filling in the following
-method:
-<code python>
-from tensorflow.python.ops.rnn_cell import RNNCell
-class mygru( RNNCell ):
-    def __init__( self, ??? ):
-    	pass
-    @property
-    def state_size(self):
-    	pass
-    @property
-    def output_size(self):
-    	pass
-    def __call__( self, inputs, state, scope=None ):
-    	pass
-</code>
-To see how this really works, it is helpful to look at the definitions
-of other cells, found (on my machine) in the file
-<code>
-/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn_cell.py
-</code>
-**However, please try not to look at the GRU cell definition.**
-That's right, the answer is right there, and in theory, you could
-just cut-and-paste it.  This bit is on your honor!
-----
-**Part 3: Run it and generate your final text!**
-Assuming everything has gone well, you should be able to run the main
-function in the scaffold code, using your custom GRU cell, and see
-output something like this.  I trained on the "alma.txt" dataset,
-using sequence_length=50, batch_size=50, and state_dim=128.
-<code>
-0     4.2465
-And wdyunNutg weotete foe At
- tharet mhg  tssaidatl atr
- eae to
-0     2.8634
-And utns, iugiin the cyel barthes hitaicen enthere ts sulod anta
-0     2.3854
-And gokse
- ne and thaseangang of the wesd unso he tait ewor besi
-0     2.2094
-And orto lhan I fome ho soy wort to come hyo deag copbeut.
-, N
-0     2.0746
-And kend Snowor to spae dil., hulth beemirss whar to wous thes m
-0     1.9668
-And ofolr yave preante for wa to.
- And thy it we be tho eprreno
-0     1.8655
-And an whis tretersJ of cil tony, now hat is snepin thee eos be
-0     1.7861
-And eomer shem; to thit tuscigh-whis porogw the reone uut whis i
-0     1.7170
-And bamonisery ald dad pasing their wich to ho ware great maviit
-0     1.6571
-And omen propre they wis pover upon of thy Lamanied agtoren moth
-0    1.6034
-And mantherefore, whiy well pofserse wo wo a ore thour paysien b
-0    1.5553
-And God of Nephe weat they welding oul hamy; and Amaming the Was
-0    1.5127
-And falling and is the my pithert to compiebion; thereforth, and
-0    1.4740
-And tothered of Nephites, and ristinvese accorangs was domlle ur
-0    1.4385
-And tire behinden to a sabe kean to past that chich me time: I a
-0    1.4072
-And to peepired farte; therefonch Chrathe in the kis iny? Moroni
-0    1.3782
-And wa righ, to ragking for it: I his brengmine med, and it your
-0    1.3519
-And and behold, and reptoregfor; who he kning:
- And recerited;
-0    1.3284
-And didnited upon the wam with is come words, that evong that no
-0    1.3051
-And ore their youndar oven the Lamanither hams, and low that ye
-0    1.2852
-And all; therefore ceme from to was ahore, in the city, and the
-0    1.2708
-And eforto Nephites, then; alanissuons or the, countom.
- And he
-0    1.2497
-And doth he pead, with did cities; and I to that it gath.
- And
-0    1.2330
-And keagition spanth which to man unto them. Wo and hear to that
-0    1.2121
-And to sent and having hopksect the awporit was waind of the lan
-0    1.1990
-And them a cit the sla; and behold is people that your land, tha
-0    1.1915
-And ans commack that they might upon their mand men by the naw a
-0    1.1767
-And mave the citer wat and surmed ox many of armon, the said thi
-0    1.1524
-And tremery believe Soviltent not their concitn of Teen to be pr
-0    1.1308
-And shim, the man unto it came to pass the rived to pose also; a
-0    1.1168
-And invaing the were have touslesssoon, and to our city of as te
-0    1.1086
-And bring.
- Now a quich, by fiel metry ming to the commandren t
-0    1.0965
-And from them in them, caused atgourness, my son, so, was war am
-0    1.0780
-And Staring: Moroni, and a say had bound from not have have foul
-0    1.0635
-And warp be not benong all thou hast dainthers, yea, which they
-0    1.0569
-And which I wore from the Spirits of our will come flact.
- Now
-0    1.0491
-And werch ragets was etred which cast unto his armor of Amulek,
-0    1.0445
-And tought by owfs the hoig wirt lay hinds one their brethar, bu
-0    1.0294
-And word bring, and messer, now you, on the namb it come all one
-0    1.0074
-And against fantitions of Sidon, even to empt thises to sing man
-0    0.9927
-And wither the people which is fear ago not who thou hastinushs
-0    0.9840
-And not agam and armies for the words of Christ, inquer.
- For b
-0    0.9749
-And wish spake tightiston slain who trought a privitions and sup
-0    0.9747
-And great as the eviek of the repturely from, and the people of
-0    0.9650
-And might they mickoning thick them rest exceedingly Lamoni land
-0    0.9476
-And is the hapthers?
- And ye was as my preacherce and of the li
-0    0.9373
-And might God, but now end their first, and according to to fait
-0    0.9230
-And that the Lamanites Lamanites for the words of Milshent up im
-0    0.9183
-And given to the king; then fore this givened that reternets to
-0    0.9082
-And thit had slaked all men what had gathere, which bringeded hi
-0    0.9186
-And wish power that ye shall pray, all this cursions a can to th
-0    0.9022
-And not giving with dimsed were trader, and place unto you.
- Be
-0    0.8966
-And never of the Lord; and he repentate of the sinces, whoughter
-0    0.8917
-And being the provored to the churchder.
- And that we should no
-0    0.8790
-And indachers more congen.
- And they were define among the peop
-0    0.8667
-And him he might sayed the Lord from the cry to repent deliver,
-0    0.8601
-And might on the nint; and these chuncrusios of their hearts; ne
-0    0.8604
-And begirish--Zerahomnant the wide upon the Son of uster you, ha
-0    0.8481
-And to comeans, therefore they slougness for this great faith hi
-0    0.8438
-And who we had lid hath mansh, yea; and they twey swolling but f
-0    0.8352
-And again they did comvant of the people of pony city of Nephiha
-0    0.8333
-And who with piscorst.
- Now bon to take fallerant these wond as
-0    0.8206
-And poftor, they did take up rich unto him: I commencement food
-0    0.8202
-And to passed them.
- And now, do not a shange a string, ye lony
-0    0.8148
-And befiel Requestion, but that therefore God Moroni, and their
-0    0.8107
-And their hapfenembly as done might not great, yea, and the land
-0    0.8024
-And with the king would no romp not and he wilterness of God, fo
-0    0.7942
-And for that it was not doot.
- And now it came to pass te his k
-0    0.7862
-And buch against the church in the power of God; therefore I hav
-0    0.7872
-And were sins--the ether came pertays puran to wax liget.
- Cond
-0    0.7935
-And frich mestireded to the Zoramites came of the crues had come
-0    0.8013
-And to pass, the Lord sit ad not blood; yea, this ye who will be
-0    0.7764
-And was given for the eart no men; and this was the word of Neph
-0    0.7634
-And free over the chief judge our implaties appointited from the
-0    0.7532
-And from himsmed, that the liveth by the mins and prisoner, and
-0    0.7604
-And hazrmen and phourth Aaron possession of the blessed against
-0    0.7584
-And buither many which and dilled that his people; whose the eas
-0    0.7512
-And whith him that he also smitting of the fled over working man
-0    0.7417
-And are not vost number of Care to the eard be ye think possessi
-0    0.7434
-And freich which that was not have aboume of our brethren, that
-0    0.7329
-And arw these after them in fone Ishlat in a happen up and take
-0    0.7394
-And with bring of his remainder and sent Curean by none wats, O
-0    0.7304
-And mind animed among them towarsh recommence of their coursend-
-0    0.7285
-And is them, and the wird Antipus sawe thy loft understands rete
-0    0.7160
-And mite thy are mottlo, yea, except they murnogeffers preparati
-0    0.7093
-And being had obear.
- Nevertheless, they were past into those c
-0    0.7045
-And friont pervest should cut out God.
- Now the consop gave the
-0    0.7014
-And with him; for know their people, become, and God the Lamanit
-0    0.7094
-And mind a. the things salver the command of their slave upon th
-0    0.6970
-And with his propheti, Ammon was a many thy fulfied after the Sp
-0    0.6904
-And havid; and thus beginn as plavellwary misenaged, saying, com
-0    0.6901
-And with their lives; and behold, jestinity. Now, let forth year
-0    0.6868
-And awau hard he swell these things pertaingd Middoni had set en
-0    0.6855
-And are care over the people of Nephi-Lehi, which he did jush as
-0    0.6819
-And freidded my son.
- And now, I say unto Zarahemla.
- Comfalal
-0    0.6808
-And pewer to place in Christ, but believe.
- Seet the last dain,
-0    0.6670
-And bite them they were men, that Lehon, as I say unto him: What
-0    0.6642
-And him did that they had saw this children by Teancums and the
-0    0.6643
-And minam id ye beginged with the armies of the Lamanites was th
-0    0.6613
-And which had said that retain, ye begliving among all manner ov
-0   0.6599
-And age is very cones power in the sword.
- Therefore thou wilt
-0   0.6740
-And praties of Moroni him about.
- And now these askengeh exerci
-0   0.6613
-And are brought torded fordate werf and shedding unto repentance
-0   0.6589
-And awe the works; and this man of plessed under the manst beste
-0   0.6489
-And from preserve them by the people to small their teother: I a
-0   0.6448
-And awai had great nardening the wickedness and the pride of the
-0   0.6427
-And bring when his bend of the flocks say unto his brethren may
-0   0.6378
-And firitious armide me stand give hast, saying: Telt redeem; be
-0   0.6317
-And are murders shall be angel because of given up just in their
-0   0.6341
-And about, which arthy forth Laman servants, brought up at a str
-0   0.6202
-And abority befold, this believer his laid cosumbly to executed
-0   0.6182
-And are.
- And now Nephi.
- I would and men bar delighed, that w
-0   0.6216
-And freites, and be brought out of the land of Zarahemla. Ned th
-0   0.6273
-And and Said hath not teot were cried--also be delivered by our
-0   0.6127
-And and time, their swords, and all the borders which laves amon
-0   0.6112
-And abehor is hearts of their long shord; yea, and also that it
-0   0.6051
-And awe this began to command him goreven; therefore there was I
-0   0.6054
-And into-tright, Zenamewhered abepting denis; and silver, they w
-0   0.6030
-And into the knewsed, and ablieve thing call, with him; for we w
-0   0.6028
-And and did this the judge Ammon, and his fathers year all our p
-0   0.6024
-And angifes; therefore they had fight and a sword, and there is
-0   0.5996
-And abie it supposed to deliver them.
- And it came to pass that
-0   0.6005
-And bith a give as be saved--is to rememe far insteovert known u
-0   0.5872
-And fapis of its; and also deny; their camp with him, and had de
-0   0.5854
-And into with his inuase men, on hercite watkennes of perime!
-0   0.5817
-And accord him suppose that there was ascomel not to hear; for t
-0   0.5809
-And accien Siedgmentes of his brethren, which camp words his gre
-0   0.5765
-And awe this never his walls powers by the loss whereolah.
- Ant
-0   0.5747
-And eatek in there spiritted by mine say, because they had been
-0   0.5714
-And ir thirst; therefore a by your wives again; therefore he sai
-0   0.5662
-And enject his hand, they gathered tomsor; and he said unto men,
-0   0.5668
-And awain the coming of Noahor hath he said: Yea, they did not d
-0   0.5687
-And gid tike up their God.
- And it came to pass that the man? W
-0   0.5634
-And aid the Nephites were spneed, and his brethren done of I hav
-0   0.5688
-And appeace the endemed with inhelvist destroy things know the l
-0   0.5628
-And afpiining this holy according to to learness before the Lama
-0   0.5638
-And accircures up their house to bring my buriaged which had int
-0   0.5534
-And freit. And it came to pass thas when Moroni had same pointen
-0   0.5485
-And firing to pass the Lord sixtt them.
- And now, behold, O the
-0   0.5502
-And awau Ziesout had good were seniat in my righteok; and they a
-0   0.5409
-And accien hampleate him, as I have some his dettrore to maintai
-0   0.5462
-And firen brethren, be relepor of king-men) sore wither every di
-0   0.5383
-And misins who were pleasated against the record rembant and the
-0   0.5419
-And iffinish to bahy tranch out of the land whatsoever no knowle
-0   0.5335
-And iffling their things, faith from the commencement rejoining
-0   0.5355
-And great pure good and to termy, in with their tenth in the pla
-0   0.5282
-And priesting him that might destroy their myselfings, insomperi
-0   0.5257
-And from with his ower of Pahoran, came over which they anger ag
-0   0.5308
-And friam timperisted against the man who had no power of whom t
-0   0.5230
-And acciritishe it corder, having fied against them by those men
-0   0.5254
-And cie tiand of joy, as the west to the power.
- And it came to
-0   0.5281
-And awind him that my brethren speak unto him forward frighteent
-0   0.5270
-And appeice when he led to back equace of for God, or by the Lam
-0   0.5190
-And afte it hat his preserving the plan eviltont of the hearts o
-0   0.5106
-And atininiful that there should death, and also me cometh, and
-0   0.5162
-And ifte the sword, who seon and disperibee unto me, shook feed
-0   0.5143
-And mitiquity mind, and according to the chief judge, so make at
-0   0.5192
-And ifte thin forgision forward thene over up to a fear not your
-0   0.5097
-And freitions, which is great God. Behold these are the loss sub
-0   0.5079
-And ache with the Lord hath bloes, which was done to the holy Gr
-0   0.5074
-And appeicis arm vinimonahites strong in name, to doth piseling
-0   0.5073
-And miniquithers these words, he commanded order not; neither sa
-0   0.5030
-And min for many would happine even to the earth, to said unto m
-0   0.5053
-And mie first be traditions? Behold, you, because it was a sound
-0   0.4915
-And from tike ended the Lamanites had administered, and I say bi
-</code>

BYU CS classes

User Tools

Site Tools

Differences

Page Tools