To learn about recurrent neural networks, LSTMs, GRUs and Tensorflow's sequence-to-sequence capabilities. To be able to read the core Tensorflow codebase.
For this lab, you will need to implement the `char-rnn` model of Karpathy. You will train it on a text corpus that you're interested in, and then show samples from the model.
You should turn in your python code (not necessarily in a notebook, if you'd rather not), along with novel text samples showing that your code is working: first, after training on the provided “alma.txt” dataset, and second, samples from your network trained on a dataset of your choice.
An example of my final samples are shown below (more detail in the final section of this writeup), after 150 passes through the data. Please generate about 15 samples for each dataset.
And ifte thin forgision forward thene over up to a fear not your And freitions, which is great God. Behold these are the loss sub And ache with the Lord hath bloes, which was done to the holy Gr And appeicis arm vinimonahites strong in name, to doth piseling And miniquithers these words, he commanded order not; neither sa And min for many would happine even to the earth, to said unto m And mie first be traditions? Behold, you, because it was a sound And from tike ended the Lamanites had administered, and I say bi
You are welcome to turn in your samples and your code separately.
Your code/image will be graded on the following:
For this lab, you will code up the char-rnn model of Karpathy. This is a recurrent neural network that is trained probabilistically on sequences of characters, and that can then be used to sample new sequences that are like the original.
This lab will help you develop several new tensorflow skills, as well as understand some best practices needed for building large models. In addition, we'll be able to create networks that generate neat text!
There are two parts of this lab: first, wiring up a basic sequence-to-sequence computation graph, and second, implementing your own GRU cell.
To help you get started, you should download and look at the
which contains a few extra goodies to help you: a TextLoader class, and some simple text files that you can use for debugging, and a little script called `switch.sh` that switches between text files.
After unpacking the scaffold tarball, you should run switch.sh to select one of the provided training files:
Part 0: Creating a basic sequence-to-sequence model
Tensorflow has sequence modeling capabilities built in as a library. There is a tutorial here
However, I found the tutorial somewhat confusing. Instead, I found it easier to read the relevant code; on my system, it was located at
You should look at the `lab8_scaffold.py` code, where you see some helper code, outlining how to create your computation graph. Most of the hard work of massaging inputs has already been done for you!
For the first part of this lab, you should use the default LSTM cell included with Tensorflow. Possibly like this:
... from tensorflow.python.ops.rnn_cell import BasicLSTMCell cell = BasicLSTMCell( state_dim ) ...
Note that your computation graph will have a fixed `batch_size` – unfortunately, TF can't automatically scale that dimension for us. :(
Part 1: Sampling new sequences
Ok, great. So we're training an RNN. But how can we sample new text from it?
The basic idea is to carefully execute one step at a time. To show you how to do this, the scaffold code has a “sampler” method in it that feeds characters one at a time into a “sampler graph”, tracking the state and sampling from the resulting probabilities.
You are welcome to modify this code however you want.
You need to create a separate computation graph for this sampling step. It will look identical to your main computation graph (and should share all of the same parameters!), but it will be for a fixed “batch_size” of 1 and “sequence_length” of 1.
If you name your variables appropriately, you should be able to just use my sampler function.
Part 2: Creating your own GRU cell
The cell that you used in Part 0 was a pre-defined Tensorflow cell. The Tensorflow RNN API allows you to define your own cell by subclassing the `RNNCell` class. For this part, you should create a custom GRU cell by creating a new class by filling in the following method:
from tensorflow.python.ops.rnn_cell import RNNCell class mygru( RNNCell ): def __init__( self, ??? ): pass @property def state_size(self): pass @property def output_size(self): pass def __call__( self, inputs, state, scope=None ): pass
To see how this really works, it is helpful to look at the definitions of other cells, found (on my machine) in the file
However, please try not to look at the GRU cell definition. That's right, the answer is right there, and in theory, you could just cut-and-paste it. This bit is on your honor!
Part 3: Run it and generate your final text!
Assuming everything has gone well, you should be able to run the main function in the scaffold code, using your custom GRU cell, and see output something like this. I trained on the “alma.txt” dataset, using sequence_length=50, batch_size=50, and state_dim=128.
0 0 4.2465 And wdyunNutg weotete foe At tharet mhg tssaidatl atr eae to 1 0 2.8634 And utns, iugiin the cyel barthes hitaicen enthere ts sulod anta 2 0 2.3854 And gokse ne and thaseangang of the wesd unso he tait ewor besi 3 0 2.2094 And orto lhan I fome ho soy wort to come hyo deag copbeut. , N 4 0 2.0746 And kend Snowor to spae dil., hulth beemirss whar to wous thes m 5 0 1.9668 And ofolr yave preante for wa to. And thy it we be tho eprreno 6 0 1.8655 And an whis tretersJ of cil tony, now hat is snepin thee eos be 7 0 1.7861 And eomer shem; to thit tuscigh-whis porogw the reone uut whis i 8 0 1.7170 And bamonisery ald dad pasing their wich to ho ware great maviit 9 0 1.6571 And omen propre they wis pover upon of thy Lamanied agtoren moth 10 0 1.6034 And mantherefore, whiy well pofserse wo wo a ore thour paysien b 11 0 1.5553 And God of Nephe weat they welding oul hamy; and Amaming the Was 12 0 1.5127 And falling and is the my pithert to compiebion; thereforth, and 13 0 1.4740 And tothered of Nephites, and ristinvese accorangs was domlle ur 14 0 1.4385 And tire behinden to a sabe kean to past that chich me time: I a 15 0 1.4072 And to peepired farte; therefonch Chrathe in the kis iny? Moroni 16 0 1.3782 And wa righ, to ragking for it: I his brengmine med, and it your 17 0 1.3519 And and behold, and reptoregfor; who he kning: And recerited; 18 0 1.3284 And didnited upon the wam with is come words, that evong that no 19 0 1.3051 And ore their youndar oven the Lamanither hams, and low that ye 20 0 1.2852 And all; therefore ceme from to was ahore, in the city, and the 21 0 1.2708 And eforto Nephites, then; alanissuons or the, countom. And he 22 0 1.2497 And doth he pead, with did cities; and I to that it gath. And 23 0 1.2330 And keagition spanth which to man unto them. Wo and hear to that 24 0 1.2121 And to sent and having hopksect the awporit was waind of the lan 25 0 1.1990 And them a cit the sla; and behold is people that your land, tha 26 0 1.1915 And ans commack that they might upon their mand men by the naw a 27 0 1.1767 And mave the citer wat and surmed ox many of armon, the said thi 28 0 1.1524 And tremery believe Soviltent not their concitn of Teen to be pr 29 0 1.1308 And shim, the man unto it came to pass the rived to pose also; a 30 0 1.1168 And invaing the were have touslesssoon, and to our city of as te 31 0 1.1086 And bring. Now a quich, by fiel metry ming to the commandren t 32 0 1.0965 And from them in them, caused atgourness, my son, so, was war am 33 0 1.0780 And Staring: Moroni, and a say had bound from not have have foul 34 0 1.0635 And warp be not benong all thou hast dainthers, yea, which they 35 0 1.0569 And which I wore from the Spirits of our will come flact. Now 36 0 1.0491 And werch ragets was etred which cast unto his armor of Amulek, 37 0 1.0445 And tought by owfs the hoig wirt lay hinds one their brethar, bu 38 0 1.0294 And word bring, and messer, now you, on the namb it come all one 39 0 1.0074 And against fantitions of Sidon, even to empt thises to sing man 40 0 0.9927 And wither the people which is fear ago not who thou hastinushs 41 0 0.9840 And not agam and armies for the words of Christ, inquer. For b 42 0 0.9749 And wish spake tightiston slain who trought a privitions and sup 43 0 0.9747 And great as the eviek of the repturely from, and the people of 44 0 0.9650 And might they mickoning thick them rest exceedingly Lamoni land 45 0 0.9476 And is the hapthers? And ye was as my preacherce and of the li 46 0 0.9373 And might God, but now end their first, and according to to fait 47 0 0.9230 And that the Lamanites Lamanites for the words of Milshent up im 48 0 0.9183 And given to the king; then fore this givened that reternets to 49 0 0.9082 And thit had slaked all men what had gathere, which bringeded hi 50 0 0.9186 And wish power that ye shall pray, all this cursions a can to th 51 0 0.9022 And not giving with dimsed were trader, and place unto you. Be 52 0 0.8966 And never of the Lord; and he repentate of the sinces, whoughter 53 0 0.8917 And being the provored to the churchder. And that we should no 54 0 0.8790 And indachers more congen. And they were define among the peop 55 0 0.8667 And him he might sayed the Lord from the cry to repent deliver, 56 0 0.8601 And might on the nint; and these chuncrusios of their hearts; ne 57 0 0.8604 And begirish--Zerahomnant the wide upon the Son of uster you, ha 58 0 0.8481 And to comeans, therefore they slougness for this great faith hi 59 0 0.8438 And who we had lid hath mansh, yea; and they twey swolling but f 60 0 0.8352 And again they did comvant of the people of pony city of Nephiha 61 0 0.8333 And who with piscorst. Now bon to take fallerant these wond as 62 0 0.8206 And poftor, they did take up rich unto him: I commencement food 63 0 0.8202 And to passed them. And now, do not a shange a string, ye lony 64 0 0.8148 And befiel Requestion, but that therefore God Moroni, and their 65 0 0.8107 And their hapfenembly as done might not great, yea, and the land 66 0 0.8024 And with the king would no romp not and he wilterness of God, fo 67 0 0.7942 And for that it was not doot. And now it came to pass te his k 68 0 0.7862 And buch against the church in the power of God; therefore I hav 69 0 0.7872 And were sins--the ether came pertays puran to wax liget. Cond 70 0 0.7935 And frich mestireded to the Zoramites came of the crues had come 71 0 0.8013 And to pass, the Lord sit ad not blood; yea, this ye who will be 72 0 0.7764 And was given for the eart no men; and this was the word of Neph 73 0 0.7634 And free over the chief judge our implaties appointited from the 74 0 0.7532 And from himsmed, that the liveth by the mins and prisoner, and 75 0 0.7604 And hazrmen and phourth Aaron possession of the blessed against 76 0 0.7584 And buither many which and dilled that his people; whose the eas 77 0 0.7512 And whith him that he also smitting of the fled over working man 78 0 0.7417 And are not vost number of Care to the eard be ye think possessi 79 0 0.7434 And freich which that was not have aboume of our brethren, that 80 0 0.7329 And arw these after them in fone Ishlat in a happen up and take 81 0 0.7394 And with bring of his remainder and sent Curean by none wats, O 82 0 0.7304 And mind animed among them towarsh recommence of their coursend- 83 0 0.7285 And is them, and the wird Antipus sawe thy loft understands rete 84 0 0.7160 And mite thy are mottlo, yea, except they murnogeffers preparati 85 0 0.7093 And being had obear. Nevertheless, they were past into those c 86 0 0.7045 And friont pervest should cut out God. Now the consop gave the 87 0 0.7014 And with him; for know their people, become, and God the Lamanit 88 0 0.7094 And mind a. the things salver the command of their slave upon th 89 0 0.6970 And with his propheti, Ammon was a many thy fulfied after the Sp 90 0 0.6904 And havid; and thus beginn as plavellwary misenaged, saying, com 91 0 0.6901 And with their lives; and behold, jestinity. Now, let forth year 92 0 0.6868 And awau hard he swell these things pertaingd Middoni had set en 93 0 0.6855 And are care over the people of Nephi-Lehi, which he did jush as 94 0 0.6819 And freidded my son. And now, I say unto Zarahemla. Comfalal 95 0 0.6808 And pewer to place in Christ, but believe. Seet the last dain, 96 0 0.6670 And bite them they were men, that Lehon, as I say unto him: What 97 0 0.6642 And him did that they had saw this children by Teancums and the 98 0 0.6643 And minam id ye beginged with the armies of the Lamanites was th 99 0 0.6613 And which had said that retain, ye begliving among all manner ov 100 0 0.6599 And age is very cones power in the sword. Therefore thou wilt 101 0 0.6740 And praties of Moroni him about. And now these askengeh exerci 102 0 0.6613 And are brought torded fordate werf and shedding unto repentance 103 0 0.6589 And awe the works; and this man of plessed under the manst beste 104 0 0.6489 And from preserve them by the people to small their teother: I a 105 0 0.6448 And awai had great nardening the wickedness and the pride of the 106 0 0.6427 And bring when his bend of the flocks say unto his brethren may 107 0 0.6378 And firitious armide me stand give hast, saying: Telt redeem; be 108 0 0.6317 And are murders shall be angel because of given up just in their 109 0 0.6341 And about, which arthy forth Laman servants, brought up at a str 110 0 0.6202 And abority befold, this believer his laid cosumbly to executed 111 0 0.6182 And are. And now Nephi. I would and men bar delighed, that w 112 0 0.6216 And freites, and be brought out of the land of Zarahemla. Ned th 113 0 0.6273 And and Said hath not teot were cried--also be delivered by our 114 0 0.6127 And and time, their swords, and all the borders which laves amon 115 0 0.6112 And abehor is hearts of their long shord; yea, and also that it 116 0 0.6051 And awe this began to command him goreven; therefore there was I 117 0 0.6054 And into-tright, Zenamewhered abepting denis; and silver, they w 118 0 0.6030 And into the knewsed, and ablieve thing call, with him; for we w 119 0 0.6028 And and did this the judge Ammon, and his fathers year all our p 120 0 0.6024 And angifes; therefore they had fight and a sword, and there is 121 0 0.5996 And abie it supposed to deliver them. And it came to pass that 122 0 0.6005 And bith a give as be saved--is to rememe far insteovert known u 123 0 0.5872 And fapis of its; and also deny; their camp with him, and had de 124 0 0.5854 And into with his inuase men, on hercite watkennes of perime! 125 0 0.5817 And accord him suppose that there was ascomel not to hear; for t 126 0 0.5809 And accien Siedgmentes of his brethren, which camp words his gre 127 0 0.5765 And awe this never his walls powers by the loss whereolah. Ant 128 0 0.5747 And eatek in there spiritted by mine say, because they had been 129 0 0.5714 And ir thirst; therefore a by your wives again; therefore he sai 130 0 0.5662 And enject his hand, they gathered tomsor; and he said unto men, 131 0 0.5668 And awain the coming of Noahor hath he said: Yea, they did not d 132 0 0.5687 And gid tike up their God. And it came to pass that the man? W 133 0 0.5634 And aid the Nephites were spneed, and his brethren done of I hav 134 0 0.5688 And appeace the endemed with inhelvist destroy things know the l 135 0 0.5628 And afpiining this holy according to to learness before the Lama 136 0 0.5638 And accircures up their house to bring my buriaged which had int 137 0 0.5534 And freit. And it came to pass thas when Moroni had same pointen 138 0 0.5485 And firing to pass the Lord sixtt them. And now, behold, O the 139 0 0.5502 And awau Ziesout had good were seniat in my righteok; and they a 140 0 0.5409 And accien hampleate him, as I have some his dettrore to maintai 141 0 0.5462 And firen brethren, be relepor of king-men) sore wither every di 142 0 0.5383 And misins who were pleasated against the record rembant and the 143 0 0.5419 And iffinish to bahy tranch out of the land whatsoever no knowle 144 0 0.5335 And iffling their things, faith from the commencement rejoining 145 0 0.5355 And great pure good and to termy, in with their tenth in the pla 146 0 0.5282 And priesting him that might destroy their myselfings, insomperi 147 0 0.5257 And from with his ower of Pahoran, came over which they anger ag 148 0 0.5308 And friam timperisted against the man who had no power of whom t 149 0 0.5230 And acciritishe it corder, having fied against them by those men 150 0 0.5254 And cie tiand of joy, as the west to the power. And it came to 151 0 0.5281 And awind him that my brethren speak unto him forward frighteent 152 0 0.5270 And appeice when he led to back equace of for God, or by the Lam 153 0 0.5190 And afte it hat his preserving the plan eviltont of the hearts o 154 0 0.5106 And atininiful that there should death, and also me cometh, and 155 0 0.5162 And ifte the sword, who seon and disperibee unto me, shook feed 156 0 0.5143 And mitiquity mind, and according to the chief judge, so make at 157 0 0.5192 And ifte thin forgision forward thene over up to a fear not your 158 0 0.5097 And freitions, which is great God. Behold these are the loss sub 159 0 0.5079 And ache with the Lord hath bloes, which was done to the holy Gr 160 0 0.5074 And appeicis arm vinimonahites strong in name, to doth piseling 161 0 0.5073 And miniquithers these words, he commanded order not; neither sa 162 0 0.5030 And min for many would happine even to the earth, to said unto m 163 0 0.5053 And mie first be traditions? Behold, you, because it was a sound 164 0 0.4915 And from tike ended the Lamanites had administered, and I say bi