====Objective:====
To learn about recurrent neural networks, LSTMs, GRUs and
Tensorflow's sequence-to-sequence capabilities. To be able to read
the core Tensorflow codebase.
----
====Deliverable:====
For this lab, you will need to implement the `char-rnn` model of Karpathy.
You will train it on a text corpus that you're interested in, and then
show samples from the model.
You should turn in your python code (not necessarily in a notebook, if you'd rather not), along
with novel text samples showing that your code is working: first, after training on
the provided "alma.txt" dataset, and second, samples from your
network trained on a dataset of your choice.
An example of my final samples are shown below (more detail in the
final section of this writeup), after 150 passes through the data.
Please generate about 15 samples for each dataset.
And ifte thin forgision forward thene over up to a fear not your
And freitions, which is great God. Behold these are the loss sub
And ache with the Lord hath bloes, which was done to the holy Gr
And appeicis arm vinimonahites strong in name, to doth piseling
And miniquithers these words, he commanded order not; neither sa
And min for many would happine even to the earth, to said unto m
And mie first be traditions? Behold, you, because it was a sound
And from tike ended the Lamanites had administered, and I say bi
You are welcome to turn in your samples and your code separately.
----
====Grading standards:====
Your code/image will be graded on the following:
* 20% Correct use of sequence encoder
* 20% Correct use of sequence decoder
* 20% Correct implementation of GRU cell
* 20% Correct sharing of variables
* 20% Training and sampling on a novel text dataset
----
====Description:====
For this lab, you will code up the
[[http://karpathy.github.io/2015/05/21/rnn-effectiveness/|char-rnn
model of Karpathy]]. This is a recurrent neural network that is
trained probabilistically on sequences of characters, and that can
then be used to sample new sequences that are like the original.
This lab will help you develop several new tensorflow skills, as well
as understand some best practices needed for building large models.
In addition, we'll be able to create networks that generate neat text!
There are two parts of this lab: first, wiring up a basic
sequence-to-sequence computation graph, and second, implementing your
own GRU cell.
To help you get started, you should download and look at the
[[http://liftothers.org/byu/lab8_scaffold.tar.gz|Lab 8 scaffold code]]
which contains a few extra goodies to help you: a TextLoader class,
and some simple text files that you can use for debugging, and a
little script called `switch.sh` that switches between text files.
After unpacking the scaffold tarball, you should run switch.sh to select one of the provided training files:
./switch.sh alma.txt
----
**Part 0: Creating a basic sequence-to-sequence model**
Tensorflow has sequence modeling capabilities built in as a library. There is a tutorial here
[[https://www.tensorflow.org/versions/r0.11/tutorials/seq2seq/index.html|Tensorflow
sequence-to-sequence tutorial]]
However, I found the tutorial somewhat confusing. Instead, I found it
easier to read the relevant code; on my system, it was located at
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/seq2seq.py
You should look at the `lab8_scaffold.py` code, where you see some
helper code, outlining how to create your computation graph. Most of
the hard work of massaging inputs has already been done for you!
For the first part of this lab, you should use the default LSTM cell
included with Tensorflow. Possibly like this:
...
from tensorflow.python.ops.rnn_cell import BasicLSTMCell
cell = BasicLSTMCell( state_dim )
...
Note that your computation graph will have a fixed `batch_size` --
unfortunately, TF can't automatically scale that dimension for us. :(
----
**Part 1: Sampling new sequences**
Ok, great. So we're training an RNN. But how can we sample new text
from it?
The basic idea is to carefully execute one step at a time. To show
you how to do this, the scaffold code has a "sampler" method in it
that feeds characters one at a time into a "sampler graph", tracking
the state and sampling from the resulting probabilities.
You are welcome to modify this code however you want.
You need to create a separate computation graph for this sampling
step. It will look identical to your main computation graph (**and
should share all of the same parameters!**), but it will be for a
fixed "batch_size" of 1 and "sequence_length" of 1.
If you name your variables appropriately, you should be able to just
use my sampler function.
----
**Part 2: Creating your own GRU cell**
The cell that you used in Part 0 was a pre-defined Tensorflow cell.
The Tensorflow RNN API allows you to define your own cell by
subclassing the `RNNCell` class. For this part, you should create a
custom GRU cell by creating a new class by filling in the following
method:
from tensorflow.python.ops.rnn_cell import RNNCell
class mygru( RNNCell ):
def __init__( self, ??? ):
pass
@property
def state_size(self):
pass
@property
def output_size(self):
pass
def __call__( self, inputs, state, scope=None ):
pass
To see how this really works, it is helpful to look at the definitions
of other cells, found (on my machine) in the file
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/rnn_cell.py
**However, please try not to look at the GRU cell definition.**
That's right, the answer is right there, and in theory, you could
just cut-and-paste it. This bit is on your honor!
----
**Part 3: Run it and generate your final text!**
Assuming everything has gone well, you should be able to run the main
function in the scaffold code, using your custom GRU cell, and see
output something like this. I trained on the "alma.txt" dataset,
using sequence_length=50, batch_size=50, and state_dim=128.
0 0 4.2465
And wdyunNutg weotete foe At
tharet mhg tssaidatl atr
eae to
1 0 2.8634
And utns, iugiin the cyel barthes hitaicen enthere ts sulod anta
2 0 2.3854
And gokse
ne and thaseangang of the wesd unso he tait ewor besi
3 0 2.2094
And orto lhan I fome ho soy wort to come hyo deag copbeut.
, N
4 0 2.0746
And kend Snowor to spae dil., hulth beemirss whar to wous thes m
5 0 1.9668
And ofolr yave preante for wa to.
And thy it we be tho eprreno
6 0 1.8655
And an whis tretersJ of cil tony, now hat is snepin thee eos be
7 0 1.7861
And eomer shem; to thit tuscigh-whis porogw the reone uut whis i
8 0 1.7170
And bamonisery ald dad pasing their wich to ho ware great maviit
9 0 1.6571
And omen propre they wis pover upon of thy Lamanied agtoren moth
10 0 1.6034
And mantherefore, whiy well pofserse wo wo a ore thour paysien b
11 0 1.5553
And God of Nephe weat they welding oul hamy; and Amaming the Was
12 0 1.5127
And falling and is the my pithert to compiebion; thereforth, and
13 0 1.4740
And tothered of Nephites, and ristinvese accorangs was domlle ur
14 0 1.4385
And tire behinden to a sabe kean to past that chich me time: I a
15 0 1.4072
And to peepired farte; therefonch Chrathe in the kis iny? Moroni
16 0 1.3782
And wa righ, to ragking for it: I his brengmine med, and it your
17 0 1.3519
And and behold, and reptoregfor; who he kning:
And recerited;
18 0 1.3284
And didnited upon the wam with is come words, that evong that no
19 0 1.3051
And ore their youndar oven the Lamanither hams, and low that ye
20 0 1.2852
And all; therefore ceme from to was ahore, in the city, and the
21 0 1.2708
And eforto Nephites, then; alanissuons or the, countom.
And he
22 0 1.2497
And doth he pead, with did cities; and I to that it gath.
And
23 0 1.2330
And keagition spanth which to man unto them. Wo and hear to that
24 0 1.2121
And to sent and having hopksect the awporit was waind of the lan
25 0 1.1990
And them a cit the sla; and behold is people that your land, tha
26 0 1.1915
And ans commack that they might upon their mand men by the naw a
27 0 1.1767
And mave the citer wat and surmed ox many of armon, the said thi
28 0 1.1524
And tremery believe Soviltent not their concitn of Teen to be pr
29 0 1.1308
And shim, the man unto it came to pass the rived to pose also; a
30 0 1.1168
And invaing the were have touslesssoon, and to our city of as te
31 0 1.1086
And bring.
Now a quich, by fiel metry ming to the commandren t
32 0 1.0965
And from them in them, caused atgourness, my son, so, was war am
33 0 1.0780
And Staring: Moroni, and a say had bound from not have have foul
34 0 1.0635
And warp be not benong all thou hast dainthers, yea, which they
35 0 1.0569
And which I wore from the Spirits of our will come flact.
Now
36 0 1.0491
And werch ragets was etred which cast unto his armor of Amulek,
37 0 1.0445
And tought by owfs the hoig wirt lay hinds one their brethar, bu
38 0 1.0294
And word bring, and messer, now you, on the namb it come all one
39 0 1.0074
And against fantitions of Sidon, even to empt thises to sing man
40 0 0.9927
And wither the people which is fear ago not who thou hastinushs
41 0 0.9840
And not agam and armies for the words of Christ, inquer.
For b
42 0 0.9749
And wish spake tightiston slain who trought a privitions and sup
43 0 0.9747
And great as the eviek of the repturely from, and the people of
44 0 0.9650
And might they mickoning thick them rest exceedingly Lamoni land
45 0 0.9476
And is the hapthers?
And ye was as my preacherce and of the li
46 0 0.9373
And might God, but now end their first, and according to to fait
47 0 0.9230
And that the Lamanites Lamanites for the words of Milshent up im
48 0 0.9183
And given to the king; then fore this givened that reternets to
49 0 0.9082
And thit had slaked all men what had gathere, which bringeded hi
50 0 0.9186
And wish power that ye shall pray, all this cursions a can to th
51 0 0.9022
And not giving with dimsed were trader, and place unto you.
Be
52 0 0.8966
And never of the Lord; and he repentate of the sinces, whoughter
53 0 0.8917
And being the provored to the churchder.
And that we should no
54 0 0.8790
And indachers more congen.
And they were define among the peop
55 0 0.8667
And him he might sayed the Lord from the cry to repent deliver,
56 0 0.8601
And might on the nint; and these chuncrusios of their hearts; ne
57 0 0.8604
And begirish--Zerahomnant the wide upon the Son of uster you, ha
58 0 0.8481
And to comeans, therefore they slougness for this great faith hi
59 0 0.8438
And who we had lid hath mansh, yea; and they twey swolling but f
60 0 0.8352
And again they did comvant of the people of pony city of Nephiha
61 0 0.8333
And who with piscorst.
Now bon to take fallerant these wond as
62 0 0.8206
And poftor, they did take up rich unto him: I commencement food
63 0 0.8202
And to passed them.
And now, do not a shange a string, ye lony
64 0 0.8148
And befiel Requestion, but that therefore God Moroni, and their
65 0 0.8107
And their hapfenembly as done might not great, yea, and the land
66 0 0.8024
And with the king would no romp not and he wilterness of God, fo
67 0 0.7942
And for that it was not doot.
And now it came to pass te his k
68 0 0.7862
And buch against the church in the power of God; therefore I hav
69 0 0.7872
And were sins--the ether came pertays puran to wax liget.
Cond
70 0 0.7935
And frich mestireded to the Zoramites came of the crues had come
71 0 0.8013
And to pass, the Lord sit ad not blood; yea, this ye who will be
72 0 0.7764
And was given for the eart no men; and this was the word of Neph
73 0 0.7634
And free over the chief judge our implaties appointited from the
74 0 0.7532
And from himsmed, that the liveth by the mins and prisoner, and
75 0 0.7604
And hazrmen and phourth Aaron possession of the blessed against
76 0 0.7584
And buither many which and dilled that his people; whose the eas
77 0 0.7512
And whith him that he also smitting of the fled over working man
78 0 0.7417
And are not vost number of Care to the eard be ye think possessi
79 0 0.7434
And freich which that was not have aboume of our brethren, that
80 0 0.7329
And arw these after them in fone Ishlat in a happen up and take
81 0 0.7394
And with bring of his remainder and sent Curean by none wats, O
82 0 0.7304
And mind animed among them towarsh recommence of their coursend-
83 0 0.7285
And is them, and the wird Antipus sawe thy loft understands rete
84 0 0.7160
And mite thy are mottlo, yea, except they murnogeffers preparati
85 0 0.7093
And being had obear.
Nevertheless, they were past into those c
86 0 0.7045
And friont pervest should cut out God.
Now the consop gave the
87 0 0.7014
And with him; for know their people, become, and God the Lamanit
88 0 0.7094
And mind a. the things salver the command of their slave upon th
89 0 0.6970
And with his propheti, Ammon was a many thy fulfied after the Sp
90 0 0.6904
And havid; and thus beginn as plavellwary misenaged, saying, com
91 0 0.6901
And with their lives; and behold, jestinity. Now, let forth year
92 0 0.6868
And awau hard he swell these things pertaingd Middoni had set en
93 0 0.6855
And are care over the people of Nephi-Lehi, which he did jush as
94 0 0.6819
And freidded my son.
And now, I say unto Zarahemla.
Comfalal
95 0 0.6808
And pewer to place in Christ, but believe.
Seet the last dain,
96 0 0.6670
And bite them they were men, that Lehon, as I say unto him: What
97 0 0.6642
And him did that they had saw this children by Teancums and the
98 0 0.6643
And minam id ye beginged with the armies of the Lamanites was th
99 0 0.6613
And which had said that retain, ye begliving among all manner ov
100 0 0.6599
And age is very cones power in the sword.
Therefore thou wilt
101 0 0.6740
And praties of Moroni him about.
And now these askengeh exerci
102 0 0.6613
And are brought torded fordate werf and shedding unto repentance
103 0 0.6589
And awe the works; and this man of plessed under the manst beste
104 0 0.6489
And from preserve them by the people to small their teother: I a
105 0 0.6448
And awai had great nardening the wickedness and the pride of the
106 0 0.6427
And bring when his bend of the flocks say unto his brethren may
107 0 0.6378
And firitious armide me stand give hast, saying: Telt redeem; be
108 0 0.6317
And are murders shall be angel because of given up just in their
109 0 0.6341
And about, which arthy forth Laman servants, brought up at a str
110 0 0.6202
And abority befold, this believer his laid cosumbly to executed
111 0 0.6182
And are.
And now Nephi.
I would and men bar delighed, that w
112 0 0.6216
And freites, and be brought out of the land of Zarahemla. Ned th
113 0 0.6273
And and Said hath not teot were cried--also be delivered by our
114 0 0.6127
And and time, their swords, and all the borders which laves amon
115 0 0.6112
And abehor is hearts of their long shord; yea, and also that it
116 0 0.6051
And awe this began to command him goreven; therefore there was I
117 0 0.6054
And into-tright, Zenamewhered abepting denis; and silver, they w
118 0 0.6030
And into the knewsed, and ablieve thing call, with him; for we w
119 0 0.6028
And and did this the judge Ammon, and his fathers year all our p
120 0 0.6024
And angifes; therefore they had fight and a sword, and there is
121 0 0.5996
And abie it supposed to deliver them.
And it came to pass that
122 0 0.6005
And bith a give as be saved--is to rememe far insteovert known u
123 0 0.5872
And fapis of its; and also deny; their camp with him, and had de
124 0 0.5854
And into with his inuase men, on hercite watkennes of perime!
125 0 0.5817
And accord him suppose that there was ascomel not to hear; for t
126 0 0.5809
And accien Siedgmentes of his brethren, which camp words his gre
127 0 0.5765
And awe this never his walls powers by the loss whereolah.
Ant
128 0 0.5747
And eatek in there spiritted by mine say, because they had been
129 0 0.5714
And ir thirst; therefore a by your wives again; therefore he sai
130 0 0.5662
And enject his hand, they gathered tomsor; and he said unto men,
131 0 0.5668
And awain the coming of Noahor hath he said: Yea, they did not d
132 0 0.5687
And gid tike up their God.
And it came to pass that the man? W
133 0 0.5634
And aid the Nephites were spneed, and his brethren done of I hav
134 0 0.5688
And appeace the endemed with inhelvist destroy things know the l
135 0 0.5628
And afpiining this holy according to to learness before the Lama
136 0 0.5638
And accircures up their house to bring my buriaged which had int
137 0 0.5534
And freit. And it came to pass thas when Moroni had same pointen
138 0 0.5485
And firing to pass the Lord sixtt them.
And now, behold, O the
139 0 0.5502
And awau Ziesout had good were seniat in my righteok; and they a
140 0 0.5409
And accien hampleate him, as I have some his dettrore to maintai
141 0 0.5462
And firen brethren, be relepor of king-men) sore wither every di
142 0 0.5383
And misins who were pleasated against the record rembant and the
143 0 0.5419
And iffinish to bahy tranch out of the land whatsoever no knowle
144 0 0.5335
And iffling their things, faith from the commencement rejoining
145 0 0.5355
And great pure good and to termy, in with their tenth in the pla
146 0 0.5282
And priesting him that might destroy their myselfings, insomperi
147 0 0.5257
And from with his ower of Pahoran, came over which they anger ag
148 0 0.5308
And friam timperisted against the man who had no power of whom t
149 0 0.5230
And acciritishe it corder, having fied against them by those men
150 0 0.5254
And cie tiand of joy, as the west to the power.
And it came to
151 0 0.5281
And awind him that my brethren speak unto him forward frighteent
152 0 0.5270
And appeice when he led to back equace of for God, or by the Lam
153 0 0.5190
And afte it hat his preserving the plan eviltont of the hearts o
154 0 0.5106
And atininiful that there should death, and also me cometh, and
155 0 0.5162
And ifte the sword, who seon and disperibee unto me, shook feed
156 0 0.5143
And mitiquity mind, and according to the chief judge, so make at
157 0 0.5192
And ifte thin forgision forward thene over up to a fear not your
158 0 0.5097
And freitions, which is great God. Behold these are the loss sub
159 0 0.5079
And ache with the Lord hath bloes, which was done to the holy Gr
160 0 0.5074
And appeicis arm vinimonahites strong in name, to doth piseling
161 0 0.5073
And miniquithers these words, he commanded order not; neither sa
162 0 0.5030
And min for many would happine even to the earth, to said unto m
163 0 0.5053
And mie first be traditions? Behold, you, because it was a sound
164 0 0.4915
And from tike ended the Lamanites had administered, and I say bi