BYU CS classes

This is an old revision of the document!

---

Preliminary Syllabus and topics to be covered:

1. Basics of DNNs
   1. Convolution layers
   2. Maxpooling layers
   3. Relu units
   4. Softmax units
   5. Local response normalization / contrast normalization
2. Regularization strategies
   1. Dropout
   2. Dropconnect
   3. Batch normalization
   4. Adversarial networks
   5. Data augmentation
3. High-level implementation packages - pros and cons
   1. Tensorflow, Theano, Caffe, Keras, Torch, Mocha
4. Case studies / existing networks and why they're interesting
   1. AlexNet
   2. VGG
   3. GoogLeNet / Inception
   4. ZFNet
5. Training & initialization
   1. Initialization strategies: Xavier, Gaussian, Identity, Sparse
   2. Optimization theory and algorithms
   3. Local minima; saddle points; plateaus
   4. SGD
   5. RPROP
   6. RMS prop
   7. Adagrad
   8. Adam
   9. Higher-order algorithms (LBFGS; Hessian-free; trust-region)
   10. Nesterov and momentum
6. Large-scale distributed learning
   1. Parameter servers
   2. Asynchronous vs. synchronous architectures
7. Temporal networks and how to train them
   1. Basic RNNs and Backprop-through-time
   2. LSTMs
   3. Deep Memory Nets
8. Application areas
   1. Deep reinforcement learning
   2. NN models of style vs. content (deepart.io)
   3. Imagenet classification
   4. The Neural Turing Machine
   5. Sentiment classification
   6. Word embeddings
9. Understanding and visualizing CNNs
   1. tSNE embeddings
   2. deconvnets
   3. data gradients / inceptionism
10. Misc
    1. Network compression
    2. Low bit-precision networks
    3. Sum-product networks
    4. Evolutionary approaches to topology discovery
    5. Spatial transformer networks
    6. Network-in-network
    7. Regions-with-CNN