BYU CS classes

This is an old revision of the document!

---

Preliminary Syllabus and topics to be covered:

1. Basics of DNNs
   1. Convolution layers
   2. Maxpooling layers
   3. Relu units
   4. Softmax units
   5. Local response normalization / contrast normalization
2. Regularization strategies
   1. Dropout
   2. Dropconnect
   3. Batch normalization
   4. Adversarial networks
   5. Data augmentation

1. High-level implementation packages - pros and cons
   1. Tensorflow, Theano, Caffe, Keras, Torch, Mocha

Case studies / existing networks and why they're interesting

AlexNet
VGG
GoogLeNet / Inception
ZFNet

Training & initialization

Initialization strategies: Xavier, Gaussian, Identity, Sparse
Optimization theory and algorithms
Local minima; saddle points; plateaus
SGD
RPROP
RMS prop
Adagrad
Adam
Higher-order algorithms (LBFGS; Hessian-free; trust-region)
Nesterov and momentum

Large-scale distributed learning

Parameter servers
Asynchronous vs. synchronous architectures

Temporal networks and how to train them

Basic RNNs
LSTMs
Deep Memory Nets

Application areas

Deep reinforcement learning
NN models of style vs. content (deepart.io)
Imagenet classification
The Neural Turing Machine
Sentiment classification
Word embeddings

Understanding and visualizing CNNs

tSNE embeddings
deconvnets
data gradients / inceptionism

Misc

Network compression
Low bit-precision networks
Sum-product networks
Evolutionary approaches to topology discovery
Spatial transformer networks
Network-in-network
Regions-with-CNN