Week 2: Sequence models: RNNs, LSTMs, and GRUs

DATA 443/643 · Week 2

Module 1 — Sequence models: RNNs, LSTMs, and GRUs

Notes

• 2.1 Sequential Models and RNNs(slides)
• 2.2 Vanishing and Exploding Gradients(slides)
• 2.3 LSTMs and GRUs(slides)

Labs

• Lab 2: Long-Range Memory(PDF)

Videos

• StatQuest: Recurrent Neural Networks, Clearly Explained (~21 min)
• RNNs and vanishing gradients (watch the vanishing/exploding segment)
• StatQuest: Long Short-Term Memory (LSTM), Clearly Explained (~21 min)
• StatQuest: Gated Recurrent Units (GRU), Clearly Explained

Additional readings

• Goodfellow, Bengio, Courville: Sequence Modeling, Ch. 10 opening (vanilla RNN; stop before recursive nets)
• Pascanu, Mikolov, Bengio 2013: On the difficulty of training recurrent neural networks (clipping)
• Hochreiter and Schmidhuber 1997: Long short-term memory (cite, do not assign the whole paper)
• Stanford CS224N W26 L4: language models and RNNs (copy intuitions; skip NMT)
• Stanford CS231N 2025 L7: RNNs, BPTT, singular values (copy the computational-graph form)
• CMU 11-785 L14: stability, memory, and the LSTM cell (copy the eigenvalue slogan)
• Understanding LSTM networks (Olah), after 2.3

Discussion

• Why is a shared recurrent cell a better default than a wider n-gram table or a fixed-window MLP?
• Clipping and truncated BPTT both appear in RNN papers. Which problem does each one actually address?
• If the forget gate is 1 and the input gate is 0, what happens to the cell, and to the gradient along that path? Why does that not mean LSTMs solved vanishing gradients?
• Lab 2 copies the first bit. What would a vanished gradient look like in your table as T grows, and what would you report in a project?