Week 3: Attention and transformers

DATA 443/643 · Week 3

Module 1 — Attention and transformers

Notes

• 3.1 From Recurrence to Attention(slides)
• 3.2 Self-Attention (Q, K, V)(slides)
• 3.3 The Transformer Block(slides)
• 3.4 GPT and BERT(slides)

Labs

• Lab 3: Self-Attention by Hand(PDF)

Videos

• 3Blue1Brown: But what is a GPT? Visual intro to transformers (~27 min)
• 3Blue1Brown: Attention in transformers, step-by-step (~26 min)
• Umar Jamil: Transformer explained (architecture + code; watch the block walkthrough)
• Karpathy: Let's build GPT (watch through the first attention implementation)
• CodeEmporium: BERT Neural Network — EXPLAINED!

Additional readings

• Vaswani et al., Attention is all you need (architecture; skip the appendix training tricks)
• Bahdanau, Cho, Bengio 2015: Neural machine translation by jointly learning to align (cite the reread)
• Jurafsky & Martin, Speech and Language Processing, Ch. 9 The Transformer (SLP3 draft)
• Jurafsky & Martin, Ch. 10 Masked and Generative Language Models (SLP3 draft)
• Stanford CS224N W26 L5: attention and transformers (copy the bottleneck; skip NMT training)
• CMU 11-711: attention and transformers (copy scaled dots, multi-head, the causal mask)
• Princeton COS 484 L8 and L10: encoder vs decoder, then GPT vs BERT
• Illustrated transformer (Alammar), after 3.3

Discussion

• Path length 1 versus cost T squared: which one is the reason we left RNNs, and which one is the reason Week 7 exists?
• Why mix values rather than keys? What goes wrong in Lab 3 if you softmax over the wrong axis?
• GPT versus BERT: which mask, which loss, and which one would you pick if the demo must stream tokens?
• If you drop positional encodings, which two sentences become the same set? Why is that fatal for language?