Week 1: Neural networks and word embeddings

DATA 443/643 · Week 1

Module 1 — Neural networks and word embeddings

Notes

• 1.1 Course Map and the Semester Project(slides)
• 1.2 Neurons, Activations, and Feedforward Nets(slides)
• 1.3 Gradient Descent(slides)
• 1.4 Word Embeddings and Semantic Geometry(slides)

Labs

• Lab 1: XOR, Autograd, and a Tiny Embedding Space(PDF)

Videos

• Karpathy: Intro to Large Language Models (0:00–8:00 in class; finish 0:00–20:00 as homework)
• MIT 6.S191 Lecture 1: perceptron through loss and gradient descent (homework, ~17:20–44:22)
• 3Blue1Brown: But what is a neural network? (~19 min)
• 3Blue1Brown: Gradient descent, how neural networks learn (~21 min)
• 3Blue1Brown: What is backpropagation really doing? (~14 min, after class)
• StatQuest: Word Embedding and Word2Vec, Clearly Explained (~16 min)

Additional readings

• Goodfellow, Bengio, Courville: Deep feedforward networks, Ch. 6 opening and §6.1 XOR
• Nielsen, Neural Networks and Deep Learning, Ch. 1 (gradient descent; stop before long MNIST)
• Jurafsky & Martin, Speech and Language Processing, Ch. 5 Embeddings (draft of 19 Aug 2026)
• Mikolov et al. 2013, Efficient estimation of word representations, §§1–3
• Stanford CS224N: word vectors and neural nets (copy intuitions; skip the skip-gram Jacobian)
• CS231N notes: neuron, backprop as gates, SGD

Discussion

• Intrinsic vs. extrinsic evaluation of embeddings: which number belongs in a project report, and which one is for debugging?
• Name one way embedding geometry can encode a stereotype. What would you measure with the Lab 1 projection?
• Why can this course skip DATA 641 and 642 and still start from neurons this week?
• Goodfellow’s two-ReLU XOR vs. Lab 1’s 33-parameter MLP: what did the extra units buy you that the table already showed?