Week 7: Scaling, efficiency, and deployment

DATA 443/643 · Week 7

Module 3 — Scaling, efficiency, and deployment

Notes

• 7.1 Scaling Laws(slides)
• 7.2 Mixture of Experts(slides)
• 7.3 Compression and Deployment(slides)

Labs

• Lab 7: A Scaling Curve and Int8(PDF)

Videos

• Karpathy: Intro to Large Language Models (scaling and pretraining after 20:00)
• Karpathy: Deep Dive into LLMs like ChatGPT (internals 20:11–26:01; routing on the board)
• Umar Jamil: LoRA explained (small deltas at deploy time; pair with the quantization section)

Additional readings

• Kaplan et al., Scaling laws for neural language models
• Stanford CS224N 2025 L9: GPT-3 tokens versus Chinchilla
• Hoffmann et al., Chinchilla / training compute-optimal LLMs
• Stanford CS336 2025 L4: Mixtral 8 experts, top-2 active

Discussion

• GPT-3 was 175B on 300B tokens. Who does Chinchilla pick at the same FLOPs, and why?
• If you can 10× parameters or 10× tokens but not both, which does Chinchilla push you toward?
• Mixtral stores 8 MLPs and runs 2. What did you just train if the router always picks expert 2?
• Quantization is not a free lunch. What would you measure besides perplexity?