Week 4: Multimodal foundations and vision transformers

DATA 443/643 · Week 4

Module 2 — Multimodal foundations and vision transformers

Notes

• 4.1 Multimodal Foundations(slides)
• 4.2 Vision Transformers(slides)
• 4.3 Contrastive Learning and Zero-Shot Transfer(slides)

Labs

• Lab 4: Patches and a Tiny Contrastive Loss(PDF)

Videos

• Yannic Kilcher: OpenAI CLIP, Connecting Text and Images (watch the two-tower setup)
• Umar Jamil: Vision Transformer explained (watch the patch embedding section)

Additional readings

• Baltrušaitis, Ahuja, Morency: Multimodal machine learning (the five challenges)
• CMU 11-777: Multimodal Machine Learning (joint vs coordinated; copy slogans, skip DCCA)
• Dosovitskiy et al., An image is worth 16x16 words (ViT; architecture, not the full scaling appendix)
• Stanford CS231N 2025 L8: attention and transformers, including ViT patchification
• Oord et al., Contrastive predictive coding (InfoNCE; cite the loss, skip the audio CPC setup)
• Stanford CS231N 2025 L16: vision and language (CLIP contrastive, zero-shot, a photo of)
• Radford et al., CLIP (Week 5 owns the paper; this week is the loss and zero-shot)

Discussion

• Search a photo gallery with a typed query: why is a coordinated space the default, and what breaks in a joint concat model?
• A classifier ViT sees all patches at once. Why is a GPT causal mask the wrong extra?
• InfoNCE treats the rest of the batch as negatives. What goes wrong if every caption is the word photo?
• Linear probe versus zero-shot: which one needs a labeled C-way training set, and which one needs the text tower?