Week 5: Vision–language models (CLIP and BLIP)

DATA 443/643 · Week 5

Module 2 — Vision–language models (CLIP and BLIP)

Notes

• 5.1 CLIP(slides)
• 5.2 BLIP and Captioning(slides)
• 5.3 Retrieval, Bias, and Robustness(slides)

Labs

• Lab 5: Retrieval in a Toy CLIP Space(PDF)

Videos

• Yannic Kilcher: OpenAI CLIP, Connecting Text and Images (full paper)
• What's AI: OpenAI's Whisper Model Explained (encoder–decoder analog; pair with the BLIP paper)

Additional readings

• Radford et al., CLIP (towers, InfoNCE, zero-shot; skip the full WIT appendix)
• Stanford CS231N 2025 L16: CLIP is a similarity score; CoCa adds a decoder
• Li et al., BLIP (ITC + ITM + LM, and the bootstrap filter)

Discussion

• CLIP retrieves; BLIP can write. Which one belongs in a captioning project, and which loss is the reason?
• A batch of 8 versus 1024: what did you add, and what still fails (mug in grass)?
• Name one retrieval probe you would run before ranking people or jobs, and one typographic failure you would log.
• ITM versus ITC: when is a binary matcher the right tool and a batch softmax the wrong one?