Week 6: Audio and cross-modal integration

DATA 443/643 · Week 6

Module 2 — Audio and cross-modal integration

Notes

• 6.1 Waveforms, Spectrograms, and Time–Frequency Tokens(slides)
• 6.2 Audio Encoders and Speech Models(slides)
• 6.3 Fusion, Unified Embeddings, and Data Scarcity(slides)

Labs

• Lab 6: A Spectrogram and a Tiny Fusion(PDF)

Videos

• Valerio Velardo: Short-Time Fourier Transform explained (Sound of AI)
• What's AI: OpenAI's Whisper Model Explained (~12 min)
• Yannic Kilcher: OpenAI CLIP (coordinated towers as a fusion baseline)

Additional readings

• Stanford CS224S 2025 L2: spectrogram as spectrum plus time
• Stanford CS224S 2025 L11: Whisper architecture, 30 s chunks, no CTC
• Radford et al., Whisper (weak supervision, encoder–decoder)
• Elizalde et al., CLAP (CLIP for audio)
• CMU 11-777: multimodal fusion (early / late / cross-attention)

Discussion

• Why is a spectrogram a more natural transformer input than the raw waveform at 16 kHz?
• Whisper versus CLAP: both take spectrograms. Which head do you want for subtitles, and which for a typed search?
• Name one fusion choice you would defend if audio is often missing at test time.
• Audio is 64-D and vision is 256-D. What linear map lets you add them, and what map do you need after a concat?