6.1 Waveforms, Spectrograms, and Time–Frequency Tokens

STFT frames, not 16 kHz samples. Patch the picture the way ViT patched pixels.

These notes match the lecture slides. Use (slides) on the course hub for the deck.

A waveform is a 1-D list of samples. A spectrogram is that list cut into short windows, each turned into a frequency snapshot. Once you have a 2-D time–frequency picture, every vision trick from Week 4 (patches, a transformer, CLIP-style towers) applies to audio. Lab 6 will cut a sine mixture by hand; this hour is why the picture exists.

1. What a spectrogram is

A spectrogram is a picture of sound. Models do not eat 16 kHz samples as a raw list. You record pressure $x [n]$ at a sampling rate $f_{s}$ (16 kHz is a common speech default). A sine of $f$ hertz is $x [n] = \sin (2 π f n / f_{s})$ . Mixtures add; that is still one stream.

The short-time Fourier transform (STFT) windows the signal (Hann, hop $H$ , FFT size $N$ ), then takes a DFT of each frame. The magnitude spectrogram $S (t, f) = | X (t, f) |$ is what you plot. Phase exists, but most audio encoders drop it and keep magnitudes or a mel compression of them (log-mel filterbank). A mel spectrogram warps frequency to a perceptual scale.

Waveform next to its spectrogram

A tall band that stays put is a steady tone. A sweep that climbs is a chirp. Speech looks like stacked formants that move.

Log-mel is a compressive front end: fewer bands than a linear STFT, spaced the way hearing is. It is still a spectrogram. Do not skip the plot; if the 440 Hz band is missing, the rest of the pipeline is theater.

2. Why we use it

Waveforms are long and oscillatory. A transformer on 16 kHz samples would see 16{,}000 tokens per second. An STFT with a 25 ms window and a 10 ms hop sees 100 frames per second. Those frames last long enough to see a phoneme, not a single pressure sample. A 2-D time–frequency map can then be patched like a ViT image.

You cannot make time resolution and frequency resolution arbitrarily fine at once. Hop $H$ is the time step; $N$ is the frequency resolution. Small $H$ means more frames (and more tokens later); large $N$ means narrower Hertz bins and a longer window that smears onsets.

If your project uses audio, write $f_{s}$ , window, hop, and whether you used mel or raw STFT. Those choices change the tokens.

3. Architecture

The boxes connect in that order: waveform, then frames, then STFT magnitudes (optionally mel + log), then patch embed, then transformer.

Sliding windows

Treat $S$ as a single-channel image. A CNN can ingest it. A ViT can patch it: tiles of time $\times$ frequency, flatten, linear map to width $d$ , add positions. Lab 6 does that cut by hand.

Spectrogram cut into patch tokens

Time is not quite space. Adjacent frequency bins are related (harmonics); adjacent frames are related (smooth speech). Positions should know which axis is which. Still, “patch then attend” is the default modern front end.

Non-overlapping $p \times p$ tiles on an $F \times T$ grid give $(F / p) \times (T / p)$ tokens if both axes divide. Remainders are padded or cropped; Lab 6 crops.

This front end is not a transcriber. Whisper (note 6.2) will decode text from this picture. CLAP will embed the picture next to a caption.

4. How it works, step by step

Lab 6: sine mixture, STFT, patch. Hop length and window are method knobs.

Record. Length $L$ samples at rate $f_{s}$ . Duration in seconds is $L / f_{s}$ , not the frame count.
Window. Place frames of length $N$ with hop $H$ . Hann is the usual taper.
FFT each frame. Keep magnitudes, or a log-mel filterbank. Phase is usually dropped.
Check a known tone. Bin $k$ of an $N$ -point FFT sits near $k \cdot f_{s} / N$ hertz (real FFT: $k = 0, \dots, N / 2$ ). That conversion is how you check Lab 6’s two sines.
Patch. Cut the $F \times T$ grid into non-overlapping tiles, flatten, linear map, add positions that know which axis is frequency.
Hand the tokens to a transformer. Encoding, retrieval, or transcription is a later head, not this picture.

A larger FFT does not “hear more of the clip.” It hears a longer window, so onsets blur; hop still sets the time grid.

5. Mathematical formulas

Frame $m$ of the STFT:

X (m, ω) = \sum_{n} x [n] w [n - m H] e^{- j ω n} .

The plot is usually $S (m, ω) = | X (m, ω) |$ , or a mel filterbank of those magnitudes. Frame count, without padding, is

T = 1 + ⌊ \frac{L - N}{H} ⌋

when the length $L \geq N$ . Sequence length for the transformer is then about $T$ frames, or the number of patches of those frames.

Bin $k$ sits near $k \cdot f_{s} / N$ hertz. Non-overlapping $p \times p$ tiles on an $F \times T$ grid give $(F / p) \times (T / p)$ tokens when both axes divide.

6. Positive points and negative points

Positive.

A spectrogram shares ViT code: patch, position, attend.
100 frames per second is a phoneme-scale sequence; 16{,}000 samples per second is not.
Lab 6 can plot two sines and check the Hertz bins by hand.

Negative.

Phase is often dropped; reconstruction is not the job of this encoder.
Time resolution versus frequency resolution is a real tradeoff, not a software default.
A spectrogram is not yet a transcriber, a speaker id, or a music tag (next note).
Patching $F \times T$ as if it were a square image forgets which axis is frequency; remainders can drop a formant band if you crop carelessly.

When not to. Do not feed raw 16 kHz samples to a transformer and call the spectrogram optional. Do not count samples as frames.

7. What this is not

A spectrogram is not a transcript, a speaker id, or a music tag. Those are tasks on top of the tokens. Whisper (note 6.2) will decode text from this picture. CLAP will embed the picture next to a caption. If your project uses audio, write $f_{s}$ , window, hop, and whether you used mel or raw STFT. Those choices change the tokens.

Bin $k$ of an $N$ -point FFT sits near $k \cdot f_{s} / N$ hertz (real FFT: $k = 0, \dots, N / 2$ ). That conversion is how you check Lab 6’s two sines.

8. Teaching this note

About 35 minutes at the board in the two-hour week: draw one second of a 440 Hz sine, mark a window of length $N$ and hop $H$ , count frames, then cut a toy mel grid into patches. Play the STFT video (~10–12 min; see Video). Lab 6 (sine mixture, STFT, concat vs add) is studio time after notes 6.2–6.3, not this block.

9. Worked example

One second of speech at $f_{s} = 16 k H z$ : $L = 16000$ samples. Take $N = 400$ , hop $H = 160$ . Window $400 / 16000 = 25 m s$ , hop $10 m s$ . Those are Whisper’s default front-end numbers (note 6.2).

T = 1 + ⌊ \frac{16000 - 400}{160} ⌋ = 1 + 97 = 98

frames. Each real FFT has $N / 2 + 1 = 201$ bins. A 440 Hz tone sits near bin $k \approx 440 \cdot 400 / 16000 = 11$ .

Double the hop to $H = 320$ and keep $N$ fixed:

T^{'} = 1 + ⌊ \frac{15600}{320} ⌋ = 1 + 48 = 49 .

About half as many frames; each frame is twice as far apart in time (20 ms). Frequency bins did not change, because $N$ did not.

Now patch a log-mel grid of shape $80 \times 400$ with non-overlapping $16 \times 16$ tiles (ignore remainder):

\frac{80}{16} \times \frac{400}{16} = 5 \times 25 = 125

tokens. That is the number Lab 6 is practicing in miniature, and the number a Whisper-style encoder would attend over.

10. Where students get stuck

Counting samples as frames. $L / f_{s}$ is duration in seconds; $T$ is how many windows you placed.
Thinking a larger FFT “hears more of the clip.” It hears a longer window, so onsets blur; hop still sets the time grid.
Patching $F \times T$ as if it were a square image and forgetting which axis is frequency. Remainders: crop in the lab; do not silently drop a formant band.

11. Video

Valerio Velardo — Short-Time Fourier Transform Explained Easily. Play the STFT / spectrogram walkthrough in class (~10–12 min). Pause when the window slides: that hop is $H$ in the formula above. Search “Valerio Velardo Short-Time Fourier Transform” if you need the rest of the Sound of AI series. The board hour is the frame count and the patch count, not the complex-phase algebra.

12. Practice

You double the hop length and keep $N$ fixed. What happens to the number of time frames, and to time resolution?
Why is a spectrogram a better input to a transformer than the raw waveform at 16 kHz?
Count patch tokens for a mel grid of $80 \times 400$ with $16 \times 16$ tiles (ignore remainder).
$f_{s} = 8000$ , one second, $N = 256$ , hop $H = 128$ . How many STFT frames? What Hertz spacing is one FFT bin ( $f_{s} / N$ )?
Same clip as (4), but you pad so the last frame is included and $T = ⌈ L / H ⌉$ . What is $T$ , and why might a library report that instead of $1 + ⌊ (L - N) / H ⌋$ ?

Ahmad Mousavi