Log-Mel Spectrogram

A log-mel spectrogram is a two-dimensional summary of an audio signal: a matrix with one row per short time step and one column per frequency band, where each entry is the logarithm of the energy in that band at that moment. It is the standard input representation for speech and audio models, because it turns a long one-dimensional waveform into a compact, image-like array that preserves what is being said or played and discards details that matter little for recognition.

Why not the raw waveform

Speech is commonly sampled at 16,000 samples per second, so 30 seconds is 480,000 numbers, and neighboring samples differ by tiny amounts. The content of speech, such as which phoneme is being spoken, is carried by how energy is distributed across frequencies over tens of milliseconds, which is not visible in individual samples. A time-frequency representation exposes that directly.

Construction

1. Framing. Cut the waveform into overlapping windows of \(w\) samples, advancing by a hop of \(h\) samples. An input of \(n\) samples gives \(1+\lfloor (n-w)/h\rfloor\) frames (more if the ends are padded). Overlap prevents events that fall on a window edge from being lost.
2. Windowing and Fourier transform. Multiply each frame by a taper such as a Hann window, which reduces artifacts from cutting the signal abruptly, then apply the short-time Fourier transform (STFT): a discrete Fourier transform per frame. It gives, per frame, complex coefficients at \(n_\text{fft}/2+1\) frequency bins from 0 Hz to half the sample rate. For a power spectrogram, take the squared magnitude \( |\mathrm{STFT}|^2 \) before summing through the mel filters; a magnitude frontend uses \( |\mathrm{STFT}| \) instead and gives different values. Window length trades time for frequency resolution: a longer window separates frequencies more finely and blurs timing.
3. Mel filterbank. Group the frequency bins into bands spaced on the mel scale, which spaces bands more finely at low frequencies, where hearing resolves pitch more precisely. A common conversion is \(m=2595\log_{10}(1+f/700)\), a convention fitted to perceptual experiments and not a law. Each band is a triangular filter that sums nearby bins’ energy, so the spectrum shrinks from hundreds of bins to a few dozen bands.
4. Log compression. Take the logarithm of band energies. Sound energy spans many orders of magnitude, and the log matches how loudness is perceived and keeps loud and quiet parts on a similar numerical scale. A power ratio of \(10^6\) becomes a difference of 6 in \(\log_{10}\).

With time on rows the result has shape (frames, mel bands); implementations may transpose it. Whisper returns (mel bands, frames).

Typical sizes

The original 80-band Whisper configuration (Radford et al.) uses audio at 16 kHz, 25 ms windows, and a 10 ms stride. Its frontend projects STFT power through mel filters, applies log compression, then clamps and scales it. Implementations also support 128-band variants; match the frontend to the model. Computing the sizes for 80 bands:

• Window \(400\) samples and hop \(160\) samples. With \(n_\text{fft}=400\) there are 201 frequency bins, 40 Hz apart.
• 1 second is 100 frames; 30 seconds is 3,000 frames (with edge padding; without it \(1+\lfloor(480000-400)/160\rfloor=2998\)).
• A 30-second clip holds 480,000 waveform samples and \(80\times3000=240{,}000\) log-mel values, half as many numbers, in a form where the time axis moves at 100 frames per second.
• Whisper builds its filters with librosa’s default Slaney mel scale (linear below 1 kHz, logarithmic above). Over 0 to 8 kHz in 80 bands, the first triangular filter spans about 74 Hz and the last about 591 Hz from edge to edge: fine resolution at low frequency, coarse at high. The HTK conversion above would give about 45 Hz and 525 Hz, so the exact widths depend on which mel definition an implementation uses.

The encoder then treats the matrix as a sequence of 80-dimensional vectors. In Whisper’s encoder, a convolutional stem with stride two halves the frame rate, so a 30-second window becomes 1,500 encoder positions of 20 ms each. That sequence is what attention runs over, and what a connector later maps toward a language model.

What the model sees, and what is lost

• Kept. Energy by frequency band over time: which tones, formants, and noise bursts occur when, at 10 ms time resolution and band-wide frequency resolution.
• Lost. The phase of the signal, so the waveform cannot be exactly reconstructed from the spectrogram (a vocoder has to estimate it); fine frequency detail inside a band; and absolute level if the signal is normalized.
• Frequencies above half the sample rate. A 16 kHz front end contains nothing above 8 kHz.

Pitfalls

Encoders are trained on one specific front end: sample rate, window, hop, number of mel bands, scaling. Changing any of them at inference changes the numbers the encoder sees without raising an error. A 44.1 kHz recording passed to a front end that assumes 16 kHz looks to the encoder like slower, lower-pitched audio, and recognition quality drops quietly. Take the front end from the model’s own specification, share one implementation between training and serving, and test it with fixed inputs whose expected features are known.

References

• Radford et al., “Robust Speech Recognition via Large-Scale Weak Supervision” (Whisper) — https://arxiv.org/abs/2212.04356