Mel Spectrograms

A mel spectrogram is a two-dimensional array that describes a sound. One axis is time, divided into short overlapping frames. The other axis is frequency, divided into bands. Each cell holds the energy of the sound in that band during that frame. What makes it a mel spectrogram, rather than an ordinary spectrogram, is how the bands are spaced. They follow the mel scale, a perceptual pitch scale on which equal steps sound roughly equally far apart to a listener. On that scale low frequencies are spread out and high frequencies are squeezed together, so a mel spectrogram spends most of its resolution where speech and music carry the most perceptually important detail. Researchers usually take the logarithm of the energies, producing a log-mel spectrogram, because loudness perception is also roughly logarithmic.

Computing one takes four steps. First, the waveform, a list of air-pressure samples such as 16,000 per second, is cut into frames using a window function, typically a Hann window that tapers each frame's edges. The window length sets frequency resolution and the hop, the gap between frame starts, sets time resolution. Second, each frame goes through a short-time Fourier transform (STFT), which measures how much of each frequency is present, producing a linear spectrogram with n_fft/2 + 1 frequency bins. Third, the magnitudes, or their squares for power, are multiplied by a mel filterbank: a set of overlapping triangular filters, each centered on a point evenly spaced on the mel scale, which pool neighboring frequency bins into one band. Fourth, a logarithm compresses the dynamic range, usually after clipping very small values so the log does not explode.

The scale itself comes from psychology. In 1937 S. S. Stevens and J. Volkmann of Harvard and E. B. Newman of Swarthmore published "A Scale for the Measurement of the Psychological Magnitude Pitch" in the Journal of the Acoustical Society of America. They built a subjective pitch scale by asking listeners to judge when one tone sounded half as high as another, at many reference frequencies. Two engineering formulas descended from that work. The HTK formula, named after the Hidden Markov Model Toolkit and the default in torchaudio, is mel = 2595 times log10(1 + f/700), which maps 1,000 Hz to about 1,000 mels. The Slaney formula from Slaney's MATLAB Auditory Toolbox, the default in librosa, is linear below 1,000 Hz and logarithmic above. They give different band edges, so features computed one way do not match a model trained the other way.

Mel features have been used for speech recognition for decades, as the Tacotron 2 paper notes, but deep learning made them a common interface between models. Tacotron 2, posted by Google researchers in December 2017, split text-to-speech into two networks connected by a mel spectrogram: one predicts 80-channel mel frames from characters, and a modified WaveNet vocoder, the network that turns spectral features back into a waveform, generates audio from them. The authors used 50 ms frames, a 12.5 ms hop, a filterbank from 125 Hz to 7.6 kHz, and clipped magnitudes at 0.01 before taking the log. They reported a mean opinion score of 4.53, against 4.58 for professional recordings. HiFi-GAN in 2020 made the vocoder step fast, and Whisper in 2022 trained a speech recognizer on 680,000 hours of audio using 80-channel log-mel input.

The main alternatives each trade something. A raw waveform keeps everything, including phase, but is long and hard to model directly: 30 seconds at 16 kHz is 480,000 numbers, while Whisper's mel representation of the same audio is 3,000 frames of 80 values, half as many numbers arranged in a far more structured form. A linear-frequency STFT spectrogram keeps more high-frequency detail and can be roughly inverted with the Griffin-Lim algorithm, but uses many more frequency bins. Mel-frequency cepstral coefficients (MFCCs), made by applying a further cosine transform to the log-mel bands, compress further and were the classic input to older recognizers. Learned audio tokenizers produce discrete codes instead of a fixed transform. The Tacotron 2 paper's argument for mel is that it is easy to compute from audio, smoother than raw samples, and indifferent to phase within each frame, which makes squared-error training easier.

Those same properties are the limitations. A mel spectrogram discards phase, and the Tacotron 2 authors note that it discards even more information than a linear spectrogram, so turning it back into audio is a hard inverse problem that needs a trained vocoder or an approximation. The pooling of high frequencies blurs detail that matters for some tasks, such as fricative sounds, music cymbals or non-speech signals; Tacotron 2's authors explicitly accepted this trade, arguing that high-frequency noise bursts rarely need high fidelity for intelligibility. Settings matter: Tacotron 2's authors tried a 5 ms hop and found that the extra time resolution caused significantly more pronunciation problems. A mel spectrogram is also not a transcript, a pitch track or a quality score; it is a signal representation that a model must still interpret.

In practice the transform is cheap and the configuration is the hard part. Every model expects a specific sample rate, FFT size, window length, hop, number of mel bands, frequency range, mel formula, normalization and log scaling, and mismatching any of them silently degrades results. Whisper large-v3, for example, needs 128 bands where earlier Whisper models used 80, and HiFi-GAN's published setup uses an FFT size of 1,024 and a hop of 256 samples at 22.05 kHz. The safest approach is to use the feature extractor shipped with a model rather than reimplementing it.