SoundStream Neural Audio Codec

A codec, short for coder-decoder, is software that shrinks audio for sending or storage and rebuilds it on the other side. Traditional speech codecs such as Opus and EVS are designed by hand using signal processing and models of human hearing. SoundStream, described by Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund and Marco Tagliasacchi of Google Research in a July 2021 paper, replaces those hand-built parts with neural networks trained end to end. It handles speech, music and general audio with one model and can stream, meaning it encodes and decodes as audio arrives rather than waiting for a whole file.

The pipeline has three parts. An encoder, a stack of one-dimensional convolutional layers, reads the raw waveform sampled at 24 kHz and downsamples it with strides of 2, 4, 5 and 8, producing one embedding vector for every 320 samples. That is 75 vectors per second, one every 13.3 milliseconds. A quantizer then replaces each vector with a short list of codebook indices, which is what actually gets transmitted. A decoder, built like the encoder in reverse with transposed convolutions, turns the indices back into a waveform. All three are trained together so that the output sounds like the input.

The quantizer is the key idea. Vector quantization maps a vector to the nearest entry in a learned codebook and sends only that entry's number. The paper shows why plain vector quantization fails at useful bitrates: at 6 kbps and 75 frames per second each frame gets 80 bits, which would require a codebook of 2 to the power 80 entries. SoundStream instead uses residual vector quantization (RVQ). The first quantizer picks the closest codebook entry, the difference between the vector and that entry, called the residual, goes to a second quantizer, its residual goes to a third, and so on. With 8 quantizers of 1,024 entries each, every quantizer spends 10 bits and the total is 80 bits per frame, using only 8,192 stored vectors in total.

Training combines two kinds of loss. A reconstruction loss compares spectrograms of the input and output at several time-frequency resolutions, and a feature loss compares how the discriminators respond to real and decoded audio. Adversarial losses come from discriminators, networks trained to tell real audio from decoded audio, which push the decoder toward natural-sounding output rather than a blurry average. A technique called quantizer dropout makes the bitrate adjustable: during training, the model randomly uses only the first n quantizers, so a single trained model can later run anywhere from 3 to 18 kbps, and the authors report negligible quality loss compared with models trained for one fixed bitrate.

The headline results come from crowd-sourced listening tests inspired by MUSHRA, a standard method in which listeners rate several versions of a clip against a hidden reference. On 200 clips of clean speech, noisy speech, reverberant speech and music, each rated 20 times, SoundStream at 3 kbps outperformed Opus at 12 kbps and approached EVS at 9.6 kbps, which Google's blog post describes as using 3.2 to 4 times fewer bits. The model also proved fast: on a single CPU thread of a Pixel 4 phone, the default configuration ran more than 2.3 times faster than real time, and a smaller version more than 7.1 times. Training used LibriTTS for clean speech, LibriTTS mixed with Freesound noise for noisy speech, and MagnaTagATune for music.

Compared with traditional codecs, SoundStream's strength is quality at very low bitrates and its ability to treat any kind of sound with one model. Its costs are the usual neural ones: it must be trained on large audio collections, and both ends must run the network rather than a compact hand-written algorithm. Compared with Google's first Lyra codec, an autoregressive generative model at 3 kbps, SoundStream is not autoregressive and was designed for low latency. Later codecs built on the same recipe include Meta's EnCodec, released in October 2022 for 24 kHz mono and 48 kHz stereo audio, and the Improved RVQGAN codec, which reports roughly 90 times compression of 44.1 kHz audio into 8 kbps.

The codec's limits are documented in its own paper and its successors. Quality falls as fewer quantizers are used, and latency is a trade-off: doubling the frame length to 26.6 ms requires 160 bits per frame for the same 6 kbps budget. Bitstreams are tied to a specific trained model, so both ends must run the same weights, unlike standardized codecs with fixed formats. The authors also note in the paper that PESQ and POLQA, the usual ITU-T quality metrics, are inconvenient to use because of licensing restrictions, so they relied on the open ViSQOL metric and human tests for evaluation.

SoundStream's second life is as an audio tokenizer. Because RVQ turns audio into a grid of discrete integers, those integers can be modeled like words. Google's AudioLM, announced in October 2022, uses SoundStream codes as acoustic tokens, generating them with language models and decoding them with the SoundStream decoder. In practice, developers who want SoundStream for calls can use Lyra V2, released under the Apache license in September 2022 for Android and Linux with experimental Mac and Windows support.