Codec Audio Neural

A codec, short for coder-decoder, turns audio into a compact bitstream and back. Traditional codecs such as MP3, Opus and EVS are designed by engineers using signal processing: a transform splits the sound into frequency components, a perceptual model decides which components the ear will not miss, and the rest are quantized, meaning rounded to a limited set of values, then entropy coded. A neural audio codec replaces most of that hand-designed pipeline with networks trained on large amounts of audio. An encoder network turns the waveform into a short sequence of vectors, a quantizer turns each vector into a few integers, and a decoder network rebuilds a waveform from those integers. Because the whole chain is trained end to end, the system learns what to keep and what to discard.

Google's SoundStream, posted in July 2021 by Neil Zeghidour and colleagues, set the template. Its encoder is a stack of convolutions that downsamples by strides of 2, 4, 5 and 8, so one embedding is produced for every 2 x 4 x 5 x 8 = 320 samples. At 24,000 samples per second, that is 75 frames per second. To hit 6 kbps, each frame gets 80 bits. A plain vector quantizer would need a codebook, a lookup table of learned vectors, with 2 to the 80th power entries, which the paper calls obviously unfeasible. SoundStream instead uses residual vector quantization (RVQ): the first quantizer picks the nearest of 1,024 codebook vectors, the second quantizes what the first missed, and so on. Eight quantizers with 1,024 entries each, 10 bits apiece, give the 80 bits.

RVQ also makes bitrate adjustable. SoundStream trained with quantizer dropout, randomly using only the first few quantizers in each training example, so a single model works at 3 to 18 kbps by sending more or fewer codebook indices. Training combines reconstruction losses, which compare the output with the original, and adversarial losses from discriminator networks that try to tell decoded audio from real audio, a technique borrowed from neural speech synthesis. Both encoder and decoder use only causal convolutions, which look only at past samples, so the codec can stream with latency set by the 320-sample frame. In the paper's listening tests, SoundStream at 3 kbps outperformed Opus at 12 kbps and approached EVS at 9.6 kbps.

Meta's EnCodec, posted in October 2022 by Alexandre Defossez, Jade Copet, Gabriel Synnaeve and Yossi Adi, kept the encoder, RVQ and decoder design but simplified training with a single multiscale spectrogram discriminator and added a loss balancer, which sets each loss weight as a fraction of the total gradient. It also used a small transformer language model with entropy coding to shrink the bitstream by up to 40% while staying faster than real time. Its models use up to 32 codebooks of 1,024 entries, 75 frames per second at 24 kHz, and in MUSHRA listening tests, a standard method in which listeners score clips against a hidden reference, it beat Opus, EVS and Google's Lyra-v2 across speech, noisy speech and music.

Descript's Descript Audio Codec (DAC), from the June 2023 paper on Improved RVQGAN by Rithesh Kumar, Prem Seetharaman and colleagues, pushed quality further. It compresses 44.1 kHz audio about 90 times, to 8 kbps, with one model for speech, music and environmental sound. The authors identified codebook collapse, in which many codebook entries go unused and bandwidth is wasted, and fixed it with two techniques from image models: projecting vectors to a low-dimensional space before lookup and normalizing them. They also found that standard quantizer dropout hurt full-bitrate quality and applied it to only some training examples. In their listening tests DAC scored much higher than EnCodec at all bitrates tested, though still below the original audio.

The second role of these codecs turned out to be as important as compression. Their integer codes are discrete tokens, the same kind of input language models are built to predict. VALL-E trained a language model on codec tokens to clone a voice from 3 seconds of audio, MusicGen interleaves several codebook streams so one transformer can generate music, and Kyutai's Mimi codec, described in the September 2024 Moshi paper, distills semantic information from the WavLM speech model into its first quantizer so that a single token stream carries both meaning and sound at 12.5 frames per second.

Compared with traditional codecs, neural codecs win most clearly at low bitrates, where engineered codecs introduce audible artifacts. At higher bitrates, established codecs remain efficient, standardized and supported by every device. Neural codecs cost more computation, need a matching decoder on both ends, and can fail in unfamiliar ways: rather than adding hiss, a generative decoder can produce plausible but wrong detail, such as smeared consonants or altered timbre, especially on sounds unlike its training data. Because codec tokens let a language model continue a voice from a 3-second sample, as VALL-E showed, the same technology that compresses speech also makes convincing voice imitation easier.

In practice, using a neural codec is simple: EnCodec and DAC publish code and pretrained weights, and encoding or decoding runs on a laptop. EnCodec's paper targeted models that run in real time on a single CPU core. Training a new codec is a larger job, because it needs large and varied audio collections and adversarial training with several discriminator networks.