SesLM

AudioLM is a framework for generating audio with the same technique that powers text language models: predict the next token in a sequence. It was introduced by Zalan Borsos and colleagues at Google Research in a paper first posted in September 2022 and described in a Google blog post on October 6, 2022. Given a short audio prompt, it produces a continuation that keeps the speaker's voice, accent, recording conditions and, for music, the melody and rhythm. It was trained on raw audio only, with no text transcripts or musical notation, and Google said at the time that it was for research and had no plans to release it more broadly.

The central problem AudioLM solves is that audio needs two kinds of information at very different time scales. Long-range structure, which words are said and how a sentence or musical phrase unfolds, changes slowly. Fine acoustic detail, such as a voice's timbre and room echo, changes many times per second. A single token type rarely captures both well. The authors report that tokens from self-supervised speech models capture content but reconstruct audio poorly, while tokens from a neural codec reconstruct audio well but, when modeled alone, produce output without coherent long-term structure. AudioLM's answer is a hybrid tokenization that uses both.

Semantic tokens come from w2v-BERT, a self-supervised speech model from Google that combines contrastive learning with masked language modeling. AudioLM takes w2v-BERT's internal features from the 7th layer of its masked language model module, at 25 vectors per second, and maps each to the nearest of 1,024 cluster centers found with k-means, so each 40 ms of audio becomes one integer. Acoustic tokens come from SoundStream, Google's neural audio codec. AudioLM's SoundStream encodes 16 kHz audio into 50 frames per second, each described by 12 codebook indices from residual vector quantization, a scheme in which each quantizer encodes what the previous ones missed. With 1,024 entries per codebook this is 6,000 bits per second, or 600 tokens per second.

Generation runs in three stages, each a decoder-only Transformer of about 0.3 billion parameters. The semantic stage predicts future semantic tokens from past ones, modeling what is said. The coarse acoustic stage predicts the first 4 SoundStream quantizer levels, conditioned on the semantic tokens and on earlier coarse tokens, which adds speaker identity and recording conditions. The fine acoustic stage predicts the remaining 8 levels from the coarse ones, raising quality from 2,000 to 6,000 bits per second of detail. The SoundStream decoder then turns the tokens back into a waveform. For speech, every component was trained on the unlab-60k split of Libri-Light, about 60,000 hours of English audiobook speech, with each stage trained on 16 TPU v4 chips for 1 million steps.

The evaluations support the design. A speech recognizer transcribing audio generated from ground-truth semantic tokens had a 6.0% word error rate, compared with 2.5% on the original recordings and 6.6% for an earlier system, GSLM. A speaker classifier recognized the prompt's speaker in 92.6% of continuations, but in only 3.2% of samples generated from semantic tokens without an acoustic prompt, confirming that the semantic tokens carry content while the acoustic tokens carry voice. In the human test, 10 raters each judged 100 ten-second clips, half real and half made of a 3-second real prompt plus 7 generated seconds, and their 51.2% success rate was not statistically different from guessing.

AudioLM differs from text-to-speech (TTS) systems, which turn written text into speech and need paired text and audio for training. AudioLM needs no text at all, which means it can model any speech or music it can hear, but it also cannot be told what to say. It differs from codec-only language models, which the authors found sound clean but drift in content, and from earlier spoken language models such as GSLM, which also generate speech from discrete units; on the paper's transcription test, AudioLM's output had a slightly lower word error rate, 6.0% against 6.6%. Later Google work built on the same token split: SoundStorm (May 2023) kept the semantic tokens and replaced the two acoustic stages with a non-autoregressive model, which its authors report is two orders of magnitude faster.

The limits are clear from the paper and the follow-ups. Generation is slow because three autoregressive models produce hundreds of tokens per second of audio one at a time, the bottleneck SoundStorm targets. The speech model was trained on English audiobooks, so it reflects that style and language. Because it is unconditioned, a continuation can wander off topic or produce meaningless but fluent speech. Above all, the ability to continue a stranger's voice from 3 seconds of audio creates obvious risks of impersonation and fraud. The authors trained a detector that caught AudioLM speech with 98.6% accuracy, and Google mentioned audio watermarking as a possible further safeguard.

The practical requirements put AudioLM out of reach for most individuals. Google did not release the trained model, and reproducing it requires tens of thousands of hours of audio, a SoundStream codec, a w2v-BERT-style feature extractor and three 0.3-billion-parameter Transformers trained on TPU clusters. Its main value for learners is as a design pattern: separate what is said from how it sounds, model each with its own tokens, and decode with a neural codec.