MusicLM: Hierarchische semantische und akustische Token

MusicLM is a generative model that writes music as audio, not as sheet music or MIDI notes. A user types a description of genre, mood, instruments and tempo, and the system produces a recording that tries to match it. The research paper, MusicLM: Generating Music From Text by Andrea Agostinelli, Timo Denk and eleven colleagues at Google Research, was posted on January 26, 2023. It reports music at a 24 kHz sample rate that remains coherent over several minutes, and it compares favorably with two earlier text-to-music systems, Mubert and Riffusion, on both audio quality and faithfulness to the text.

The central idea is to treat music like language. Raw audio has tens of thousands of samples per second, far too many for a model to predict one by one over minutes. MusicLM therefore converts audio into tokens, short discrete codes, at two levels of detail. Semantic tokens come from w2v-BERT, a 600-million-parameter self-supervised audio model; MusicLM takes embeddings from its seventh layer and maps each to the nearest of 1,024 clusters, giving 25 tokens per second that capture melody, rhythm and long-range structure. Acoustic tokens come from SoundStream, a neural audio codec that compresses 24 kHz audio into 50 frames per second, each described by 12 codes from a residual vector quantizer, which works out to 600 tokens per second and a bitrate of 6 kbps. Acoustic tokens preserve timbre and recording detail well enough to be decoded back into sound.

The third ingredient connects text to music. MuLan, a Google model described in August 2022, was trained on 44 million music recordings totaling 370,000 hours paired with loosely associated text, so that a clip and a description of it land near each other in a shared embedding space. This lets MusicLM train on audio alone: during training it conditions on the MuLan embedding of the music itself, and at generation time it swaps in the MuLan embedding of the user's text. That trick removes the need for millions of hand-written captions.

Generation then runs as a hierarchy of separate transformer models, each predicting the next token from the ones before it. The semantic stage reads the MuLan tokens and generates semantic tokens, which sketch the musical plan. The acoustic stage, split into a coarse model for the first four SoundStream quantizer levels and a fine model for the remaining eight, generates acoustic tokens conditioned on both the MuLan tokens and the semantic tokens. The SoundStream decoder finally turns acoustic tokens into a waveform. Each stage is a decoder-only transformer of 24 layers with about 430 million parameters, and sampling temperatures of 1.0, 0.95 and 0.4 for the semantic, coarse and fine stages trade variety against stability.

MusicLM builds directly on earlier Google work. SoundStream, the codec, appeared in July 2021. AudioLM, posted in September 2022, introduced the hybrid of semantic and acoustic tokens and showed it could continue speech and produce coherent piano music from raw audio alone. MuLan supplied the text link in August 2022. MusicLM combined the three in January 2023, added melody conditioning through a small vision-transformer-style model trained on mel spectrograms, and released the MusicCaps evaluation set. In May 2023 Google opened sign-ups for a public version in AI Test Kitchen.

The paper measures quality in three ways on MusicCaps captions. Frechet Audio Distance (FAD) compares statistics of generated and real audio, where lower is better; MusicLM scored 4.0 on the VGGish variant against 9.6 for Mubert and 13.4 for Riffusion. MuLan Cycle Consistency, which checks how well the generated audio matches the text in MuLan's embedding space, was 0.51 for MusicLM versus 0.32 and 0.34 for the baselines. In 1,200 pairwise listening ratings, MusicLM won 312 comparisons, Riffusion 158 and Mubert 97, while the real reference recordings won 472, showing a clear remaining gap to human-made music. Compared with Riffusion, which fine-tunes an image diffusion model on spectrograms, and Mubert, which assembles pre-recorded sounds, MusicLM generates the audio itself. Compared with later single-stage models such as MusicGen, its cascade of several models is more complex to train and run.

The authors document several limits. Training data shapes the output, so music from cultures underrepresented in the data may be poorly served, and the paper raises concerns about cultural appropriation. Lyrics, vocal quality and song structure such as verse and chorus are all listed as future work. To test copying, the team measured memorization: exact matches stayed below 0.2% even with a ten-second prompt, but for about 1% of examples they found an approximate match to training audio. Because of the risk of misappropriating creative content, the paper states that the team had no plans to release the models at that point, so MusicLM itself is not available as open weights.

In practice, MusicLM is expensive to reproduce. The semantic and acoustic stages were trained on about five million audio clips, roughly 280,000 hours of music, and three 430-million-parameter transformers plus w2v-BERT, SoundStream and MuLan must run in sequence to produce a clip. For most people, the realistic ways to engage with the work are listening to the official examples page, using MusicCaps to evaluate other models, or experimenting with openly released successors that adopt similar token-based ideas.