Orin Gen

MusicGen is a model that generates music from a text description, optionally guided by a melody. It was introduced by Jade Copet and colleagues at Meta AI in the paper "Simple and Controllable Music Generation", first posted to arXiv in June 2023. It works the way a text language model does, predicting the next token from the previous ones, except that its tokens represent short slices of compressed audio rather than words. A separate neural audio codec turns those tokens back into sound. The word simple in the title refers to the design: one transformer generates everything in a single stage, instead of the chain of models that earlier systems such as Google's MusicLM used to go from coarse to fine detail.

The first ingredient is EnCodec, a neural audio codec from the same Meta team, described in an October 2022 paper. A codec compresses audio into a compact form and reconstructs it. The MusicGen version takes 32 kHz mono audio, downsamples it by a stride of 640 samples to 50 frames per second, and represents each frame with residual vector quantization (RVQ): four codebooks of 2,048 entries each, where the first codebook captures the coarse content and each later codebook encodes what the previous ones missed. Thirty seconds of music therefore becomes 1,500 frames, each with four tokens.

The second ingredient is how those four parallel token streams are fed to one transformer. Predicting all 6,000 tokens of a 30-second clip one by one, called flattening, is accurate but slow. Predicting all four codebooks of a frame at once, called parallel, is fast but ignores dependencies between them. MusicGen's default "delay" pattern offsets each codebook by one step, so at each step the model predicts codebook 1 for the current frame, codebook 2 for the previous frame, and so on. The paper says this turns 30 seconds of audio into 1,500 autoregressive steps. In its ablation, flattening still scored best, delay came close, and parallel was worst.

Conditioning steers the generation. Text prompts are encoded by a frozen T5 text encoder, and the paper also tested FLAN-T5 and CLAP. For melody, the model reads a chromagram, a representation of how much energy falls in each of the 12 pitch classes over time. The authors found that the raw chromagram led the model to copy the original track, so they kept only the dominant pitch class at each step and used Demucs, a source-separation model, to remove drums and bass first, because those dominated the chromagram. Generation uses classifier-free guidance: during training the condition is dropped 20% of the time, and at inference the model blends conditioned and unconditioned predictions with a guidance scale of 3.0 to follow the prompt more closely.

Training used about 20,000 hours of licensed music: an internal set of 10,000 high-quality tracks plus 25,000 and 365,000 instrument-only tracks from the ShutterStock and Pond5 collections, all at 32 kHz with text descriptions and metadata. The team trained models of 300 million, 1.5 billion and 3.3 billion parameters for one million steps on 30-second crops, using 32, 64 and 96 GPUs respectively. Meta announced the model publicly on 2 August 2023 as part of AudioCraft, alongside AudioGen for sound effects and an improved EnCodec decoder. The Hugging Face model card says the version 1 models were trained between April and May 2023.

On the MusicCaps benchmark of expert-written music descriptions, MusicGen compared well. Frechet Audio Distance (FAD), which measures how far generated audio statistics are from real audio, was 3.1 for the 300M model, against 4.0 reported for MusicLM, 7.5 for Mousai and 14.8 for Riffusion; Noise2Music reported the best FAD at 2.1. In human ratings on a 1 to 100 scale, the 3.3B model scored 84.81 for overall quality and the 1.5B model 83.70 for relevance to the text, against 80.51 and 82.35 for the public MusicLM demo. The authors note that larger models did not always improve automatic metrics even when listeners preferred them. With melody conditioning, the similarity between the reference and generated chromagrams rose from 0.10 to 0.66. A stereo variant, fine-tuned from the mono model with eight codebooks, was rated higher than mono.

Compared with alternatives, MusicGen's main trade is simplicity for some control. Hierarchical systems like MusicLM cascade several models, which the MusicGen authors avoided. Diffusion-based systems such as Riffusion and Mousai generate spectrograms or latents through iterative denoising rather than token by token. Within token-based models, MusicGen's interleaving patterns are its distinguishing choice. Its open release is also a difference: Meta published weights and code, while MusicLM's implementation was not public at the time of the paper.

The limitations are stated by Meta itself. The model card says MusicGen cannot generate realistic vocals, because vocals were removed from training data; works best with English descriptions; performs unevenly across musical styles and cultures; sometimes collapses to silence at the end of a song; and may need prompt engineering. The paper's authors note that their method offers no fine-grained control over how closely the output follows the conditioning beyond classifier-free guidance, and that the dataset contains a larger proportion of Western-style music. They also raise the concern that generative models could compete unfairly with artists. The weights are licensed CC-BY-NC 4.0, which rules out commercial use, even though the code is MIT-licensed. In practice, running MusicGen requires Python, PyTorch and ffmpeg per the AudioCraft README, and a GPU is advisable for the larger checkpoints; the model card's examples generate clips of a few seconds at a time.