Jukebox

Jukebox is a neural network that generates music as raw audio, the actual waveform you hear, rather than as sheet music or MIDI note data. Given an artist, a genre and optionally lyrics, it produces a new song in that style, with instruments and a singing voice, at CD-quality sample rates. It was described in the paper Jukebox: A Generative Model for Music by Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford and Ilya Sutskever of OpenAI, submitted to arXiv on April 30, 2020. It was a research release, not a product: OpenAI published the paper, code, model weights and thousands of uncurated samples.

The central difficulty is length. Audio at 44.1 kHz has 44,100 numbers per second, so a four-minute song is over 10 million values, far more than a Transformer can attend to at once. MIT Technology Review put it as millions of timesteps per song, against about a thousand for text in GPT-2. Jukebox's answer was to compress first and generate second. It uses a VQ-VAE, a vector-quantized variational autoencoder introduced by Aaron van den Oord and colleagues in 2017, which encodes a signal into a sequence of discrete codes, each code being the index of the nearest entry in a learned codebook, and a decoder that turns codes back into audio.

Jukebox trains three separate VQ-VAEs at different compression levels, with hop lengths of 8, 32 and 128, meaning one code per 8, 32 or 128 audio samples. Each has a codebook of 2,048 entries. The top level, at 128 times compression, keeps mostly melody, harmony and rhythm; the bottom level, at 8 times, keeps the detail needed for good audio quality. The authors trained the levels separately rather than as one hierarchy so the top level would carry real information, reset rarely used codebook entries to avoid codebook collapse, and added a spectral loss to preserve mid and high frequencies.

Generation then happens in three stages. A top-level prior, an autoregressive Transformer with about 5 billion parameters and 72 layers, generates the coarse codes one at a time, each conditioned on the previous ones and on artist, genre and timing information. Its context of 8,192 tokens covers about 24 seconds of music. Two upsamplers of about 1 billion parameters each then generate the middle and bottom codes, conditioned on the level above, with contexts covering about 6 and 1.5 seconds. Finally, the bottom-level VQ-VAE decoder converts codes to audio. For lyrics, the top-level prior adds an encoder for the lyric text and attention between music and lyric tokens; the training lyrics were aligned to audio by extracting vocals with the Spleeter separation tool and then running forced alignment.

The training effort was large. The dataset was about 1.2 million songs, roughly 600,000 of them in English, paired with lyrics and metadata such as artist, album, genre and year from LyricWiki. The VQ-VAE trained on 256 NVIDIA V100 GPUs for 3 days, the upsamplers on 128 V100s for 2 weeks, the top-level prior on 512 V100s for 4 weeks, and lyrics conditioning on 512 V100s for another 2 weeks. Sampling was very slow: the paper reports about 1 hour to generate one minute of top-level codes and about 8 hours to upsample them. The repository similarly estimates about 3 hours on a V100 to fully sample 20 seconds of music.

The results drew wide attention. MIT Technology Review's Will Douglas Heaven called them technically impressive but deep in the uncanny valley, and noted samples imitating Katy Perry, Elvis Presley and Nas. The paper claimed coherence over several minutes, recognizable artist and genre styles, and singing that often followed the given lyrics. Because it produced timbre and voice, not just notes, it demonstrated that the compress-then-model recipe could work for music at scale.

Its limitations were stated plainly by its authors. Musical coherence holds mainly within the top-level context of about 24 seconds, so songs lack long-term structure such as returning choruses or call-and-response melodies. Melodies were often less interesting than human ones. The audio has audible noise and scratchiness from the compression. Training data was mostly English. Linear alignment of lyrics struggled with fast genres such as hip hop. And the sampling time made interactive use impossible. Rights questions were also raised from the start: Technology Review linked the release to Jay-Z's legal action against deepfake recordings of his voice, and OpenAI said it planned research on intellectual property.

Later systems kept the core idea of discrete audio tokens but became far more practical. Google's MusicLM, posted in January 2023, generates 24 kHz music from text descriptions with consistency over several minutes using a hierarchical sequence-to-sequence approach, and released the MusicCaps dataset of 5,500 music-text pairs. Meta's MusicGen, posted in June 2023 and published at NeurIPS 2023, uses a single-stage Transformer over interleaved token streams, avoiding the cascade of upsamplers that made Jukebox slow, and accepts text or melody conditioning. Running Jukebox today means using its archived repository, which expects no updates and tunes its sampling settings for a 16 GB V100 GPU.