Stable Audio and Timing-Conditioned Latent Audio Diffusion

Stable Audio is a set of generative models from Stability AI that turn a text description, such as a style, mood, tempo and instrumentation, into an audio file. The technique underneath is latent diffusion, the same idea behind the Stable Diffusion image models. Instead of generating 44,100 sound samples per second per channel directly, the system first learns a compact code for audio, called a latent representation, and then learns to generate those codes. A decoder turns the generated code back into a waveform. The original Stable Audio research model also introduced timing conditioning, which lets a user ask for a specific duration instead of always receiving a fixed-length clip.

The pipeline in the original model has three parts. First, a variational autoencoder (VAE), a network trained to compress and then reconstruct its input, squeezes 44.1 kHz stereo audio along the time axis by a factor of 1,024 into a 64-channel latent sequence, an overall compression ratio of 32. Second, a text encoder based on CLAP, a model trained to match audio with descriptions, turns the prompt into a sequence of feature vectors. Third, a diffusion model, a 907-million-parameter U-Net, starts from random noise in the latent space and removes it step by step, guided by the text features through cross-attention and by the timing information. After denoising, the VAE decoder converts the latent sequence into stereo audio.

Timing conditioning is what made the original model distinctive. During training, each example was a window cut from a longer file, and the model received two learned embeddings: seconds_start, where the window began in the source file, and seconds_total, the full file's length. Files shorter than the training window were padded with silence. At generation time, a user sets seconds_start to 0 and seconds_total to the length they want, and the model fills that much of its 95-second window with content and the rest with silence, which is trimmed afterwards. This gives control over length, and because the model has seen where windows sit within whole files, it is designed to produce clips that begin and end rather than ones cut mid-phrase.

The timeline runs as follows. Stability AI launched Stable Audio as a product in September 2023, with a research post describing a model trained on over 800,000 files, about 19,500 hours of music, sound effects and instrument stems licensed from the stock music library AudioSparx. The full paper, Fast Timing-Conditioned Latent Audio Diffusion by Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley and Jordi Pons, was posted in February 2024 and accepted to ICML 2024. It reports generating up to 95 seconds of stereo audio in about 8 seconds on an A100 using 100 sampling steps. Stable Audio 2.0, announced on April 3, 2024, extended tracks to three minutes, added audio-to-audio transformation, and replaced the U-Net with a diffusion transformer (DiT), which the company said handles long sequences better. Stable Audio Open 1.0, described in a July 2024 paper, released weights trained only on Creative Commons audio.

The main alternative design is autoregressive generation, used by models such as MusicGen, which predict discrete audio tokens one after another. The Stable Audio paper compared against MusicGen and AudioLDM2 variants and reports better scores on two of its three main MusicCaps metrics, Fréchet distance computed on OpenL3 embeddings and KL divergence computed with the PaSST classifier, along with much faster generation than the autoregressive baselines. Diffusion generates the whole clip in parallel over a fixed number of steps, so generation time grows slowly with length, while autoregressive models must produce tokens sequentially. The paper presents full-band stereo output and long-form musical structure as the advantages over some earlier state-of-the-art models.

The limitations are concrete and partly acknowledged by the authors. In human evaluations, stereo correctness on AudioCaps sound-effect prompts was only 57%, and text alignment was weaker for sound effects, which the authors link to sound effects being only about a quarter of the training files. Timing accuracy had more errors for requested lengths around 40 to 60 seconds. The autoencoder itself causes a small loss of fidelity compared with real recordings. The open model's card adds that Stable Audio Open cannot generate realistic vocals, works best with English prompts, performs unevenly across musical styles and cultures, and is stronger at sound effects than music. Bias toward the cultures most represented in the training data is flagged in both the paper and the model card.

Data provenance is a central issue for music generation, and Stability AI has addressed it publicly. The original and 2.0 models were trained on a licensed AudioSparx catalog, and the 2.0 announcement says artists could opt out. Stable Audio Open used CC0, CC BY and CC Sampling+ recordings from Freesound and the Free Music Archive, with copyrighted music screened out using a classifier, Audible Magic matching and human review. These choices reduce some legal risk but also limit the styles the models know, which helps explain uneven quality across genres.

In practice, using Stable Audio comes in three forms. The hosted service requires no hardware. Stable Audio Open can be run locally with Stability AI's stable-audio-tools code, which is developed on Python 3.10 and requires PyTorch 2.5 or later for its attention kernels; the weights are released under the Stability AI Community License, which directs commercial users to Stability AI's licensing terms. Training a model from scratch requires a large, rights-cleared dataset and multi-GPU compute, which is why most users fine-tune or prompt existing models. For any published work, users should check the license attached to the specific model version they used, since terms differ between the hosted product and the open weights.