Demucs ati HT-Demucs Orin Orisun Iyapa

Music source separation is the task of taking a mixed recording, the single stereo file that ends up on a streaming service, and recovering the individual parts that were added together to make it. The standard research setting asks for four stems: vocals, drums, bass and other, where other holds everything else. Demucs is a family of deep learning models for this task created by Alexandre Défossez and colleagues at Meta AI (then Facebook AI Research). The code and pretrained models are free on GitHub, and they became a common default for musicians, remixers, educators and researchers.

Most earlier systems worked on a spectrogram, a picture of sound that shows how much energy each frequency has at each moment, computed with the short-time Fourier transform (STFT). A network predicts a mask for each source, a set of values between 0 and 1 that says what fraction of each time-frequency cell belongs to, say, the vocals, and the masked spectrogram is turned back into audio. The weakness is phase, the timing information in each frequency component, which masks usually copy from the mixture rather than estimate. The original Demucs paper, posted in November 2019 by Défossez, Nicolas Usunier, Léon Bottou and Francis Bach, instead worked waveform to waveform: raw audio samples in, raw audio samples out, so the network learns phase as well as loudness.

That first model was a U-Net, an encoder-decoder network with skip connections that pass detail from each encoder layer straight to the matching decoder layer. The encoder is a stack of strided one-dimensional convolutions that shrink the time axis while adding channels, so deep layers summarize longer stretches of sound. A bidirectional LSTM, a recurrent layer that reads the compressed sequence forwards and backwards, sits in the middle to capture longer context. The decoder mirrors the encoder with transposed convolutions and outputs one waveform per source. The paper reported an average of 6.3 dB signal-to-distortion ratio (SDR) on the MUSDB18 test set with data augmentation, beating every architecture it compared against, and 6.8 dB with 150 extra training songs. For bass, that extra-data result surpassed the ideal ratio mask, an oracle that knows the true sources.

The second milestone was Hybrid Demucs, described by Défossez in November 2021 for the ISMIR 2021 Music Demixing (MDX) Workshop. It runs two U-Net branches in parallel, one on the waveform and one on the spectrogram, which share layers in the middle, and lets training decide which representation suits each source. Hybrid Demucs won the Music Demixing Challenge 2021 organized by Sony. It improved SDR by 1.4 dB across sources on MUSDB HQ, and in human listening tests scored 2.83 out of 5 for overall quality against 2.36 for the waveform-only version, with fewer contamination artifacts than the second-place challenge entry.

The third milestone, Hybrid Transformer Demucs (HT Demucs), was posted to arXiv in November 2022 by Simon Rouard, Francisco Massa and Défossez. It keeps the two-branch hybrid design but replaces the innermost layers with a cross-domain Transformer encoder, a stack of attention layers that let every time step look at every other and let the waveform and spectrogram branches exchange information. The paper found the Transformer performed poorly when trained only on the 100 MUSDB training songs but beat Hybrid Demucs by 0.45 dB when both used 800 extra songs. With sparse attention, which skips most pairwise comparisons so longer clips fit in memory, and per-source fine-tuning, it reached 9.20 dB SDR, then state of the art on MUSDB with extra data. HT Demucs is the v4 release and the default model in the package.

Compared with alternatives, Demucs sits between pure spectrogram maskers such as Open-Unmix, which scored 5.33 dB in the 2019 paper's comparison table, and pure waveform models such as Conv-TasNet, a speech separation network the 2019 paper adapted to music. The paper found Conv-TasNet scored well on numbers but human listeners heard significant artifacts, while Demucs sounded more natural. The hybrid design was a direct answer to the finding that some sources, such as sustained vocals, are easier in the frequency domain while transient drums are easier in the time domain.

Limits are audible. The 2019 paper's listeners noticed bleeding, where one instrument leaks into another stem, especially between vocals and other. The six-source model's piano stem is described in the README as weak. Scores come from MUSDB18, which holds 150 tracks and about 10 hours of mostly Western popular music, so results on orchestral, heavily processed or non-Western material may differ. The models also cannot recover what was never recorded separately: reverb shared by all parts, or two instruments playing the same note in the same register, will blur. Training data rights matter too, since MUSDB18 is restricted to academic use.

In practice, Demucs installs as a Python package and runs from the command line. The README says it needs at least 3 GB of GPU memory, about 7 GB with default settings, and a segment option lowers memory use at some cost in quality. On a CPU it takes roughly 1.5 times the length of the track. The original Meta repository is now archived and read-only because the author no longer works at Meta, and the README points to a personal fork maintained for important bug fixes. Separated stems are estimates, not the original multitracks, so anyone releasing a remix still needs the rights to the source recording.