重點摘要
- Demucs 是一個開源模型系列,它將混合歌曲分為人聲、鼓、貝斯和其他部分。
- 2019 年的 Demucs 論文直接研究波形,在 MUSDB18 上平均達到 6.3 dB SDR,擊敗了與之比較的頻譜圖方法。
- Hybrid Demucs 結合了波形和頻譜圖分支,贏得了索尼 2021 年音樂解混挑戰賽的冠軍。
- 混合 Transformer Demucs 使用跨域 Transformer 和 800 首額外訓練歌曲,於 2022 年在 MUSDB 上達到 9.20 dB SDR。
- 分離的主幹仍然包含樂器之間的出血,並且六源模型的鋼琴輸出被自己的文檔描述為較弱。
- 官方 Meta 儲存庫已存檔,運行預設模型需要大約 7 GB 的 GPU 記憶體或在 CPU 上運行緩慢。
深入探討
Music source separation is the task of taking a mixed recording, the single stereo file that ends up on a streaming service, and recovering the individual parts that were added together to make it. The standard research setting asks for four stems: vocals, drums, bass and other, where other holds everything else. Demucs is a family of deep learning models for this task created by Alexandre Défossez and colleagues at Meta AI (then Facebook AI Research). The code and pretrained models are free on GitHub, and they became a common default for musicians, remixers, educators and researchers.
Most earlier systems worked on a spectrogram, a picture of sound that shows how much energy each frequency has at each moment, computed with the short-time Fourier transform (STFT). A network predicts a mask for each source, a set of values between 0 and 1 that says what fraction of each time-frequency cell belongs to, say, the vocals, and the masked spectrogram is turned back into audio. The weakness is phase, the timing information in each frequency component, which masks usually copy from the mixture rather than estimate. The original Demucs paper, posted in November 2019 by Défossez, Nicolas Usunier, Léon Bottou and Francis Bach, instead worked waveform to waveform: raw audio samples in, raw audio samples out, so the network learns phase as well as loudness.
That first model was a U-Net, an encoder-decoder network with skip connections that pass detail from each encoder layer straight to the matching decoder layer. The encoder is a stack of strided one-dimensional convolutions that shrink the time axis while adding channels, so deep layers summarize longer stretches of sound. A bidirectional LSTM, a recurrent layer that reads the compressed sequence forwards and backwards, sits in the middle to capture longer context. The decoder mirrors the encoder with transposed convolutions and outputs one waveform per source. The paper reported an average of 6.3 dB signal-to-distortion ratio (SDR) on the MUSDB18 test set with data augmentation, beating every architecture it compared against, and 6.8 dB with 150 extra training songs. For bass, that extra-data result surpassed the ideal ratio mask, an oracle that knows the true sources.
The second milestone was Hybrid Demucs, described by Défossez in November 2021 for the ISMIR 2021 Music Demixing (MDX) Workshop. It runs two U-Net branches in parallel, one on the waveform and one on the spectrogram, which share layers in the middle, and lets training decide which representation suits each source. Hybrid Demucs won the Music Demixing Challenge 2021 organized by Sony. It improved SDR by 1.4 dB across sources on MUSDB HQ, and in human listening tests scored 2.83 out of 5 for overall quality against 2.36 for the waveform-only version, with fewer contamination artifacts than the second-place challenge entry.
The third milestone, Hybrid Transformer Demucs (HT Demucs), was posted to arXiv in November 2022 by Simon Rouard, Francisco Massa and Défossez. It keeps the two-branch hybrid design but replaces the innermost layers with a cross-domain Transformer encoder, a stack of attention layers that let every time step look at every other and let the waveform and spectrogram branches exchange information. The paper found the Transformer performed poorly when trained only on the 100 MUSDB training songs but beat Hybrid Demucs by 0.45 dB when both used 800 extra songs. With sparse attention, which skips most pairwise comparisons so longer clips fit in memory, and per-source fine-tuning, it reached 9.20 dB SDR, then state of the art on MUSDB with extra data. HT Demucs is the v4 release and the default model in the package.
Compared with alternatives, Demucs sits between pure spectrogram maskers such as Open-Unmix, which scored 5.33 dB in the 2019 paper's comparison table, and pure waveform models such as Conv-TasNet, a speech separation network the 2019 paper adapted to music. The paper found Conv-TasNet scored well on numbers but human listeners heard significant artifacts, while Demucs sounded more natural. The hybrid design was a direct answer to the finding that some sources, such as sustained vocals, are easier in the frequency domain while transient drums are easier in the time domain.
Limits are audible. The 2019 paper's listeners noticed bleeding, where one instrument leaks into another stem, especially between vocals and other. The six-source model's piano stem is described in the README as weak. Scores come from MUSDB18, which holds 150 tracks and about 10 hours of mostly Western popular music, so results on orchestral, heavily processed or non-Western material may differ. The models also cannot recover what was never recorded separately: reverb shared by all parts, or two instruments playing the same note in the same register, will blur. Training data rights matter too, since MUSDB18 is restricted to academic use.
In practice, Demucs installs as a Python package and runs from the command line. The README says it needs at least 3 GB of GPU memory, about 7 GB with default settings, and a segment option lowers memory use at some cost in quality. On a CPU it takes roughly 1.5 times the length of the track. The original Meta repository is now archived and read-only because the author no longer works at Meta, and the README points to a personal fork maintained for important bug fixes. Separated stems are estimates, not the original multitracks, so anyone releasing a remix still needs the rights to the source recording.
03工作範例
使用 Demucs 將歌曲分成四個主幹
在具有 Python 的電腦上使用 pip install demucs 安裝軟體包,最好具有至少 3 GB 記憶體的 GPU。
對您有權使用的歌曲檔案執行 demucs;預設的 htdemucs 模型寫入四個 WAV 檔案:鼓、貝斯、人聲等。
如果記憶體耗盡,請添加較小的段值,或添加 -d cpu 並預計處理時間約為歌曲長度的 1.5 倍。
用耳機聆聽每個主幹並注意滲色,例如在聲主幹中微弱地聽到鈸聲。
使用 -n htdemucs_ft 重新運行並比較聲幹;微調後的模型速度較慢,但可能稍微乾淨一些。
以零增益將所有四個主幹加載到音訊編輯器中,然後一起播放它們以檢查它們是否添加回原始混音。
它顯示了什麼
四個主幹文件和每個主幹的可聽音損的書面註釋。總和檢查聽起來應該與原始歌曲幾乎相同;任何明顯的差異都意味著主幹缺少內容或文件未對齊。
現實世界的實施
透過在歌曲上運行 demucs --two-stems=vocals 來製作卡拉 OK 曲目,官方自述文件將其記錄為一種將人聲與其他所有內容一次性分離的方法。
使用預設的 htdemucs 模型隔離鼓或貝斯聲部進行練習或轉錄,然後透過耳朵檢查琴幹是否有其他樂器的滲色。
在 Ultimate Vocal Remover 中使用 Demucs,這是一款 MIT 許可的桌面應用程序,其製作人員將 Demucs 與 MDX-Net 和 VR 架構一起列為捆綁分離模型。
在 MUSDB18-HQ 上對新的分離研究模型 Demucs 進行基準測試,MUSDB18-HQ 是 Demucs 論文中使用的 44.1 kHz 未壓縮立體聲幹的 150 軌資料集。
使用實驗性 htdemucs_6s 模型將吉他和鋼琴拆分為額外的主幹,同時注意到自述文件警告其鋼琴輸出具有明顯的滲色和偽影。
Democs 和 HT-Demucs 音樂來源分離的未來
Open problems in music source separation are now less about average benchmark scores and more about artifacts, data and scope. MUSDB18 is small and genre-limited, and the 2022 HT Demucs paper showed that its Transformer only beat older designs once 800 extra training songs were added, which points to data rather than architecture as the main constraint. Licensed multitrack data is scarce, and the benchmark itself is restricted to academic use. A second direction is more and finer stems, such as guitar, piano, strings and individual voices; the experimental six-source Demucs model shows the difficulty, since its own README calls the piano output weak. Evaluation is a third issue: SDR rewards low error energy but the 2021 Hybrid Demucs paper still relied on human listening tests to judge contamination and naturalness. Finally, the original Meta repository has been archived and is no longer maintained, so continued development depends on forks and on newer research systems that build on Demucs ideas.
付諸實踐
為什麼這很重要
- 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
- 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
- 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。
留意
- 如果未徵得同意,語音濫用和冒充風險就會增加。
- 由於口音、方言或嘈雜的環境,準確性可能會下降。
- 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。
後續步驟
- 獲得語音捕獲、克隆和重用的明確同意。
- 測試不同揚聲器和背景條件下的品質。
- 定義人員必須審查或批准輸出的時間。
- 標記合成音訊並保留來源記錄以供問責。
資料來源與延伸閱讀
- arXiv波形域中的音樂來源分離
- arXiv (ISMIR 2021 MDX Workshop)混合頻譜圖和波形源分離
- arXiv (ICASSP 2023)用於音樂來源分離的混合變壓器
- GitHub (Meta Research)facebookresearch/demucs:論文《混合頻譜圖與波形源分離》的程式碼
- SigSepMUSDB18資料集
- GitHub終極人聲去除器 GUI
不斷探索
常見問題
什麼是 Democs 和 HT-Demucs 音樂來源分離?
Democs是Meta的開源音樂來源分離模型; Hybrid Transformer Demucs (HT-Demucs) 使用波形和頻譜圖處理將歌曲分成人聲、鼓、貝斯和其他主幹。
Demucs 對完成的歌曲做了什麼?
Democs 執行音樂來源分離,將立體聲混音分成單獨的樂器和聲樂部分。
是什麼讓 Hybrid Demucs 成為「混合體」?
Hybrid Demucs 結合了時域波形分支和頻譜圖分支,融合了它們的優勢。
原始 Demucs 論文報告了 MUSDB 上分離品質的哪個指標?
原始論文報告了 MUSDB 上音樂來源分離的平均 SDR;結果屬於該基準和設定。
哪個研究組織與原始 Demucs 專案相關?
原始的 Demucs 論文列出了 FAIR 隸屬關係,官方專案儲存庫位於 facebookresearch 下。這描述了該項目的研究起源。
為什麼早期的 Democs 直接在原始波形上運作?
在波形域中操作可以保留相位,而頻譜圖幅度方法經常會丟棄該相位,而必須對其進行估計。