音訊人工智慧指南

音樂產生器

MusicGen 是 Meta 的開放權重文字轉音樂模型,於 2023 年 6 月發布,使用單一 transformer 語言模型將音樂寫成壓縮音訊代幣序列。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 產生與檢查短篇 MusicGen 片段
  5. 現實世界的實施
  6. MusicGen 的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

它能依照文字描述(如類型與情緒)或哼唱、演奏的旋律,輸出 32 kHz 音訊。其程式碼採 MIT 授權,權重則授予非商業使用。

重點摘要

  1. MusicGen 是單階段 transformer,根據文字描述以及(可選的)旋律產生 EnCodec 音訊代幣。
  2. Meta 於 2023 年 8 月 2 日在 AudioCraft 中釋出 MusicGen 權重與程式碼。
  3. MusicGen 在約 20,000 小時的授權音樂上訓練,提供 300M、1.5B 與 3.3B 參數規模。
  4. 延遲交錯模式將 30 秒的四碼本音訊轉換為約 1,500 個自回歸步驟。
  5. MusicGen 無法產生逼真的人聲,因為訓練資料中已移除人聲。
  6. MusicGen 程式碼採 MIT 授權,但模型權重採 CC-BY-NC 4.0,禁止商業使用。

深入探討

MusicGen is a model that generates music from a text description, optionally guided by a melody. It was introduced by Jade Copet and colleagues at Meta AI in the paper "Simple and Controllable Music Generation", first posted to arXiv in June 2023. It works the way a text language model does, predicting the next token from the previous ones, except that its tokens represent short slices of compressed audio rather than words. A separate neural audio codec turns those tokens back into sound. The word simple in the title refers to the design: one transformer generates everything in a single stage, instead of the chain of models that earlier systems such as Google's MusicLM used to go from coarse to fine detail.

The first ingredient is EnCodec, a neural audio codec from the same Meta team, described in an October 2022 paper. A codec compresses audio into a compact form and reconstructs it. The MusicGen version takes 32 kHz mono audio, downsamples it by a stride of 640 samples to 50 frames per second, and represents each frame with residual vector quantization (RVQ): four codebooks of 2,048 entries each, where the first codebook captures the coarse content and each later codebook encodes what the previous ones missed. Thirty seconds of music therefore becomes 1,500 frames, each with four tokens.

The second ingredient is how those four parallel token streams are fed to one transformer. Predicting all 6,000 tokens of a 30-second clip one by one, called flattening, is accurate but slow. Predicting all four codebooks of a frame at once, called parallel, is fast but ignores dependencies between them. MusicGen's default "delay" pattern offsets each codebook by one step, so at each step the model predicts codebook 1 for the current frame, codebook 2 for the previous frame, and so on. The paper says this turns 30 seconds of audio into 1,500 autoregressive steps. In its ablation, flattening still scored best, delay came close, and parallel was worst.

Conditioning steers the generation. Text prompts are encoded by a frozen T5 text encoder, and the paper also tested FLAN-T5 and CLAP. For melody, the model reads a chromagram, a representation of how much energy falls in each of the 12 pitch classes over time. The authors found that the raw chromagram led the model to copy the original track, so they kept only the dominant pitch class at each step and used Demucs, a source-separation model, to remove drums and bass first, because those dominated the chromagram. Generation uses classifier-free guidance: during training the condition is dropped 20% of the time, and at inference the model blends conditioned and unconditioned predictions with a guidance scale of 3.0 to follow the prompt more closely.

Training used about 20,000 hours of licensed music: an internal set of 10,000 high-quality tracks plus 25,000 and 365,000 instrument-only tracks from the ShutterStock and Pond5 collections, all at 32 kHz with text descriptions and metadata. The team trained models of 300 million, 1.5 billion and 3.3 billion parameters for one million steps on 30-second crops, using 32, 64 and 96 GPUs respectively. Meta announced the model publicly on 2 August 2023 as part of AudioCraft, alongside AudioGen for sound effects and an improved EnCodec decoder. The Hugging Face model card says the version 1 models were trained between April and May 2023.

On the MusicCaps benchmark of expert-written music descriptions, MusicGen compared well. Frechet Audio Distance (FAD), which measures how far generated audio statistics are from real audio, was 3.1 for the 300M model, against 4.0 reported for MusicLM, 7.5 for Mousai and 14.8 for Riffusion; Noise2Music reported the best FAD at 2.1. In human ratings on a 1 to 100 scale, the 3.3B model scored 84.81 for overall quality and the 1.5B model 83.70 for relevance to the text, against 80.51 and 82.35 for the public MusicLM demo. The authors note that larger models did not always improve automatic metrics even when listeners preferred them. With melody conditioning, the similarity between the reference and generated chromagrams rose from 0.10 to 0.66. A stereo variant, fine-tuned from the mono model with eight codebooks, was rated higher than mono.

Compared with alternatives, MusicGen's main trade is simplicity for some control. Hierarchical systems like MusicLM cascade several models, which the MusicGen authors avoided. Diffusion-based systems such as Riffusion and Mousai generate spectrograms or latents through iterative denoising rather than token by token. Within token-based models, MusicGen's interleaving patterns are its distinguishing choice. Its open release is also a difference: Meta published weights and code, while MusicLM's implementation was not public at the time of the paper.

The limitations are stated by Meta itself. The model card says MusicGen cannot generate realistic vocals, because vocals were removed from training data; works best with English descriptions; performs unevenly across musical styles and cultures; sometimes collapses to silence at the end of a song; and may need prompt engineering. The paper's authors note that their method offers no fine-grained control over how closely the output follows the conditioning beyond classifier-free guidance, and that the dataset contains a larger proportion of Western-style music. They also raise the concern that generative models could compete unfairly with artists. The weights are licensed CC-BY-NC 4.0, which rules out commercial use, even though the code is MIT-licensed. In practice, running MusicGen requires Python, PyTorch and ffmpeg per the AudioCraft README, and a GPU is advisable for the larger checkpoints; the model card's examples generate clips of a few seconds at a time.

04工作範例

產生與檢查短篇 MusicGen 片段

  1. 安裝 Hugging Face Transformers 版本 4.31.0 或更新版本以及 scipy,並確認使用符合 CC‑BY‑NC 4.0 權重授權的非商業用途。

  2. 透過文字到音訊管線載入 facebook/musicgen-small,以降低首次測試的記憶體使用量。

  3. 以提示「lo‑fi hip hop beat with soft piano and vinyl crackle」產生一段片段,並將回傳的 32 kHz 陣列存為 WAV 檔。

  4. 以相同提示產生第二段片段並聆聽兩者;使用 top‑k 抽樣會使每次執行結果不同。

  5. 將提示中的一個元素更換,例如把鋼琴換成薩克斯風,並檢查配器是否相應改變。

  6. 比較時,提供一段短小的參考旋律並聆聽旋律是否延續。

它顯示了什麼

兩段或以上32 kHz的短樂器片段,反映提示的風格與樂器。透過聆聽確認是否符合提示、是否在結尾附近崩潰為靜音,以及是否沒有出現逼真的人聲,兩項皆符合Meta模型卡的限制。

現實世界的實施

Meta於2023年8月2日以AudioCraft框架發布MusicGen、AudioGen與EnCodec,並公布模型權重與訓練程式碼,讓研究人員能在自己的資料上訓練模型。

旋律調節:MusicGen旋律檢查點在去除鼓與低音後,從參考曲目提取色譜圖,然後產生跟隨該旋律且符合文字提示的新音樂。

研究人員可透過Hugging Face Transformers(版本4.31.0或以上)載入小型、中型、大型或旋律檢查點,並使用文字到音訊管線產生單聲道32 kHz片段。

在MusicCaps的聆聽測試中,3.3B參數的MusicGen在整體品質上得分84.81/100,高於公開MusicLM示範測試的80.51。

AudioCraft倉庫後續加入相關研究模型,例如MAGNeT(非自回歸文字到音樂模型)與AudioSeal(音訊水印),均建立於相同程式碼庫。

MusicGen 的未來

The MusicGen paper names its own open problems: finer control over how strictly output follows its conditioning, and better data augmentation and guidance for audio conditioning such as melodies. Meta's August 2023 announcement acknowledged that the training data skews toward Western-style music and English-only descriptions, which limits the model's range. Since 2023, the AudioCraft repository has grown to include MAGNeT, a non-autoregressive text-to-music and text-to-sound model, MusicGen Style, JASCO for conditioning on chords, melodies and drum tracks, and AudioSeal for watermarking generated audio, which shows the research direction toward faster generation, richer musical controls and provenance tools. Questions about training-data licensing, attribution and the economic effect on working musicians remain unresolved across the field, and the MusicGen authors flagged unfair competition for artists as an open issue. Evaluations using shared benchmarks such as MusicCaps, together with human listening tests, remain the main way to compare newer music models with MusicGen.

付諸實踐

為什麼這很重要

  • 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
  • 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
  • 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。

留意

  • 如果未徵得同意,語音濫用和冒充風險就會增加。
  • 由於口音、方言或嘈雜的環境,準確性可能會下降。
  • 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。

後續步驟

  1. 獲得語音捕獲、克隆和重用的明確同意。
  2. 測試不同揚聲器和背景條件下的品質。
  3. 定義人員必須審查或批准輸出的時間。
  4. 標記合成音訊並保留來源記錄以供問責。

資料來源與延伸閱讀

  1. arXiv (Meta AI)簡單可控的音樂生成
  2. GitHub (facebookresearch)AudioCraft:音訊生成的開源程式碼
  3. Hugging Facefacebook/musicgen-large 模型卡
  4. Meta AIAudioCraft:一個簡單的一站式音訊建模平台
  5. arXiv (Meta AI)高保真神經音訊壓縮

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加 MusicGen 測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

MusicGen是什麼?

MusicGen 是 Meta 的開放權重文字轉音樂模型,於 2023 年 6 月發布,使用單一 transformer 語言模型將音樂寫成壓縮音訊代幣序列。它可以根據文字描述(如曲風與情緒)或哼唱、演奏的旋律,輸出 32 kHz 的音訊。其程式碼採用 MIT 授權,而模型權重則僅限非商業使用。

除了文字之外,旋律條件的 MusicGen 變體還能接受哪種輸入?

旋律條件變體可以使用文字描述與參考旋律;此輸入在所有僅文字的 checkpoint 中並不提供。

哪個神經編解碼器產生 MusicGen 預測的音訊標記?

MusicGen 對 Meta 的 EnCodec 編解碼器產生的離散令牌進行建模,該編解碼器還將預測的令牌解碼回音訊。

MusicGen 如何對多條 EnCodec 代碼流建模?

該架構在單階段自回歸模型中使用延遲交錯;「單階段」並不意味著整首歌曲在一次前向傳播中生成。

MusicGen 如何在不複製精確錄音的情況下遵循提供的旋律?

色譜圖捕捉隨時間變化出現的音級,讓 MusicGen 匹配曲調的和聲和輪廓,而無需再現原始音色。

MusicGen 是哪個專案和公司發布的?

MusicGen 作為 Meta AI 的 AudioCraft 專案的一部分於 2023 年發布,具有開放程式碼和模型權重。