概述
僅使用音訊進行訓練,沒有文字記錄,它以相同的聲音和風格繼續 3 秒的語音提示,其令牌層次結構成為後來的音訊產生器的模板。
重點摘要
- AudioLM 透過使用 Transformer 預測離散標記來產生音頻,使用內容語義標記和音質 SoundStream 聲學標記。
- 其語意標記來自 w2v-BERT 特徵,以每秒 25 個標記的速度聚類為 1,024 個值。
- 生成分三個階段進行:語意標記、來自 4 個量化器的粗聲學標記和來自 8 個以上量化器的精細聲學標記。
- 人類評分者辨識真實語音與 AudioLM 連續語音的正確率僅為 51.2%,幾乎是偶然。
- 一個簡單的分類器偵測到 AudioLM 語音的準確率高達 98.6%,Google 並未公開發布該模型。
深入探討
AudioLM is a framework for generating audio with the same technique that powers text language models: predict the next token in a sequence. It was introduced by Zalan Borsos and colleagues at Google Research in a paper first posted in September 2022 and described in a Google blog post on October 6, 2022. Given a short audio prompt, it produces a continuation that keeps the speaker's voice, accent, recording conditions and, for music, the melody and rhythm. It was trained on raw audio only, with no text transcripts or musical notation, and Google said at the time that it was for research and had no plans to release it more broadly.
The central problem AudioLM solves is that audio needs two kinds of information at very different time scales. Long-range structure, which words are said and how a sentence or musical phrase unfolds, changes slowly. Fine acoustic detail, such as a voice's timbre and room echo, changes many times per second. A single token type rarely captures both well. The authors report that tokens from self-supervised speech models capture content but reconstruct audio poorly, while tokens from a neural codec reconstruct audio well but, when modeled alone, produce output without coherent long-term structure. AudioLM's answer is a hybrid tokenization that uses both.
Semantic tokens come from w2v-BERT, a self-supervised speech model from Google that combines contrastive learning with masked language modeling. AudioLM takes w2v-BERT's internal features from the 7th layer of its masked language model module, at 25 vectors per second, and maps each to the nearest of 1,024 cluster centers found with k-means, so each 40 ms of audio becomes one integer. Acoustic tokens come from SoundStream, Google's neural audio codec. AudioLM's SoundStream encodes 16 kHz audio into 50 frames per second, each described by 12 codebook indices from residual vector quantization, a scheme in which each quantizer encodes what the previous ones missed. With 1,024 entries per codebook this is 6,000 bits per second, or 600 tokens per second.
Generation runs in three stages, each a decoder-only Transformer of about 0.3 billion parameters. The semantic stage predicts future semantic tokens from past ones, modeling what is said. The coarse acoustic stage predicts the first 4 SoundStream quantizer levels, conditioned on the semantic tokens and on earlier coarse tokens, which adds speaker identity and recording conditions. The fine acoustic stage predicts the remaining 8 levels from the coarse ones, raising quality from 2,000 to 6,000 bits per second of detail. The SoundStream decoder then turns the tokens back into a waveform. For speech, every component was trained on the unlab-60k split of Libri-Light, about 60,000 hours of English audiobook speech, with each stage trained on 16 TPU v4 chips for 1 million steps.
The evaluations support the design. A speech recognizer transcribing audio generated from ground-truth semantic tokens had a 6.0% word error rate, compared with 2.5% on the original recordings and 6.6% for an earlier system, GSLM. A speaker classifier recognized the prompt's speaker in 92.6% of continuations, but in only 3.2% of samples generated from semantic tokens without an acoustic prompt, confirming that the semantic tokens carry content while the acoustic tokens carry voice. In the human test, 10 raters each judged 100 ten-second clips, half real and half made of a 3-second real prompt plus 7 generated seconds, and their 51.2% success rate was not statistically different from guessing.
AudioLM differs from text-to-speech (TTS) systems, which turn written text into speech and need paired text and audio for training. AudioLM needs no text at all, which means it can model any speech or music it can hear, but it also cannot be told what to say. It differs from codec-only language models, which the authors found sound clean but drift in content, and from earlier spoken language models such as GSLM, which also generate speech from discrete units; on the paper's transcription test, AudioLM's output had a slightly lower word error rate, 6.0% against 6.6%. Later Google work built on the same token split: SoundStorm (May 2023) kept the semantic tokens and replaced the two acoustic stages with a non-autoregressive model, which its authors report is two orders of magnitude faster.
The limits are clear from the paper and the follow-ups. Generation is slow because three autoregressive models produce hundreds of tokens per second of audio one at a time, the bottleneck SoundStorm targets. The speech model was trained on English audiobooks, so it reflects that style and language. Because it is unconditioned, a continuation can wander off topic or produce meaningless but fluent speech. Above all, the ability to continue a stranger's voice from 3 seconds of audio creates obvious risks of impersonation and fraud. The authors trained a detector that caught AudioLM speech with 98.6% accuracy, and Google mentioned audio watermarking as a possible further safeguard.
The practical requirements put AudioLM out of reach for most individuals. Google did not release the trained model, and reproducing it requires tens of thousands of hours of audio, a SoundStream codec, a w2v-BERT-style feature extractor and three 0.3-billion-parameter Transformers trained on TPU clusters. Its main value for learners is as a design pattern: separate what is said from how it sounds, model each with its own tokens, and decode with a neural codec.
04工作範例
計算 10 秒剪輯的 AudioLM 令牌
以 16 kHz 取樣的 10 秒語音片段,即 160,000 個樣本。
語意標記:每秒 25 個的 w2v-BERT 特徵給予 250 個語意標記,每個語意標記包含 1,024 個群集 ID。
聲學幀:SoundStream 的總步幅為 320,即 160,000 / 320 = 500 幀,即每秒 50 幀。
粗略令牌:500 幀乘以 4 個量化器等於第二階段的 2,000 個令牌。
精細令牌:500 幀乘以 8 個量化器等於第三階段的 4,000 個令牌。
位元率檢查:500 幀乘以 12 個量化器乘以每個量化器 10 位,除以 10 秒,等於每秒 6,000 位元。
它顯示了什麼
學習者看到 10 秒的語音變成了 250 個語義標記和 6,000 個聲學標記,這解釋了為什麼 AudioLM 將生成分為多個階段,以及為什麼 SoundStorm 後來專注於加速聲學部分。
現實世界的實施
語音延續:給 LibriSpeech 揚聲器 3 秒的時間,AudioLM 以相同的聲音再寫 7 秒,在包含 1,000 個評分的聽力測試中,人們選擇真實語音與生成正確語音的比例只有 51.2%。
鋼琴延續:AudioLM 在內部 40,000 小時的鋼琴錄音集上進行重新訓練,繼續來自 Maestro 資料集的 4 秒提示,具有一致的旋律和節奏,無需任何 MIDI 或樂譜輸入。
合成語音偵測:除了模型之外,Google 還訓練了一個簡單的音訊分類器,該分類器告訴 AudioLM 來自真實 SoundStream 壓縮語音的延續,在平衡測試集上的準確率達到 98.6%。
生成速度更快:Google 的 SoundStorm(2023 年 5 月)保留了 AudioLM 的語義標記,但用並行解碼替換了其聲學階段,在 TPU-v4 上在 0.5 秒內生成 30 秒的音頻,並從腳本生成多揚聲器多揚聲器對話。
AudioLM 的未來
AudioLM 在語意和聲學標記之間的劃分影響了後來音訊產生器的建構方式。 Google 自己的 SoundStorm(2023 年 5 月)保留了 AudioLM 的語義標記,並用基於置信度的並行解碼取代了緩慢的自回歸聲學階段,在 TPU-v4 上半秒內生成 30 秒的音頻,並將該方法從文字記錄和簡短的語音提示擴展到多對話。開放的研究問題包括如何使基於標記的生成速度足夠快以供實時使用,如何在不失去從未轉錄音頻中學習的能力的情況下添加文本、風格或情感等控制,以及如何將主要在英語有聲讀物上訓練的模型擴展到多種語言和錄音條件。安全問題至少與技術問題一樣突出。 AudioLM 論文報告稱,人們無法可靠地區分 7 秒的連續語音和真實語音,而簡單的分類器可以以 98.6% 的準確率檢測它們。隨著生成器的改進,此類檢測器是否能繼續工作,以及 Google 所說的 2022 年可能方向的水印是否可以對編輯和壓縮變得穩健,這些都尚未解決。如何處理聲音克隆和合成音訊披露的同意也是一個僅靠研究無法解決的懸而未決的問題。
付諸實踐
為什麼這很重要
- 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
- 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
- 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。
留意
- 如果未徵得同意,語音濫用和冒充風險就會增加。
- 由於口音、方言或嘈雜的環境,準確性可能會下降。
- 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。
後續步驟
- 獲得語音捕獲、克隆和重用的明確同意。
- 測試不同揚聲器和背景條件下的品質。
- 定義人員必須審查或批准輸出的時間。
- 標記合成音訊並保留來源記錄以供問責。
資料來源與延伸閱讀
- arXivAudioLM:音訊產生的語言建模方法
- Google Research BlogAudioLM:音訊產生的語言建模方法
- arXivSoundStream:端對端神經音訊編解碼器
- arXivw2v-BERT:結合對比學習和掩碼語言建模進行自監督語音預訓練
- arXivSoundStorm:高效的平行音訊生成
不斷探索
常見問題
什麼是 AudioLM?
AudioLM 是一個 Google 研究系統,於 2022 年 9 月首次描述,它透過將音訊視為一系列離散標記並使用語言模型對其進行預測來產生逼真的語音和鋼琴音樂。僅使用音訊進行訓練,沒有文字記錄,它以相同的聲音和風格繼續 3 秒的語音提示,其令牌層次結構成為後來的音訊產生器的模板。
AudioLM 使用什麼核心思想來產生音訊?
AudioLM 將波形轉換為離散標記並按順序預測它們,將語言模型的下一個標記方法應用於原始聲音。
AudioLM 中「語意」標記的作用是什麼?
語義標記(來自 w2v-BERT)對高級結構和連貫性進行編碼,而聲學標記則處理精細的音訊細節。
哪個神經編解碼器產生 AudioLM 的聲學標記?
SoundStream 使用殘差向量量化將音訊壓縮為聲學標記,然後將預測標記解碼回波形。
AudioLM 需要什麼作為訓練資料?
一個顯著的特點是 AudioLM 純粹基於音訊進行訓練,沒有任何文字標籤,直接從聲音中學習結構。
AudioLM 預測語意標記後,其分層產生如何添加聲學細節?
论文的层次结构从语义标记到粗略的声音标记,然后是精细的声学标记;这个阶段的顺序与语义表示的角色不同。