音訊人工智慧指南
Voicebox 流程匹配語音生成
Voicebox is a speech generation model from Meta AI, announced June 16, 2023, that learns one task, filling in masked stretches of speech given the surrounding audio and a transcript, and uses it for text-to-speech, editing, denoising and cross-lingual voice transfer.
聆聽此指南AI 語音敘述 · ~7 min
本頁8 分鐘閱讀
概述
It matters as an early large demonstration that flow matching, a faster alternative to diffusion, can power general-purpose speech generation, and as a case where the model was withheld over misuse risk.
重點摘要
- Voicebox, from Meta AI in June 2023, learns to fill in masked speech from surrounding audio and a transcript, then reuses that skill for many tasks.
- It uses flow matching with straight-line optimal transport paths, generating all missing frames at once by solving an ODE.
- The audio model is a 24-layer Transformer with 330 million parameters, trained on 60,000 hours of English or 50,000 hours of multilingual audiobooks.
- Meta reported 1.9% word error rate for zero-shot text-to-speech versus 5.9% for VALL-E, and up to 20 times faster generation.
- Meta did not release Voicebox's model or code, citing potential misuse, and built a classifier to detect its generated speech.
深入探討
Voicebox is a research model that generates speech. It was described in 'Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale' by Matthew Le, Apoorv Vyas, Bowen Shi, Wei-Ning Hsu and colleagues, posted to arXiv on June 23, 2023 and accepted to NeurIPS 2023. Meta announced it on June 16, 2023. The core idea is that many speech tasks are the same problem in disguise. If a model can fill in a missing stretch of audio, given the audio around it and the words that should be spoken there, it can synthesize a sentence from scratch, fix a mistake, cover up noise, or continue a speaker's voice in another language.
The training task is called text-guided speech infilling. Each training clip comes with a transcript turned into phones, the basic sound units of a language, aligned to the audio frame by frame. During training, a large random chunk of the audio, usually 70% to 100% of its length, is hidden. The model sees the remaining audio and the full phone sequence and must produce the hidden part. Because the visible context carries the speaker's voice, emotion, microphone and background, the model learns to continue all of those in the missing region. Audio is represented as an 80-dimensional log Mel spectrogram at 100 frames per second, a compact picture of energy across frequencies over time, and a separate HiFi-GAN vocoder converts spectrograms back into waveforms.
Voicebox splits the work into two models. A duration model predicts how long each phone should last, which determines how many frames to generate. An audio model, a 24-layer Transformer with 330 million parameters, then generates the spectrogram frames. Unlike autoregressive systems that produce audio one token at a time from left to right, Voicebox is non-autoregressive: it produces all missing frames together and can use context from both before and after the gap.
The generative method is flow matching, introduced by Yaron Lipman and colleagues in a paper posted October 6, 2022. Like diffusion models, a flow-matching model turns random noise into data. Instead of learning to remove noise step by step, it learns a velocity field that tells each point how to move from noise toward data over an imaginary time running from 0 to 1. Voicebox uses the optimal transport path, in which each noise sample moves in a straight line toward its target. Training needs no simulation of the process, only a regression target at a random time. At inference, an ordinary differential equation (ODE) solver follows the learned velocity field from noise to speech. Each step is one function evaluation of the network, so the number of steps directly sets the cost, and a user can trade quality for speed at inference time without retraining.
The paper trained an English model on 60,000 hours of English audiobooks and a multilingual model on 50,000 hours of audiobooks in six languages. The paper stresses that this speech was neither filtered nor enhanced, so the model learned from audio with real-world noise and variation. Meta described the source as public domain audiobooks. Reported results include zero-shot text-to-speech with 1.9% word error rate and 0.681 speaker similarity, against 5.9% and 0.580 for VALL-E. For a 10-second clip with two function evaluations and no guidance, Voicebox took about 0.31 seconds, around 20 times faster than VALL-E; at 64 evaluations it was only 4% slower than VALL-E. On cross-lingual transfer it cut average word error rate from 10.9% to 5.2% compared with YourTTS.
The obvious comparison is VALL-E, described in a paper posted January 5, 2023. VALL-E treats speech as a language-modeling problem over discrete tokens from a neural audio codec, trained on 60,000 hours of English, and continues a 3-second prompt. That design generates left to right, so it cannot easily use future context for editing. Voicebox instead models continuous spectrograms and infills in both directions. Compared with diffusion-based speech models such as NaturalSpeech 2, the Voicebox paper notes it produces high-quality output with about 16 ODE steps where NaturalSpeech 2 uses 150 diffusion steps.
The limitations are stated in the paper itself. Training data is read audiobook speech, so the models may not transfer well to casual conversation with laughter and filler sounds. Voicebox depends on a phonemizer and a forced aligner to produce frame-level phone transcripts, and word-based phonemizers miss context-dependent pronunciations such as French liaisons. Style attributes are not separately controllable: a user cannot borrow the voice of one sample and the emotion of another. Meta also built a classifier to detect Voicebox audio. It separated original from generated audio perfectly in the paper's tests, most likely by spotting vocoder artifacts, but distinguishing Voicebox output from vocoder-resynthesized real speech was harder, with accuracy from 0.704 to 0.907 depending on how much was masked.
The largest practical constraint is access. Citing potential misuse, Meta did not release the Voicebox model or code, sharing only audio samples and the paper. Anyone wanting to use the approach must reimplement it and assemble tens of thousands of hours of transcribed speech, plus a vocoder and alignment tools. Generation cost depends on the number of ODE steps chosen, and classifier-free guidance, used to strengthen conditioning, doubles the computation per step.
04工作範例
Tracing a Voicebox-style speech edit
Start with a 6-second recording of 'I will meet you on Tuesday' and its transcript, where the speaker should have said Thursday.
Convert the audio to an 80-bin log Mel spectrogram at 100 frames per second, giving about 600 frames.
Align the phones to frames with a forced aligner, then replace the phones for 'Tuesday' with those for 'Thursday', using the duration model to set their length.
Build the mask: mark the frames of the new word as missing and keep every other frame as context.
Sample Gaussian noise for the masked frames and integrate the learned velocity field from t = 0 to 1 with about 32 solver evaluations.
Splice the generated frames into the context and convert the full spectrogram to audio with the vocoder.
它顯示了什麼
A recording that says Thursday in the same voice, room sound and pace, with the rest of the sentence unchanged. Check it by transcribing the output with a speech recognizer to confirm the new word, and by comparing the unmasked portion against the original, which should match closely.
現實世界的實施
Zero-shot text-to-speech: given a few seconds of an unseen speaker, Voicebox speaks new text in that voice; Meta reported 1.9% word error rate versus 5.9% for VALL-E.
Editing a recorded sentence by masking a misspoken word and regenerating only that span from corrected text, while the surrounding audio stays untouched.
Removing a burst of noise such as a car horn by masking the noisy segment and infilling clean speech that matches the speaker and transcript.
Cross-lingual style transfer across English, French, Spanish, German, Polish and Portuguese, where Meta reported average word error rate falling from 10.9% with YourTTS to 5.2%.
Generating diverse synthetic training data: a speech recognizer trained only on Voicebox output lost just 0.4 to 1.7 points of word error rate on LibriSpeech, per the paper.
Voicebox 流程配對語音生成的未來
Meta's follow-up, Audiobox, posted December 25, 2023, extended the flow-matching approach to a unified model for speech and sound effects, added natural-language description prompts alongside example-based prompts, and reported that Bespoke Solvers sped up generation more than 25 times over the default ODE solver. That addresses one of Voicebox's stated gaps, independent control of voice, style and acoustic conditions. Other open problems are listed in the Voicebox paper: moving from read audiobooks to conversational speech, removing the dependence on phonemizers and forced aligners by taking raw text with punctuation, and covering more languages. On safety, the authors proposed embedding detectable fingerprints in generated speech rather than relying only on after-the-fact classifiers, since their own detector had a harder time separating generated speech from vocoder-resynthesized real speech. How openly to release voice-cloning models of this capability remains an open policy question, and Meta's June 2023 decision to withhold Voicebox is one documented reference point.
付諸實踐
為什麼這很重要
- 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
- 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
- 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。
留意
- 如果未徵得同意,語音濫用和冒充風險就會增加。
- 由於口音、方言或嘈雜的環境,準確性可能會下降。
- 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。
後續步驟
- 獲得語音捕獲、克隆和重用的明確同意。
- 測試不同揚聲器和背景條件下的品質。
- 定義人員必須審查或批准輸出的時間。
- 標記合成音訊並保留來源記錄以供問責。
資料來源與延伸閱讀
- arXiv (NeurIPS 2023)Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
- Meta AIIntroducing Voicebox: The first generative AI model for speech to generalize across tasks with state-of-the-art performance
- arXivFlow Matching for Generative Modeling
- arXivNeural Codec Language Models are Zero-Shot Text to Speech Synthesizers
- arXivAudiobox: Unified Audio Generation with Natural Language Prompts
不斷探索
常見問題
什麼是 Voicebox 流程匹配語音生成?
Voicebox is a speech generation model from Meta AI, announced June 16, 2023, that learns one task, filling in masked stretches of speech given the surrounding audio and a transcript, and uses it for text-to-speech, editing, denoising and cross-lingual voice transfer. It matters as an early large demonstration that flow matching, a faster alternative to diffusion, can power general-purpose speech generation, and as a case where the model was withheld over misuse risk.
Voicebox 針對哪些單一訓練任務進行了最佳化?
Voicebox 經過訓練,可以使用周圍的音訊和文字來填充語音的屏蔽部分,這是一種受語言模型啟發的上下文表述。
Voicebox 使用哪種生成技術來取代擴散?
Voicebox 使用條件流匹配,學習將噪音傳輸為語音特徵的速度場,並且可以使用 ODE 求解器高效求解。
Voicebox 如何在不重新訓練的情況下調節目標說話者的語音生成?
該論文演示了使用參考音訊作為上下文的零樣本語音生成;這是一個推理時間條件,而不是模型重新訓練。
Voicebox 論文報告的零樣本 TTS 比較的效率結果為何?
作者報告稱,在比較中,生成速度比 VALL-E 快 20 倍;這是特定於論文的結果和設定。
為什麼 Meta 最初保留完整的 Voicebox 模型?
由於模型可以令人信服地克隆聲音,Meta 推遲了它並強調了檢測合成語音的研究。