Voicebox: generación de voz con flow matching

Voicebox is a research model that generates speech. It was described in 'Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale' by Matthew Le, Apoorv Vyas, Bowen Shi, Wei-Ning Hsu and colleagues, posted to arXiv on June 23, 2023 and accepted to NeurIPS 2023. Meta announced it on June 16, 2023. The core idea is that many speech tasks are the same problem in disguise. If a model can fill in a missing stretch of audio, given the audio around it and the words that should be spoken there, it can synthesize a sentence from scratch, fix a mistake, cover up noise, or continue a speaker's voice in another language.

The training task is called text-guided speech infilling. Each training clip comes with a transcript turned into phones, the basic sound units of a language, aligned to the audio frame by frame. During training, a large random chunk of the audio, usually 70% to 100% of its length, is hidden. The model sees the remaining audio and the full phone sequence and must produce the hidden part. Because the visible context carries the speaker's voice, emotion, microphone and background, the model learns to continue all of those in the missing region. Audio is represented as an 80-dimensional log Mel spectrogram at 100 frames per second, a compact picture of energy across frequencies over time, and a separate HiFi-GAN vocoder converts spectrograms back into waveforms.

Voicebox splits the work into two models. A duration model predicts how long each phone should last, which determines how many frames to generate. An audio model, a 24-layer Transformer with 330 million parameters, then generates the spectrogram frames. Unlike autoregressive systems that produce audio one token at a time from left to right, Voicebox is non-autoregressive: it produces all missing frames together and can use context from both before and after the gap.

The generative method is flow matching, introduced by Yaron Lipman and colleagues in a paper posted October 6, 2022. Like diffusion models, a flow-matching model turns random noise into data. Instead of learning to remove noise step by step, it learns a velocity field that tells each point how to move from noise toward data over an imaginary time running from 0 to 1. Voicebox uses the optimal transport path, in which each noise sample moves in a straight line toward its target. Training needs no simulation of the process, only a regression target at a random time. At inference, an ordinary differential equation (ODE) solver follows the learned velocity field from noise to speech. Each step is one function evaluation of the network, so the number of steps directly sets the cost, and a user can trade quality for speed at inference time without retraining.

The paper trained an English model on 60,000 hours of English audiobooks and a multilingual model on 50,000 hours of audiobooks in six languages. The paper stresses that this speech was neither filtered nor enhanced, so the model learned from audio with real-world noise and variation. Meta described the source as public domain audiobooks. Reported results include zero-shot text-to-speech with 1.9% word error rate and 0.681 speaker similarity, against 5.9% and 0.580 for VALL-E. For a 10-second clip with two function evaluations and no guidance, Voicebox took about 0.31 seconds, around 20 times faster than VALL-E; at 64 evaluations it was only 4% slower than VALL-E. On cross-lingual transfer it cut average word error rate from 10.9% to 5.2% compared with YourTTS.

The obvious comparison is VALL-E, described in a paper posted January 5, 2023. VALL-E treats speech as a language-modeling problem over discrete tokens from a neural audio codec, trained on 60,000 hours of English, and continues a 3-second prompt. That design generates left to right, so it cannot easily use future context for editing. Voicebox instead models continuous spectrograms and infills in both directions. Compared with diffusion-based speech models such as NaturalSpeech 2, the Voicebox paper notes it produces high-quality output with about 16 ODE steps where NaturalSpeech 2 uses 150 diffusion steps.

The limitations are stated in the paper itself. Training data is read audiobook speech, so the models may not transfer well to casual conversation with laughter and filler sounds. Voicebox depends on a phonemizer and a forced aligner to produce frame-level phone transcripts, and word-based phonemizers miss context-dependent pronunciations such as French liaisons. Style attributes are not separately controllable: a user cannot borrow the voice of one sample and the emotion of another. Meta also built a classifier to detect Voicebox audio. It separated original from generated audio perfectly in the paper's tests, most likely by spotting vocoder artifacts, but distinguishing Voicebox output from vocoder-resynthesized real speech was harder, with accuracy from 0.704 to 0.907 depending on how much was masked.

The largest practical constraint is access. Citing potential misuse, Meta did not release the Voicebox model or code, sharing only audio samples and the paper. Anyone wanting to use the approach must reimplement it and assemble tens of thousands of hours of transcribed speech, plus a vocoder and alignment tools. Generation cost depends on the number of ODE steps chosen, and classifier-free guidance, used to strengthen conditioning, doubles the computation per step.