VITS 端到端语音合成

VITS stands for Variational Inference with adversarial learning for end-to-end Text-to-Speech. It was introduced by Jaehyeon Kim, Jungil Kong and Juhee Son in the paper "Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech", posted in June 2021 and presented at ICML 2021. Its main idea is to generate the final audio samples directly from text in a single model trained in one stage. Earlier neural text-to-speech systems were split in two: an acoustic model such as Tacotron 2 or Glow-TTS predicted a mel spectrogram, a compact picture of sound energy over time and frequency, and a separate vocoder such as HiFi-GAN converted that picture into a waveform. Two-stage systems sounded good but had to be trained and tuned separately, and errors in the first stage passed into the second.

VITS is built around a conditional variational autoencoder (VAE). An autoencoder learns to compress data into a hidden code, called a latent variable, and rebuild it; a variational one learns a probability distribution over codes, so new samples can be drawn. During training, a posterior encoder reads the linear spectrogram of a real recording and produces a latent sequence z describing that audio. A decoder, built like the HiFi-GAN generator, turns z back into a waveform. Separately, a text encoder reads the phonemes, the basic speech sounds of the sentence, and predicts a distribution for each one. A normalizing flow, a chain of invertible transformations, bends the audio latent into the shape the text encoder predicts, so text and sound meet in a shared space. Training pushes the two sides to agree while the decoder learns to reproduce the recording.

Two problems remain: which audio frames belong to which phoneme, and how long each phoneme should last. For alignment, VITS reuses monotonic alignment search (MAS), introduced in Glow-TTS at NeurIPS 2020. MAS uses dynamic programming to find the most likely left-to-right assignment of audio frames to phonemes, with no external aligner. For duration, VITS adds a stochastic duration predictor, a small flow-based model that learns a distribution of possible lengths rather than one fixed value. The same sentence can therefore be spoken with different rhythms on different runs. Finally, adversarial training adds a discriminator network that tries to tell real audio from generated audio, and the generator improves by fooling it, which sharpens fine detail.

At synthesis time the path is short. Text becomes phonemes, the text encoder predicts a distribution per phoneme, the duration predictor samples how many frames each phoneme gets, a latent sequence is sampled, the flow is run in reverse, and the decoder outputs audio. The VITS paper reports a speed of about 1,480 kHz on one NVIDIA V100 GPU, meaning roughly 1.48 million audio samples per second, or 67 times faster than real time, compared with about 27 times for Glow-TTS plus HiFi-GAN measured the same way.

The quality results explain its influence. On LJ Speech, a public dataset of 13,100 clips from a single reader totaling about 24 hours, listeners rated VITS 4.43 on a 5-point mean opinion score (MOS) scale, against 4.46 for the real recordings, 4.32 for Glow-TTS with a fine-tuned HiFi-GAN and 4.25 for Tacotron 2 with a fine-tuned HiFi-GAN. An ablation, a test with one component removed, showed how much the flow mattered: without it, MOS fell from 4.50 to 2.98. The paper also trained a multi-speaker version on VCTK, a dataset of 109 English speakers. Follow-ups built directly on the code: YourTTS added zero-shot voice cloning across languages, and Meta's Massively Multilingual Speech project, posted in May 2023, released text-to-speech for 1,107 languages using the VITS architecture.

Compared with alternatives, VITS sits between older two-stage systems and newer large generative models. Against two-stage pipelines it offers simpler training and faster synthesis with equal or better quality on standard tests. Against autoregressive models such as Tacotron 2, which produce output one step at a time, it generates all frames in parallel and is less prone to skipping or repeating words, because MAS forces a strictly left-to-right alignment. Large language-model-style speech generators trained on far more data can imitate voices and emotions more flexibly, but they are heavier to run. VITS remains attractive where a small, fast, single-voice or few-voice model is enough.

VITS has known limits. VITS2's authors list intermittent unnaturalness, computational inefficiency and strong dependence on phoneme conversion, meaning text must first be turned into phonemes by a separate tool, which breaks on unusual names and languages without good pronunciation rules. Output is random by design; the Transformers documentation tells users to set a seed for reproducible audio. Expressiveness is bounded by training data: a model trained on one calm reader will sound like one calm reader. Meta's MMS data came from readings of religious texts, which shapes vocabulary and speaking style. As with any voice synthesis, models fine-tuned on a real person's voice raise consent and impersonation concerns.

What VITS requires in practice is modest by current standards. The original model was trained with mixed precision on 4 NVIDIA V100 GPUs, a batch of 64 per GPU, for up to 800,000 steps, using about 24 hours of clean single-speaker audio. Fine-tuning a pretrained checkpoint to a new voice needs far less; YourTTS reported useful results from under a minute of speech. Running a trained model needs only a single GPU or a capable CPU, and the Hugging Face implementation produces 16 kHz audio by default for MMS checkpoints with a few lines of code.