Audio AI-GIDS

VITS End-to-End Spraaksynthese

VITS is a text-to-speech model, published at ICML 2021, that turns text directly into an audio waveform in one network instead of the usual two-stage pipeline of spectrogram predictor plus vocoder.

Luister naar deze gidsAI-verteld · ~7 min

Op deze pagina8 min leestijd
  1. Overzicht
  2. Belangrijkste inzichten
  3. Diepe duik
  4. Generate and vary speech with a VITS model
  5. Implementatie in de echte wereld
  6. De toekomst van VITS end-to-end spraaksynthese
  7. Het in de praktijk brengen
  8. Bronnen en verdere literatuur
  9. Blijf verkennen
  10. Veelgestelde vragen

Overzicht

It scored 4.43 out of 5 in listening tests against 4.46 for real recordings, runs about 67 times faster than real time on one GPU, and underpins Meta's speech models for 1,107 languages.

Belangrijkste inzichten

  1. VITS generates waveforms directly from text in one model, removing the separate vocoder used by two-stage text-to-speech systems.
  2. On LJ Speech, VITS scored 4.43 mean opinion score against 4.46 for real recordings in its 2021 ICML paper.
  3. Monotonic alignment search lets VITS learn which audio frames match which phoneme without any external alignment tool.
  4. A stochastic duration predictor lets VITS speak the same sentence with different rhythms, so output varies unless a random seed is fixed.
  5. Meta's 2023 Massively Multilingual Speech text-to-speech models for 1,107 languages use the VITS architecture.

Diepe duik

VITS stands for Variational Inference with adversarial learning for end-to-end Text-to-Speech. It was introduced by Jaehyeon Kim, Jungil Kong and Juhee Son in the paper "Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech", posted in June 2021 and presented at ICML 2021. Its main idea is to generate the final audio samples directly from text in a single model trained in one stage. Earlier neural text-to-speech systems were split in two: an acoustic model such as Tacotron 2 or Glow-TTS predicted a mel spectrogram, a compact picture of sound energy over time and frequency, and a separate vocoder such as HiFi-GAN converted that picture into a waveform. Two-stage systems sounded good but had to be trained and tuned separately, and errors in the first stage passed into the second.

VITS is built around a conditional variational autoencoder (VAE). An autoencoder learns to compress data into a hidden code, called a latent variable, and rebuild it; a variational one learns a probability distribution over codes, so new samples can be drawn. During training, a posterior encoder reads the linear spectrogram of a real recording and produces a latent sequence z describing that audio. A decoder, built like the HiFi-GAN generator, turns z back into a waveform. Separately, a text encoder reads the phonemes, the basic speech sounds of the sentence, and predicts a distribution for each one. A normalizing flow, a chain of invertible transformations, bends the audio latent into the shape the text encoder predicts, so text and sound meet in a shared space. Training pushes the two sides to agree while the decoder learns to reproduce the recording.

Two problems remain: which audio frames belong to which phoneme, and how long each phoneme should last. For alignment, VITS reuses monotonic alignment search (MAS), introduced in Glow-TTS at NeurIPS 2020. MAS uses dynamic programming to find the most likely left-to-right assignment of audio frames to phonemes, with no external aligner. For duration, VITS adds a stochastic duration predictor, a small flow-based model that learns a distribution of possible lengths rather than one fixed value. The same sentence can therefore be spoken with different rhythms on different runs. Finally, adversarial training adds a discriminator network that tries to tell real audio from generated audio, and the generator improves by fooling it, which sharpens fine detail.

At synthesis time the path is short. Text becomes phonemes, the text encoder predicts a distribution per phoneme, the duration predictor samples how many frames each phoneme gets, a latent sequence is sampled, the flow is run in reverse, and the decoder outputs audio. The VITS paper reports a speed of about 1,480 kHz on one NVIDIA V100 GPU, meaning roughly 1.48 million audio samples per second, or 67 times faster than real time, compared with about 27 times for Glow-TTS plus HiFi-GAN measured the same way.

The quality results explain its influence. On LJ Speech, a public dataset of 13,100 clips from a single reader totaling about 24 hours, listeners rated VITS 4.43 on a 5-point mean opinion score (MOS) scale, against 4.46 for the real recordings, 4.32 for Glow-TTS with a fine-tuned HiFi-GAN and 4.25 for Tacotron 2 with a fine-tuned HiFi-GAN. An ablation, a test with one component removed, showed how much the flow mattered: without it, MOS fell from 4.50 to 2.98. The paper also trained a multi-speaker version on VCTK, a dataset of 109 English speakers. Follow-ups built directly on the code: YourTTS added zero-shot voice cloning across languages, and Meta's Massively Multilingual Speech project, posted in May 2023, released text-to-speech for 1,107 languages using the VITS architecture.

Compared with alternatives, VITS sits between older two-stage systems and newer large generative models. Against two-stage pipelines it offers simpler training and faster synthesis with equal or better quality on standard tests. Against autoregressive models such as Tacotron 2, which produce output one step at a time, it generates all frames in parallel and is less prone to skipping or repeating words, because MAS forces a strictly left-to-right alignment. Large language-model-style speech generators trained on far more data can imitate voices and emotions more flexibly, but they are heavier to run. VITS remains attractive where a small, fast, single-voice or few-voice model is enough.

VITS has known limits. VITS2's authors list intermittent unnaturalness, computational inefficiency and strong dependence on phoneme conversion, meaning text must first be turned into phonemes by a separate tool, which breaks on unusual names and languages without good pronunciation rules. Output is random by design; the Transformers documentation tells users to set a seed for reproducible audio. Expressiveness is bounded by training data: a model trained on one calm reader will sound like one calm reader. Meta's MMS data came from readings of religious texts, which shapes vocabulary and speaking style. As with any voice synthesis, models fine-tuned on a real person's voice raise consent and impersonation concerns.

What VITS requires in practice is modest by current standards. The original model was trained with mixed precision on 4 NVIDIA V100 GPUs, a batch of 64 per GPU, for up to 800,000 steps, using about 24 hours of clean single-speaker audio. Fine-tuning a pretrained checkpoint to a new voice needs far less; YourTTS reported useful results from under a minute of speech. Running a trained model needs only a single GPU or a capable CPU, and the Hugging Face implementation produces 16 kHz audio by default for MMS checkpoints with a few lines of code.

04Uitgewerkt voorbeeld

Generate and vary speech with a VITS model

  1. Install the Transformers library and load VitsTokenizer and VitsModel from the facebook/mms-tts-eng checkpoint.

  2. Tokenize a sentence such as "Turn left at the second traffic light." and call set_seed(555) before running the model under torch.no_grad().

  3. Save outputs.waveform at the model's sampling rate (16 kHz for MMS) as a WAV file and listen to it.

  4. Run it again with the same seed and confirm the audio is identical, then change the seed and note small differences in rhythm.

  5. Set model.speaking_rate to 0.8 and regenerate to hear slower speech, then raise model.noise_scale_duration to hear more varied timing.

  6. Compare the number of output samples across runs to see how duration sampling changes total length.

Wat het laat zien

Several WAV files of the same sentence with controlled differences in speed and rhythm. Check by comparing durations: a speaking rate of 0.8 should produce a clip about 25% longer, and identical seeds should produce byte-identical audio.

Implementatie in de echte wereld

Meta's Massively Multilingual Speech project released text-to-speech models for 1,107 languages in 2023; Hugging Face Transformers loads them as VITS models, for example facebook/mms-tts-eng.

YourTTS (ICML 2022) extends VITS for zero-shot multi-speaker and multilingual speech, and its authors report strong voice similarity after fine-tuning on less than one minute of a new speaker's audio.

Developers adjust delivery at inference time through the speaking_rate, noise_scale and noise_scale_duration settings exposed by the Transformers VitsModel, for example slowing narration for language learners.

VITS is a standard research baseline on the LJ Speech dataset of about 24 hours of one reader, where the original paper reported a mean opinion score of 4.43.

VITS2 (Interspeech 2023) from Jungil Kong and colleagues improves on VITS's occasional unnatural passages, speeds up training and inference, and reduces dependence on phoneme conversion.

De toekomst van VITS end-to-end spraaksynthese

Work after VITS has followed three lines. The first is refinement of the same design: VITS2, published at Interspeech 2023, targeted the original's occasional unnatural passages, its efficiency and its reliance on phoneme conversion, moving toward a fully end-to-end single-stage system. The second is scale across speakers and languages: YourTTS in 2022 showed zero-shot multi-speaker and cross-lingual synthesis on a VITS base, and Meta's 2023 Massively Multilingual Speech release extended VITS-style synthesis to 1,107 languages, many with little prior speech technology. The third is competition from much larger generative speech models that treat audio as sequences of tokens, which trade VITS's compactness for flexibility in voice, emotion and style. Open problems for the VITS family include robust handling of text without reliable phoneme dictionaries, controllable emotion and emphasis, evaluation beyond small listening tests on read audiobook speech, and safeguards against cloning voices without consent. Whether compact flow-and-GAN models like VITS will keep their role on phones and embedded devices as larger models become more efficient is an open question.

Het in de praktijk brengen

Waarom het ertoe doet

  • Toegang en bereik. Het verbetert de toegankelijkheid via transcriptie, gesproken tekst en spraakinterfaces.
  • Kosten en budget. Mediateams kunnen met kleinere budgetten sneller gepolijste audio leveren.
  • Snelheid en schaal. Klantgerichte systemen kunnen gesproken interacties op grotere schaal verwerken.

Pas op voor

  • Het risico op stemmisbruik en imitatie neemt toe als de toestemming ontbreekt.
  • De nauwkeurigheid kan afnemen bij accenten, dialecten of luidruchtige omgevingen.
  • Synthetische audio kan worden aangezien voor authentieke spraak zonder duidelijke labels.

Volgende stappen

  1. Verkrijg expliciete toestemming voor het vastleggen, klonen en hergebruiken van spraak.
  2. Test de kwaliteit van diverse sprekers en achtergrondomstandigheden.
  3. Bepaal wanneer een mens de output moet beoordelen of goedkeuren.
  4. Label synthetische audio en houd de herkomstgegevens bij voor verantwoording.

Bronnen en verdere literatuur

  1. arXiv (ICML 2021)Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
  2. arXiv (NeurIPS 2020)Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
  3. arXiv (NeurIPS 2020)HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
  4. arXiv (ICML 2022)YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone
  5. arXiv (Interspeech 2023)VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design
  6. arXiv (Meta AI)Scaling Speech Technology to 1,000+ Languages
  7. Hugging Face TransformersVITS model documentation

Blijf verkennen

Gratis nieuwsbrief

Ontvang de dagelijkse AI-briefing

Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.

Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.

Test jezelf

Take the VITS End-to-End Speech Synthesis quiz

Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.

Quiz starten

Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie

Veelgestelde vragen

What is VITS End-to-End Speech Synthesis?

VITS is a text-to-speech model, published at ICML 2021, that turns text directly into an audio waveform in one network instead of the usual two-stage pipeline of spectrogram predictor plus vocoder. It scored 4.43 out of 5 in listening tests against 4.46 for real recordings, runs about 67 times faster than real time on one GPU, and underpins Meta's speech models for 1,107 languages.

Waar staat de afkorting VITS voor?

VITS staat voor Variational Inference met vijandig leren voor end-to-end tekst-naar-spraak en weerspiegelt de drie kerningrediënten.

Wat is het belangrijkste architectonische verschil tussen VITS- en traditionele TTS-pijpleidingen?

VITS is end-to-end: het leert tekst-naar-golfvorm in één model, waardoor de discrepantie tussen een afzonderlijk akoestisch model en een afzonderlijke vocoder wordt vermeden.

Hoe leert VITS de uitlijning tussen tekst- en audioframes?

VITS maakt gebruik van Monotonic Alignment Search om intern de beste tekst-naar-frame-uitlijning te ontdekken, zonder dat een externe uitlijning nodig is.

Waarom bevat VITS een stochastische duurvoorspeller?

De stochastische duurvoorspeller laat dezelfde zin met verschillende natuurlijke ritmes uitspreken, waardoor een vlakke, robotachtige cadans wordt vermeden.

Welk onderdeel stuurt de VITS-uitvoer naar realistisch klinkende audio?

VITS maakt gebruik van adversarial (GAN) training, waarbij een discriminator beoordeelt of golfvormen echt klinken, waardoor de perceptuele kwaliteit wordt verbeterd.