Audio AI ÚTMUTATÓ

VITS End-to-End beszédszintézis

VITS is a text-to-speech model, published at ICML 2021, that turns text directly into an audio waveform in one network instead of the usual two-stage pipeline of spectrogram predictor plus vocoder.

Hallgassa meg ezt az útmutatótAI-narrált · ~7 min

Ezen az oldalon8 perc olvasás
  1. Áttekintés
  2. Főbb tanulságok
  3. Mély merülés
  4. Generate and vary speech with a VITS model
  5. Valós megvalósítás
  6. A VITS end-to-end beszédszintézis jövője
  7. A gyakorlatba ültetés
  8. Források és további olvasmányok
  9. Folytassa a felfedezést
  10. Gyakran ismételt kérdések

Áttekintés

It scored 4.43 out of 5 in listening tests against 4.46 for real recordings, runs about 67 times faster than real time on one GPU, and underpins Meta's speech models for 1,107 languages.

Főbb tanulságok

  1. VITS generates waveforms directly from text in one model, removing the separate vocoder used by two-stage text-to-speech systems.
  2. On LJ Speech, VITS scored 4.43 mean opinion score against 4.46 for real recordings in its 2021 ICML paper.
  3. Monotonic alignment search lets VITS learn which audio frames match which phoneme without any external alignment tool.
  4. A stochastic duration predictor lets VITS speak the same sentence with different rhythms, so output varies unless a random seed is fixed.
  5. Meta's 2023 Massively Multilingual Speech text-to-speech models for 1,107 languages use the VITS architecture.

Mély merülés

VITS stands for Variational Inference with adversarial learning for end-to-end Text-to-Speech. It was introduced by Jaehyeon Kim, Jungil Kong and Juhee Son in the paper "Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech", posted in June 2021 and presented at ICML 2021. Its main idea is to generate the final audio samples directly from text in a single model trained in one stage. Earlier neural text-to-speech systems were split in two: an acoustic model such as Tacotron 2 or Glow-TTS predicted a mel spectrogram, a compact picture of sound energy over time and frequency, and a separate vocoder such as HiFi-GAN converted that picture into a waveform. Two-stage systems sounded good but had to be trained and tuned separately, and errors in the first stage passed into the second.

VITS is built around a conditional variational autoencoder (VAE). An autoencoder learns to compress data into a hidden code, called a latent variable, and rebuild it; a variational one learns a probability distribution over codes, so new samples can be drawn. During training, a posterior encoder reads the linear spectrogram of a real recording and produces a latent sequence z describing that audio. A decoder, built like the HiFi-GAN generator, turns z back into a waveform. Separately, a text encoder reads the phonemes, the basic speech sounds of the sentence, and predicts a distribution for each one. A normalizing flow, a chain of invertible transformations, bends the audio latent into the shape the text encoder predicts, so text and sound meet in a shared space. Training pushes the two sides to agree while the decoder learns to reproduce the recording.

Two problems remain: which audio frames belong to which phoneme, and how long each phoneme should last. For alignment, VITS reuses monotonic alignment search (MAS), introduced in Glow-TTS at NeurIPS 2020. MAS uses dynamic programming to find the most likely left-to-right assignment of audio frames to phonemes, with no external aligner. For duration, VITS adds a stochastic duration predictor, a small flow-based model that learns a distribution of possible lengths rather than one fixed value. The same sentence can therefore be spoken with different rhythms on different runs. Finally, adversarial training adds a discriminator network that tries to tell real audio from generated audio, and the generator improves by fooling it, which sharpens fine detail.

At synthesis time the path is short. Text becomes phonemes, the text encoder predicts a distribution per phoneme, the duration predictor samples how many frames each phoneme gets, a latent sequence is sampled, the flow is run in reverse, and the decoder outputs audio. The VITS paper reports a speed of about 1,480 kHz on one NVIDIA V100 GPU, meaning roughly 1.48 million audio samples per second, or 67 times faster than real time, compared with about 27 times for Glow-TTS plus HiFi-GAN measured the same way.

The quality results explain its influence. On LJ Speech, a public dataset of 13,100 clips from a single reader totaling about 24 hours, listeners rated VITS 4.43 on a 5-point mean opinion score (MOS) scale, against 4.46 for the real recordings, 4.32 for Glow-TTS with a fine-tuned HiFi-GAN and 4.25 for Tacotron 2 with a fine-tuned HiFi-GAN. An ablation, a test with one component removed, showed how much the flow mattered: without it, MOS fell from 4.50 to 2.98. The paper also trained a multi-speaker version on VCTK, a dataset of 109 English speakers. Follow-ups built directly on the code: YourTTS added zero-shot voice cloning across languages, and Meta's Massively Multilingual Speech project, posted in May 2023, released text-to-speech for 1,107 languages using the VITS architecture.

Compared with alternatives, VITS sits between older two-stage systems and newer large generative models. Against two-stage pipelines it offers simpler training and faster synthesis with equal or better quality on standard tests. Against autoregressive models such as Tacotron 2, which produce output one step at a time, it generates all frames in parallel and is less prone to skipping or repeating words, because MAS forces a strictly left-to-right alignment. Large language-model-style speech generators trained on far more data can imitate voices and emotions more flexibly, but they are heavier to run. VITS remains attractive where a small, fast, single-voice or few-voice model is enough.

VITS has known limits. VITS2's authors list intermittent unnaturalness, computational inefficiency and strong dependence on phoneme conversion, meaning text must first be turned into phonemes by a separate tool, which breaks on unusual names and languages without good pronunciation rules. Output is random by design; the Transformers documentation tells users to set a seed for reproducible audio. Expressiveness is bounded by training data: a model trained on one calm reader will sound like one calm reader. Meta's MMS data came from readings of religious texts, which shapes vocabulary and speaking style. As with any voice synthesis, models fine-tuned on a real person's voice raise consent and impersonation concerns.

What VITS requires in practice is modest by current standards. The original model was trained with mixed precision on 4 NVIDIA V100 GPUs, a batch of 64 per GPU, for up to 800,000 steps, using about 24 hours of clean single-speaker audio. Fine-tuning a pretrained checkpoint to a new voice needs far less; YourTTS reported useful results from under a minute of speech. Running a trained model needs only a single GPU or a capable CPU, and the Hugging Face implementation produces 16 kHz audio by default for MMS checkpoints with a few lines of code.

04Működő példa

Generate and vary speech with a VITS model

  1. Install the Transformers library and load VitsTokenizer and VitsModel from the facebook/mms-tts-eng checkpoint.

  2. Tokenize a sentence such as "Turn left at the second traffic light." and call set_seed(555) before running the model under torch.no_grad().

  3. Save outputs.waveform at the model's sampling rate (16 kHz for MMS) as a WAV file and listen to it.

  4. Run it again with the same seed and confirm the audio is identical, then change the seed and note small differences in rhythm.

  5. Set model.speaking_rate to 0.8 and regenerate to hear slower speech, then raise model.noise_scale_duration to hear more varied timing.

  6. Compare the number of output samples across runs to see how duration sampling changes total length.

Mit mutat

Several WAV files of the same sentence with controlled differences in speed and rhythm. Check by comparing durations: a speaking rate of 0.8 should produce a clip about 25% longer, and identical seeds should produce byte-identical audio.

Valós megvalósítás

Meta's Massively Multilingual Speech project released text-to-speech models for 1,107 languages in 2023; Hugging Face Transformers loads them as VITS models, for example facebook/mms-tts-eng.

YourTTS (ICML 2022) extends VITS for zero-shot multi-speaker and multilingual speech, and its authors report strong voice similarity after fine-tuning on less than one minute of a new speaker's audio.

Developers adjust delivery at inference time through the speaking_rate, noise_scale and noise_scale_duration settings exposed by the Transformers VitsModel, for example slowing narration for language learners.

VITS is a standard research baseline on the LJ Speech dataset of about 24 hours of one reader, where the original paper reported a mean opinion score of 4.43.

VITS2 (Interspeech 2023) from Jungil Kong and colleagues improves on VITS's occasional unnatural passages, speeds up training and inference, and reduces dependence on phoneme conversion.

A VITS end-to-end beszédszintézis jövője

Work after VITS has followed three lines. The first is refinement of the same design: VITS2, published at Interspeech 2023, targeted the original's occasional unnatural passages, its efficiency and its reliance on phoneme conversion, moving toward a fully end-to-end single-stage system. The second is scale across speakers and languages: YourTTS in 2022 showed zero-shot multi-speaker and cross-lingual synthesis on a VITS base, and Meta's 2023 Massively Multilingual Speech release extended VITS-style synthesis to 1,107 languages, many with little prior speech technology. The third is competition from much larger generative speech models that treat audio as sequences of tokens, which trade VITS's compactness for flexibility in voice, emotion and style. Open problems for the VITS family include robust handling of text without reliable phoneme dictionaries, controllable emotion and emphasis, evaluation beyond small listening tests on read audiobook speech, and safeguards against cloning voices without consent. Whether compact flow-and-GAN models like VITS will keep their role on phones and embedded devices as larger models become more efficient is an open question.

A gyakorlatba ültetés

Miért számít

  • Hozzáférés és elérés. Javítja a hozzáférhetőséget az átírás, a narráció és a hangfelületek révén.
  • Költség és költségvetés. A médiacsapatok kisebb költségvetéssel gyorsabban szállíthatják a csiszolt hanganyagot.
  • Sebesség és méretarány. Az ügyfélközpontú rendszerek nagyobb léptékben képesek feldolgozni a beszélt interakciókat.

Vigyázz

  • A beleegyezés hiányában nő a hanggal való visszaélés és a megszemélyesítés kockázata.
  • A pontosság csökkenhet az akcentusok, dialektusok vagy zajos környezetben.
  • A szintetikus hang összetéveszthető a hiteles beszéddel egyértelmű címkézés nélkül.

Következő lépések

  1. Kérjen kifejezett hozzájárulást a hangrögzítéshez, klónozáshoz és újrafelhasználáshoz.
  2. Tesztelje a minőséget különféle hangszórókon és háttérviszonyok között.
  3. Határozza meg, mikor kell egy embernek felülvizsgálnia vagy jóváhagynia a kimeneteket.
  4. Címkézze fel a szintetikus hanganyagot, és vezessen származási nyilvántartást az elszámoltathatóság érdekében.

Források és további olvasmányok

  1. arXiv (ICML 2021)Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
  2. arXiv (NeurIPS 2020)Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
  3. arXiv (NeurIPS 2020)HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
  4. arXiv (ICML 2022)YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone
  5. arXiv (Interspeech 2023)VITS2: Improving Quality and Efficiency of Single-Stage Text-to-Speech with Adversarial Learning and Architecture Design
  6. arXiv (Meta AI)Scaling Speech Technology to 1,000+ Languages
  7. Hugging Face TransformersVITS model documentation

Folytassa a felfedezést

Ingyenes hírlevél

Szerezze meg a napi AI összefoglalót

Minden hétköznap reggel három ellenőrzött AI történet, egyszerű angol nyelven. Örökké ingyen, hirdetések nélkül.

Minden hétköznap egy e-mail. Egy kattintással lemondhat. Soha nem értékesítjük vagy osztjuk meg az Ön címét.

Teszteld magad

Take the VITS End-to-End Speech Synthesis quiz

Azonnali visszajelzés minden válaszra, és megosztható tanúsítvány egy ellenőrizhető azonosítóval, miután sikeresen teljesít egy kurzust.

Kezdő kvíz

Támogassa az ingyenes AI oktatást. AI Understanding egy 501(c)(3) nonprofit – nincs hirdetés, nincs fizetőfal, soha. Tegyen adományt

Gyakran ismételt kérdések

Mi az a VITS végponttól végpontig beszédszintézis?

VITS is a text-to-speech model, published at ICML 2021, that turns text directly into an audio waveform in one network instead of the usual two-stage pipeline of spectrogram predictor plus vocoder. It scored 4.43 out of 5 in listening tests against 4.46 for real recordings, runs about 67 times faster than real time on one GPU, and underpins Meta's speech models for 1,107 languages.

Mit jelent a VITS mozaikszó?

A VITS a Variational Inference (Variational Inference) rövidítése, amely a végpontok közötti szövegfelolvasó (text-to-speech) ellentmondásos tanulását jelenti, tükrözve annak három alapvető összetevőjét.

Mi a fő építészeti különbség a VITS és a hagyományos TTS csővezetékek között?

A VITS végponttól végpontig működik: egy modellben tanulja meg a szöveget hullámformává, elkerülve a különálló akusztikus modell és a vokóder közötti eltéréseket.

Hogyan tanulja meg a VITS a szöveg- és hangkockák igazítását?

A VITS a Monotonic Alignment Search segítségével fedezi fel a legjobb szöveg-keret igazítást belsőleg, külső igazítás nélkül.

Miért tartalmaz a VITS sztochasztikus időtartam-előrejelzőt?

A sztochasztikus időtartam-előrejelző lehetővé teszi ugyanazt a mondatot különböző természetes ritmusokkal, elkerülve a lapos, robotos ütemet.

Melyik komponens tolja el a VITS kimenetet a valósághű hangzás felé?

A VITS adversariális (GAN) képzést alkalmaz, ahol a diszkriminátor megítéli, hogy a hullámformák valódiak-e, javítva az észlelési minőséget.