Clonage vocal multilingue XTTS

XTTS is a zero-shot, multi-speaker, multilingual text-to-speech (TTS) model. Text-to-speech means turning written text into spoken audio. Zero-shot means the model can imitate a voice it never heard during training, given only a short reference recording. Cross-lingual means the reference can be in one language and the output in another, so an English speaker's voice can read Spanish or Japanese text. The model was built at Coqui, a speech startup, and described in the paper "XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model" by Edresson Casanova and colleagues, posted to arXiv on 7 June 2024. The paper covers 16 languages. The public XTTS-v2 checkpoint on Hugging Face lists 17, adding Hindi, says a 6-second clip is enough to clone a voice, and outputs 24 kHz audio.

The architecture has three parts. First, a vector-quantized variational autoencoder (VQ-VAE), a small network with 13 million parameters, compresses a mel-spectrogram (a picture of sound energy across time and pitch bands) into a stream of discrete codes at 21.53 codes per second, using one codebook of 8,192 entries that the authors pruned to the 1,024 most frequent. Second, a GPT-2 style transformer with 443 million parameters reads the input text, split into tokens by a 6,681-entry byte-pair-encoding vocabulary, and predicts those audio codes one after another. Third, a HiFi-GAN vocoder with 26 million parameters turns the transformer's internal states into a waveform.

Voice identity enters in two places. A conditioning encoder reads the reference clip's mel-spectrogram through six attention layers and a Perceiver Resampler, a module that always outputs a fixed set of 32 vectors of 1,024 numbers regardless of clip length. Those vectors steer the transformer. The paper reports that the single embedding used by the earlier Tortoise model weakened cloning once many languages were trained together, which is why the fixed set of 32 was added. The vocoder is separately conditioned on a speaker embedding from a speaker-recognition model and trained with a speaker consistency loss that rewards output sounding like the reference. Because voice comes from the conditioning path and language comes from the text, the model can mix them freely, and the authors stress that no parallel recordings of the same speaker in two languages were needed.

XTTS sits in a short lineage. Tortoise, released by James Betker in May 2023, applied autoregressive transformers and diffusion to speech and was trained on 49,000 hours of English. VALL-E, posted to arXiv in January 2023, treated TTS as language modeling over EnCodec tokens with 8 codebooks at 75 frames per second. Multilingual zero-shot systems existed but were narrow: the XTTS paper counts three languages for YourTTS, two each for VALL-E X and Mega-TTS 2, and six for Voicebox. XTTS reused the Tortoise design and scaled it to 16 languages with a 27,281.6-hour dataset. English made up 14,513.1 hours, mostly audiobook-style internal data plus LibriTTS-R and LibriLight, while most other languages came from Mozilla Common Voice. Training ran for about 2.5 million steps on four NVIDIA A100 80 GB GPUs.

The results are measured with three automatic metrics. Character error rate (CER) is computed by transcribing the output with Whisper Large v3; lower means clearer pronunciation. UTMOS is a model that predicts a human naturalness rating on a 1 to 5 scale. Speaker encoder cosine similarity (SECS) compares voice fingerprints of output and reference; 1.0 would be identical. In English, XTTS had the lowest CER in the paper's table, 0.54, against 0.68 for StyleTTS 2 and 0.77 for HierSpeech++, with UTMOS 4.01 and SECS 0.64. HierSpeech++ scored higher on naturalness (4.46) and similarity (0.65). In listening tests with at least 15 raters per comparison, listeners preferred XTTS for naturalness over HierSpeech++ (CMOS +0.41) and Mega-TTS 2 (+0.92) but rated its speaker similarity slightly lower (-0.31 and -0.39). Averaged over 16 languages, XTTS reached CER 2.06 and SECS 0.50, against 4.79 and 0.47 for a YourTTS model trained on the same data.

Compared with alternatives, XTTS trades some voice similarity for breadth. Monolingual English models such as HierSpeech++ matched voices more closely in the paper's tests, which the authors attribute to the difficulty of massive multilingual training. Against VALL-E, XTTS generates far fewer tokens per second of audio, 21.53 versus 75 frames, which the authors cite as a speed advantage. Against fine-tuning a model on one speaker, zero-shot cloning is instant but less faithful: fine-tuning on about 10 minutes of speech raised cross-lingual SECS from 0.59 to 0.72 for the authors' eight test voices.

The limitations are visible in the numbers. Error rates are much higher for some languages: CER was 5.37 for Japanese, 5.20 for Chinese and 4.06 for Korean, and those three languages are romanized before tokenization. Several languages had little data, for example 52.4 hours of Czech and 57.3 of Japanese. Cross-lingual similarity stayed around 0.46 to 0.54 SECS, well below the 0.90 measured between real recordings of the same English speakers. Output is sampled, so two runs on the same sentence differ; the paper used temperature 0.75 and a repetition penalty of 10 to keep generation stable. Reference clips with unusual recording conditions or voices far from the training data are harder to clone, as the authors note. Voice cloning also carries obvious consent and impersonation risks, and any use should involve the speaker's permission.

In practice the model is free to download but not free to use commercially. The XTTS-v2 weights are released under the Coqui Public Model License, which a maintainer of the community fork quotes as allowing only non-commercial use of the model and its outputs. The original coqui-ai/TTS repository is described by that fork, maintained under the idiap GitHub organization, as unmaintained; the fork ships as the coqui-tts package, and since version 0.27.4 PyTorch must be installed separately. By the paper's counts the three components total about 482 million parameters, and the reference audio needs only a few seconds of clean speech.