歌声合成

Singing voice synthesis is the singing counterpart of text-to-speech. Instead of only a sentence to read, the system receives a score: the lyrics, the note each syllable falls on, how long each note lasts, and often tempo, dynamics and expression marks. Its job is to produce audio of a specific voice singing that score in tune and in time. Singing is harder than speech for several reasons. Pitch must land on exact notes, vowels are stretched over long durations, vibrato and breath must sound natural, and the vocal range covered is much wider than in conversation.

Modern neural systems usually split the work into stages. First, a front end converts lyrics into phonemes (the individual speech sounds) and aligns each phoneme with a note. Second, a duration model decides how many audio frames each phoneme receives inside its note; consonants are short and vowels absorb most of the note length. Third, an acoustic model predicts a mel-spectrogram, a picture of how energy is spread across frequencies over time, along with the fundamental frequency (F0) curve that defines pitch, including vibrato and slides between notes. Fourth, a vocoder turns that spectrogram into an audio waveform. Some newer systems merge the last two stages and generate audio directly from the score.

The first commercial generation used concatenation. Yamaha began developing VOCALOID in 2000, revealed it at Musikmesse in Frankfurt in 2003, and shipped the first products, the English voicebanks Leon and Lola by Zero-G, at the NAMM Show in 2004. As described by Yamaha engineers Hideki Kenmochi and Hayato Ohshita at Interspeech 2007, VOCALOID stitched together short samples recorded from a real singer and adjusted their pitch and timing to match the score. Users edited pitch curves and expression parameters by hand in a piano-roll editor. VOCALOID2 and the Hatsune Miku voicebank arrived in 2007, and versions 3, 4 and 5 followed in 2011, 2014 and 2018, according to Yamaha's official timeline.

The second generation was statistical. The Sinsy project at Nagoya Institute of Technology modeled singing with hidden Markov models (HMMs), probabilistic models that generate smooth parameter trajectories. Statistical models needed less storage than sample libraries and could produce any note smoothly, but they tended to sound muffled. Its developers later rebuilt Sinsy around deep neural networks, reporting in IEEE/ACM Transactions on Audio, Speech, and Language Processing (2021) that the DNN version improved timing, vibrato and pitch accuracy and scored higher in listening tests. That version has four modules: a time-lag model, a duration model, an acoustic model and a pitch-robust neural vocoder called PeriodNet.

The third generation is deep generative modeling, borrowed largely from speech synthesis. XiaoiceSing (2020) adapted the FastSpeech text-to-speech architecture, adding note pitch and length to each phoneme, a residual connection so predicted pitch stays near the written note, and a loss that keeps the summed phoneme durations equal to the note length. Against a convolutional baseline, its authors reported mean opinion score (MOS, a 1 to 5 listener rating) gains of 1.44 for sound quality and 1.38 for naturalness. DiffSinger from Zhejiang University (AAAI 2022) used a diffusion model, which learns to turn random noise into a mel-spectrogram step by step, and introduced a "shallow diffusion" shortcut that starts from a rough prediction instead of pure noise. VISinger (ICASSP 2022) adapted the end-to-end VITS speech model to produce waveforms directly from lyrics and score.

The approaches trade off differently. Concatenative systems preserve the exact timbre of the recorded singer but can produce audible joins and need large, carefully recorded libraries. HMM systems are compact and controllable but sound buzzy or flat. Neural acoustic models produce far more natural output but need hours of clean, annotated singing; DiffSinger's PopCS dataset, for example, holds 117 Mandarin pop songs, 5.89 hours from one female vocalist, recorded in a studio and annotated with phoneme alignments and pitch. A separate family, singing voice conversion, takes an existing sung recording and changes whose voice it sounds like; it needs no score but depends on a human performance as input.

The limits are both technical and social. Data is scarce because singing must be recorded dry, without accompaniment, and labeled note by note. Models trained on one singer struggle with styles outside that data, such as growls, falsetto or extreme ranges. Pitch errors are easy to hear, which is why Sinsy added automatic pitch correction for out-of-tune training phrases. Expressive control is still often done by hand, as in the VOCALOID editors. On the social side, convincing synthetic singing enables voice cloning of real artists. The SingFake study (ICASSP 2024) found that four leading detectors built for spoken deepfakes performed poorly on singing, and that even detectors retrained on singing struggled with unseen singers, languages and audio codecs.

In practice, training a research model is modest by modern AI standards. DiffSinger's authors trained on one Nvidia V100 GPU, with about 16 hours for a warm-up stage and 12 for the main stage, and the model generated a second of mel-spectrogram in 0.191 seconds before vocoding. The harder cost is data: recording, cleaning and aligning hours of a consenting singer's voice. Commercial products bundle that work into voicebanks that musicians license and control through note editors inside digital audio workstations.