概述
它從 2004 年雅馬哈基於樣本的 VOCALOID 發展到 DiffSinger 和 VISinger 等神經系統,現在它為音樂製作工具提供動力,同時提出了有關語音權利和深度偽造的新問題。
重點摘要
- 歌聲合成根據歌詞、音高和音符時長而不是純文字產生演唱音訊。
- Yamaha 的 VOCALOID 於 2004 年首次與 Leon 和 Lola 音庫一起出售,它透過連接真實歌手錄製的簡短樣本來建立歌曲。
- 現代神經系統預測音素持續時間、音高曲線和梅爾頻譜圖,然後使用聲碼器產生最終波形。
- DiffSinger 的淺層擴散技巧從粗略的預測開始降噪,並將推理時間縮短了 45.1%,同時提高了聽眾的收視率。
- 在 2023 年 SingFake 研究中,經過口語 Deepfake 訓練的偵測器在合成歌聲方面表現不佳,因此歌聲需要自己的偵測模型。
深入探討
Singing voice synthesis is the singing counterpart of text-to-speech. Instead of only a sentence to read, the system receives a score: the lyrics, the note each syllable falls on, how long each note lasts, and often tempo, dynamics and expression marks. Its job is to produce audio of a specific voice singing that score in tune and in time. Singing is harder than speech for several reasons. Pitch must land on exact notes, vowels are stretched over long durations, vibrato and breath must sound natural, and the vocal range covered is much wider than in conversation.
Modern neural systems usually split the work into stages. First, a front end converts lyrics into phonemes (the individual speech sounds) and aligns each phoneme with a note. Second, a duration model decides how many audio frames each phoneme receives inside its note; consonants are short and vowels absorb most of the note length. Third, an acoustic model predicts a mel-spectrogram, a picture of how energy is spread across frequencies over time, along with the fundamental frequency (F0) curve that defines pitch, including vibrato and slides between notes. Fourth, a vocoder turns that spectrogram into an audio waveform. Some newer systems merge the last two stages and generate audio directly from the score.
The first commercial generation used concatenation. Yamaha began developing VOCALOID in 2000, revealed it at Musikmesse in Frankfurt in 2003, and shipped the first products, the English voicebanks Leon and Lola by Zero-G, at the NAMM Show in 2004. As described by Yamaha engineers Hideki Kenmochi and Hayato Ohshita at Interspeech 2007, VOCALOID stitched together short samples recorded from a real singer and adjusted their pitch and timing to match the score. Users edited pitch curves and expression parameters by hand in a piano-roll editor. VOCALOID2 and the Hatsune Miku voicebank arrived in 2007, and versions 3, 4 and 5 followed in 2011, 2014 and 2018, according to Yamaha's official timeline.
The second generation was statistical. The Sinsy project at Nagoya Institute of Technology modeled singing with hidden Markov models (HMMs), probabilistic models that generate smooth parameter trajectories. Statistical models needed less storage than sample libraries and could produce any note smoothly, but they tended to sound muffled. Its developers later rebuilt Sinsy around deep neural networks, reporting in IEEE/ACM Transactions on Audio, Speech, and Language Processing (2021) that the DNN version improved timing, vibrato and pitch accuracy and scored higher in listening tests. That version has four modules: a time-lag model, a duration model, an acoustic model and a pitch-robust neural vocoder called PeriodNet.
The third generation is deep generative modeling, borrowed largely from speech synthesis. XiaoiceSing (2020) adapted the FastSpeech text-to-speech architecture, adding note pitch and length to each phoneme, a residual connection so predicted pitch stays near the written note, and a loss that keeps the summed phoneme durations equal to the note length. Against a convolutional baseline, its authors reported mean opinion score (MOS, a 1 to 5 listener rating) gains of 1.44 for sound quality and 1.38 for naturalness. DiffSinger from Zhejiang University (AAAI 2022) used a diffusion model, which learns to turn random noise into a mel-spectrogram step by step, and introduced a "shallow diffusion" shortcut that starts from a rough prediction instead of pure noise. VISinger (ICASSP 2022) adapted the end-to-end VITS speech model to produce waveforms directly from lyrics and score.
The approaches trade off differently. Concatenative systems preserve the exact timbre of the recorded singer but can produce audible joins and need large, carefully recorded libraries. HMM systems are compact and controllable but sound buzzy or flat. Neural acoustic models produce far more natural output but need hours of clean, annotated singing; DiffSinger's PopCS dataset, for example, holds 117 Mandarin pop songs, 5.89 hours from one female vocalist, recorded in a studio and annotated with phoneme alignments and pitch. A separate family, singing voice conversion, takes an existing sung recording and changes whose voice it sounds like; it needs no score but depends on a human performance as input.
The limits are both technical and social. Data is scarce because singing must be recorded dry, without accompaniment, and labeled note by note. Models trained on one singer struggle with styles outside that data, such as growls, falsetto or extreme ranges. Pitch errors are easy to hear, which is why Sinsy added automatic pitch correction for out-of-tune training phrases. Expressive control is still often done by hand, as in the VOCALOID editors. On the social side, convincing synthetic singing enables voice cloning of real artists. The SingFake study (ICASSP 2024) found that four leading detectors built for spoken deepfakes performed poorly on singing, and that even detectors retrained on singing struggled with unseen singers, languages and audio codecs.
In practice, training a research model is modest by modern AI standards. DiffSinger's authors trained on one Nvidia V100 GPU, with about 16 hours for a warm-up stage and 12 for the main stage, and the model generated a second of mel-spectrogram in 0.191 seconds before vocoding. The harder cost is data: recording, cleaning and aligning hours of a consenting singer's voice. Commercial products bundle that work into voicebanks that musicians license and control through note editors inside digital audio workstations.
04工作範例
透過神經 SVS 管道追蹤一個歌唱短語
在兩個四分音符上為歌詞「hello」寫出樂譜,先是 C4,然後是 E4,每分鐘 120 拍,因此每個音符持續 0.5 秒。
將歌詞轉換為 h、e、l、ou 等音素,並將 h 和 e 指派給第一個音符,將 l 和 ou 指派給第二個音符。
讓時長模型將每個 0.5 秒的音符分割成幀,為短輔音提供幾幀,為元音提供其餘幀。
讓音高模型產生一條 F0 曲線,該曲線對於 C4 來說位於 261.6 Hz 附近,然後向上滑動,並且對於帶有輕顫音的 E4 來說在 329.6 Hz 附近穩定下來。
將音素、幀計數和 F0 輸入到聲學模型中,產生覆蓋一秒的 80 組梅爾頻譜圖。
將梅爾頻譜圖傳遞給神經聲碼器以渲染波形。
它顯示了什麼
輸出是所選聲音以兩個音符唱“hello”的大約一秒的音頻。透過在音訊上運行音高追蹤器並確認 F0 與 C4 和 E4 的匹配在幾分之內進行檢查,並在音符變化時聆聽乾淨的輔音計時。
現實世界的實施
Crypton Future Media於2007年8月31日發布了雅馬哈VOCALOID2引擎的初音未來語音庫,讓用戶可以透過輸入音符和歌詞來製作由虛擬角色演唱的歌曲。
雅馬哈的 VOCALOID:AI 計畫為已故日本歌手美空雲雀 (Misora Hibari) 去世 30 週年的 2019 年 NHK 特別紀念活動重現了她的聲音。
雅馬哈於 2022 年發布的 VOCALOID6 為長期運行的產品線添加了基於人工智慧的引擎,隨後於 2024 年推出了新的人工智慧語音庫。
名古屋工業大學的 Sinsy 系統從統計 HMM 模型開始,並圍繞深度神經網路進行了重建,其開發人員報告稱,到 2021 年,其計時、顫音和音調精度將得到提高。
研究人員於 2023 年發布了 SingFake,這是一個由 40 名歌手的 28.93 小時真實演唱和 29.40 小時人工智慧生成的演唱組成的數據集,用於訓練深度偽造演唱檢測器。
歌聲合成的未來
Research directions visible in recent work include end-to-end systems that skip the separate vocoder, as VISinger did in 2022 and as the DiffSinger authors listed as future work; multilingual and cross-lingual voices, which VOCALOID3 began commercially in 2011 with five languages; and finer expressive control so musicians can shape breath, vibrato and timbre without hand-drawing every curve. Data remains the bottleneck, so methods that learn from fewer hours of a singer or from mixed recordings separated by source-separation tools are active areas. Commercial products have moved toward AI engines. Yamaha released VOCALOID6 with an AI-based engine in 2022 and distributed the experimental VX-beta as a free VOCALOID6 add-on in 2024, according to its official history. In parallel, detection research is growing: the SingFake authors (2023) reported that unseen singers, languages, codecs and musical contexts remain hard for detectors. How consent, licensing and attribution for synthetic singing voices will be handled, especially when a voice resembles a living or deceased artist, is an open question for both industry and regulators.
付諸實踐
為什麼這很重要
- 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
- 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
- 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。
留意
- 如果未徵得同意,語音濫用和冒充風險就會增加。
- 由於口音、方言或嘈雜的環境,準確性可能會下降。
- 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。
後續步驟
- 獲得語音捕獲、克隆和重用的明確同意。
- 測試不同揚聲器和背景條件下的品質。
- 定義人員必須審查或批准輸出的時間。
- 標記合成音訊並保留來源記錄以供問責。
資料來源與延伸閱讀
- ISCA Archive (Interspeech 2007)VOCALOID - 基於樣本串聯的商業歌唱合成器
- Yamaha CorporationVOCALOID 20週年歷史
- arXiv (IEEE/ACM TASLP 2021)Sinsy:基於深度神經網路的歌聲合成系統
- arXiv小冰Sing:高品質、一體化的歌聲合成系統
- arXiv (AAAI 2022)DiffSinger:透過淺擴散機制進行歌聲合成
- arXiv (ICASSP 2022)VISinger:用於端對端歌聲合成的對抗性學習的變分推理
- arXiv (ICASSP 2024)SingFake:歌声 Deepfake 检测
不斷探索
常見問題
什么是歌声合成?
歌聲合成 (SVS) 是一種將樂譜(即歌詞加上帶有音高和節奏的音符)轉換為演唱聲音錄音的技術。它從 2004 年雅馬哈基於樣本的 VOCALOID 發展到 DiffSinger 和 VISinger 等神經系統,現在它為音樂製作工具提供動力,同時提出了有關語音權利和深度偽造的新問題。
樂譜條件歌聲系統使用哪些核心輸入?
歌词和乐谱提供了内容和时间目标。歌手身份調節是可選的並且取決於系統。
雅马哈在 2003 年公开宣布了什么?
雅馬哈研究員 Hideki Kenmochi 回憶道,該公司於 2003 年宣布正在開發名為 VOCALOID 的歌唱合成技術。
F0 轮廓在浊音歌唱中代表什么?
F0 追蹤有聲幀的基頻;它代表音高輪廓,而不是保證精確的音高估計。
在所描述的兩階段神經歌唱合成設定中,聲學模型可以在聲碼器之前產生什麼?
在這個兩階段設計中,聲學模型預測梅爾頻譜圖,聲碼器將該表示轉換為波形音訊。
DiffSinger 如何在其聲學模型中使用擴散?
DiffSinger 引入了一種淺擴散機制,用於根據樂譜條件生成歌唱聲學特徵;這描述了本文的模型,而不是每個 SVS 系統。