概述
諸如 TimbreTron (2018)、Google Magenta 的 DDSP (2020) 以及 IRCAM 的 RAVE (2021) 等研究系統使其成為可行,而 Tone Transfer 等瀏覽器工具則將其帶給了非專業人士。
重點摘要
- 音色轉換會在保留表演的音高、節奏與響度的同時,將其音色替換為另一種樂器的音色。
- Google Magenta 的 DDSP 透過萃取出的音高與響度來驅動諧波加雜訊合成器,其示範模型僅使用了不到 13 分鐘的音訊。
- TimbreTron 透過對常數 Q 頻譜圖(CQT)應用 CycleGAN 並使用 WaveNet 重建音訊來轉換音色。
- RAVE 報告指出,其在筆電 CPU 上的 48 kHz 合成速度比即時快約 20 倍,從而實現了即時音色轉換。
- 音高驅動的方法一次只能處理一個音符,因此和弦與完整的混音仍然很難處理。
深入探討
Timbre is the quality that makes two sounds with the same pitch and loudness sound different, the reason a violin and a flute playing the same note at the same volume are easy to tell apart. Physically, it comes from how energy is spread across the harmonics, the whole-number multiples of a note's fundamental frequency, how that spread changes over the course of a note, and the noisy parts of the sound such as breath or bow scrape. Timbre transfer is the task of keeping a performance's musical content, which notes are played, when and how loudly, while replacing its timbre with that of a target instrument. It is the audio counterpart of image style transfer, where the content of a photo is kept and its style is changed.
Most systems follow the same three-step plan. First, they separate content from timbre, either with explicit features such as pitch and loudness curves or with a learned representation that is trained to ignore the source instrument. Second, they generate a new representation of the target instrument conditioned on that content. Third, they turn the representation back into a waveform. The approaches differ mainly in how they perform each step.
An early approach treated audio as an image. TimbreTron, posted in November 2018 by Sicong Huang and colleagues at the University of Toronto and presented at ICLR 2019, converted audio to a constant-Q transform (CQT), a time-frequency picture whose frequency bins are spaced like musical notes, so shifting pitch roughly shifts the picture. A CycleGAN, an image translation network that learns from unpaired examples of two domains, changed the CQT from one instrument's style to another's, and a conditional WaveNet, a neural vocoder, synthesized the final waveform. Human listeners judged that pitch, rhythm and loudness were preserved while timbre was recognizably transferred. In May 2018, Noam Mor and colleagues at Facebook described the Universal Music Translation Network, which used a single shared encoder for all instruments and a WaveNet autoencoder trained directly on waveforms, and reported convincing translations even from domains not seen in training, such as whistling.
Google Magenta's Differentiable Digital Signal Processing (DDSP), posted on January 14, 2020 by Jesse Engel, Lamtharn Hantrakul, Chenjie Gu and Adam Roberts and published at ICLR 2020, took a different route. Instead of having a neural network output every audio sample, it has the network control a classic synthesizer. An encoder extracts the fundamental frequency, using a pretrained pitch tracker called CREPE, and the loudness of the input. A decoder network turns those curves into synthesizer settings: the amplitude of up to 100 harmonics and the shape of a filtered noise signal, frame by frame. A harmonic-plus-noise synthesizer then produces the sound, and a learned reverb adds the room. Because every step is differentiable, the whole chain can be trained by comparing spectrograms of the output and the original. To transfer timbre, the pitch and loudness from a singer are fed into a decoder trained on violin, and the violin synthesizer plays the singer's melody.
DDSP's design makes timbre transfer data-efficient. Magenta reported that all the models in its demonstrations were trained on less than 13 minutes of audio, and the paper's violin model used 13 minutes of solo violin from the MusOpen royalty-free library. RAVE, a variational autoencoder by Antoine Caillon and Philippe Esling posted in November 2021, learns a compact latent representation in a first training stage and then fine-tunes its decoder adversarially, producing 48 kHz audio and running about 20 times faster than real time on a laptop CPU. Feeding one instrument's audio into a RAVE model trained on another gives real-time timbre transfer.
Compared with alternatives, timbre transfer occupies a middle ground. Traditional audio effects such as filters or pitch-tracking synthesizers can change tone in real time but rarely sound like a convincing acoustic instrument. Transcribing audio to MIDI and replaying it with a sampled instrument keeps notes but loses expressive detail such as vibrato and slides, which DDSP-style systems preserve because they follow the continuous pitch and loudness curves. Fully generative music models can produce any instrument but are harder to steer note by note.
The limitations are well defined. Systems driven by a single pitch curve, as DDSP's harmonic synthesizer is, model one note at a time, so chords and mixed recordings need source separation or a different method. Transfer quality depends on the source performance staying within the target instrument's plausible range: a voice singing below a flute's range or a vocal glide that a piano cannot produce will yield unrealistic results. Pitch trackers make errors on noisy, breathy or very fast passages, and those errors are passed straight to the output. Learned-representation systems such as RAVE can carry some of the source instrument's character into the output when the training data does not fully separate timbre from content. Evaluation is also hard, because there is no correct answer to compare against, so most papers rely on listening tests.
In practice, timbre transfer is accessible. The DDSP library is open source on GitHub, Tone Transfer runs in a browser, and RAVE's source code and examples are public. Training a personal DDSP model on an instrument needs only minutes of clean, single-instrument recordings, though recordings with one instrument in one room give the best results because DDSP's violin model used a single fixed reverb for its single acoustic environment.
04工作範例
使用 DDSP 將哼唱的旋律變成小提琴
在安靜的房間錄製一段短小、乾淨的哼唱旋律,最好一次只唱一個音符。
使用像 CREPE 這樣的音高追蹤器提取基頻曲線,並從音訊中取得音量曲線。
將音高曲線與目標音域比較,若旋律低於小提琴的最低音,則將其上移一個八度。
將音高與音量曲線餵入在獨奏小提琴上訓練的 DDSP 解碼器,即可取得每幀的諧波振幅與噪聲濾波器設定。
使用諧波加雜訊合成器合成波形並套用模型學習的混響效果。
聆聽結果並將其音高曲線與原始曲線比較,以確認音調與節奏已被保留。
它顯示了什麼
學習者聽到以小提琴般音色演奏的哼唱旋律,跟隨原曲的滑音與颤音。匹配的音高曲線證實內容被保留,任何瑕疵通常可追溯至呼吸音或不清晰音符的音高追蹤錯誤。
現實世界的實施
在瀏覽器中將聲音轉為樂器:Google 的 Tone Transfer,於 2020 年 10 月 1 日由 Magenta 與 AIUX 團隊發布,允許使用者錄製或上傳聲音,並將其重新渲染為薩克斯風、長笛等樂器,完全在瀏覽器中使用 TensorFlow.js 執行 DDSP 模型。
小資料量樂器模型:Magenta 在 2020 年 1 月的 DDSP 公告展示了聲音轉小提琴、小提琴轉長笛以及 mbira 轉小提琴、長笛和泰國 salo,每個模型的訓練資料少於 13 分鐘。
即時表現:IRCAM 研究人員的 RAVE 模型報告在標準筆記型電腦 CPU 上以 48 kHz 速度運行,速度為實時的 20 倍,使得樂器或聲音的即時音色轉移成為可能。
跨域翻譯:Facebook 研究人員的 Universal Music Translation Network(2018)使用一個共享編碼器和 WaveNet 自編碼器,在沒有配對訓練樣本的情況下,將樂器與風格之間進行翻譯,包括從口哨聲。
多聲部轉移:TimbreTron(ICLR 2019)將圖像風格的 CycleGAN 轉移應用於常數 Q 频谱圖,並且人類聽者判斷音色已明顯改變,且音樂內容在單線和多聲部樣本中均被保留。
音樂音色傳輸的未來
Timbre transfer research since 2020 has pushed in three directions. One is speed and deployment: RAVE showed real-time 48 kHz synthesis on a laptop CPU in 2021, and Magenta's 2020 Tone Transfer announcement said the team planned to let users train their own DDSP models and run them on phones, audio plugins and websites. Another is polyphony, since the most data-efficient methods model one pitched line at a time, while real recordings often contain chords and several instruments. A third is control, letting musicians adjust how much of the source's character is kept and how much of the target's is imposed. Open problems remain. There is no standard objective measure of transfer quality, so comparisons depend on listening tests with different protocols. Systems trained on one recording setup often degrade on different microphones and rooms. Expressive techniques that do not exist in the target instrument, such as vocal consonants mapped onto a violin, have no obvious correct rendering. There are also questions outside engineering. Timbre models trained on a specific performer's recordings can imitate that performer's sound, which raises questions about consent and credit similar to those around voice cloning. Small-data methods such as DDSP, which needed under 13 minutes of audio in Magenta's demonstrations, make such models easy to build, so norms for training data and disclosure are likely to matter as much as technical progress.
付諸實踐
為什麼這很重要
- 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
- 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
- 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。
留意
- 如果未徵得同意,語音濫用和冒充風險就會增加。
- 由於口音、方言或嘈雜的環境,準確性可能會下降。
- 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。
後續步驟
- 獲得語音捕獲、克隆和重用的明確同意。
- 測試不同揚聲器和背景條件下的品質。
- 定義人員必須審查或批准輸出的時間。
- 標記合成音訊並保留來源記錄以供問責。
資料來源與延伸閱讀
- arXiv (ICLR 2020)DDSP:可微分數位訊號處理
- Google MagentaDDSP:可微分數位訊號處理
- Google Magenta音色轉換
- arXiv (ICLR 2019)TimbreTron:一個 WaveNet(CycleGAN(CQT(Audio)))管線,用於音樂音色轉移
- arXiv一個 Universal Music Translation Network
- arXivRAVE:一個快速且高品質的變分自編碼器,用於神經音訊合成
不斷探索
常見問題
什麼是音樂音色轉錄?
音樂音色轉移會改變錄音的音色,使其聽起來像不同的樂器或聲音,同時保持原始音高、節奏和動態,例如將歌唱旋律轉換為小提琴線。像 TimbreTron(2018)、Google Magenta 的 DDSP(2020)和 IRCAM 的 RAVE(2021)等研究系統使其變得實用,而像 Tone Transfer 這樣的瀏覽器工具則將其帶給非專業人士。
在音色傳輸中,原始音訊中保留了什麼?
音色傳輸保留內容、音調、響度和節奏,同時交換音色(樂器的音調特徵)。
「音色」實際上指的是什麼?
音色是小提琴和小號即使在相同的音高和音量下聽起來也不同的原因,它是聲音的頻譜特徵。
是什麼讓 Google 的 DDSP 方法特別具有資料效率?
通过嵌入可微分的真实 DSP 组件(振荡器、滤波器),DDSP 需要的训练数据少得多,并且可以从短单声道录音中进行概括。
為什麼 DDSP 中的合成器必須是「可微分的」?
可微性使頻譜損失透過合成步驟反向傳播,因此網路學會設定與目標聲音相符的合成器參數。
DDSP 通常從輸入效能中提取哪兩個控制訊號?
DDSP 透過擷取的音調(基頻)曲線和隨時間變化的響度包絡來驅動其振盪器組。