Musikalische Klangfarbenübertragung

Timbre is the quality that makes two sounds with the same pitch and loudness sound different, the reason a violin and a flute playing the same note at the same volume are easy to tell apart. Physically, it comes from how energy is spread across the harmonics, the whole-number multiples of a note's fundamental frequency, how that spread changes over the course of a note, and the noisy parts of the sound such as breath or bow scrape. Timbre transfer is the task of keeping a performance's musical content, which notes are played, when and how loudly, while replacing its timbre with that of a target instrument. It is the audio counterpart of image style transfer, where the content of a photo is kept and its style is changed.

Most systems follow the same three-step plan. First, they separate content from timbre, either with explicit features such as pitch and loudness curves or with a learned representation that is trained to ignore the source instrument. Second, they generate a new representation of the target instrument conditioned on that content. Third, they turn the representation back into a waveform. The approaches differ mainly in how they perform each step.

An early approach treated audio as an image. TimbreTron, posted in November 2018 by Sicong Huang and colleagues at the University of Toronto and presented at ICLR 2019, converted audio to a constant-Q transform (CQT), a time-frequency picture whose frequency bins are spaced like musical notes, so shifting pitch roughly shifts the picture. A CycleGAN, an image translation network that learns from unpaired examples of two domains, changed the CQT from one instrument's style to another's, and a conditional WaveNet, a neural vocoder, synthesized the final waveform. Human listeners judged that pitch, rhythm and loudness were preserved while timbre was recognizably transferred. In May 2018, Noam Mor and colleagues at Facebook described the Universal Music Translation Network, which used a single shared encoder for all instruments and a WaveNet autoencoder trained directly on waveforms, and reported convincing translations even from domains not seen in training, such as whistling.

Google Magenta's Differentiable Digital Signal Processing (DDSP), posted on January 14, 2020 by Jesse Engel, Lamtharn Hantrakul, Chenjie Gu and Adam Roberts and published at ICLR 2020, took a different route. Instead of having a neural network output every audio sample, it has the network control a classic synthesizer. An encoder extracts the fundamental frequency, using a pretrained pitch tracker called CREPE, and the loudness of the input. A decoder network turns those curves into synthesizer settings: the amplitude of up to 100 harmonics and the shape of a filtered noise signal, frame by frame. A harmonic-plus-noise synthesizer then produces the sound, and a learned reverb adds the room. Because every step is differentiable, the whole chain can be trained by comparing spectrograms of the output and the original. To transfer timbre, the pitch and loudness from a singer are fed into a decoder trained on violin, and the violin synthesizer plays the singer's melody.

DDSP's design makes timbre transfer data-efficient. Magenta reported that all the models in its demonstrations were trained on less than 13 minutes of audio, and the paper's violin model used 13 minutes of solo violin from the MusOpen royalty-free library. RAVE, a variational autoencoder by Antoine Caillon and Philippe Esling posted in November 2021, learns a compact latent representation in a first training stage and then fine-tunes its decoder adversarially, producing 48 kHz audio and running about 20 times faster than real time on a laptop CPU. Feeding one instrument's audio into a RAVE model trained on another gives real-time timbre transfer.

Compared with alternatives, timbre transfer occupies a middle ground. Traditional audio effects such as filters or pitch-tracking synthesizers can change tone in real time but rarely sound like a convincing acoustic instrument. Transcribing audio to MIDI and replaying it with a sampled instrument keeps notes but loses expressive detail such as vibrato and slides, which DDSP-style systems preserve because they follow the continuous pitch and loudness curves. Fully generative music models can produce any instrument but are harder to steer note by note.

The limitations are well defined. Systems driven by a single pitch curve, as DDSP's harmonic synthesizer is, model one note at a time, so chords and mixed recordings need source separation or a different method. Transfer quality depends on the source performance staying within the target instrument's plausible range: a voice singing below a flute's range or a vocal glide that a piano cannot produce will yield unrealistic results. Pitch trackers make errors on noisy, breathy or very fast passages, and those errors are passed straight to the output. Learned-representation systems such as RAVE can carry some of the source instrument's character into the output when the training data does not fully separate timbre from content. Evaluation is also hard, because there is no correct answer to compare against, so most papers rely on listening tests.

In practice, timbre transfer is accessible. The DDSP library is open source on GitHub, Tone Transfer runs in a browser, and RAVE's source code and examples are public. Training a personal DDSP model on an instrument needs only minutes of clean, single-instrument recordings, though recordings with one instrument in one room give the best results because DDSP's violin model used a single fixed reverb for its single acoustic environment.