Yuan-Chiao Cheng (Music Informatics Group, Georgia Institute of Technology) and Zhiyao Duan (University of Rochester)
Instrument timbre transfer from an audio reference clip, driven by two plug-in control modules on a pretrained music generator: a pitch module that reads a binary piano roll through decoupled cross-attention, and a timbre module that reads a fine-tuned CLAP embedding through adaptive layer norm.
Each row takes the notes of the source clip and the timbre of the reference clip. Source and reference come from different datasets and different instruments; none of the recordings was seen in training. Baselines that read the timbre from audio: CTD, SS-VQ-VAE, TokenSynth-A. Baselines that name the target instrument by text: TokenSynth-T, MuseControlLite, DDIM-inv, DDPM-inv. Headphones recommended.
| source → target | Source | Reference | MuseTimbre (ours) | CTD | SS-VQ-VAE | TokenSynth-A | TokenSynth-T | MuseControlLite | DDIM-inv | DDPM-inv |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 bassoon → piano | ||||||||||
| 2 electric guitar → piano | ||||||||||
| 3 electric guitar → horn | ||||||||||
| 4 viola → electric guitar | ||||||||||
| 5 viola → saxophone | ||||||||||
| 6 clarinet → viola | ||||||||||
| 7 clarinet → electric guitar | ||||||||||
| 8 electric guitar → clarinet | ||||||||||
| 9 double bass → clarinet | ||||||||||
| 10 clarinet → flute | ||||||||||
| 11 flute → violin | ||||||||||
| 12 piano → violin |
The pitch module accepts a symbolic score in place of a transcription, and the generator's original text conditioner still works on top of the transferred timbre. The last column adds the text prompt shown under the player.
| example | Source audio | Reference | Ours, pitch from audio | Ours, pitch from MIDI score | Ours, MIDI + text prompt |
|---|---|---|---|---|---|
| Guitar to electric piano guitar (GOAT) → electric piano |
“in a large reverberant concert hall” | ||||
| Guitar to organ guitar (GOAT) → organ |
“with heavy rain and distant thunder in the background” | ||||
| Violin to distorted guitar violin (URMP) → distorted guitar |
“in a huge cathedral with a long reverb tail” | ||||
| Flute to electric piano flute (URMP) → electric piano |
“lo-fi vintage radio recording, warm and distant” |
Submitted to ICASSP 2027. Link and BibTeX will appear here.