viola → electric guitarphenicx → goat
Viola melody in the timbre of an electric guitar
0:00
ICASSP 2027 submission
MuseTimbre: Zero-Shot Timbre Transfer by Controlling a Frozen Music Generator
MuseTimbre plays the notes of a source recording in the timbre of an audio reference clip. Two plug-in control modules steer a frozen Stable Audio 3 Medium: a pitch module reads a piano roll through cross-attention, and a timbre module reads a fine-tuned CLAP embedding through adaptive layer norm.
Source and reference come from different instruments and different datasets, and none of the recordings was seen in training. All systems run on the same pairs. Clips are 5-second mono. Each example has one player: switch versions with the buttons below it, and playback continues from the same point.
MuseTimbre (ours)baselines reading timbre from audiobaselines naming the instrument by text
Zero-shot timbre transfer
The notes come from the source and the timbre from the reference. Audio-reference baselines: CTD, SS-VQ-VAE, TokenSynth-A. Text baselines: TokenSynth-T, MuseControlLite, DDIM-inv, DDPM-inv.
viola → saxophonephenicx → bach10
Viola melody in the timbre of a saxophone
0:00
electric guitar → pianogoat → maestro
Electric guitar melody in the timbre of a piano
0:00
bassoon → electric guitarphenicx → goat
Bassoon melody in the timbre of an electric guitar
0:00
electric guitar → clarinetgoat → bach10
Electric guitar melody in the timbre of a clarinet
0:00
violin → pianophenicx → maestro
Violin melody in the timbre of a piano
0:00
double bass → clarinetphenicx → bach10
Double bass melody in the timbre of a clarinet
0:00
clarinet → flutebach10 → phenicx
Clarinet melody in the timbre of a flute
0:00
flute → violinphenicx → bach10
Flute melody in the timbre of a violin
0:00
electric guitar → double bassgoat → phenicx
Electric guitar melody in the timbre of a double bass
0:00
oboe → electric guitarphenicx → goat
Oboe melody in the timbre of an electric guitar
0:00
MIDI input and text control
The pitch module takes a MIDI score in place of a transcription, and the backbone's original text control still works on top of the transferred timbre.
guitar (GOAT) → electric piano
“in a large reverberant concert hall” (text prompt for the last version)
0:00
guitar (GOAT) → organ
“with heavy rain and distant thunder in the background” (text prompt for the last version)
0:00
violin (URMP) → distorted guitar
“in a huge cathedral with a long reverb tail” (text prompt for the last version)
0:00
flute (URMP) → electric piano
“lo-fi vintage radio recording, warm and distant” (text prompt for the last version)
0:00
Paper and code
Paper: arXiv:2609.30548. Code: github.com/barry-mir/muse-timbre. Weights: huggingface.co/barry-mir/muse-timbre.