Timbral Transfer: Continuous Latent Diffusion for Multi-Instrument Audio Synthesis

Master's Thesis Project — California Institute of the Arts (CalArts)
Colton Arnold (@carnol16)

Overview

Timbral Transfer is a continuous latent diffusion transformer (DiT) system designed for neural musical instrument timbre conversion. The model accepts arbitrary melodic audio (whistling, singing, acoustic instruments), extracts its fundamental frequency ($F_0$) pitch contour via TorchCREPE and RMS loudness envelope, and synthesizes 44.1 kHz stereo audio in the target instrument's acoustic profile using the Stability AI Stable Audio continuous latent space.

Key Highlights

18 Slakh Instrument Classes: Full coverage across acoustic piano, guitars, strings, brass, woodwinds, organs, and synthesizers.
v2 High-Capacity Studio LoRAs ($r=32$): Fine-tuned on uncompressed solo performer tracks from Medley-solos-DB, delivering acoustic body resonance, breath turbulence, bow scrape, and up to 50% pitch error reduction.
Adaptive Hybrid Chunking: Dynamic onset density routing ($W=32$ for rapid riffs $>3.0$ onsets/s; $W=48$ for legato phrasing $\le 3.0$ onsets/s; single-pass for $\le 3.7$s) with RMS energy valley seam snapping.
GPU-Batched Single-Pass DDIM: Parallel denoises all temporal windows in ~35s for a 24s audio clip on an NVIDIA RTX 3080.
Dynamic RMS Calibration & Headroom Limiter: Prevents feature saturation on mastered inputs ($>0.04$ RMS) and preserves natural dynamic range with a $-0.5$ dBFS safety limiter.

Checkpoint Catalog

File / Folder	Architecture / Type	Parameters / Size	Description
`diT_best_val.pt`	Multi-Instrument DiT (Base)	114M (~690 MB)	Epoch 84 best validation loss checkpoint trained across 18 Slakh2100 instrument classes.
`lora_norm_v2/`	High-Capacity LoRAs ($r=32, \alpha=32$)	9.45 MB each	Fine-tuned on solo artist tracks with clean normalization and attention projection ranks.
↳ `violin_song121_r32_best.pt`	Soloist LoRA (Violin)	9.45 MB	Expressive vibrato, wooden body resonance, rich harmonic overtone series.
↳ `tenor_saxophone_song85_r32_best.pt`	Soloist LoRA (Saxophone)	9.45 MB	Acoustic reed bite, smoky subtone, tight intonation locking.
↳ `flute_song56_r32_best.pt`	Soloist LoRA (Concert Flute)	9.45 MB	Airy breath turbulence, pure sinusoidal harmonic profile.
↳ `trumpet_song94_r32_best.pt`	Soloist LoRA (Trumpet)	9.45 MB	Brassy harmonic drive, crisp attack flare, centered projection.
↳ `clarinet_song201_r32_best.pt`	Soloist LoRA (Clarinet)	9.45 MB	Hollow cylindrical woodwind acoustic profile, prominent odd harmonics.
`lora/`	Curated Adapters (v1 $r=8$ & v2 $r=32$)	2.38 MB / 9.45 MB	Backward-compatible mirrors for automated GUI preset downloaders.

Benchmark Results

1. Held-Out Test Set Performance (Slakh2100 & Medley-solos-DB)

Across held-out validation takes, the $r=32$ v2 LoRA suite substantially improves spectral reconstruction fidelity and pitch alignment:

Spectral Loss (MSSL): Reduced from $2.35 \to 2.13$ in high-noise registers.
F0 Correlation: Improved from $0.81 \to 0.94$.
Intonation Error: Median cents error reduced by up to $42%$.

##Quickstart & Inference

Web GUI (Gradio)

To run the interactive web application with automatic checkpoint downloading:

git clone https://github.com/carnol16/Thesis-timbral-transfer.git
cd Thesis-timbral-transfer/universal_deployable
bash install_linux.sh   # Or install_mac.sh / install_pc.bat
python run_app.py

Programmatic Python Download

from huggingface_hub import hf_hub_download

# Download base DiT checkpoint
base_model = hf_hub_download(
    repo_id="carnol16/timbral-transfer",
    filename="diT_best_val.pt",
    local_dir="checkpoints"
)

# Download high-capacity v2 Violin LoRA
violin_lora = hf_hub_download(
    repo_id="carnol16/timbral-transfer",
    subfolder="lora_norm_v2",
    filename="violin_song121_r32_best.pt",
    local_dir="checkpoints"
)

License & Attribution

Code & Model Weights: Non-Commercial Research & Educational License (CC-BY-NC-4.0).
Base Datasets: Slakh2100 (CC-BY 4.0), Medley-solos-DB.
Autoencoder: Stability AI Stable Audio 1.0 (SAME Autoencoder, MIT License).

Downloads last month: -; Downloads are not tracked for this model. How to track