training_dataset_info.md

<center>
<h1 style="font-size:34px;"><u>Foundation-1 Training & Dataset Notes</u></h1>
</center>

<center>
<i>High-level overview of the training setup, dataset composition, and design philosophy behind Foundation-1.</i>
</center>

<br>

This page provides a high-level overview of the **training setup**, **dataset composition**, and **design philosophy** behind Foundation-1.

It is intended as a companion to the main model page, which focuses on capabilities, prompting, and audio examples.

---

<center>

## Training Summary

</center>

Foundation-1 was trained as a structured **text-to-sample diffusion model** designed for **music production workflows**, rather than general-purpose music captioning.

### Hardware

- **GPUs:** 2 × NVIDIA RTX A6000  
- **System RAM:** 128 GB  

### Training Run

- **Training stopped at step:** **183,474**

### Audio Configuration

- **Sample Rate:** **44,100 Hz**
- **Bit Depth:** **16-bit**
- **Channels:** **Stereo**
- **Sample Length:** **882,000 samples (~20 seconds)**

---

<center>

## Technical Training Notes

</center>

Foundation-1 was fine-tuned from **`stabilityai/stable-audio-open-1.0`** using a diffusion transformer architecture with structured prompt conditioning.

### Model Architecture

- **Backbone:** Diffusion Transformer (DiT)
- **Transformer Depth:** 24 layers
- **Attention Heads:** 24
- **Embedding Dimension:** 1536
- **Conditioning Dimension:** 768
- **Text Encoder:** `t5-base`

### Optimization

- **Optimizer:** AdamW
- **Learning Rate:** `5e-5`
- **Weight Decay:** `1e-3`
- **Scheduler:** InverseLR
- **EMA:** Enabled

---

<center>

## Dataset Overview

</center>

### Dataset Totals

- **Total WAV files:** **3,784,862**
- **Total Dataset Size:** **7.100 TiB**

---

<center>

## Important Note on Dataset Scale

</center>

While the dataset appears quite large, its scale reflects the **post-augmentation training set**, not a flat count of completely unique and unrelated one-off melodic phrases.

All samples were **hand-labeled first**, then processed through a **controlled augmentation pipeline** designed to expand sonic and conditioning coverage.

As a result, a single melodic phrase may be represented across many different contexts, including variations in:

- timbre
- FX treatment
- key
- BPM
- loop structure
- tonal emphasis

This design was intentional. The goal was not simply to maximize the number of isolated phrases, but to teach the model how **musical ideas translate across different sonic identities and production scenarios**.

---

<center>

## Dataset and Training Philosophy

</center>

Foundation-1 was built around a **structured sample-generation philosophy**, rather than generic or genre-based audio captioning.

The dataset consists entirely of **hand-labeled audio**, organized around a layered prompt and conditioning system designed to reflect how producers actually think about sound.

At a high level, the training design emphasizes:

- structured musical loops
- instrument hierarchy
- explicit timbre representation
- dedicated FX descriptors
- notation-aware prompt terms
- key / tempo / bar-aware looping
- strong production relevance
- broad reuse for compositional workflows

This design is central to the model’s **musical coherence**, **prompt controllability**, and **production-facing behavior**.

---

<center>

## Why the Dataset Was Structured This Way

</center>

Most audio generation systems treat prompts as broad descriptive captions.

Foundation-1 instead was trained to understand sound as a **layered system with separable controls**.

That structure includes:

**Instrument Identity**  
Broad family and sub-family control over what kind of sound is being generated.

**Timbre**  
Direct conditioning over tonal character, texture, density, brightness, width, grit, warmth, and other sonic traits.

**FX Context**  
Dedicated processing descriptors such as reverb, delay, distortion, phasing, and bitcrushing.

**Musical Structure**  
Notation-aware terms that encourage coherent phrasing, melodic behavior, rhythmic structure, and harmonic motion.

**Timing and Tonality**  
Explicit support for BPM, bar count, and key-aware sample generation.

This layered design is one of the main reasons Foundation-1 can produce outputs that feel both **musically structured** and **sonically steerable**.

---

<center>

## Coverage and Augmentation

</center>

The augmentation strategy was designed to improve **coverage**, **control**, and **generalization**.

Rather than treating each source phrase as a single static example, labeled material was expanded across multiple sonic and musical contexts.

This helps the model learn relationships between:

- melody and timbre
- phrase behavior and instrumentation
- sound design and FX treatment
- tonal setting and loop structure
- tempo and musical feel

In practice, this encourages the model to learn **reusable musical relationships**, rather than simply memorizing isolated recordings.

---

<center>

## Generalization and Overfitting

</center>

Because structured augmentation can increase dataset size rapidly, special care was taken to reduce melodic overfitting and encourage broader generalization.

The objective was to help the model learn:

- how similar phrase structures can exist across many timbral identities
- how sound design changes affect musical material
- how prompts can steer sonic outcomes without collapsing variety
- how the same conditioning vocabulary can remain useful across many production contexts

This is one reason Foundation-1 can produce **multiple distinct outputs from the same prompt** while still preserving the requested timbral and structural identity.

---

<center>

## What the Dataset Size Represents

</center>

The full size of the dataset should be understood as a measure of **coverage and structured variation**, not simply a count of unrelated melodies.

Foundation-1 was not designed around “more files for the sake of more files,” which is often seen with large scraped or loosely structured datasets.

Instead, the dataset was built from the ground up to teach the model how:

- phrases behave across different instruments
- timbral tags influence sonic identity
- FX descriptors shape the output
- timing and tonality influence loop behavior
- musical structure can remain coherent under many sonic conditions

In other words, the dataset size reflects the **breadth of the conditioning system** as much as it reflects the raw quantity of audio.

---

<center>

## Charts at a Glance

</center>

The following charts illustrate the distribution of key components of the Foundation-1 training dataset.

<center>

<table>
<tr>
<td align="center">
<img src="./Charts/families_pie.PNG" width="1100">
</td>

<td align="center">
<img src="./Charts/subfamilites_pie.PNG" width="1100">
</td>
</tr>

<tr>
<td align="center">
<img src="./Charts/timbre_tags_pie.PNG" width="1100">
</td>

<td align="center">
<img src="./Charts/fx_pie.PNG" width="1100">
</td>
</tr>
</table>

</center>

These visualizations provide a quick overview of the dataset’s coverage across:

- instrument families
- instrument sub-families
- timbral descriptors
- FX conditioning tags

---

<center>

## Scope

</center>

Foundation-1 is focused specifically on **sample-generation workflows**.

It was trained to generate:

- musical loops
- melodic phrases
- chordal material
- arps
- top-line ideas
- basslines
- textures
- production-ready instrumental content

It was **not designed** as:

- a full song generator
- a drum generator
- a general-purpose music captioning model

---

<center>

## Final Note

</center>

Foundation-1 was built to give producers structured control over:

- what the sound is
- how it behaves musically
- how it feels sonically
- how it sits in a production context

The dataset and training design were built around that exact goal.

For examples, prompting guidance, and model capabilities, see the **[main repository page](./README.md)**.

Additional information on the model and design philosophy can be found in the companion video:

🎥 **[Watch the Foundation-1 overview and design philosophy video](https://www.youtube.com/watch?v=O2iBBWeWaL8)**