nshuster/gemma4-31b-augmental-sft-ep3

gemma4-31b-augmental-sft — epoch 3 (final)

Full-parameter SFT of google/gemma-4-31B (base) on Heralax/Augmental-Dataset (7,831 rows, visual-novel style multi-character roleplay dialogue).

This is the end-of-epoch-3 (final, step 720) checkpoint. Sibling repos: epoch 1 (-ep1), epoch 2 (-ep2).

Eval

Checkpoint	eval_loss	eval_ppl
base (step 0)	1.753	5.77
epoch 1 (-ep1)	1.572	4.81
epoch 2 (-ep2)	1.521 (best held-out)	4.58
epoch 3 (this repo)	1.707	5.51

Epoch 3 overfits the training set (train loss ~0.42 at end): held-out loss is worse than epochs 1–2, but it imitates the dataset's style most strongly. Pick by use-case; -ep2 generalizes best.

Prompt format

Trained with a plain-text scenario format (no chat template). Prompt the model exactly like this, then let it continue after the trailing speaker tag:

Scenario: {scenario description}

{Speaker A}: "…"
{Speaker B}: "…"
{target speaker}:

Generation ends with <eos>.

Training config

• 8× H200, DeepSpeed ZeRO-3, bf16, SDPA attention (FlashAttention-2 is incompatible with Gemma-4's 512-dim global-attention layers)
• seq len 4096, global batch 32 (micro 2 × accum 2 × 8 GPUs)
• lr 5e-6 cosine, warmup 25 steps, 3 epochs, prompt tokens masked (train_on_inputs: false)
• Trained with axolotl; 720 steps in 1h09m

License

Gemma derivatives are governed by the Gemma Terms of Use, including the Gemma Prohibited Use Policy.