nshuster/gemma4-31b-augmental-sft-ep2

gemma4-31b-augmental-sft — epoch 2

Full-parameter SFT of google/gemma-4-31B (base) on Heralax/Augmental-Dataset (7,831 rows, visual-novel style multi-character roleplay dialogue).

This is the end-of-epoch-2 checkpoint (480/720 steps). Sibling repos: epoch 1 (-ep1), epoch 3 (-ep3).

Eval

Checkpoint	eval_loss	eval_ppl
base (step 0)	1.753	5.77
epoch 1 (-ep1)	1.572	4.81
epoch 2 (this repo)	1.521	4.58

Prompt format

Trained with a plain-text scenario format (no chat template). Prompt the model exactly like this, then let it continue after the trailing speaker tag:

Scenario: {scenario description}

{Speaker A}: "…"
{Speaker B}: "…"
{target speaker}:

Generation ends with <eos>.

Training config

• 8× H200, DeepSpeed ZeRO-3, bf16, SDPA attention (FlashAttention-2 is incompatible with Gemma-4's 512-dim global-attention layers)
• seq len 4096, global batch 32 (micro 2 × accum 2 × 8 GPUs)
• lr 5e-6 cosine, warmup 25 steps, 3 epochs total, prompt tokens masked (train_on_inputs: false)
• Trained with axolotl

License

Gemma derivatives are governed by the Gemma Terms of Use, including the Gemma Prohibited Use Policy.