Training status: step 256000

Checkpoint weights: step 256000 (1667MB)

Training status: step 240000

Checkpoint weights: step 240000 (1667MB)

Training status: step 224000

Checkpoint weights: step 224000 (1667MB)

Training status: step 208000

Checkpoint weights: step 208000 (1666MB)

Training status: step 192000

Checkpoint weights: step 192000 (1665MB)

Training status: step 176000

Checkpoint weights: step 176000 (1666MB)

Training status: step 160000

Checkpoint weights: step 160000 (1666MB)

Training status: step 144000

Checkpoint weights: step 144000 (1666MB)

Training status: step 128000

Checkpoint weights: step 128000 (1666MB)

Training status: step 112000

Checkpoint weights: step 112000 (1664MB)

Training status: step 96000

Checkpoint weights: step 96000 (1666MB)

Training status: step 80000

Checkpoint weights: step 80000 (1666MB)

Training status: step 64000

Checkpoint weights: step 64000 (1666MB)

Training status: step 48000

Checkpoint weights: step 48000 (1666MB)

Remove stale training_status.json from diverged run

Always pass --resume so train.py checks Hub when local is empty"

Auto-resume from Hub when local checkpoints are cleared by Kaggle\n\nNow --resume checks: local disk first → Hub download fallback → fresh start\nNo more lost progress across Kaggle sessions."

Training status: step 32000

Checkpoint weights: step 32000 (1668MB)

Training status: step 16000

Checkpoint weights: step 16000 (1671MB)

Fix divergence: peak_lr 1e-3 → 3e-4, max_grad_norm 1.0 → 0.5 (batch=8 too small for high LR)"

Delete stale checkpoint from diverged run

Training status: step 16000

Checkpoint weights: step 16000 (1688MB)

Fix data.py: handle edge cases, add fallback for failed datasets, fix orca-agentinstruct split names

CRITICAL FIX: Wire up real data pipeline (was training on random tokens!)\n\nChanges:\n- Replace random.randint batch with TAPDataPipeline streaming real data\n- Add tokenizer initialization with TAP special tokens\n- Add pad_fraction diagnostic to detect data issues\n- Keep upd_rms diagnostic\n- Stage-aware curriculum switching\n- Proper pad token masking in loss (was already in build_model_mtp)\n\nThis fixes the root cause of loss stuck at ln(vocab)=10.85"

Remove stale training_status.json from old run

Delete stale checkpoints from stuck training (loss=10.95, bad init + low LR)

Fix config.py default peak_lr to 1e-3

Fix stuck training: peak_lr 5e-4 → 1e-3, increase weight decay for stability

Fix stuck loss: increase LR to 1e-3, fix initialization scales, add diagnostics

Training status: step 176000

Checkpoint weights: step 176000 (1670MB)

Training status: step 160000

Checkpoint weights: step 160000 (1678MB)