README.md

DragonCode-150M

DragonCode family — 150M-parameter code-domain language model (pretraining tier).

---

⚠️ Training reset — issue log

Date: 2026-08-25

What happened: All prior pretraining checkpoints (steps 2000 → 46000, ~780M tokens) were deleted and discarded. Pretraining is being restarted from step 0 against the clean code-domain token cache.

Why (root cause): The early checkpoints were produced while the training script contained four critical bugs, which made the saved weights unsafe to continue from:

1. Gradient-accumulation dead-loop — (step+1) % grad_acc == 0 used the resumed step counter (misaligned), so optimizer.step() never fired and gradients accumulated indefinitely (looked like a hang; GPU pegged at 100%).
2. LR decayed to 0 — the LR schedule computed decay from the raw (resumed) step counter instead of token progress, so the learning rate collapsed to 0.0 after resume.
3. bytes not JSON-serializable — TrainState.rng_torch (bytes) crashed checkpoint save.
4. allow_duplicate_filename param — huggingface_hub 1.24.0's upload_folder() has no such arg, causing checkpoint push to retry-and-fail.

Additionally, an earlier script-version of the pipeline had trained on generic web corpora (fineweb-edu, SlimPaj627B, The-Pile) instead of the intended code-domain sources (codeparrot-clean, codeforces-cots) — further invalidating those checkpoints.

Decision: Rather than resume from contaminated optimizer/weights state, all checkpoints were purged and training restarts clean from zero. Tokenized code-domain data (DragonCode-Tokenized-Pretrain, ~3.42B tokens) is retained and unaffected.

---