README.md

# DragonCode-150M

DragonCode family — 150M-parameter code-domain language model (pretraining tier).

---

## ⚠️ Training reset — issue log

**Date:** 2026-08-25

**What happened:** All prior pretraining checkpoints (steps `2000` → `46000`, ~780M tokens) were **deleted and discarded**. Pretraining is being **restarted from step 0** against the clean code-domain token cache.

**Why (root cause):** The early checkpoints were produced while the training script contained **four critical bugs**, which made the saved weights unsafe to continue from:

1. **Gradient-accumulation dead-loop** — `(step+1) % grad_acc == 0` used the *resumed* `step` counter (misaligned), so `optimizer.step()` never fired and gradients accumulated indefinitely (looked like a hang; GPU pegged at 100%).
2. **LR decayed to 0** — the LR schedule computed decay from the raw (resumed) step counter instead of token progress, so the learning rate collapsed to `0.0` after resume.
3. **`bytes` not JSON-serializable** — `TrainState.rng_torch` (`bytes`) crashed checkpoint save.
4. **`allow_duplicate_filename` param** — `huggingface_hub 1.24.0`'s `upload_folder()` has no such arg, causing checkpoint push to retry-and-fail.

Additionally, an earlier script-version of the pipeline had trained on **generic web corpora** (`fineweb-edu`, `SlimPaj627B`, `The-Pile`) instead of the intended **code-domain** sources (`codeparrot-clean`, `codeforces-cots`) — further invalidating those checkpoints.

**Decision:** Rather than resume from contaminated optimizer/weights state, all checkpoints were purged and training restarts clean from zero. Tokenized code-domain data (`DragonCode-Tokenized-Pretrain`, ~3.42B tokens) is retained and unaffected.

---