| # DragonCode-150M | |
| DragonCode family — 150M-parameter code-domain language model (pretraining tier). | |
| --- | |
| ## ⚠️ Training reset — issue log | |
| **Date:** 2026-08-25 | |
| **What happened:** All prior pretraining checkpoints (steps `2000` → `46000`, ~780M tokens) were **deleted and discarded**. Pretraining is being **restarted from step 0** against the clean code-domain token cache. | |
| **Why (root cause):** The early checkpoints were produced while the training script contained **four critical bugs**, which made the saved weights unsafe to continue from: | |
| 1. **Gradient-accumulation dead-loop** — `(step+1) % grad_acc == 0` used the *resumed* `step` counter (misaligned), so `optimizer.step()` never fired and gradients accumulated indefinitely (looked like a hang; GPU pegged at 100%). | |
| 2. **LR decayed to 0** — the LR schedule computed decay from the raw (resumed) step counter instead of token progress, so the learning rate collapsed to `0.0` after resume. | |
| 3. **`bytes` not JSON-serializable** — `TrainState.rng_torch` (`bytes`) crashed checkpoint save. | |
| 4. **`allow_duplicate_filename` param** — `huggingface_hub 1.24.0`'s `upload_folder()` has no such arg, causing checkpoint push to retry-and-fail. | |
| Additionally, an earlier script-version of the pipeline had trained on **generic web corpora** (`fineweb-edu`, `SlimPaj627B`, `The-Pile`) instead of the intended **code-domain** sources (`codeparrot-clean`, `codeforces-cots`) — further invalidating those checkpoints. | |
| **Decision:** Rather than resume from contaminated optimizer/weights state, all checkpoints were purged and training restarts clean from zero. Tokenized code-domain data (`DragonCode-Tokenized-Pretrain`, ~3.42B tokens) is retained and unaffected. | |
| --- | |