Extreme Overtraining in Tiny Language Models
But how much higher is safe? This post documents what happens when you push a 0.9M parameter model to a ratio of ~222,000 tokens per parameter. Short version: the model overtrained hard, and every metric peaked around 20B tokens (22K:1) before degrading for the remaining 160B tokens.
How this happened
We wanted to train a 0.9M parameter model and figured a 222K:1 ratio would give us maximum intelligence. It didn't.
The config:
- Architecture: 6 layers, 96 hidden dim, SwiGLU intermediate 380, GQA 6Q/2KV, 384 vocab, 8K context
- Optimizer: Muon (peak LR 7e-2) for 2D params, AdamW (peak LR 4e-3) for the rest
- Data: FineWeb-HQ + Cosmopedia v2
- Budget: 200B tokens
That's ~222,000 tokens per parameter — Chinchilla × ~220. The reasoning at the time: the vocab is very small (384), so each token carries less information than a normal 32K-vocab tokenizer, and the model should be able to absorb more raw tokens as a result. Small vocab also frees embedding parameters, leaving proportionally more budget for the transformer stack itself.
Checkpoints were evaluated on the standard Open SLM benchmark suite (ARC-Easy, ARC-Challenge, HellaSwag, PIQA) plus ArithMark-2/3, aggregated into an INT Index score.
What happened
The INT Index peaked at 4.55 at 20B tokens and then declined monotonically (with noise) to 3.31 at 180B tokens — a −27.3% loss across the last 160B tokens of training.
Individual benchmarks tell the same story. Comparing the 10% checkpoint (20B tokens) to the 90% final (180B tokens):
| Benchmark | 20B (10%) | 180B (90%) | Δ |
|---|---|---|---|
| ARC-Easy | 26.98 | 28.32 | +1.34 |
| PIQA | 53.54 | 52.07 | −1.47 |
| ARC-Challenge | 22.27 | 21.25 | −1.02 |
| HellaSwag | 29.01 | 28.06 | −0.95 |
| Average | 32.95 | 32.43 | −0.52 |
Three of four benchmarks are worse at 180B than they were at 20B. Only ARC-Easy improved, and by less than PIQA/HS/ARC-C each lost.
To rule out the possibility that the last-10% LR decay caused the drop, we also evaluated the 80% checkpoint (~160B tokens). It landed at avg 32.68 — much closer to the 90% final than to the 40% peak (33.44). Most of the damage was already done by 160B, so this isn't a schedule artifact a shorter cosine would fix. It's real overtraining.
The Chinchilla control
As a control, the same architecture was trained on 18M tokens — a proper Chinchilla ratio of ~20:1. Result: INT Index 1.53. Essentially chance-level on every benchmark (ARC-E 26.64, PIQA 49.78, ARC-C 26.54, HS 24.88).
So the useful range is bracketed on both sides:
- 20:1 (18M tokens): INT Index 1.53 — undertrained, chance-level
- 22K:1 (20B tokens): INT Index 4.55 — peak
- 200K:1 (180B tokens): INT Index 3.31 — overtrained, ~27% below peak
The useful token-per-parameter range
The good news: normal high overtraining ratios (~7K:1, ~15K:1, ~22K:1) work fine. Community models trained at these ratios routinely produce healthy scaling curves — TinyStories at ~10K:1, many small models on the sub-3M leaderboards at 15–30K:1. This is well-established territory.
The failure mode is what happens past that range. Here's the full shape from this run:
| Tokens | Ratio | INT Index |
|---|---|---|
| 18M | 20:1 | 1.53 (chance) |
| 10B | 11K:1 | 4.01 |
| 20B | 22K:1 | 4.55 (peak) |
| 40B | 44K:1 | 4.12 |
| 80B | 88K:1 | 4.13 |
| 160B | 175K:1 | 3.82 |
| 180B | 200K:1 | 3.31 |
The ascent from Chinchilla-optimal (chance-level) to peak is very fast — ~0.05 INT Index per B tokens between 10B and 20B. The descent from peak to overtrained wreckage is much slower — ~0.008 per B — but relentless. Every checkpoint past 20B was worse than 20B, with the trend line pointing straight down.
Takeaway
Based on this run, the useful compute-optimal range for a Pico-tier (~1M param) model looks like 22K–30K tokens per parameter. Below that (down to a few K:1), models are still climbing. Above that, benchmarks start rotting.
If you're planning a small-model training run, the practical advice is: start with a small budget (20K–30K tokens per parameter), evaluate, then decide whether to extend. Blindly scaling to 200K:1 costs a lot of GPU-hours to produce a worse model than the 20K:1 run would have.
