---
license: apache-2.0
library_name: pytorch
tags:
  - causal-lm
  - mla
  - muon
  - historical
---

# TinyLM v1 Checkpoint (historical)

Single MLA+Muon training run from the v1 TinyLM effort (RunPod A100-80GB,
May 2026). Trained on **1B unique** FineWeb-Edu tokens **repeated ~21×**
over 20k steps — the data bug the HPC re-run later fixed.

**Preserved here for historical contrast — not the recommended model.**

- **Recommended model:** [`Shiv-22/tinylm`](https://huggingface.co/Shiv-22/tinylm) (Run D from the HPC re-run, 8B unique tokens, +3.97 avg pts above this v1 on the same architecture)
- **Full 4-arm ablation checkpoints:** [`Shiv-22/tinylm-checkpoints-v2`](https://huggingface.co/Shiv-22/tinylm-checkpoints-v2)
- **Source code:** [github.com/shivnarainms22/TinyLM](https://github.com/shivnarainms22/TinyLM)

## v1 eval (0-shot)

| Benchmark | Metric | v1 D |
|---|---|:---:|
| HellaSwag | acc_norm | 37.1% |
| ARC-Easy | acc_norm | 48.4% |
| LAMBADA | acc | 29.2% |
| Winogrande | acc | 50.0% |
| **Average** | | **41.18%** |

The notably weak LAMBADA (long-range coherence) was the main signal that
repeated data was hurting; the HPC re-run with 8B unique tokens lifted
LAMBADA to 36.81% (+7.61) on the same arm.

## License

Apache 2.0.