metadata
license: apache-2.0
library_name: pytorch
tags:
- causal-lm
- mla
- muon
- historical
TinyLM v1 Checkpoint (historical)
Single MLA+Muon training run from the v1 TinyLM effort (RunPod A100-80GB, May 2026). Trained on 1B unique FineWeb-Edu tokens repeated ~21× over 20k steps — the data bug the HPC re-run later fixed.
Preserved here for historical contrast — not the recommended model.
- Recommended model:
Shiv-22/tinylm(Run D from the HPC re-run, 8B unique tokens, +3.97 avg pts above this v1 on the same architecture) - Full 4-arm ablation checkpoints:
Shiv-22/tinylm-checkpoints-v2 - Source code: github.com/shivnarainms22/TinyLM
v1 eval (0-shot)
| Benchmark | Metric | v1 D |
|---|---|---|
| HellaSwag | acc_norm | 37.1% |
| ARC-Easy | acc_norm | 48.4% |
| LAMBADA | acc | 29.2% |
| Winogrande | acc | 50.0% |
| Average | 41.18% |
The notably weak LAMBADA (long-range coherence) was the main signal that repeated data was hurting; the HPC re-run with 8B unique tokens lifted LAMBADA to 36.81% (+7.61) on the same arm.
License
Apache 2.0.