| --- |
| license: apache-2.0 |
| tags: [tiny, synthetic, sorted-tokens, runtime-verification] |
| --- |
| |
| # Consecutive-token tiny llama (14.2M params) |
|
|
| Causal LM trained from scratch on consecutive token-id runs |
| ``[k, k+1, ...]`` mod 50000 (ids 0..49999; 49999 wraps to 0). Greedy |
| decoding continues the run exactly, which makes it a hand-checkable oracle |
| for verifying inference runtimes (vLLM, etc.) token-for-token against the |
| HuggingFace reference. |
|
|
| - Architecture: `llama` (random init, trained 3000 steps, |
| lr 0.0005, batch 16, boundary-crossing frac 0.25) |
| - Data: fully synthetic, generated on the fly (`train.py::consecutive_seq`) |
| - Eval (greedy exact-match of the hand-computed continuation): |
| in-distribution `1.000`, |
| longer-than-train `1.000` |
|
|
