File size: 743 Bytes
7abee35 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | ---
license: apache-2.0
tags: [tiny, synthetic, sorted-tokens, runtime-verification]
---
# Consecutive-token tiny llama (14.2M params)
Causal LM trained from scratch on consecutive token-id runs
``[k, k+1, ...]`` mod 50000 (ids 0..49999; 49999 wraps to 0). Greedy
decoding continues the run exactly, which makes it a hand-checkable oracle
for verifying inference runtimes (vLLM, etc.) token-for-token against the
HuggingFace reference.
- Architecture: `llama` (random init, trained 3000 steps,
lr 0.0005, batch 16, boundary-crossing frac 0.25)
- Data: fully synthetic, generated on the fly (`train.py::consecutive_seq`)
- Eval (greedy exact-match of the hand-computed continuation):
in-distribution `1.000`,
longer-than-train `1.000`
|