Claim 01 — TinyStories byte-LM benchmark vs vanilla

Status: preliminary single-seed directional finding. Not a defensible architecture claim.

This file is a condensed, public-facing copy of the audit-rewritten BENCHMARKS.md from the working repo. The honest headline:

On the only Tilelli-Lite seed whose log.jsonl is preserved (seed 1234), the win against a 1-seed vanilla baseline depends on which best_val definition is used. Within-training periodic eval: Lite loses by 0.6 %. Post-training "final" extra eval: Lite wins by 0.4 % on a different validation-batch composition. Two additional Lite seeds (5678, 9012) were reported at 0.5679 and 0.5693 but their log.jsonl files are not preserved.

Why this isn't yet a real result

best_val was heterogeneous. The training loop saved min(within-training-best, post-training-extra-eval). For vanilla the post-training eval was higher (0.5761 vs 0.5707) so stored best_val was the within-training value. For Lite seed 1234 it was lower (0.5685 vs 0.5742) so stored best_val was the noisier single-batch post-training value.
eval_every differed. Vanilla evaluated every 2000 steps (25 draws); Lite every 2500 (20 draws). Vanilla had more chances at a low within-training value.
2 / 3 Lite seed logs are not preserved. The numbers 0.5679 and 0.5693 for seeds 5678 and 9012 live only in the original RunPod REPORT.md. Not auditable from shipped artifacts.

What would convert this from directional to formal

Re-run vanilla with eval_every=2500 to match, or re-run Lite with eval_every=2000, so the within-training best is computed on equal numbers of evaluations.
Run all 4 seeds (vanilla + 3 Lite) with identical val_stream RNG initial state.
K=10 independent post-training eval passes with a fixed RNG.
Mean ± std with a two-sample test.

Estimated cost: ~$2.60 on an A40 SXM. Script lives in the working repo (not in this public kit) at scripts/reproduce_benchmark.py. Queued, not run.

Full provenance

See BENCHMARKS.md and BENCHMARK_AUDIT.md in the working repo (tilelli-kit/) for the per-seed, per-eval-event raw numbers and the preserved-vs-not-preserved log audit.