results/claim_01_benchmark.md

# Claim 01 — TinyStories byte-LM benchmark vs vanilla

**Status:** preliminary single-seed directional finding. Not a defensible
architecture claim.

This file is a condensed, public-facing copy of the audit-rewritten
`BENCHMARKS.md` from the working repo. The honest headline:

> On the only Tilelli-Lite seed whose `log.jsonl` is preserved (seed
> 1234), the win against a 1-seed vanilla baseline depends on which
> `best_val` definition is used. Within-training periodic eval: Lite
> loses by 0.6 %. Post-training "final" extra eval: Lite wins by 0.4 %
> on a different validation-batch composition. Two additional Lite
> seeds (5678, 9012) were reported at 0.5679 and 0.5693 but their
> `log.jsonl` files are not preserved.

## Why this isn't yet a real result

1. **`best_val` was heterogeneous.** The training loop saved
   `min(within-training-best, post-training-extra-eval)`. For vanilla
   the post-training eval was higher (0.5761 vs 0.5707) so stored
   best_val was the within-training value. For Lite seed 1234 it was
   lower (0.5685 vs 0.5742) so stored best_val was the noisier
   single-batch post-training value.
2. **`eval_every` differed.** Vanilla evaluated every 2000 steps (25
   draws); Lite every 2500 (20 draws). Vanilla had more chances at a
   low within-training value.
3. **2 / 3 Lite seed logs are not preserved.** The numbers 0.5679 and
   0.5693 for seeds 5678 and 9012 live only in the original RunPod
   `REPORT.md`. Not auditable from shipped artifacts.

## What would convert this from directional to formal

- Re-run vanilla with `eval_every=2500` to match, or re-run Lite with
  `eval_every=2000`, so the within-training best is computed on equal
  numbers of evaluations.
- Run all 4 seeds (vanilla + 3 Lite) with identical `val_stream` RNG
  initial state.
- K=10 independent post-training eval passes with a fixed RNG.
- Mean ± std with a two-sample test.

Estimated cost: ~$2.60 on an A40 SXM. Script lives in the working repo
(not in this public kit) at `scripts/reproduce_benchmark.py`. Queued,
not run.

## Full provenance

See `BENCHMARKS.md` and `BENCHMARK_AUDIT.md` in the working repo
(`tilelli-kit/`) for the per-seed, per-eval-event raw numbers and the
preserved-vs-not-preserved log audit.