| # Audit trail — from the original kit to this one |
|
|
| This file documents what was removed, changed, or rewritten when the |
| project went from `tilelli-kit-2026-05-23-audited.zip` (the original |
| shipping kit, SHA `00af96d8…`) to this humble public repo. |
|
|
| ## Headline |
|
|
| The original kit shipped 6 forward-looking claims ("emergent |
| metacognition", "3/3 seeds beat vanilla", "matched at parameter count", |
| etc.). This kit ships **3 verifiable positives + 3 verified negatives**, |
| and only the v4 deployed ckpt. |
|
|
| ## Removed |
|
|
| | Artifact | Why | |
| |---|---| |
| | `BENCHMARKS.md` (original) | Headline overstated: "3/3 seeds beat vanilla" rested on heterogeneous `best_val` definition + asymmetric `eval_every`. Replaced by `results/claim_01_benchmark.md` (preliminary 1-seed directional). | |
| | `HOW_TILELLI_WORKS.html` | 17-scene marketing-style explainer. Replaced by this README + the audit trail. | |
| | `PAPER.md` §4.2 "emergent metacognition" | Empirically disproven, see `results/claim_02_metacog.md`. | |
| | Ckpts: `tilelli_chat_v5_metacog_cpu.pt`, `tilelli_chat_v6_metacog_chat_step100.pt`, `tilelli_chat_v7_metacog_v2.pt`, `tilelli_chat_v4splice_v7abstain.pt` | All failed their respective deploy gates. Available on request for negative-result replication. | |
| | `runs/` (28 of 30 sub-dirs) | Historical engineering churn. Only the 3 audit-relevant ones (`metacog_2026-05-23`, `metacog_v7_2026-05-23`, `metacog_v4v7splice_2026-05-23`) are referenced by claims here. | |
| | `swap_weights.sh` | Deprecated per 2026-05-20 memory; could cause prod regression. | |
|
|
| ## Rewritten |
|
|
| | File | Change | |
| |---|---| |
| | `README.md` | Karpathy-style honest framing. Every claim links to a script. Negative results first-class. | |
| | `results/claim_01_benchmark.md` | Inherits the audited `BENCHMARKS.md` (21:39 2026-05-23) honest version. | |
| | `results/claim_02_metacog.md` | New, documents the DISPROVEN router-entropy claim + the v7 sub-result + the splice failure. | |
| | `pyproject.toml` | Trimmed to torch + numpy only. | |
|
|
| ## Preserved as-is |
|
|
| | File | Notes | |
| |---|---| |
| | `src/tilelli/` (the package) | The actual model code. Unchanged from the kit version. | |
| | `checkpoints/tilelli_chat_v4.pt` | The deployed ckpt. SHA-pinned in `results/claim_01_benchmark.md`. | |
| | `prompts/probe_210.jsonl` | The audited eval set. | |
| | `LICENSE` | Apache 2.0. | |
|
|
| ## What replication looks like |
|
|
| If you want to reproduce the negative results (not just the positives), |
| the original kit + the metacog run dirs (v5 / v6 / v7 / v8a / v8b ckpts |
| and the full probe JSONLs) are preserved in the lineage's working repo — |
| available on request via `hello@tilelli.tech`. |
|
|
| ## What "humble" means here |
|
|
| Three rules: |
|
|
| 1. **The README is the contract.** Every numerical claim has a script |
| that produces it from a bundled artifact, and the script exits |
| non-zero if it doesn't. |
| 2. **Negative results are first-class.** A finding of "this doesn't |
| work, and here's exactly how we know" is as worth shipping as a |
| positive result of equal effort. |
| 3. **One ckpt, one story.** Don't ship five ckpts and let the user |
| guess which one corresponds to the headline. Ship the one that |
| anchors the claims; document the others as audit trail. |
|
|
| This is the Karpathy-style framing of `nanoGPT` and `minBPE` adapted to a |
| small routed LM with an audited claim trail. |
|
|
