Audit trail — from the original kit to this one
This file documents what was removed, changed, or rewritten when the
project went from tilelli-kit-2026-05-23-audited.zip (the original
shipping kit, SHA 00af96d8…) to this humble public repo.
Headline
The original kit shipped 6 forward-looking claims ("emergent metacognition", "3/3 seeds beat vanilla", "matched at parameter count", etc.). This kit ships 3 verifiable positives + 3 verified negatives, and only the v4 deployed ckpt.
Removed
| Artifact | Why |
|---|---|
BENCHMARKS.md (original) |
Headline overstated: "3/3 seeds beat vanilla" rested on heterogeneous best_val definition + asymmetric eval_every. Replaced by results/claim_01_benchmark.md (preliminary 1-seed directional). |
HOW_TILELLI_WORKS.html |
17-scene marketing-style explainer. Replaced by this README + the audit trail. |
PAPER.md §4.2 "emergent metacognition" |
Empirically disproven, see results/claim_02_metacog.md. |
Ckpts: tilelli_chat_v5_metacog_cpu.pt, tilelli_chat_v6_metacog_chat_step100.pt, tilelli_chat_v7_metacog_v2.pt, tilelli_chat_v4splice_v7abstain.pt |
All failed their respective deploy gates. Available on request for negative-result replication. |
runs/ (28 of 30 sub-dirs) |
Historical engineering churn. Only the 3 audit-relevant ones (metacog_2026-05-23, metacog_v7_2026-05-23, metacog_v4v7splice_2026-05-23) are referenced by claims here. |
swap_weights.sh |
Deprecated per 2026-05-20 memory; could cause prod regression. |
Rewritten
| File | Change |
|---|---|
README.md |
Karpathy-style honest framing. Every claim links to a script. Negative results first-class. |
results/claim_01_benchmark.md |
Inherits the audited BENCHMARKS.md (21:39 2026-05-23) honest version. |
results/claim_02_metacog.md |
New, documents the DISPROVEN router-entropy claim + the v7 sub-result + the splice failure. |
pyproject.toml |
Trimmed to torch + numpy only. |
Preserved as-is
| File | Notes |
|---|---|
src/tilelli/ (the package) |
The actual model code. Unchanged from the kit version. |
checkpoints/tilelli_chat_v4.pt |
The deployed ckpt. SHA-pinned in results/claim_01_benchmark.md. |
prompts/probe_210.jsonl |
The audited eval set. |
LICENSE |
Apache 2.0. |
What replication looks like
If you want to reproduce the negative results (not just the positives),
the original kit + the metacog run dirs (v5 / v6 / v7 / v8a / v8b ckpts
and the full probe JSONLs) are preserved in the lineage's working repo —
available on request via hello@tilelli.tech.
What "humble" means here
Three rules:
- The README is the contract. Every numerical claim has a script that produces it from a bundled artifact, and the script exits non-zero if it doesn't.
- Negative results are first-class. A finding of "this doesn't work, and here's exactly how we know" is as worth shipping as a positive result of equal effort.
- One ckpt, one story. Don't ship five ckpts and let the user guess which one corresponds to the headline. Ship the one that anchors the claims; document the others as audit trail.
This is the Karpathy-style framing of nanoGPT and minBPE adapted to a
small routed LM with an audited claim trail.