babel-codec-gpt2 / paper /PLAIN_SUMMARY.md
wpferrell's picture
BABEL v1.1 -- review-hardened revision
f0a48ea verified
|
Raw
History Blame Contribute Delete
9.15 kB

PLAIN_SUMMARY.md — the paper in one page (v1.1, 2026-07-08; revised per external agentic review (paperreview.ai))

What the paper claims. We took one production language model — GPT-2 — and built what the paper claims in its locked form: the first complete, certified, bidirectional decode of an entire production language model. We accounted for its ENTIRE internal state, at every one of its 13 layer checkpoints, in three kinds of text, against a pass bar the model itself sets. Not "here are some interesting features" — an account: throw away the model's whole hidden state at a checkpoint, rebuild it from only the things our decoder can read, and the model's behavior stays inside its own noise floor at 39 out of 39 boundary×regime cells — 13 checkpoints × 3 kinds of text (36/39 on the older, stricter ruler, and those 3 are priced ruler-geometry, not model behavior). The amount of behavior we could NOT explain went from 11.2 units of surprise to exactly zero, in public, dated steps.

And then we translated it — the BABEL codec (your ratified name; the paper's long form, used once: "a certified bidirectional codec for GPT-2's residual-stream language"). Every one of the decoder's 351 channels was put on trial: 54% carry an explicit English meaning (a "naval/warship" field, a "clause boundary" word, an "operator" anchor); the other 46% are PROVEN to carry no word — proven, because random directions of the same loudness move the model more. (v1.1 scope, answering the reviewer: that 54%/46% split is the frozen per-channel rate under the naming battery at 20 null draws; under explicit multiple-comparison control the named fraction is lower and correction-dependent — about a quarter of channels survive channel-level false-discovery control at q = 0.05, about 7% under the strictest per-gate control, with a 9% core that no correction removes and a built-in "hold in ≥2 of 3 text regimes" rule that already caps the false-named fraction near 10%; the 46% word-less figure is, if anything, an under-estimate. Paper §6.1 + Appendix B.) How the language composes from layer to layer is certified linear at all 36 layer-seam tests. We built the exact inverse (English → state) and proved the round trip is behaviorally invisible at all 39 cells. And the model obeys hand edits: turn the "naval" field up and GPT-2 starts predicting amphib, sunk, ashore, reefs, sailed, submarine. Three of the four axes we hand-edited steer the model in their own vocabulary; the fourth (the "rung", an executable formula rather than a static dial) is now CERTIFIED UNUSABLE AS A STEERING LEVER: a same-day follow-up (L5) probed it through the channel matched to what it IS — the repetition behavior itself — plus two internal readouts, and its tiny effect cleared none of the internal readouts, while a genuine onset direction did move the behavior. A second follow-up (L6) tested it against an honest 20-draw random null at both doses: at ±6 it does not separate from a random nudge of the same size, and at ±3 its tiny (0.4%-probability) effect is indistinguishable from the honest random floor itself (two pre-registered 20-draw nulls straddle it — one lands just below the effect, one above), growing SUB-linearly with dose — pushing harder buys nothing. A gauge you can read, not a lever you can pull, at both doses we tested. One new fact came out of the hunt: this bounds what can be written INTO the rung, not what reads out of it — one candidate certified channel (the naval/warship field, one layer later; a thin 2.8% margin over its multiplicity null) consumes the rung's output. Perfectly readable, not a steering handle.

Why anyone should believe it. This is the paper's real weapon. Every experiment had its pass bar written down and locked BEFORE the data, with a stated bet; the completeness verdict came back NOT-YET six consecutive times and we published the gap table and stopped each time; the meter was recalibrated once, under pre-registered sanity gates, and both meters are reported everywhere, forever. The hardest object in the model — a repetition-keeping computation at layer 5 — got three certified impossibility results (can't be read, can't be looked up, can't be forged by its own circuit) before the thing that finally worked: we taught a tiny LINEAR student to compute it from readable inputs, and the test was built so that bigger students who merely memorize get caught — and they were caught (they ace training, fail on never-seen repeat periods; the linear one passes). One instrument bug happened all week; our own replay gate caught it. Every number in the paper traces to a frozen, hash-stamped artifact (Appendix A maps each one), and everything ran on the one A4500 in this machine — any skeptic can re-run any row.

What we do NOT claim. Not the first "activations → English" concept — Anthropic's Natural Language Autoencoders (May 2026) and the Cycle-Consistent Activation Oracles published that idea first, and the paper says so generously. Our claim is the locked one — the first complete, certified, bidirectional decode of an entire production language model — on four axes they don't touch: all boundaries priced (they read middle-to-late-layer samples); falsifiable verdicts incl. certified-word-less channels (they score plausible glosses by reconstruction); a decoder built from the model's own certified channels with an algebraic inverse (theirs is a trained external translator); and a round trip scored in BEHAVIOR against matched-random nulls — edit the English, the model obeys (theirs is scored in activation space, with a qualitative steering demo on top and no matched-null certification — the paper credits the demo). Also honestly scoped: one model, one grain, three regimes; the 46% word-less fraction and the un-steerable rung are named, not hidden.

The three honest percentages (the paper states all three, always together): 100% of the pre-registered definition met (certified-no-word counts as an answer); behavioral round trip 100% / 94.7% / 3-of-4 (reconstruct / meaning-transplant / human-edit; the transplant number is 16 prose pairs at one mid-stack checkpoint — a v1.1 boundary×regime sweep confirms it is not special to that checkpoint: median transplant 0.94–0.98 in prose and 0.82–0.98 in repetition across early, mid and late checkpoints, and 0.89 in code at late depth, though code is heavy-tailed at early/mid depth; paper §6.4); 53.6% of channels carry an actual English word (frozen gate-level rate under the 20-draw battery; materially lower under multiple-comparison control — see the naming note above and §6.1).

The two loose ends are now finished — as certified negatives (L5, same day). Neither headline number moved, and both favorite bets lost. The missing 5.3% of transplantable meaning is NOT hiding in the certified door channels: adding the certified door read (summarized or in full) moves the model exactly 0% further — it is certified to live in genuinely un-charted dark mass outside the whole dictionary. It resisted every translation method we tried: it transfers only as its exact raw configuration, never through any compressed or named form. And editing the rung cleared none of the four channels we can read it through (the honest dose story is above). Both remainders are closed properties now, not open wounds; the honest claim is sharper, not bigger.

And the questions those answers opened are finished too (L6, same day). We hunted the 5.3% to ground. It is DIFFUSE: smeared across the entire 329-dimension dark space — no hidden low-rank concept (we tried every rank up to 256; none captures 80% of the gap, and a RANDOM 256-dim slice of the dark does nearly as well as the best hand-picked one). And it is mostly word-less: of its 8 biggest directions, only 2 faintly cross the naming bar, through side channels, with no stable meaning (they're recorded as provisional dark signatures, NOT dictionary entries). The rung stays steering-unusable at double dose (above). And the one L4 oddity we'd never explained — the "operator" axis appearing to steer on-manifold when it was predicted inert — dissolves: it was read-out ECHO (~94% of the response is the injected word-vector riding straight through to the output; the computed part is inside the noise), so no new capability, and the L4 result's own control always held. L6's scoreboard: 2 favorite bets hit, 3 lost — every loss logged as a certified finding. No headline number moved, in any of it.

Deliverables ready for your review: PAPER_DRAFT_V1.md (+ 6 figures in paper_figs/, all regenerable from the frozen JSONs by one script; L5 verdicts in §6.6, L6 verdicts in §6.7, new Fig. 6 = the dark-mass rank ladder) and the outreach kit (AF post, 4 emails, Zenodo + GitHub checklists, release sequence, and the new GitHub-front-door README_DRAFT.md you asked for) in outreach_kit/, all updated post-L6 and standardized to the ratified name "the BABEL codec". Nothing has been sent, posted, uploaded, or committed anywhere — every send is yours to fire.