Upload TAEBENCH_V2_DESIGN.md with huggingface_hub

TAEBENCH_V2_DESIGN.md

@@ -0,0 +1,509 @@
+# TAE-BENCH v2 — What it would mean to FULLY UNDERSTAND a text autoencoder
+*Design doc, 2026-06-12. Successor to TAE-Bench v1 (`parascopes/layerwise/src/taebench.py`:
+recon/causal/manifold/sparsity). Grounded in INTERP_RESULTS.md §10–14, QUESTIONS.md Q1–Q5,
+PLANS_SCORED.md. Adversarially refereed (sub-agent critique integrated throughout).*
+
+---
+
+## 0. The framing: understanding as a CERTIFICATE, not a vibe
+
+v1 measures *dictionary quality*. v2 measures *our understanding of SONAR itself*. The
+difference: v1 asks "how good is this SAE?"; v2 asks "could we pass the exams that only
+someone who understands the codec can pass?" Each criterion below guards against a specific,
+already-observed failure mode of pseudo-understanding (the FVU lie, the read/write gap, the
+DAS template artifact, seed-disjoint dictionaries). The capstone — not a suite, but the
+north star — is the **clean-room test**: a team given only our written account (no weights)
+should be able to predict, edit, construct, and re-implement a functionally matching codec.
+Weights are not transmissible through prose, so this is the one test distillation can't game.
+
+**Cross-cutting anti-gaming rules** (apply to every suite):
+1. **Compression budget.** Any learned predictor/constructor used to pass a suite has a hard
+   parameter budget (default ≤1% of the component it explains; "laws" ≤10 named parameters).
+   Otherwise "the weights themselves" pass everything. Report params alongside every score.
+2. **Frozen eval families.** Perturbation families, edit types, transfer targets are fixed
+   (or adversary-chosen) AFTER the interpretive account is written, never tuned against.
+3. **Honest ceilings/baselines everywhere.** Decoder-prior baseline (shuffled-z), corpus-prior
+   baseline, LLM-rewrite+re-encode ceiling, encoder round-trip ceiling — the battery's audits
+   showed every headline needs these (belief-state, DAS, steering all moved under them).
+4. **Impossibility clause.** A criterion can be passed by a *demonstrated structural
+   impossibility* (e.g. "no low-rank wording subspace exists — margin collapse 0.000 at every
+   rank, §14"). Understanding that the object LACKS asked-for structure is understanding —
+   the deepest kind. Without this clause, weakest-link scoring is hostage to category errors
+   about a holographic code.
+5. **Read/write separation.** Every claim is tagged READ (decodable from z) or WRITE
+   (controllable via z). The battery's single most replicated law is read ≫ write; a suite
+   that conflates them measures nothing.
+
+**Aggregation.** Understanding Index = the full 7-vector, headline = min over criteria
+(conjunctive: you don't understand a codec you can't edit), with the impossibility clause as
+the escape valve. Mean is reported but never headlined — a mean lets the easy reading half
+(probes ≈ .99) launder the failing writing half.
+
+---
+
+## PART 1 — THE SEVEN CRITERIA
+
+Scores are 0–1: (current best − baseline)/(ceiling − baseline), my calibrated judgment from
+§10–14 evidence cited inline.
+
+### (A) PREDICT — simulate the decoder from z without running it. **Current ≈ 0.35**
+**What:** given z (plus our interpretive features: probes, SAE codes, spectral coords), and
+optionally a perturbation δ, predict what the decoder will emit — attributes, fidelity,
+degeneracy, per-token reliance on z, collapse behavior — under a parameter-budgeted
+predictor, better than corpus priors, approaching decode-then-measure.
+**Why it matters:** prediction-without-simulation is the canonical operationalization of
+having a *causal model* rather than a copy. An interpretation that doesn't constrain the
+decoder's behavior is post-hoc storytelling. The budget clause is essential: an unbounded
+predictor is just a distilled second decoder (refereed objection #1).
+**Have:** qualitative laws — decoder fills function words (AtP 1.37×, §10), n_sentences is
+THE lossy attribute (§10), language fingerprint decode-inert (§12), dose→grapheme-attractor
+census with per-feature collapse doses (§10), noise robustness curves, attr probes ~.99 READ.
+**Gap:** all of it is retrodictive. Nobody has forecast a decode outcome ex ante; no
+quantitative simulator; perturbation responses only catalogued, never predicted on held-out
+directions.
+**Closes it:** a ≤10-param law per phenomenon + a budgeted forecaster hitting ≥80% of
+explainable variance on FROZEN held-out perturbation families.
+
+### (B) DECOMPOSE — z as interpretable parts; reassembly lossless; parts dual-use. **Current ≈ 0.45**
+**What:** an inventory z = Σ parts + characterized residual, where parts are (i) labeled and
+detection-validated, (ii) canonical across seeds, (iii) aligned with what the decoder
+actually reads (Jacobian coverage), (iv) reassembly reproduces the TEXT (CE/exact-match, not
+just SBERT), and (i)–(iv) hold for the SAME inventory — each is separately purchasable and
+separately worthless (refereed objection: identity map is lossless+canonical; L2 dial buys
+canonicity without coverage).
+**Have:** behavioral recon .89 SBERT / .92 attr (text-loss SAE, §10–11); canonicity is now a
+recipe (l2≈1e-3, k256 → 96.7% aligned at FVU .435, §12); dark matter is structured (error
+term recovers 64% of decode gap, §10); detection-scored labels .66–.78.
+**Gap:** the *same* dictionary never has all four: Jacobian coverage stuck at 18.5% ≈ random
+(triple-confirmed, §10); features aren't compositional atoms (17% survival under composition,
+§12); no losslessness standard exists — everything scored in SBERT, never in nats/exact text.
+**Closes it:** one dictionary passing canonicity ≥.9 AND coverage ≥2.5× random AND reassembly
+within 0.15 nats of the encode→decode ceiling AND labels ≥.7 detection — or an impossibility
+argument for whichever conjunct the hologram forbids.
+
+### (C) EDIT — any specified text edit via in-z operations, bounded collateral. **Current ≈ 0.40**
+**What:** an edit calculus: spec → Δz built from documented parts (NO re-encoding the target
+text — that's the rewrite pipeline, which passes with zero understanding), scoring both
+edit-success and collateral damage to everything else, on WILD text, generalizing to held-out
+edit types.
+**Why:** edit is where read/write conflation dies or survives. Reading is solved; if our
+parts can't be used as levers with predictable side effects, we've labeled the code without
+understanding its causal grammar. Caveat honestly: edit failure can be a fact about the
+channel (decoder error-correction regenerates erased attributes, §10) — that's what the
+impossibility clause + regeneration tests are for.
+**Have:** word-swap arithmetic verified (88% global / 67% position-matched, §12); gradient
+surgery 100% success at ‖Δz‖≈0.2 RMS but directions are LOCAL (cos .17–.35 across bases,
+§10); steering 0/6 vs rewrite Pareto (§10); rank-1 DAS dead in the wild (0%, §14); erasure
+incomplete for nonlinearly-stored attrs (§10).
+**Gap:** no standardized edit suite, no collateral metric at all (v1's biggest hole), no
+transport rule for local edit directions, no held-out-edit-type generalization test.
+**Closes it:** an in-z method matching the rewrite Pareto frontier on ≥5/10 frozen edit types
+at collateral ≤ rewrite's, including ≥2 edit types never seen during method development.
+
+### (D) CONSTRUCT — synthesize z from a spec at encoder quality, white-box. **Current ≈ 0.30**
+**What:** spec (word sequence / attribute set + content words / parts inventory) → ẑ whose
+decode satisfies the spec at ≥0.9× the encoder's own round-trip quality — with the
+constructor assembled from the *documented* operators (role rotations, size-adder PC0 law,
+scaffold + retraction), under a near-zero learned-capacity budget, extrapolating to longer
+and unseen-combination specs. A black-box GRU at encoder quality only proves the encoding is
+learnable — known — not that we know what it is (refereed objection #4).
+**Why:** Feynman's criterion. Construction is the only test that catches one-way
+understanding: every holographic claim, order-code claim, and content-mass claim becomes
+load-bearing when you must BUILD the vector.
+**Have:** GRU constructor recall .31 / order τ .75 / cos .655 vs ceiling .96 (§14) — order is
+SOLVED (shuffled-input control dissociates), content mass is the wall; normalized bag .31
+SBERT .44; composition operator fully characterized and invertible (§12–13); word-folding
+through the composer transmits order τ .92 but attenuates content (§14).
+**Gap:** 3× below ceiling on content; no white-box constructor exists; no spec-bandwidth
+accounting (how much structure must the spec supply — Q5's core question).
+**Closes it:** a constructor from documented parts at recall/SBERT ≥0.9× ceiling, plus the
+measured bits-of-structure curve.
+
+### (E) MECHANISM — how the encoder builds and the decoder reads each part. **Current ≈ 0.45**
+**What:** a causal account of (i) pooling: which token-state properties determine pooled-z
+behavior, predicting ablation outcomes ex ante; (ii) the decoder's readout: which
+layers/heads read which z-subspaces for which token types, validated by targeted ablation
+matching prediction; (iii) replaceability: modules rebuilt from the account (transcoders)
+swap in with bounded behavioral loss; (iv) where the order/position code arises in the
+encoder stack. Validation must be causal-scrubbing-grade on the natural distribution, not
+cherry-picked templates (the DAS lesson).
+**Why:** input-output laws without internals break OOD silently; mechanism is what makes
+PREDICT extrapolate and EDIT transportable. "Understood = replaceable" is the strongest
+mechanical standard available.
+**Have:** pooling = masked mean, verified cos>0.999 (`sonar_prepool`); token states = lexical
+ID 75% + position R² .88 (§14); pooling holographic (1-token ablation deletes the word only
+4–6%, §14); content = first ~256 spectral dims (§14); decoder cross-attn = near-full-rank
+readers off the PCs (§12, weight-SVD, confirming AtP); mid-layer transcoders FVU .32–.33,
+layers 0/23 near-linear (§10); attractor census (descriptive).
+**Gap:** no account predicts WHICH ablation case occurs; transcoders never tested as
+replacements (behavioral retention unknown); no per-head readout map; no story for how
+encoder attention assembles the order rotation; attractor basins unexplained.
+**Closes it:** ablation-outcome prediction ≫ base rate, a per-head map validated by
+intervention, ≥1 decoder layer behaviorally replaced by its transcoder at ≤5% SBERT loss.
+
+### (F) TRANSFER — everything above survives language/domain/seed/template shift. **Current ≈ 0.55**
+**What:** the other six criteria re-scored under shift: languages (incl. non-Latin),
+domains/registers (code, dialogue, markup, long text), dictionary seeds, and — the killer —
+template-vs-wild. Targets chosen after methods are frozen.
+**Why:** the battery's audits show distribution shift is THE claim-killer (multiclass DAS
+reversed under held-out values; rank-1 tense dial 0% wild; seed-disjoint dictionaries). A
+result that holds only on its training template is a description of the template.
+**Have:** strongest result in the program — 18/18 feature×language transfers, 10 languages,
+4 scripts (§10); composer language-robust (96% of monolingual, §12); soft-pivot decode
+mechanics (§10); canonicity dial makes seed transfer purchasable (§12).
+**Gap:** the BENCH itself is English-paragraph-shard-40 only; edits/construction/prediction
+never scored under any shift; no adversarial/off-manifold z family.
+**Closes it:** the v2 core suites run on ≥3 languages + 2 domains + 2 seeds with ≤20%
+degradation, template-vs-wild gap reported on every causal number.
+
+### (G) CALIBRATE — know what we don't know, and predict failures ex ante. **Current ≈ 0.25**
+*(New criterion, from the adversarial review: the hexad let us pass while the dark matter is
+load-bearing — the 81.5% of the decoder's sensitivity subspace our dictionaries miss IS what
+the decoder reads.)*
+**What:** (i) a signed dark-matter ledger: % of behavioral (decoder-CE) variance attributed
+per named part, with the residual located (spectral region, MELBO directions) and tracked
+release-over-release; (ii) ex-ante failure forecasting: before each new suite runs, written
+predictions of outcomes with confidences, Brier-scored.
+**Why:** a claimed understanding with unaccounted residual is exactly how the FVU lie
+happened. And predicting successes is cheap — predicting the .31 recall wall *before*
+measuring it is what understanding feels like from the inside.
+**Have:** dark-matter accounting exists (24.5% of SAE error linearly predictable; structured
+term recovers 64% of decode gap, §10); MELBO finds dictionary holes (§10); dose census. All
+retrodictive.
+**Gap:** no ledger is maintained; no forecast has ever been registered and scored.
+**Closes it:** ledger ≥80% of decoder-CE variance attributed to named parts; forecast Brier
+beating an uninformed-prior forecaster on a season of suites.
+
+---
+
+## PART 2 — THE v2 SUITES (criterion → aspects → protocols)
+
+Format per aspect: WHY (certification argument) · HOW (protocol ≤5 sentences) · metric /
+baseline / ceiling / pass · cost · **[TONIGHT]** = implementable now with `sonar_prepool.py`,
+`sonar_grad.py` (SonarDecoderCE: text_to_ids, differentiable ce_loss), `taebench.py` harness,
+`tae_probe` attributes, dose-response census artifacts, data_v2 (834k sentences + fr/de/zh
+parallel), trained transcoders, GRU constructor.
+
+Shared fixtures: eval shard 40 + data_v2 held-out chunk, seeds {0,1}, N=150–500 texts,
+SBERT all-MiniLM-L6-v2, attr battery from `tae_probe`, leaderboard row schema extended with
+`suite_version: 2`.
+
+### A. PREDICT suite
+
+**P1 — Blind decode forecasting. [TONIGHT]**
+WHY: the minimal honest version of "simulate the decoder": if our reading of z (probes +
+SAE code + spectral stats) can't say what text properties come out, the reading is
+decorative. The budget cap (linear/GBM ≤50k params over interpretive features, never raw-z
+MLPs) blocks the distillation exploit; the frozen split blocks tuning-to-eval.
+HOW: for 500 held-out z, compute interpretive features (12 attr probes, SAE acts, spectral
+band energies, ‖z‖, fingerprint). Fit budgeted predictors of the DECODE's properties:
+each attribute, SBERT-to-source, degeneracy, length. Evaluate on a frozen 500-z test split;
+compare per-target against corpus-prior baseline and against decode-then-measure (ceiling).
+METRIC: mean fraction-of-explainable-variance (FEV) captured. BASELINE: corpus prior (~0).
+CEILING: 1.0. PASS: FEV ≥ .8. COST: ~1 A40-h (decodes) + CPU.
+
+**P2 — Perturbation-response forecasting. [TONIGHT]**
+WHY: counterfactual depth — predicting what the decoder does to z+δ *without decoding* is
+the difference between a catalogue and a theory; the dose-response census (64 features, 3
+regimes, per-feature letter attractors) is sitting there as training data and has never been
+asked to generalize. Held-out DIRECTION FAMILIES (not just held-out features) are the
+anti-gaming constraint.
+HOW: from census + direction-zoo artifacts, fit a budgeted classifier {no-change /
+semantic-edit / attractor-collapse} + regressors (collapse dose, decode-SBERT drop) from
+direction features only (norm, spectral profile, SAE/PC/grad-family alignment, manifold
+survival). Evaluate on held-out feature directions AND two unseen families (fresh MELBO
+directions, random tail-band directions); decode to verify. Report acc/MAE vs base-rate and
+family-permuted baselines.
+METRIC: 3-class acc + dose MAE. BASELINE: base rate (~.55 acc). CEILING: test-retest of the
+census itself. PASS: acc ≥ .75 AND beats base rate on unseen families. COST: ~2 A40-h.
+
+**P3 — z-reliance law (per-token).**
+WHY: "the decoder reads content words 1.37× harder" is a correlation; understanding means a
+stated ≤10-parameter formula (token frequency, POS class, position, length) that predicts
+the per-token gradient attribution profile of UNSEEN sentences — a law, not a learned blob.
+HOW: compute AtP per-token attributions (`sonar_grad`) on 300 sentences; fit the ≤10-param
+formula on 150, test rank-correlation per sentence on the rest; ablation-validate on 20
+sentences by checking that high-law tokens are the ones whose z-information (Q3-020 style
+prepool deletion) moves decode CE most.
+METRIC: mean Spearman ρ. BASELINE: uniform/length-only (~.2). CEILING: attribution
+test-retest (~.9). PASS: ρ ≥ .7. COST: ~2 A40-h. [partial tonight — AtP exists, deletion arm new]
+
+### B. DECOMPOSE suite
+
+**D1 — Lossless reassembly round-trip. [TONIGHT]**
+WHY: v1 scores reconstruction in SBERT — meaning-similarity — but §14 proved z is a
+transcript; the honest losslessness unit is nats of teacher-forced CE / exact text. This is
+the round-trip test of whether our parts inventory IS the vector or merely gestures at it,
+with the residual term forced into the open (ledger input for G).
+HOW: decompose each held-out z as SAE reconstruction + structured-error term (darkmatter
+predictor) + documented residual; reassemble at each truncation level (SAE only / +error /
++residual). Score teacher-forced CE to the ORIGINAL text (`SonarDecoderCE`), exact-match and
+charF of greedy decodes, vs the encode→decode ceiling (CE ≈ .22, §14) and a
+paraphrase-of-source floor (CE ≈ 1.25).
+METRIC: ΔCE-to-ceiling (nats/token). BASELINE: paraphrase floor (≥1.0 over ceiling).
+CEILING: 0. PASS: ≤ .15 nats with all named parts. COST: ~1 A40-h.
+
+**D2 — Dual-use parts (canonicity × coverage × labels, one dictionary). [TONIGHT]**
+WHY: each sub-virtue is independently purchasable and independently worthless (L2 buys
+canonicity, k buys recon, text-loss buys behavior — and coverage stayed at chance through all
+of it). Only the conjunction on a single inventory certifies "these are the parts." This is
+also where the impossibility clause may legitimately fire — the hologram might forbid the
+conjunction, and proving that is a pass.
+HOW: for each benched dictionary run the existing three measurements as ONE row: PW-MCC
+across 2 seeds (no shared data, no post-hoc rotation), Jacobian sensitivity coverage
+(`tae_jacobian_audit`) vs random-feature control, detection-scored labels (flipbook, α=20)
+on the SAME features. Score = min of the three normalized sub-scores.
+METRIC: min(canonicity, coverage/2.5×random, detection/.7). PASS: min ≥ 1, i.e. ≥.9 / ≥2.5×
+/ ≥.7. BASELINE: current best joint ≈ (.97, 1.0×, .66) — fails on coverage. COST: ~3 A40-h.
+
+**D3 — Part compositionality. [TONIGHT]**
+WHY: parts that don't survive composition aren't parts of the CODE, they're features of one
+distribution slice — and §12 measured exactly this (17% survival, additivity R² .38–.60,
+Matryoshka best). v2 promotes it from a finding to a standing score so dictionary training
+can optimize for it.
+HOW: for 500 (A,B) pairs, compose via the trained operator and via real encode of "A then
+B"; measure feature survival, union Jaccard, and held-out activation additivity R² for the
+benched dictionary; report against the Matryoshka-k128 reference (.60) and flat-k128 (.38).
+METRIC: additivity R². PASS: ≥ .8. COST: <1 A40-h (embeds cached).
+
+**D4 — Decomposition of the WORDING code (or its impossibility certificate).**
+WHY: §14's deepest result — wording is in z at 1.2–1.5 nats but in NO low-rank subspace —
+is currently a negative; the criterion needs it as a scored boundary: either some nonlinear/
+local-frame decomposition isolates wording from meaning, or we certify the impossibility
+with a tight bound (the escape-clause pass).
+HOW: take length-matched paraphrase pairs (Q3-009 hygiene); attempt wording/meaning split
+with the best available tools (local PCA frames, SSAE shift dictionaries, gradient
+subspaces); score the CE wording-margin retained after removing the candidate wording
+component while SBERT meaning stays ≥.9. Sweep rank/locality; the impossibility certificate
+is the measured margin-retention curve pinned at ~0 across all methods.
+METRIC: margin removed at meaning ≥.9. PASS: ≥80% removed, OR certified flat curve across ≥3
+method families. COST: ~3 A40-h. [partial tonight]
+
+### C. EDIT suite
+
+**E1 — Edit precision with collateral (flagship; v1's biggest hole). [TONIGHT]**
+WHY: v1's causal suite scores potency×coherence — it literally cannot see side effects, and
+read≫write means side effects are where understanding fails. A standardized edit-spec suite
+with a collateral score and the rewrite Pareto ceiling turns "can we edit" into one number,
+and the held-out-edit-type split tests the calculus, not the memorized direction.
+HOW: 10 frozen edit types (word-swap, tense, person, negation, number-change, entity-swap,
+register, insert-clause, delete-clause, reorder) × 50 WILD bases each; methods = arithmetic
+delta, diff-of-means, gradient surgery (`sonar_grad`), feature steering — in-z only, no
+re-encoding of any target text. Edit-success = LLM-judge + regex on the specified change;
+collateral = 1 − (token-F1 on unchanged spans) combined with drift on the 11 untargeted
+attributes. Report exact-edit@collateral≤.1 per type and the Pareto AUC vs LLM-rewrite+
+re-encode; 2 of the 10 types are held out from all method development.
+METRIC: mean exact-edit@collateral≤.1. BASELINE: random-direction edit (~0). CEILING:
+rewrite pipeline (~.85). PASS: ≥.6 incl. held-out types. COST: ~4 A40-h + LLM-judge calls.
+
+**E2 — Erasure completeness with regeneration audit. [TONIGHT]**
+WHY: "erasure ≠ removal" (§10) is the cleanest demonstration that probe-space success can be
+behaviorally void — the decoder regenerates attributes from nonlinear traces. A standing
+score with the re-encode→re-decode regeneration test certifies WRITE-removal, not READ-removal,
+and bounds what the channel's error-correction allows (impossibility clause eligible).
+HOW: for each erasable attribute, apply best-known erasure (LEACE + iterated variant);
+measure attribute presence in the decode, then re-encode the decode and decode again
+(regeneration probe), with content-SBERT ≥.9 required throughout. Score = post-regeneration
+absence rate, against the do-nothing baseline and a rewrite-erasure ceiling.
+METRIC: regeneration-proof removal rate at content ≥.9. PASS: ≥.8 per attribute or certified
+channel-regeneration bound. COST: ~1.5 A40-h.
+
+**E3 — Edit transport across bases.**
+WHY: gradient surgery succeeds per-instance but its directions are local (cos .17–.35) —
+so we currently possess edit *instances*, not an edit *calculus*. A transport rule (local
+frame map / parallel transport along the manifold) recovering most of per-base success is
+exactly what upgrades the local-frames picture from descriptive to operational (Q4-003).
+HOW: compute minimal edit Δz at 30 source bases per edit type; transport each to 10 target
+bases via (i) naive copy, (ii) local-PCA frame alignment, (iii) learned budgeted transport
+map; score edit-success and collateral at targets vs per-base recomputation (ceiling).
+METRIC: transported/recomputed success ratio. BASELINE: naive copy (low). PASS: ≥.7.
+COST: ~3 A40-h. [not tonight — transport machinery is new]
+
+### D. CONSTRUCT suite
+
+**C1 — Construction vs spec. [TONIGHT]**
+WHY: certifies the Feynman criterion at the system level: a spec→z→text pipeline scored
+against what the encoder itself achieves makes every claimed piece of knowledge (order code,
+content mass, scaffold) load-bearing. v1 has no construction suite at all; the GRU
+constructor and the bag ladder already exist as the first two leaderboard rows.
+HOW: 300 frozen specs in three formats (word sequence; attribute-set + content words; parts
+inventory). Run each registered constructor; decode; score word recall, order τ,
+attr-satisfaction, SBERT to a reference rendering, degeneracy. Ceiling = encode(reference
+text) round-trip; baseline = normalized bag-of-word-vectors (.31/.44).
+METRIC: composite (recall, τ, attr, SBERT) vs ceiling ratio. PASS: ≥.9× ceiling. CURRENT:
+GRU ≈ .4–.5×. COST: ~1 A40-h per constructor.
+
+**C2 — Spec bandwidth (minimal structure curve). [TONIGHT]**
+WHY: Q5's actual question is not "can a net do it" but "HOW MUCH structure must be
+supplied" — the bits-of-structure curve is the quantitative form of the transcript thesis
+(if z is a transcript, order+length scaffolding should be worth more than better word
+vectors, which §14 already hints at).
+HOW: re-run C1 with spec ablations: shuffled order, dropped function words, no length,
+quantized word vectors; measure degradation per ablation; express as marginal value per
+structure channel. Deliverable = the curve + the single number "minimum spec achieving .8×
+ceiling".
+METRIC: the curve (descriptive) + min-spec point. PASS: n/a (measurement aspect). COST:
+~1.5 A40-h.
+
+**C3 — White-box constructor from documented operators.**
+WHY: the anti-gaming heart of CONSTRUCT — assemble z using only the account's published
+machinery (role-rotation composer, PC0 size-adder, scaffold templates, retraction onto local
+frames; learned glue ≤100k params) and require extrapolation (longer specs, unseen attribute
+combinations) where a memorizing constructor fails. Passing this approaches the clean-room
+capstone for the encoder side.
+HOW: implement spec→z as: embed words, fold via the trained composition operator with
+documented depth-decay compensation, add size/attr offsets from the documented dials,
+retract onto the local manifold frame (Q5-021 recipe); evaluate on C1's frozen specs PLUS an
+extrapolation split (1.5× length, novel attr pairs). Compare to the black-box GRU at matched
+score.
+METRIC: C1 composite + extrapolation ratio (extrap/in-dist ≥ .8). PASS: ≥ GRU at ≤1/10 its
+learned params, extrapolation holds. COST: ~4 A40-h. [not tonight — research]
+
+### E. MECHANISM suite
+
+**M1 — Pooling accounting. [TONIGHT]**
+WHY: we know the pooling op exactly (masked mean, verified) and know ablation outcomes are
+trimodal (4–6% clean delete / 78% no-change / global perturbation, §14) — but no account
+predicts WHICH outcome a given token yields. A predictive accounting of the hologram's
+entry point is the first mechanism number the bench can hold us to, using machinery
+(`pool_ablate`) that already exists.
+HOW: for 200 sentences × all tokens, run `pool_ablate` + decode, label each token's outcome
+class; fit a budgeted (≤10-feature) predictor from token-state properties (norm, ID
+frequency, position, content/function class, state–z cosine); test on 100 held-out
+sentences; report confusion vs base rate.
+METRIC: 3-class balanced acc. BASELINE: base rate (~.33 balanced). CEILING: label
+test-retest. PASS: ≥.6 balanced. COST: ~2 A40-h.
+
+**M2 — Decoder readout map (per-head cross-attention accounting).**
+WHY: "the decoder reads broadly, off the PCs" (§12 weight-SVD + AtP) is an aggregate; a
+mechanism account must say WHICH heads read WHICH z-subspaces for WHICH token classes, and
+be validated by intervention — predicted head importance must match ablation-measured
+importance, the causal-scrubbing standard the referee demanded.
+HOW: per-head AtP attribution of z-gradient flow across decoder cross-attn (extend
+`sonar_grad` hooks); build the head×subspace×token-class map; pick the top-k and bottom-k
+heads per claim and ablate them, measuring CE/SBERT damage; score rank-correlation of
+predicted vs measured damage across heads.
+METRIC: Spearman ρ predicted-vs-ablated. BASELINE: 0. PASS: ≥.7. COST: ~4 A40-h.
+[partial tonight — hooks are new code]
+
+**M3 — Replacement test (transcoder swap). [TONIGHT]**
+WHY: "understood = replaceable" is the strongest mechanical certificate that survives the
+narrative objection — a module rebuilt from the account either carries the behavior or it
+doesn't. The transcoders are TRAINED (FVU .32–.33 mid, .005–.015 ends) but have never been
+behaviorally swapped in; this is free signal sitting on disk.
+HOW: replace decoder FFN layers with their transcoders, one layer at a time then cumulative
+{0,23} / {6} / {6,12,18} / all-5; decode 300 held-out texts; score SBERT, attr preservation,
+degeneracy vs the intact decoder; report behavioral-retention per layer against an
+FVU-matched random-projection control.
+METRIC: SBERT retention. BASELINE: random-proj control. CEILING: 1.0. PASS: ≥.95 single
+mid-layer, ≥.85 all-5. COST: ~2 A40-h.
+
+**M4 — Encoder assembly account (where the order code arises).**
+WHY: everything so far reads the encoder's OUTPUT; mechanism requires knowing where in the
+stack the position/order code (R² .88 at the top) is assembled and verifying causally by
+cross-layer patching — without it, CONSTRUCT-side claims about scaffolds are unanchored
+(Q3-033 + patch arm).
+HOW: capture encoder hidden states at every layer (extend `sonar_prepool` hooks); probe
+token-ID/position/order per layer to find the assembly depth profile; then patch order-
+carrying components from a scrambled sentence's layer-l states into the original and test
+whether the decode follows the donor order (the §14 scramble readout as the behavioral
+verdict).
+METRIC: layer profile + patch flip rate. PASS: identified layer band with patch flip ≥.6.
+COST: ~4 A40-h. [not tonight — capture is new]
+
+### F. TRANSFER suite
+
+**T1 — Language battery. [TONIGHT]**
+WHY: the bench is English-only while the program's strongest claim (18/18 feature-language
+transfers) is multilingual — and the soft-pivot + re-encode-tax findings (§10) predict
+WHERE degradation should appear (re-encode-heavy suites suffer in ja/zh), making this a
+prediction test too (feeds G2).
+HOW: run the v2 core (D1 reassembly, E1 edits on 3 edit types, P1 forecasting, C1
+construction) on the fr/de/zh parallel chunks in data_v2; score the per-suite degradation
+ratio vs English; compare against the re-encode-tax prediction.
+METRIC: mean degradation ratio. PASS: ≤20% drop (fr/de), documented + predicted gap for zh.
+COST: ~3 A40-h.
+
+**T2 — Domain & wild-shift battery + template-vs-wild gap. [partial TONIGHT]**
+WHY: every reversed claim in the program died at this boundary (multiclass DAS, rank-1 tense
+dial) — so v2 makes the template-vs-wild GAP a first-class reported number on every causal
+score, and adds code/dialogue/markup/long-text domains where the length-PC0 and markup-PC1
+structure predict specific failures.
+HOW: mint 4 domain eval sets (200 texts each); run D1/E1/P1; additionally re-score every E1
+edit type on its development templates vs wild bases and report the paired gap. Adversary
+(or a held-out collaborator) picks one extra domain after methods freeze.
+METRIC: degradation ratios + per-claim template-wild gap. PASS: gap ≤.15 on passing claims.
+COST: ~3 A40-h.
+
+**T3 — Seed & dictionary transfer. [TONIGHT]**
+WHY: seed-disjointness (match cos .149) means any dictionary-dependent score might be a fact
+about one draw; v2 requires every dictionary-dependent row to carry its cross-seed spread,
+and tests whether EDITS and PARTS transport across seeds (attribute-level rescue, Q2-007) —
+cheap, and it operationalizes "frames, not atoms" as a measured penalty.
+HOW: run D2/D3/E1-feature-steering with both seeds of the benched dictionary (canonical
+recipe + unregularized control); report per-score spread; for 20 labeled features, test
+whether the cross-seed matched feature performs the same edit.
+METRIC: cross-seed score spread + edit-transport rate. PASS: spread ≤10%, transport ≥.7 for
+canonical-recipe dictionaries. COST: ~2 A40-h.
+
+### G. CALIBRATE suite
+
+**G1 — Dark-matter ledger. [TONIGHT]**
+WHY: the review's sharpest point — the hexad could be passed while the unexplained 81.5% of
+the decoder's sensitivity subspace does the real work. A signed, versioned ledger (% of
+decoder-CE variance attributed per named part; residual located by spectral region and
+MELBO-direction overlap) makes ignorance a tracked quantity that must shrink release over
+release, like a test-coverage number.
+HOW: on 500 held-out z, attribute decoder-CE variance to: SAE part, structured-error part,
+fingerprint, scaffold/length, residual (via nested-ablation deltas with the D1 machinery);
+locate the residual (band energies, cos to MELBO set); publish as a leaderboard artifact
+with version + date.
+METRIC: % behavioral variance attributed. BASELINE: SAE-only (~64% of decode gap, §10).
+PASS: ≥80% attributed, residual located. COST: ~1.5 A40-h (rides on D1).
+
+**G2 — Registered failure forecasts.** [process, free]
+WHY: predicting the .31 recall wall before measuring it is what understanding feels like;
+Brier-scoring our own pre-registered forecasts on each new suite is the cheapest possible
+test of whether the account has counterfactual depth — and it generates the frozen-eval
+discipline rule 2 requires.
+HOW: before each v2 suite first runs, write 5–10 quantitative predictions with confidence
+intervals into a FORECASTS.md (e.g. "M3 single-mid-layer retention .93±.03", "T1 ja
+re-encode-tax suites drop ≥30%"); score Brier/coverage after; the running score is a
+leaderboard row of the ACCOUNT itself.
+METRIC: Brier vs uninformed-prior forecaster. PASS: beat it over a season. COST: ~0.
+
+---
+
+## 3. Scoring & leaderboard schema
+
+- Each aspect → normalized 0–1: (score − baseline)/(ceiling − baseline), clamped.
+- Criterion score = min over its aspects (impossibility certificate ⇒ that aspect = 1.0 with
+  flag `certified_impossible`).
+- **Understanding Index = the 7-vector; headline = min.** Current estimated vector:
+  PREDICT .35 · DECOMPOSE .45 · EDIT .40 · CONSTRUCT .30 · MECHANISM .45 · TRANSFER .55 ·
+  CALIBRATE .25 → **headline ≈ 0.25 (CALIBRATE-bound), honest "we ~quarter-understand SONAR."**
+- Leaderboard: extend `taebench_leaderboard.json` rows with `suite_version: 2`, per-aspect
+  fields, params-budget fields, and the forecast Brier row. v1 suites keep running unchanged
+  (they remain the dictionary-quality lane; v2 is the understanding lane).
+
+## 4. Implementation order
+
+**Tonight (existing tools, ~12 A40-h total, all [TONIGHT]-flagged):**
+1. **E1 edit-precision+collateral** — the single biggest v1 hole; word-swap/diff-means/
+   gradient-surgery code all exist; new code = collateral scorer + spec runner.
+2. **D1 lossless reassembly (CE units)** — `SonarDecoderCE` + darkmatter predictor exist;
+   establishes the nats-based losslessness standard and feeds G1 directly.
+3. **P2 perturbation forecasting** — census artifacts are training data on disk; pure CPU
+   fit + ~2h decode verification; first-ever ex-ante prediction number.
+4. **C1+C2 construction-vs-spec + bandwidth** — GRU + bag baselines become leaderboard rows;
+   spec ablations are config, not code.
+5. **M1 pooling accounting / M3 transcoder swap** — `pool_ablate` and trained transcoders
+   are idle capability; both are ≤2 A40-h each.
+   Then G1 (rides on D1), T3, T1, P1, D2, D3, E2, G2 (free), T2.
+**Next wave (new machinery):** P3, D4, E3 (transport), M2 (per-head hooks), M4 (encoder
+capture), C3 (white-box constructor), T2 adversarial domain.
+**Capstone (quarter-scale):** the clean-room test — freeze the written account, hand it to
+an independent agent team without weights, score their P/E/C performance using only the doc.
+
+*Dependencies: D1→G1; C1→C2/C3; E1→E3/T2-gap; M1 capture reusable for M4; P2 needs census
+JSONs (HF: tae_dose_response, direction-zoo).*