Rename FABLE_PLAN.MD to FABLE_PLAN.md

9aae406 verified 9 days ago

14.8 kB

license: mit
tags:
  - aleph
  - geometry
  - differentiation
  - rotary
  - rotational
  - differential
  - geometric
  - conjunctive
  - additive-conjunctive
  - marginal-bits
  - ADC
  - a.d.c.

exp_011 — Additive-Conjunctive Differentiation (ACD)

Enriching the aleph: composition operators, marginal-bits methodology, and the Forge

Status: PLAN (pre-registration draft, awaiting Captain sign-off on §9) Date: 2026-07-01 · Prior art anchor: exp_007–010, RESEARCH_HISTORY.md Statutes inherited: pure Adam · clip max(loss,1.0) · no GAP/BN/Dropout on geometric paths · orthogonal init · employment law (no champion without prediction through the structure) · complete-file artifacts · smoke before ship · backup after every run.

1. The problem, formalized

A single aleph (K=64, D=4) is one soft partition of the input: a signed-projective address on the 2K-simplex, carrying at most ~log2(2K) ≈ 7 bits, in practice fewer (soft mass, eff-rank ≤ D). It is not enriched enough for tasks that need deep discrimination.

The naive enrichment — stack more alephs and add their addresses — was observed to produce cascade noise divergence: accumulated addresses got noisier, not sharper. Mechanism (preregistered explanation, to be measured in Phase 2):

Redundancy. Free codebooks trained on the same signal against the same loss fall into the same geometric attractor (the CV-band physics). m independent alephs learn ~the same partition. Their sum ≈ one address + disagreement noise.
Signed interference. Antipodal structure means disagreement doesn't average neutrally: a p⁺ vote on one stage against a p⁻ vote on another cancels signal in the downstream linear-attention features. Noise magnitude grows ~√m; signal grows < m.
Rank ceiling. Each stage caps at eff-rank D=4. Sums of aligned rank-4 objects stay rank-4 (redundant); sums of misaligned ones go isotropic (mush). Neither adds adjudication capacity.

2. The design law

The chain rule of mutual information is the whole game:

I(Y; A₁,…,A_m) = Σₜ I(Y; Aₜ | A₁..Aₜ₋₁)

Additive differentiation is only additive in information if each stage is conditioned on the outcome of the previous stages. Unconditioned addition adds redundant signal + independent noise → divergence (his observation). Conditioning can enter by four routes, which define the operator taxonomy:

operate on the residual of what previous stages explained (RES),
select branch-specific geometry given the previous address (TREE),
make stages independent by construction over disjoint subspaces, so the conditional equals the marginal (PROD — conjunctive),
adjudicate which stage speaks, input-dependently (GATE).

Headline gauge of the entire study: the marginal-bits curve — estimated Î(Y; Aₜ | A₍<t₎) per stage t. A structure is enriched iff the curve stays positive as m grows. Divergence = the curve crossing zero while address variance keeps rising.

3. Operator taxonomy (the formulas)

All operators compose m micro-alephs Aᵢ (signed-projective address, verbatim _address from aleph_routed_attention.py). All feed the same head. Notation: aᵢ = Aᵢ(·) ∈ Δ^{2Kᵢ}, per-stage codebook Mᵢ (Kᵢ×Dᵢ).

op	formula	conditioning route	closest prior
SUM (control)	a = Σ wᵢ Aᵢ(x)	none — predicted to diverge	naive ensembling
GATE (MoA)	a = Σ gᵢ(x) Aᵢ(x), g = A_meta(x)	adjudication	MoE routing
RES	r₀=x; aₜ=Aₜ(rₜ₋₁); rₜ = rₜ₋₁ − Wₜᵀâₜ	residual subtraction	RQ / RQ-MoE / REVQ
PROD	x → R x → split m subspaces; a = ⊕ᵢ Aᵢ(xᵢ) (joint read = implicit ⊗)	independence by construction	PQ / product-key memory
TREE	b ~ adjudicate(A₀(x)); a₁ = A₁^{(b)}(x), branch-specific M^{(b)}	explicit chain rule	HME / S'MoRE hierarchical routing
CROSS	features ⊇ low-rank proj of aᵢ ⊗ aⱼ	second-order conjunction	bilinear/FM models
ANNEAL	same M, temperature ladder τ₁>τ₂>…; coarse-to-fine re-address	curriculum conditioning	deterministic annealing

Sub-arms: TREE soft (mixture over branches, differentiable) vs TREE hard (top-1 + straight-through); RES with tied vs free per-stage D; PROD with learned rotation R vs fixed permutation. Stage counts m ∈ {1,2,3,4,6,8}.

Preregistered predictions: P1 SUM: marginal bits → ≤0 by m=3; address SNR falls (this is the observed divergence, now as a measured control). P2 RES & TREE: positive marginal bits through m=4; TREE best bits/stage, RES best on reconstruction-flavored targets. P3 PROD: best bits/parameter (K^m cells from m·K·D params); redundancy gauge ≈ 0 by construction. P4 GATE beats SUM but plateaus (adjudication without refinement caps at max-of-stages, not sum-of-stages). P5 Statute-by-construction (frozen-spread per stage) ≥ free per stage, replicating exp_010's construction law compositionally.

4. Task tiers (the "many small prototypes")

Tier-P — probe (seconds/arm). Synthetic nested globular clusters: Gaussian bubbles with sub-bubbles, 2–4 levels, controlled branching factor, separation, and noise floor — the VAE-bubble picture made exact. Ground-truth hierarchy known ⇒ exact per-level information, so marginal-bits is measured against truth, not a proxy. Model = composition operator + linear head, ~50–500K params. Target = immediate prediction of leaf/level labels + next-token-style transition prediction on bubble walks (the "rapid entropic decision adjudication" measured directly: entropy of the label posterior after each stage).
Tier-P2 (optional). Same probe over real VAE latents (SDXL/geolip VAE encodings) with pseudo-hierarchy from agglomerative clustering — bridges synthetic → real geometry.
Tier-L — LM (minutes/arm). The exp_010 instrument: 6.75M byte-trigram backbone, apmix head reading the composed address (employment law enforced), dense bank, 5k steps. Winners only.
Tier-A — transfer. Graft the winning operator into the Tier-A donor (exp_010 grafting playbook). Final phase only.

Budget matching: primary = equal total codebook params (Σ Kᵢ·Dᵢ const vs the K=64,D=4 single-aleph baseline); secondary = equal address bandwidth (Σ 2Kᵢ). Every arm is compared against its budget-matched single-aleph twin.

5. Gauge battery (uniform, every arm, every eval step)

Effectiveness: task NLL/accuracy (Tier-P), bpb (Tier-L); marginal-bits per stage (exact on Tier-P; staged-probe estimate on Tier-L: ΔH(Y|A≤t) via small frozen readouts).
Scaling: curves over m, K, D, params; marginal-bits-vs-m is the divergence detector; bits/param and bits/lat for efficiency fronts.
Deviance (statute gauges, per stage): dev, eff-rank, min-angle spread, mixing entropy, routing concentration — plus three new composition gauges: cross-stage redundancy (pairwise MI / distributional cosine between stage addresses; want ~0 for PROD, structured for TREE), hemisphere cancellation rate (mass-weighted sign conflicts across stages on shared axes), stage SNR (between-cluster / within-cluster variance of the accumulated address).
Cost: params, step latency, peak mem.
Verdict function (automated): PROMOTE if marginal-bits(m)>ε ∧ dev in band ∧ beats budget-matched twin; PARK if neutral; KILL on NaN, rank collapse (<50% of ceiling), marginal-bits<0 twice consecutively, or grad blowup. Thresholds are config, logged with every verdict.

6. The Forge (automation — "thousands of decisions, curated")

One system, four parts, all resumable via HF pushes:

Grammar. An arm is a JSON spec: {op, stages:[{K,D,freeze}], head, tier, budget_mode, seed} → canonical hash = arm id. Human-readable, diff-able, ledger-able.
Generator. Expands grids (op × m × K/D × freeze × seed) + random proposals; dedups against the ledger; auto-inserts every arm's budget-matched twin and the SUM control at matching m.
Scheduler — successive halving (ASHA-style rungs). rung0: Tier-P 200 steps (all arms) → top ⅓ → rung1: Tier-P 1k steps → top ⅓ → rung2: Tier-L 1k steps → top ⅓ → rung3: Tier-L 5k steps (exp_010-grade). Kill rules fire within rungs via the gauge battery, so a diverging arm dies in seconds, not minutes. Lane parallelism: Tier-P micro-models are batched — many arms trained simultaneously as one vectorized super-batch on the Blackwell (the WideCompiler instinct, natively; 96GB comfortably holds 32–64 lanes of 500K-param models).
Ledger + verdicts. Append-only results.csv + per-arm JSON + auto-generated SWEEP.md leaderboard + verdict log (why each arm was promoted/killed, with gauge values) → pushed to HF each rung. Checkpoints pushed only from rung2 up (storage sanity). Captain reviews verdicts, not arms.

Deliverable files (Phase 1): acd_structures.py (the 7 operators, pluggable, verbatim _address core) · acd_probe.py (nested-bubble generator + exact-MI gauges + staged probes) · acd_forge.py (grammar, generator, rungs, kill rules, ledger, HF push, lanes) · acd_lm_adapter.py (composed-address feature into the 6.75M backbone + apmix).

7. Phases and gates

Phase 0 — theory + prereg (this document). Gate: Captain approves §9.
Phase 1 — instrument. Build the four files, smoke every operator fwd/bwd, verify exact-MI gauge against analytic values on 2-level bubbles, verify lane parallelism bit-equivalence to sequential. Gate: all smokes green.
Phase 2 — mass screen. ~1–2k arms through rung0–1 (hours). Gate: SUM control reproduces the divergence signature (validates the instrument against the founding observation — if SUM doesn't diverge here, the theory is wrong and we stop and think).
Phase 3 — LM validation. Rung2–3 survivors (~50→15 arms). Per-stage controller-contrast ablation on finalists (permute stage-t codebook; is each stage load-bearing?). Gate: ≥1 operator with positive marginal bits at m≥3 AND bpb win over budget-matched single aleph.
Phase 4 — scale + graft. Winner's scaling ladder; graft single-aleph Tier-A donor → composite (does pretrained geometry transplant into stage-1?).
Phase 5 — statutes + publication. New temple facts, -lm curation, article.

8. Prior-art map (what exists / what doesn't)

Bubbles-in-bubbles with learned trees: TreeVAE, DeepECT, nCRP-VAE, TreeDiffusion. Subsequent differentiation: RQ lineage → RQ-MoE (2026: two-level MoE, input-dependent codebooks, unifies RQ as degenerate cases) and SwitchCodec/REVQ (2026: shared base quantizer + sparsely routed expert quantizers in a residual hierarchy). Conditional tree routing: HME → S'MoRE (p(child|parent,x) top-down). Conjunctive addressing: PQ, product-key memory. Unoccupied conjunction: signed antipodal addresses; statute-governed stage geometry; prediction through the composed structure at every stage (their routing serves compute/reconstruction, not the predictive head); marginal-bits as the design criterion. That conjunction is exp_011's territory — same shape of gap the single aleph occupied, one level up.

9. Decision points — RESOLVED 2026-07-01

#5 repo: NEW dedicated repo geolip-aleph-differentiation (aleph-lm line stays separate — Captain's call, supersedes the -void proposal). #1–#4: proceeding with stated defaults per go-ahead (all 7 operators; Tier-P2 deferred to Phase 4; param-matched primary; TREE hard+soft both in v1).

9b. Original decision points (record)

Operator set v1: the seven in §3 — cut/add? (CROSS and ANNEAL are the cheapest to defer if we want a tighter first screen.)
Tier-P substrate: synthetic nested bubbles primary; include Tier-P2 (real VAE latents) in Phase 2 or defer to Phase 4?
Budget matching: param-matched primary + bandwidth-matched secondary — agreed?
TREE hard mode: allow top-1 + straight-through sub-arms in v1, or soft-only first?
Repo target: Forge pushes → geolip-aleph-void (raw), curated finals → geolip-aleph-lm (resolves the standing REPO_ID note) — agreed?

10. Risks and honesty notes

Marginal-bits on Tier-L is a probe estimate, biased low; Tier-P exact values are the calibration anchor. Report both, never conflate.
Straight-through gradients (TREE hard) add estimator variance — seed-replicate (2 seeds minimum at rung2+, the exp_007 discipline).
Redundancy vs capacity confound: a stage can look non-redundant by being noise. The SNR gauge exists precisely to separate these; verdicts require both.
Lane parallelism must be proven bit-equivalent before trusting screen results (Phase 1 gate) — shared-RNG contamination across lanes is the known trap.
Tier-P conclusions transfer to LM only as ordering hypotheses, never as numbers; rung2 exists to check the ordering, and disagreement is a finding, not a failure.

Amendment — 2026-07-01 (post Phase-2b, 84 arms, 2 seeds, 8-bit task)

P1 verdict: refuted in the letter, confirmed in the spirit. SUM's probe-channel marginals never went negative through m=8 — necessarily, since the staged probe reads raw per-stage addresses and is therefore itself a conditioned aggregator; it cannot see aggregation damage. The divergence lives in the delivered channel (the head's cross-entropy): SUM's delivered bits plateau flat from m=4 (6.13 → 6.13 → 6.15; acc pinned 0.51; redundancy 0.16 → 0.25) while conditioned operators keep climbing (prod 6.55 → 6.78, acc 0.70). Refined law: unconditioned addition does not stop stages from learning; it destroys the aggregate channel. The divergence gauge v2 is the marginal delivered-bits curve (Δdelivered/Δm), now logged first-class as delivered_bits.

Enrichment confirmed at depth: monotone in m on both channels for res/prod/cross; prod_m8 = 6.78 delivered / 6.43 held of 8.0; seeds agree within 0.04; stage-knowledge saturation unfound at m=8 (final held-marginals ~ +0.14 for all ops).

Instrument note: the 250-step probe underfits at 256 classes (single: held 3.11 vs delivered 5.34). Probe budget now scales as max(probe_steps, 2.5·n_classes). Held values from the pre-fix 2b ledger are low-biased; within-op cross-m comparisons remain valid (shared bias); held-vs-delivered gaps from that ledger are directional only.

Authors

Author: AbstractPhil + Claude Fable 5 Repo: https://huggingface.co/AbstractPhil/geolip-aleph-differentiation/ License: MIT Date: July 1 2026