exp014_gd/article.md

Genetic Distillation and the Memory Substrate: What Inherits, What Merely Travels

exp014 — the third campaign of the differentiation series. Sequel to exp012 (article) and exp013. Code, full ledger (including the aborted run — it is the evidence), and 23 champion genomes: exp014_gd/.

---

TL;DR

We ran multi-generational tournaments of small byte-LMs — the geometric-memory-ft3 evolutionary paradigm with one upgrade: the aleph codebook as an explicit heritable genome, inherited by Procrustes/GPA consensus (placement by construction; this deliberately replaces GM3's k-means-on-consensus anchor init). Six lineages, a catastrophic-parent probe, and a two-track implantation study. Findings, each with a built-in control:

1. Inverse evolution through logit inheritance. Knowledge distillation at full weight from near-parity teachers doesn't just fail to help — it compounds downward (best-of-generation 2.4301 → 2.5046 → 2.5603 bits/byte), because selection feeds each generation's degraded champions back as teachers. The gene-flow founder — same generation, no inheritance — degraded identically, isolating KD as the cause. KD's classic regime is a much better teacher; at teacher≈student parity it is an anchor, not a lift.
2. Inheritance pays iff trunk continuity. With KD tamed (α=0.25, founders exempt), all three full-weight lineages ascend monotonically and near-identically (−0.049 to −0.054 over three generations); both organ-only lineages (head projection + consensus book transplanted onto fresh trunks) never beat their own founders; and the no-inheritance floor's best lucky draw (2.4177) beats organ heredity outright. The co-adapted whole transfers. Organs alone are a below-random initialization.
3. The germline buys stability, not score. On top of full-weight continuity, the consensus-book overwrite is task-neutral (final best 2.4077 vs the pure-continuity control's 2.4046) — but it drives the codebook to a lineage fixed point by generation 2 (parent→child drift 0.24 → 0.083 → 0.003) while un-overwritten books drift ~0.20 every generation, forever. Heredity manufactures its own attractor; the task score is indifferent to it at this scale.
4. Catastrophic parents are NOT absorbed. One zero-step random book in a three-way consensus poisons the generation (+0.10 bpb) — the opposite of GM3's celebrated robustness. A projective consensus is precise, and precision is fragile to garbage. Gate your germline.
5. Implantation transfers stability, not score. Mature (fixed-point) books implant into a 3× larger organism at parity (2.1014 vs 2.1000 fresh) where immature books cost ~0.02 and freezing ~0.015; implanted into GPT-2's relay adapters, both donors sit at exact score parity with random books — but the mature book drifts less inside the foreign trunk. Nothing accelerated; nothing broke; stability traveled.

The unifying insight — and the compass it leaves: score-wise, one healthy near-uniform codebook is as good as another. That is the flip side of the projective-universality result from exp012/013 (every healthy book converges to the same near-uniform RP³ statistics). What heredity and maturity buy is stability. So genetic methods can only pay where the book's content is load-bearing — sign-code deployment, embedding-faithful reads, addressed conditioning — not where any healthy scaffold suffices.

---

1. The design

Organism: the exp012 byte-LM bed (d=192, 4 layers, wikitext-2 bytes, 2000 steps/member, pure Adam wd=0, ~3.5 min each on one RTX 4090). Tournament: population 4, four generations, top-2 selection by validation bits/byte, one fresh founder per generation (GM3 gene flow). All founders of a lineage share a common-ancestor codebook so genome rows correspond across the population — which makes consensus well-defined.

The consensus operator (the germline): projective Procrustes with iterated per-row sign flips (books live on RP³), GPA-iterated to the mean shape and anchored back to the best parent's frame. Validated on planted ground truth before any tournament use: two rotated, sign-flipped copies of one book realign to it at |cos| = 1.000 in five iterations. This is placement by construction — no k-means anywhere near the germline.

Lineages:

lineage	inherits
mlp_kd	full best-parent weights + KD (the traditional distillation lineage)
aleph_weights	full best-parent weights, aleph head (pure-continuity control)
aleph_full	full weights + consensus-book overwrite (germline on continuity)
aleph_flat	head organ only: best-parent projection + consensus book
aleph_tree	structured organ: root book + 4 branch books (tree head)
no_inherit	nothing (the evolution floor)

The tree head — a root aleph producing dense soft weights over four branch books — needed its own pre-tournament fix: a single hard-τ 4-way root partially collapsed (usage [.85, .12, .01, .02]), extending exp012's consumption law to low-cardinality gates; slot-parallel root consumption (4 root slots averaged, τ=0.3) restored health. Same disease, same cure, smaller gate.

2. The brackets

Best-of-generation, bits/byte (full per-member ledger in results/ledger.jsonl; build_results.py re-asserts every claim below):

lineage	g0	g1	g2	g3
mlp_kd	2.4106	2.3707	2.3662	2.3594
aleph_weights	2.4585	2.4183	2.4116	2.4046
aleph_full	2.4564	2.4202	2.4143	2.4077
aleph_flat (organ)	2.4580	2.4952	2.4653	2.4731
aleph_tree (organ)	2.6274	2.6896	2.6522	2.6488
no_inherit (floor)	2.4560	2.4527	2.4298	2.4177

The three full-weight lineages ascend in lockstep; heirs dominate their own gene-flow founders every generation. The two organ lineages never recover their founders' level — the heirs' books also under-cultivate (drift 0.14 vs the founders' 0.24): a head organ co-adapted to its parent's trunk forces a fresh trunk to bend around it. And the floor's sixteen lottery tickets (2.4177) beat everything organ heredity produced. The aborted first campaign (kept in the ledger as superseded rows) is finding #1: at KD α=1.0, best-of-generation degraded monotonically — inverse evolution, isolated by the founder control.

The fixed point. In aleph_full, parent→child book drift collapses across generations: 0.24 → 0.083 → 0.003 → 0.006. By generation 2 the consensus book is stationary while trunks keep improving around it. In aleph_weights (no overwrite) the book drifts ~0.20 in every generation indefinitely. Same scores, categorically different germline dynamics.

The probe. A separate bracket injected a zero-step random book into the three-way consensus at generation 2: clean g1 best 2.4360 → poisoned g2 best 2.5354. Not absorbed. GM3's catastrophic-robustness does not transfer to projective book-consensus at this scale.

3. The implants (memory substrate)

The D=4 home makes books size- and architecture-agnostic; we tested both directions.

Cross-size (d=128/2-layer donor → d=384/6-layer organism, 2000 steps):

arm	single-run donor	fixed-point champion donor
fresh random book	2.0960	2.1000
implant, trainable	2.1171	2.1014
implant, frozen	2.1134	2.1134
MLP head control	2.1214	2.1257

The mature champion book implants at parity; the immature book costs ~0.02; freezing costs ~0.015 regardless of donor. No arm accelerated anything. (Side result: at d=384 co-trained, the aleph head beats the matched MLP head — the exp012 co-training advantage at double width.)

Cross-architecture (books into GPT-2's twelve relay adapters, exp013 bed): 25.983–26.010 ppl across all four cells — exact parity — with one tell: the champion book drifts less inside the foreign trunk (0.28–0.32 vs 0.32–0.44). Stability transfers even where score cannot.

4. What this means

• For distillation practice: logit-KD between near-parity models inside a selection loop is actively harmful — a feedback anchor. Weight continuity is the inheritance that works; geometry rides on top of it safely but silently.
• For the aleph program: the codebook's task value at this scale is its health, not its identity — any near-uniform book serves. Its identity becomes valuable exactly where the code itself is consumed: discrete sign-code conditioning, metric-faithful retrieval, addressed lookup. Genetic methods should therefore inherit content-bearing structures, and be judged on content tasks.
• For the memory-substrate idea: it is legal (nothing breaks across size or architecture), stable when mature, and not yet profitable. The profitable version must make the transferred content do work the recipient cannot re-derive in 2000 steps — longer budgets, content tasks, or both.

5. Limitations

One tournament seed for the main brackets (the aborted campaign doubles as a replicate only for the inverse-evolution finding); 2000-step organisms (maturity and transfer effects may scale differently); one bed (wikitext-2 bytes); KD explored at only two α values; the tree lineage was tested only in the (failing) organ regime — its continuity-regime bracket is future work; per-generation projective reads were logged as vitals, not deep-read.

6. Reproduction

The package is standalone — every code dependency ships in exp014_gd/ itself (geolip_vitals.py, ar_differentiation_bed.py, exp013_augmentation_bed.py for the GPT-2 relays, read_codebook.py). From inside the folder:

pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install pyarrow huggingface_hub          # + transformers for --mode b2
python repro.py                              # CPU smoke: parse + shapes + GPA sanity
python repro.py --mode tournament --lineage aleph_full   # one bracket (GPU, ~20 min)
python repro.py --mode b1                    # cross-size implants (GPU)
python repro.py --mode b2 --genome genomes/champion_aleph_full_t0_g3.pt
python build_results.py   # re-asserts every claim in this article from the ledger

Data lands in ./data (override with GEOLIP_DATA). In Colab, paste geolip_vitals.py, ar_differentiation_bed.py, then exp014_genetic_distillation.py into cells in that order and call the runners directly.

genomes/ holds the 23 per-generation champion books (champion_{lineage}_t{seed}_g{gen}.pt) — the germline fossil record: python read_codebook.py genomes/.

Citations

• GM3 paradigm (consensus distillation, evolution, geometric autograd): geometric-memory-ft3
• The bed, the consumption law, sign codes: aleph-autoregressive-differentiation-ft1
• Pretrained-substrate results this builds on: exp013_aug
• Projective codebook law: geometric-tri-band-ft2
• Generalized Procrustes analysis: Gower, Psychometrika 1975
• Knowledge distillation: Hinton, Vinyals & Dean, 2015

---

Author: AbstractPhil + Claude Fable 5 Repo: https://huggingface.co/AbstractPhil/geolip-aleph-differentiation/ License: MIT · Date: July 10, 2026