exp014_gd/article.md

# Genetic Distillation and the Memory Substrate: What Inherits, What Merely Travels

**exp014 — the third campaign of the differentiation series.** Sequel to
[exp012](../exp012_ar/) ([article](https://huggingface.co/blog/AbstractPhil/aleph-autoregressive-differentiation-ft1))
and [exp013](../exp013_aug/). Code, full ledger (including the aborted run — it is
the evidence), and 23 champion genomes:
[`exp014_gd/`](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation/tree/main/exp014_gd).

---

## TL;DR

We ran multi-generational tournaments of small byte-LMs — the
[geometric-memory-ft3](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft3)
evolutionary paradigm with one upgrade: the aleph codebook as an **explicit heritable
genome**, inherited by Procrustes/GPA consensus (placement by construction; this
deliberately replaces GM3's k-means-on-consensus anchor init). Six lineages, a
catastrophic-parent probe, and a two-track implantation study. Findings, each with a
built-in control:

1. **Inverse evolution through logit inheritance.** Knowledge distillation at full
   weight from near-parity teachers doesn't just fail to help — it *compounds
   downward* (best-of-generation 2.4301 → 2.5046 → 2.5603 bits/byte), because
   selection feeds each generation's degraded champions back as teachers. The
   gene-flow founder — same generation, no inheritance — degraded identically,
   isolating KD as the cause. KD's classic regime is a *much better* teacher;
   at teacher≈student parity it is an anchor, not a lift.
2. **Inheritance pays iff trunk continuity.** With KD tamed (α=0.25, founders
   exempt), all three full-weight lineages ascend monotonically and near-identically
   (−0.049 to −0.054 over three generations); both organ-only lineages (head
   projection + consensus book transplanted onto fresh trunks) never beat their own
   founders; and the no-inheritance floor's best lucky draw (2.4177) beats organ
   heredity outright. The co-adapted whole transfers. Organs alone are a
   *below-random* initialization.
3. **The germline buys stability, not score.** On top of full-weight continuity, the
   consensus-book overwrite is task-neutral (final best 2.4077 vs the pure-continuity
   control's 2.4046) — but it drives the codebook to a **lineage fixed point** by
   generation 2 (parent→child drift 0.24 → 0.083 → 0.003) while un-overwritten books
   drift ~0.20 every generation, forever. Heredity manufactures its own attractor;
   the task score is indifferent to it at this scale.
4. **Catastrophic parents are NOT absorbed.** One zero-step random book in a
   three-way consensus poisons the generation (+0.10 bpb) — the opposite of GM3's
   celebrated robustness. A projective consensus is precise, and precision is
   fragile to garbage. Gate your germline.
5. **Implantation transfers stability, not score.** Mature (fixed-point) books
   implant into a 3× larger organism at parity (2.1014 vs 2.1000 fresh) where
   immature books cost ~0.02 and freezing ~0.015; implanted into GPT-2's relay
   adapters, both donors sit at exact score parity with random books — but the
   mature book drifts less inside the foreign trunk. Nothing accelerated; nothing
   broke; stability traveled.

**The unifying insight** — and the compass it leaves: score-wise, one healthy
near-uniform codebook is as good as another. That is the flip side of the
projective-universality result from exp012/013 (every healthy book converges to the
same near-uniform RP³ statistics). What heredity and maturity buy is *stability*.
So genetic methods can only pay where the book's **content** is load-bearing —
sign-code deployment, embedding-faithful reads, addressed conditioning — not where
any healthy scaffold suffices.

---

## 1. The design

**Organism**: the exp012 byte-LM bed (d=192, 4 layers, wikitext-2 bytes, 2000
steps/member, pure Adam wd=0, ~3.5 min each on one RTX 4090). **Tournament**:
population 4, four generations, top-2 selection by validation bits/byte, one fresh
founder per generation (GM3 gene flow). All founders of a lineage share a
**common-ancestor codebook** so genome rows correspond across the population — which
makes consensus well-defined.

**The consensus operator** (the germline): projective Procrustes with iterated
per-row sign flips (books live on RP³), GPA-iterated to the mean shape and anchored
back to the best parent's frame. Validated on planted ground truth before any
tournament use: two rotated, sign-flipped copies of one book realign to it at
|cos| = 1.000 in five iterations. This is placement by construction — no k-means
anywhere near the germline.

**Lineages**:

| lineage | inherits |
|---|---|
| `mlp_kd` | full best-parent weights + KD (the traditional distillation lineage) |
| `aleph_weights` | full best-parent weights, aleph head (pure-continuity control) |
| `aleph_full` | full weights **+ consensus-book overwrite** (germline on continuity) |
| `aleph_flat` | head organ only: best-parent projection + consensus book |
| `aleph_tree` | structured organ: root book + 4 branch books (tree head) |
| `no_inherit` | nothing (the evolution floor) |

The tree head — a root aleph producing dense soft weights over four branch books —
needed its own pre-tournament fix: a single hard-τ 4-way root partially collapsed
(usage [.85, .12, .01, .02]), extending exp012's consumption law to low-cardinality
gates; slot-parallel root consumption (4 root slots averaged, τ=0.3) restored health.
Same disease, same cure, smaller gate.

## 2. The brackets

Best-of-generation, bits/byte (full per-member ledger in `results/ledger.jsonl`;
`build_results.py` re-asserts every claim below):

| lineage | g0 | g1 | g2 | g3 |
|---|---|---|---|---|
| mlp_kd | 2.4106 | 2.3707 | 2.3662 | **2.3594** |
| aleph_weights | 2.4585 | 2.4183 | 2.4116 | **2.4046** |
| aleph_full | 2.4564 | 2.4202 | 2.4143 | **2.4077** |
| aleph_flat (organ) | 2.4580 | 2.4952 | 2.4653 | 2.4731 |
| aleph_tree (organ) | 2.6274 | 2.6896 | 2.6522 | 2.6488 |
| no_inherit (floor) | 2.4560 | 2.4527 | 2.4298 | 2.4177 |

The three full-weight lineages ascend in lockstep; heirs dominate their own gene-flow
founders every generation. The two organ lineages never recover their founders' level
— the heirs' books also *under-cultivate* (drift 0.14 vs the founders' 0.24): a head
organ co-adapted to its parent's trunk forces a fresh trunk to bend around it.
And the floor's sixteen lottery tickets (2.4177) beat everything organ heredity
produced. The aborted first campaign (kept in the ledger as `superseded` rows) is
finding #1: at KD α=1.0, best-of-generation *degraded monotonically* — inverse
evolution, isolated by the founder control.

**The fixed point.** In `aleph_full`, parent→child book drift collapses across
generations: 0.24 → 0.083 → 0.003 → 0.006. By generation 2 the consensus book is
stationary while trunks keep improving around it. In `aleph_weights` (no overwrite)
the book drifts ~0.20 in every generation indefinitely. Same scores, categorically
different germline dynamics.

**The probe.** A separate bracket injected a zero-step random book into the
three-way consensus at generation 2: clean g1 best 2.4360 → poisoned g2 best 2.5354.
Not absorbed. GM3's catastrophic-robustness does not transfer to projective
book-consensus at this scale.

## 3. The implants (memory substrate)

The D=4 home makes books size- and architecture-agnostic; we tested both directions.

**Cross-size** (d=128/2-layer donor → d=384/6-layer organism, 2000 steps):

| arm | single-run donor | fixed-point champion donor |
|---|---|---|
| fresh random book | 2.0960 | 2.1000 |
| implant, trainable | 2.1171 | **2.1014** |
| implant, frozen | 2.1134 | 2.1134 |
| MLP head control | 2.1214 | 2.1257 |

The mature champion book implants at parity; the immature book costs ~0.02; freezing
costs ~0.015 regardless of donor. No arm accelerated anything. (Side result: at
d=384 co-trained, the aleph head beats the matched MLP head — the exp012 co-training
advantage at double width.)

**Cross-architecture** (books into GPT-2's twelve relay adapters, exp013 bed):
25.983–26.010 ppl across all four cells — exact parity — with one tell: the champion
book drifts less inside the foreign trunk (0.28–0.32 vs 0.32–0.44). Stability
transfers even where score cannot.

## 4. What this means

- **For distillation practice**: logit-KD between near-parity models inside a
  selection loop is actively harmful — a feedback anchor. Weight continuity is the
  inheritance that works; geometry rides on top of it safely but silently.
- **For the aleph program**: the codebook's *task value* at this scale is its
  health, not its identity — any near-uniform book serves. Its *identity* becomes
  valuable exactly where the code itself is consumed: discrete sign-code
  conditioning, metric-faithful retrieval, addressed lookup. Genetic methods should
  therefore inherit **content-bearing** structures, and be judged on content tasks.
- **For the memory-substrate idea**: it is legal (nothing breaks across size or
  architecture), stable when mature, and not yet profitable. The profitable version
  must make the transferred content do work the recipient cannot re-derive in 2000
  steps — longer budgets, content tasks, or both.

## 5. Limitations

One tournament seed for the main brackets (the aborted campaign doubles as a
replicate only for the inverse-evolution finding); 2000-step organisms (maturity and
transfer effects may scale differently); one bed (wikitext-2 bytes); KD explored at
only two α values; the tree lineage was tested only in the (failing) organ regime —
its continuity-regime bracket is future work; per-generation projective reads were
logged as vitals, not deep-read.

## 6. Reproduction

The package is standalone — every code dependency ships in
[`exp014_gd/`](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation/tree/main/exp014_gd)
itself (`geolip_vitals.py`, `ar_differentiation_bed.py`,
`exp013_augmentation_bed.py` for the GPT-2 relays, `read_codebook.py`).
From inside the folder:

```bash
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install pyarrow huggingface_hub          # + transformers for --mode b2
python repro.py                              # CPU smoke: parse + shapes + GPA sanity
python repro.py --mode tournament --lineage aleph_full   # one bracket (GPU, ~20 min)
python repro.py --mode b1                    # cross-size implants (GPU)
python repro.py --mode b2 --genome genomes/champion_aleph_full_t0_g3.pt
python build_results.py   # re-asserts every claim in this article from the ledger
```

Data lands in `./data` (override with `GEOLIP_DATA`). In Colab, paste
`geolip_vitals.py`, `ar_differentiation_bed.py`, then
`exp014_genetic_distillation.py` into cells in that order and call the runners
directly.

`genomes/` holds the 23 per-generation champion books (`champion_{lineage}_t{seed}_g{gen}.pt`)
— the germline fossil record: `python read_codebook.py genomes/`.

## Citations

- GM3 paradigm (consensus distillation, evolution, geometric autograd):
  [geometric-memory-ft3](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft3)
- The bed, the consumption law, sign codes:
  [aleph-autoregressive-differentiation-ft1](https://huggingface.co/blog/AbstractPhil/aleph-autoregressive-differentiation-ft1)
- Pretrained-substrate results this builds on: [exp013_aug](../exp013_aug/)
- Projective codebook law: [geometric-tri-band-ft2](https://huggingface.co/blog/AbstractPhil/geometric-tri-band-ft2)
- Generalized Procrustes analysis: Gower, *Psychometrika* 1975
- Knowledge distillation: Hinton, Vinyals & Dean, 2015

---
Author: AbstractPhil + Claude Fable 5
Repo: https://huggingface.co/AbstractPhil/geolip-aleph-differentiation/
License: MIT · Date: July 10, 2026