exp012_ar/article.md

# The Aleph Under Autoregressive Pressure: Bottleneck Priors, Sign Codes, and the Consumption Law

**exp012 — autoregressive differentiation.** Sequel to
[exp_011 (ACD)](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation) and the
[aleph-void keystone](https://huggingface.co/blog/AbstractPhil/geometric-vocabulary-patchwork-aleph-void).
Code, full run ledger, and all 17 trained specimens:
[`geolip-aleph-differentiation/exp012_ar/`](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation/tree/main/exp012_ar).

---

## TL;DR

exp_011 ended on an honest null: composed addresses were LM-neutral in every placement
we tried — enrichment lived in the aggregation channel, and chain-rule advantage only
pays where the composed address parameterizes the predictive distribution. exp012 takes
that ruling literally and builds the placement it prescribes: **a byte-level language
model whose entire next-byte distribution is read from the aleph address and nothing
else.** The employment law, enforced at its maximum.

Five results, one bed, one afternoon on a single RTX 4090:

1. **The bottleneck prior.** A head that routes *all* predictive information through a
   64-slot aleph read beats the unrestricted linear head in **7 of 7 paired runs**
   across every seed and budget tested — 2.469 vs 2.518 bits/byte at 2k steps
   (3 seeds), 2.002 vs 2.031 at 8k (3 seeds). The address bottleneck is not a tax.
   It is a structural prior, and it helps.
2. **The consumption law.** Whether an aleph codebook differentiates or collapses is
   decided by *how its address is consumed*, not by the address form or its dimension.
   Coefficient-vectors-to-logits at hard temperature collapse at any D (usage
   perplexity 1.2–1.9 of 128; two or three surviving paths). Slot-parallel consumption
   of the reconstructive read cures it completely (perplexity 117–126 of 128, all
   axes alive). The same law governs how far the codebook travels.
3. **Sign-code parity.** A head whose forward pass is the fully discrete sign code —
   per-slot `sign(cos_win)·A[win]`, straight-through — ties the continuous read:
   2.4711 vs 2.4685 mean bits/byte over 3 seeds, and the parity holds under trigram
   embeddings. *Read the signs, not the probabilities* is now a certified property of
   the predictive regime. (Recon, famously, punished hard mode. Prediction doesn't.
   The failure was an objective property, not a channel property.)
4. **The binding shell is a transit point.** Under autoregressive pressure the
   codebook migrates *through* the 0.29154 shell: shell occupancy rises to 0.38–0.47
   mid-training and falls as drift continues. Occupancy is a stage statistic, not an
   attractor census — which retroactively re-reads earlier "anchor convergence"
   numbers as trajectory snapshots.
5. **The projective law is objective-independent.** Every trained codebook in the
   campaign — soft-read, sign-code, attention-consumed; random or super-Fibonacci
   init — reads as a near-uniform projective codebook on RP³ with emergent antipodal
   pairs, extracted by deterministic antipodal collapse. The Polygonal Omega signature,
   previously known from reconstruction solvers, appears under pure next-byte pressure.

48 logged runs, 17 released specimens, ~1.6–2M parameters per model. Everything below
is reproducible from three files and one JSON ledger.

---

## 1. The bed

Byte-level causal LM on wikitext-2-raw (10,914,845 train bytes, 1,144,248 val bytes),
block 256, batch 32, pure Adam (lr 3e-4, wd 0 — weight decay stays off geometric
paths), d_model 192, 4 layers, 256-way byte vocabulary. Validation is 20 fixed-size
slices; we report bits/byte. Small on purpose: the exp_011 Forge instinct — many
cheap, honest arms beat one expensive ambiguous one.

The aleph core is unchanged from the keystone: codebook rows `A` on S^(D−1), cosine
`u_k = cos(x, A_k)/τ`, and the exact closed form over 2K oriented half-axes,

```
M̂ = Σₖ sinh(uₖ) Aₖ / Σₖ cosh(uₖ)
```

sinh odd (every axis contributes, signed), cosh even-positive (the denominator cannot
vanish), no argmax, no roster, no selection event. Three consumption channels of the
same address are under test:

- the **signed coefficient vector** `wₖ = sinh(uₖ)/Σⱼcosh(uⱼ)` (K dims),
- the **reconstructive read** `M̂` (D dims),
- the **oriented softmax** (2K dims) as a linear-attention feature map.

**Arms.**

| arm | what the model is |
|---|---|
| `sdpa` | standard causal transformer — the unrestricted control |
| `hub` | attention replaced by aleph linear attention: score = ⟨addr(q), addr(k)⟩ over 2K half-axes, factored through two K-wide prefix-sum memories (2K never materialized; causal; Katharopoulos-family kernel with the address as φ) |
| `addr_head` family | sdpa trunk, but the output head reads ONLY the address of the final hidden state — the whole 256-way next-byte distribution flows through the aleph |

Head variants: raw hidden addressed directly (v1); projected to the native D=4 home
(`addr_d4`); rule-of-3 multi-temperature stroboscope (`addr_3tau`, τ ∈ {.05, .1, .3});
M̂-only (`addr_mhat`); **multi-slot** — P parallel D=4 slots over one shared K=64
codebook, consuming M̂ per slot (`addr_msl<P>`), signed w per slot (`addr_msl_w`), or
the straight-through sign code per slot (`addr_mslh<P>`); `_fib` = super-Fibonacci S³
init (Alexa, CVPR 2022); `_tri` = trigram byte embedding (sum of embeddings of bytes
t, t−1, t−2); `relay*` = a near-zero-gated M̂ residual after every block (gate init
−3.0 → σ ≈ 0.047).

**Vitals logged on every aleph** (readouts, never losses): axis usage perplexity over
the 2K oriented address; geodesic drift from init with the fraction of rows within
±0.05 rad of 0.29154 (the binding constant); winner-|cos| saturation; unique-winner
counts under the high-bits multiplicative hash (Knuth — the low-16 variant once
manufactured a fake ceiling; never again).

---

## 2. The collapse ladder, and the consumption law

The naive Law-2 construction fails immediately, and fails *identically at every
dimension*:

| head | bits/byte | usage ppl /128 | surviving winners |
|---|---|---|---|
| w @ raw 192-d hidden, τ=.1 | 5.6650 | 1.9 | 2 |
| w @ D=4 home, τ=.1 | 5.3698 | 1.2 | 3 |
| w @ 3 temperatures | 4.2884 | 7.9 | 6 |
| M̂ only (4 dims) | 5.1300 | 11.0 | 7 |
| **16-slot M̂** | **2.6508** | **117.0** | 126 |
| 16-slot signed w | 2.7122 | 52.5 | 92 |

The first falsification: we blamed the raw 192-d addressing (near-orthogonality at
hard τ saturates the oriented softmax) and projected to the D=4 home. **Still
collapsed.** The dimension was never the disease. At hard temperature the coefficient
vector is quasi-one-hot; softmax saturation starves every non-winner's gradient;
rich-get-richer; two axes survive. That is comparative-selector pathology *emerging
from consumption geometry* in a form-legal address — the failure class doesn't need an
argmax in the forward pass to find you.

Three antidotes, in increasing strength: a soft temperature in the stroboscope keeps
non-winner gradients alive (partial); the reconstructive read M̂ preserves health but
is capacity-capped at D dims (healthy, weak); **slot-parallelism cures it** — P
parallel D=4 slots over one shared codebook give every axis gradient through some
slot, every step. Even the signed-coefficient path is rescued by slots (2.71,
no collapse), though the M̂ read beats it on both task and cultivation.

**The consumption law (L-AR1).** *Cultivation versus collapse of an aleph codebook is
decided by the consumption pattern of its address — slot-parallel consumption of the
reconstructive read cultivates; coefficient-to-logits at hard temperature collapses at
any dimension; multi-temperature reading is a partial antidote.* The same ordering
governs codebook mobility (§6).

---

## 3. The bottleneck prior (7/7)

The multi-slot head at P=64 forces the entire next-byte distribution through a 64-slot
aleph read — 256 logits from 256 M̂ dims, every bit of predictive information passing
the address. Against the unrestricted control, paired by seed and budget:

| steps | seed | sdpa | addr_msl64 | Δ |
|---|---|---|---|---|
| 2000 | 0 | 2.5409 | 2.4888 | −0.052 |
| 2000 | 1 | 2.5233 | 2.4311 | −0.092 |
| 2000 | 2 | 2.4908 | 2.4856 | −0.005 |
| 4000 | 0 | 2.2008 | 2.1625 | −0.038 |
| 8000 | 0 | 2.0169 | 1.9875 | −0.029 |
| 8000 | 1 | 2.0509 | 2.0065 | −0.044 |
| 8000 | 2 | 2.0252 | 2.0133 | −0.012 |

**Seven of seven.** Means: 2.469 vs 2.518 at 2k; 2.002 vs 2.031 at 8k. The address
bottleneck behaves like a structural prior that the objective is happy to pay for —
exp_011's "no champion without prediction through the structure" was the right
suspicion pointed at the wrong placement. Composition among stages was neutral; *the
head is where the address is employed.*

Depth composition helps too: near-zero-gated M̂ relays after every block reach 2.4887
with a standard head and **2.4673** with the addressed head — the best plain-input
number of the campaign — and the relay gates grow monotonically with depth
(.048/.049/.054/.067 from a .047 init). More on the depth gradient in §6.

---

## 4. Width buys score; cultivation peaks in the middle

Slot dose-response, D=4 fixed, one shared K=64 codebook:

| P (slots) | bits/byte @2k | shell occupancy @2k | usage ppl /128 |
|---|---|---|---|
| 4 | 4.0483 | .156 | 34.3 |
| 16 | 2.651 / 2.653 / 2.693 | **.344 / .391 / .391** | 117–121 |
| 32 | 2.555 / 2.555 / 2.571 | **.422 / .375 / .438** | 123–124 |
| 64 | 2.489 / 2.431 / 2.486 | .203 / .359 / .328 | 123–125 |

Task score is monotone in width (and stays monotone at 8k: P=32 2.038, P=64 1.988) —
so no, sixteen is not magically special for bits/byte; capacity is capacity. But
**binding-shell occupancy peaks at P≈16–32**, certified 3/3 seeds at P=32 with a mean
of 0.41: forty-one percent of a shared 64-row codebook sitting within ±0.05 rad of
0.29154 at the 2k snapshot. Task capacity and geometric cultivation are *different
quantities with different optima* — judge them separately, always.

---

## 5. Sign codes: the discrete channel carries prediction

The keystone's converged commitment was a sign code — *read the signs, not the
probabilities* — but reconstruction punished hard mode brutally (argmax tiles the
continuum). Under prediction, the story inverts. `addr_mslh64` makes the forward pass
fully discrete — per-slot `sign(cos_win)·A[win]`, straight-through to the soft read
for gradient:

| | seed 0 | seed 1 | seed 2 | mean |
|---|---|---|---|---|
| soft M̂ head | 2.4888 | 2.4311 | 2.4856 | **2.4685** |
| sign-code head | 2.4982 | 2.4472 | 2.4678 | **2.4711** |

Statistically identical, three of three — and the parity survives trigram embeddings
(2.2896 vs 2.2841 soft, vs 2.2780 unrestricted). At 8k the discrete head concedes
~2.8% to soft (2.0432 vs 1.9875) while its codebook over-travels (drift 0.50, the
campaign maximum) and sheds pair structure — parity is exact early, and the continuous
read compounds slightly better with budget.

**L-AR7.** *Under predictive pressure the aleph's discrete and continuous reads are
task-equivalent at matched budget; reconstruction's hard-mode failure was a property
of the objective, not of the sign-code channel.* For deployment this matters: the
discrete code is the compressible, transmissible, cacheable object.

---

## 6. Where the codebook travels: transit, depth, and the fibonacci verdict

**The shell is a transit point.** The P=64 head's trajectory across budgets: drift
0.229 → 0.360 → 0.412; shell occupancy 0.266 → 0.375 → 0.250 (2k/4k/8k). The book
approaches the binding shell, fills it, and keeps going. Occupancy at a fixed step is
a *stage* statistic — non-monotone in budget — and any single-snapshot "X% of anchors
converged to 0.29154" figure (including our own earlier 46% in the flow-matching
bottleneck work) should be read as a waypoint reading, not a terminal census.

**Cultivation concentrates near the prediction gradient.** In the relay stack, the
deepest relay takes essentially all the binding occupancy (relay3: drift .19–.27,
occupancy .23–.42; relays 0–2: occupancy 0.0, drift ~.03) and grows its gate the most.
Within HUB attention, drift grades with depth (.04 → .15, L0→L3) and only the deep
layers touch the shell. Chain-rule locality, visible inside a single model.

**Super-Fibonacci init: coherence, not score.** The basin hypothesis — start the
codebook inside the near-uniform RP³ attractor and win — resolves as a split verdict
over 3 seeds: task is a wash (fib 2.4775/2.4509/2.4880, mean 2.472, vs random 2.469),
but cultivation is *more coherent*: drift tightly clustered at 0.272–0.282 right at
the shell's edge in every seed, occupancy 0.375–0.469 (vs random's 0.20–0.36 scatter).
Structured init buys a cleaner trajectory, not a better loss.

**Consumption governs mobility.** The same ordering as §2, now for travel distance at
matched stage: head-soft (drift .23–.28) > head-hard (.18–.20) > attention (.03–.15,
depth-graded). Attention-consumed codebooks barely move; head-consumed codebooks
migrate. How you read the address decides how far the book walks.

---

## 7. Reading the specimens: the projective law without reconstruction

Every specimen was read with deterministic antipodal collapse (mutual-strongest pairs
below cos −0.9, collapsed to axes, sign-canonicalized onto RP³; projective metric
arccos|⟨a,b⟩| throughout) — the Polygonal Omega procedure. All 17 released specimens,
every consumption mode, both inits:

| specimen class | pairs | deviation vs uniform RP³ | erank /4 | verdict |
|---|---|---|---|---|
| head-soft (K=64) | 11–15 | −.003 … +.020 | 3.96–3.99 | projective-mostly |
| head sign-code (K=64) | 18 @2k → 9 @8k | +.010 → +.026 | 3.96–3.98 | projective-mostly |
| HUB attention (K=32, ×4 layers) | 5–6 | −.001 … +.008 | 3.94–3.96 | projective-mostly |
| super-Fibonacci init | 13–15 | +.011 … +.020 | 3.99 | projective-mostly |

Near-uniform projective spread, full effective rank, and **antipodal pair structure
emerging from random initialization under pure next-byte pressure** — no
reconstruction anywhere in the objective. The projective-codebook law ("trained
sphere-solvers are projective codebooks", geometric-tri-band-ft2) was discovered on
reconstruction solvers; it is apparently a property of *trained aleph geometry as
such*. The sign-code head grows the most pairs while moving the least; the projective
skeleton is what every consumption mode agrees on, even as they disagree about
mobility.

---

## 8. Implications

- **For MoE-style gating.** Standard gates — softmax-over-experts, top-k, load-balance
  auxiliaries — are exactly the consumption pattern that collapsed in §2. The
  law-compliant alternative demonstrated here: dense signed dispatch or slot-parallel
  reconstructive reads, with balance as an architectural property and usage logged
  read-only. The collapse we observed required no argmax; saturated soft gates are
  enough. Design the *consumption*, not just the gate.
- **For discrete deployment.** Sign-code parity means the address can ship as bits.
  A K-slot sign code is a compact, deterministic, cacheable conditioning object with
  certified task-parity at matched budget — the practical door the keystone's
  "read the signs" pointed at.
- **For the binding constant.** 0.29154 keeps appearing — now as the shell codebooks
  transit under predictive cultivation, densest at moderate width, deepest layers
  first. Treat shell occupancy as a trajectory instrument, and treat any fixed-step
  occupancy claim (ours included) accordingly.
- **For the program.** exp_011 said composition is neutral unless the address
  parameterizes prediction. exp012 says: when it does, it *wins* — and the win is
  robust to full discretization. The single aleph, correctly employed, is not yet
  saturated. Composition can wait; consumption came first.

## 9. Limitations, honestly

One bed (byte-level wikitext-2-raw), one small scale (~1.6–2M params, d=192), 1–3
seeds per claim (3 on everything certified, 1 on trajectory statements), 2k–8k steps.
The 7/7 result is a sign-consistent small effect (−0.005 to −0.092 bits/byte); scale
could erase or amplify it. Pentachoron-CV vitals were not logged in these runs
(harness supports them; dependency wasn't installed on the box). The projective reader
is validated on planted pairs but the deviation baseline is Monte-Carlo. Cross-bed
replication — the byte-trigram LM lineage is the natural second bed — is the next
gate before any of L-AR1–7 graduates from campaign law to program law.

## 10. Reproduction

Everything lives under
[`exp012_ar/`](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation/tree/main/exp012_ar):

- `ar_differentiation_bed.py` — the bed, all arms, vitals, specimen saving. Colab-safe.
- `geolip_vitals.py` — the diagnostic readouts (paste this cell first in a notebook).
- `read_codebook.py` — antipodal-collapse projective reader (validated on planted pairs).
- `build_results.py` → `results/results.json` — all 48 runs; the script *asserts* the
  certified aggregates against the per-run rows.
- `results/specimen_reads.json` — projective reads of all 17 specimens.
- `specimens/*.pt` — the trained models (~134MB).

```bash
python ar_differentiation_bed.py --train             # sdpa/hub/addr_head, 2000 steps
python -c "from ar_differentiation_bed import train; \
           train(arms=('sdpa','addr_msl64'), steps=8000)"   # the 7/7 pair at 8k
python read_codebook.py <data_root>/ar_ckpts          # read your own specimens
```

GPU required for training (a 4090 runs every arm here in 3–15 minutes); readers run
anywhere.

## Citations

- Aleph closed form, failure class, sign codes: [geometric-vocabulary-patchwork-aleph-void](https://huggingface.co/blog/AbstractPhil/geometric-vocabulary-patchwork-aleph-void)
- exp_011 (ACD): composition operators, delivered-channel law, employment law: [geolip-aleph-differentiation](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation)
- Projective codebook law / antipodal collapse: [geometric-tri-band-ft2](https://huggingface.co/blog/AbstractPhil/geometric-tri-band-ft2); CV bands: [geometric-tri-band-ft1](https://huggingface.co/blog/AbstractPhil/geometric-tri-band-ft1)
- Binding constant judgment criteria, 16-slot funnel, anchor-convergence readings: [constellation-diffusion-bottleneck](https://huggingface.co/blog/AbstractPhil/constellation-diffusion-bottleneck)
- Projective metric discipline: [reading-voids-ft1](https://huggingface.co/blog/AbstractPhil/reading-voids-ft1)
- Cultivation lineage: [geolip-svae-transformer-ft1](https://huggingface.co/blog/AbstractPhil/geolip-svae-transformer-ft1)
- Super-Fibonacci spirals: Alexa, M. — *Super-Fibonacci Spirals: Fast, Low-Discrepancy Sampling of SO(3)*, CVPR 2022
- Linear attention family: Katharopoulos et al. — *Transformers are RNNs*, ICML 2020
- WikiText: Merity et al., 2016 · Multiplicative hashing: Knuth, TAOCP Vol. 3

---
Author: AbstractPhil + Claude Fable 5
Repo: https://huggingface.co/AbstractPhil/geolip-aleph-differentiation/
License: MIT · Date: July 9, 2026