| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - aleph-routed-attention |
| - linear-attention |
| - geometric-deep-learning |
| - language-model |
| - byte-level |
| - mixture-of-pointers |
| - research |
| library_name: pytorch |
| pipeline_tag: text-generation |
| --- |
| |
| # geolip-aleph-lm |
|
|
| **Aleph-routed attention: a signed-projective geometric codebook serving as |
| both the attention feature map and the prediction substrate of a byte-level |
| language model.** |
|
|
| This repository holds the code, checkpoints, and experiment ledger for the |
| *aleph* line of the GeoLIP geometric program. It is a **research instrument**, |
| not a product model — the artifacts here are language models in the strict |
| density-estimation sense (they predict the next bytes of English), and they |
| are studied for what their geometry does, not deployed for what they say. |
|
|
| > Part of the GeoLIP geometric ecosystem. Companion repositories: |
| > [`procrustes-analysis`](https://huggingface.co/AbstractPhil) (the 17-model |
| > cross-architecture survey), `geolip-deep-embedding-analysis` (the |
| > 65,536-config CV-band study), and the GeoLIP constellation packages. |
|
|
| --- |
|
|
| ## What an "aleph" is |
|
|
| An aleph is a learned codebook of **K** unit directions in a low-dimensional |
| address space (**D**-dimensional, default D=4). At each position the model |
| computes a **signed antipodal address** over those directions, |
|
|
| ``` |
| p± = exp(±u − m) / Z |
| ``` |
|
|
| a pair of probability vectors (positive and negative hemisphere) on the |
| **2K**-simplex, with a guaranteed-positive denominator that makes the whole |
| operation a valid linear-attention feature map and exactly streamable. The |
| same codebook can serve two roles: |
|
|
| 1. **Routing** — the address is the attention kernel feature map, giving |
| O(n·K) attention instead of O(n²) softmax. *Every* configuration in this |
| repo is aleph-**routed**. |
| 2. **Prediction** — when the output head scores candidates *through* the |
| codebook (the `kernel` and `apmix` heads), prediction error flows back into |
| the codebook geometry, closing a control loop. This is aleph **prediction**, |
| and it is the load-bearing claim of the program. |
|
|
| The distinction between routing and prediction is not cosmetic — it is the |
| single most important finding here, measured directly (see |
| [The controller contrast](#the-controller-contrast)). |
|
|
| --- |
|
|
| ## Architecture |
|
|
| `AlephLM` is a byte-level trigram model: it predicts the next three bytes |
| jointly through a **gate × bank + byte-tail** mixture. |
|
|
| - **Backbone** — a stack of aleph-routed attention layers (signed-projective |
| addressing, linear in sequence length, constant recurrent state). The 23M |
| configuration carries ~532K floats of recurrent state and an effective |
| context of ~12 KB through streaming, at constant memory. |
| - **Heads** (selectable, this is what the experiments compare): |
| - `byte` — plain byte softmaxes (the control). |
| - `kernel` — π·κ log-kernel over the bank, *through the codebook* (the |
| original aleph predictor; rank-bounded on dense banks by construction). |
| - `pmix` — a mixture-of-pointers over an independent learned sphere |
| (escapes the rank bound, but **bypasses the codebook** — see the audit). |
| - `apmix` — mixture-of-pointers aiming in the **Hellinger embedding of the |
| signed codebook address** (√κ on S^(2K−1), Bhattacharyya affinity). Keeps |
| the rank escape *and* restores full prediction pressure through the |
| codebook. This is the synthesis head. |
| - **Bank** — an external tensor of candidate trigrams drawn from a stratified |
| lexical atlas (12.9M n-grams across char/word/unicode strata). The bank is |
| *data, not parameters* — swappable without retraining (demonstrated below). |
| - **Statute instrumentation** — a live gauge reports the codebook's geometric |
| state every log step: deviation on a degenerate↔uniform↔polytope axis, |
| effective rank, minimum-angle spread, routing concentration. Collapse and |
| health are visible in real time. |
| |
| --- |
|
|
| ## The two-hemisphere law |
|
|
| The codebook is governed by two opposing forces, both observed directly: |
|
|
| - **Routing pressure is rank-destructive.** Under a head that does not employ |
| the codebook for prediction, streamed long-context routing drives the |
| codebook toward low rank (a 23M streamed run fell from effective rank 3.66 to |
| 1.89; see the calibration run below). |
| - **Prediction pressure is rank-constructive.** Under a head that scores |
| through the codebook, discrimination demand *builds* rank (kernel runs climb |
| monotonically, effective rank 3.82 → 3.93 over 10K steps, four-way |
| replicated). |
|
|
| A codebook collapses not because something attacks it but because **its only |
| demanding employer was removed**. This reframes "preservation" as |
| *employment*. |
|
|
| --- |
|
|
| ## The controller contrast |
|
|
| The decisive measurement. Two trained 23M checkpoints — one with a |
| codebook-employing `kernel` head, one with a codebook-bypassing `pmix` head — |
| subjected to the *identical* codebook ablation (permute the codebook rows, |
| re-measure bits-per-byte): |
|
|
| | model | head | permute Δbpb | randomize Δbpb | |
| |---|---|---|---| |
| | kernel (closed loop) | `kernel` | **+2.33** | +2.26 | |
| | pmix (open loop) | `pmix` | **+0.009** | +0.020 | |
|
|
| Same formulas, same K=64/D=4 codebook, same corpus. A **~250× dependence |
| ratio**. The aleph is a genuine *structural controller* — load-bearing to the |
| tune of 2.3 bits per byte — **exactly where prediction closes the loop, and |
| vestigial where it does not.** The bypassing model had learned to route around |
| its own codebook. |
|
|
| This is why the program's verdict is *no champion without the aleph doing |
| work*: a strong bits-per-byte number from a bypassing head is a remix of |
| mixture-of-softmaxes and linear attention, not a test of the geometric thesis. |
|
|
| --- |
|
|
| ## What's in the repo |
|
|
| ``` |
| 1_lexical_atlas.py stratified n-gram atlas (bank construction) |
| 2_aleph_routed_attention.py the aleph: signed-projective addressing, |
| hub/bucket routed attention, streaming state |
| 3_aleph_trigram_lm.py exp_007-era byte-trigram LM (seed-battery arch) |
| 4_aleph_lm.py current AlephLM: byte/kernel/pmix/apmix heads, |
| bank loading, statute gauge, sweep hooks |
| checkpoints/ every LM checkpoint, flat (83 files) |
| exp_008_tier_a.pt the 23M calibration model |
| exp004_d4.pt … exp004_d1024.pt exp_004 multiscale-lens ladder (D=4…1024) |
| {byte,kernel_*,pmix*_*}_s*.pt exp_007 seed battery (18 runs, two seeds) |
| {free,div_*,frozen,slowlane}.pt exp_009 preservation arms |
| {ctx,kaux,head,bld,apx,pmx,bk,wr,gr}*.pt exp_010 component survey |
| aleph_byte_trigram_*.pt pre-exp_007 hard-routing prototypes |
| CHECKPOINTS.md load-and-purpose quick reference (in checkpoints/) |
| EXPERIMENTAL_CHECKPOINTS.md per-checkpoint statistics (bpb, dev, rank, params) |
| RESEARCH_HISTORY.md the full discovery ledger and theorem table |
| ``` |
|
|
| The numbered files are self-contained cells — paste them in order into a |
| notebook (or import them as modules) to reconstruct the full stack. Checkpoints |
| were migrated flat from the experiment tree of the source research repo; each |
| carries the model state dict and config, and the `AlephLM`-era files also carry |
| the candidate `bank`. The two manifest files document purpose and statistics |
| respectively. Codebook snapshots and cumulative result tables remain in the |
| source experiment repo, |
| [`geolip-aleph-void`](https://huggingface.co/AbstractPhil/geolip-aleph-void), |
| where training runs push automatically. |
|
|
| --- |
|
|
| ## Headline results |
|
|
| ### The 23M calibration run (exp_008) |
| |
| A 23,114,451-parameter model, 8000 steps on WikiText-103-raw, streamed to ~4096 |
| effective trigrams, `pmix` head. Reached **2.16 bits/byte** (train-stream), |
| bracketing English's trigram branching factor at the measured mixture entropy. |
| Honestly positioned: this is **2013–2015 LSTM-class** statistics — a better |
| compressor than bzip2, behind modern same-parameter models, and beatable by a |
| classical Kneser-Ney 5-gram on this corpus. It proved the *scaffolding* scales; |
| it also collapsed its (unemployed) codebook, which is what motivated the |
| identity audit and the restoration head. |
| |
| ### The identity audit and `apmix` (exp_009 + the audit) |
|
|
| A code trace plus the ablation above established that the `pmix` champion was |
| not performing aleph prediction. The `apmix` head was designed to re-close the |
| loop, and a five-arm preservation battery confirmed the open-loop neutrality |
| prediction sharply: free, diversity-regularized, frozen, and slow-lane |
| codebooks all landed within **0.001 bpb** of each other under `pmix` — the |
| codebook is pure ornament when the loop is open. |
|
|
| ### Earlier experiments present in the repo |
|
|
| Alongside the runs detailed below, `checkpoints/` includes two earlier lines: |
| the **exp_004 multiscale-lens ladder** (single-aleph models at D = 4 … 1024, |
| the empirical basis for choosing D=4 as the working address dimension) and the |
| **exp_007 seed battery** (18 two-seed runs establishing the statute |
| dose-response and the scorer×bank crossover that exposed the T7 rank bound; |
| champion arm `pmix8_dense` at 2.323 bpb). Statistics for these are in their |
| source experiment directories; the per-checkpoint table in |
| `EXPERIMENTAL_CHECKPOINTS.md` focuses on the exp_008–010 runs profiled in |
| detail. |
| |
| ### The component-potential survey (exp_010, 44 arms) |
|
|
| A small-backbone (6.75M) sweep across eight families, ~4 minutes per arm: |
|
|
| - **Hemisphere dial is free.** A kernel-auxiliary loss through the codebook |
| moves deviation −0.004 → +0.007 and effective rank 3.79 → 3.90 dose-monotone, |
| while in-bank NLL, coverage, and gate accuracy stay identical to three |
| decimals. Codebook employment at **zero LM-quality cost**. |
| - **Statute-by-construction earns its keep.** A frozen, repulsion-optimized |
| maximal-spread codebook (the "spread" construction) beat the free codebook |
| under *both* employed heads (apmix 2.785 < 2.796; kernel 2.877 < 2.889), |
| while a frozen-*random* control beat neither — so the gain is the **geometry**, |
| not the freezing. |
| - **Grafting works — three ways.** Warm-starting an expanded model from a |
| trained donor (2000 steps) versus cold training: |
|
|
| | graft | bpb @2k | reference | |
| |---|---|---| |
| | depth 4→6 (layer interpolation) | **2.484** | beats its own 5k donor (2.577) and 5k cold (2.718) | |
| | mixture J→8 (tile + jitter) | **2.468** | beats donor | |
| | head `pmix`→`apmix` (identity retrofit) | 2.950 | beats cold same-arch (3.093) | |
|
|
| The depth graft **beat the model it was grown from**, in 40% of the steps. |
| None collapsed on transplant. |
| - **Two clean negatives.** The write-head auxiliary is bits-per-byte-neutral at |
| this scale (drop candidate). The bank survives a full vocabulary swap with |
| continuous geometry (vocabulary is data). |
|
|
| The practical upshot: **employment + construction + grafting** is the cohesive |
| shape. A pretrained model can be retrofitted to the closed-loop `apmix` head, |
| deepened, or widened — each as a cheap warm-start rather than a retrain. |
|
|
| --- |
|
|
| ## How the aleph relates to known work |
|
|
| No major model uses the aleph as constructed. Organ-level cousins exist and the |
| program names them honestly: MoE routers (an unsigned, single-hemisphere |
| address that routes FFN compute, with router-collapse mirroring codebook |
| degeneracy and balance losses mirroring the diversity term); Performer/FAVOR+ |
| (positive feature maps for linear attention, but random and high-dimensional |
| rather than a learned low-D codebook); product-key memory; Perceiver inducing |
| points; VQ-VAE codebooks (hard quantization vs. the aleph's soft addressing). |
| What is unoccupied: the **signed antipodal address** serving attention *and* |
| prediction through one geometry, the statute/basin theory treating that |
| geometry as a first-class governed object, and the controller-contrast |
| methodology itself. A formal prior-art sweep is recommended before any formal |
| publication. |
|
|
| --- |
|
|
| ## Connections to the wider GeoLIP program |
|
|
| The aleph's home dimension D=4 sits **deliberately in the volatile regime**: the |
| CV-band physics measured across 65,536 configurations and 17+ models place the |
| phase boundary at the binding constant **0.29154** (with the trained-activation |
| CV converging to the uniform-sphere value for each ambient dimension). D=4 |
| extrapolates to CV ≈ 0.9, far above the stable band — which is exactly why the |
| aleph's defense stack (sphere-normalization, bounded scaling, the statute |
| gauge) is load-bearing rather than decorative. The aleph line is the |
| language-modeling probe of that same geometric substrate. |
|
|
| --- |
|
|
| ## Status and honesty notes |
|
|
| - These models **complete text; they do not converse or follow instructions.** |
| Nearest legitimate-LLM-fluency tier (TinyStories-class) is ~10× the |
| calibration run's compute — roughly one GPU-day — and remains future work. |
| - Bits-per-byte figures from training streams are *not* held-out validation and |
| run slightly optimistic; WikiText-103 is cleaner than enwik8. |
| - The geometric findings (controller contrast, two-hemisphere law, |
| construction advantage) are the contribution. The bits-per-byte ladder is |
| where the rest of the field lives; the geometry is where this program lives. |
| - Reliable as of the experiments dated in `RESEARCH_HISTORY.md` (2026-06). |
| Claims there carry their evidence basis and date. |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{geolip_aleph_lm_2026, |
| title = {geolip-aleph-lm: Aleph-Routed Attention as a Structural Controller}, |
| author = {Coelho, Philip}, |
| year = {2026}, |
| note = {GeoLIP geometric program}, |
| url = {https://huggingface.co/AbstractPhil/geolip-aleph-lm} |
| } |
| ``` |
|
|
| ## License |
|
|
| MIT. |
