exp015_ch package (content-bearing heredity: 5 brackets, content gauge, 20 genomes) + repro retrofit: every package standalone (own harness copies, portable data roots, real CLIs, repro.py loaders, genome-aware reader) - all README snippets verified by execution
Browse files- README.md +27 -0
- exp012_ar/README.md +7 -3
- exp012_ar/read_codebook.py +11 -4
- exp012_ar/repro.py +29 -0
- exp013_aug/README.md +16 -4
- exp013_aug/ar_differentiation_bed.py +487 -0
- exp013_aug/exp013_augmentation_bed.py +5 -3
- exp013_aug/geolip_vitals.py +184 -0
- exp013_aug/read_codebook.py +159 -0
- exp013_aug/repro.py +32 -0
- exp014_gd/README.md +25 -4
- exp014_gd/ar_differentiation_bed.py +487 -0
- exp014_gd/article.md +17 -6
- exp014_gd/exp013_augmentation_bed.py +506 -0
- exp014_gd/exp014_genetic_distillation.py +23 -3
- exp014_gd/geolip_vitals.py +184 -0
- exp014_gd/read_codebook.py +159 -0
- exp014_gd/repro.py +39 -0
- exp015_ch/README.md +100 -0
- exp015_ch/ar_differentiation_bed.py +487 -0
- exp015_ch/build_results.py +100 -0
- exp015_ch/exp014_genetic_distillation.py +515 -0
- exp015_ch/exp015_content_heredity.py +188 -0
- exp015_ch/genomes/champion_sign_full_t0_g0.pt +3 -0
- exp015_ch/genomes/champion_sign_full_t0_g1.pt +3 -0
- exp015_ch/genomes/champion_sign_full_t0_g2.pt +3 -0
- exp015_ch/genomes/champion_sign_full_t0_g3.pt +3 -0
- exp015_ch/genomes/champion_sign_none_t0_g0.pt +3 -0
- exp015_ch/genomes/champion_sign_none_t0_g1.pt +3 -0
- exp015_ch/genomes/champion_sign_none_t0_g2.pt +3 -0
- exp015_ch/genomes/champion_sign_none_t0_g3.pt +3 -0
- exp015_ch/genomes/champion_sign_weights_t0_g0.pt +3 -0
- exp015_ch/genomes/champion_sign_weights_t0_g1.pt +3 -0
- exp015_ch/genomes/champion_sign_weights_t0_g2.pt +3 -0
- exp015_ch/genomes/champion_sign_weights_t0_g3.pt +3 -0
- exp015_ch/genomes/champion_tree_full_t0_g0.pt +3 -0
- exp015_ch/genomes/champion_tree_full_t0_g1.pt +3 -0
- exp015_ch/genomes/champion_tree_full_t0_g2.pt +3 -0
- exp015_ch/genomes/champion_tree_full_t0_g3.pt +3 -0
- exp015_ch/genomes/champion_tree_weights_t0_g0.pt +3 -0
- exp015_ch/genomes/champion_tree_weights_t0_g1.pt +3 -0
- exp015_ch/genomes/champion_tree_weights_t0_g2.pt +3 -0
- exp015_ch/genomes/champion_tree_weights_t0_g3.pt +3 -0
- exp015_ch/geolip_vitals.py +184 -0
- exp015_ch/read_codebook.py +159 -0
- exp015_ch/repro.py +39 -0
- exp015_ch/results/ledger.jsonl +80 -0
- exp015_ch/results/results.json +107 -0
README.md
CHANGED
|
@@ -145,6 +145,33 @@ Unifying insight: near-uniform books are near-interchangeable scaffolds — gene
|
|
| 145 |
methods pay only where the book's CONTENT is load-bearing. Write-up:
|
| 146 |
[exp014_gd/article.md](./exp014_gd/article.md); ledger + 23 champion genomes included.
|
| 147 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
## Relation to prior work
|
| 149 |
|
| 150 |
Residual-expert quantization (RQ-MoE, SwitchCodec/REVQ), hierarchical
|
|
|
|
| 145 |
methods pay only where the book's CONTENT is load-bearing. Write-up:
|
| 146 |
[exp014_gd/article.md](./exp014_gd/article.md); ledger + 23 champion genomes included.
|
| 147 |
|
| 148 |
+
## exp015 — content-bearing heredity (July 10, 2026)
|
| 149 |
+
|
| 150 |
+
[`exp015_ch/`](./exp015_ch): exp014's compass executed — tournaments where the
|
| 151 |
+
prediction channel consumes book IDENTITY (the sign-code head: features are the
|
| 152 |
+
±A\[win\] rows themselves), plus the tree lineage under fair full-weight
|
| 153 |
+
inheritance. Verdicts: content consumption cultivates LSH fidelity by itself
|
| 154 |
+
(0.942 → 0.954 with no germline), compressing the germline's score headroom to
|
| 155 |
+
founder-luck scale; the lineage fixed point replicates on the discrete channel
|
| 156 |
+
and LOCKS fidelity (<0.001 inter-member spread); the tree inherits under
|
| 157 |
+
continuity (both lineages monotone; whole-structure fixed point by gen 2);
|
| 158 |
+
branch revival belongs to continuity — the germline's stationarity freezes
|
| 159 |
+
routing at zero score cost; and heredity maintains root-routing health that
|
| 160 |
+
every fresh founder loses. The content gauge separates heredity from lottery
|
| 161 |
+
far more cleanly than score does. 80-run ledger + 20 champion genomes;
|
| 162 |
+
write-up in [exp015_ch/README.md](./exp015_ch/README.md).
|
| 163 |
+
|
| 164 |
+
## Reproducibility
|
| 165 |
+
|
| 166 |
+
Every experiment package (`exp012_ar/`, `exp013_aug/`, `exp014_gd/`,
|
| 167 |
+
`exp015_ch/`) is **standalone**: it carries its own copies of the shared
|
| 168 |
+
harness (`geolip_vitals.py`, `ar_differentiation_bed.py`, `read_codebook.py`)
|
| 169 |
+
and a `repro.py` loader — `python repro.py` smokes on CPU, flags forward to
|
| 170 |
+
verdict runs on GPU. Data roots default to `./data` (override with
|
| 171 |
+
`GEOLIP_DATA`). Each README's snippet has been verified by execution; the
|
| 172 |
+
exp013 track-b1 snippet reproduces its published table exactly from a fresh
|
| 173 |
+
cache.
|
| 174 |
+
|
| 175 |
## Relation to prior work
|
| 176 |
|
| 177 |
Residual-expert quantization (RQ-MoE, SwitchCodec/REVQ), hierarchical
|
exp012_ar/README.md
CHANGED
|
@@ -20,18 +20,22 @@ transit point · projective-codebook law holds under pure predictive pressure.
|
|
| 20 |
pentachoron-CV via geovocab2).
|
| 21 |
- `read_codebook.py` — antipodal-collapse projective reader (Polygonal Omega recipe),
|
| 22 |
validated on planted pairs. `python read_codebook.py specimens/` reads everything.
|
|
|
|
|
|
|
| 23 |
- `build_results.py` → `results/results.json` — the complete 48-run ledger; running
|
| 24 |
it re-asserts every certified aggregate against the per-run rows.
|
| 25 |
- `results/specimen_reads.json` — projective reads of all 17 specimens.
|
| 26 |
- `specimens/*.pt` — trained models, `{arm}_s{seed}_t{steps}.pt` (~134MB total).
|
| 27 |
|
| 28 |
-
## Quick start
|
| 29 |
```bash
|
| 30 |
pip install torch --index-url https://download.pytorch.org/whl/cu128
|
| 31 |
pip install pyarrow huggingface_hub
|
| 32 |
-
python
|
|
|
|
| 33 |
python -c "from ar_differentiation_bed import train; train(arms=('sdpa','addr_msl64'), steps=8000)"
|
| 34 |
-
python read_codebook.py
|
|
|
|
| 35 |
```
|
| 36 |
Bed constants: wikitext-2-raw bytes (HF parquet), block 256, batch 32, pure Adam
|
| 37 |
lr 3e-4 wd 0, d=192, 4 layers. Single RTX 4090: 3–15 min per arm.
|
|
|
|
| 20 |
pentachoron-CV via geovocab2).
|
| 21 |
- `read_codebook.py` — antipodal-collapse projective reader (Polygonal Omega recipe),
|
| 22 |
validated on planted pairs. `python read_codebook.py specimens/` reads everything.
|
| 23 |
+
- `repro.py` — loads the code files from this folder and runs them (smoke or
|
| 24 |
+
`--train`).
|
| 25 |
- `build_results.py` → `results/results.json` — the complete 48-run ledger; running
|
| 26 |
it re-asserts every certified aggregate against the per-run rows.
|
| 27 |
- `results/specimen_reads.json` — projective reads of all 17 specimens.
|
| 28 |
- `specimens/*.pt` — trained models, `{arm}_s{seed}_t{steps}.pt` (~134MB total).
|
| 29 |
|
| 30 |
+
## Quick start
|
| 31 |
```bash
|
| 32 |
pip install torch --index-url https://download.pytorch.org/whl/cu128
|
| 33 |
pip install pyarrow huggingface_hub
|
| 34 |
+
python repro.py # CPU smoke: parse + shapes + causality
|
| 35 |
+
python repro.py --train # the original three arms (GPU)
|
| 36 |
python -c "from ar_differentiation_bed import train; train(arms=('sdpa','addr_msl64'), steps=8000)"
|
| 37 |
+
python read_codebook.py specimens/ # projective reads of shipped specimens
|
| 38 |
+
python read_codebook.py ./data/ar_ckpts # ...or of your own trained arms
|
| 39 |
```
|
| 40 |
Bed constants: wikitext-2-raw bytes (HF parquet), block 256, batch 32, pure Adam
|
| 41 |
lr 3e-4 wd 0, d=192, 4 layers. Single RTX 4090: 3–15 min per arm.
|
exp012_ar/read_codebook.py
CHANGED
|
@@ -85,13 +85,20 @@ def projective_stats(axes: torch.Tensor, n_baseline: int = 20000,
|
|
| 85 |
@torch.no_grad()
|
| 86 |
def read_specimen(path: str) -> dict:
|
| 87 |
ck = torch.load(path, map_location="cpu", weights_only=True)
|
| 88 |
-
sd = ck["state_dict"]
|
| 89 |
out = {"file": path.split("\\")[-1].split("/")[-1],
|
| 90 |
"arm": ck.get("arm"), "seed": ck.get("seed"),
|
| 91 |
"steps": ck.get("steps"), "val_bpb": round(ck.get("val_bpb", -1), 4)}
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
reads = {}
|
| 96 |
for name, cb in books.items():
|
| 97 |
col = antipodal_collapse(cb)
|
|
|
|
| 85 |
@torch.no_grad()
|
| 86 |
def read_specimen(path: str) -> dict:
|
| 87 |
ck = torch.load(path, map_location="cpu", weights_only=True)
|
|
|
|
| 88 |
out = {"file": path.split("\\")[-1].split("/")[-1],
|
| 89 |
"arm": ck.get("arm"), "seed": ck.get("seed"),
|
| 90 |
"steps": ck.get("steps"), "val_bpb": round(ck.get("val_bpb", -1), 4)}
|
| 91 |
+
if "state_dict" in ck: # full specimen checkpoint
|
| 92 |
+
sd = ck["state_dict"]
|
| 93 |
+
books = {k[:-len(".codebook")]: sd[k] for k in sd
|
| 94 |
+
if k.endswith("addr.codebook") or k.endswith("head_addr.codebook")}
|
| 95 |
+
homes = {k[:-len(".home")]: sd[k] for k in sd if k.endswith(".home")}
|
| 96 |
+
else: # bare genome dict (exp014+ champion files):
|
| 97 |
+
# books under flat/root/branch* keys; *_proj entries are projections
|
| 98 |
+
books = {k: v for k, v in ck.items()
|
| 99 |
+
if torch.is_tensor(v) and v.ndim == 2
|
| 100 |
+
and (k in ("flat", "root") or k.startswith("branch"))}
|
| 101 |
+
homes = {}
|
| 102 |
reads = {}
|
| 103 |
for name, cb in books.items():
|
| 104 |
col = antipodal_collapse(cb)
|
exp012_ar/repro.py
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""repro.py — standalone loader/runner for exp012_ar. Every code dependency
|
| 2 |
+
lives in THIS folder (geolip_vitals.py, ar_differentiation_bed.py,
|
| 3 |
+
read_codebook.py) — no cross-package paths.
|
| 4 |
+
|
| 5 |
+
python repro.py # CPU smoke: parse + shapes + causality
|
| 6 |
+
python repro.py --train # the original three-arm verdict run (GPU)
|
| 7 |
+
python repro.py --train --steps 8000
|
| 8 |
+
|
| 9 |
+
Data + checkpoints land in ./data (checkpoints under ./data/ar_ckpts).
|
| 10 |
+
Reading the shipped specimens:
|
| 11 |
+
python read_codebook.py specimens/
|
| 12 |
+
"""
|
| 13 |
+
import os
|
| 14 |
+
import runpy
|
| 15 |
+
import sys
|
| 16 |
+
|
| 17 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 18 |
+
sys.path.insert(0, HERE)
|
| 19 |
+
|
| 20 |
+
if __name__ == "__main__":
|
| 21 |
+
if len(sys.argv) == 1:
|
| 22 |
+
import geolip_vitals # noqa: F401 (paste order)
|
| 23 |
+
import ar_differentiation_bed as bed
|
| 24 |
+
bed.smoke()
|
| 25 |
+
print("repro smoke passed — run with --train for verdict runs")
|
| 26 |
+
else:
|
| 27 |
+
target = os.path.join(HERE, "ar_differentiation_bed.py")
|
| 28 |
+
sys.argv = [target] + sys.argv[1:]
|
| 29 |
+
runpy.run_path(target, run_name="__main__")
|
exp013_aug/README.md
CHANGED
|
@@ -79,16 +79,28 @@ pretrained trunk* and on pooled semantic embeddings.
|
|
| 79 |
|
| 80 |
## Files
|
| 81 |
- `exp013_augmentation_bed.py` — the bed (all tracks/arms, caching, vitals,
|
| 82 |
-
specimen saving).
|
| 83 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
- `build_results.py` → `results/results.json` — full ledger, self-asserting.
|
| 85 |
- `results/specimen_reads.json` — projective reads of all specimens.
|
| 86 |
|
|
|
|
| 87 |
```bash
|
| 88 |
pip install torch --index-url https://download.pytorch.org/whl/cu128
|
| 89 |
pip install transformers tokenizers pyarrow huggingface_hub
|
| 90 |
-
python
|
| 91 |
-
python -
|
|
|
|
|
|
|
|
|
|
| 92 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
|
| 94 |
License: MIT · AbstractPhil + Claude Fable 5 · July 10, 2026
|
|
|
|
| 79 |
|
| 80 |
## Files
|
| 81 |
- `exp013_augmentation_bed.py` — the bed (all tracks/arms, caching, vitals,
|
| 82 |
+
specimen saving).
|
| 83 |
+
- `geolip_vitals.py` / `ar_differentiation_bed.py` / `read_codebook.py` — this
|
| 84 |
+
package's own copies of the shared harness (readout diagnostics; the
|
| 85 |
+
AlephAddress + MslRelay attention mechanisms; the projective specimen
|
| 86 |
+
reader). Identical to the `exp012_ar/` copies — the package is standalone.
|
| 87 |
+
- `repro.py` — loads the code files from this folder and runs them.
|
| 88 |
- `build_results.py` → `results/results.json` — full ledger, self-asserting.
|
| 89 |
- `results/specimen_reads.json` — projective reads of all specimens.
|
| 90 |
|
| 91 |
+
## Reproduce (from inside this folder)
|
| 92 |
```bash
|
| 93 |
pip install torch --index-url https://download.pytorch.org/whl/cu128
|
| 94 |
pip install transformers tokenizers pyarrow huggingface_hub
|
| 95 |
+
python repro.py # CPU smoke: parse + shapes, no downloads
|
| 96 |
+
python repro.py --track b1 # the gate + baselines (GPU)
|
| 97 |
+
python repro.py --track a # CLIP-L dual-layer token-AR (GPU)
|
| 98 |
+
python repro.py --track c # GPT-2 relay augmentation (GPU)
|
| 99 |
+
python read_codebook.py specimens/ # projective reads of shipped specimens
|
| 100 |
```
|
| 101 |
+
Data + model caches land in `./data` (override with the `GEOLIP_DATA` env var);
|
| 102 |
+
the first verdict run downloads CLIP-L / BERT / GPT-2 from the Hub. In Colab,
|
| 103 |
+
paste `geolip_vitals.py`, `ar_differentiation_bed.py`, then the bed into cells
|
| 104 |
+
in that order — same files, no path setup.
|
| 105 |
|
| 106 |
License: MIT · AbstractPhil + Claude Fable 5 · July 10, 2026
|
exp013_aug/ar_differentiation_bed.py
ADDED
|
@@ -0,0 +1,487 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""ar_differentiation_bed.py — exp012: autoregressive differentiation of the aleph.
|
| 2 |
+
|
| 3 |
+
Differentiation is cultivated by PREDICTIVE pressure along the sequence — the
|
| 4 |
+
address parameterizing the next-byte distribution (Law 2: chain-rule advantage pays
|
| 5 |
+
ONLY where the composed address directly parameterizes the predictive distribution).
|
| 6 |
+
This bed puts the aleph in the autoregressive gradient path and measures what
|
| 7 |
+
differentiates. The head arms enforce the employment law at its maximum: the
|
| 8 |
+
ENTIRE next-byte distribution is parameterized by the address.
|
| 9 |
+
|
| 10 |
+
Byte-level causal LM on wikitext-2-raw (HF parquet, CDN-fast), block 256. ARMS:
|
| 11 |
+
sdpa — standard causal transformer control (matched trunk).
|
| 12 |
+
hub — attention replaced by CAUSAL HUB: linear attention whose feature map
|
| 13 |
+
is the 2K-oriented aleph address, prefix-sum memories (no selection
|
| 14 |
+
event; O(n*K*d)). Differentiation cultivated INSIDE attention.
|
| 15 |
+
addr_head — sdpa trunk, but the OUTPUT HEAD reads ONLY the signed aleph
|
| 16 |
+
coefficient vector w_k = sinh(u_k)/sum_j cosh(u_j) of the final
|
| 17 |
+
hidden state (K -> 256 logits). The address MUST carry every bit of
|
| 18 |
+
next-byte information — the hardest Law-2 bottleneck.
|
| 19 |
+
|
| 20 |
+
JUDGED BY: val bits-per-byte per arm (task) + CULTIVATION VITALS on every aleph
|
| 21 |
+
codebook (readouts, never losses): axis aliveness/hppl, drift-from-init +
|
| 22 |
+
binding fraction @0.29154, winner-|cos| saturation (sign-code emergence), shadow
|
| 23 |
+
path diversity (fixed high-bits hash). Never by recon.
|
| 24 |
+
|
| 25 |
+
Riders: pure Adam wd=0; no BN/Dropout/GAP on geometric paths; orthogonal init;
|
| 26 |
+
Colab-cell-safe (paste-ahead imports, no bare argparse, no __file__ reliance);
|
| 27 |
+
GPU-only for verdict runs.
|
| 28 |
+
|
| 29 |
+
Terminal: python ar_differentiation_bed.py # shapes/parse smoke
|
| 30 |
+
python ar_differentiation_bed.py --train # verdict run
|
| 31 |
+
Colab: paste geolip_vitals.py cell, then this file (smoke auto-runs),
|
| 32 |
+
then train(steps=2000, data_root="/content/data") in the next cell.
|
| 33 |
+
"""
|
| 34 |
+
from __future__ import annotations
|
| 35 |
+
import math
|
| 36 |
+
import torch
|
| 37 |
+
import torch.nn as nn
|
| 38 |
+
import torch.nn.functional as F
|
| 39 |
+
|
| 40 |
+
if "anchor_drift" not in globals():
|
| 41 |
+
try:
|
| 42 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 43 |
+
except ImportError:
|
| 44 |
+
_here = globals().get("__file__")
|
| 45 |
+
if _here is not None:
|
| 46 |
+
import sys, pathlib
|
| 47 |
+
sys.path.insert(0, str(pathlib.Path(_here).parent))
|
| 48 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 49 |
+
else:
|
| 50 |
+
raise ImportError(
|
| 51 |
+
"geolip_vitals not found — paste/run its cell first, or "
|
| 52 |
+
"hf_hub_download exp012_ar/geolip_vitals.py from "
|
| 53 |
+
"AbstractPhil/geolip-aleph-differentiation.")
|
| 54 |
+
|
| 55 |
+
VOCAB = 256 # bytes
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
# ------------------------------------------------------------------ aleph address
|
| 59 |
+
def _super_fibonacci_s3(n: int) -> torch.Tensor:
|
| 60 |
+
"""Near-uniform unit quaternions (Alexa CVPR'22) —
|
| 61 |
+
starts the codebook INSIDE the RP^3 attractor basin. D=4 only."""
|
| 62 |
+
PHI, PSI = math.sqrt(2.0), 1.533751168755204288118041
|
| 63 |
+
i = torch.arange(n, dtype=torch.float64)
|
| 64 |
+
s = (i + 0.5) / n
|
| 65 |
+
r, R = torch.sqrt(s), torch.sqrt(1.0 - s)
|
| 66 |
+
a, b = 2 * math.pi * i / PHI, 2 * math.pi * i / PSI
|
| 67 |
+
q = torch.stack([r * torch.sin(a), r * torch.cos(a),
|
| 68 |
+
R * torch.sin(b), R * torch.cos(b)], dim=-1)
|
| 69 |
+
return F.normalize(q, dim=-1).float()
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
class AlephAddress(nn.Module):
|
| 73 |
+
"""Closed-form aleph over 2K oriented half-axes (aleph-void article).
|
| 74 |
+
signed(x): (..., K) w_k = sinh(u_k)/sum_j cosh(u_j) — the Law-2 head feature.
|
| 75 |
+
oriented(x): ((..., K), (..., K)) positive halves of the 2K softmax — HUB map."""
|
| 76 |
+
|
| 77 |
+
def __init__(self, K: int, D: int, tau: float = 0.1, init: str = "random"):
|
| 78 |
+
super().__init__()
|
| 79 |
+
self.K, self.D, self.tau = K, D, tau
|
| 80 |
+
if init == "fibonacci":
|
| 81 |
+
assert D == 4, "fibonacci init lives on S^3 (D=4)"
|
| 82 |
+
A = _super_fibonacci_s3(K)
|
| 83 |
+
else:
|
| 84 |
+
A = F.normalize(torch.randn(K, D), dim=-1)
|
| 85 |
+
self.codebook = nn.Parameter(A)
|
| 86 |
+
self.register_buffer("home", self.codebook.detach().clone())
|
| 87 |
+
|
| 88 |
+
def _u(self, x):
|
| 89 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 90 |
+
return (F.normalize(x, dim=-1) @ A.transpose(-1, -2)) / self.tau
|
| 91 |
+
|
| 92 |
+
def oriented(self, x):
|
| 93 |
+
u = self._u(x)
|
| 94 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 95 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 96 |
+
Z = (ep + en).sum(dim=-1, keepdim=True)
|
| 97 |
+
return ep / Z, en / Z
|
| 98 |
+
|
| 99 |
+
def signed(self, x):
|
| 100 |
+
u = self._u(x)
|
| 101 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 102 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 103 |
+
return (ep - en) / (ep + en).sum(dim=-1, keepdim=True)
|
| 104 |
+
|
| 105 |
+
def signed_at(self, x, taus):
|
| 106 |
+
"""Multi-tau stroboscope (rule of 3): signed coefficients at several
|
| 107 |
+
temperatures, concatenated — softer taus keep the vector dense while a
|
| 108 |
+
hard tau supplies the sign-code sharpness. v2 refinement (b)."""
|
| 109 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 110 |
+
cos = F.normalize(x, dim=-1) @ A.transpose(-1, -2)
|
| 111 |
+
outs = []
|
| 112 |
+
for t in taus:
|
| 113 |
+
u = cos / t
|
| 114 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 115 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 116 |
+
outs.append((ep - en) / (ep + en).sum(dim=-1, keepdim=True))
|
| 117 |
+
return torch.cat(outs, dim=-1)
|
| 118 |
+
|
| 119 |
+
def m_hat(self, x):
|
| 120 |
+
"""Closed-form soft read (decoders read M_hat, never M). v2 control (c)."""
|
| 121 |
+
u = self._u(x)
|
| 122 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 123 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 124 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 125 |
+
return ((ep - en) @ A) / (ep + en).sum(dim=-1, keepdim=True)
|
| 126 |
+
|
| 127 |
+
def m_hard_ste(self, x):
|
| 128 |
+
"""Hard mode (aleph-void article): M_hard = sign(cos_win) * A[win], straight-through to
|
| 129 |
+
the soft read — forward fully discrete SIGN CODE, backward soft gradient.
|
| 130 |
+
Legal per theme A (reconstructive sign code, not a one-hot roster pick)."""
|
| 131 |
+
u = self._u(x)
|
| 132 |
+
soft = self.m_hat(x)
|
| 133 |
+
win = u.abs().argmax(dim=-1)
|
| 134 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 135 |
+
sign = torch.sign(torch.gather(u, -1, win.unsqueeze(-1))).squeeze(-1)
|
| 136 |
+
hard = sign.unsqueeze(-1) * A[win]
|
| 137 |
+
return hard + soft - soft.detach()
|
| 138 |
+
|
| 139 |
+
@torch.no_grad()
|
| 140 |
+
def vitals(self, x_sample) -> dict:
|
| 141 |
+
u = self._u(x_sample.reshape(-1, x_sample.shape[-1]))
|
| 142 |
+
p, n = self.oriented(x_sample.reshape(-1, x_sample.shape[-1]))
|
| 143 |
+
two_k = torch.cat([p, n], dim=-1)
|
| 144 |
+
win = two_k.argmax(dim=-1)
|
| 145 |
+
cos_win = (u.abs().amax(dim=-1) * self.tau) # winner |cos| — sign-code sat.
|
| 146 |
+
d = anchor_drift(self.codebook, self.home)
|
| 147 |
+
return {"drift": round(d["mean"], 4),
|
| 148 |
+
"binding_frac": round(d["binding_fraction"], 4),
|
| 149 |
+
"aliveness": axis_aliveness(two_k),
|
| 150 |
+
"win_cos_mean": round(cos_win.mean().item(), 4),
|
| 151 |
+
"paths": path_diversity(win)}
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
# ------------------------------------------------------------------------- blocks
|
| 155 |
+
class CausalSDPA(nn.Module):
|
| 156 |
+
def __init__(self, d: int, heads: int = 4):
|
| 157 |
+
super().__init__()
|
| 158 |
+
self.h = heads
|
| 159 |
+
self.qkv = nn.Linear(d, 3 * d, bias=False)
|
| 160 |
+
self.o = nn.Linear(d, d, bias=False)
|
| 161 |
+
nn.init.orthogonal_(self.qkv.weight); nn.init.orthogonal_(self.o.weight)
|
| 162 |
+
|
| 163 |
+
def forward(self, x):
|
| 164 |
+
B, n, d = x.shape
|
| 165 |
+
q, k, v = self.qkv(x).chunk(3, dim=-1)
|
| 166 |
+
q, k, v = (t.view(B, n, self.h, d // self.h).transpose(1, 2) for t in (q, k, v))
|
| 167 |
+
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
| 168 |
+
return self.o(y.transpose(1, 2).reshape(B, n, d))
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
class CausalHUB(nn.Module):
|
| 172 |
+
"""Causal aleph linear attention: prefix-sum memories over the two K-wide
|
| 173 |
+
halves of the oriented address; 2K never materialized; no selection event."""
|
| 174 |
+
|
| 175 |
+
def __init__(self, d: int, K: int = 32, D: int = 4, tau: float = 0.1):
|
| 176 |
+
super().__init__()
|
| 177 |
+
self.addr = AlephAddress(K, D, tau)
|
| 178 |
+
self.q = nn.Linear(d, D, bias=False)
|
| 179 |
+
self.k = nn.Linear(d, D, bias=False)
|
| 180 |
+
self.v = nn.Linear(d, d, bias=False)
|
| 181 |
+
self.o = nn.Linear(d, d, bias=False)
|
| 182 |
+
for m in (self.q, self.k, self.v, self.o):
|
| 183 |
+
nn.init.orthogonal_(m.weight)
|
| 184 |
+
|
| 185 |
+
def forward(self, x):
|
| 186 |
+
qp, qn = self.addr.oriented(self.q(x)) # (B, n, K)
|
| 187 |
+
kp, kn = self.addr.oriented(self.k(x))
|
| 188 |
+
v = self.v(x) # (B, n, d)
|
| 189 |
+
Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
|
| 190 |
+
Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
|
| 191 |
+
zp = torch.cumsum(kp, dim=1)
|
| 192 |
+
zn = torch.cumsum(kn, dim=1)
|
| 193 |
+
num = torch.einsum("bnk,bnkd->bnd", qp, Sp) + torch.einsum("bnk,bnkd->bnd", qn, Sn)
|
| 194 |
+
den = (qp * zp).sum(-1, keepdim=True) + (qn * zn).sum(-1, keepdim=True)
|
| 195 |
+
return self.o(num / den.clamp_min(1e-12))
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
class MslRelay(nn.Module):
|
| 199 |
+
"""Depth-composition unit (chain-rule probe): multi-slot M_hat read entering
|
| 200 |
+
the trunk as a NEAR-ZERO gated residual (gate init -3.0, sigma~0.047 — theme D:
|
| 201 |
+
geometry enters as a nudge and grows only if it earns gradient)."""
|
| 202 |
+
|
| 203 |
+
def __init__(self, d: int, n_slots: int = 16, K: int = 64):
|
| 204 |
+
super().__init__()
|
| 205 |
+
self.n_slots = n_slots
|
| 206 |
+
self.proj = nn.Linear(d, n_slots * 4, bias=False)
|
| 207 |
+
self.out = nn.Linear(n_slots * 4, d, bias=False)
|
| 208 |
+
nn.init.orthogonal_(self.proj.weight)
|
| 209 |
+
nn.init.orthogonal_(self.out.weight)
|
| 210 |
+
self.addr = AlephAddress(K, 4)
|
| 211 |
+
self.gate = nn.Parameter(torch.tensor(-3.0))
|
| 212 |
+
|
| 213 |
+
def forward(self, x):
|
| 214 |
+
B, n, _ = x.shape
|
| 215 |
+
slots = self.proj(x).view(B, n, self.n_slots, 4)
|
| 216 |
+
m = self.addr.m_hat(slots).reshape(B, n, -1)
|
| 217 |
+
return x + self.gate.sigmoid() * self.out(m)
|
| 218 |
+
|
| 219 |
+
|
| 220 |
+
class Block(nn.Module):
|
| 221 |
+
def __init__(self, d: int, attn: nn.Module):
|
| 222 |
+
super().__init__()
|
| 223 |
+
self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
|
| 224 |
+
self.attn = attn
|
| 225 |
+
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
|
| 226 |
+
|
| 227 |
+
def forward(self, x):
|
| 228 |
+
x = x + self.attn(self.n1(x))
|
| 229 |
+
return x + self.mlp(self.n2(x))
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
class ByteLM(nn.Module):
|
| 233 |
+
def __init__(self, arm: str, d: int = 192, layers: int = 4, block: int = 256,
|
| 234 |
+
K: int = 32, D: int = 4):
|
| 235 |
+
super().__init__()
|
| 236 |
+
# "<arm>_tri" suffix = trigram byte embedding (AlephLM byte_emb x3 lineage):
|
| 237 |
+
# token embedding is the sum of embeddings of bytes t, t-1, t-2.
|
| 238 |
+
self.trigram = arm.endswith("_tri")
|
| 239 |
+
if self.trigram:
|
| 240 |
+
arm = arm[:-4]
|
| 241 |
+
# "_fib" = super-Fibonacci S^3 codebook init (basin test: starts INSIDE
|
| 242 |
+
# the RP^3 attractor; primary observable is init->final geodesic drift).
|
| 243 |
+
self.fib = arm.endswith("_fib")
|
| 244 |
+
if self.fib:
|
| 245 |
+
arm = arm[:-4]
|
| 246 |
+
# "relay*" = stacked addresses in depth: MslRelay after every block.
|
| 247 |
+
# relay -> sdpa trunk + standard head; relay_msl64 -> + addressed head.
|
| 248 |
+
self.use_relay = arm.startswith("relay")
|
| 249 |
+
if arm == "relay":
|
| 250 |
+
arm = "sdpa"
|
| 251 |
+
elif arm == "relay_msl64":
|
| 252 |
+
arm = "addr_msl64"
|
| 253 |
+
self.arm, self.block = arm, block
|
| 254 |
+
self.emb = nn.Embedding(VOCAB, d)
|
| 255 |
+
if self.trigram:
|
| 256 |
+
self.emb1 = nn.Embedding(VOCAB, d)
|
| 257 |
+
self.emb2 = nn.Embedding(VOCAB, d)
|
| 258 |
+
self.pos = nn.Parameter(torch.zeros(1, block, d) + 0.01 * torch.randn(1, block, d))
|
| 259 |
+
mk_attn = (lambda: CausalHUB(d, K, D)) if arm == "hub" else (lambda: CausalSDPA(d))
|
| 260 |
+
self.blocks = nn.ModuleList([Block(d, mk_attn()) for _ in range(layers)])
|
| 261 |
+
if self.use_relay:
|
| 262 |
+
self.relays = nn.ModuleList([MslRelay(d) for _ in range(layers)])
|
| 263 |
+
self.nf = nn.LayerNorm(d)
|
| 264 |
+
if arm == "addr_head":
|
| 265 |
+
self.head_addr = AlephAddress(K, d) # v1: codebook in model dim — COLLAPSED
|
| 266 |
+
self.head = nn.Linear(K, VOCAB, bias=True)
|
| 267 |
+
elif arm in ("addr_d4", "addr_3tau", "addr_mhat"):
|
| 268 |
+
# v2 refinements: LOW-D HOME — learned projection to the native D=4 home
|
| 269 |
+
# before addressing (mirrors the healthy HUB arms), K=64.
|
| 270 |
+
self.head_proj = nn.Linear(d, 4, bias=False)
|
| 271 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 272 |
+
self.head_addr = AlephAddress(64, 4)
|
| 273 |
+
if arm == "addr_d4":
|
| 274 |
+
self.head = nn.Linear(64, VOCAB, bias=True) # w alone, D=4 home
|
| 275 |
+
elif arm == "addr_3tau":
|
| 276 |
+
self.taus = (0.05, 0.1, 0.3) # rule-of-3 strobe
|
| 277 |
+
self.head = nn.Linear(64 * 3, VOCAB, bias=True)
|
| 278 |
+
else: # addr_mhat
|
| 279 |
+
self.head = nn.Linear(4, VOCAB, bias=True) # tightest: M_hat
|
| 280 |
+
elif arm.startswith("addr_msl"):
|
| 281 |
+
# v3: MULTI-SLOT heads — the 16s funnel widening: P parallel D=4 slots
|
| 282 |
+
# over a SHARED codebook. addr_msl consumes the reconstructive M_hat per
|
| 283 |
+
# slot (Px4 dims); addr_msl_w consumes signed w per slot (Px64) — tests
|
| 284 |
+
# whether slot-parallel consumption alone rescues the coefficient path.
|
| 285 |
+
# addr_msl<P> = slot-count dose-response. addr_mslh<P> = HARD sign-code
|
| 286 |
+
# consumption (straight-through M_hard per slot).
|
| 287 |
+
self.hard = arm.startswith("addr_mslh")
|
| 288 |
+
if arm in ("addr_msl", "addr_msl_w"):
|
| 289 |
+
self.n_slots = 16
|
| 290 |
+
else:
|
| 291 |
+
self.n_slots = int(arm[len("addr_mslh" if self.hard else "addr_msl"):])
|
| 292 |
+
self.head_proj = nn.Linear(d, self.n_slots * 4, bias=False)
|
| 293 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 294 |
+
self.head_addr = AlephAddress(
|
| 295 |
+
64, 4, init="fibonacci" if self.fib else "random")
|
| 296 |
+
width = self.n_slots * (64 if arm == "addr_msl_w" else 4)
|
| 297 |
+
self.head = nn.Linear(width, VOCAB, bias=True)
|
| 298 |
+
elif arm == "addr_3tau_mhat":
|
| 299 |
+
# v3: combine the two v2 winners — 3-tau stroboscope + reconstructive read.
|
| 300 |
+
self.head_proj = nn.Linear(d, 4, bias=False)
|
| 301 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 302 |
+
self.head_addr = AlephAddress(64, 4)
|
| 303 |
+
self.taus = (0.05, 0.1, 0.3)
|
| 304 |
+
self.head = nn.Linear(64 * 3 + 4, VOCAB, bias=True)
|
| 305 |
+
else:
|
| 306 |
+
self.head = nn.Linear(d, VOCAB, bias=True)
|
| 307 |
+
self._last_h = None
|
| 308 |
+
|
| 309 |
+
def forward(self, idx):
|
| 310 |
+
x = self.emb(idx)
|
| 311 |
+
if self.trigram: # past-only shifts — causality preserved
|
| 312 |
+
x = x + self.emb1(F.pad(idx, (1, 0), value=0)[:, :-1]) \
|
| 313 |
+
+ self.emb2(F.pad(idx, (2, 0), value=0)[:, :-2])
|
| 314 |
+
x = x + self.pos[:, : idx.shape[1]]
|
| 315 |
+
if self.use_relay:
|
| 316 |
+
for b, r in zip(self.blocks, self.relays):
|
| 317 |
+
x = r(b(x))
|
| 318 |
+
else:
|
| 319 |
+
for b in self.blocks:
|
| 320 |
+
x = b(x)
|
| 321 |
+
h = self.nf(x)
|
| 322 |
+
self._last_h = h.detach()
|
| 323 |
+
if self.arm == "addr_head":
|
| 324 |
+
return self.head(self.head_addr.signed(h))
|
| 325 |
+
if self.arm == "addr_d4":
|
| 326 |
+
return self.head(self.head_addr.signed(self.head_proj(h)))
|
| 327 |
+
if self.arm == "addr_3tau":
|
| 328 |
+
return self.head(self.head_addr.signed_at(self.head_proj(h), self.taus))
|
| 329 |
+
if self.arm == "addr_mhat":
|
| 330 |
+
return self.head(self.head_addr.m_hat(self.head_proj(h)))
|
| 331 |
+
if self.arm.startswith("addr_msl"):
|
| 332 |
+
B, n, _ = h.shape
|
| 333 |
+
slots = self.head_proj(h).view(B, n, self.n_slots, 4)
|
| 334 |
+
if self.arm == "addr_msl_w":
|
| 335 |
+
feats = self.head_addr.signed(slots).reshape(B, n, -1)
|
| 336 |
+
elif getattr(self, "hard", False):
|
| 337 |
+
feats = self.head_addr.m_hard_ste(slots).reshape(B, n, -1)
|
| 338 |
+
else:
|
| 339 |
+
feats = self.head_addr.m_hat(slots).reshape(B, n, -1)
|
| 340 |
+
return self.head(feats)
|
| 341 |
+
if self.arm == "addr_3tau_mhat":
|
| 342 |
+
p = self.head_proj(h)
|
| 343 |
+
feats = torch.cat([self.head_addr.signed_at(p, self.taus),
|
| 344 |
+
self.head_addr.m_hat(p)], dim=-1)
|
| 345 |
+
return self.head(feats)
|
| 346 |
+
return self.head(h)
|
| 347 |
+
|
| 348 |
+
@torch.no_grad()
|
| 349 |
+
def vitals(self) -> dict:
|
| 350 |
+
out = {}
|
| 351 |
+
if self.arm == "hub":
|
| 352 |
+
for i, b in enumerate(self.blocks):
|
| 353 |
+
if self._last_h is not None:
|
| 354 |
+
out[f"L{i}"] = b.attn.addr.vitals(b.attn.q(self._last_h[:2]))
|
| 355 |
+
elif self.arm == "addr_head" and self._last_h is not None:
|
| 356 |
+
out["head"] = self.head_addr.vitals(self._last_h[:2])
|
| 357 |
+
elif self.arm in ("addr_d4", "addr_3tau", "addr_mhat",
|
| 358 |
+
"addr_3tau_mhat") and self._last_h is not None:
|
| 359 |
+
out["head"] = self.head_addr.vitals(self.head_proj(self._last_h[:2]))
|
| 360 |
+
elif self.arm.startswith("addr_msl") and self._last_h is not None:
|
| 361 |
+
slots = self.head_proj(self._last_h[:2])
|
| 362 |
+
out["head"] = self.head_addr.vitals(
|
| 363 |
+
slots.reshape(*slots.shape[:-1], self.n_slots, 4))
|
| 364 |
+
if self.use_relay and self._last_h is not None:
|
| 365 |
+
for i, r in enumerate(self.relays):
|
| 366 |
+
s = r.proj(self._last_h[:2])
|
| 367 |
+
v = r.addr.vitals(s.reshape(*s.shape[:-1], r.n_slots, 4))
|
| 368 |
+
out[f"relay{i}"] = {"gate": round(r.gate.sigmoid().item(), 4),
|
| 369 |
+
"drift": v["drift"],
|
| 370 |
+
"binding_frac": v["binding_frac"],
|
| 371 |
+
"ppl": round(v["aliveness"]["usage_ppl"], 1)}
|
| 372 |
+
return out
|
| 373 |
+
|
| 374 |
+
|
| 375 |
+
# --------------------------------------------------------------------------- data
|
| 376 |
+
def _wikitext_bytes(data_root: str):
|
| 377 |
+
"""wikitext-2-raw as flat uint8 tensors via the HF parquet CDN."""
|
| 378 |
+
from huggingface_hub import hf_hub_download
|
| 379 |
+
import pyarrow.parquet as pq
|
| 380 |
+
|
| 381 |
+
def load(split):
|
| 382 |
+
p = hf_hub_download("Salesforce/wikitext",
|
| 383 |
+
f"wikitext-2-raw-v1/{split}-00000-of-00001.parquet",
|
| 384 |
+
repo_type="dataset", local_dir=data_root)
|
| 385 |
+
text = "".join(pq.read_table(p).column("text").to_pylist())
|
| 386 |
+
return torch.frombuffer(bytearray(text.encode("utf-8")), dtype=torch.uint8).clone()
|
| 387 |
+
|
| 388 |
+
return load("train"), load("validation")
|
| 389 |
+
|
| 390 |
+
|
| 391 |
+
def _batch(data: torch.Tensor, batch: int, block: int, device, g: torch.Generator):
|
| 392 |
+
ix = torch.randint(0, data.numel() - block - 1, (batch,), generator=g)
|
| 393 |
+
x = torch.stack([data[i:i + block] for i in ix]).long().to(device)
|
| 394 |
+
y = torch.stack([data[i + 1:i + block + 1] for i in ix]).long().to(device)
|
| 395 |
+
return x, y
|
| 396 |
+
|
| 397 |
+
|
| 398 |
+
# -------------------------------------------------------------------- train/smoke
|
| 399 |
+
def train(arms=("sdpa", "hub", "addr_head"), steps: int = 2000, batch: int = 32,
|
| 400 |
+
block: int = 256, device: str = "cuda", data_root: str = "./data",
|
| 401 |
+
seed: int = 0, eval_every: int = 500, save: bool = True):
|
| 402 |
+
"""Verdict run — GPU only. Pure Adam wd=0. Reports val bits-per-byte + vitals.
|
| 403 |
+
save=True writes {data_root}/ar_ckpts/{arm}_s{seed}_t{steps}.pt per arm —
|
| 404 |
+
the cultivated codebooks are SPECIMENS for the projective reading instruments."""
|
| 405 |
+
import os
|
| 406 |
+
if device == "cuda" and not torch.cuda.is_available():
|
| 407 |
+
raise RuntimeError("Verdict runs are GPU-only (never CPU-train for accuracy).")
|
| 408 |
+
ckpt_dir = os.path.join(data_root, "ar_ckpts")
|
| 409 |
+
os.makedirs(ckpt_dir, exist_ok=True)
|
| 410 |
+
tr, va = _wikitext_bytes(data_root)
|
| 411 |
+
print(f"data ready: train {tr.numel():,} bytes, val {va.numel():,} bytes", flush=True)
|
| 412 |
+
results = {}
|
| 413 |
+
for arm in arms:
|
| 414 |
+
torch.manual_seed(seed)
|
| 415 |
+
g = torch.Generator().manual_seed(seed)
|
| 416 |
+
model = ByteLM(arm, block=block).to(device)
|
| 417 |
+
n_params = sum(p.numel() for p in model.parameters())
|
| 418 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 419 |
+
for step in range(1, steps + 1):
|
| 420 |
+
x, y = _batch(tr, batch, block, device, g)
|
| 421 |
+
logits = model(x)
|
| 422 |
+
loss = F.cross_entropy(logits.reshape(-1, VOCAB), y.reshape(-1))
|
| 423 |
+
opt.zero_grad(set_to_none=True)
|
| 424 |
+
loss.backward()
|
| 425 |
+
opt.step()
|
| 426 |
+
if step % eval_every == 0 or step == steps:
|
| 427 |
+
model.eval()
|
| 428 |
+
with torch.no_grad():
|
| 429 |
+
losses = []
|
| 430 |
+
for _ in range(20):
|
| 431 |
+
xv, yv = _batch(va, batch, block, device, g)
|
| 432 |
+
lv = F.cross_entropy(model(xv).reshape(-1, VOCAB),
|
| 433 |
+
yv.reshape(-1))
|
| 434 |
+
losses.append(lv.item())
|
| 435 |
+
bpb = sum(losses) / len(losses) / math.log(2)
|
| 436 |
+
print(f"[{arm}] step {step} val_bpb={bpb:.4f} vitals={model.vitals()}",
|
| 437 |
+
flush=True)
|
| 438 |
+
model.train()
|
| 439 |
+
results[arm] = {"val_bpb": bpb, "params": n_params, "vitals": model.vitals()}
|
| 440 |
+
if save:
|
| 441 |
+
path = os.path.join(ckpt_dir, f"{arm}_s{seed}_t{steps}.pt")
|
| 442 |
+
torch.save({"arm": arm, "seed": seed, "steps": steps, "val_bpb": bpb,
|
| 443 |
+
"state_dict": {k: v.cpu() for k, v in
|
| 444 |
+
model.state_dict().items()}}, path)
|
| 445 |
+
print(f"saved specimen: {path}", flush=True)
|
| 446 |
+
print(results, flush=True)
|
| 447 |
+
return results
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
def smoke():
|
| 451 |
+
"""Shapes/parse only — no accuracy claims."""
|
| 452 |
+
x = torch.randint(0, VOCAB, (2, 64))
|
| 453 |
+
for arm in ("sdpa", "hub", "addr_head"):
|
| 454 |
+
m = ByteLM(arm, d=96, layers=2, block=64, K=16)
|
| 455 |
+
logits = m(x)
|
| 456 |
+
assert logits.shape == (2, 64, VOCAB)
|
| 457 |
+
logits.sum().backward()
|
| 458 |
+
# causality check: future byte must not affect past logits
|
| 459 |
+
with torch.no_grad():
|
| 460 |
+
a = m(x)[0, 10]
|
| 461 |
+
x2 = x.clone(); x2[0, 40] = (x2[0, 40] + 7) % 256
|
| 462 |
+
b = m(x2)[0, 10]
|
| 463 |
+
assert torch.allclose(a, b, atol=1e-4), f"{arm} leaks future context"
|
| 464 |
+
print(f"{arm}: OK params={sum(p.numel() for p in m.parameters()):,} "
|
| 465 |
+
f"vitals={m.vitals()}", flush=True)
|
| 466 |
+
print("OK — AR bed smoke passed (verdict run: train() on GPU)", flush=True)
|
| 467 |
+
|
| 468 |
+
|
| 469 |
+
def _in_notebook() -> bool:
|
| 470 |
+
try:
|
| 471 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 472 |
+
return True
|
| 473 |
+
except NameError:
|
| 474 |
+
return False
|
| 475 |
+
|
| 476 |
+
|
| 477 |
+
if __name__ == "__main__":
|
| 478 |
+
if _in_notebook():
|
| 479 |
+
smoke()
|
| 480 |
+
print("Notebook mode: call train(steps=2000) in the next cell (GPU).")
|
| 481 |
+
else:
|
| 482 |
+
import argparse
|
| 483 |
+
ap = argparse.ArgumentParser()
|
| 484 |
+
ap.add_argument("--train", action="store_true")
|
| 485 |
+
ap.add_argument("--steps", type=int, default=2000)
|
| 486 |
+
a, _ = ap.parse_known_args()
|
| 487 |
+
train(steps=a.steps) if a.train else smoke()
|
exp013_aug/exp013_augmentation_bed.py
CHANGED
|
@@ -48,6 +48,8 @@ if "AlephAddress" not in globals():
|
|
| 48 |
raise ImportError("paste/run ar_differentiation_bed.py first")
|
| 49 |
from ar_differentiation_bed import AlephAddress, MslRelay, _wikitext_bytes
|
| 50 |
|
|
|
|
|
|
|
| 51 |
|
| 52 |
class SquaredReLU(nn.Module):
|
| 53 |
def forward(self, x):
|
|
@@ -205,7 +207,7 @@ def cache_word_embeddings(data_root, n_words=10000, device="cuda", batch=256):
|
|
| 205 |
# ============================================================ track A =============
|
| 206 |
def track_a(arms=("linear", "mlp", "aleph", "sign"), layers=("final", "penult"),
|
| 207 |
steps=1500, batch=64, seed=0, device="cuda",
|
| 208 |
-
data_root=
|
| 209 |
"""Next-CLIP-token prediction from frozen CLIP-L hidden states."""
|
| 210 |
if not torch.cuda.is_available():
|
| 211 |
raise RuntimeError("verdict runs are GPU-only")
|
|
@@ -298,7 +300,7 @@ class CharDecoder(nn.Module):
|
|
| 298 |
|
| 299 |
def track_b1(arms=("linear", "mlp"), substrates=("clip_final", "clip_penult",
|
| 300 |
"bert_cls", "bert_mean"), steps=3000, batch=128, seed=0,
|
| 301 |
-
device="cuda", data_root=
|
| 302 |
"""Spelling-AR from pooled embeddings. Run baselines first (the GATE:
|
| 303 |
qualify the task only if linear/mlp exact-match < 0.50), then aleph/sign."""
|
| 304 |
if not torch.cuda.is_available():
|
|
@@ -387,7 +389,7 @@ class _BlockWithAdapter(nn.Module):
|
|
| 387 |
|
| 388 |
|
| 389 |
def track_c(arms=("frozen", "aleph", "mlp"), steps=1500, batch=8, block=256,
|
| 390 |
-
seed=0, device="cuda", data_root=
|
| 391 |
eval_every=500, save=True):
|
| 392 |
"""GPT-2 124M frozen; adapters after every block; train adapters only."""
|
| 393 |
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
|
|
|
|
| 48 |
raise ImportError("paste/run ar_differentiation_bed.py first")
|
| 49 |
from ar_differentiation_bed import AlephAddress, MslRelay, _wikitext_bytes
|
| 50 |
|
| 51 |
+
DATA_ROOT = os.environ.get("GEOLIP_DATA", "./data")
|
| 52 |
+
|
| 53 |
|
| 54 |
class SquaredReLU(nn.Module):
|
| 55 |
def forward(self, x):
|
|
|
|
| 207 |
# ============================================================ track A =============
|
| 208 |
def track_a(arms=("linear", "mlp", "aleph", "sign"), layers=("final", "penult"),
|
| 209 |
steps=1500, batch=64, seed=0, device="cuda",
|
| 210 |
+
data_root=DATA_ROOT, eval_every=500, save=True):
|
| 211 |
"""Next-CLIP-token prediction from frozen CLIP-L hidden states."""
|
| 212 |
if not torch.cuda.is_available():
|
| 213 |
raise RuntimeError("verdict runs are GPU-only")
|
|
|
|
| 300 |
|
| 301 |
def track_b1(arms=("linear", "mlp"), substrates=("clip_final", "clip_penult",
|
| 302 |
"bert_cls", "bert_mean"), steps=3000, batch=128, seed=0,
|
| 303 |
+
device="cuda", data_root=DATA_ROOT, save=True):
|
| 304 |
"""Spelling-AR from pooled embeddings. Run baselines first (the GATE:
|
| 305 |
qualify the task only if linear/mlp exact-match < 0.50), then aleph/sign."""
|
| 306 |
if not torch.cuda.is_available():
|
|
|
|
| 389 |
|
| 390 |
|
| 391 |
def track_c(arms=("frozen", "aleph", "mlp"), steps=1500, batch=8, block=256,
|
| 392 |
+
seed=0, device="cuda", data_root=DATA_ROOT,
|
| 393 |
eval_every=500, save=True):
|
| 394 |
"""GPT-2 124M frozen; adapters after every block; train adapters only."""
|
| 395 |
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
|
exp013_aug/geolip_vitals.py
ADDED
|
@@ -0,0 +1,184 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""geolip_vitals.py — shared diagnostic harness for the GeoLIP aleph experiments.
|
| 2 |
+
ALL functions are READOUTS: no gradients, no losses. CV is a readout, never a
|
| 3 |
+
force. Addressing is judged by drift->0.29154 and CV->0.20, never by recon cosine
|
| 4 |
+
(judgment criteria per the aleph-void article: https://huggingface.co/blog/AbstractPhil/geometric-vocabulary-patchwork-aleph-void).
|
| 5 |
+
|
| 6 |
+
Vitals provided:
|
| 7 |
+
anchor_drift — geodesic drift of anchors from init; binding fraction @0.29154
|
| 8 |
+
pentachoron_cv — CM 4-volume CV over random 5-row subsets (geovocab2 import)
|
| 9 |
+
axis_aliveness — oriented-address usage: axes alive, hppl, collapse flag
|
| 10 |
+
gate_stats — gate means vs the 0.012-0.03 band
|
| 11 |
+
path_diversity — unique-path counting, FIXED high-bits hash (low-16 bug is the
|
| 12 |
+
retracted artifact — never use the low bits)
|
| 13 |
+
grad_norm_spread — gradient democracy monitor (orders-of-magnitude spread)
|
| 14 |
+
CVScreen — CV@1000-batch early band screen (<0.30 LOW / .35-.50 MID / >.80 HIGH)
|
| 15 |
+
|
| 16 |
+
Smoke on a torch-capable env: python geolip_vitals.py
|
| 17 |
+
"""
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
import math
|
| 20 |
+
import torch
|
| 21 |
+
|
| 22 |
+
BINDING = 0.29154 # radians; the binding/separation constant
|
| 23 |
+
CV_BAND = (0.13, 0.30) # CM CV band (discovery_catalog #4)
|
| 24 |
+
GATE_BAND = (0.012, 0.03) # live invariant candidate (acd_campaign)
|
| 25 |
+
KNUTH32 = 2654435761
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
# ----------------------------------------------------------------------------- drift
|
| 29 |
+
@torch.no_grad()
|
| 30 |
+
def anchor_drift(current: torch.Tensor, init: torch.Tensor, tol: float = 0.05) -> dict:
|
| 31 |
+
"""Geodesic drift (radians) of each row of `current` from its row in `init`,
|
| 32 |
+
both row-normalized. Returns mean/std/per-row drift and the fraction of rows
|
| 33 |
+
within +/-tol of BINDING (the GLFM '46%' readout)."""
|
| 34 |
+
a = torch.nn.functional.normalize(current.float(), dim=-1)
|
| 35 |
+
b = torch.nn.functional.normalize(init.float(), dim=-1)
|
| 36 |
+
cos = (a * b).sum(-1).clamp(-1.0, 1.0)
|
| 37 |
+
drift = torch.arccos(cos)
|
| 38 |
+
frac = ((drift - BINDING).abs() <= tol).float().mean()
|
| 39 |
+
return {"mean": drift.mean().item(), "std": drift.std().item(),
|
| 40 |
+
"per_row": drift, "binding_fraction": frac.item()}
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
# -------------------------------------------------------------------------------- cv
|
| 44 |
+
@torch.no_grad()
|
| 45 |
+
def pentachoron_cv(rows: torch.Tensor, n_samples: int = 200,
|
| 46 |
+
generator: torch.Generator | None = None) -> float:
|
| 47 |
+
"""CV (std/mean) of Cayley-Menger 4-simplex volumes over n_samples random
|
| 48 |
+
5-row subsets. CM comes from geovocab2 — never inlined (program rule);
|
| 49 |
+
fp32 det only. Rows are row-normalized before measurement."""
|
| 50 |
+
try:
|
| 51 |
+
from geovocab2.formula.cayley_menger import simplex_volume # noqa: F401
|
| 52 |
+
except Exception as e: # pragma: no cover
|
| 53 |
+
raise ImportError(
|
| 54 |
+
"pentachoron_cv requires geovocab2 (CM must be imported, never inlined "
|
| 55 |
+
"— program rule). Install/point PYTHONPATH at geovocab2; expected "
|
| 56 |
+
"geovocab2.formula.cayley_menger.simplex_volume(points fp32)."
|
| 57 |
+
) from e
|
| 58 |
+
x = torch.nn.functional.normalize(rows.float(), dim=-1)
|
| 59 |
+
n = x.shape[0]
|
| 60 |
+
if n < 5:
|
| 61 |
+
raise ValueError(f"pentachoron_cv needs >=5 rows, got {n}")
|
| 62 |
+
g = generator or torch.Generator(device="cpu").manual_seed(0)
|
| 63 |
+
vols = []
|
| 64 |
+
for _ in range(n_samples):
|
| 65 |
+
idx = torch.randperm(n, generator=g)[:5]
|
| 66 |
+
vols.append(simplex_volume(x[idx].to(torch.float32)))
|
| 67 |
+
v = torch.stack([torch.as_tensor(v, dtype=torch.float32) for v in vols])
|
| 68 |
+
return (v.std() / v.mean().clamp_min(1e-12)).item()
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
# ------------------------------------------------------------------------- aliveness
|
| 72 |
+
@torch.no_grad()
|
| 73 |
+
def axis_aliveness(oriented_weights: torch.Tensor, alive_thresh: float = 1e-3) -> dict:
|
| 74 |
+
"""`oriented_weights`: (..., 2K) nonnegative oriented-softmax address rows
|
| 75 |
+
(sum to 1 on the last dim). Returns axes-alive count, mean-usage perplexity
|
| 76 |
+
(hppl analogue; healthy hosted reference 125-126/128), and a collapse flag.
|
| 77 |
+
Reference behavior: near-uniform aliveness at div_weight=0 (discovery #22)."""
|
| 78 |
+
w = oriented_weights.reshape(-1, oriented_weights.shape[-1]).float()
|
| 79 |
+
usage = w.mean(0)
|
| 80 |
+
usage = usage / usage.sum().clamp_min(1e-12)
|
| 81 |
+
# an axis is alive if its mean usage exceeds alive_thresh x the uniform share
|
| 82 |
+
alive = int((usage > alive_thresh * (1.0 / usage.numel())).sum())
|
| 83 |
+
ent = -(usage.clamp_min(1e-12) * usage.clamp_min(1e-12).log()).sum()
|
| 84 |
+
ppl = float(ent.exp())
|
| 85 |
+
return {"axes_total": usage.numel(), "axes_alive": alive, "usage_ppl": ppl,
|
| 86 |
+
"collapsed": ppl < 0.05 * usage.numel()}
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
# ------------------------------------------------------------------------------ gates
|
| 90 |
+
@torch.no_grad()
|
| 91 |
+
def gate_stats(gates: torch.Tensor) -> dict:
|
| 92 |
+
"""Gate values (post-sigmoid/clamp). Reports mean and whether it sits in the
|
| 93 |
+
0.012-0.03 band (read-only — the band is a candidate invariant, never a target)."""
|
| 94 |
+
g = gates.float().flatten()
|
| 95 |
+
m = g.mean().item()
|
| 96 |
+
return {"mean": m, "std": g.std().item(),
|
| 97 |
+
"in_band": GATE_BAND[0] <= m <= GATE_BAND[1]}
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
# ------------------------------------------------------------------------------ paths
|
| 101 |
+
@torch.no_grad()
|
| 102 |
+
def path_diversity(ids: torch.Tensor) -> dict:
|
| 103 |
+
"""Unique-path counting with the FIXED multiplicative hash:
|
| 104 |
+
((ids * 2654435761) % 2^32) >> 16 — Knuth needs the HIGH bits; the low-16
|
| 105 |
+
variant produced a retracted ~1,500 path ceiling in a prior campaign.
|
| 106 |
+
`ids`: integer tensor, one composed path id per row (any shape)."""
|
| 107 |
+
x = ids.reshape(-1).to(torch.int64)
|
| 108 |
+
hashed = ((x * KNUTH32) % (1 << 32)) >> 16
|
| 109 |
+
return {"n": int(x.numel()),
|
| 110 |
+
"unique_raw": int(torch.unique(x).numel()),
|
| 111 |
+
"unique_hashed": int(torch.unique(hashed).numel())}
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
@torch.no_grad()
|
| 115 |
+
def compose_path_ids(stage_indices: list[torch.Tensor], radix: int) -> torch.Tensor:
|
| 116 |
+
"""Compose per-stage discrete indices (each (...,) int in [0, radix)) into a
|
| 117 |
+
single path id, positional base-`radix` — construction, not hashing."""
|
| 118 |
+
out = torch.zeros_like(stage_indices[0], dtype=torch.int64)
|
| 119 |
+
for s in stage_indices:
|
| 120 |
+
out = out * radix + s.to(torch.int64)
|
| 121 |
+
return out
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
# --------------------------------------------------------------------- grad democracy
|
| 125 |
+
@torch.no_grad()
|
| 126 |
+
def grad_norm_spread(groups: dict[str, list[torch.nn.Parameter]]) -> dict:
|
| 127 |
+
"""Gradient-democracy monitor. `groups`: name -> params of one parallel member
|
| 128 |
+
(tower/expert). Reports per-group grad norms and the orders-of-magnitude spread.
|
| 129 |
+
Reference: unequalized heterogeneous towers spread ~20 orders (fibonacci dead at
|
| 130 |
+
2.25e-21 under helix); equalized ~0.0 (geofractal gradient-democracy result)."""
|
| 131 |
+
norms = {}
|
| 132 |
+
for name, params in groups.items():
|
| 133 |
+
gs = [p.grad for p in params if p.grad is not None]
|
| 134 |
+
norms[name] = float(torch.sqrt(sum((g.float() ** 2).sum() for g in gs)).item()) \
|
| 135 |
+
if gs else 0.0
|
| 136 |
+
vals = [v for v in norms.values() if v > 0]
|
| 137 |
+
spread = (math.log10(max(vals)) - math.log10(min(vals))) if len(vals) >= 2 else 0.0
|
| 138 |
+
return {"norms": norms, "spread_orders": spread, "dead": [k for k, v in norms.items() if v == 0.0]}
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
# ----------------------------------------------------------------------------- screen
|
| 142 |
+
class CVScreen:
|
| 143 |
+
"""CV@N early band screen (tri-band ft1): record pentachoron CV at `step_mark`
|
| 144 |
+
batches; classify <0.30 LOW / 0.35-0.50 MID / >0.80 HIGH. Turns ~2h/config
|
| 145 |
+
into ~7min. Readout only."""
|
| 146 |
+
def __init__(self, step_mark: int = 1000):
|
| 147 |
+
self.step_mark = step_mark
|
| 148 |
+
self.recorded: float | None = None
|
| 149 |
+
|
| 150 |
+
def maybe_record(self, step: int, rows: torch.Tensor) -> float | None:
|
| 151 |
+
if self.recorded is None and step >= self.step_mark:
|
| 152 |
+
self.recorded = pentachoron_cv(rows)
|
| 153 |
+
return self.recorded
|
| 154 |
+
|
| 155 |
+
@property
|
| 156 |
+
def band(self) -> str | None:
|
| 157 |
+
c = self.recorded
|
| 158 |
+
if c is None:
|
| 159 |
+
return None
|
| 160 |
+
if c < 0.30:
|
| 161 |
+
return "LOW"
|
| 162 |
+
if 0.35 <= c <= 0.50:
|
| 163 |
+
return "MID"
|
| 164 |
+
if c > 0.80:
|
| 165 |
+
return "HIGH"
|
| 166 |
+
return "BETWEEN"
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
# ------------------------------------------------------------------------------ smoke
|
| 170 |
+
if __name__ == "__main__": # shapes/parse smoke ONLY — no training, ever.
|
| 171 |
+
g = torch.Generator().manual_seed(0)
|
| 172 |
+
K, D = 64, 4
|
| 173 |
+
init = torch.nn.functional.normalize(torch.randn(K, D, generator=g), dim=-1)
|
| 174 |
+
cur = torch.nn.functional.normalize(init + 0.29 * torch.randn(K, D, generator=g), dim=-1)
|
| 175 |
+
print("drift:", {k: v for k, v in anchor_drift(cur, init).items() if k != "per_row"})
|
| 176 |
+
w = torch.softmax(torch.randn(32, 2 * K, generator=g), dim=-1)
|
| 177 |
+
print("aliveness:", axis_aliveness(w))
|
| 178 |
+
print("gates:", gate_stats(torch.full((8,), 0.024)))
|
| 179 |
+
ids = compose_path_ids([torch.randint(0, 16, (4096,), generator=g) for _ in range(4)], 16)
|
| 180 |
+
print("paths:", path_diversity(ids))
|
| 181 |
+
lin = torch.nn.Linear(8, 8)
|
| 182 |
+
lin(torch.randn(4, 8)).sum().backward()
|
| 183 |
+
print("democracy:", grad_norm_spread({"a": list(lin.parameters())}))
|
| 184 |
+
print("OK — vitals smoke passed (pentachoron_cv needs geovocab2; run on GPU env)")
|
exp013_aug/read_codebook.py
ADDED
|
@@ -0,0 +1,159 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""read_codebook.py — projective reading of cultivated aleph codebooks.
|
| 2 |
+
Antipodal-collapse extraction on trained codebooks + projective statistics on
|
| 3 |
+
RP^(D-1) — applied to the exp012 AR-bed specimens.
|
| 4 |
+
|
| 5 |
+
Recipe per the Polygonal Omega article (geometric-tri-band-ft2): collapse = (row_i - row_j)/2 normalized for each MUTUAL-STRONGEST
|
| 6 |
+
pair with cos < -0.9 — "a deterministic tensor operation," not clustering.
|
| 7 |
+
Projective metric ALWAYS arccos|<a,b>| (metric-alignment rule, reading-voids-ft1).
|
| 8 |
+
D=4 scope is the validated regime (D=5 walked back; axis count grows with D).
|
| 9 |
+
|
| 10 |
+
Readouts per specimen:
|
| 11 |
+
pairs / n_axes / unpaired — antipodal structure
|
| 12 |
+
proj_angle mean vs uniform baseline, deviation — near-uniform RP^(D-1)?
|
| 13 |
+
drift from home + binding fraction @0.29154 — cultivation record
|
| 14 |
+
erank of the axis set — spectral occupancy
|
| 15 |
+
verdict: PROJECTIVE-CLEAN (|dev|<0.05, util>0.95, secondary pairs<=3) /
|
| 16 |
+
-MOSTLY / STRUCTURED / DEGENERATE (per Polygonal Omega thresholds)
|
| 17 |
+
|
| 18 |
+
Usage (terminal): python read_codebook.py <ckpt_or_dir> [more paths...]
|
| 19 |
+
Colab: paste geolip_vitals.py cell first (optional), then this file, then
|
| 20 |
+
read_all(r"/content/data/ar_ckpts").
|
| 21 |
+
"""
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
import math
|
| 24 |
+
import sys
|
| 25 |
+
import torch
|
| 26 |
+
import torch.nn.functional as F
|
| 27 |
+
|
| 28 |
+
BINDING = 0.29154
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
@torch.no_grad()
|
| 32 |
+
def antipodal_collapse(codebook: torch.Tensor, thresh: float = -0.9) -> dict:
|
| 33 |
+
"""Mutual-strongest antipodal pairing + collapse to axes on RP^(D-1)."""
|
| 34 |
+
A = F.normalize(codebook.float(), dim=-1)
|
| 35 |
+
K = A.shape[0]
|
| 36 |
+
cos = A @ A.T
|
| 37 |
+
cos.fill_diagonal_(2.0) # exclude self from minima
|
| 38 |
+
nearest_neg = cos.argmin(dim=-1) # most-antipodal partner
|
| 39 |
+
pairs = []
|
| 40 |
+
used = set()
|
| 41 |
+
for i in range(K):
|
| 42 |
+
j = int(nearest_neg[i])
|
| 43 |
+
if i < j and int(nearest_neg[j]) == i and cos[i, j] < thresh:
|
| 44 |
+
pairs.append((i, j))
|
| 45 |
+
used.update((i, j))
|
| 46 |
+
axes = [F.normalize((A[i] - A[j]) / 2.0, dim=-1) for i, j in pairs]
|
| 47 |
+
axes += [A[i] for i in range(K) if i not in used] # unpaired rows as axes
|
| 48 |
+
axes = torch.stack(axes) if axes else A[:0]
|
| 49 |
+
# sign-canon onto RP: first nonzero coordinate positive
|
| 50 |
+
for r in range(axes.shape[0]):
|
| 51 |
+
nz = torch.nonzero(axes[r].abs() > 1e-8)
|
| 52 |
+
if nz.numel() and axes[r, nz[0, 0]] < 0:
|
| 53 |
+
axes[r] = -axes[r]
|
| 54 |
+
return {"pairs": len(pairs), "n_axes": axes.shape[0],
|
| 55 |
+
"unpaired": K - 2 * len(pairs), "axes": axes}
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
@torch.no_grad()
|
| 59 |
+
def projective_stats(axes: torch.Tensor, n_baseline: int = 20000,
|
| 60 |
+
seed: int = 0) -> dict:
|
| 61 |
+
"""Mean projective angle arccos|<a,b>| vs a uniform-RP baseline at same (n, D)."""
|
| 62 |
+
n, D = axes.shape
|
| 63 |
+
if n < 2:
|
| 64 |
+
return {"proj_angle_mean": None, "uniform_baseline": None,
|
| 65 |
+
"deviation": None, "erank": None}
|
| 66 |
+
def mean_angle(rows):
|
| 67 |
+
c = (rows @ rows.T).abs().clamp(max=1.0)
|
| 68 |
+
iu = torch.triu_indices(rows.shape[0], rows.shape[0], offset=1)
|
| 69 |
+
return torch.arccos(c[iu[0], iu[1]]).mean().item()
|
| 70 |
+
obs = mean_angle(axes)
|
| 71 |
+
g = torch.Generator().manual_seed(seed)
|
| 72 |
+
base_angles = []
|
| 73 |
+
m = max(2, n)
|
| 74 |
+
for _ in range(max(1, n_baseline // max(1, m * (m - 1) // 2))):
|
| 75 |
+
r = F.normalize(torch.randn(m, D, generator=g), dim=-1)
|
| 76 |
+
base_angles.append(mean_angle(r))
|
| 77 |
+
base = sum(base_angles) / len(base_angles)
|
| 78 |
+
s = torch.linalg.svdvals(axes)
|
| 79 |
+
p = (s / s.sum().clamp_min(1e-12))
|
| 80 |
+
erank = float(torch.exp(-(p.clamp_min(1e-12) * p.clamp_min(1e-12).log()).sum()))
|
| 81 |
+
return {"proj_angle_mean": round(obs, 4), "uniform_baseline": round(base, 4),
|
| 82 |
+
"deviation": round(obs - base, 4), "erank": round(erank, 3)}
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
@torch.no_grad()
|
| 86 |
+
def read_specimen(path: str) -> dict:
|
| 87 |
+
ck = torch.load(path, map_location="cpu", weights_only=True)
|
| 88 |
+
out = {"file": path.split("\\")[-1].split("/")[-1],
|
| 89 |
+
"arm": ck.get("arm"), "seed": ck.get("seed"),
|
| 90 |
+
"steps": ck.get("steps"), "val_bpb": round(ck.get("val_bpb", -1), 4)}
|
| 91 |
+
if "state_dict" in ck: # full specimen checkpoint
|
| 92 |
+
sd = ck["state_dict"]
|
| 93 |
+
books = {k[:-len(".codebook")]: sd[k] for k in sd
|
| 94 |
+
if k.endswith("addr.codebook") or k.endswith("head_addr.codebook")}
|
| 95 |
+
homes = {k[:-len(".home")]: sd[k] for k in sd if k.endswith(".home")}
|
| 96 |
+
else: # bare genome dict (exp014+ champion files):
|
| 97 |
+
# books under flat/root/branch* keys; *_proj entries are projections
|
| 98 |
+
books = {k: v for k, v in ck.items()
|
| 99 |
+
if torch.is_tensor(v) and v.ndim == 2
|
| 100 |
+
and (k in ("flat", "root") or k.startswith("branch"))}
|
| 101 |
+
homes = {}
|
| 102 |
+
reads = {}
|
| 103 |
+
for name, cb in books.items():
|
| 104 |
+
col = antipodal_collapse(cb)
|
| 105 |
+
stats = projective_stats(col["axes"])
|
| 106 |
+
home = homes.get(name)
|
| 107 |
+
drift = None
|
| 108 |
+
binding = None
|
| 109 |
+
if home is not None and home.shape == cb.shape:
|
| 110 |
+
a = F.normalize(cb.float(), dim=-1)
|
| 111 |
+
b = F.normalize(home.float(), dim=-1)
|
| 112 |
+
dr = torch.arccos((a * b).sum(-1).clamp(-1, 1))
|
| 113 |
+
drift = round(dr.mean().item(), 4)
|
| 114 |
+
binding = round(((dr - BINDING).abs() <= 0.05).float().mean().item(), 4)
|
| 115 |
+
util = col["n_axes"] / cb.shape[0]
|
| 116 |
+
dev = stats["deviation"]
|
| 117 |
+
if dev is not None and abs(dev) < 0.05 and util > 0.95 and col["pairs"] <= 3:
|
| 118 |
+
verdict = "PROJECTIVE-CLEAN"
|
| 119 |
+
elif dev is not None and abs(dev) < 0.05:
|
| 120 |
+
verdict = "PROJECTIVE-MOSTLY"
|
| 121 |
+
elif dev is not None and dev > 0.05:
|
| 122 |
+
verdict = "STRUCTURED(repulsive)"
|
| 123 |
+
else:
|
| 124 |
+
verdict = "DEGENERATE/CLUMPED" if dev is not None else "TOO-FEW-AXES"
|
| 125 |
+
reads[name] = {
|
| 126 |
+
"pairs": col["pairs"], "n_axes": col["n_axes"], **stats,
|
| 127 |
+
"drift": drift, "binding_frac": binding, "verdict": verdict}
|
| 128 |
+
out["codebooks"] = reads
|
| 129 |
+
return out
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def read_all(root: str) -> list:
|
| 133 |
+
import glob, os
|
| 134 |
+
results = []
|
| 135 |
+
for p in sorted(glob.glob(os.path.join(root, "*.pt"))):
|
| 136 |
+
r = read_specimen(p)
|
| 137 |
+
print(r, flush=True)
|
| 138 |
+
results.append(r)
|
| 139 |
+
return results
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def _in_notebook() -> bool:
|
| 143 |
+
try:
|
| 144 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 145 |
+
return True
|
| 146 |
+
except NameError:
|
| 147 |
+
return False
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
if __name__ == "__main__":
|
| 151 |
+
if _in_notebook():
|
| 152 |
+
print("Notebook mode: call read_all(r'<data_root>/ar_ckpts') in the next cell.")
|
| 153 |
+
else:
|
| 154 |
+
args = [a for a in sys.argv[1:] if not a.startswith("-")]
|
| 155 |
+
if not args:
|
| 156 |
+
print("usage: python read_codebook.py <ckpt_or_dir> [...]")
|
| 157 |
+
for a in args:
|
| 158 |
+
import os
|
| 159 |
+
read_all(a) if os.path.isdir(a) else print(read_specimen(a))
|
exp013_aug/repro.py
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""repro.py — standalone loader/runner for exp013_aug. Every code dependency
|
| 2 |
+
lives in THIS folder (geolip_vitals.py, ar_differentiation_bed.py,
|
| 3 |
+
exp013_augmentation_bed.py, read_codebook.py) — no cross-package paths.
|
| 4 |
+
|
| 5 |
+
python repro.py # CPU smoke: parse + shapes, no downloads
|
| 6 |
+
python repro.py --track b1 # the gate + baselines (GPU; caches models)
|
| 7 |
+
python repro.py --track a # CLIP-L dual-layer token-AR (GPU)
|
| 8 |
+
python repro.py --track c # GPT-2 relay augmentation (GPU)
|
| 9 |
+
|
| 10 |
+
Data + caches land in ./data (override with the GEOLIP_DATA env var).
|
| 11 |
+
Verdict-run costs on one RTX 4090: caching ~5-10 min (downloads CLIP-L/BERT/GPT-2
|
| 12 |
+
on first run), track b1 ~20 min, track a ~30 min, track c ~40 min.
|
| 13 |
+
"""
|
| 14 |
+
import os
|
| 15 |
+
import runpy
|
| 16 |
+
import sys
|
| 17 |
+
|
| 18 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 19 |
+
sys.path.insert(0, HERE)
|
| 20 |
+
|
| 21 |
+
if __name__ == "__main__":
|
| 22 |
+
if len(sys.argv) == 1:
|
| 23 |
+
import geolip_vitals # noqa: F401 (paste order)
|
| 24 |
+
import ar_differentiation_bed as bed
|
| 25 |
+
import exp013_augmentation_bed as aug
|
| 26 |
+
bed.smoke()
|
| 27 |
+
aug.smoke()
|
| 28 |
+
print("repro smoke passed — run with --track {a,b1,c} for verdict runs")
|
| 29 |
+
else:
|
| 30 |
+
target = os.path.join(HERE, "exp013_augmentation_bed.py")
|
| 31 |
+
sys.argv = [target] + sys.argv[1:]
|
| 32 |
+
runpy.run_path(target, run_name="__main__")
|
exp014_gd/README.md
CHANGED
|
@@ -17,13 +17,34 @@ parity; nothing accelerates).
|
|
| 17 |
- `article.md` — the write-up.
|
| 18 |
- `exp014_genetic_distillation.py` — consensus operator (projective ICP + GPA,
|
| 19 |
planted-truth validated), TreeHead (slot-parallel root), KD inheritance,
|
| 20 |
-
tournament + implant runners
|
| 21 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
- `build_results.py` → `results/results.json` — reads the ledger and RE-ASSERTS
|
| 23 |
every article claim (keep-last dedupe; superseded rows = the aborted-campaign
|
| 24 |
inverse-evolution record, preserved on purpose).
|
| 25 |
- `results/ledger.jsonl` — the append-only per-member record (138 rows).
|
| 26 |
-
- `genomes/` — 23 per-generation champion codebooks (the germline fossil record
|
| 27 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
License: MIT · AbstractPhil + Claude Fable 5 · July 10, 2026
|
|
|
|
| 17 |
- `article.md` — the write-up.
|
| 18 |
- `exp014_genetic_distillation.py` — consensus operator (projective ICP + GPA,
|
| 19 |
planted-truth validated), TreeHead (slot-parallel root), KD inheritance,
|
| 20 |
+
tournament + implant runners, real CLI (`--mode tournament|b1|b2`).
|
| 21 |
+
- `geolip_vitals.py` / `ar_differentiation_bed.py` /
|
| 22 |
+
`exp013_augmentation_bed.py` / `read_codebook.py` — this package's own copies
|
| 23 |
+
of the shared harness (readout diagnostics; the ByteLM organism + AlephAddress
|
| 24 |
+
+ MslRelay attention mechanisms; the GPT-2 relay bed used by `run_b2`; the
|
| 25 |
+
projective specimen reader). Identical to the copies in their home packages —
|
| 26 |
+
the package is standalone.
|
| 27 |
+
- `repro.py` — loads the code files from this folder and runs them.
|
| 28 |
- `build_results.py` → `results/results.json` — reads the ledger and RE-ASSERTS
|
| 29 |
every article claim (keep-last dedupe; superseded rows = the aborted-campaign
|
| 30 |
inverse-evolution record, preserved on purpose).
|
| 31 |
- `results/ledger.jsonl` — the append-only per-member record (138 rows).
|
| 32 |
+
- `genomes/` — 23 per-generation champion codebooks (the germline fossil record).
|
| 33 |
+
|
| 34 |
+
## Reproduce (from inside this folder)
|
| 35 |
+
```bash
|
| 36 |
+
pip install torch --index-url https://download.pytorch.org/whl/cu128
|
| 37 |
+
pip install pyarrow huggingface_hub # + transformers for --mode b2
|
| 38 |
+
python repro.py # CPU smoke: parse + shapes + GPA sanity
|
| 39 |
+
python repro.py --mode tournament --lineage aleph_full # one bracket (GPU)
|
| 40 |
+
python repro.py --mode b1 # cross-size implants (GPU)
|
| 41 |
+
python repro.py --mode b2 --genome genomes/champion_aleph_full_t0_g3.pt # GPT-2 relays
|
| 42 |
+
python read_codebook.py genomes/ # read the germline fossil record
|
| 43 |
+
python build_results.py # re-assert every article claim
|
| 44 |
+
```
|
| 45 |
+
Data lands in `./data` (override with the `GEOLIP_DATA` env var); tournament
|
| 46 |
+
ledgers + champion genomes in `./data/exp014`. In Colab, paste
|
| 47 |
+
`geolip_vitals.py`, `ar_differentiation_bed.py` (+ `exp013_augmentation_bed.py`
|
| 48 |
+
for b2), then `exp014_genetic_distillation.py` into cells in that order.
|
| 49 |
|
| 50 |
License: MIT · AbstractPhil + Claude Fable 5 · July 10, 2026
|
exp014_gd/ar_differentiation_bed.py
ADDED
|
@@ -0,0 +1,487 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""ar_differentiation_bed.py — exp012: autoregressive differentiation of the aleph.
|
| 2 |
+
|
| 3 |
+
Differentiation is cultivated by PREDICTIVE pressure along the sequence — the
|
| 4 |
+
address parameterizing the next-byte distribution (Law 2: chain-rule advantage pays
|
| 5 |
+
ONLY where the composed address directly parameterizes the predictive distribution).
|
| 6 |
+
This bed puts the aleph in the autoregressive gradient path and measures what
|
| 7 |
+
differentiates. The head arms enforce the employment law at its maximum: the
|
| 8 |
+
ENTIRE next-byte distribution is parameterized by the address.
|
| 9 |
+
|
| 10 |
+
Byte-level causal LM on wikitext-2-raw (HF parquet, CDN-fast), block 256. ARMS:
|
| 11 |
+
sdpa — standard causal transformer control (matched trunk).
|
| 12 |
+
hub — attention replaced by CAUSAL HUB: linear attention whose feature map
|
| 13 |
+
is the 2K-oriented aleph address, prefix-sum memories (no selection
|
| 14 |
+
event; O(n*K*d)). Differentiation cultivated INSIDE attention.
|
| 15 |
+
addr_head — sdpa trunk, but the OUTPUT HEAD reads ONLY the signed aleph
|
| 16 |
+
coefficient vector w_k = sinh(u_k)/sum_j cosh(u_j) of the final
|
| 17 |
+
hidden state (K -> 256 logits). The address MUST carry every bit of
|
| 18 |
+
next-byte information — the hardest Law-2 bottleneck.
|
| 19 |
+
|
| 20 |
+
JUDGED BY: val bits-per-byte per arm (task) + CULTIVATION VITALS on every aleph
|
| 21 |
+
codebook (readouts, never losses): axis aliveness/hppl, drift-from-init +
|
| 22 |
+
binding fraction @0.29154, winner-|cos| saturation (sign-code emergence), shadow
|
| 23 |
+
path diversity (fixed high-bits hash). Never by recon.
|
| 24 |
+
|
| 25 |
+
Riders: pure Adam wd=0; no BN/Dropout/GAP on geometric paths; orthogonal init;
|
| 26 |
+
Colab-cell-safe (paste-ahead imports, no bare argparse, no __file__ reliance);
|
| 27 |
+
GPU-only for verdict runs.
|
| 28 |
+
|
| 29 |
+
Terminal: python ar_differentiation_bed.py # shapes/parse smoke
|
| 30 |
+
python ar_differentiation_bed.py --train # verdict run
|
| 31 |
+
Colab: paste geolip_vitals.py cell, then this file (smoke auto-runs),
|
| 32 |
+
then train(steps=2000, data_root="/content/data") in the next cell.
|
| 33 |
+
"""
|
| 34 |
+
from __future__ import annotations
|
| 35 |
+
import math
|
| 36 |
+
import torch
|
| 37 |
+
import torch.nn as nn
|
| 38 |
+
import torch.nn.functional as F
|
| 39 |
+
|
| 40 |
+
if "anchor_drift" not in globals():
|
| 41 |
+
try:
|
| 42 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 43 |
+
except ImportError:
|
| 44 |
+
_here = globals().get("__file__")
|
| 45 |
+
if _here is not None:
|
| 46 |
+
import sys, pathlib
|
| 47 |
+
sys.path.insert(0, str(pathlib.Path(_here).parent))
|
| 48 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 49 |
+
else:
|
| 50 |
+
raise ImportError(
|
| 51 |
+
"geolip_vitals not found — paste/run its cell first, or "
|
| 52 |
+
"hf_hub_download exp012_ar/geolip_vitals.py from "
|
| 53 |
+
"AbstractPhil/geolip-aleph-differentiation.")
|
| 54 |
+
|
| 55 |
+
VOCAB = 256 # bytes
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
# ------------------------------------------------------------------ aleph address
|
| 59 |
+
def _super_fibonacci_s3(n: int) -> torch.Tensor:
|
| 60 |
+
"""Near-uniform unit quaternions (Alexa CVPR'22) —
|
| 61 |
+
starts the codebook INSIDE the RP^3 attractor basin. D=4 only."""
|
| 62 |
+
PHI, PSI = math.sqrt(2.0), 1.533751168755204288118041
|
| 63 |
+
i = torch.arange(n, dtype=torch.float64)
|
| 64 |
+
s = (i + 0.5) / n
|
| 65 |
+
r, R = torch.sqrt(s), torch.sqrt(1.0 - s)
|
| 66 |
+
a, b = 2 * math.pi * i / PHI, 2 * math.pi * i / PSI
|
| 67 |
+
q = torch.stack([r * torch.sin(a), r * torch.cos(a),
|
| 68 |
+
R * torch.sin(b), R * torch.cos(b)], dim=-1)
|
| 69 |
+
return F.normalize(q, dim=-1).float()
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
class AlephAddress(nn.Module):
|
| 73 |
+
"""Closed-form aleph over 2K oriented half-axes (aleph-void article).
|
| 74 |
+
signed(x): (..., K) w_k = sinh(u_k)/sum_j cosh(u_j) — the Law-2 head feature.
|
| 75 |
+
oriented(x): ((..., K), (..., K)) positive halves of the 2K softmax — HUB map."""
|
| 76 |
+
|
| 77 |
+
def __init__(self, K: int, D: int, tau: float = 0.1, init: str = "random"):
|
| 78 |
+
super().__init__()
|
| 79 |
+
self.K, self.D, self.tau = K, D, tau
|
| 80 |
+
if init == "fibonacci":
|
| 81 |
+
assert D == 4, "fibonacci init lives on S^3 (D=4)"
|
| 82 |
+
A = _super_fibonacci_s3(K)
|
| 83 |
+
else:
|
| 84 |
+
A = F.normalize(torch.randn(K, D), dim=-1)
|
| 85 |
+
self.codebook = nn.Parameter(A)
|
| 86 |
+
self.register_buffer("home", self.codebook.detach().clone())
|
| 87 |
+
|
| 88 |
+
def _u(self, x):
|
| 89 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 90 |
+
return (F.normalize(x, dim=-1) @ A.transpose(-1, -2)) / self.tau
|
| 91 |
+
|
| 92 |
+
def oriented(self, x):
|
| 93 |
+
u = self._u(x)
|
| 94 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 95 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 96 |
+
Z = (ep + en).sum(dim=-1, keepdim=True)
|
| 97 |
+
return ep / Z, en / Z
|
| 98 |
+
|
| 99 |
+
def signed(self, x):
|
| 100 |
+
u = self._u(x)
|
| 101 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 102 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 103 |
+
return (ep - en) / (ep + en).sum(dim=-1, keepdim=True)
|
| 104 |
+
|
| 105 |
+
def signed_at(self, x, taus):
|
| 106 |
+
"""Multi-tau stroboscope (rule of 3): signed coefficients at several
|
| 107 |
+
temperatures, concatenated — softer taus keep the vector dense while a
|
| 108 |
+
hard tau supplies the sign-code sharpness. v2 refinement (b)."""
|
| 109 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 110 |
+
cos = F.normalize(x, dim=-1) @ A.transpose(-1, -2)
|
| 111 |
+
outs = []
|
| 112 |
+
for t in taus:
|
| 113 |
+
u = cos / t
|
| 114 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 115 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 116 |
+
outs.append((ep - en) / (ep + en).sum(dim=-1, keepdim=True))
|
| 117 |
+
return torch.cat(outs, dim=-1)
|
| 118 |
+
|
| 119 |
+
def m_hat(self, x):
|
| 120 |
+
"""Closed-form soft read (decoders read M_hat, never M). v2 control (c)."""
|
| 121 |
+
u = self._u(x)
|
| 122 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 123 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 124 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 125 |
+
return ((ep - en) @ A) / (ep + en).sum(dim=-1, keepdim=True)
|
| 126 |
+
|
| 127 |
+
def m_hard_ste(self, x):
|
| 128 |
+
"""Hard mode (aleph-void article): M_hard = sign(cos_win) * A[win], straight-through to
|
| 129 |
+
the soft read — forward fully discrete SIGN CODE, backward soft gradient.
|
| 130 |
+
Legal per theme A (reconstructive sign code, not a one-hot roster pick)."""
|
| 131 |
+
u = self._u(x)
|
| 132 |
+
soft = self.m_hat(x)
|
| 133 |
+
win = u.abs().argmax(dim=-1)
|
| 134 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 135 |
+
sign = torch.sign(torch.gather(u, -1, win.unsqueeze(-1))).squeeze(-1)
|
| 136 |
+
hard = sign.unsqueeze(-1) * A[win]
|
| 137 |
+
return hard + soft - soft.detach()
|
| 138 |
+
|
| 139 |
+
@torch.no_grad()
|
| 140 |
+
def vitals(self, x_sample) -> dict:
|
| 141 |
+
u = self._u(x_sample.reshape(-1, x_sample.shape[-1]))
|
| 142 |
+
p, n = self.oriented(x_sample.reshape(-1, x_sample.shape[-1]))
|
| 143 |
+
two_k = torch.cat([p, n], dim=-1)
|
| 144 |
+
win = two_k.argmax(dim=-1)
|
| 145 |
+
cos_win = (u.abs().amax(dim=-1) * self.tau) # winner |cos| — sign-code sat.
|
| 146 |
+
d = anchor_drift(self.codebook, self.home)
|
| 147 |
+
return {"drift": round(d["mean"], 4),
|
| 148 |
+
"binding_frac": round(d["binding_fraction"], 4),
|
| 149 |
+
"aliveness": axis_aliveness(two_k),
|
| 150 |
+
"win_cos_mean": round(cos_win.mean().item(), 4),
|
| 151 |
+
"paths": path_diversity(win)}
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
# ------------------------------------------------------------------------- blocks
|
| 155 |
+
class CausalSDPA(nn.Module):
|
| 156 |
+
def __init__(self, d: int, heads: int = 4):
|
| 157 |
+
super().__init__()
|
| 158 |
+
self.h = heads
|
| 159 |
+
self.qkv = nn.Linear(d, 3 * d, bias=False)
|
| 160 |
+
self.o = nn.Linear(d, d, bias=False)
|
| 161 |
+
nn.init.orthogonal_(self.qkv.weight); nn.init.orthogonal_(self.o.weight)
|
| 162 |
+
|
| 163 |
+
def forward(self, x):
|
| 164 |
+
B, n, d = x.shape
|
| 165 |
+
q, k, v = self.qkv(x).chunk(3, dim=-1)
|
| 166 |
+
q, k, v = (t.view(B, n, self.h, d // self.h).transpose(1, 2) for t in (q, k, v))
|
| 167 |
+
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
| 168 |
+
return self.o(y.transpose(1, 2).reshape(B, n, d))
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
class CausalHUB(nn.Module):
|
| 172 |
+
"""Causal aleph linear attention: prefix-sum memories over the two K-wide
|
| 173 |
+
halves of the oriented address; 2K never materialized; no selection event."""
|
| 174 |
+
|
| 175 |
+
def __init__(self, d: int, K: int = 32, D: int = 4, tau: float = 0.1):
|
| 176 |
+
super().__init__()
|
| 177 |
+
self.addr = AlephAddress(K, D, tau)
|
| 178 |
+
self.q = nn.Linear(d, D, bias=False)
|
| 179 |
+
self.k = nn.Linear(d, D, bias=False)
|
| 180 |
+
self.v = nn.Linear(d, d, bias=False)
|
| 181 |
+
self.o = nn.Linear(d, d, bias=False)
|
| 182 |
+
for m in (self.q, self.k, self.v, self.o):
|
| 183 |
+
nn.init.orthogonal_(m.weight)
|
| 184 |
+
|
| 185 |
+
def forward(self, x):
|
| 186 |
+
qp, qn = self.addr.oriented(self.q(x)) # (B, n, K)
|
| 187 |
+
kp, kn = self.addr.oriented(self.k(x))
|
| 188 |
+
v = self.v(x) # (B, n, d)
|
| 189 |
+
Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
|
| 190 |
+
Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
|
| 191 |
+
zp = torch.cumsum(kp, dim=1)
|
| 192 |
+
zn = torch.cumsum(kn, dim=1)
|
| 193 |
+
num = torch.einsum("bnk,bnkd->bnd", qp, Sp) + torch.einsum("bnk,bnkd->bnd", qn, Sn)
|
| 194 |
+
den = (qp * zp).sum(-1, keepdim=True) + (qn * zn).sum(-1, keepdim=True)
|
| 195 |
+
return self.o(num / den.clamp_min(1e-12))
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
class MslRelay(nn.Module):
|
| 199 |
+
"""Depth-composition unit (chain-rule probe): multi-slot M_hat read entering
|
| 200 |
+
the trunk as a NEAR-ZERO gated residual (gate init -3.0, sigma~0.047 — theme D:
|
| 201 |
+
geometry enters as a nudge and grows only if it earns gradient)."""
|
| 202 |
+
|
| 203 |
+
def __init__(self, d: int, n_slots: int = 16, K: int = 64):
|
| 204 |
+
super().__init__()
|
| 205 |
+
self.n_slots = n_slots
|
| 206 |
+
self.proj = nn.Linear(d, n_slots * 4, bias=False)
|
| 207 |
+
self.out = nn.Linear(n_slots * 4, d, bias=False)
|
| 208 |
+
nn.init.orthogonal_(self.proj.weight)
|
| 209 |
+
nn.init.orthogonal_(self.out.weight)
|
| 210 |
+
self.addr = AlephAddress(K, 4)
|
| 211 |
+
self.gate = nn.Parameter(torch.tensor(-3.0))
|
| 212 |
+
|
| 213 |
+
def forward(self, x):
|
| 214 |
+
B, n, _ = x.shape
|
| 215 |
+
slots = self.proj(x).view(B, n, self.n_slots, 4)
|
| 216 |
+
m = self.addr.m_hat(slots).reshape(B, n, -1)
|
| 217 |
+
return x + self.gate.sigmoid() * self.out(m)
|
| 218 |
+
|
| 219 |
+
|
| 220 |
+
class Block(nn.Module):
|
| 221 |
+
def __init__(self, d: int, attn: nn.Module):
|
| 222 |
+
super().__init__()
|
| 223 |
+
self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
|
| 224 |
+
self.attn = attn
|
| 225 |
+
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
|
| 226 |
+
|
| 227 |
+
def forward(self, x):
|
| 228 |
+
x = x + self.attn(self.n1(x))
|
| 229 |
+
return x + self.mlp(self.n2(x))
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
class ByteLM(nn.Module):
|
| 233 |
+
def __init__(self, arm: str, d: int = 192, layers: int = 4, block: int = 256,
|
| 234 |
+
K: int = 32, D: int = 4):
|
| 235 |
+
super().__init__()
|
| 236 |
+
# "<arm>_tri" suffix = trigram byte embedding (AlephLM byte_emb x3 lineage):
|
| 237 |
+
# token embedding is the sum of embeddings of bytes t, t-1, t-2.
|
| 238 |
+
self.trigram = arm.endswith("_tri")
|
| 239 |
+
if self.trigram:
|
| 240 |
+
arm = arm[:-4]
|
| 241 |
+
# "_fib" = super-Fibonacci S^3 codebook init (basin test: starts INSIDE
|
| 242 |
+
# the RP^3 attractor; primary observable is init->final geodesic drift).
|
| 243 |
+
self.fib = arm.endswith("_fib")
|
| 244 |
+
if self.fib:
|
| 245 |
+
arm = arm[:-4]
|
| 246 |
+
# "relay*" = stacked addresses in depth: MslRelay after every block.
|
| 247 |
+
# relay -> sdpa trunk + standard head; relay_msl64 -> + addressed head.
|
| 248 |
+
self.use_relay = arm.startswith("relay")
|
| 249 |
+
if arm == "relay":
|
| 250 |
+
arm = "sdpa"
|
| 251 |
+
elif arm == "relay_msl64":
|
| 252 |
+
arm = "addr_msl64"
|
| 253 |
+
self.arm, self.block = arm, block
|
| 254 |
+
self.emb = nn.Embedding(VOCAB, d)
|
| 255 |
+
if self.trigram:
|
| 256 |
+
self.emb1 = nn.Embedding(VOCAB, d)
|
| 257 |
+
self.emb2 = nn.Embedding(VOCAB, d)
|
| 258 |
+
self.pos = nn.Parameter(torch.zeros(1, block, d) + 0.01 * torch.randn(1, block, d))
|
| 259 |
+
mk_attn = (lambda: CausalHUB(d, K, D)) if arm == "hub" else (lambda: CausalSDPA(d))
|
| 260 |
+
self.blocks = nn.ModuleList([Block(d, mk_attn()) for _ in range(layers)])
|
| 261 |
+
if self.use_relay:
|
| 262 |
+
self.relays = nn.ModuleList([MslRelay(d) for _ in range(layers)])
|
| 263 |
+
self.nf = nn.LayerNorm(d)
|
| 264 |
+
if arm == "addr_head":
|
| 265 |
+
self.head_addr = AlephAddress(K, d) # v1: codebook in model dim — COLLAPSED
|
| 266 |
+
self.head = nn.Linear(K, VOCAB, bias=True)
|
| 267 |
+
elif arm in ("addr_d4", "addr_3tau", "addr_mhat"):
|
| 268 |
+
# v2 refinements: LOW-D HOME — learned projection to the native D=4 home
|
| 269 |
+
# before addressing (mirrors the healthy HUB arms), K=64.
|
| 270 |
+
self.head_proj = nn.Linear(d, 4, bias=False)
|
| 271 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 272 |
+
self.head_addr = AlephAddress(64, 4)
|
| 273 |
+
if arm == "addr_d4":
|
| 274 |
+
self.head = nn.Linear(64, VOCAB, bias=True) # w alone, D=4 home
|
| 275 |
+
elif arm == "addr_3tau":
|
| 276 |
+
self.taus = (0.05, 0.1, 0.3) # rule-of-3 strobe
|
| 277 |
+
self.head = nn.Linear(64 * 3, VOCAB, bias=True)
|
| 278 |
+
else: # addr_mhat
|
| 279 |
+
self.head = nn.Linear(4, VOCAB, bias=True) # tightest: M_hat
|
| 280 |
+
elif arm.startswith("addr_msl"):
|
| 281 |
+
# v3: MULTI-SLOT heads — the 16s funnel widening: P parallel D=4 slots
|
| 282 |
+
# over a SHARED codebook. addr_msl consumes the reconstructive M_hat per
|
| 283 |
+
# slot (Px4 dims); addr_msl_w consumes signed w per slot (Px64) — tests
|
| 284 |
+
# whether slot-parallel consumption alone rescues the coefficient path.
|
| 285 |
+
# addr_msl<P> = slot-count dose-response. addr_mslh<P> = HARD sign-code
|
| 286 |
+
# consumption (straight-through M_hard per slot).
|
| 287 |
+
self.hard = arm.startswith("addr_mslh")
|
| 288 |
+
if arm in ("addr_msl", "addr_msl_w"):
|
| 289 |
+
self.n_slots = 16
|
| 290 |
+
else:
|
| 291 |
+
self.n_slots = int(arm[len("addr_mslh" if self.hard else "addr_msl"):])
|
| 292 |
+
self.head_proj = nn.Linear(d, self.n_slots * 4, bias=False)
|
| 293 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 294 |
+
self.head_addr = AlephAddress(
|
| 295 |
+
64, 4, init="fibonacci" if self.fib else "random")
|
| 296 |
+
width = self.n_slots * (64 if arm == "addr_msl_w" else 4)
|
| 297 |
+
self.head = nn.Linear(width, VOCAB, bias=True)
|
| 298 |
+
elif arm == "addr_3tau_mhat":
|
| 299 |
+
# v3: combine the two v2 winners — 3-tau stroboscope + reconstructive read.
|
| 300 |
+
self.head_proj = nn.Linear(d, 4, bias=False)
|
| 301 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 302 |
+
self.head_addr = AlephAddress(64, 4)
|
| 303 |
+
self.taus = (0.05, 0.1, 0.3)
|
| 304 |
+
self.head = nn.Linear(64 * 3 + 4, VOCAB, bias=True)
|
| 305 |
+
else:
|
| 306 |
+
self.head = nn.Linear(d, VOCAB, bias=True)
|
| 307 |
+
self._last_h = None
|
| 308 |
+
|
| 309 |
+
def forward(self, idx):
|
| 310 |
+
x = self.emb(idx)
|
| 311 |
+
if self.trigram: # past-only shifts — causality preserved
|
| 312 |
+
x = x + self.emb1(F.pad(idx, (1, 0), value=0)[:, :-1]) \
|
| 313 |
+
+ self.emb2(F.pad(idx, (2, 0), value=0)[:, :-2])
|
| 314 |
+
x = x + self.pos[:, : idx.shape[1]]
|
| 315 |
+
if self.use_relay:
|
| 316 |
+
for b, r in zip(self.blocks, self.relays):
|
| 317 |
+
x = r(b(x))
|
| 318 |
+
else:
|
| 319 |
+
for b in self.blocks:
|
| 320 |
+
x = b(x)
|
| 321 |
+
h = self.nf(x)
|
| 322 |
+
self._last_h = h.detach()
|
| 323 |
+
if self.arm == "addr_head":
|
| 324 |
+
return self.head(self.head_addr.signed(h))
|
| 325 |
+
if self.arm == "addr_d4":
|
| 326 |
+
return self.head(self.head_addr.signed(self.head_proj(h)))
|
| 327 |
+
if self.arm == "addr_3tau":
|
| 328 |
+
return self.head(self.head_addr.signed_at(self.head_proj(h), self.taus))
|
| 329 |
+
if self.arm == "addr_mhat":
|
| 330 |
+
return self.head(self.head_addr.m_hat(self.head_proj(h)))
|
| 331 |
+
if self.arm.startswith("addr_msl"):
|
| 332 |
+
B, n, _ = h.shape
|
| 333 |
+
slots = self.head_proj(h).view(B, n, self.n_slots, 4)
|
| 334 |
+
if self.arm == "addr_msl_w":
|
| 335 |
+
feats = self.head_addr.signed(slots).reshape(B, n, -1)
|
| 336 |
+
elif getattr(self, "hard", False):
|
| 337 |
+
feats = self.head_addr.m_hard_ste(slots).reshape(B, n, -1)
|
| 338 |
+
else:
|
| 339 |
+
feats = self.head_addr.m_hat(slots).reshape(B, n, -1)
|
| 340 |
+
return self.head(feats)
|
| 341 |
+
if self.arm == "addr_3tau_mhat":
|
| 342 |
+
p = self.head_proj(h)
|
| 343 |
+
feats = torch.cat([self.head_addr.signed_at(p, self.taus),
|
| 344 |
+
self.head_addr.m_hat(p)], dim=-1)
|
| 345 |
+
return self.head(feats)
|
| 346 |
+
return self.head(h)
|
| 347 |
+
|
| 348 |
+
@torch.no_grad()
|
| 349 |
+
def vitals(self) -> dict:
|
| 350 |
+
out = {}
|
| 351 |
+
if self.arm == "hub":
|
| 352 |
+
for i, b in enumerate(self.blocks):
|
| 353 |
+
if self._last_h is not None:
|
| 354 |
+
out[f"L{i}"] = b.attn.addr.vitals(b.attn.q(self._last_h[:2]))
|
| 355 |
+
elif self.arm == "addr_head" and self._last_h is not None:
|
| 356 |
+
out["head"] = self.head_addr.vitals(self._last_h[:2])
|
| 357 |
+
elif self.arm in ("addr_d4", "addr_3tau", "addr_mhat",
|
| 358 |
+
"addr_3tau_mhat") and self._last_h is not None:
|
| 359 |
+
out["head"] = self.head_addr.vitals(self.head_proj(self._last_h[:2]))
|
| 360 |
+
elif self.arm.startswith("addr_msl") and self._last_h is not None:
|
| 361 |
+
slots = self.head_proj(self._last_h[:2])
|
| 362 |
+
out["head"] = self.head_addr.vitals(
|
| 363 |
+
slots.reshape(*slots.shape[:-1], self.n_slots, 4))
|
| 364 |
+
if self.use_relay and self._last_h is not None:
|
| 365 |
+
for i, r in enumerate(self.relays):
|
| 366 |
+
s = r.proj(self._last_h[:2])
|
| 367 |
+
v = r.addr.vitals(s.reshape(*s.shape[:-1], r.n_slots, 4))
|
| 368 |
+
out[f"relay{i}"] = {"gate": round(r.gate.sigmoid().item(), 4),
|
| 369 |
+
"drift": v["drift"],
|
| 370 |
+
"binding_frac": v["binding_frac"],
|
| 371 |
+
"ppl": round(v["aliveness"]["usage_ppl"], 1)}
|
| 372 |
+
return out
|
| 373 |
+
|
| 374 |
+
|
| 375 |
+
# --------------------------------------------------------------------------- data
|
| 376 |
+
def _wikitext_bytes(data_root: str):
|
| 377 |
+
"""wikitext-2-raw as flat uint8 tensors via the HF parquet CDN."""
|
| 378 |
+
from huggingface_hub import hf_hub_download
|
| 379 |
+
import pyarrow.parquet as pq
|
| 380 |
+
|
| 381 |
+
def load(split):
|
| 382 |
+
p = hf_hub_download("Salesforce/wikitext",
|
| 383 |
+
f"wikitext-2-raw-v1/{split}-00000-of-00001.parquet",
|
| 384 |
+
repo_type="dataset", local_dir=data_root)
|
| 385 |
+
text = "".join(pq.read_table(p).column("text").to_pylist())
|
| 386 |
+
return torch.frombuffer(bytearray(text.encode("utf-8")), dtype=torch.uint8).clone()
|
| 387 |
+
|
| 388 |
+
return load("train"), load("validation")
|
| 389 |
+
|
| 390 |
+
|
| 391 |
+
def _batch(data: torch.Tensor, batch: int, block: int, device, g: torch.Generator):
|
| 392 |
+
ix = torch.randint(0, data.numel() - block - 1, (batch,), generator=g)
|
| 393 |
+
x = torch.stack([data[i:i + block] for i in ix]).long().to(device)
|
| 394 |
+
y = torch.stack([data[i + 1:i + block + 1] for i in ix]).long().to(device)
|
| 395 |
+
return x, y
|
| 396 |
+
|
| 397 |
+
|
| 398 |
+
# -------------------------------------------------------------------- train/smoke
|
| 399 |
+
def train(arms=("sdpa", "hub", "addr_head"), steps: int = 2000, batch: int = 32,
|
| 400 |
+
block: int = 256, device: str = "cuda", data_root: str = "./data",
|
| 401 |
+
seed: int = 0, eval_every: int = 500, save: bool = True):
|
| 402 |
+
"""Verdict run — GPU only. Pure Adam wd=0. Reports val bits-per-byte + vitals.
|
| 403 |
+
save=True writes {data_root}/ar_ckpts/{arm}_s{seed}_t{steps}.pt per arm —
|
| 404 |
+
the cultivated codebooks are SPECIMENS for the projective reading instruments."""
|
| 405 |
+
import os
|
| 406 |
+
if device == "cuda" and not torch.cuda.is_available():
|
| 407 |
+
raise RuntimeError("Verdict runs are GPU-only (never CPU-train for accuracy).")
|
| 408 |
+
ckpt_dir = os.path.join(data_root, "ar_ckpts")
|
| 409 |
+
os.makedirs(ckpt_dir, exist_ok=True)
|
| 410 |
+
tr, va = _wikitext_bytes(data_root)
|
| 411 |
+
print(f"data ready: train {tr.numel():,} bytes, val {va.numel():,} bytes", flush=True)
|
| 412 |
+
results = {}
|
| 413 |
+
for arm in arms:
|
| 414 |
+
torch.manual_seed(seed)
|
| 415 |
+
g = torch.Generator().manual_seed(seed)
|
| 416 |
+
model = ByteLM(arm, block=block).to(device)
|
| 417 |
+
n_params = sum(p.numel() for p in model.parameters())
|
| 418 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 419 |
+
for step in range(1, steps + 1):
|
| 420 |
+
x, y = _batch(tr, batch, block, device, g)
|
| 421 |
+
logits = model(x)
|
| 422 |
+
loss = F.cross_entropy(logits.reshape(-1, VOCAB), y.reshape(-1))
|
| 423 |
+
opt.zero_grad(set_to_none=True)
|
| 424 |
+
loss.backward()
|
| 425 |
+
opt.step()
|
| 426 |
+
if step % eval_every == 0 or step == steps:
|
| 427 |
+
model.eval()
|
| 428 |
+
with torch.no_grad():
|
| 429 |
+
losses = []
|
| 430 |
+
for _ in range(20):
|
| 431 |
+
xv, yv = _batch(va, batch, block, device, g)
|
| 432 |
+
lv = F.cross_entropy(model(xv).reshape(-1, VOCAB),
|
| 433 |
+
yv.reshape(-1))
|
| 434 |
+
losses.append(lv.item())
|
| 435 |
+
bpb = sum(losses) / len(losses) / math.log(2)
|
| 436 |
+
print(f"[{arm}] step {step} val_bpb={bpb:.4f} vitals={model.vitals()}",
|
| 437 |
+
flush=True)
|
| 438 |
+
model.train()
|
| 439 |
+
results[arm] = {"val_bpb": bpb, "params": n_params, "vitals": model.vitals()}
|
| 440 |
+
if save:
|
| 441 |
+
path = os.path.join(ckpt_dir, f"{arm}_s{seed}_t{steps}.pt")
|
| 442 |
+
torch.save({"arm": arm, "seed": seed, "steps": steps, "val_bpb": bpb,
|
| 443 |
+
"state_dict": {k: v.cpu() for k, v in
|
| 444 |
+
model.state_dict().items()}}, path)
|
| 445 |
+
print(f"saved specimen: {path}", flush=True)
|
| 446 |
+
print(results, flush=True)
|
| 447 |
+
return results
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
def smoke():
|
| 451 |
+
"""Shapes/parse only — no accuracy claims."""
|
| 452 |
+
x = torch.randint(0, VOCAB, (2, 64))
|
| 453 |
+
for arm in ("sdpa", "hub", "addr_head"):
|
| 454 |
+
m = ByteLM(arm, d=96, layers=2, block=64, K=16)
|
| 455 |
+
logits = m(x)
|
| 456 |
+
assert logits.shape == (2, 64, VOCAB)
|
| 457 |
+
logits.sum().backward()
|
| 458 |
+
# causality check: future byte must not affect past logits
|
| 459 |
+
with torch.no_grad():
|
| 460 |
+
a = m(x)[0, 10]
|
| 461 |
+
x2 = x.clone(); x2[0, 40] = (x2[0, 40] + 7) % 256
|
| 462 |
+
b = m(x2)[0, 10]
|
| 463 |
+
assert torch.allclose(a, b, atol=1e-4), f"{arm} leaks future context"
|
| 464 |
+
print(f"{arm}: OK params={sum(p.numel() for p in m.parameters()):,} "
|
| 465 |
+
f"vitals={m.vitals()}", flush=True)
|
| 466 |
+
print("OK — AR bed smoke passed (verdict run: train() on GPU)", flush=True)
|
| 467 |
+
|
| 468 |
+
|
| 469 |
+
def _in_notebook() -> bool:
|
| 470 |
+
try:
|
| 471 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 472 |
+
return True
|
| 473 |
+
except NameError:
|
| 474 |
+
return False
|
| 475 |
+
|
| 476 |
+
|
| 477 |
+
if __name__ == "__main__":
|
| 478 |
+
if _in_notebook():
|
| 479 |
+
smoke()
|
| 480 |
+
print("Notebook mode: call train(steps=2000) in the next cell (GPU).")
|
| 481 |
+
else:
|
| 482 |
+
import argparse
|
| 483 |
+
ap = argparse.ArgumentParser()
|
| 484 |
+
ap.add_argument("--train", action="store_true")
|
| 485 |
+
ap.add_argument("--steps", type=int, default=2000)
|
| 486 |
+
a, _ = ap.parse_known_args()
|
| 487 |
+
train(steps=a.steps) if a.train else smoke()
|
exp014_gd/article.md
CHANGED
|
@@ -175,18 +175,29 @@ logged as vitals, not deep-read.
|
|
| 175 |
|
| 176 |
## 6. Reproduction
|
| 177 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 178 |
```bash
|
| 179 |
pip install torch --index-url https://download.pytorch.org/whl/cu128
|
| 180 |
-
pip install pyarrow huggingface_hub
|
| 181 |
-
#
|
| 182 |
-
|
| 183 |
-
python -
|
| 184 |
-
|
| 185 |
python build_results.py # re-asserts every claim in this article from the ledger
|
| 186 |
```
|
| 187 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 188 |
`genomes/` holds the 23 per-generation champion books (`champion_{lineage}_t{seed}_g{gen}.pt`)
|
| 189 |
-
— the germline fossil record
|
| 190 |
|
| 191 |
## Citations
|
| 192 |
|
|
|
|
| 175 |
|
| 176 |
## 6. Reproduction
|
| 177 |
|
| 178 |
+
The package is standalone — every code dependency ships in
|
| 179 |
+
[`exp014_gd/`](https://huggingface.co/AbstractPhil/geolip-aleph-differentiation/tree/main/exp014_gd)
|
| 180 |
+
itself (`geolip_vitals.py`, `ar_differentiation_bed.py`,
|
| 181 |
+
`exp013_augmentation_bed.py` for the GPT-2 relays, `read_codebook.py`).
|
| 182 |
+
From inside the folder:
|
| 183 |
+
|
| 184 |
```bash
|
| 185 |
pip install torch --index-url https://download.pytorch.org/whl/cu128
|
| 186 |
+
pip install pyarrow huggingface_hub # + transformers for --mode b2
|
| 187 |
+
python repro.py # CPU smoke: parse + shapes + GPA sanity
|
| 188 |
+
python repro.py --mode tournament --lineage aleph_full # one bracket (GPU, ~20 min)
|
| 189 |
+
python repro.py --mode b1 # cross-size implants (GPU)
|
| 190 |
+
python repro.py --mode b2 --genome genomes/champion_aleph_full_t0_g3.pt
|
| 191 |
python build_results.py # re-asserts every claim in this article from the ledger
|
| 192 |
```
|
| 193 |
|
| 194 |
+
Data lands in `./data` (override with `GEOLIP_DATA`). In Colab, paste
|
| 195 |
+
`geolip_vitals.py`, `ar_differentiation_bed.py`, then
|
| 196 |
+
`exp014_genetic_distillation.py` into cells in that order and call the runners
|
| 197 |
+
directly.
|
| 198 |
+
|
| 199 |
`genomes/` holds the 23 per-generation champion books (`champion_{lineage}_t{seed}_g{gen}.pt`)
|
| 200 |
+
— the germline fossil record: `python read_codebook.py genomes/`.
|
| 201 |
|
| 202 |
## Citations
|
| 203 |
|
exp014_gd/exp013_augmentation_bed.py
ADDED
|
@@ -0,0 +1,506 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""exp013_augmentation_bed.py — augmenting pretrained models with the aleph.
|
| 2 |
+
Three tracks, one file (sequel to exp012's ar_differentiation_bed):
|
| 3 |
+
|
| 4 |
+
A. AUTOREGRESSION FROM CLIP-L: next-token prediction over CLIP-tokenized text,
|
| 5 |
+
reading FROZEN openai/clip-vit-large-patch14 text-tower hidden states.
|
| 6 |
+
FACTOR: extraction layer in {final, penultimate} (the last two layers — the
|
| 7 |
+
penultimate is what diffusion stacks consume). Heads at ~matched params:
|
| 8 |
+
linear | mlp | aleph multi-slot M_hat (P=64, D=4, shared K=64 — the exp012
|
| 9 |
+
certified construction) | sign-code (straight-through).
|
| 10 |
+
B. JOINT-FAILURE PROBES on frozen pooled embeddings (CLIP-L both layers + BERT):
|
| 11 |
+
b1 SPELLING-AR — decode a word's characters from ONLY the head's read of its
|
| 12 |
+
pooled embedding (GATE: linear/mlp must fail <50% exact first);
|
| 13 |
+
b2 ORDER — original-vs-shuffled discrimination (secondary; info may be absent).
|
| 14 |
+
C. GPT-2 (124M) AUGMENTATION: frozen trunk + trainable adapters after every block —
|
| 15 |
+
aleph MslRelay adapters vs param-matched MLP adapters vs frozen baseline;
|
| 16 |
+
gate growth by depth is a first-class readout (exp012 depth-gradient law).
|
| 17 |
+
|
| 18 |
+
Riders: pure Adam wd=0; no contrastive/InfoNCE into address paths; vitals are
|
| 19 |
+
readouts; GPU-only for verdict runs; caches/specimens live OUTSIDE the repo.
|
| 20 |
+
Colab: paste geolip_vitals.py, then ar_differentiation_bed.py, then this file.
|
| 21 |
+
"""
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
import json
|
| 24 |
+
import math
|
| 25 |
+
import os
|
| 26 |
+
import re
|
| 27 |
+
import torch
|
| 28 |
+
import torch.nn as nn
|
| 29 |
+
import torch.nn.functional as F
|
| 30 |
+
|
| 31 |
+
# ---- paste-ahead imports (notebook-safe) -----------------------------------------
|
| 32 |
+
if "anchor_drift" not in globals():
|
| 33 |
+
try:
|
| 34 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 35 |
+
except ImportError:
|
| 36 |
+
_here = globals().get("__file__")
|
| 37 |
+
if _here is None:
|
| 38 |
+
raise ImportError("paste/run geolip_vitals.py first")
|
| 39 |
+
import sys, pathlib
|
| 40 |
+
sys.path.insert(0, str(pathlib.Path(_here).parent))
|
| 41 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 42 |
+
if "AlephAddress" not in globals():
|
| 43 |
+
try:
|
| 44 |
+
from ar_differentiation_bed import AlephAddress, MslRelay, _wikitext_bytes
|
| 45 |
+
except ImportError:
|
| 46 |
+
_here = globals().get("__file__")
|
| 47 |
+
if _here is None:
|
| 48 |
+
raise ImportError("paste/run ar_differentiation_bed.py first")
|
| 49 |
+
from ar_differentiation_bed import AlephAddress, MslRelay, _wikitext_bytes
|
| 50 |
+
|
| 51 |
+
DATA_ROOT = os.environ.get("GEOLIP_DATA", "./data")
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
class SquaredReLU(nn.Module):
|
| 55 |
+
def forward(self, x):
|
| 56 |
+
return F.relu(x) ** 2
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
# =============================================================== head arm family ==
|
| 60 |
+
class HeadArm(nn.Module):
|
| 61 |
+
"""Conditioning transform d_in -> F_OUT (256), then a shared-shape task head.
|
| 62 |
+
Arms: linear | mlp | aleph (multi-slot M_hat) | sign (straight-through M_hard).
|
| 63 |
+
The aleph arms are the exp012-certified construction: P=64 slots x D=4 over one
|
| 64 |
+
shared K=64 codebook."""
|
| 65 |
+
F_OUT = 256
|
| 66 |
+
|
| 67 |
+
def __init__(self, arm: str, d_in: int, mlp_hidden: int = 192):
|
| 68 |
+
# mlp_hidden=192 param-matches the aleph arm at d_in=768:
|
| 69 |
+
# mlp ~ 192*(768+256)+LN ~ 197K vs aleph proj 768*256 + codebook = 196.9K
|
| 70 |
+
super().__init__()
|
| 71 |
+
self.arm = arm
|
| 72 |
+
if arm == "linear":
|
| 73 |
+
self.net = nn.Linear(d_in, self.F_OUT, bias=False)
|
| 74 |
+
elif arm == "mlp":
|
| 75 |
+
self.net = nn.Sequential(nn.Linear(d_in, mlp_hidden), SquaredReLU(),
|
| 76 |
+
nn.LayerNorm(mlp_hidden),
|
| 77 |
+
nn.Linear(mlp_hidden, self.F_OUT))
|
| 78 |
+
elif arm in ("aleph", "sign"):
|
| 79 |
+
self.proj = nn.Linear(d_in, 64 * 4, bias=False)
|
| 80 |
+
nn.init.orthogonal_(self.proj.weight)
|
| 81 |
+
self.addr = AlephAddress(64, 4)
|
| 82 |
+
else:
|
| 83 |
+
raise ValueError(arm)
|
| 84 |
+
|
| 85 |
+
def forward(self, x):
|
| 86 |
+
if self.arm in ("linear", "mlp"):
|
| 87 |
+
return self.net(x)
|
| 88 |
+
slots = self.proj(x).reshape(*x.shape[:-1], 64, 4)
|
| 89 |
+
read = self.addr.m_hard_ste(slots) if self.arm == "sign" \
|
| 90 |
+
else self.addr.m_hat(slots)
|
| 91 |
+
return read.reshape(*x.shape[:-1], 256)
|
| 92 |
+
|
| 93 |
+
@torch.no_grad()
|
| 94 |
+
def vitals(self, x_sample) -> dict:
|
| 95 |
+
if self.arm in ("linear", "mlp"):
|
| 96 |
+
return {}
|
| 97 |
+
slots = self.proj(x_sample).reshape(*x_sample.shape[:-1], 64, 4)
|
| 98 |
+
p, n = self.addr.oriented(slots)
|
| 99 |
+
two_k = torch.cat([p, n], -1).reshape(-1, 128)
|
| 100 |
+
d = anchor_drift(self.addr.codebook, self.addr.home)
|
| 101 |
+
return {"drift": round(d["mean"], 4),
|
| 102 |
+
"binding_frac": round(d["binding_fraction"], 4),
|
| 103 |
+
"usage_ppl": round(axis_aliveness(two_k)["usage_ppl"], 1),
|
| 104 |
+
"paths": path_diversity(two_k.argmax(-1))["unique_hashed"]}
|
| 105 |
+
|
| 106 |
+
def param_count(self):
|
| 107 |
+
return sum(p.numel() for p in self.parameters())
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
# ================================================================== caches ========
|
| 111 |
+
def _wikitext_lines(data_root, min_chars=40, max_lines=None):
|
| 112 |
+
from huggingface_hub import hf_hub_download
|
| 113 |
+
import pyarrow.parquet as pq
|
| 114 |
+
out = {}
|
| 115 |
+
for split, cap in (("train", max_lines), ("validation", None)):
|
| 116 |
+
p = hf_hub_download("Salesforce/wikitext",
|
| 117 |
+
f"wikitext-2-raw-v1/{split}-00000-of-00001.parquet",
|
| 118 |
+
repo_type="dataset", local_dir=data_root)
|
| 119 |
+
lines = [t.strip() for t in pq.read_table(p).column("text").to_pylist()
|
| 120 |
+
if len(t.strip()) >= min_chars]
|
| 121 |
+
out[split] = lines[:cap] if cap else lines
|
| 122 |
+
return out["train"], out["validation"]
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
@torch.no_grad()
|
| 126 |
+
def cache_clip(data_root, n_train=12000, n_val=1500, device="cuda", batch=64):
|
| 127 |
+
"""CLIP-L text tower over wikitext lines; caches BOTH of the last two layers.
|
| 128 |
+
hidden_states[-1] == the final encoder layer output (pre final-LN),
|
| 129 |
+
last_hidden_state == final-LN(final layer). We cache:
|
| 130 |
+
'final' = last_hidden_state (what the projection head consumes),
|
| 131 |
+
'penult' = hidden_states[-2] (the layer diffusion stacks consume).
|
| 132 |
+
Also caches pooled (EOS-position) vectors for both layers, and token ids."""
|
| 133 |
+
from transformers import CLIPTextModel, CLIPTokenizerFast
|
| 134 |
+
path = os.path.join(data_root, "exp013", "clip_cache.pt")
|
| 135 |
+
if os.path.exists(path):
|
| 136 |
+
return torch.load(path, map_location="cpu", weights_only=True)
|
| 137 |
+
os.makedirs(os.path.dirname(path), exist_ok=True)
|
| 138 |
+
tok = CLIPTokenizerFast.from_pretrained("openai/clip-vit-large-patch14")
|
| 139 |
+
model = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14").to(device).eval()
|
| 140 |
+
tr_lines, va_lines = _wikitext_lines(data_root, max_lines=n_train)
|
| 141 |
+
va_lines = va_lines[:n_val]
|
| 142 |
+
def encode(lines):
|
| 143 |
+
H_f, H_p, IDS, EOS = [], [], [], []
|
| 144 |
+
for i in range(0, len(lines), batch):
|
| 145 |
+
enc = tok(lines[i:i + batch], padding="max_length", truncation=True,
|
| 146 |
+
max_length=77, return_tensors="pt").to(device)
|
| 147 |
+
out = model(**enc, output_hidden_states=True)
|
| 148 |
+
H_f.append(out.last_hidden_state.half().cpu())
|
| 149 |
+
H_p.append(out.hidden_states[-2].half().cpu())
|
| 150 |
+
IDS.append(enc.input_ids.cpu())
|
| 151 |
+
EOS.append(enc.input_ids.argmax(-1).cpu()) # EOT id is the max token id
|
| 152 |
+
return (torch.cat(H_f), torch.cat(H_p), torch.cat(IDS), torch.cat(EOS))
|
| 153 |
+
tr = encode(tr_lines)
|
| 154 |
+
va = encode(va_lines)
|
| 155 |
+
blob = {"train": {"final": tr[0], "penult": tr[1], "ids": tr[2], "eos": tr[3]},
|
| 156 |
+
"val": {"final": va[0], "penult": va[1], "ids": va[2], "eos": va[3]},
|
| 157 |
+
"vocab": tok.vocab_size}
|
| 158 |
+
torch.save(blob, path)
|
| 159 |
+
print(f"clip cache: train {tr[0].shape}, val {va[0].shape} -> {path}", flush=True)
|
| 160 |
+
return blob
|
| 161 |
+
|
| 162 |
+
|
| 163 |
+
@torch.no_grad()
|
| 164 |
+
def cache_word_embeddings(data_root, n_words=10000, device="cuda", batch=256):
|
| 165 |
+
"""Pooled embeddings of frequent wikitext words for the spelling probe:
|
| 166 |
+
CLIP-L final + penultimate (EOS-pooled) and BERT (CLS + mean of last layer)."""
|
| 167 |
+
from transformers import (CLIPTextModel, CLIPTokenizerFast,
|
| 168 |
+
BertModel, BertTokenizerFast)
|
| 169 |
+
path = os.path.join(data_root, "exp013", "word_cache.pt")
|
| 170 |
+
if os.path.exists(path):
|
| 171 |
+
return torch.load(path, map_location="cpu", weights_only=True)
|
| 172 |
+
os.makedirs(os.path.dirname(path), exist_ok=True)
|
| 173 |
+
tr_lines, _ = _wikitext_lines(data_root)
|
| 174 |
+
from collections import Counter
|
| 175 |
+
cnt = Counter(w for l in tr_lines for w in re.findall(r"[a-z]{3,12}", l.lower()))
|
| 176 |
+
words = [w for w, _ in cnt.most_common(n_words)]
|
| 177 |
+
blob = {"words": words}
|
| 178 |
+
ct = CLIPTokenizerFast.from_pretrained("openai/clip-vit-large-patch14")
|
| 179 |
+
cm = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14").to(device).eval()
|
| 180 |
+
fin, pen = [], []
|
| 181 |
+
for i in range(0, len(words), batch):
|
| 182 |
+
enc = ct(words[i:i + batch], padding="max_length", truncation=True,
|
| 183 |
+
max_length=77, return_tensors="pt").to(device)
|
| 184 |
+
out = cm(**enc, output_hidden_states=True)
|
| 185 |
+
eos = enc.input_ids.argmax(-1)
|
| 186 |
+
idx = torch.arange(eos.numel(), device=device)
|
| 187 |
+
fin.append(out.last_hidden_state[idx, eos].half().cpu())
|
| 188 |
+
pen.append(out.hidden_states[-2][idx, eos].half().cpu())
|
| 189 |
+
blob["clip_final"], blob["clip_penult"] = torch.cat(fin), torch.cat(pen)
|
| 190 |
+
del cm
|
| 191 |
+
bt = BertTokenizerFast.from_pretrained("bert-base-uncased")
|
| 192 |
+
bm = BertModel.from_pretrained("bert-base-uncased").to(device).eval()
|
| 193 |
+
cls, mean = [], []
|
| 194 |
+
for i in range(0, len(words), batch):
|
| 195 |
+
enc = bt(words[i:i + batch], padding=True, truncation=True,
|
| 196 |
+
max_length=16, return_tensors="pt").to(device)
|
| 197 |
+
out = bm(**enc).last_hidden_state
|
| 198 |
+
m = enc.attention_mask.unsqueeze(-1)
|
| 199 |
+
cls.append(out[:, 0].half().cpu())
|
| 200 |
+
mean.append(((out * m).sum(1) / m.sum(1)).half().cpu())
|
| 201 |
+
blob["bert_cls"], blob["bert_mean"] = torch.cat(cls), torch.cat(mean)
|
| 202 |
+
torch.save(blob, path)
|
| 203 |
+
print(f"word cache: {len(words)} words -> {path}", flush=True)
|
| 204 |
+
return blob
|
| 205 |
+
|
| 206 |
+
|
| 207 |
+
# ============================================================ track A =============
|
| 208 |
+
def track_a(arms=("linear", "mlp", "aleph", "sign"), layers=("final", "penult"),
|
| 209 |
+
steps=1500, batch=64, seed=0, device="cuda",
|
| 210 |
+
data_root=DATA_ROOT, eval_every=500, save=True):
|
| 211 |
+
"""Next-CLIP-token prediction from frozen CLIP-L hidden states."""
|
| 212 |
+
if not torch.cuda.is_available():
|
| 213 |
+
raise RuntimeError("verdict runs are GPU-only")
|
| 214 |
+
blob = cache_clip(data_root, device=device)
|
| 215 |
+
V = blob["vocab"]
|
| 216 |
+
results = {}
|
| 217 |
+
ck_dir = os.path.join(data_root, "exp013", "ckpts")
|
| 218 |
+
os.makedirs(ck_dir, exist_ok=True)
|
| 219 |
+
for layer in layers:
|
| 220 |
+
Htr = blob["train"][layer].float()
|
| 221 |
+
ids_tr = blob["train"]["ids"]
|
| 222 |
+
Hva = blob["val"][layer].float()
|
| 223 |
+
ids_va = blob["val"]["ids"]
|
| 224 |
+
for arm in arms:
|
| 225 |
+
torch.manual_seed(seed)
|
| 226 |
+
g = torch.Generator().manual_seed(seed)
|
| 227 |
+
head = HeadArm(arm, Htr.shape[-1]).to(device)
|
| 228 |
+
out_proj = nn.Linear(HeadArm.F_OUT, V).to(device)
|
| 229 |
+
params = list(head.parameters()) + list(out_proj.parameters())
|
| 230 |
+
opt = torch.optim.Adam(params, lr=3e-4, weight_decay=0.0)
|
| 231 |
+
n_par = sum(p.numel() for p in params)
|
| 232 |
+
for step in range(1, steps + 1):
|
| 233 |
+
ix = torch.randint(0, Htr.shape[0], (batch,), generator=g)
|
| 234 |
+
h = Htr[ix].to(device)
|
| 235 |
+
y = ids_tr[ix].to(device)
|
| 236 |
+
logits = out_proj(head(h[:, :-1]))
|
| 237 |
+
loss = F.cross_entropy(logits.reshape(-1, V), y[:, 1:].reshape(-1))
|
| 238 |
+
opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
|
| 239 |
+
if step % eval_every == 0 or step == steps:
|
| 240 |
+
with torch.no_grad():
|
| 241 |
+
ls = []
|
| 242 |
+
for j in range(0, min(1024, Hva.shape[0]), batch):
|
| 243 |
+
h = Hva[j:j + batch].to(device)
|
| 244 |
+
y = ids_va[j:j + batch].to(device)
|
| 245 |
+
lg = out_proj(head(h[:, :-1]))
|
| 246 |
+
ls.append(F.cross_entropy(
|
| 247 |
+
lg.reshape(-1, V), y[:, 1:].reshape(-1)).item())
|
| 248 |
+
ce = sum(ls) / len(ls)
|
| 249 |
+
vit = head.vitals(Hva[:2, :8].to(device))
|
| 250 |
+
print(f"[A {layer} {arm} s{seed}] step {step} val_ce={ce:.4f} "
|
| 251 |
+
f"params={n_par:,} vitals={vit}", flush=True)
|
| 252 |
+
results[f"{layer}/{arm}/s{seed}"] = {"val_ce": ce, "params": n_par,
|
| 253 |
+
"vitals": vit}
|
| 254 |
+
if save and arm in ("aleph", "sign"):
|
| 255 |
+
torch.save({"track": "A", "layer": layer, "arm": arm, "seed": seed,
|
| 256 |
+
"val_ce": ce, "state_dict": {k: v.cpu() for k, v in
|
| 257 |
+
head.state_dict().items()}},
|
| 258 |
+
os.path.join(ck_dir, f"A_{layer}_{arm}_s{seed}.pt"))
|
| 259 |
+
print(results, flush=True)
|
| 260 |
+
return results
|
| 261 |
+
|
| 262 |
+
|
| 263 |
+
# ============================================================ track B =============
|
| 264 |
+
class CharDecoder(nn.Module):
|
| 265 |
+
"""Tiny GRU char decoder conditioned ONLY on the arm's 256-d read."""
|
| 266 |
+
CHARS = "abcdefghijklmnopqrstuvwxyz"
|
| 267 |
+
def __init__(self, cond_dim=256, hidden=256):
|
| 268 |
+
super().__init__()
|
| 269 |
+
self.V = len(self.CHARS) + 2 # +BOS +EOS
|
| 270 |
+
self.emb = nn.Embedding(self.V, 64)
|
| 271 |
+
self.init = nn.Linear(cond_dim, hidden)
|
| 272 |
+
self.gru = nn.GRU(64, hidden, batch_first=True)
|
| 273 |
+
self.out = nn.Linear(hidden, self.V)
|
| 274 |
+
|
| 275 |
+
def encode_word(self, w):
|
| 276 |
+
return [1] + [2 + self.CHARS.index(c) for c in w] + [0] # BOS..EOS(0)
|
| 277 |
+
|
| 278 |
+
def forward(self, cond, tgt): # tgt: (B, L) int, teacher-forced
|
| 279 |
+
h0 = torch.tanh(self.init(cond)).unsqueeze(0)
|
| 280 |
+
x = self.emb(tgt[:, :-1])
|
| 281 |
+
y, _ = self.gru(x, h0)
|
| 282 |
+
return self.out(y) # predict tgt[:,1:]
|
| 283 |
+
|
| 284 |
+
@torch.no_grad()
|
| 285 |
+
def greedy(self, cond, max_len=14):
|
| 286 |
+
B = cond.shape[0]
|
| 287 |
+
h = torch.tanh(self.init(cond)).unsqueeze(0)
|
| 288 |
+
t = torch.ones(B, 1, dtype=torch.long, device=cond.device)
|
| 289 |
+
done = torch.zeros(B, dtype=torch.bool, device=cond.device)
|
| 290 |
+
outs = []
|
| 291 |
+
for _ in range(max_len):
|
| 292 |
+
y, h = self.gru(self.emb(t), h)
|
| 293 |
+
t = self.out(y).argmax(-1)
|
| 294 |
+
outs.append(t)
|
| 295 |
+
done |= (t.squeeze(1) == 0)
|
| 296 |
+
if done.all():
|
| 297 |
+
break
|
| 298 |
+
return torch.cat(outs, 1)
|
| 299 |
+
|
| 300 |
+
|
| 301 |
+
def track_b1(arms=("linear", "mlp"), substrates=("clip_final", "clip_penult",
|
| 302 |
+
"bert_cls", "bert_mean"), steps=3000, batch=128, seed=0,
|
| 303 |
+
device="cuda", data_root=DATA_ROOT, save=True):
|
| 304 |
+
"""Spelling-AR from pooled embeddings. Run baselines first (the GATE:
|
| 305 |
+
qualify the task only if linear/mlp exact-match < 0.50), then aleph/sign."""
|
| 306 |
+
if not torch.cuda.is_available():
|
| 307 |
+
raise RuntimeError("verdict runs are GPU-only")
|
| 308 |
+
blob = cache_word_embeddings(data_root, device=device)
|
| 309 |
+
words = blob["words"]
|
| 310 |
+
dec_tpl = CharDecoder()
|
| 311 |
+
enc = [dec_tpl.encode_word(w) for w in words]
|
| 312 |
+
L = max(len(e) for e in enc)
|
| 313 |
+
tgt = torch.zeros(len(enc), L, dtype=torch.long)
|
| 314 |
+
for i, e in enumerate(enc):
|
| 315 |
+
tgt[i, :len(e)] = torch.tensor(e)
|
| 316 |
+
g0 = torch.Generator().manual_seed(1234) # fixed split across arms
|
| 317 |
+
perm = torch.randperm(len(words), generator=g0)
|
| 318 |
+
tr_ix, va_ix = perm[:9000], perm[9000:]
|
| 319 |
+
ck_dir = os.path.join(data_root, "exp013", "ckpts")
|
| 320 |
+
os.makedirs(ck_dir, exist_ok=True)
|
| 321 |
+
results = {}
|
| 322 |
+
for sub in substrates:
|
| 323 |
+
E = blob[sub].float()
|
| 324 |
+
for arm in arms:
|
| 325 |
+
torch.manual_seed(seed)
|
| 326 |
+
g = torch.Generator().manual_seed(seed)
|
| 327 |
+
head = HeadArm(arm, E.shape[-1]).to(device)
|
| 328 |
+
dec = CharDecoder().to(device)
|
| 329 |
+
params = list(head.parameters()) + list(dec.parameters())
|
| 330 |
+
opt = torch.optim.Adam(params, lr=1e-3, weight_decay=0.0)
|
| 331 |
+
for step in range(1, steps + 1):
|
| 332 |
+
ix = tr_ix[torch.randint(0, tr_ix.numel(), (batch,), generator=g)]
|
| 333 |
+
cond = head(E[ix].to(device))
|
| 334 |
+
t = tgt[ix].to(device)
|
| 335 |
+
lg = dec(cond, t)
|
| 336 |
+
mask = (t[:, 1:] != 0) | (torch.cumsum(t[:, 1:] == 0, 1) == 1)
|
| 337 |
+
loss = F.cross_entropy(lg[mask], t[:, 1:][mask])
|
| 338 |
+
opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
|
| 339 |
+
with torch.no_grad():
|
| 340 |
+
cond = head(E[va_ix].to(device))
|
| 341 |
+
pred = dec.greedy(cond)
|
| 342 |
+
t = tgt[va_ix, 1:].to(device)
|
| 343 |
+
n = min(pred.shape[1], t.shape[1])
|
| 344 |
+
pad_ok = torch.ones_like(t[:, :n], dtype=torch.bool)
|
| 345 |
+
seen_eos = torch.cumsum(t[:, :n] == 0, 1) > 0
|
| 346 |
+
match = ((pred[:, :n] == t[:, :n]) | seen_eos).all(-1)
|
| 347 |
+
exact = match.float().mean().item()
|
| 348 |
+
vit = head.vitals(E[va_ix[:16]].to(device))
|
| 349 |
+
print(f"[B1 {sub} {arm} s{seed}] exact={exact:.4f} vitals={vit}", flush=True)
|
| 350 |
+
results[f"{sub}/{arm}/s{seed}"] = {"exact": exact, "vitals": vit}
|
| 351 |
+
if save and arm in ("aleph", "sign"):
|
| 352 |
+
torch.save({"track": "B1", "sub": sub, "arm": arm, "seed": seed,
|
| 353 |
+
"exact": exact, "state_dict": {k: v.cpu() for k, v in
|
| 354 |
+
head.state_dict().items()}},
|
| 355 |
+
os.path.join(ck_dir, f"B1_{sub}_{arm}_s{seed}.pt"))
|
| 356 |
+
print(results, flush=True)
|
| 357 |
+
return results
|
| 358 |
+
|
| 359 |
+
|
| 360 |
+
# ============================================================ track C =============
|
| 361 |
+
class MLPAdapter(nn.Module):
|
| 362 |
+
"""Param-matched plain adapter (the ablation twin of MslRelay).
|
| 363 |
+
hidden=64 matches MslRelay at d=768 (2*768*64=98.3K vs 98.6K incl codebook).
|
| 364 |
+
Output layer ZERO-INIT (standard adapter stabilization — the first version
|
| 365 |
+
diverged at lr 1e-3 with random init; the aleph relay needed no such aid,
|
| 366 |
+
which is itself a datapoint, but the control gets its best shot)."""
|
| 367 |
+
def __init__(self, d, hidden=64):
|
| 368 |
+
super().__init__()
|
| 369 |
+
out = nn.Linear(hidden, d)
|
| 370 |
+
nn.init.zeros_(out.weight)
|
| 371 |
+
nn.init.zeros_(out.bias)
|
| 372 |
+
self.net = nn.Sequential(nn.Linear(d, hidden), SquaredReLU(), out)
|
| 373 |
+
self.gate = nn.Parameter(torch.tensor(-3.0))
|
| 374 |
+
|
| 375 |
+
def forward(self, x):
|
| 376 |
+
return x + self.gate.sigmoid() * self.net(x)
|
| 377 |
+
|
| 378 |
+
|
| 379 |
+
class _BlockWithAdapter(nn.Module):
|
| 380 |
+
def __init__(self, block, adapter):
|
| 381 |
+
super().__init__()
|
| 382 |
+
self.block, self.adapter = block, adapter
|
| 383 |
+
|
| 384 |
+
def forward(self, *a, **k):
|
| 385 |
+
out = self.block(*a, **k)
|
| 386 |
+
if isinstance(out, tuple):
|
| 387 |
+
return (self.adapter(out[0]),) + out[1:]
|
| 388 |
+
return self.adapter(out)
|
| 389 |
+
|
| 390 |
+
|
| 391 |
+
def track_c(arms=("frozen", "aleph", "mlp"), steps=1500, batch=8, block=256,
|
| 392 |
+
seed=0, device="cuda", data_root=DATA_ROOT,
|
| 393 |
+
eval_every=500, save=True):
|
| 394 |
+
"""GPT-2 124M frozen; adapters after every block; train adapters only."""
|
| 395 |
+
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
|
| 396 |
+
if not torch.cuda.is_available():
|
| 397 |
+
raise RuntimeError("verdict runs are GPU-only")
|
| 398 |
+
tok = GPT2TokenizerFast.from_pretrained("gpt2")
|
| 399 |
+
tr_lines, va_lines = _wikitext_lines(data_root)
|
| 400 |
+
def to_stream(lines):
|
| 401 |
+
ids = tok("\n\n".join(lines), return_tensors="pt").input_ids[0]
|
| 402 |
+
return ids
|
| 403 |
+
stream_tr = to_stream(tr_lines[:8000])
|
| 404 |
+
stream_va = to_stream(va_lines[:1000])
|
| 405 |
+
ck_dir = os.path.join(data_root, "exp013", "ckpts")
|
| 406 |
+
os.makedirs(ck_dir, exist_ok=True)
|
| 407 |
+
results = {}
|
| 408 |
+
for arm in arms:
|
| 409 |
+
torch.manual_seed(seed)
|
| 410 |
+
g = torch.Generator().manual_seed(seed)
|
| 411 |
+
model = GPT2LMHeadModel.from_pretrained("gpt2").to(device)
|
| 412 |
+
for p in model.parameters():
|
| 413 |
+
p.requires_grad_(False)
|
| 414 |
+
adapters = []
|
| 415 |
+
if arm != "frozen":
|
| 416 |
+
d = model.config.n_embd
|
| 417 |
+
for i, blk in enumerate(model.transformer.h):
|
| 418 |
+
ad = (MslRelay(d) if arm == "aleph" else MLPAdapter(d)).to(device)
|
| 419 |
+
model.transformer.h[i] = _BlockWithAdapter(blk, ad)
|
| 420 |
+
adapters.append(ad)
|
| 421 |
+
params = [p for ad in adapters for p in ad.parameters()]
|
| 422 |
+
n_par = sum(p.numel() for p in params)
|
| 423 |
+
opt = torch.optim.Adam(params, lr=1e-3, weight_decay=0.0)
|
| 424 |
+
else:
|
| 425 |
+
params, n_par = [], 0
|
| 426 |
+
def eval_ppl():
|
| 427 |
+
model.eval()
|
| 428 |
+
with torch.no_grad():
|
| 429 |
+
ls = []
|
| 430 |
+
for j in range(0, stream_va.numel() - block - 1, block * 4):
|
| 431 |
+
x = stream_va[j:j + block].unsqueeze(0).to(device)
|
| 432 |
+
out = model(x, labels=x)
|
| 433 |
+
ls.append(out.loss.item())
|
| 434 |
+
model.train()
|
| 435 |
+
return math.exp(sum(ls) / len(ls))
|
| 436 |
+
if arm == "frozen":
|
| 437 |
+
ppl = eval_ppl()
|
| 438 |
+
print(f"[C frozen] ppl={ppl:.3f}", flush=True)
|
| 439 |
+
results["frozen"] = {"ppl": ppl}
|
| 440 |
+
continue
|
| 441 |
+
for step in range(1, steps + 1):
|
| 442 |
+
ix = torch.randint(0, stream_tr.numel() - block - 1, (batch,), generator=g)
|
| 443 |
+
x = torch.stack([stream_tr[i:i + block] for i in ix]).to(device)
|
| 444 |
+
loss = model(x, labels=x).loss
|
| 445 |
+
opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
|
| 446 |
+
if step % eval_every == 0 or step == steps:
|
| 447 |
+
ppl = eval_ppl()
|
| 448 |
+
gates = [round(ad.gate.sigmoid().item(), 4) for ad in adapters]
|
| 449 |
+
vit = {}
|
| 450 |
+
if arm == "aleph":
|
| 451 |
+
drifts = [round(anchor_drift(ad.addr.codebook, ad.addr.home)
|
| 452 |
+
["mean"], 3) for ad in adapters]
|
| 453 |
+
vit = {"drift_by_depth": drifts}
|
| 454 |
+
print(f"[C {arm} s{seed}] step {step} ppl={ppl:.3f} "
|
| 455 |
+
f"params={n_par:,} gates={gates} {vit}", flush=True)
|
| 456 |
+
results[f"{arm}/s{seed}"] = {"ppl": ppl, "params": n_par, "gates": gates,
|
| 457 |
+
**vit}
|
| 458 |
+
if save and arm == "aleph":
|
| 459 |
+
torch.save({"track": "C", "arm": arm, "seed": seed, "ppl": ppl,
|
| 460 |
+
"state_dict": {f"relay{i}.{k}": v.cpu()
|
| 461 |
+
for i, ad in enumerate(adapters)
|
| 462 |
+
for k, v in ad.state_dict().items()}},
|
| 463 |
+
os.path.join(ck_dir, f"C_{arm}_s{seed}.pt"))
|
| 464 |
+
print(results, flush=True)
|
| 465 |
+
return results
|
| 466 |
+
|
| 467 |
+
|
| 468 |
+
# ================================================================ smoke ===========
|
| 469 |
+
def smoke():
|
| 470 |
+
"""Shapes/parse only — no substrates, no training."""
|
| 471 |
+
for arm in ("linear", "mlp", "aleph", "sign"):
|
| 472 |
+
h = HeadArm(arm, 768)
|
| 473 |
+
y = h(torch.randn(2, 10, 768))
|
| 474 |
+
assert y.shape == (2, 10, 256)
|
| 475 |
+
y.sum().backward()
|
| 476 |
+
print(arm, "OK", f"{h.param_count():,}", h.vitals(torch.randn(2, 4, 768)))
|
| 477 |
+
dec = CharDecoder()
|
| 478 |
+
t = torch.tensor([dec.encode_word("hello") + [0] * 3,
|
| 479 |
+
dec.encode_word("worlds") + [0] * 2])
|
| 480 |
+
lg = dec(torch.randn(2, 256), t)
|
| 481 |
+
assert lg.shape[:2] == (2, t.shape[1] - 1)
|
| 482 |
+
print("decoder OK; greedy:", dec.greedy(torch.randn(2, 256)).shape)
|
| 483 |
+
ad = MLPAdapter(768)
|
| 484 |
+
assert ad(torch.randn(2, 4, 768)).shape == (2, 4, 768)
|
| 485 |
+
print("adapter OK — exp013 smoke passed (caches+tracks need GPU+transformers)")
|
| 486 |
+
|
| 487 |
+
|
| 488 |
+
def _in_notebook():
|
| 489 |
+
try:
|
| 490 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 491 |
+
return True
|
| 492 |
+
except NameError:
|
| 493 |
+
return False
|
| 494 |
+
|
| 495 |
+
|
| 496 |
+
if __name__ == "__main__":
|
| 497 |
+
if _in_notebook():
|
| 498 |
+
smoke()
|
| 499 |
+
print("Notebook: cache_clip()/cache_word_embeddings() then "
|
| 500 |
+
"track_b1() gate -> track_a() -> track_c().")
|
| 501 |
+
else:
|
| 502 |
+
import argparse
|
| 503 |
+
ap = argparse.ArgumentParser()
|
| 504 |
+
ap.add_argument("--track", default="smoke")
|
| 505 |
+
a, _ = ap.parse_known_args()
|
| 506 |
+
{"smoke": smoke, "a": track_a, "b1": track_b1, "c": track_c}[a.track]()
|
exp014_gd/exp014_genetic_distillation.py
CHANGED
|
@@ -44,7 +44,7 @@ if "ByteLM" not in globals():
|
|
| 44 |
except ImportError:
|
| 45 |
raise ImportError("paste/run ar_differentiation_bed.py first")
|
| 46 |
|
| 47 |
-
DATA_ROOT =
|
| 48 |
EXP_DIR = os.path.join(DATA_ROOT, "exp014")
|
| 49 |
|
| 50 |
|
|
@@ -490,6 +490,26 @@ if __name__ == "__main__":
|
|
| 490 |
else:
|
| 491 |
import argparse
|
| 492 |
ap = argparse.ArgumentParser()
|
| 493 |
-
ap.add_argument("--mode", default="smoke"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 494 |
a, _ = ap.parse_known_args()
|
| 495 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
except ImportError:
|
| 45 |
raise ImportError("paste/run ar_differentiation_bed.py first")
|
| 46 |
|
| 47 |
+
DATA_ROOT = os.environ.get("GEOLIP_DATA", "./data")
|
| 48 |
EXP_DIR = os.path.join(DATA_ROOT, "exp014")
|
| 49 |
|
| 50 |
|
|
|
|
| 490 |
else:
|
| 491 |
import argparse
|
| 492 |
ap = argparse.ArgumentParser()
|
| 493 |
+
ap.add_argument("--mode", default="smoke",
|
| 494 |
+
choices=["smoke", "tournament", "b1", "b2"])
|
| 495 |
+
ap.add_argument("--lineage", default="aleph_full",
|
| 496 |
+
help="tournament lineage: aleph_flat|aleph_full|"
|
| 497 |
+
"aleph_weights|aleph_tree|mlp_kd|no_inherit")
|
| 498 |
+
ap.add_argument("--seed", type=int, default=0)
|
| 499 |
+
ap.add_argument("--steps", type=int, default=2000)
|
| 500 |
+
ap.add_argument("--genome", default="genomes/champion_aleph_full_t0_g3.pt",
|
| 501 |
+
help="donor genome .pt for --mode b1/b2 (uses its 'flat' book)")
|
| 502 |
a, _ = ap.parse_known_args()
|
| 503 |
+
if a.mode == "smoke":
|
| 504 |
+
smoke()
|
| 505 |
+
elif a.mode == "tournament":
|
| 506 |
+
run_tournament(a.lineage, steps=a.steps, seed=a.seed)
|
| 507 |
+
elif a.mode == "b1":
|
| 508 |
+
donor = (torch.load(a.genome, map_location="cpu")["flat"]
|
| 509 |
+
if os.path.exists(a.genome) else None)
|
| 510 |
+
run_b1(steps=a.steps, seed=a.seed, donor_book=donor,
|
| 511 |
+
tag=os.path.basename(a.genome) if donor is not None
|
| 512 |
+
else "small_cultivated")
|
| 513 |
+
elif a.mode == "b2":
|
| 514 |
+
donor = torch.load(a.genome, map_location="cpu")["flat"]
|
| 515 |
+
run_b2(donor, seed=a.seed, tag=os.path.basename(a.genome))
|
exp014_gd/geolip_vitals.py
ADDED
|
@@ -0,0 +1,184 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""geolip_vitals.py — shared diagnostic harness for the GeoLIP aleph experiments.
|
| 2 |
+
ALL functions are READOUTS: no gradients, no losses. CV is a readout, never a
|
| 3 |
+
force. Addressing is judged by drift->0.29154 and CV->0.20, never by recon cosine
|
| 4 |
+
(judgment criteria per the aleph-void article: https://huggingface.co/blog/AbstractPhil/geometric-vocabulary-patchwork-aleph-void).
|
| 5 |
+
|
| 6 |
+
Vitals provided:
|
| 7 |
+
anchor_drift — geodesic drift of anchors from init; binding fraction @0.29154
|
| 8 |
+
pentachoron_cv — CM 4-volume CV over random 5-row subsets (geovocab2 import)
|
| 9 |
+
axis_aliveness — oriented-address usage: axes alive, hppl, collapse flag
|
| 10 |
+
gate_stats — gate means vs the 0.012-0.03 band
|
| 11 |
+
path_diversity — unique-path counting, FIXED high-bits hash (low-16 bug is the
|
| 12 |
+
retracted artifact — never use the low bits)
|
| 13 |
+
grad_norm_spread — gradient democracy monitor (orders-of-magnitude spread)
|
| 14 |
+
CVScreen — CV@1000-batch early band screen (<0.30 LOW / .35-.50 MID / >.80 HIGH)
|
| 15 |
+
|
| 16 |
+
Smoke on a torch-capable env: python geolip_vitals.py
|
| 17 |
+
"""
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
import math
|
| 20 |
+
import torch
|
| 21 |
+
|
| 22 |
+
BINDING = 0.29154 # radians; the binding/separation constant
|
| 23 |
+
CV_BAND = (0.13, 0.30) # CM CV band (discovery_catalog #4)
|
| 24 |
+
GATE_BAND = (0.012, 0.03) # live invariant candidate (acd_campaign)
|
| 25 |
+
KNUTH32 = 2654435761
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
# ----------------------------------------------------------------------------- drift
|
| 29 |
+
@torch.no_grad()
|
| 30 |
+
def anchor_drift(current: torch.Tensor, init: torch.Tensor, tol: float = 0.05) -> dict:
|
| 31 |
+
"""Geodesic drift (radians) of each row of `current` from its row in `init`,
|
| 32 |
+
both row-normalized. Returns mean/std/per-row drift and the fraction of rows
|
| 33 |
+
within +/-tol of BINDING (the GLFM '46%' readout)."""
|
| 34 |
+
a = torch.nn.functional.normalize(current.float(), dim=-1)
|
| 35 |
+
b = torch.nn.functional.normalize(init.float(), dim=-1)
|
| 36 |
+
cos = (a * b).sum(-1).clamp(-1.0, 1.0)
|
| 37 |
+
drift = torch.arccos(cos)
|
| 38 |
+
frac = ((drift - BINDING).abs() <= tol).float().mean()
|
| 39 |
+
return {"mean": drift.mean().item(), "std": drift.std().item(),
|
| 40 |
+
"per_row": drift, "binding_fraction": frac.item()}
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
# -------------------------------------------------------------------------------- cv
|
| 44 |
+
@torch.no_grad()
|
| 45 |
+
def pentachoron_cv(rows: torch.Tensor, n_samples: int = 200,
|
| 46 |
+
generator: torch.Generator | None = None) -> float:
|
| 47 |
+
"""CV (std/mean) of Cayley-Menger 4-simplex volumes over n_samples random
|
| 48 |
+
5-row subsets. CM comes from geovocab2 — never inlined (program rule);
|
| 49 |
+
fp32 det only. Rows are row-normalized before measurement."""
|
| 50 |
+
try:
|
| 51 |
+
from geovocab2.formula.cayley_menger import simplex_volume # noqa: F401
|
| 52 |
+
except Exception as e: # pragma: no cover
|
| 53 |
+
raise ImportError(
|
| 54 |
+
"pentachoron_cv requires geovocab2 (CM must be imported, never inlined "
|
| 55 |
+
"— program rule). Install/point PYTHONPATH at geovocab2; expected "
|
| 56 |
+
"geovocab2.formula.cayley_menger.simplex_volume(points fp32)."
|
| 57 |
+
) from e
|
| 58 |
+
x = torch.nn.functional.normalize(rows.float(), dim=-1)
|
| 59 |
+
n = x.shape[0]
|
| 60 |
+
if n < 5:
|
| 61 |
+
raise ValueError(f"pentachoron_cv needs >=5 rows, got {n}")
|
| 62 |
+
g = generator or torch.Generator(device="cpu").manual_seed(0)
|
| 63 |
+
vols = []
|
| 64 |
+
for _ in range(n_samples):
|
| 65 |
+
idx = torch.randperm(n, generator=g)[:5]
|
| 66 |
+
vols.append(simplex_volume(x[idx].to(torch.float32)))
|
| 67 |
+
v = torch.stack([torch.as_tensor(v, dtype=torch.float32) for v in vols])
|
| 68 |
+
return (v.std() / v.mean().clamp_min(1e-12)).item()
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
# ------------------------------------------------------------------------- aliveness
|
| 72 |
+
@torch.no_grad()
|
| 73 |
+
def axis_aliveness(oriented_weights: torch.Tensor, alive_thresh: float = 1e-3) -> dict:
|
| 74 |
+
"""`oriented_weights`: (..., 2K) nonnegative oriented-softmax address rows
|
| 75 |
+
(sum to 1 on the last dim). Returns axes-alive count, mean-usage perplexity
|
| 76 |
+
(hppl analogue; healthy hosted reference 125-126/128), and a collapse flag.
|
| 77 |
+
Reference behavior: near-uniform aliveness at div_weight=0 (discovery #22)."""
|
| 78 |
+
w = oriented_weights.reshape(-1, oriented_weights.shape[-1]).float()
|
| 79 |
+
usage = w.mean(0)
|
| 80 |
+
usage = usage / usage.sum().clamp_min(1e-12)
|
| 81 |
+
# an axis is alive if its mean usage exceeds alive_thresh x the uniform share
|
| 82 |
+
alive = int((usage > alive_thresh * (1.0 / usage.numel())).sum())
|
| 83 |
+
ent = -(usage.clamp_min(1e-12) * usage.clamp_min(1e-12).log()).sum()
|
| 84 |
+
ppl = float(ent.exp())
|
| 85 |
+
return {"axes_total": usage.numel(), "axes_alive": alive, "usage_ppl": ppl,
|
| 86 |
+
"collapsed": ppl < 0.05 * usage.numel()}
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
# ------------------------------------------------------------------------------ gates
|
| 90 |
+
@torch.no_grad()
|
| 91 |
+
def gate_stats(gates: torch.Tensor) -> dict:
|
| 92 |
+
"""Gate values (post-sigmoid/clamp). Reports mean and whether it sits in the
|
| 93 |
+
0.012-0.03 band (read-only — the band is a candidate invariant, never a target)."""
|
| 94 |
+
g = gates.float().flatten()
|
| 95 |
+
m = g.mean().item()
|
| 96 |
+
return {"mean": m, "std": g.std().item(),
|
| 97 |
+
"in_band": GATE_BAND[0] <= m <= GATE_BAND[1]}
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
# ------------------------------------------------------------------------------ paths
|
| 101 |
+
@torch.no_grad()
|
| 102 |
+
def path_diversity(ids: torch.Tensor) -> dict:
|
| 103 |
+
"""Unique-path counting with the FIXED multiplicative hash:
|
| 104 |
+
((ids * 2654435761) % 2^32) >> 16 — Knuth needs the HIGH bits; the low-16
|
| 105 |
+
variant produced a retracted ~1,500 path ceiling in a prior campaign.
|
| 106 |
+
`ids`: integer tensor, one composed path id per row (any shape)."""
|
| 107 |
+
x = ids.reshape(-1).to(torch.int64)
|
| 108 |
+
hashed = ((x * KNUTH32) % (1 << 32)) >> 16
|
| 109 |
+
return {"n": int(x.numel()),
|
| 110 |
+
"unique_raw": int(torch.unique(x).numel()),
|
| 111 |
+
"unique_hashed": int(torch.unique(hashed).numel())}
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
@torch.no_grad()
|
| 115 |
+
def compose_path_ids(stage_indices: list[torch.Tensor], radix: int) -> torch.Tensor:
|
| 116 |
+
"""Compose per-stage discrete indices (each (...,) int in [0, radix)) into a
|
| 117 |
+
single path id, positional base-`radix` — construction, not hashing."""
|
| 118 |
+
out = torch.zeros_like(stage_indices[0], dtype=torch.int64)
|
| 119 |
+
for s in stage_indices:
|
| 120 |
+
out = out * radix + s.to(torch.int64)
|
| 121 |
+
return out
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
# --------------------------------------------------------------------- grad democracy
|
| 125 |
+
@torch.no_grad()
|
| 126 |
+
def grad_norm_spread(groups: dict[str, list[torch.nn.Parameter]]) -> dict:
|
| 127 |
+
"""Gradient-democracy monitor. `groups`: name -> params of one parallel member
|
| 128 |
+
(tower/expert). Reports per-group grad norms and the orders-of-magnitude spread.
|
| 129 |
+
Reference: unequalized heterogeneous towers spread ~20 orders (fibonacci dead at
|
| 130 |
+
2.25e-21 under helix); equalized ~0.0 (geofractal gradient-democracy result)."""
|
| 131 |
+
norms = {}
|
| 132 |
+
for name, params in groups.items():
|
| 133 |
+
gs = [p.grad for p in params if p.grad is not None]
|
| 134 |
+
norms[name] = float(torch.sqrt(sum((g.float() ** 2).sum() for g in gs)).item()) \
|
| 135 |
+
if gs else 0.0
|
| 136 |
+
vals = [v for v in norms.values() if v > 0]
|
| 137 |
+
spread = (math.log10(max(vals)) - math.log10(min(vals))) if len(vals) >= 2 else 0.0
|
| 138 |
+
return {"norms": norms, "spread_orders": spread, "dead": [k for k, v in norms.items() if v == 0.0]}
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
# ----------------------------------------------------------------------------- screen
|
| 142 |
+
class CVScreen:
|
| 143 |
+
"""CV@N early band screen (tri-band ft1): record pentachoron CV at `step_mark`
|
| 144 |
+
batches; classify <0.30 LOW / 0.35-0.50 MID / >0.80 HIGH. Turns ~2h/config
|
| 145 |
+
into ~7min. Readout only."""
|
| 146 |
+
def __init__(self, step_mark: int = 1000):
|
| 147 |
+
self.step_mark = step_mark
|
| 148 |
+
self.recorded: float | None = None
|
| 149 |
+
|
| 150 |
+
def maybe_record(self, step: int, rows: torch.Tensor) -> float | None:
|
| 151 |
+
if self.recorded is None and step >= self.step_mark:
|
| 152 |
+
self.recorded = pentachoron_cv(rows)
|
| 153 |
+
return self.recorded
|
| 154 |
+
|
| 155 |
+
@property
|
| 156 |
+
def band(self) -> str | None:
|
| 157 |
+
c = self.recorded
|
| 158 |
+
if c is None:
|
| 159 |
+
return None
|
| 160 |
+
if c < 0.30:
|
| 161 |
+
return "LOW"
|
| 162 |
+
if 0.35 <= c <= 0.50:
|
| 163 |
+
return "MID"
|
| 164 |
+
if c > 0.80:
|
| 165 |
+
return "HIGH"
|
| 166 |
+
return "BETWEEN"
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
# ------------------------------------------------------------------------------ smoke
|
| 170 |
+
if __name__ == "__main__": # shapes/parse smoke ONLY — no training, ever.
|
| 171 |
+
g = torch.Generator().manual_seed(0)
|
| 172 |
+
K, D = 64, 4
|
| 173 |
+
init = torch.nn.functional.normalize(torch.randn(K, D, generator=g), dim=-1)
|
| 174 |
+
cur = torch.nn.functional.normalize(init + 0.29 * torch.randn(K, D, generator=g), dim=-1)
|
| 175 |
+
print("drift:", {k: v for k, v in anchor_drift(cur, init).items() if k != "per_row"})
|
| 176 |
+
w = torch.softmax(torch.randn(32, 2 * K, generator=g), dim=-1)
|
| 177 |
+
print("aliveness:", axis_aliveness(w))
|
| 178 |
+
print("gates:", gate_stats(torch.full((8,), 0.024)))
|
| 179 |
+
ids = compose_path_ids([torch.randint(0, 16, (4096,), generator=g) for _ in range(4)], 16)
|
| 180 |
+
print("paths:", path_diversity(ids))
|
| 181 |
+
lin = torch.nn.Linear(8, 8)
|
| 182 |
+
lin(torch.randn(4, 8)).sum().backward()
|
| 183 |
+
print("democracy:", grad_norm_spread({"a": list(lin.parameters())}))
|
| 184 |
+
print("OK — vitals smoke passed (pentachoron_cv needs geovocab2; run on GPU env)")
|
exp014_gd/read_codebook.py
ADDED
|
@@ -0,0 +1,159 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""read_codebook.py — projective reading of cultivated aleph codebooks.
|
| 2 |
+
Antipodal-collapse extraction on trained codebooks + projective statistics on
|
| 3 |
+
RP^(D-1) — applied to the exp012 AR-bed specimens.
|
| 4 |
+
|
| 5 |
+
Recipe per the Polygonal Omega article (geometric-tri-band-ft2): collapse = (row_i - row_j)/2 normalized for each MUTUAL-STRONGEST
|
| 6 |
+
pair with cos < -0.9 — "a deterministic tensor operation," not clustering.
|
| 7 |
+
Projective metric ALWAYS arccos|<a,b>| (metric-alignment rule, reading-voids-ft1).
|
| 8 |
+
D=4 scope is the validated regime (D=5 walked back; axis count grows with D).
|
| 9 |
+
|
| 10 |
+
Readouts per specimen:
|
| 11 |
+
pairs / n_axes / unpaired — antipodal structure
|
| 12 |
+
proj_angle mean vs uniform baseline, deviation — near-uniform RP^(D-1)?
|
| 13 |
+
drift from home + binding fraction @0.29154 — cultivation record
|
| 14 |
+
erank of the axis set — spectral occupancy
|
| 15 |
+
verdict: PROJECTIVE-CLEAN (|dev|<0.05, util>0.95, secondary pairs<=3) /
|
| 16 |
+
-MOSTLY / STRUCTURED / DEGENERATE (per Polygonal Omega thresholds)
|
| 17 |
+
|
| 18 |
+
Usage (terminal): python read_codebook.py <ckpt_or_dir> [more paths...]
|
| 19 |
+
Colab: paste geolip_vitals.py cell first (optional), then this file, then
|
| 20 |
+
read_all(r"/content/data/ar_ckpts").
|
| 21 |
+
"""
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
import math
|
| 24 |
+
import sys
|
| 25 |
+
import torch
|
| 26 |
+
import torch.nn.functional as F
|
| 27 |
+
|
| 28 |
+
BINDING = 0.29154
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
@torch.no_grad()
|
| 32 |
+
def antipodal_collapse(codebook: torch.Tensor, thresh: float = -0.9) -> dict:
|
| 33 |
+
"""Mutual-strongest antipodal pairing + collapse to axes on RP^(D-1)."""
|
| 34 |
+
A = F.normalize(codebook.float(), dim=-1)
|
| 35 |
+
K = A.shape[0]
|
| 36 |
+
cos = A @ A.T
|
| 37 |
+
cos.fill_diagonal_(2.0) # exclude self from minima
|
| 38 |
+
nearest_neg = cos.argmin(dim=-1) # most-antipodal partner
|
| 39 |
+
pairs = []
|
| 40 |
+
used = set()
|
| 41 |
+
for i in range(K):
|
| 42 |
+
j = int(nearest_neg[i])
|
| 43 |
+
if i < j and int(nearest_neg[j]) == i and cos[i, j] < thresh:
|
| 44 |
+
pairs.append((i, j))
|
| 45 |
+
used.update((i, j))
|
| 46 |
+
axes = [F.normalize((A[i] - A[j]) / 2.0, dim=-1) for i, j in pairs]
|
| 47 |
+
axes += [A[i] for i in range(K) if i not in used] # unpaired rows as axes
|
| 48 |
+
axes = torch.stack(axes) if axes else A[:0]
|
| 49 |
+
# sign-canon onto RP: first nonzero coordinate positive
|
| 50 |
+
for r in range(axes.shape[0]):
|
| 51 |
+
nz = torch.nonzero(axes[r].abs() > 1e-8)
|
| 52 |
+
if nz.numel() and axes[r, nz[0, 0]] < 0:
|
| 53 |
+
axes[r] = -axes[r]
|
| 54 |
+
return {"pairs": len(pairs), "n_axes": axes.shape[0],
|
| 55 |
+
"unpaired": K - 2 * len(pairs), "axes": axes}
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
@torch.no_grad()
|
| 59 |
+
def projective_stats(axes: torch.Tensor, n_baseline: int = 20000,
|
| 60 |
+
seed: int = 0) -> dict:
|
| 61 |
+
"""Mean projective angle arccos|<a,b>| vs a uniform-RP baseline at same (n, D)."""
|
| 62 |
+
n, D = axes.shape
|
| 63 |
+
if n < 2:
|
| 64 |
+
return {"proj_angle_mean": None, "uniform_baseline": None,
|
| 65 |
+
"deviation": None, "erank": None}
|
| 66 |
+
def mean_angle(rows):
|
| 67 |
+
c = (rows @ rows.T).abs().clamp(max=1.0)
|
| 68 |
+
iu = torch.triu_indices(rows.shape[0], rows.shape[0], offset=1)
|
| 69 |
+
return torch.arccos(c[iu[0], iu[1]]).mean().item()
|
| 70 |
+
obs = mean_angle(axes)
|
| 71 |
+
g = torch.Generator().manual_seed(seed)
|
| 72 |
+
base_angles = []
|
| 73 |
+
m = max(2, n)
|
| 74 |
+
for _ in range(max(1, n_baseline // max(1, m * (m - 1) // 2))):
|
| 75 |
+
r = F.normalize(torch.randn(m, D, generator=g), dim=-1)
|
| 76 |
+
base_angles.append(mean_angle(r))
|
| 77 |
+
base = sum(base_angles) / len(base_angles)
|
| 78 |
+
s = torch.linalg.svdvals(axes)
|
| 79 |
+
p = (s / s.sum().clamp_min(1e-12))
|
| 80 |
+
erank = float(torch.exp(-(p.clamp_min(1e-12) * p.clamp_min(1e-12).log()).sum()))
|
| 81 |
+
return {"proj_angle_mean": round(obs, 4), "uniform_baseline": round(base, 4),
|
| 82 |
+
"deviation": round(obs - base, 4), "erank": round(erank, 3)}
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
@torch.no_grad()
|
| 86 |
+
def read_specimen(path: str) -> dict:
|
| 87 |
+
ck = torch.load(path, map_location="cpu", weights_only=True)
|
| 88 |
+
out = {"file": path.split("\\")[-1].split("/")[-1],
|
| 89 |
+
"arm": ck.get("arm"), "seed": ck.get("seed"),
|
| 90 |
+
"steps": ck.get("steps"), "val_bpb": round(ck.get("val_bpb", -1), 4)}
|
| 91 |
+
if "state_dict" in ck: # full specimen checkpoint
|
| 92 |
+
sd = ck["state_dict"]
|
| 93 |
+
books = {k[:-len(".codebook")]: sd[k] for k in sd
|
| 94 |
+
if k.endswith("addr.codebook") or k.endswith("head_addr.codebook")}
|
| 95 |
+
homes = {k[:-len(".home")]: sd[k] for k in sd if k.endswith(".home")}
|
| 96 |
+
else: # bare genome dict (exp014+ champion files):
|
| 97 |
+
# books under flat/root/branch* keys; *_proj entries are projections
|
| 98 |
+
books = {k: v for k, v in ck.items()
|
| 99 |
+
if torch.is_tensor(v) and v.ndim == 2
|
| 100 |
+
and (k in ("flat", "root") or k.startswith("branch"))}
|
| 101 |
+
homes = {}
|
| 102 |
+
reads = {}
|
| 103 |
+
for name, cb in books.items():
|
| 104 |
+
col = antipodal_collapse(cb)
|
| 105 |
+
stats = projective_stats(col["axes"])
|
| 106 |
+
home = homes.get(name)
|
| 107 |
+
drift = None
|
| 108 |
+
binding = None
|
| 109 |
+
if home is not None and home.shape == cb.shape:
|
| 110 |
+
a = F.normalize(cb.float(), dim=-1)
|
| 111 |
+
b = F.normalize(home.float(), dim=-1)
|
| 112 |
+
dr = torch.arccos((a * b).sum(-1).clamp(-1, 1))
|
| 113 |
+
drift = round(dr.mean().item(), 4)
|
| 114 |
+
binding = round(((dr - BINDING).abs() <= 0.05).float().mean().item(), 4)
|
| 115 |
+
util = col["n_axes"] / cb.shape[0]
|
| 116 |
+
dev = stats["deviation"]
|
| 117 |
+
if dev is not None and abs(dev) < 0.05 and util > 0.95 and col["pairs"] <= 3:
|
| 118 |
+
verdict = "PROJECTIVE-CLEAN"
|
| 119 |
+
elif dev is not None and abs(dev) < 0.05:
|
| 120 |
+
verdict = "PROJECTIVE-MOSTLY"
|
| 121 |
+
elif dev is not None and dev > 0.05:
|
| 122 |
+
verdict = "STRUCTURED(repulsive)"
|
| 123 |
+
else:
|
| 124 |
+
verdict = "DEGENERATE/CLUMPED" if dev is not None else "TOO-FEW-AXES"
|
| 125 |
+
reads[name] = {
|
| 126 |
+
"pairs": col["pairs"], "n_axes": col["n_axes"], **stats,
|
| 127 |
+
"drift": drift, "binding_frac": binding, "verdict": verdict}
|
| 128 |
+
out["codebooks"] = reads
|
| 129 |
+
return out
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def read_all(root: str) -> list:
|
| 133 |
+
import glob, os
|
| 134 |
+
results = []
|
| 135 |
+
for p in sorted(glob.glob(os.path.join(root, "*.pt"))):
|
| 136 |
+
r = read_specimen(p)
|
| 137 |
+
print(r, flush=True)
|
| 138 |
+
results.append(r)
|
| 139 |
+
return results
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def _in_notebook() -> bool:
|
| 143 |
+
try:
|
| 144 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 145 |
+
return True
|
| 146 |
+
except NameError:
|
| 147 |
+
return False
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
if __name__ == "__main__":
|
| 151 |
+
if _in_notebook():
|
| 152 |
+
print("Notebook mode: call read_all(r'<data_root>/ar_ckpts') in the next cell.")
|
| 153 |
+
else:
|
| 154 |
+
args = [a for a in sys.argv[1:] if not a.startswith("-")]
|
| 155 |
+
if not args:
|
| 156 |
+
print("usage: python read_codebook.py <ckpt_or_dir> [...]")
|
| 157 |
+
for a in args:
|
| 158 |
+
import os
|
| 159 |
+
read_all(a) if os.path.isdir(a) else print(read_specimen(a))
|
exp014_gd/repro.py
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""repro.py — standalone loader/runner for exp014_gd. Every code dependency
|
| 2 |
+
lives in THIS folder (geolip_vitals.py, ar_differentiation_bed.py,
|
| 3 |
+
exp013_augmentation_bed.py for run_b2, exp014_genetic_distillation.py,
|
| 4 |
+
read_codebook.py) — no cross-package paths.
|
| 5 |
+
|
| 6 |
+
python repro.py # CPU smoke: parse + shapes + GPA sanity
|
| 7 |
+
python repro.py --mode tournament # one full lineage bracket (GPU, ~20 min)
|
| 8 |
+
python repro.py --mode b1 # cross-size implants (GPU)
|
| 9 |
+
python repro.py --mode b2 # GPT-2 relay implants (GPU; downloads GPT-2)
|
| 10 |
+
|
| 11 |
+
Or drive it directly, e.g. a specific lineage:
|
| 12 |
+
python -c "import sys; sys.path.insert(0, '.'); \
|
| 13 |
+
from exp014_genetic_distillation import run_tournament; \
|
| 14 |
+
run_tournament('aleph_full')"
|
| 15 |
+
|
| 16 |
+
Data lands in ./data (override with the GEOLIP_DATA env var); the tournament
|
| 17 |
+
ledger + champion genomes land in ./data/exp014. Reading the shipped genomes:
|
| 18 |
+
python read_codebook.py genomes/
|
| 19 |
+
"""
|
| 20 |
+
import os
|
| 21 |
+
import runpy
|
| 22 |
+
import sys
|
| 23 |
+
|
| 24 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 25 |
+
sys.path.insert(0, HERE)
|
| 26 |
+
|
| 27 |
+
if __name__ == "__main__":
|
| 28 |
+
if len(sys.argv) == 1:
|
| 29 |
+
import geolip_vitals # noqa: F401 (paste order)
|
| 30 |
+
import ar_differentiation_bed as bed
|
| 31 |
+
import exp014_genetic_distillation as gd
|
| 32 |
+
bed.smoke()
|
| 33 |
+
gd.smoke()
|
| 34 |
+
print("repro smoke passed — run with --mode {tournament,b1,b2} for "
|
| 35 |
+
"verdict runs")
|
| 36 |
+
else:
|
| 37 |
+
target = os.path.join(HERE, "exp014_genetic_distillation.py")
|
| 38 |
+
sys.argv = [target] + sys.argv[1:]
|
| 39 |
+
runpy.run_path(target, run_name="__main__")
|
exp015_ch/README.md
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# exp015_ch — content-bearing heredity
|
| 2 |
+
|
| 3 |
+
Sequel to [exp014](../exp014_gd/) ([article](../exp014_gd/article.md)), executing
|
| 4 |
+
its closing compass. exp014 found that consensus-book heredity is task-neutral
|
| 5 |
+
because any healthy near-uniform book is as good as another — so genetic methods
|
| 6 |
+
can only pay where the book's **content** is load-bearing. exp015 makes content
|
| 7 |
+
load-bearing in the consumption itself: organisms whose entire prediction channel
|
| 8 |
+
is the **sign-code read** (features are literally the ±A\[win\] rows of the book),
|
| 9 |
+
plus the tree lineage finally tested under fair full-weight inheritance. Every
|
| 10 |
+
exp014 lesson is baked in (full-weight continuity everywhere, KD α=0.25 with
|
| 11 |
+
founders exempt, consensus overwrite only on the `_full` lineages).
|
| 12 |
+
|
| 13 |
+
New instrument — the **content gauge** `sign_fidelity`: Spearman correlation of
|
| 14 |
+
book-Hamming distance vs true angle over random sphere pairs — the LSH
|
| 15 |
+
faithfulness of the cultivated book, logged per member.
|
| 16 |
+
|
| 17 |
+
## The brackets (best-of-generation, bits/byte; 4 gens × pop 4, seed 0)
|
| 18 |
+
|
| 19 |
+
| lineage | inherits | g0 | g1 | g2 | g3 |
|
| 20 |
+
|---|---|---|---|---|---|
|
| 21 |
+
| sign_weights | full weights (continuity control) | 2.4542 | 2.4149 | 2.4110 | **2.4098** |
|
| 22 |
+
| sign_full | full weights + consensus book | 2.4518 | 2.4120 | 2.4083 | **2.4081** |
|
| 23 |
+
| tree_weights | full weights (tree control) | 2.6228 | 2.5889 | 2.5790 | **2.5721** |
|
| 24 |
+
| tree_full | full weights + root+branch consensus | 2.6307 | 2.5901 | 2.5815 | **2.5750** |
|
| 25 |
+
| sign_none | nothing (floor) | 2.4472 | 2.4448 | 2.4365 | 2.4152 |
|
| 26 |
+
|
| 27 |
+
`build_results.py` re-asserts every claim below from `results/ledger.jsonl`.
|
| 28 |
+
|
| 29 |
+
## Findings
|
| 30 |
+
|
| 31 |
+
1. **Content consumption cultivates fidelity by itself — and that compresses the
|
| 32 |
+
germline's headroom.** The control lineage's books climb from 0.942 to a 0.954
|
| 33 |
+
fidelity plateau with no consensus operator anywhere; every fresh founder's
|
| 34 |
+
book sits at 0.910–0.935. Under sign-code consumption, task pressure aligns
|
| 35 |
+
with LSH faithfulness. Consequence: the germline is ahead on bpb in 3/3
|
| 36 |
+
post-divergence generations (+0.003, +0.003, +0.002) — directionally consistent
|
| 37 |
+
but at founder-luck scale (the g0 gap was 0.0024). Content **consumption**
|
| 38 |
+
prices book identity; training pays that price on its own. The compass now
|
| 39 |
+
points at content **tasks** (retrieval, spelling) rather than content channels.
|
| 40 |
+
2. **The lineage fixed point replicates on the discrete channel, and it locks
|
| 41 |
+
fidelity.** Germline champion drift: 0.175 → 0.112 → 0.011 → 0.009 (stationary
|
| 42 |
+
by gen 2, exactly the exp014 dynamic); control champions drift ~0.20 every
|
| 43 |
+
generation forever. Germline heirs hold fidelity 0.9555–0.9558 with < 0.001
|
| 44 |
+
inter-member spread — all members carry the same stationary book — while the
|
| 45 |
+
control's plateau keeps wandering. The floor's luckiest draw gets within 0.007
|
| 46 |
+
bpb of the germline champion, but its best fidelity (0.9355) never approaches
|
| 47 |
+
the lineage plateau: **the content gauge separates heredity from lottery far
|
| 48 |
+
more cleanly than score does.**
|
| 49 |
+
3. **The tree inherits — continuity was the missing ingredient.** Both tree
|
| 50 |
+
lineages ascend monotonically (−0.051 / −0.056) where exp014's organ-regime
|
| 51 |
+
tree never beat its founders. The whole structure hits the fixed point: root
|
| 52 |
+
drift 0.144 → 0.004 and all four branch drifts to ≤ 0.005 by gen 2, branch
|
| 53 |
+
fidelity locked at 0.945–0.962. Structured consensus works as a germline
|
| 54 |
+
operator; the ~0.16 architecture tax vs flat heads is unchanged.
|
| 55 |
+
4. **Branch revival belongs to continuity, not the germline.** The control's
|
| 56 |
+
weakest branch rises from 8.8% to 13.3% usage (root routing entropy ppl4
|
| 57 |
+
3.28 → 3.77) while the germline's fixed point freezes routing at
|
| 58 |
+
\[0.41, 0.34, 0.087, 0.17\] (ppl4 3.47) — stationarity and adaptability trade
|
| 59 |
+
off directly, at zero score difference.
|
| 60 |
+
5. **Heredity maintains routing health that fresh draws repeatedly lose.** Every
|
| 61 |
+
fresh tree founder develops root imbalance (one branch at 67–76% usage, ppl4
|
| 62 |
+
as low as 1.96); lineage champions never do (ppl4 ≥ 3.28, no branch below
|
| 63 |
+
8%). Selection + inheritance preserve a property that single-organism
|
| 64 |
+
training reliably fails to find.
|
| 65 |
+
|
| 66 |
+
## Files
|
| 67 |
+
- `exp015_content_heredity.py` — the bed: sign-code + tree organisms, fair
|
| 68 |
+
inheritance with optional consensus overwrite, the `sign_fidelity` content
|
| 69 |
+
gauge, the tournament runner.
|
| 70 |
+
- `geolip_vitals.py` / `ar_differentiation_bed.py` /
|
| 71 |
+
`exp014_genetic_distillation.py` / `read_codebook.py` — this package's own
|
| 72 |
+
copies of the shared harness (readout diagnostics; the ByteLM organism +
|
| 73 |
+
attention mechanisms; the consensus operator + TreeHead; the projective
|
| 74 |
+
specimen/genome reader). The package is standalone.
|
| 75 |
+
- `repro.py` — loads the code files from this folder and runs them.
|
| 76 |
+
- `build_results.py` → `results/results.json` — re-asserts every claim above
|
| 77 |
+
from the ledger.
|
| 78 |
+
- `results/ledger.jsonl` — the append-only per-member record (80 rows).
|
| 79 |
+
- `genomes/` — 20 per-generation champion codebooks.
|
| 80 |
+
|
| 81 |
+
## Reproduce (from inside this folder)
|
| 82 |
+
```bash
|
| 83 |
+
pip install torch --index-url https://download.pytorch.org/whl/cu128
|
| 84 |
+
pip install pyarrow huggingface_hub
|
| 85 |
+
python repro.py # CPU smoke: parse + shapes + inheritance
|
| 86 |
+
python repro.py --lineage sign_full # one full bracket (GPU, ~20 min)
|
| 87 |
+
python read_codebook.py genomes/ # read the germline fossil record
|
| 88 |
+
python build_results.py # re-assert every claim from the ledger
|
| 89 |
+
```
|
| 90 |
+
Data lands in `./data` (override with the `GEOLIP_DATA` env var); bracket ledgers
|
| 91 |
+
+ champion genomes in `./data/exp015`. In Colab, paste `geolip_vitals.py`,
|
| 92 |
+
`ar_differentiation_bed.py`, `exp014_genetic_distillation.py`, then
|
| 93 |
+
`exp015_content_heredity.py` into cells in that order.
|
| 94 |
+
|
| 95 |
+
## Limitations
|
| 96 |
+
One tournament seed; one bed (wikitext-2 bytes); 2000-step organisms; the
|
| 97 |
+
content gauge is a geometry readout (LSH faithfulness), not yet a content
|
| 98 |
+
*task* — the retrieval/spelling-judged tournament is the named next step.
|
| 99 |
+
|
| 100 |
+
License: MIT · AbstractPhil + Claude Fable 5 · July 10, 2026
|
exp015_ch/ar_differentiation_bed.py
ADDED
|
@@ -0,0 +1,487 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""ar_differentiation_bed.py — exp012: autoregressive differentiation of the aleph.
|
| 2 |
+
|
| 3 |
+
Differentiation is cultivated by PREDICTIVE pressure along the sequence — the
|
| 4 |
+
address parameterizing the next-byte distribution (Law 2: chain-rule advantage pays
|
| 5 |
+
ONLY where the composed address directly parameterizes the predictive distribution).
|
| 6 |
+
This bed puts the aleph in the autoregressive gradient path and measures what
|
| 7 |
+
differentiates. The head arms enforce the employment law at its maximum: the
|
| 8 |
+
ENTIRE next-byte distribution is parameterized by the address.
|
| 9 |
+
|
| 10 |
+
Byte-level causal LM on wikitext-2-raw (HF parquet, CDN-fast), block 256. ARMS:
|
| 11 |
+
sdpa — standard causal transformer control (matched trunk).
|
| 12 |
+
hub — attention replaced by CAUSAL HUB: linear attention whose feature map
|
| 13 |
+
is the 2K-oriented aleph address, prefix-sum memories (no selection
|
| 14 |
+
event; O(n*K*d)). Differentiation cultivated INSIDE attention.
|
| 15 |
+
addr_head — sdpa trunk, but the OUTPUT HEAD reads ONLY the signed aleph
|
| 16 |
+
coefficient vector w_k = sinh(u_k)/sum_j cosh(u_j) of the final
|
| 17 |
+
hidden state (K -> 256 logits). The address MUST carry every bit of
|
| 18 |
+
next-byte information — the hardest Law-2 bottleneck.
|
| 19 |
+
|
| 20 |
+
JUDGED BY: val bits-per-byte per arm (task) + CULTIVATION VITALS on every aleph
|
| 21 |
+
codebook (readouts, never losses): axis aliveness/hppl, drift-from-init +
|
| 22 |
+
binding fraction @0.29154, winner-|cos| saturation (sign-code emergence), shadow
|
| 23 |
+
path diversity (fixed high-bits hash). Never by recon.
|
| 24 |
+
|
| 25 |
+
Riders: pure Adam wd=0; no BN/Dropout/GAP on geometric paths; orthogonal init;
|
| 26 |
+
Colab-cell-safe (paste-ahead imports, no bare argparse, no __file__ reliance);
|
| 27 |
+
GPU-only for verdict runs.
|
| 28 |
+
|
| 29 |
+
Terminal: python ar_differentiation_bed.py # shapes/parse smoke
|
| 30 |
+
python ar_differentiation_bed.py --train # verdict run
|
| 31 |
+
Colab: paste geolip_vitals.py cell, then this file (smoke auto-runs),
|
| 32 |
+
then train(steps=2000, data_root="/content/data") in the next cell.
|
| 33 |
+
"""
|
| 34 |
+
from __future__ import annotations
|
| 35 |
+
import math
|
| 36 |
+
import torch
|
| 37 |
+
import torch.nn as nn
|
| 38 |
+
import torch.nn.functional as F
|
| 39 |
+
|
| 40 |
+
if "anchor_drift" not in globals():
|
| 41 |
+
try:
|
| 42 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 43 |
+
except ImportError:
|
| 44 |
+
_here = globals().get("__file__")
|
| 45 |
+
if _here is not None:
|
| 46 |
+
import sys, pathlib
|
| 47 |
+
sys.path.insert(0, str(pathlib.Path(_here).parent))
|
| 48 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 49 |
+
else:
|
| 50 |
+
raise ImportError(
|
| 51 |
+
"geolip_vitals not found — paste/run its cell first, or "
|
| 52 |
+
"hf_hub_download exp012_ar/geolip_vitals.py from "
|
| 53 |
+
"AbstractPhil/geolip-aleph-differentiation.")
|
| 54 |
+
|
| 55 |
+
VOCAB = 256 # bytes
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
# ------------------------------------------------------------------ aleph address
|
| 59 |
+
def _super_fibonacci_s3(n: int) -> torch.Tensor:
|
| 60 |
+
"""Near-uniform unit quaternions (Alexa CVPR'22) —
|
| 61 |
+
starts the codebook INSIDE the RP^3 attractor basin. D=4 only."""
|
| 62 |
+
PHI, PSI = math.sqrt(2.0), 1.533751168755204288118041
|
| 63 |
+
i = torch.arange(n, dtype=torch.float64)
|
| 64 |
+
s = (i + 0.5) / n
|
| 65 |
+
r, R = torch.sqrt(s), torch.sqrt(1.0 - s)
|
| 66 |
+
a, b = 2 * math.pi * i / PHI, 2 * math.pi * i / PSI
|
| 67 |
+
q = torch.stack([r * torch.sin(a), r * torch.cos(a),
|
| 68 |
+
R * torch.sin(b), R * torch.cos(b)], dim=-1)
|
| 69 |
+
return F.normalize(q, dim=-1).float()
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
class AlephAddress(nn.Module):
|
| 73 |
+
"""Closed-form aleph over 2K oriented half-axes (aleph-void article).
|
| 74 |
+
signed(x): (..., K) w_k = sinh(u_k)/sum_j cosh(u_j) — the Law-2 head feature.
|
| 75 |
+
oriented(x): ((..., K), (..., K)) positive halves of the 2K softmax — HUB map."""
|
| 76 |
+
|
| 77 |
+
def __init__(self, K: int, D: int, tau: float = 0.1, init: str = "random"):
|
| 78 |
+
super().__init__()
|
| 79 |
+
self.K, self.D, self.tau = K, D, tau
|
| 80 |
+
if init == "fibonacci":
|
| 81 |
+
assert D == 4, "fibonacci init lives on S^3 (D=4)"
|
| 82 |
+
A = _super_fibonacci_s3(K)
|
| 83 |
+
else:
|
| 84 |
+
A = F.normalize(torch.randn(K, D), dim=-1)
|
| 85 |
+
self.codebook = nn.Parameter(A)
|
| 86 |
+
self.register_buffer("home", self.codebook.detach().clone())
|
| 87 |
+
|
| 88 |
+
def _u(self, x):
|
| 89 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 90 |
+
return (F.normalize(x, dim=-1) @ A.transpose(-1, -2)) / self.tau
|
| 91 |
+
|
| 92 |
+
def oriented(self, x):
|
| 93 |
+
u = self._u(x)
|
| 94 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 95 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 96 |
+
Z = (ep + en).sum(dim=-1, keepdim=True)
|
| 97 |
+
return ep / Z, en / Z
|
| 98 |
+
|
| 99 |
+
def signed(self, x):
|
| 100 |
+
u = self._u(x)
|
| 101 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 102 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 103 |
+
return (ep - en) / (ep + en).sum(dim=-1, keepdim=True)
|
| 104 |
+
|
| 105 |
+
def signed_at(self, x, taus):
|
| 106 |
+
"""Multi-tau stroboscope (rule of 3): signed coefficients at several
|
| 107 |
+
temperatures, concatenated — softer taus keep the vector dense while a
|
| 108 |
+
hard tau supplies the sign-code sharpness. v2 refinement (b)."""
|
| 109 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 110 |
+
cos = F.normalize(x, dim=-1) @ A.transpose(-1, -2)
|
| 111 |
+
outs = []
|
| 112 |
+
for t in taus:
|
| 113 |
+
u = cos / t
|
| 114 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 115 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 116 |
+
outs.append((ep - en) / (ep + en).sum(dim=-1, keepdim=True))
|
| 117 |
+
return torch.cat(outs, dim=-1)
|
| 118 |
+
|
| 119 |
+
def m_hat(self, x):
|
| 120 |
+
"""Closed-form soft read (decoders read M_hat, never M). v2 control (c)."""
|
| 121 |
+
u = self._u(x)
|
| 122 |
+
m = u.abs().amax(dim=-1, keepdim=True)
|
| 123 |
+
ep, en = torch.exp(u - m), torch.exp(-u - m)
|
| 124 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 125 |
+
return ((ep - en) @ A) / (ep + en).sum(dim=-1, keepdim=True)
|
| 126 |
+
|
| 127 |
+
def m_hard_ste(self, x):
|
| 128 |
+
"""Hard mode (aleph-void article): M_hard = sign(cos_win) * A[win], straight-through to
|
| 129 |
+
the soft read — forward fully discrete SIGN CODE, backward soft gradient.
|
| 130 |
+
Legal per theme A (reconstructive sign code, not a one-hot roster pick)."""
|
| 131 |
+
u = self._u(x)
|
| 132 |
+
soft = self.m_hat(x)
|
| 133 |
+
win = u.abs().argmax(dim=-1)
|
| 134 |
+
A = F.normalize(self.codebook, dim=-1)
|
| 135 |
+
sign = torch.sign(torch.gather(u, -1, win.unsqueeze(-1))).squeeze(-1)
|
| 136 |
+
hard = sign.unsqueeze(-1) * A[win]
|
| 137 |
+
return hard + soft - soft.detach()
|
| 138 |
+
|
| 139 |
+
@torch.no_grad()
|
| 140 |
+
def vitals(self, x_sample) -> dict:
|
| 141 |
+
u = self._u(x_sample.reshape(-1, x_sample.shape[-1]))
|
| 142 |
+
p, n = self.oriented(x_sample.reshape(-1, x_sample.shape[-1]))
|
| 143 |
+
two_k = torch.cat([p, n], dim=-1)
|
| 144 |
+
win = two_k.argmax(dim=-1)
|
| 145 |
+
cos_win = (u.abs().amax(dim=-1) * self.tau) # winner |cos| — sign-code sat.
|
| 146 |
+
d = anchor_drift(self.codebook, self.home)
|
| 147 |
+
return {"drift": round(d["mean"], 4),
|
| 148 |
+
"binding_frac": round(d["binding_fraction"], 4),
|
| 149 |
+
"aliveness": axis_aliveness(two_k),
|
| 150 |
+
"win_cos_mean": round(cos_win.mean().item(), 4),
|
| 151 |
+
"paths": path_diversity(win)}
|
| 152 |
+
|
| 153 |
+
|
| 154 |
+
# ------------------------------------------------------------------------- blocks
|
| 155 |
+
class CausalSDPA(nn.Module):
|
| 156 |
+
def __init__(self, d: int, heads: int = 4):
|
| 157 |
+
super().__init__()
|
| 158 |
+
self.h = heads
|
| 159 |
+
self.qkv = nn.Linear(d, 3 * d, bias=False)
|
| 160 |
+
self.o = nn.Linear(d, d, bias=False)
|
| 161 |
+
nn.init.orthogonal_(self.qkv.weight); nn.init.orthogonal_(self.o.weight)
|
| 162 |
+
|
| 163 |
+
def forward(self, x):
|
| 164 |
+
B, n, d = x.shape
|
| 165 |
+
q, k, v = self.qkv(x).chunk(3, dim=-1)
|
| 166 |
+
q, k, v = (t.view(B, n, self.h, d // self.h).transpose(1, 2) for t in (q, k, v))
|
| 167 |
+
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
| 168 |
+
return self.o(y.transpose(1, 2).reshape(B, n, d))
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
class CausalHUB(nn.Module):
|
| 172 |
+
"""Causal aleph linear attention: prefix-sum memories over the two K-wide
|
| 173 |
+
halves of the oriented address; 2K never materialized; no selection event."""
|
| 174 |
+
|
| 175 |
+
def __init__(self, d: int, K: int = 32, D: int = 4, tau: float = 0.1):
|
| 176 |
+
super().__init__()
|
| 177 |
+
self.addr = AlephAddress(K, D, tau)
|
| 178 |
+
self.q = nn.Linear(d, D, bias=False)
|
| 179 |
+
self.k = nn.Linear(d, D, bias=False)
|
| 180 |
+
self.v = nn.Linear(d, d, bias=False)
|
| 181 |
+
self.o = nn.Linear(d, d, bias=False)
|
| 182 |
+
for m in (self.q, self.k, self.v, self.o):
|
| 183 |
+
nn.init.orthogonal_(m.weight)
|
| 184 |
+
|
| 185 |
+
def forward(self, x):
|
| 186 |
+
qp, qn = self.addr.oriented(self.q(x)) # (B, n, K)
|
| 187 |
+
kp, kn = self.addr.oriented(self.k(x))
|
| 188 |
+
v = self.v(x) # (B, n, d)
|
| 189 |
+
Sp = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kp, v), dim=1)
|
| 190 |
+
Sn = torch.cumsum(torch.einsum("bnk,bnd->bnkd", kn, v), dim=1)
|
| 191 |
+
zp = torch.cumsum(kp, dim=1)
|
| 192 |
+
zn = torch.cumsum(kn, dim=1)
|
| 193 |
+
num = torch.einsum("bnk,bnkd->bnd", qp, Sp) + torch.einsum("bnk,bnkd->bnd", qn, Sn)
|
| 194 |
+
den = (qp * zp).sum(-1, keepdim=True) + (qn * zn).sum(-1, keepdim=True)
|
| 195 |
+
return self.o(num / den.clamp_min(1e-12))
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
class MslRelay(nn.Module):
|
| 199 |
+
"""Depth-composition unit (chain-rule probe): multi-slot M_hat read entering
|
| 200 |
+
the trunk as a NEAR-ZERO gated residual (gate init -3.0, sigma~0.047 — theme D:
|
| 201 |
+
geometry enters as a nudge and grows only if it earns gradient)."""
|
| 202 |
+
|
| 203 |
+
def __init__(self, d: int, n_slots: int = 16, K: int = 64):
|
| 204 |
+
super().__init__()
|
| 205 |
+
self.n_slots = n_slots
|
| 206 |
+
self.proj = nn.Linear(d, n_slots * 4, bias=False)
|
| 207 |
+
self.out = nn.Linear(n_slots * 4, d, bias=False)
|
| 208 |
+
nn.init.orthogonal_(self.proj.weight)
|
| 209 |
+
nn.init.orthogonal_(self.out.weight)
|
| 210 |
+
self.addr = AlephAddress(K, 4)
|
| 211 |
+
self.gate = nn.Parameter(torch.tensor(-3.0))
|
| 212 |
+
|
| 213 |
+
def forward(self, x):
|
| 214 |
+
B, n, _ = x.shape
|
| 215 |
+
slots = self.proj(x).view(B, n, self.n_slots, 4)
|
| 216 |
+
m = self.addr.m_hat(slots).reshape(B, n, -1)
|
| 217 |
+
return x + self.gate.sigmoid() * self.out(m)
|
| 218 |
+
|
| 219 |
+
|
| 220 |
+
class Block(nn.Module):
|
| 221 |
+
def __init__(self, d: int, attn: nn.Module):
|
| 222 |
+
super().__init__()
|
| 223 |
+
self.n1, self.n2 = nn.LayerNorm(d), nn.LayerNorm(d)
|
| 224 |
+
self.attn = attn
|
| 225 |
+
self.mlp = nn.Sequential(nn.Linear(d, 4 * d), nn.GELU(), nn.Linear(4 * d, d))
|
| 226 |
+
|
| 227 |
+
def forward(self, x):
|
| 228 |
+
x = x + self.attn(self.n1(x))
|
| 229 |
+
return x + self.mlp(self.n2(x))
|
| 230 |
+
|
| 231 |
+
|
| 232 |
+
class ByteLM(nn.Module):
|
| 233 |
+
def __init__(self, arm: str, d: int = 192, layers: int = 4, block: int = 256,
|
| 234 |
+
K: int = 32, D: int = 4):
|
| 235 |
+
super().__init__()
|
| 236 |
+
# "<arm>_tri" suffix = trigram byte embedding (AlephLM byte_emb x3 lineage):
|
| 237 |
+
# token embedding is the sum of embeddings of bytes t, t-1, t-2.
|
| 238 |
+
self.trigram = arm.endswith("_tri")
|
| 239 |
+
if self.trigram:
|
| 240 |
+
arm = arm[:-4]
|
| 241 |
+
# "_fib" = super-Fibonacci S^3 codebook init (basin test: starts INSIDE
|
| 242 |
+
# the RP^3 attractor; primary observable is init->final geodesic drift).
|
| 243 |
+
self.fib = arm.endswith("_fib")
|
| 244 |
+
if self.fib:
|
| 245 |
+
arm = arm[:-4]
|
| 246 |
+
# "relay*" = stacked addresses in depth: MslRelay after every block.
|
| 247 |
+
# relay -> sdpa trunk + standard head; relay_msl64 -> + addressed head.
|
| 248 |
+
self.use_relay = arm.startswith("relay")
|
| 249 |
+
if arm == "relay":
|
| 250 |
+
arm = "sdpa"
|
| 251 |
+
elif arm == "relay_msl64":
|
| 252 |
+
arm = "addr_msl64"
|
| 253 |
+
self.arm, self.block = arm, block
|
| 254 |
+
self.emb = nn.Embedding(VOCAB, d)
|
| 255 |
+
if self.trigram:
|
| 256 |
+
self.emb1 = nn.Embedding(VOCAB, d)
|
| 257 |
+
self.emb2 = nn.Embedding(VOCAB, d)
|
| 258 |
+
self.pos = nn.Parameter(torch.zeros(1, block, d) + 0.01 * torch.randn(1, block, d))
|
| 259 |
+
mk_attn = (lambda: CausalHUB(d, K, D)) if arm == "hub" else (lambda: CausalSDPA(d))
|
| 260 |
+
self.blocks = nn.ModuleList([Block(d, mk_attn()) for _ in range(layers)])
|
| 261 |
+
if self.use_relay:
|
| 262 |
+
self.relays = nn.ModuleList([MslRelay(d) for _ in range(layers)])
|
| 263 |
+
self.nf = nn.LayerNorm(d)
|
| 264 |
+
if arm == "addr_head":
|
| 265 |
+
self.head_addr = AlephAddress(K, d) # v1: codebook in model dim — COLLAPSED
|
| 266 |
+
self.head = nn.Linear(K, VOCAB, bias=True)
|
| 267 |
+
elif arm in ("addr_d4", "addr_3tau", "addr_mhat"):
|
| 268 |
+
# v2 refinements: LOW-D HOME — learned projection to the native D=4 home
|
| 269 |
+
# before addressing (mirrors the healthy HUB arms), K=64.
|
| 270 |
+
self.head_proj = nn.Linear(d, 4, bias=False)
|
| 271 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 272 |
+
self.head_addr = AlephAddress(64, 4)
|
| 273 |
+
if arm == "addr_d4":
|
| 274 |
+
self.head = nn.Linear(64, VOCAB, bias=True) # w alone, D=4 home
|
| 275 |
+
elif arm == "addr_3tau":
|
| 276 |
+
self.taus = (0.05, 0.1, 0.3) # rule-of-3 strobe
|
| 277 |
+
self.head = nn.Linear(64 * 3, VOCAB, bias=True)
|
| 278 |
+
else: # addr_mhat
|
| 279 |
+
self.head = nn.Linear(4, VOCAB, bias=True) # tightest: M_hat
|
| 280 |
+
elif arm.startswith("addr_msl"):
|
| 281 |
+
# v3: MULTI-SLOT heads — the 16s funnel widening: P parallel D=4 slots
|
| 282 |
+
# over a SHARED codebook. addr_msl consumes the reconstructive M_hat per
|
| 283 |
+
# slot (Px4 dims); addr_msl_w consumes signed w per slot (Px64) — tests
|
| 284 |
+
# whether slot-parallel consumption alone rescues the coefficient path.
|
| 285 |
+
# addr_msl<P> = slot-count dose-response. addr_mslh<P> = HARD sign-code
|
| 286 |
+
# consumption (straight-through M_hard per slot).
|
| 287 |
+
self.hard = arm.startswith("addr_mslh")
|
| 288 |
+
if arm in ("addr_msl", "addr_msl_w"):
|
| 289 |
+
self.n_slots = 16
|
| 290 |
+
else:
|
| 291 |
+
self.n_slots = int(arm[len("addr_mslh" if self.hard else "addr_msl"):])
|
| 292 |
+
self.head_proj = nn.Linear(d, self.n_slots * 4, bias=False)
|
| 293 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 294 |
+
self.head_addr = AlephAddress(
|
| 295 |
+
64, 4, init="fibonacci" if self.fib else "random")
|
| 296 |
+
width = self.n_slots * (64 if arm == "addr_msl_w" else 4)
|
| 297 |
+
self.head = nn.Linear(width, VOCAB, bias=True)
|
| 298 |
+
elif arm == "addr_3tau_mhat":
|
| 299 |
+
# v3: combine the two v2 winners — 3-tau stroboscope + reconstructive read.
|
| 300 |
+
self.head_proj = nn.Linear(d, 4, bias=False)
|
| 301 |
+
nn.init.orthogonal_(self.head_proj.weight)
|
| 302 |
+
self.head_addr = AlephAddress(64, 4)
|
| 303 |
+
self.taus = (0.05, 0.1, 0.3)
|
| 304 |
+
self.head = nn.Linear(64 * 3 + 4, VOCAB, bias=True)
|
| 305 |
+
else:
|
| 306 |
+
self.head = nn.Linear(d, VOCAB, bias=True)
|
| 307 |
+
self._last_h = None
|
| 308 |
+
|
| 309 |
+
def forward(self, idx):
|
| 310 |
+
x = self.emb(idx)
|
| 311 |
+
if self.trigram: # past-only shifts — causality preserved
|
| 312 |
+
x = x + self.emb1(F.pad(idx, (1, 0), value=0)[:, :-1]) \
|
| 313 |
+
+ self.emb2(F.pad(idx, (2, 0), value=0)[:, :-2])
|
| 314 |
+
x = x + self.pos[:, : idx.shape[1]]
|
| 315 |
+
if self.use_relay:
|
| 316 |
+
for b, r in zip(self.blocks, self.relays):
|
| 317 |
+
x = r(b(x))
|
| 318 |
+
else:
|
| 319 |
+
for b in self.blocks:
|
| 320 |
+
x = b(x)
|
| 321 |
+
h = self.nf(x)
|
| 322 |
+
self._last_h = h.detach()
|
| 323 |
+
if self.arm == "addr_head":
|
| 324 |
+
return self.head(self.head_addr.signed(h))
|
| 325 |
+
if self.arm == "addr_d4":
|
| 326 |
+
return self.head(self.head_addr.signed(self.head_proj(h)))
|
| 327 |
+
if self.arm == "addr_3tau":
|
| 328 |
+
return self.head(self.head_addr.signed_at(self.head_proj(h), self.taus))
|
| 329 |
+
if self.arm == "addr_mhat":
|
| 330 |
+
return self.head(self.head_addr.m_hat(self.head_proj(h)))
|
| 331 |
+
if self.arm.startswith("addr_msl"):
|
| 332 |
+
B, n, _ = h.shape
|
| 333 |
+
slots = self.head_proj(h).view(B, n, self.n_slots, 4)
|
| 334 |
+
if self.arm == "addr_msl_w":
|
| 335 |
+
feats = self.head_addr.signed(slots).reshape(B, n, -1)
|
| 336 |
+
elif getattr(self, "hard", False):
|
| 337 |
+
feats = self.head_addr.m_hard_ste(slots).reshape(B, n, -1)
|
| 338 |
+
else:
|
| 339 |
+
feats = self.head_addr.m_hat(slots).reshape(B, n, -1)
|
| 340 |
+
return self.head(feats)
|
| 341 |
+
if self.arm == "addr_3tau_mhat":
|
| 342 |
+
p = self.head_proj(h)
|
| 343 |
+
feats = torch.cat([self.head_addr.signed_at(p, self.taus),
|
| 344 |
+
self.head_addr.m_hat(p)], dim=-1)
|
| 345 |
+
return self.head(feats)
|
| 346 |
+
return self.head(h)
|
| 347 |
+
|
| 348 |
+
@torch.no_grad()
|
| 349 |
+
def vitals(self) -> dict:
|
| 350 |
+
out = {}
|
| 351 |
+
if self.arm == "hub":
|
| 352 |
+
for i, b in enumerate(self.blocks):
|
| 353 |
+
if self._last_h is not None:
|
| 354 |
+
out[f"L{i}"] = b.attn.addr.vitals(b.attn.q(self._last_h[:2]))
|
| 355 |
+
elif self.arm == "addr_head" and self._last_h is not None:
|
| 356 |
+
out["head"] = self.head_addr.vitals(self._last_h[:2])
|
| 357 |
+
elif self.arm in ("addr_d4", "addr_3tau", "addr_mhat",
|
| 358 |
+
"addr_3tau_mhat") and self._last_h is not None:
|
| 359 |
+
out["head"] = self.head_addr.vitals(self.head_proj(self._last_h[:2]))
|
| 360 |
+
elif self.arm.startswith("addr_msl") and self._last_h is not None:
|
| 361 |
+
slots = self.head_proj(self._last_h[:2])
|
| 362 |
+
out["head"] = self.head_addr.vitals(
|
| 363 |
+
slots.reshape(*slots.shape[:-1], self.n_slots, 4))
|
| 364 |
+
if self.use_relay and self._last_h is not None:
|
| 365 |
+
for i, r in enumerate(self.relays):
|
| 366 |
+
s = r.proj(self._last_h[:2])
|
| 367 |
+
v = r.addr.vitals(s.reshape(*s.shape[:-1], r.n_slots, 4))
|
| 368 |
+
out[f"relay{i}"] = {"gate": round(r.gate.sigmoid().item(), 4),
|
| 369 |
+
"drift": v["drift"],
|
| 370 |
+
"binding_frac": v["binding_frac"],
|
| 371 |
+
"ppl": round(v["aliveness"]["usage_ppl"], 1)}
|
| 372 |
+
return out
|
| 373 |
+
|
| 374 |
+
|
| 375 |
+
# --------------------------------------------------------------------------- data
|
| 376 |
+
def _wikitext_bytes(data_root: str):
|
| 377 |
+
"""wikitext-2-raw as flat uint8 tensors via the HF parquet CDN."""
|
| 378 |
+
from huggingface_hub import hf_hub_download
|
| 379 |
+
import pyarrow.parquet as pq
|
| 380 |
+
|
| 381 |
+
def load(split):
|
| 382 |
+
p = hf_hub_download("Salesforce/wikitext",
|
| 383 |
+
f"wikitext-2-raw-v1/{split}-00000-of-00001.parquet",
|
| 384 |
+
repo_type="dataset", local_dir=data_root)
|
| 385 |
+
text = "".join(pq.read_table(p).column("text").to_pylist())
|
| 386 |
+
return torch.frombuffer(bytearray(text.encode("utf-8")), dtype=torch.uint8).clone()
|
| 387 |
+
|
| 388 |
+
return load("train"), load("validation")
|
| 389 |
+
|
| 390 |
+
|
| 391 |
+
def _batch(data: torch.Tensor, batch: int, block: int, device, g: torch.Generator):
|
| 392 |
+
ix = torch.randint(0, data.numel() - block - 1, (batch,), generator=g)
|
| 393 |
+
x = torch.stack([data[i:i + block] for i in ix]).long().to(device)
|
| 394 |
+
y = torch.stack([data[i + 1:i + block + 1] for i in ix]).long().to(device)
|
| 395 |
+
return x, y
|
| 396 |
+
|
| 397 |
+
|
| 398 |
+
# -------------------------------------------------------------------- train/smoke
|
| 399 |
+
def train(arms=("sdpa", "hub", "addr_head"), steps: int = 2000, batch: int = 32,
|
| 400 |
+
block: int = 256, device: str = "cuda", data_root: str = "./data",
|
| 401 |
+
seed: int = 0, eval_every: int = 500, save: bool = True):
|
| 402 |
+
"""Verdict run — GPU only. Pure Adam wd=0. Reports val bits-per-byte + vitals.
|
| 403 |
+
save=True writes {data_root}/ar_ckpts/{arm}_s{seed}_t{steps}.pt per arm —
|
| 404 |
+
the cultivated codebooks are SPECIMENS for the projective reading instruments."""
|
| 405 |
+
import os
|
| 406 |
+
if device == "cuda" and not torch.cuda.is_available():
|
| 407 |
+
raise RuntimeError("Verdict runs are GPU-only (never CPU-train for accuracy).")
|
| 408 |
+
ckpt_dir = os.path.join(data_root, "ar_ckpts")
|
| 409 |
+
os.makedirs(ckpt_dir, exist_ok=True)
|
| 410 |
+
tr, va = _wikitext_bytes(data_root)
|
| 411 |
+
print(f"data ready: train {tr.numel():,} bytes, val {va.numel():,} bytes", flush=True)
|
| 412 |
+
results = {}
|
| 413 |
+
for arm in arms:
|
| 414 |
+
torch.manual_seed(seed)
|
| 415 |
+
g = torch.Generator().manual_seed(seed)
|
| 416 |
+
model = ByteLM(arm, block=block).to(device)
|
| 417 |
+
n_params = sum(p.numel() for p in model.parameters())
|
| 418 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 419 |
+
for step in range(1, steps + 1):
|
| 420 |
+
x, y = _batch(tr, batch, block, device, g)
|
| 421 |
+
logits = model(x)
|
| 422 |
+
loss = F.cross_entropy(logits.reshape(-1, VOCAB), y.reshape(-1))
|
| 423 |
+
opt.zero_grad(set_to_none=True)
|
| 424 |
+
loss.backward()
|
| 425 |
+
opt.step()
|
| 426 |
+
if step % eval_every == 0 or step == steps:
|
| 427 |
+
model.eval()
|
| 428 |
+
with torch.no_grad():
|
| 429 |
+
losses = []
|
| 430 |
+
for _ in range(20):
|
| 431 |
+
xv, yv = _batch(va, batch, block, device, g)
|
| 432 |
+
lv = F.cross_entropy(model(xv).reshape(-1, VOCAB),
|
| 433 |
+
yv.reshape(-1))
|
| 434 |
+
losses.append(lv.item())
|
| 435 |
+
bpb = sum(losses) / len(losses) / math.log(2)
|
| 436 |
+
print(f"[{arm}] step {step} val_bpb={bpb:.4f} vitals={model.vitals()}",
|
| 437 |
+
flush=True)
|
| 438 |
+
model.train()
|
| 439 |
+
results[arm] = {"val_bpb": bpb, "params": n_params, "vitals": model.vitals()}
|
| 440 |
+
if save:
|
| 441 |
+
path = os.path.join(ckpt_dir, f"{arm}_s{seed}_t{steps}.pt")
|
| 442 |
+
torch.save({"arm": arm, "seed": seed, "steps": steps, "val_bpb": bpb,
|
| 443 |
+
"state_dict": {k: v.cpu() for k, v in
|
| 444 |
+
model.state_dict().items()}}, path)
|
| 445 |
+
print(f"saved specimen: {path}", flush=True)
|
| 446 |
+
print(results, flush=True)
|
| 447 |
+
return results
|
| 448 |
+
|
| 449 |
+
|
| 450 |
+
def smoke():
|
| 451 |
+
"""Shapes/parse only — no accuracy claims."""
|
| 452 |
+
x = torch.randint(0, VOCAB, (2, 64))
|
| 453 |
+
for arm in ("sdpa", "hub", "addr_head"):
|
| 454 |
+
m = ByteLM(arm, d=96, layers=2, block=64, K=16)
|
| 455 |
+
logits = m(x)
|
| 456 |
+
assert logits.shape == (2, 64, VOCAB)
|
| 457 |
+
logits.sum().backward()
|
| 458 |
+
# causality check: future byte must not affect past logits
|
| 459 |
+
with torch.no_grad():
|
| 460 |
+
a = m(x)[0, 10]
|
| 461 |
+
x2 = x.clone(); x2[0, 40] = (x2[0, 40] + 7) % 256
|
| 462 |
+
b = m(x2)[0, 10]
|
| 463 |
+
assert torch.allclose(a, b, atol=1e-4), f"{arm} leaks future context"
|
| 464 |
+
print(f"{arm}: OK params={sum(p.numel() for p in m.parameters()):,} "
|
| 465 |
+
f"vitals={m.vitals()}", flush=True)
|
| 466 |
+
print("OK — AR bed smoke passed (verdict run: train() on GPU)", flush=True)
|
| 467 |
+
|
| 468 |
+
|
| 469 |
+
def _in_notebook() -> bool:
|
| 470 |
+
try:
|
| 471 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 472 |
+
return True
|
| 473 |
+
except NameError:
|
| 474 |
+
return False
|
| 475 |
+
|
| 476 |
+
|
| 477 |
+
if __name__ == "__main__":
|
| 478 |
+
if _in_notebook():
|
| 479 |
+
smoke()
|
| 480 |
+
print("Notebook mode: call train(steps=2000) in the next cell (GPU).")
|
| 481 |
+
else:
|
| 482 |
+
import argparse
|
| 483 |
+
ap = argparse.ArgumentParser()
|
| 484 |
+
ap.add_argument("--train", action="store_true")
|
| 485 |
+
ap.add_argument("--steps", type=int, default=2000)
|
| 486 |
+
a, _ = ap.parse_known_args()
|
| 487 |
+
train(steps=a.steps) if a.train else smoke()
|
exp015_ch/build_results.py
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""build_results.py — exp015_ch: read results/ledger.jsonl, aggregate, and
|
| 2 |
+
RE-ASSERT every claim in the README. Run from inside this folder:
|
| 3 |
+
python build_results.py
|
| 4 |
+
"""
|
| 5 |
+
import json
|
| 6 |
+
import os
|
| 7 |
+
|
| 8 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 9 |
+
LEDGER = os.path.join(HERE, "results", "ledger.jsonl")
|
| 10 |
+
OUT = os.path.join(HERE, "results", "results.json")
|
| 11 |
+
|
| 12 |
+
rows = [json.loads(l) for l in open(LEDGER, encoding="utf-8")]
|
| 13 |
+
assert all(r["exp"] == "15" for r in rows)
|
| 14 |
+
|
| 15 |
+
LINEAGES = ["sign_weights", "sign_full", "tree_weights", "tree_full", "sign_none"]
|
| 16 |
+
best = {} # lineage -> [best bpb per gen]
|
| 17 |
+
champ = {} # lineage -> [champion row per gen]
|
| 18 |
+
fresh = {} # lineage -> fresh-founder rows
|
| 19 |
+
for lin in LINEAGES:
|
| 20 |
+
lr = [r for r in rows if r["lineage"] == lin]
|
| 21 |
+
gens = sorted({r["gen"] for r in lr})
|
| 22 |
+
assert gens == [0, 1, 2, 3], (lin, gens)
|
| 23 |
+
best[lin] = []
|
| 24 |
+
champ[lin] = []
|
| 25 |
+
for g in gens:
|
| 26 |
+
gr = [r for r in lr if r["gen"] == g]
|
| 27 |
+
assert len(gr) == 4, (lin, g, len(gr))
|
| 28 |
+
c = min(gr, key=lambda r: r["bpb"])
|
| 29 |
+
best[lin].append(round(c["bpb"], 4))
|
| 30 |
+
champ[lin].append(c)
|
| 31 |
+
fresh[lin] = [r for r in lr if r["fresh"]]
|
| 32 |
+
|
| 33 |
+
# ---- claim 1: founder parity across the F1 pair (sign channel = healthy bed)
|
| 34 |
+
g0_gap = abs(best["sign_weights"][0] - best["sign_full"][0])
|
| 35 |
+
assert g0_gap <= 0.005, g0_gap
|
| 36 |
+
assert 2.40 < best["sign_weights"][0] < 2.50
|
| 37 |
+
|
| 38 |
+
# ---- claim 2: every inheriting lineage ascends monotonically (best-of-gen)
|
| 39 |
+
for lin in ("sign_weights", "sign_full", "tree_weights", "tree_full"):
|
| 40 |
+
bs = best[lin]
|
| 41 |
+
assert all(bs[i + 1] < bs[i] for i in range(3)), (lin, bs)
|
| 42 |
+
|
| 43 |
+
# ---- claim 3: germline directionally ahead 3/3 post-divergence gens on the
|
| 44 |
+
# sign channel, but margin stays at founder-luck scale (<= g0 gap + 0.002)
|
| 45 |
+
margins = [best["sign_weights"][g] - best["sign_full"][g] for g in (1, 2, 3)]
|
| 46 |
+
assert all(m > 0 for m in margins), margins
|
| 47 |
+
assert all(m <= g0_gap + 0.002 for m in margins), (margins, g0_gap)
|
| 48 |
+
|
| 49 |
+
# ---- claim 4: the fixed point — germline champion books stationary by gen 2,
|
| 50 |
+
# continuity champions drift ~0.2 every generation forever
|
| 51 |
+
for g in (2, 3):
|
| 52 |
+
assert champ["sign_full"][g]["content"]["drift"] < 0.02
|
| 53 |
+
assert champ["sign_weights"][g]["content"]["drift"] > 0.15
|
| 54 |
+
tc = champ["tree_full"][g]["content"]
|
| 55 |
+
assert tc["root_drift"] < 0.02 and max(tc["branch_drift"]) < 0.02
|
| 56 |
+
tw = champ["tree_weights"][g]["content"]
|
| 57 |
+
assert tw["root_drift"] > 0.1 or max(tw["branch_drift"]) > 0.15
|
| 58 |
+
|
| 59 |
+
# ---- claim 5: fidelity — content consumption cultivates it (heirs beat every
|
| 60 |
+
# fresh founder), and the germline locks it (near-zero inter-member spread)
|
| 61 |
+
for lin in ("sign_weights", "sign_full"):
|
| 62 |
+
heir_fids = [r["content"]["book_fid"] for r in rows
|
| 63 |
+
if r["lineage"] == lin and r["gen"] >= 1 and not r["fresh"]]
|
| 64 |
+
fresh_fids = [r["content"]["book_fid"] for r in fresh[lin]]
|
| 65 |
+
assert min(heir_fids) > max(fresh_fids), lin
|
| 66 |
+
g3_full = [r["content"]["book_fid"] for r in rows if r["lineage"] == "sign_full"
|
| 67 |
+
and r["gen"] == 3 and not r["fresh"]]
|
| 68 |
+
assert max(g3_full) - min(g3_full) < 0.001, g3_full
|
| 69 |
+
|
| 70 |
+
# ---- claim 6: tree routing — every fresh tree founder develops root imbalance
|
| 71 |
+
# (ppl4 < 3.2); lineage champions never do (ppl4 > 3.2, no dead branch)
|
| 72 |
+
for lin in ("tree_weights", "tree_full"):
|
| 73 |
+
for r in fresh[lin]:
|
| 74 |
+
assert r["content"]["root_ppl4"] < 3.2, r
|
| 75 |
+
for g in range(4):
|
| 76 |
+
c = champ[lin][g]["content"]
|
| 77 |
+
assert c["root_ppl4"] > 3.2 and min(c["root_usage"]) > 0.05, c
|
| 78 |
+
|
| 79 |
+
# ---- claim 7: branch revival belongs to continuity — the control's weakest
|
| 80 |
+
# branch usage rises g0->g3; the germline's stays frozen after fixation
|
| 81 |
+
tw0, tw3 = champ["tree_weights"][0]["content"], champ["tree_weights"][3]["content"]
|
| 82 |
+
assert min(tw3["root_usage"]) > min(tw0["root_usage"]) + 0.02
|
| 83 |
+
tf2, tf3 = champ["tree_full"][2]["content"], champ["tree_full"][3]["content"]
|
| 84 |
+
assert abs(min(tf3["root_usage"]) - min(tf2["root_usage"])) < 0.02
|
| 85 |
+
|
| 86 |
+
out = {"best_of_gen": best,
|
| 87 |
+
"g0_founder_gap_sign": round(g0_gap, 4),
|
| 88 |
+
"post_divergence_margins_sign": [round(m, 4) for m in margins],
|
| 89 |
+
"champion_drift": {lin: [c["content"].get("drift",
|
| 90 |
+
c["content"].get("root_drift")) for c in champ[lin]]
|
| 91 |
+
for lin in LINEAGES},
|
| 92 |
+
"champion_fid": {lin: [c["content"].get("book_fid") for c in champ[lin]]
|
| 93 |
+
for lin in ("sign_weights", "sign_full", "sign_none")},
|
| 94 |
+
"tree_root_ppl4": {lin: [c["content"]["root_ppl4"] for c in champ[lin]]
|
| 95 |
+
for lin in ("tree_weights", "tree_full")},
|
| 96 |
+
"n_rows": len(rows)}
|
| 97 |
+
os.makedirs(os.path.dirname(OUT), exist_ok=True)
|
| 98 |
+
json.dump(out, open(OUT, "w", encoding="utf-8"), indent=1)
|
| 99 |
+
print(f"{len(rows)} rows -> results/results.json")
|
| 100 |
+
print("all README claims asserted OK")
|
exp015_ch/exp014_genetic_distillation.py
ADDED
|
@@ -0,0 +1,515 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""exp014_genetic_distillation.py — genetic distillation + memory substrate.
|
| 2 |
+
14_a: multi-generational tournament (GM3 paradigm) where the aleph codebook is the
|
| 3 |
+
explicit heritable genome. Lineages: ALEPH-FLAT (consensus book + KD) |
|
| 4 |
+
ALEPH-TREE (structured genome: root book + branch books) | MLP-KD
|
| 5 |
+
(traditional: best-parent weights + KD) | NO-INHERIT (evolution floor).
|
| 6 |
+
Both sides intentionally inherit logits (KD); only ours inherits geometry.
|
| 7 |
+
Consensus = Procrustes/GPA alignment of parents' books to mean shape
|
| 8 |
+
(placement by construction — replaces GM3's k-means-on-consensus init).
|
| 9 |
+
14_b: memory substrate — the D=4 home makes books size-agnostic. Implant books
|
| 10 |
+
cultivated in a small organism into a larger one (frozen / trainable), and
|
| 11 |
+
into GPT-2 relay adapters (cross-architecture frozen distillation).
|
| 12 |
+
|
| 13 |
+
Riders: pure Adam wd=0; KD = KL to detached teacher probs (predictive pressure, no
|
| 14 |
+
contrastive); tree routing is DENSE SOFT (oriented weights; collapse monitor on the
|
| 15 |
+
root); drift-check precedes every freeze claim; GPU-only verdict runs; Colab-safe.
|
| 16 |
+
Founders share a COMMON-ANCESTOR book so GPA row correspondence is inherited.
|
| 17 |
+
|
| 18 |
+
Colab paste order: geolip_vitals.py -> ar_differentiation_bed.py ->
|
| 19 |
+
exp013_augmentation_bed.py (only for run_b2) -> this file.
|
| 20 |
+
"""
|
| 21 |
+
from __future__ import annotations
|
| 22 |
+
import copy
|
| 23 |
+
import json
|
| 24 |
+
import math
|
| 25 |
+
import os
|
| 26 |
+
import torch
|
| 27 |
+
import torch.nn as nn
|
| 28 |
+
import torch.nn.functional as F
|
| 29 |
+
|
| 30 |
+
if "anchor_drift" not in globals():
|
| 31 |
+
try:
|
| 32 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 33 |
+
except ImportError:
|
| 34 |
+
_here = globals().get("__file__")
|
| 35 |
+
if _here is None:
|
| 36 |
+
raise ImportError("paste/run geolip_vitals.py first")
|
| 37 |
+
import sys, pathlib
|
| 38 |
+
sys.path.insert(0, str(pathlib.Path(_here).parent))
|
| 39 |
+
from geolip_vitals import anchor_drift, axis_aliveness, path_diversity
|
| 40 |
+
if "ByteLM" not in globals():
|
| 41 |
+
try:
|
| 42 |
+
from ar_differentiation_bed import (ByteLM, AlephAddress, _wikitext_bytes,
|
| 43 |
+
_batch, VOCAB)
|
| 44 |
+
except ImportError:
|
| 45 |
+
raise ImportError("paste/run ar_differentiation_bed.py first")
|
| 46 |
+
|
| 47 |
+
DATA_ROOT = os.environ.get("GEOLIP_DATA", "./data")
|
| 48 |
+
EXP_DIR = os.path.join(DATA_ROOT, "exp014")
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
class SquaredReLU(nn.Module):
|
| 52 |
+
def forward(self, x):
|
| 53 |
+
return F.relu(x) ** 2
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
# ===================================================== consensus (the germline) ===
|
| 57 |
+
@torch.no_grad()
|
| 58 |
+
def procrustes_rotation(A: torch.Tensor, M: torch.Tensor) -> torch.Tensor:
|
| 59 |
+
"""Orthogonal R minimizing ||A R - M||_F (rows correspond)."""
|
| 60 |
+
U, _, Vt = torch.linalg.svd(A.T.double() @ M.double())
|
| 61 |
+
return (U @ Vt).float()
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
@torch.no_grad()
|
| 65 |
+
def align_to(A: torch.Tensor, ref: torch.Tensor, iters: int = 20) -> torch.Tensor:
|
| 66 |
+
"""Projective Procrustes (rows correspond, signs free): alternate the
|
| 67 |
+
orthogonal rotation and per-row sign flips (books live on RP^(D-1))."""
|
| 68 |
+
s = torch.ones(A.shape[0], 1)
|
| 69 |
+
for _ in range(iters):
|
| 70 |
+
R = procrustes_rotation(s * A, ref)
|
| 71 |
+
AR = (s * A) @ R
|
| 72 |
+
s_upd = torch.where((AR * ref).sum(-1, keepdim=True) < 0, -s, s)
|
| 73 |
+
if torch.equal(s_upd, s):
|
| 74 |
+
return AR
|
| 75 |
+
s = s_upd
|
| 76 |
+
return (s * A) @ procrustes_rotation(s * A, ref)
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
@torch.no_grad()
|
| 80 |
+
def consensus_codebook(books: list, iters: int = 50, tol: float = 1e-8):
|
| 81 |
+
"""GPA to mean shape (GM3 machinery, applied to aleph books), anchored to the
|
| 82 |
+
FIRST parent's frame. Rows must correspond (common-ancestor convention); signs
|
| 83 |
+
are projective. Returns (consensus, n_iters, delta)."""
|
| 84 |
+
# device-pin to CPU: parent models may live on CUDA after KD teacher moves
|
| 85 |
+
Bs = [F.normalize(b.detach().float().cpu(), dim=-1).clone() for b in books]
|
| 86 |
+
# pairwise projective alignment to parent-0's frame, THEN GPA refinement
|
| 87 |
+
aligned = [Bs[0]] + [align_to(b, Bs[0]) for b in Bs[1:]]
|
| 88 |
+
M = F.normalize(torch.stack(aligned).mean(0), dim=-1)
|
| 89 |
+
delta, it = 0.0, 0
|
| 90 |
+
for it in range(1, iters + 1):
|
| 91 |
+
aligned = [align_to(b, M) for b in Bs]
|
| 92 |
+
M_new = F.normalize(torch.stack(aligned).mean(0), dim=-1)
|
| 93 |
+
delta = (M_new - M).norm().item()
|
| 94 |
+
M = M_new
|
| 95 |
+
if delta < tol:
|
| 96 |
+
break
|
| 97 |
+
# re-anchor to parent-0 (GPA drift of the global frame stays measurable)
|
| 98 |
+
M = align_to(M, Bs[0])
|
| 99 |
+
return M, it, delta
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
@torch.no_grad()
|
| 103 |
+
def implant_book(addr: "AlephAddress", book: torch.Tensor, trainable: bool = True):
|
| 104 |
+
"""Load a book into an AlephAddress: codebook + home (drift measured from the
|
| 105 |
+
implant). Freeze only via trainable=False AFTER a drift-check justifies it."""
|
| 106 |
+
b = F.normalize(book.float(), dim=-1).to(addr.codebook.device)
|
| 107 |
+
assert b.shape == addr.codebook.shape, (b.shape, addr.codebook.shape)
|
| 108 |
+
addr.codebook.data.copy_(b)
|
| 109 |
+
addr.home.copy_(b)
|
| 110 |
+
addr.codebook.requires_grad_(trainable)
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
# ============================================================= tree head ==========
|
| 114 |
+
class TreeHead(nn.Module):
|
| 115 |
+
"""Autoregressive tree (the constellation-anchor analogue, exp011 TREE operator
|
| 116 |
+
in the healthy consumption regime): a ROOT aleph (K=2, D=4) yields the 4
|
| 117 |
+
oriented weights (2K half-axes = the 4 branches, dense soft, sums to 1);
|
| 118 |
+
each BRANCH is a 64-slot... shared slot projection read by a branch-specific
|
| 119 |
+
book (K=64, D=4); output = branch-weighted mixture of branch reads -> vocab.
|
| 120 |
+
Heritable genome: root book (2,4) + 4 branch books (64,4)."""
|
| 121 |
+
|
| 122 |
+
ROOT_SLOTS = 4 # slot-parallel root consumption (the collapse cure)
|
| 123 |
+
ROOT_TAU = 0.3 # softer root temperature (wave-1 fix: single hard-tau
|
| 124 |
+
# root partially collapsed, usage [.85,.12,.01,.02])
|
| 125 |
+
|
| 126 |
+
def __init__(self, d: int, vocab: int = 256, n_slots: int = 16):
|
| 127 |
+
super().__init__()
|
| 128 |
+
self.n_slots = n_slots
|
| 129 |
+
self.root_proj = nn.Linear(d, self.ROOT_SLOTS * 4, bias=False)
|
| 130 |
+
self.slot_proj = nn.Linear(d, n_slots * 4, bias=False)
|
| 131 |
+
nn.init.orthogonal_(self.root_proj.weight)
|
| 132 |
+
nn.init.orthogonal_(self.slot_proj.weight)
|
| 133 |
+
self.root = AlephAddress(2, 4, tau=self.ROOT_TAU)
|
| 134 |
+
self.branches = nn.ModuleList([AlephAddress(64, 4) for _ in range(4)])
|
| 135 |
+
self.out = nn.Linear(n_slots * 4, vocab, bias=True)
|
| 136 |
+
self._last_root = None
|
| 137 |
+
|
| 138 |
+
def forward(self, h):
|
| 139 |
+
B, n, _ = h.shape
|
| 140 |
+
rs = self.root_proj(h).view(B, n, self.ROOT_SLOTS, 4)
|
| 141 |
+
p, m = self.root.oriented(rs) # (B,n,S,2) x2
|
| 142 |
+
w = torch.cat([p, m], dim=-1).mean(dim=-2) # slot-avg -> (B,n,4)
|
| 143 |
+
self._last_root = w.detach()
|
| 144 |
+
slots = self.slot_proj(h).view(B, n, self.n_slots, 4)
|
| 145 |
+
mix = 0
|
| 146 |
+
for b, br in enumerate(self.branches):
|
| 147 |
+
mix = mix + w[..., b:b + 1] * br.m_hat(slots).reshape(B, n, -1)
|
| 148 |
+
return self.out(mix)
|
| 149 |
+
|
| 150 |
+
def genome(self):
|
| 151 |
+
return {"root": self.root.codebook.detach().clone(),
|
| 152 |
+
**{f"branch{i}": br.codebook.detach().clone()
|
| 153 |
+
for i, br in enumerate(self.branches)}}
|
| 154 |
+
|
| 155 |
+
@torch.no_grad()
|
| 156 |
+
def inherit(self, genomes: list):
|
| 157 |
+
c, it, dl = consensus_codebook([g["root"] for g in genomes])
|
| 158 |
+
implant_book(self.root, c)
|
| 159 |
+
for i, br in enumerate(self.branches):
|
| 160 |
+
c, _, _ = consensus_codebook([g[f"branch{i}"] for g in genomes])
|
| 161 |
+
implant_book(br, c)
|
| 162 |
+
|
| 163 |
+
@torch.no_grad()
|
| 164 |
+
def vitals(self):
|
| 165 |
+
out = {"root_drift": round(anchor_drift(self.root.codebook,
|
| 166 |
+
self.root.home)["mean"], 4)}
|
| 167 |
+
if self._last_root is not None:
|
| 168 |
+
w = self._last_root.reshape(-1, 4)
|
| 169 |
+
usage = w.mean(0)
|
| 170 |
+
usage = usage / usage.sum()
|
| 171 |
+
out["root_usage"] = [round(float(u), 3) for u in usage]
|
| 172 |
+
ent = -(usage.clamp_min(1e-9) * usage.clamp_min(1e-9).log()).sum()
|
| 173 |
+
out["root_ppl4"] = round(float(ent.exp()), 3)
|
| 174 |
+
d = [anchor_drift(br.codebook, br.home)["mean"] for br in self.branches]
|
| 175 |
+
out["branch_drift"] = [round(x, 3) for x in d]
|
| 176 |
+
return out
|
| 177 |
+
|
| 178 |
+
|
| 179 |
+
# ============================================================ organisms ===========
|
| 180 |
+
def make_organism(lineage: str, d: int = 192, layers: int = 4, block: int = 256,
|
| 181 |
+
seed: int = 0):
|
| 182 |
+
"""lineage in {aleph_flat, aleph_tree, mlp_kd, no_inherit}. no_inherit uses the
|
| 183 |
+
aleph_flat architecture (the control isolates INHERITANCE, not architecture)."""
|
| 184 |
+
torch.manual_seed(seed)
|
| 185 |
+
if lineage in ("aleph_flat", "no_inherit", "aleph_full", "aleph_weights"):
|
| 186 |
+
return ByteLM("addr_msl64", d=d, layers=layers, block=block)
|
| 187 |
+
if lineage == "aleph_tree":
|
| 188 |
+
m = ByteLM("sdpa", d=d, layers=layers, block=block)
|
| 189 |
+
m.head = TreeHead(d)
|
| 190 |
+
return m
|
| 191 |
+
if lineage == "mlp_kd":
|
| 192 |
+
m = ByteLM("sdpa", d=d, layers=layers, block=block)
|
| 193 |
+
m.head = nn.Sequential(nn.Linear(d, 224), SquaredReLU(),
|
| 194 |
+
nn.LayerNorm(224), nn.Linear(224, VOCAB))
|
| 195 |
+
return m
|
| 196 |
+
raise ValueError(lineage)
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
def genome_of(model):
|
| 200 |
+
"""The heritable organ = co-adapted (projection, book) pair(s). Books inherit
|
| 201 |
+
by CONSENSUS (the geometric germline); projections inherit from the BEST
|
| 202 |
+
parent (weight copy) — implanting a book against a random projection puts the
|
| 203 |
+
child below random init (campaign-v2 lesson)."""
|
| 204 |
+
if isinstance(model.head, TreeHead):
|
| 205 |
+
g = model.head.genome()
|
| 206 |
+
g["root_proj"] = model.head.root_proj.weight.detach().cpu().clone()
|
| 207 |
+
g["slot_proj"] = model.head.slot_proj.weight.detach().cpu().clone()
|
| 208 |
+
return g
|
| 209 |
+
if hasattr(model, "head_addr"):
|
| 210 |
+
return {"flat": model.head_addr.codebook.detach().cpu().clone(),
|
| 211 |
+
"proj": model.head_proj.weight.detach().cpu().clone()}
|
| 212 |
+
return None
|
| 213 |
+
|
| 214 |
+
|
| 215 |
+
@torch.no_grad()
|
| 216 |
+
def inherit_genome(model, genomes: list):
|
| 217 |
+
"""genomes[0] = the BEST parent (selection order matters)."""
|
| 218 |
+
if isinstance(model.head, TreeHead):
|
| 219 |
+
model.head.inherit(genomes)
|
| 220 |
+
model.head.root_proj.weight.copy_(genomes[0]["root_proj"].to(
|
| 221 |
+
model.head.root_proj.weight.device))
|
| 222 |
+
model.head.slot_proj.weight.copy_(genomes[0]["slot_proj"].to(
|
| 223 |
+
model.head.slot_proj.weight.device))
|
| 224 |
+
elif hasattr(model, "head_addr"):
|
| 225 |
+
c, it, dl = consensus_codebook([g["flat"] for g in genomes])
|
| 226 |
+
implant_book(model.head_addr, c)
|
| 227 |
+
model.head_proj.weight.copy_(genomes[0]["proj"].to(
|
| 228 |
+
model.head_proj.weight.device))
|
| 229 |
+
|
| 230 |
+
|
| 231 |
+
def organism_vitals(model):
|
| 232 |
+
if isinstance(model.head, TreeHead):
|
| 233 |
+
return model.head.vitals()
|
| 234 |
+
return model.vitals() if hasattr(model, "vitals") else {}
|
| 235 |
+
|
| 236 |
+
|
| 237 |
+
# ========================================================= train one member ======
|
| 238 |
+
def train_member(model, tr, va, steps=2000, batch=32, block=256, device="cuda",
|
| 239 |
+
seed=0, teachers=None, kd_alpha=1.0):
|
| 240 |
+
"""CE (+ KL to detached mean teacher probs when teachers given). Pure Adam."""
|
| 241 |
+
g = torch.Generator().manual_seed(seed)
|
| 242 |
+
model = model.to(device)
|
| 243 |
+
opt = torch.optim.Adam(model.parameters(), lr=3e-4, weight_decay=0.0)
|
| 244 |
+
if teachers:
|
| 245 |
+
teachers = [t.to(device).eval() for t in teachers]
|
| 246 |
+
for step in range(1, steps + 1):
|
| 247 |
+
x, y = _batch(tr, batch, block, device, g)
|
| 248 |
+
logits = model(x)
|
| 249 |
+
loss = F.cross_entropy(logits.reshape(-1, VOCAB), y.reshape(-1))
|
| 250 |
+
if teachers:
|
| 251 |
+
with torch.no_grad():
|
| 252 |
+
tp = torch.stack([F.softmax(t(x), -1) for t in teachers]).mean(0)
|
| 253 |
+
loss = loss + kd_alpha * F.kl_div(
|
| 254 |
+
F.log_softmax(logits, -1), tp, reduction="batchmean")
|
| 255 |
+
opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
|
| 256 |
+
model.eval()
|
| 257 |
+
with torch.no_grad():
|
| 258 |
+
ls = []
|
| 259 |
+
for _ in range(20):
|
| 260 |
+
xv, yv = _batch(va, batch, block, device, g)
|
| 261 |
+
ls.append(F.cross_entropy(model(xv).reshape(-1, VOCAB),
|
| 262 |
+
yv.reshape(-1)).item())
|
| 263 |
+
return sum(ls) / len(ls) / math.log(2) # bpb
|
| 264 |
+
|
| 265 |
+
|
| 266 |
+
# ============================================================ the tournament ======
|
| 267 |
+
def run_tournament(lineage: str, gens: int = 4, pop: int = 4, steps: int = 2000,
|
| 268 |
+
seed: int = 0, device: str = "cuda",
|
| 269 |
+
catastrophic_at: int | None = None):
|
| 270 |
+
"""One lineage, one tournament seed. Logs per-gen to the ledger; saves the
|
| 271 |
+
champion genome per generation. catastrophic_at=G injects a 0-step random
|
| 272 |
+
parent into the consensus at generation G (the GM3 robustness probe)."""
|
| 273 |
+
if not torch.cuda.is_available():
|
| 274 |
+
raise RuntimeError("verdict runs are GPU-only")
|
| 275 |
+
os.makedirs(EXP_DIR, exist_ok=True)
|
| 276 |
+
tr, va = _wikitext_bytes(DATA_ROOT)
|
| 277 |
+
ledger = open(os.path.join(EXP_DIR, "ledger.jsonl"), "a", encoding="utf-8")
|
| 278 |
+
# common ancestor: every founder book starts identical within a tournament
|
| 279 |
+
torch.manual_seed(9000 + seed)
|
| 280 |
+
ancestor = make_organism(lineage, seed=9000 + seed)
|
| 281 |
+
anc_genome = genome_of(ancestor)
|
| 282 |
+
parents, parent_models, champion_genomes = [], [], []
|
| 283 |
+
for gen in range(gens):
|
| 284 |
+
members = []
|
| 285 |
+
for i in range(pop):
|
| 286 |
+
mseed = seed * 1000 + gen * 100 + i
|
| 287 |
+
m = make_organism(lineage, seed=mseed)
|
| 288 |
+
if anc_genome and gen == 0:
|
| 289 |
+
inherit_genome(m, [anc_genome]) # common ancestor
|
| 290 |
+
if gen > 0:
|
| 291 |
+
is_fresh = (i == pop - 1) # gene flow founder
|
| 292 |
+
if not is_fresh:
|
| 293 |
+
if lineage in ("aleph_flat", "aleph_tree"):
|
| 294 |
+
gs = [genome_of(pm) for pm in parent_models]
|
| 295 |
+
if catastrophic_at == gen:
|
| 296 |
+
bad = make_organism(lineage, seed=666 + i)
|
| 297 |
+
gs = gs + [genome_of(bad)]
|
| 298 |
+
inherit_genome(m, gs)
|
| 299 |
+
elif lineage == "aleph_full":
|
| 300 |
+
# v4 arm (v3 lesson: continuity is what pays) — inherit the
|
| 301 |
+
# WHOLE best parent, then overwrite the book with the
|
| 302 |
+
# two-parent consensus: germline ON TOP of continuity.
|
| 303 |
+
m.load_state_dict(copy.deepcopy(
|
| 304 |
+
parent_models[0].state_dict()))
|
| 305 |
+
gs = [genome_of(pm) for pm in parent_models]
|
| 306 |
+
if catastrophic_at == gen:
|
| 307 |
+
bad = make_organism(lineage, seed=666 + i)
|
| 308 |
+
gs = gs + [genome_of(bad)]
|
| 309 |
+
c, _, _ = consensus_codebook([g["flat"] for g in gs])
|
| 310 |
+
implant_book(m.head_addr, c)
|
| 311 |
+
elif lineage in ("mlp_kd", "aleph_weights"):
|
| 312 |
+
# pure continuity (no germline op) — aleph_weights is the
|
| 313 |
+
# within-architecture control for aleph_full
|
| 314 |
+
m.load_state_dict(copy.deepcopy(
|
| 315 |
+
parent_models[0].state_dict()))
|
| 316 |
+
# no_inherit: nothing
|
| 317 |
+
# KD: alpha 0.25 (campaign-v2 lesson: alpha=1.0 from near-parity
|
| 318 |
+
# teachers COMPOUNDS DOWNWARD — inverse evolution; fresh-founder
|
| 319 |
+
# control isolated it). Fresh founders get NO KD (clean gene flow).
|
| 320 |
+
is_fresh_now = (gen > 0 and i == pop - 1)
|
| 321 |
+
teachers = parent_models if (gen > 0 and not is_fresh_now
|
| 322 |
+
and lineage != "no_inherit") else None
|
| 323 |
+
bpb = train_member(m, tr, va, steps=steps, device=device,
|
| 324 |
+
seed=mseed, teachers=teachers, kd_alpha=0.25)
|
| 325 |
+
vit = organism_vitals(m)
|
| 326 |
+
members.append((bpb, m))
|
| 327 |
+
rec = {"exp": "14a", "lineage": lineage, "tseed": seed, "gen": gen,
|
| 328 |
+
"member": i, "fresh": gen > 0 and i == pop - 1,
|
| 329 |
+
"catastrophic": catastrophic_at == gen, "bpb": round(bpb, 4),
|
| 330 |
+
"vitals": vit}
|
| 331 |
+
ledger.write(json.dumps(rec) + "\n"); ledger.flush()
|
| 332 |
+
print(f"[14a {lineage} t{seed} g{gen} m{i}] bpb={bpb:.4f} {vit}",
|
| 333 |
+
flush=True)
|
| 334 |
+
members.sort(key=lambda t: t[0])
|
| 335 |
+
parent_models = [members[0][1].cpu(), members[1][1].cpu()]
|
| 336 |
+
best = members[0][0]
|
| 337 |
+
gene = genome_of(members[0][1])
|
| 338 |
+
if gene:
|
| 339 |
+
torch.save(gene, os.path.join(
|
| 340 |
+
EXP_DIR, f"champion_{lineage}_t{seed}_g{gen}.pt"))
|
| 341 |
+
champion_genomes.append(gene)
|
| 342 |
+
print(f"[14a {lineage} t{seed} g{gen}] BEST={best:.4f} "
|
| 343 |
+
f"mean={sum(b for b, _ in members)/pop:.4f}", flush=True)
|
| 344 |
+
for _, mm in members[2:]:
|
| 345 |
+
del mm
|
| 346 |
+
torch.cuda.empty_cache()
|
| 347 |
+
ledger.close()
|
| 348 |
+
return best
|
| 349 |
+
|
| 350 |
+
|
| 351 |
+
# ============================================================ 14_b implants ======
|
| 352 |
+
def run_b1(steps: int = 2000, seed: int = 0, device: str = "cuda",
|
| 353 |
+
donor_book: torch.Tensor | None = None, tag: str = "small_cultivated"):
|
| 354 |
+
"""Cross-size: donor book (default: cultivate in a small organism) implanted
|
| 355 |
+
into a LARGE organism. Arms: fresh | implant-trainable | implant-frozen | mlp."""
|
| 356 |
+
if not torch.cuda.is_available():
|
| 357 |
+
raise RuntimeError("verdict runs are GPU-only")
|
| 358 |
+
os.makedirs(EXP_DIR, exist_ok=True)
|
| 359 |
+
tr, va = _wikitext_bytes(DATA_ROOT)
|
| 360 |
+
ledger = open(os.path.join(EXP_DIR, "ledger.jsonl"), "a", encoding="utf-8")
|
| 361 |
+
if donor_book is None:
|
| 362 |
+
small = make_organism("aleph_flat", d=128, layers=2, seed=seed)
|
| 363 |
+
bpb_small = train_member(small, tr, va, steps=steps, device=device, seed=seed)
|
| 364 |
+
donor_book = genome_of(small)["flat"]
|
| 365 |
+
print(f"[14b donor small] bpb={bpb_small:.4f}", flush=True)
|
| 366 |
+
results = {}
|
| 367 |
+
for arm in ("fresh", "implant_train", "implant_frozen", "mlp"):
|
| 368 |
+
lineage = "mlp_kd" if arm == "mlp" else "aleph_flat"
|
| 369 |
+
m = make_organism(lineage, d=384, layers=6, seed=seed + 10)
|
| 370 |
+
if arm.startswith("implant"):
|
| 371 |
+
implant_book(m.head_addr, donor_book, trainable=(arm == "implant_train"))
|
| 372 |
+
bpb = train_member(m, tr, va, steps=steps, device=device, seed=seed + 10)
|
| 373 |
+
vit = organism_vitals(m)
|
| 374 |
+
results[arm] = {"bpb": round(bpb, 4), "vitals": vit}
|
| 375 |
+
rec = {"exp": "14b1", "arm": arm, "donor": tag, "seed": seed,
|
| 376 |
+
"bpb": round(bpb, 4), "vitals": vit}
|
| 377 |
+
ledger.write(json.dumps(rec) + "\n"); ledger.flush()
|
| 378 |
+
print(f"[14b1 {arm} donor={tag}] bpb={bpb:.4f} {vit}", flush=True)
|
| 379 |
+
del m; torch.cuda.empty_cache()
|
| 380 |
+
ledger.close()
|
| 381 |
+
return results, donor_book
|
| 382 |
+
|
| 383 |
+
|
| 384 |
+
def run_b2(donor_book: torch.Tensor, steps: int = 1500, seed: int = 0,
|
| 385 |
+
device: str = "cuda", tag: str = "small_cultivated"):
|
| 386 |
+
"""Cross-architecture: implant the donor book into every GPT-2 relay adapter
|
| 387 |
+
(exp013 Track C bed) vs random-init relays. Books are (64,4) — size-agnostic."""
|
| 388 |
+
from exp013_augmentation_bed import _wikitext_lines
|
| 389 |
+
from transformers import GPT2LMHeadModel, GPT2TokenizerFast
|
| 390 |
+
if "MslRelay" not in globals():
|
| 391 |
+
from ar_differentiation_bed import MslRelay
|
| 392 |
+
from exp013_augmentation_bed import _BlockWithAdapter
|
| 393 |
+
tok = GPT2TokenizerFast.from_pretrained("gpt2")
|
| 394 |
+
tr_lines, va_lines = _wikitext_lines(DATA_ROOT)
|
| 395 |
+
stream_tr = tok("\n\n".join(tr_lines[:8000]), return_tensors="pt").input_ids[0]
|
| 396 |
+
stream_va = tok("\n\n".join(va_lines[:1000]), return_tensors="pt").input_ids[0]
|
| 397 |
+
ledger = open(os.path.join(EXP_DIR, "ledger.jsonl"), "a", encoding="utf-8")
|
| 398 |
+
out = {}
|
| 399 |
+
for arm in ("random_relays", "implanted_relays"):
|
| 400 |
+
torch.manual_seed(seed)
|
| 401 |
+
g = torch.Generator().manual_seed(seed)
|
| 402 |
+
model = GPT2LMHeadModel.from_pretrained("gpt2").to(device)
|
| 403 |
+
for p in model.parameters():
|
| 404 |
+
p.requires_grad_(False)
|
| 405 |
+
adapters = []
|
| 406 |
+
for i, blk in enumerate(model.transformer.h):
|
| 407 |
+
ad = MslRelay(model.config.n_embd).to(device)
|
| 408 |
+
if arm == "implanted_relays":
|
| 409 |
+
implant_book(ad.addr, donor_book, trainable=True)
|
| 410 |
+
model.transformer.h[i] = _BlockWithAdapter(blk, ad)
|
| 411 |
+
adapters.append(ad)
|
| 412 |
+
params = [p for ad in adapters for p in ad.parameters()
|
| 413 |
+
if p.requires_grad]
|
| 414 |
+
opt = torch.optim.Adam(params, lr=1e-3, weight_decay=0.0)
|
| 415 |
+
block = 256
|
| 416 |
+
for step in range(1, steps + 1):
|
| 417 |
+
ix = torch.randint(0, stream_tr.numel() - block - 1, (8,), generator=g)
|
| 418 |
+
x = torch.stack([stream_tr[i:i + block] for i in ix]).to(device)
|
| 419 |
+
loss = model(x, labels=x).loss
|
| 420 |
+
opt.zero_grad(set_to_none=True); loss.backward(); opt.step()
|
| 421 |
+
model.eval()
|
| 422 |
+
with torch.no_grad():
|
| 423 |
+
ls = []
|
| 424 |
+
for j in range(0, stream_va.numel() - block - 1, block * 4):
|
| 425 |
+
x = stream_va[j:j + block].unsqueeze(0).to(device)
|
| 426 |
+
ls.append(model(x, labels=x).loss.item())
|
| 427 |
+
ppl = math.exp(sum(ls) / len(ls))
|
| 428 |
+
gates = [round(ad.gate.sigmoid().item(), 4) for ad in adapters]
|
| 429 |
+
drifts = [round(anchor_drift(ad.addr.codebook, ad.addr.home)["mean"], 3)
|
| 430 |
+
for ad in adapters]
|
| 431 |
+
out[arm] = {"ppl": round(ppl, 3), "gates": gates, "drift": drifts}
|
| 432 |
+
rec = {"exp": "14b2", "arm": arm, "donor": tag, "seed": seed,
|
| 433 |
+
"ppl": round(ppl, 3), "gates": gates, "drift": drifts}
|
| 434 |
+
ledger.write(json.dumps(rec) + "\n"); ledger.flush()
|
| 435 |
+
print(f"[14b2 {arm} donor={tag}] ppl={ppl:.3f} gates={gates[:3]}.. "
|
| 436 |
+
f"drift={drifts[:3]}..", flush=True)
|
| 437 |
+
del model; torch.cuda.empty_cache()
|
| 438 |
+
ledger.close()
|
| 439 |
+
return out
|
| 440 |
+
|
| 441 |
+
|
| 442 |
+
# ================================================================ smoke ===========
|
| 443 |
+
def smoke():
|
| 444 |
+
"""CPU shapes/parse only: GPA ground truth, tree causality, implant, KD path."""
|
| 445 |
+
g = torch.Generator().manual_seed(0)
|
| 446 |
+
# GPA: two rotated (+row-sign-flipped) copies of one book must align back to it
|
| 447 |
+
A = F.normalize(torch.randn(64, 4, generator=g), dim=-1)
|
| 448 |
+
q, _ = torch.linalg.qr(torch.randn(4, 4, generator=g))
|
| 449 |
+
B = A @ q
|
| 450 |
+
B[::3] = -B[::3]
|
| 451 |
+
C, it, dl = consensus_codebook([A, B])
|
| 452 |
+
cos = (F.normalize(C, dim=-1) * A).sum(-1).abs().mean()
|
| 453 |
+
assert cos > 0.999, cos
|
| 454 |
+
print(f"GPA OK (iters={it}, delta={dl:.2e}, |cos to truth|={cos:.5f})")
|
| 455 |
+
x = torch.randint(0, 256, (2, 64))
|
| 456 |
+
for lineage in ("aleph_flat", "aleph_tree", "mlp_kd", "no_inherit"):
|
| 457 |
+
m = make_organism(lineage, d=96, layers=2, block=64, seed=0)
|
| 458 |
+
lg = m(x); assert lg.shape == (2, 64, 256); lg.sum().backward()
|
| 459 |
+
with torch.no_grad():
|
| 460 |
+
a = m(x)[0, 10]; x2 = x.clone(); x2[0, 40] = (x2[0, 40] + 7) % 256
|
| 461 |
+
b = m(x2)[0, 10]
|
| 462 |
+
assert torch.allclose(a, b, atol=1e-4), lineage + " leaks"
|
| 463 |
+
gnm = genome_of(m)
|
| 464 |
+
if gnm:
|
| 465 |
+
inherit_genome(m, [gnm, gnm]) # self-consensus = identity-ish
|
| 466 |
+
print(lineage, "OK params",
|
| 467 |
+
f"{sum(p.numel() for p in m.parameters()):,}",
|
| 468 |
+
organism_vitals(m) if lineage != "mlp_kd" else {})
|
| 469 |
+
# KD path: teacher forward + KL backward
|
| 470 |
+
t = make_organism("mlp_kd", d=96, layers=2, block=64, seed=1)
|
| 471 |
+
s = make_organism("aleph_flat", d=96, layers=2, block=64, seed=2)
|
| 472 |
+
tp = F.softmax(t(x), -1).detach()
|
| 473 |
+
loss = F.kl_div(F.log_softmax(s(x), -1), tp, reduction="batchmean")
|
| 474 |
+
loss.backward()
|
| 475 |
+
print("KD OK — exp014 smoke passed (tournament on GPU: run_tournament(...))")
|
| 476 |
+
|
| 477 |
+
|
| 478 |
+
def _in_notebook():
|
| 479 |
+
try:
|
| 480 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 481 |
+
return True
|
| 482 |
+
except NameError:
|
| 483 |
+
return False
|
| 484 |
+
|
| 485 |
+
|
| 486 |
+
if __name__ == "__main__":
|
| 487 |
+
if _in_notebook():
|
| 488 |
+
smoke()
|
| 489 |
+
print("Notebook: run_tournament('aleph_flat'), run_b1(), run_b2(book).")
|
| 490 |
+
else:
|
| 491 |
+
import argparse
|
| 492 |
+
ap = argparse.ArgumentParser()
|
| 493 |
+
ap.add_argument("--mode", default="smoke",
|
| 494 |
+
choices=["smoke", "tournament", "b1", "b2"])
|
| 495 |
+
ap.add_argument("--lineage", default="aleph_full",
|
| 496 |
+
help="tournament lineage: aleph_flat|aleph_full|"
|
| 497 |
+
"aleph_weights|aleph_tree|mlp_kd|no_inherit")
|
| 498 |
+
ap.add_argument("--seed", type=int, default=0)
|
| 499 |
+
ap.add_argument("--steps", type=int, default=2000)
|
| 500 |
+
ap.add_argument("--genome", default="genomes/champion_aleph_full_t0_g3.pt",
|
| 501 |
+
help="donor genome .pt for --mode b1/b2 (uses its 'flat' book)")
|
| 502 |
+
a, _ = ap.parse_known_args()
|
| 503 |
+
if a.mode == "smoke":
|
| 504 |
+
smoke()
|
| 505 |
+
elif a.mode == "tournament":
|
| 506 |
+
run_tournament(a.lineage, steps=a.steps, seed=a.seed)
|
| 507 |
+
elif a.mode == "b1":
|
| 508 |
+
donor = (torch.load(a.genome, map_location="cpu")["flat"]
|
| 509 |
+
if os.path.exists(a.genome) else None)
|
| 510 |
+
run_b1(steps=a.steps, seed=a.seed, donor_book=donor,
|
| 511 |
+
tag=os.path.basename(a.genome) if donor is not None
|
| 512 |
+
else "small_cultivated")
|
| 513 |
+
elif a.mode == "b2":
|
| 514 |
+
donor = torch.load(a.genome, map_location="cpu")["flat"]
|
| 515 |
+
run_b2(donor, seed=a.seed, tag=os.path.basename(a.genome))
|
exp015_ch/exp015_content_heredity.py
ADDED
|
@@ -0,0 +1,188 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""exp015_content_heredity.py — does the germline pay when CONTENT is load-bearing?
|
| 2 |
+
Composes exp014-A (content-bearing heredity) + exp014-C (the tree in the continuity
|
| 3 |
+
regime). exp014's verdict: near-uniform books are near-interchangeable scaffolds —
|
| 4 |
+
the consensus germline was task-neutral because the soft task only consumed book
|
| 5 |
+
HEALTH. exp015 changes the consumption: organisms whose prediction channel consumes
|
| 6 |
+
book IDENTITY (the sign-code head: features are literally ±A[win] rows), plus the
|
| 7 |
+
TreeHead lineage finally tested with fair (full-weight) inheritance.
|
| 8 |
+
|
| 9 |
+
Lineages (all full-weight continuity; the exp014 lesson is baked in):
|
| 10 |
+
sign_weights — sign-code organism, pure continuity (control)
|
| 11 |
+
sign_full — sign-code organism, continuity + consensus-book overwrite
|
| 12 |
+
tree_weights — tree organism, pure continuity (control)
|
| 13 |
+
tree_full — tree organism, continuity + structured consensus (root+branches)
|
| 14 |
+
sign_none — sign-code no-inheritance floor
|
| 15 |
+
Per-member gauges: val bpb + CONTENT GAUGE (sign-code metric fidelity: Spearman of
|
| 16 |
+
book Hamming distance vs true angle — the LSH faithfulness of the cultivated book)
|
| 17 |
+
+ standard vitals; tree lineages log root usage (branch-revival watch).
|
| 18 |
+
Preregistered forks:
|
| 19 |
+
F1 sign_full > sign_weights on bpb (beyond seed noise) => the germline converts
|
| 20 |
+
its stability (the exp014 finding) into VALUE under content consumption.
|
| 21 |
+
Tie => content
|
| 22 |
+
consumption at this scale still doesn't price book identity — the compass
|
| 23 |
+
points to content TASKS (retrieval/spelling) rather than content CHANNELS.
|
| 24 |
+
F2 tree_full vs tree_weights: does structure inherit under fair conditions; do
|
| 25 |
+
weak branches revive across generations (root usage ppl trajectory)?
|
| 26 |
+
F3 fidelity trajectory: do germline lineages preserve LSH fidelity better than
|
| 27 |
+
continuity lineages across generations (stability -> measurable content)?
|
| 28 |
+
Riders: pure Adam wd=0; KD alpha 0.25, never on founders; drift-check before any
|
| 29 |
+
freeze; Colab-safe. Paste order: geolip_vitals -> ar_differentiation_bed ->
|
| 30 |
+
exp014_genetic_distillation -> this file.
|
| 31 |
+
"""
|
| 32 |
+
from __future__ import annotations
|
| 33 |
+
import copy
|
| 34 |
+
import json
|
| 35 |
+
import os
|
| 36 |
+
import torch
|
| 37 |
+
import torch.nn.functional as F
|
| 38 |
+
|
| 39 |
+
if "run_tournament" not in globals():
|
| 40 |
+
try:
|
| 41 |
+
from exp014_genetic_distillation import (
|
| 42 |
+
TreeHead, consensus_codebook, implant_book, train_member,
|
| 43 |
+
genome_of, inherit_genome, EXP_DIR, DATA_ROOT)
|
| 44 |
+
from ar_differentiation_bed import ByteLM, _wikitext_bytes
|
| 45 |
+
from geolip_vitals import anchor_drift
|
| 46 |
+
except ImportError:
|
| 47 |
+
_here = globals().get("__file__")
|
| 48 |
+
if _here is None:
|
| 49 |
+
raise ImportError("paste exp014_genetic_distillation.py (and its deps) first")
|
| 50 |
+
import sys, pathlib
|
| 51 |
+
sys.path.insert(0, str(pathlib.Path(_here).parent))
|
| 52 |
+
from exp014_genetic_distillation import (
|
| 53 |
+
TreeHead, consensus_codebook, implant_book, train_member,
|
| 54 |
+
genome_of, inherit_genome, EXP_DIR, DATA_ROOT)
|
| 55 |
+
from ar_differentiation_bed import ByteLM, _wikitext_bytes
|
| 56 |
+
from geolip_vitals import anchor_drift
|
| 57 |
+
|
| 58 |
+
EXP15_DIR = os.path.join(os.path.dirname(EXP_DIR), "exp015")
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
# ---------------------------------------------------------------- content gauge ---
|
| 62 |
+
@torch.no_grad()
|
| 63 |
+
def sign_fidelity(book: torch.Tensor, n: int = 3000, seed: int = 0) -> float:
|
| 64 |
+
"""LSH faithfulness of the book: Spearman(sign-code Hamming, true angle) over
|
| 65 |
+
random pairs on S^(D-1). The content gauge — measures whether the book's
|
| 66 |
+
IDENTITY still supports metric-faithful discrete reading."""
|
| 67 |
+
A = F.normalize(book.detach().float().cpu(), dim=-1)
|
| 68 |
+
g = torch.Generator().manual_seed(seed)
|
| 69 |
+
D = A.shape[1]
|
| 70 |
+
x = F.normalize(torch.randn(n, D, generator=g), dim=-1)
|
| 71 |
+
y = F.normalize(torch.randn(n, D, generator=g), dim=-1)
|
| 72 |
+
ang = torch.arccos((x * y).sum(-1).clamp(-1, 1))
|
| 73 |
+
ham = (torch.sign(x @ A.T) != torch.sign(y @ A.T)).float().mean(-1)
|
| 74 |
+
ra = ang.argsort().argsort().float()
|
| 75 |
+
rb = ham.argsort().argsort().float()
|
| 76 |
+
ra = (ra - ra.mean()) / ra.std()
|
| 77 |
+
rb = (rb - rb.mean()) / rb.std()
|
| 78 |
+
return round(float((ra * rb).mean()), 4)
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
def make_organism(lineage: str, d: int = 192, layers: int = 4, block: int = 256,
|
| 82 |
+
seed: int = 0):
|
| 83 |
+
torch.manual_seed(seed)
|
| 84 |
+
if lineage.startswith("sign"):
|
| 85 |
+
return ByteLM("addr_mslh64", d=d, layers=layers, block=block)
|
| 86 |
+
if lineage.startswith("tree"):
|
| 87 |
+
m = ByteLM("sdpa", d=d, layers=layers, block=block)
|
| 88 |
+
m.head = TreeHead(d)
|
| 89 |
+
return m
|
| 90 |
+
raise ValueError(lineage)
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def content_gauge(model) -> dict:
|
| 94 |
+
if isinstance(model.head, TreeHead):
|
| 95 |
+
out = {"root_fid": sign_fidelity(model.head.root.codebook)}
|
| 96 |
+
out["branch_fid"] = [sign_fidelity(b.codebook) for b in model.head.branches]
|
| 97 |
+
out.update(model.head.vitals())
|
| 98 |
+
return out
|
| 99 |
+
return {"book_fid": sign_fidelity(model.head_addr.codebook),
|
| 100 |
+
"drift": round(anchor_drift(model.head_addr.codebook,
|
| 101 |
+
model.head_addr.home)["mean"], 4)}
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
@torch.no_grad()
|
| 105 |
+
def inherit_15(model, lineage: str, parent_models: list, catastrophic: bool = False):
|
| 106 |
+
"""Full-weight continuity from the best parent; *_full adds the consensus
|
| 107 |
+
overwrite (flat book, or root+branches for trees)."""
|
| 108 |
+
model.load_state_dict(copy.deepcopy(parent_models[0].state_dict()))
|
| 109 |
+
if lineage.endswith("_full"):
|
| 110 |
+
gs = [genome_of(pm) for pm in parent_models]
|
| 111 |
+
if catastrophic:
|
| 112 |
+
gs = gs + [genome_of(make_organism(lineage, seed=666))]
|
| 113 |
+
if isinstance(model.head, TreeHead):
|
| 114 |
+
model.head.inherit(gs) # consensus per root/branch book
|
| 115 |
+
else:
|
| 116 |
+
c, _, _ = consensus_codebook([g["flat"] for g in gs])
|
| 117 |
+
implant_book(model.head_addr, c)
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
def run_tournament_15(lineage: str, gens: int = 4, pop: int = 4, steps: int = 2000,
|
| 121 |
+
seed: int = 0, device: str = "cuda"):
|
| 122 |
+
if not torch.cuda.is_available():
|
| 123 |
+
raise RuntimeError("verdict runs are GPU-only")
|
| 124 |
+
os.makedirs(EXP15_DIR, exist_ok=True)
|
| 125 |
+
tr, va = _wikitext_bytes(DATA_ROOT)
|
| 126 |
+
ledger = open(os.path.join(EXP15_DIR, "ledger.jsonl"), "a", encoding="utf-8")
|
| 127 |
+
torch.manual_seed(9000 + seed)
|
| 128 |
+
ancestor = make_organism(lineage, seed=9000 + seed)
|
| 129 |
+
anc = genome_of(ancestor)
|
| 130 |
+
parent_models = []
|
| 131 |
+
for gen in range(gens):
|
| 132 |
+
members = []
|
| 133 |
+
for i in range(pop):
|
| 134 |
+
mseed = seed * 1000 + gen * 100 + i
|
| 135 |
+
m = make_organism(lineage, seed=mseed)
|
| 136 |
+
if gen == 0 and anc and lineage != "sign_none":
|
| 137 |
+
inherit_genome(m, [anc]) # common ancestor (books + proj)
|
| 138 |
+
is_fresh = (gen > 0 and i == pop - 1)
|
| 139 |
+
if gen > 0 and not is_fresh and lineage != "sign_none":
|
| 140 |
+
inherit_15(m, lineage, parent_models)
|
| 141 |
+
teachers = parent_models if (gen > 0 and not is_fresh
|
| 142 |
+
and lineage != "sign_none") else None
|
| 143 |
+
bpb = train_member(m, tr, va, steps=steps, device=device,
|
| 144 |
+
seed=mseed, teachers=teachers, kd_alpha=0.25)
|
| 145 |
+
cg = content_gauge(m)
|
| 146 |
+
members.append((bpb, m))
|
| 147 |
+
rec = {"exp": "15", "lineage": lineage, "tseed": seed, "gen": gen,
|
| 148 |
+
"member": i, "fresh": is_fresh, "bpb": round(bpb, 4),
|
| 149 |
+
"content": cg}
|
| 150 |
+
ledger.write(json.dumps(rec) + "\n"); ledger.flush()
|
| 151 |
+
print(f"[15 {lineage} t{seed} g{gen} m{i}] bpb={bpb:.4f} {cg}",
|
| 152 |
+
flush=True)
|
| 153 |
+
members.sort(key=lambda t: t[0])
|
| 154 |
+
parent_models = [members[0][1].cpu(), members[1][1].cpu()]
|
| 155 |
+
gene = genome_of(members[0][1])
|
| 156 |
+
if gene:
|
| 157 |
+
torch.save(gene, os.path.join(
|
| 158 |
+
EXP15_DIR, f"champion_{lineage}_t{seed}_g{gen}.pt"))
|
| 159 |
+
print(f"[15 {lineage} t{seed} g{gen}] BEST={members[0][0]:.4f} "
|
| 160 |
+
f"mean={sum(b for b, _ in members)/pop:.4f}", flush=True)
|
| 161 |
+
for _, mm in members[2:]:
|
| 162 |
+
del mm
|
| 163 |
+
torch.cuda.empty_cache()
|
| 164 |
+
ledger.close()
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
def smoke():
|
| 168 |
+
x = torch.randint(0, 256, (2, 64))
|
| 169 |
+
for lin in ("sign_weights", "sign_full", "tree_weights", "tree_full"):
|
| 170 |
+
m = make_organism(lin, d=96, layers=2, block=64, seed=0)
|
| 171 |
+
lg = m(x); assert lg.shape == (2, 64, 256); lg.sum().backward()
|
| 172 |
+
p = make_organism(lin, d=96, layers=2, block=64, seed=1)
|
| 173 |
+
inherit_15(m, lin, [p, p])
|
| 174 |
+
print(lin, "OK", content_gauge(m))
|
| 175 |
+
print("exp015 smoke passed")
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
def _in_notebook():
|
| 179 |
+
try:
|
| 180 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 181 |
+
return True
|
| 182 |
+
except NameError:
|
| 183 |
+
return False
|
| 184 |
+
|
| 185 |
+
|
| 186 |
+
if __name__ == "__main__":
|
| 187 |
+
smoke() if not _in_notebook() else (smoke(),
|
| 188 |
+
print("Notebook: run_tournament_15('sign_full') etc."))
|
exp015_ch/genomes/champion_sign_full_t0_g0.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4f9fcc02245b1db9a8c5220866d6fdda1ca309780c22af8b4ad4984ddd39f93f
|
| 3 |
+
size 199725
|
exp015_ch/genomes/champion_sign_full_t0_g1.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:388462b1d3c8783a42ebd39d4a65c868d973c2dba60c19b1b5a36bee3e2b5a04
|
| 3 |
+
size 199725
|
exp015_ch/genomes/champion_sign_full_t0_g2.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e56719b48af127ce91e1ab0108a626fe7aac91499faa0b6434425bb76013683d
|
| 3 |
+
size 199725
|
exp015_ch/genomes/champion_sign_full_t0_g3.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:badd14c15e7aa3556331e049a04f7fa694c26268cf1347b29debe2ad07ede3df
|
| 3 |
+
size 199725
|
exp015_ch/genomes/champion_sign_none_t0_g0.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e897f10aeac8368f22dd5d6ba855b6e2efa4c50dd40b0ea4fced5e2fa4fea01e
|
| 3 |
+
size 199725
|
exp015_ch/genomes/champion_sign_none_t0_g1.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a7e95d7794c88671d75b8ae281a61f2c3b41965435833165de87b57e35cfc56b
|
| 3 |
+
size 199725
|
exp015_ch/genomes/champion_sign_none_t0_g2.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:19390461912c494057ca5e5d64b7ae983f5214f30d5e14d625e3004bc5c2fb70
|
| 3 |
+
size 199725
|
exp015_ch/genomes/champion_sign_none_t0_g3.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e8ea6e4f1273e584ead14992f68de4031f03d809edfb793eca3a7d74739170ee
|
| 3 |
+
size 199725
|
exp015_ch/genomes/champion_sign_weights_t0_g0.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:03248eace555307f81578ae7e55d92525592cd067e8dbfcdee278af9f8974ba3
|
| 3 |
+
size 199749
|
exp015_ch/genomes/champion_sign_weights_t0_g1.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8f59a8daaec93d9e92f7b858a1f76a9c6be5b89f953b703a618c45c897935d98
|
| 3 |
+
size 199749
|
exp015_ch/genomes/champion_sign_weights_t0_g2.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9a2e0a6271da786bf7f584e0453714d5ecd28ed7c72136599909290c787e1a11
|
| 3 |
+
size 199749
|
exp015_ch/genomes/champion_sign_weights_t0_g3.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:53520c5375389de407eedae2738480e4d254a9738efee2d225a91401d0ca9309
|
| 3 |
+
size 199749
|
exp015_ch/genomes/champion_tree_full_t0_g0.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fccf1f3a451fbdf11a2e926ba86c482806fb515df89a101ce7d3212b50516bf3
|
| 3 |
+
size 68974
|
exp015_ch/genomes/champion_tree_full_t0_g1.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:943674da063964852f3dfe107c72d0389aa68da7d7b6b7246ba0a95e5556d6df
|
| 3 |
+
size 68974
|
exp015_ch/genomes/champion_tree_full_t0_g2.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:44b16a514ead65a4985ef9792ec9d2a8ebf70088858f73d310008400b35269a1
|
| 3 |
+
size 68974
|
exp015_ch/genomes/champion_tree_full_t0_g3.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7ceabd944893b39a9948eb4d6c208e2699d38985d410aa52e6b9ccd1ec867003
|
| 3 |
+
size 68974
|
exp015_ch/genomes/champion_tree_weights_t0_g0.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:93ddee07fc38b78e00e10bb1f30a591662acdb60cdf7b180f2d854e786859a99
|
| 3 |
+
size 69013
|
exp015_ch/genomes/champion_tree_weights_t0_g1.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ce2d1e44c1cd62619b49432e9572ffbe9885021729df9e2e56bc562e97fc75aa
|
| 3 |
+
size 69013
|
exp015_ch/genomes/champion_tree_weights_t0_g2.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ef06cf1c38a26a14cb02cbd5048e14fb34d250d9a8d5d64256684fbfea1dca23
|
| 3 |
+
size 69013
|
exp015_ch/genomes/champion_tree_weights_t0_g3.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9b3ff71e3a61ae4ad42e40bfcdd86670a9c4aa8b6a72cc87e13813056ddcd32b
|
| 3 |
+
size 69013
|
exp015_ch/geolip_vitals.py
ADDED
|
@@ -0,0 +1,184 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""geolip_vitals.py — shared diagnostic harness for the GeoLIP aleph experiments.
|
| 2 |
+
ALL functions are READOUTS: no gradients, no losses. CV is a readout, never a
|
| 3 |
+
force. Addressing is judged by drift->0.29154 and CV->0.20, never by recon cosine
|
| 4 |
+
(judgment criteria per the aleph-void article: https://huggingface.co/blog/AbstractPhil/geometric-vocabulary-patchwork-aleph-void).
|
| 5 |
+
|
| 6 |
+
Vitals provided:
|
| 7 |
+
anchor_drift — geodesic drift of anchors from init; binding fraction @0.29154
|
| 8 |
+
pentachoron_cv — CM 4-volume CV over random 5-row subsets (geovocab2 import)
|
| 9 |
+
axis_aliveness — oriented-address usage: axes alive, hppl, collapse flag
|
| 10 |
+
gate_stats — gate means vs the 0.012-0.03 band
|
| 11 |
+
path_diversity — unique-path counting, FIXED high-bits hash (low-16 bug is the
|
| 12 |
+
retracted artifact — never use the low bits)
|
| 13 |
+
grad_norm_spread — gradient democracy monitor (orders-of-magnitude spread)
|
| 14 |
+
CVScreen — CV@1000-batch early band screen (<0.30 LOW / .35-.50 MID / >.80 HIGH)
|
| 15 |
+
|
| 16 |
+
Smoke on a torch-capable env: python geolip_vitals.py
|
| 17 |
+
"""
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
import math
|
| 20 |
+
import torch
|
| 21 |
+
|
| 22 |
+
BINDING = 0.29154 # radians; the binding/separation constant
|
| 23 |
+
CV_BAND = (0.13, 0.30) # CM CV band (discovery_catalog #4)
|
| 24 |
+
GATE_BAND = (0.012, 0.03) # live invariant candidate (acd_campaign)
|
| 25 |
+
KNUTH32 = 2654435761
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
# ----------------------------------------------------------------------------- drift
|
| 29 |
+
@torch.no_grad()
|
| 30 |
+
def anchor_drift(current: torch.Tensor, init: torch.Tensor, tol: float = 0.05) -> dict:
|
| 31 |
+
"""Geodesic drift (radians) of each row of `current` from its row in `init`,
|
| 32 |
+
both row-normalized. Returns mean/std/per-row drift and the fraction of rows
|
| 33 |
+
within +/-tol of BINDING (the GLFM '46%' readout)."""
|
| 34 |
+
a = torch.nn.functional.normalize(current.float(), dim=-1)
|
| 35 |
+
b = torch.nn.functional.normalize(init.float(), dim=-1)
|
| 36 |
+
cos = (a * b).sum(-1).clamp(-1.0, 1.0)
|
| 37 |
+
drift = torch.arccos(cos)
|
| 38 |
+
frac = ((drift - BINDING).abs() <= tol).float().mean()
|
| 39 |
+
return {"mean": drift.mean().item(), "std": drift.std().item(),
|
| 40 |
+
"per_row": drift, "binding_fraction": frac.item()}
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
# -------------------------------------------------------------------------------- cv
|
| 44 |
+
@torch.no_grad()
|
| 45 |
+
def pentachoron_cv(rows: torch.Tensor, n_samples: int = 200,
|
| 46 |
+
generator: torch.Generator | None = None) -> float:
|
| 47 |
+
"""CV (std/mean) of Cayley-Menger 4-simplex volumes over n_samples random
|
| 48 |
+
5-row subsets. CM comes from geovocab2 — never inlined (program rule);
|
| 49 |
+
fp32 det only. Rows are row-normalized before measurement."""
|
| 50 |
+
try:
|
| 51 |
+
from geovocab2.formula.cayley_menger import simplex_volume # noqa: F401
|
| 52 |
+
except Exception as e: # pragma: no cover
|
| 53 |
+
raise ImportError(
|
| 54 |
+
"pentachoron_cv requires geovocab2 (CM must be imported, never inlined "
|
| 55 |
+
"— program rule). Install/point PYTHONPATH at geovocab2; expected "
|
| 56 |
+
"geovocab2.formula.cayley_menger.simplex_volume(points fp32)."
|
| 57 |
+
) from e
|
| 58 |
+
x = torch.nn.functional.normalize(rows.float(), dim=-1)
|
| 59 |
+
n = x.shape[0]
|
| 60 |
+
if n < 5:
|
| 61 |
+
raise ValueError(f"pentachoron_cv needs >=5 rows, got {n}")
|
| 62 |
+
g = generator or torch.Generator(device="cpu").manual_seed(0)
|
| 63 |
+
vols = []
|
| 64 |
+
for _ in range(n_samples):
|
| 65 |
+
idx = torch.randperm(n, generator=g)[:5]
|
| 66 |
+
vols.append(simplex_volume(x[idx].to(torch.float32)))
|
| 67 |
+
v = torch.stack([torch.as_tensor(v, dtype=torch.float32) for v in vols])
|
| 68 |
+
return (v.std() / v.mean().clamp_min(1e-12)).item()
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
# ------------------------------------------------------------------------- aliveness
|
| 72 |
+
@torch.no_grad()
|
| 73 |
+
def axis_aliveness(oriented_weights: torch.Tensor, alive_thresh: float = 1e-3) -> dict:
|
| 74 |
+
"""`oriented_weights`: (..., 2K) nonnegative oriented-softmax address rows
|
| 75 |
+
(sum to 1 on the last dim). Returns axes-alive count, mean-usage perplexity
|
| 76 |
+
(hppl analogue; healthy hosted reference 125-126/128), and a collapse flag.
|
| 77 |
+
Reference behavior: near-uniform aliveness at div_weight=0 (discovery #22)."""
|
| 78 |
+
w = oriented_weights.reshape(-1, oriented_weights.shape[-1]).float()
|
| 79 |
+
usage = w.mean(0)
|
| 80 |
+
usage = usage / usage.sum().clamp_min(1e-12)
|
| 81 |
+
# an axis is alive if its mean usage exceeds alive_thresh x the uniform share
|
| 82 |
+
alive = int((usage > alive_thresh * (1.0 / usage.numel())).sum())
|
| 83 |
+
ent = -(usage.clamp_min(1e-12) * usage.clamp_min(1e-12).log()).sum()
|
| 84 |
+
ppl = float(ent.exp())
|
| 85 |
+
return {"axes_total": usage.numel(), "axes_alive": alive, "usage_ppl": ppl,
|
| 86 |
+
"collapsed": ppl < 0.05 * usage.numel()}
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
# ------------------------------------------------------------------------------ gates
|
| 90 |
+
@torch.no_grad()
|
| 91 |
+
def gate_stats(gates: torch.Tensor) -> dict:
|
| 92 |
+
"""Gate values (post-sigmoid/clamp). Reports mean and whether it sits in the
|
| 93 |
+
0.012-0.03 band (read-only — the band is a candidate invariant, never a target)."""
|
| 94 |
+
g = gates.float().flatten()
|
| 95 |
+
m = g.mean().item()
|
| 96 |
+
return {"mean": m, "std": g.std().item(),
|
| 97 |
+
"in_band": GATE_BAND[0] <= m <= GATE_BAND[1]}
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
# ------------------------------------------------------------------------------ paths
|
| 101 |
+
@torch.no_grad()
|
| 102 |
+
def path_diversity(ids: torch.Tensor) -> dict:
|
| 103 |
+
"""Unique-path counting with the FIXED multiplicative hash:
|
| 104 |
+
((ids * 2654435761) % 2^32) >> 16 — Knuth needs the HIGH bits; the low-16
|
| 105 |
+
variant produced a retracted ~1,500 path ceiling in a prior campaign.
|
| 106 |
+
`ids`: integer tensor, one composed path id per row (any shape)."""
|
| 107 |
+
x = ids.reshape(-1).to(torch.int64)
|
| 108 |
+
hashed = ((x * KNUTH32) % (1 << 32)) >> 16
|
| 109 |
+
return {"n": int(x.numel()),
|
| 110 |
+
"unique_raw": int(torch.unique(x).numel()),
|
| 111 |
+
"unique_hashed": int(torch.unique(hashed).numel())}
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
@torch.no_grad()
|
| 115 |
+
def compose_path_ids(stage_indices: list[torch.Tensor], radix: int) -> torch.Tensor:
|
| 116 |
+
"""Compose per-stage discrete indices (each (...,) int in [0, radix)) into a
|
| 117 |
+
single path id, positional base-`radix` — construction, not hashing."""
|
| 118 |
+
out = torch.zeros_like(stage_indices[0], dtype=torch.int64)
|
| 119 |
+
for s in stage_indices:
|
| 120 |
+
out = out * radix + s.to(torch.int64)
|
| 121 |
+
return out
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
# --------------------------------------------------------------------- grad democracy
|
| 125 |
+
@torch.no_grad()
|
| 126 |
+
def grad_norm_spread(groups: dict[str, list[torch.nn.Parameter]]) -> dict:
|
| 127 |
+
"""Gradient-democracy monitor. `groups`: name -> params of one parallel member
|
| 128 |
+
(tower/expert). Reports per-group grad norms and the orders-of-magnitude spread.
|
| 129 |
+
Reference: unequalized heterogeneous towers spread ~20 orders (fibonacci dead at
|
| 130 |
+
2.25e-21 under helix); equalized ~0.0 (geofractal gradient-democracy result)."""
|
| 131 |
+
norms = {}
|
| 132 |
+
for name, params in groups.items():
|
| 133 |
+
gs = [p.grad for p in params if p.grad is not None]
|
| 134 |
+
norms[name] = float(torch.sqrt(sum((g.float() ** 2).sum() for g in gs)).item()) \
|
| 135 |
+
if gs else 0.0
|
| 136 |
+
vals = [v for v in norms.values() if v > 0]
|
| 137 |
+
spread = (math.log10(max(vals)) - math.log10(min(vals))) if len(vals) >= 2 else 0.0
|
| 138 |
+
return {"norms": norms, "spread_orders": spread, "dead": [k for k, v in norms.items() if v == 0.0]}
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
# ----------------------------------------------------------------------------- screen
|
| 142 |
+
class CVScreen:
|
| 143 |
+
"""CV@N early band screen (tri-band ft1): record pentachoron CV at `step_mark`
|
| 144 |
+
batches; classify <0.30 LOW / 0.35-0.50 MID / >0.80 HIGH. Turns ~2h/config
|
| 145 |
+
into ~7min. Readout only."""
|
| 146 |
+
def __init__(self, step_mark: int = 1000):
|
| 147 |
+
self.step_mark = step_mark
|
| 148 |
+
self.recorded: float | None = None
|
| 149 |
+
|
| 150 |
+
def maybe_record(self, step: int, rows: torch.Tensor) -> float | None:
|
| 151 |
+
if self.recorded is None and step >= self.step_mark:
|
| 152 |
+
self.recorded = pentachoron_cv(rows)
|
| 153 |
+
return self.recorded
|
| 154 |
+
|
| 155 |
+
@property
|
| 156 |
+
def band(self) -> str | None:
|
| 157 |
+
c = self.recorded
|
| 158 |
+
if c is None:
|
| 159 |
+
return None
|
| 160 |
+
if c < 0.30:
|
| 161 |
+
return "LOW"
|
| 162 |
+
if 0.35 <= c <= 0.50:
|
| 163 |
+
return "MID"
|
| 164 |
+
if c > 0.80:
|
| 165 |
+
return "HIGH"
|
| 166 |
+
return "BETWEEN"
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
# ------------------------------------------------------------------------------ smoke
|
| 170 |
+
if __name__ == "__main__": # shapes/parse smoke ONLY — no training, ever.
|
| 171 |
+
g = torch.Generator().manual_seed(0)
|
| 172 |
+
K, D = 64, 4
|
| 173 |
+
init = torch.nn.functional.normalize(torch.randn(K, D, generator=g), dim=-1)
|
| 174 |
+
cur = torch.nn.functional.normalize(init + 0.29 * torch.randn(K, D, generator=g), dim=-1)
|
| 175 |
+
print("drift:", {k: v for k, v in anchor_drift(cur, init).items() if k != "per_row"})
|
| 176 |
+
w = torch.softmax(torch.randn(32, 2 * K, generator=g), dim=-1)
|
| 177 |
+
print("aliveness:", axis_aliveness(w))
|
| 178 |
+
print("gates:", gate_stats(torch.full((8,), 0.024)))
|
| 179 |
+
ids = compose_path_ids([torch.randint(0, 16, (4096,), generator=g) for _ in range(4)], 16)
|
| 180 |
+
print("paths:", path_diversity(ids))
|
| 181 |
+
lin = torch.nn.Linear(8, 8)
|
| 182 |
+
lin(torch.randn(4, 8)).sum().backward()
|
| 183 |
+
print("democracy:", grad_norm_spread({"a": list(lin.parameters())}))
|
| 184 |
+
print("OK — vitals smoke passed (pentachoron_cv needs geovocab2; run on GPU env)")
|
exp015_ch/read_codebook.py
ADDED
|
@@ -0,0 +1,159 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""read_codebook.py — projective reading of cultivated aleph codebooks.
|
| 2 |
+
Antipodal-collapse extraction on trained codebooks + projective statistics on
|
| 3 |
+
RP^(D-1) — applied to the exp012 AR-bed specimens.
|
| 4 |
+
|
| 5 |
+
Recipe per the Polygonal Omega article (geometric-tri-band-ft2): collapse = (row_i - row_j)/2 normalized for each MUTUAL-STRONGEST
|
| 6 |
+
pair with cos < -0.9 — "a deterministic tensor operation," not clustering.
|
| 7 |
+
Projective metric ALWAYS arccos|<a,b>| (metric-alignment rule, reading-voids-ft1).
|
| 8 |
+
D=4 scope is the validated regime (D=5 walked back; axis count grows with D).
|
| 9 |
+
|
| 10 |
+
Readouts per specimen:
|
| 11 |
+
pairs / n_axes / unpaired — antipodal structure
|
| 12 |
+
proj_angle mean vs uniform baseline, deviation — near-uniform RP^(D-1)?
|
| 13 |
+
drift from home + binding fraction @0.29154 — cultivation record
|
| 14 |
+
erank of the axis set — spectral occupancy
|
| 15 |
+
verdict: PROJECTIVE-CLEAN (|dev|<0.05, util>0.95, secondary pairs<=3) /
|
| 16 |
+
-MOSTLY / STRUCTURED / DEGENERATE (per Polygonal Omega thresholds)
|
| 17 |
+
|
| 18 |
+
Usage (terminal): python read_codebook.py <ckpt_or_dir> [more paths...]
|
| 19 |
+
Colab: paste geolip_vitals.py cell first (optional), then this file, then
|
| 20 |
+
read_all(r"/content/data/ar_ckpts").
|
| 21 |
+
"""
|
| 22 |
+
from __future__ import annotations
|
| 23 |
+
import math
|
| 24 |
+
import sys
|
| 25 |
+
import torch
|
| 26 |
+
import torch.nn.functional as F
|
| 27 |
+
|
| 28 |
+
BINDING = 0.29154
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
@torch.no_grad()
|
| 32 |
+
def antipodal_collapse(codebook: torch.Tensor, thresh: float = -0.9) -> dict:
|
| 33 |
+
"""Mutual-strongest antipodal pairing + collapse to axes on RP^(D-1)."""
|
| 34 |
+
A = F.normalize(codebook.float(), dim=-1)
|
| 35 |
+
K = A.shape[0]
|
| 36 |
+
cos = A @ A.T
|
| 37 |
+
cos.fill_diagonal_(2.0) # exclude self from minima
|
| 38 |
+
nearest_neg = cos.argmin(dim=-1) # most-antipodal partner
|
| 39 |
+
pairs = []
|
| 40 |
+
used = set()
|
| 41 |
+
for i in range(K):
|
| 42 |
+
j = int(nearest_neg[i])
|
| 43 |
+
if i < j and int(nearest_neg[j]) == i and cos[i, j] < thresh:
|
| 44 |
+
pairs.append((i, j))
|
| 45 |
+
used.update((i, j))
|
| 46 |
+
axes = [F.normalize((A[i] - A[j]) / 2.0, dim=-1) for i, j in pairs]
|
| 47 |
+
axes += [A[i] for i in range(K) if i not in used] # unpaired rows as axes
|
| 48 |
+
axes = torch.stack(axes) if axes else A[:0]
|
| 49 |
+
# sign-canon onto RP: first nonzero coordinate positive
|
| 50 |
+
for r in range(axes.shape[0]):
|
| 51 |
+
nz = torch.nonzero(axes[r].abs() > 1e-8)
|
| 52 |
+
if nz.numel() and axes[r, nz[0, 0]] < 0:
|
| 53 |
+
axes[r] = -axes[r]
|
| 54 |
+
return {"pairs": len(pairs), "n_axes": axes.shape[0],
|
| 55 |
+
"unpaired": K - 2 * len(pairs), "axes": axes}
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
@torch.no_grad()
|
| 59 |
+
def projective_stats(axes: torch.Tensor, n_baseline: int = 20000,
|
| 60 |
+
seed: int = 0) -> dict:
|
| 61 |
+
"""Mean projective angle arccos|<a,b>| vs a uniform-RP baseline at same (n, D)."""
|
| 62 |
+
n, D = axes.shape
|
| 63 |
+
if n < 2:
|
| 64 |
+
return {"proj_angle_mean": None, "uniform_baseline": None,
|
| 65 |
+
"deviation": None, "erank": None}
|
| 66 |
+
def mean_angle(rows):
|
| 67 |
+
c = (rows @ rows.T).abs().clamp(max=1.0)
|
| 68 |
+
iu = torch.triu_indices(rows.shape[0], rows.shape[0], offset=1)
|
| 69 |
+
return torch.arccos(c[iu[0], iu[1]]).mean().item()
|
| 70 |
+
obs = mean_angle(axes)
|
| 71 |
+
g = torch.Generator().manual_seed(seed)
|
| 72 |
+
base_angles = []
|
| 73 |
+
m = max(2, n)
|
| 74 |
+
for _ in range(max(1, n_baseline // max(1, m * (m - 1) // 2))):
|
| 75 |
+
r = F.normalize(torch.randn(m, D, generator=g), dim=-1)
|
| 76 |
+
base_angles.append(mean_angle(r))
|
| 77 |
+
base = sum(base_angles) / len(base_angles)
|
| 78 |
+
s = torch.linalg.svdvals(axes)
|
| 79 |
+
p = (s / s.sum().clamp_min(1e-12))
|
| 80 |
+
erank = float(torch.exp(-(p.clamp_min(1e-12) * p.clamp_min(1e-12).log()).sum()))
|
| 81 |
+
return {"proj_angle_mean": round(obs, 4), "uniform_baseline": round(base, 4),
|
| 82 |
+
"deviation": round(obs - base, 4), "erank": round(erank, 3)}
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
@torch.no_grad()
|
| 86 |
+
def read_specimen(path: str) -> dict:
|
| 87 |
+
ck = torch.load(path, map_location="cpu", weights_only=True)
|
| 88 |
+
out = {"file": path.split("\\")[-1].split("/")[-1],
|
| 89 |
+
"arm": ck.get("arm"), "seed": ck.get("seed"),
|
| 90 |
+
"steps": ck.get("steps"), "val_bpb": round(ck.get("val_bpb", -1), 4)}
|
| 91 |
+
if "state_dict" in ck: # full specimen checkpoint
|
| 92 |
+
sd = ck["state_dict"]
|
| 93 |
+
books = {k[:-len(".codebook")]: sd[k] for k in sd
|
| 94 |
+
if k.endswith("addr.codebook") or k.endswith("head_addr.codebook")}
|
| 95 |
+
homes = {k[:-len(".home")]: sd[k] for k in sd if k.endswith(".home")}
|
| 96 |
+
else: # bare genome dict (exp014+ champion files):
|
| 97 |
+
# books under flat/root/branch* keys; *_proj entries are projections
|
| 98 |
+
books = {k: v for k, v in ck.items()
|
| 99 |
+
if torch.is_tensor(v) and v.ndim == 2
|
| 100 |
+
and (k in ("flat", "root") or k.startswith("branch"))}
|
| 101 |
+
homes = {}
|
| 102 |
+
reads = {}
|
| 103 |
+
for name, cb in books.items():
|
| 104 |
+
col = antipodal_collapse(cb)
|
| 105 |
+
stats = projective_stats(col["axes"])
|
| 106 |
+
home = homes.get(name)
|
| 107 |
+
drift = None
|
| 108 |
+
binding = None
|
| 109 |
+
if home is not None and home.shape == cb.shape:
|
| 110 |
+
a = F.normalize(cb.float(), dim=-1)
|
| 111 |
+
b = F.normalize(home.float(), dim=-1)
|
| 112 |
+
dr = torch.arccos((a * b).sum(-1).clamp(-1, 1))
|
| 113 |
+
drift = round(dr.mean().item(), 4)
|
| 114 |
+
binding = round(((dr - BINDING).abs() <= 0.05).float().mean().item(), 4)
|
| 115 |
+
util = col["n_axes"] / cb.shape[0]
|
| 116 |
+
dev = stats["deviation"]
|
| 117 |
+
if dev is not None and abs(dev) < 0.05 and util > 0.95 and col["pairs"] <= 3:
|
| 118 |
+
verdict = "PROJECTIVE-CLEAN"
|
| 119 |
+
elif dev is not None and abs(dev) < 0.05:
|
| 120 |
+
verdict = "PROJECTIVE-MOSTLY"
|
| 121 |
+
elif dev is not None and dev > 0.05:
|
| 122 |
+
verdict = "STRUCTURED(repulsive)"
|
| 123 |
+
else:
|
| 124 |
+
verdict = "DEGENERATE/CLUMPED" if dev is not None else "TOO-FEW-AXES"
|
| 125 |
+
reads[name] = {
|
| 126 |
+
"pairs": col["pairs"], "n_axes": col["n_axes"], **stats,
|
| 127 |
+
"drift": drift, "binding_frac": binding, "verdict": verdict}
|
| 128 |
+
out["codebooks"] = reads
|
| 129 |
+
return out
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def read_all(root: str) -> list:
|
| 133 |
+
import glob, os
|
| 134 |
+
results = []
|
| 135 |
+
for p in sorted(glob.glob(os.path.join(root, "*.pt"))):
|
| 136 |
+
r = read_specimen(p)
|
| 137 |
+
print(r, flush=True)
|
| 138 |
+
results.append(r)
|
| 139 |
+
return results
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def _in_notebook() -> bool:
|
| 143 |
+
try:
|
| 144 |
+
get_ipython() # type: ignore[name-defined] # noqa: F821
|
| 145 |
+
return True
|
| 146 |
+
except NameError:
|
| 147 |
+
return False
|
| 148 |
+
|
| 149 |
+
|
| 150 |
+
if __name__ == "__main__":
|
| 151 |
+
if _in_notebook():
|
| 152 |
+
print("Notebook mode: call read_all(r'<data_root>/ar_ckpts') in the next cell.")
|
| 153 |
+
else:
|
| 154 |
+
args = [a for a in sys.argv[1:] if not a.startswith("-")]
|
| 155 |
+
if not args:
|
| 156 |
+
print("usage: python read_codebook.py <ckpt_or_dir> [...]")
|
| 157 |
+
for a in args:
|
| 158 |
+
import os
|
| 159 |
+
read_all(a) if os.path.isdir(a) else print(read_specimen(a))
|
exp015_ch/repro.py
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""repro.py — standalone loader/runner for exp015_ch. Every code dependency
|
| 2 |
+
lives in THIS folder (geolip_vitals.py, ar_differentiation_bed.py,
|
| 3 |
+
exp014_genetic_distillation.py, exp015_content_heredity.py, read_codebook.py)
|
| 4 |
+
— no cross-package paths.
|
| 5 |
+
|
| 6 |
+
python repro.py # CPU smoke: parse + shapes + inheritance
|
| 7 |
+
python repro.py --lineage sign_full # one full bracket (GPU, ~20 min)
|
| 8 |
+
lineages: sign_weights | sign_full | tree_weights | tree_full | sign_none
|
| 9 |
+
|
| 10 |
+
Data lands in ./data (override with the GEOLIP_DATA env var); the bracket
|
| 11 |
+
ledger + champion genomes land in ./data/exp015. Reading the shipped genomes:
|
| 12 |
+
python read_codebook.py genomes/
|
| 13 |
+
"""
|
| 14 |
+
import os
|
| 15 |
+
import sys
|
| 16 |
+
|
| 17 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 18 |
+
sys.path.insert(0, HERE)
|
| 19 |
+
|
| 20 |
+
if __name__ == "__main__":
|
| 21 |
+
import geolip_vitals # noqa: F401 (paste order)
|
| 22 |
+
import ar_differentiation_bed as bed
|
| 23 |
+
import exp014_genetic_distillation as gd # noqa: F401
|
| 24 |
+
import exp015_content_heredity as ch
|
| 25 |
+
args = sys.argv[1:]
|
| 26 |
+
if not args:
|
| 27 |
+
ch.smoke()
|
| 28 |
+
print("repro smoke passed — run with --lineage <name> for a bracket")
|
| 29 |
+
else:
|
| 30 |
+
import argparse
|
| 31 |
+
ap = argparse.ArgumentParser()
|
| 32 |
+
ap.add_argument("--lineage", required=True,
|
| 33 |
+
choices=["sign_weights", "sign_full", "tree_weights",
|
| 34 |
+
"tree_full", "sign_none"])
|
| 35 |
+
ap.add_argument("--seed", type=int, default=0)
|
| 36 |
+
ap.add_argument("--steps", type=int, default=2000)
|
| 37 |
+
ap.add_argument("--gens", type=int, default=4)
|
| 38 |
+
a, _ = ap.parse_known_args()
|
| 39 |
+
ch.run_tournament_15(a.lineage, gens=a.gens, steps=a.steps, seed=a.seed)
|
exp015_ch/results/ledger.jsonl
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 0, "member": 0, "fresh": false, "bpb": 2.4542, "content": {"book_fid": 0.9423, "drift": 0.1757}}
|
| 2 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 0, "member": 1, "fresh": false, "bpb": 2.4859, "content": {"book_fid": 0.9398, "drift": 0.1723}}
|
| 3 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 0, "member": 2, "fresh": false, "bpb": 2.4949, "content": {"book_fid": 0.9362, "drift": 0.185}}
|
| 4 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 0, "member": 3, "fresh": false, "bpb": 2.4667, "content": {"book_fid": 0.9482, "drift": 0.1538}}
|
| 5 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 1, "member": 0, "fresh": false, "bpb": 2.4149, "content": {"book_fid": 0.954, "drift": 0.196}}
|
| 6 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 1, "member": 1, "fresh": false, "bpb": 2.4248, "content": {"book_fid": 0.9531, "drift": 0.1959}}
|
| 7 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 1, "member": 2, "fresh": false, "bpb": 2.4286, "content": {"book_fid": 0.9528, "drift": 0.1968}}
|
| 8 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 1, "member": 3, "fresh": true, "bpb": 2.4833, "content": {"book_fid": 0.9102, "drift": 0.2017}}
|
| 9 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 2, "member": 0, "fresh": false, "bpb": 2.411, "content": {"book_fid": 0.9542, "drift": 0.1997}}
|
| 10 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 2, "member": 1, "fresh": false, "bpb": 2.4306, "content": {"book_fid": 0.9542, "drift": 0.1993}}
|
| 11 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 2, "member": 2, "fresh": false, "bpb": 2.4266, "content": {"book_fid": 0.9545, "drift": 0.1997}}
|
| 12 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 2, "member": 3, "fresh": true, "bpb": 2.4805, "content": {"book_fid": 0.9155, "drift": 0.2079}}
|
| 13 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 3, "member": 0, "fresh": false, "bpb": 2.4098, "content": {"book_fid": 0.9537, "drift": 0.2028}}
|
| 14 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 3, "member": 1, "fresh": false, "bpb": 2.423, "content": {"book_fid": 0.954, "drift": 0.203}}
|
| 15 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 3, "member": 2, "fresh": false, "bpb": 2.4285, "content": {"book_fid": 0.9539, "drift": 0.2028}}
|
| 16 |
+
{"exp": "15", "lineage": "sign_weights", "tseed": 0, "gen": 3, "member": 3, "fresh": true, "bpb": 2.5037, "content": {"book_fid": 0.9343, "drift": 0.1828}}
|
| 17 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 0, "member": 0, "fresh": false, "bpb": 2.4518, "content": {"book_fid": 0.9421, "drift": 0.1749}}
|
| 18 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 0, "member": 1, "fresh": false, "bpb": 2.4885, "content": {"book_fid": 0.9405, "drift": 0.172}}
|
| 19 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 0, "member": 2, "fresh": false, "bpb": 2.4917, "content": {"book_fid": 0.9363, "drift": 0.1871}}
|
| 20 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 0, "member": 3, "fresh": false, "bpb": 2.4692, "content": {"book_fid": 0.9485, "drift": 0.1531}}
|
| 21 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 1, "member": 0, "fresh": false, "bpb": 2.412, "content": {"book_fid": 0.9554, "drift": 0.1116}}
|
| 22 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 1, "member": 1, "fresh": false, "bpb": 2.4231, "content": {"book_fid": 0.9555, "drift": 0.112}}
|
| 23 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 1, "member": 2, "fresh": false, "bpb": 2.428, "content": {"book_fid": 0.9558, "drift": 0.1121}}
|
| 24 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 1, "member": 3, "fresh": true, "bpb": 2.4718, "content": {"book_fid": 0.9116, "drift": 0.1985}}
|
| 25 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 2, "member": 0, "fresh": false, "bpb": 2.4083, "content": {"book_fid": 0.9557, "drift": 0.0113}}
|
| 26 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 2, "member": 1, "fresh": false, "bpb": 2.4274, "content": {"book_fid": 0.9556, "drift": 0.0116}}
|
| 27 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 2, "member": 2, "fresh": false, "bpb": 2.4245, "content": {"book_fid": 0.9555, "drift": 0.0117}}
|
| 28 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 2, "member": 3, "fresh": true, "bpb": 2.4783, "content": {"book_fid": 0.9159, "drift": 0.2064}}
|
| 29 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 3, "member": 0, "fresh": false, "bpb": 2.4081, "content": {"book_fid": 0.9557, "drift": 0.0088}}
|
| 30 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 3, "member": 1, "fresh": false, "bpb": 2.4215, "content": {"book_fid": 0.9558, "drift": 0.0087}}
|
| 31 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 3, "member": 2, "fresh": false, "bpb": 2.4262, "content": {"book_fid": 0.9557, "drift": 0.0088}}
|
| 32 |
+
{"exp": "15", "lineage": "sign_full", "tseed": 0, "gen": 3, "member": 3, "fresh": true, "bpb": 2.4945, "content": {"book_fid": 0.9351, "drift": 0.185}}
|
| 33 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 0, "member": 0, "fresh": false, "bpb": 2.6228, "content": {"root_fid": 0.477, "branch_fid": [0.9181, 0.9273, 0.9289, 0.9397], "root_drift": 0.1393, "root_usage": [0.389, 0.413, 0.11, 0.088], "root_ppl4": 3.284, "branch_drift": [0.418, 0.318, 0.294, 0.29]}}
|
| 34 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 0, "member": 1, "fresh": false, "bpb": 2.6412, "content": {"root_fid": 0.4639, "branch_fid": [0.9343, 0.9406, 0.9032, 0.9416], "root_drift": 0.1407, "root_usage": [0.373, 0.253, 0.32, 0.054], "root_ppl4": 3.45, "branch_drift": [0.341, 0.281, 0.414, 0.309]}}
|
| 35 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 0, "member": 2, "fresh": false, "bpb": 2.6378, "content": {"root_fid": 0.4849, "branch_fid": [0.9569, 0.9281, 0.906, 0.9432], "root_drift": 0.0785, "root_usage": [0.004, 0.122, 0.832, 0.042], "root_ppl4": 1.76, "branch_drift": [0.202, 0.376, 0.324, 0.273]}}
|
| 36 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 0, "member": 3, "fresh": false, "bpb": 2.6481, "content": {"root_fid": 0.4838, "branch_fid": [0.9411, 0.9522, 0.9024, 0.9202], "root_drift": 0.1247, "root_usage": [0.295, 0.024, 0.19, 0.492], "root_ppl4": 3.042, "branch_drift": [0.355, 0.236, 0.328, 0.337]}}
|
| 37 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 1, "member": 0, "fresh": false, "bpb": 2.5889, "content": {"root_fid": 0.4691, "branch_fid": [0.9331, 0.9456, 0.9397, 0.9505], "root_drift": 0.162, "root_usage": [0.293, 0.358, 0.221, 0.128], "root_ppl4": 3.758, "branch_drift": [0.395, 0.283, 0.284, 0.267]}}
|
| 38 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 1, "member": 1, "fresh": false, "bpb": 2.5988, "content": {"root_fid": 0.4745, "branch_fid": [0.9338, 0.9463, 0.939, 0.9494], "root_drift": 0.1611, "root_usage": [0.298, 0.364, 0.219, 0.119], "root_ppl4": 3.722, "branch_drift": [0.394, 0.281, 0.284, 0.268]}}
|
| 39 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 1, "member": 2, "fresh": false, "bpb": 2.6034, "content": {"root_fid": 0.472, "branch_fid": [0.9335, 0.9455, 0.9395, 0.9498], "root_drift": 0.17, "root_usage": [0.287, 0.372, 0.22, 0.121], "root_ppl4": 3.724, "branch_drift": [0.395, 0.283, 0.285, 0.267]}}
|
| 40 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 1, "member": 3, "fresh": true, "bpb": 2.6895, "content": {"root_fid": 0.4627, "branch_fid": [0.9124, 0.957, 0.943, 0.9067], "root_drift": 0.2238, "root_usage": [0.196, 0.008, 0.032, 0.764], "root_ppl4": 1.961, "branch_drift": [0.423, 0.247, 0.247, 0.333]}}
|
| 41 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 2, "member": 0, "fresh": false, "bpb": 2.579, "content": {"root_fid": 0.4731, "branch_fid": [0.9339, 0.9465, 0.9395, 0.9504], "root_drift": 0.1626, "root_usage": [0.294, 0.358, 0.221, 0.128], "root_ppl4": 3.759, "branch_drift": [0.394, 0.282, 0.285, 0.267]}}
|
| 42 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 2, "member": 1, "fresh": false, "bpb": 2.5976, "content": {"root_fid": 0.4714, "branch_fid": [0.9336, 0.9464, 0.9397, 0.9505], "root_drift": 0.1631, "root_usage": [0.29, 0.364, 0.223, 0.124], "root_ppl4": 3.742, "branch_drift": [0.394, 0.282, 0.285, 0.267]}}
|
| 43 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 2, "member": 2, "fresh": false, "bpb": 2.5942, "content": {"root_fid": 0.4725, "branch_fid": [0.9337, 0.9468, 0.9396, 0.9507], "root_drift": 0.1629, "root_usage": [0.296, 0.359, 0.218, 0.127], "root_ppl4": 3.751, "branch_drift": [0.394, 0.282, 0.284, 0.267]}}
|
| 44 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 2, "member": 3, "fresh": true, "bpb": 2.6505, "content": {"root_fid": 0.4673, "branch_fid": [0.9367, 0.9109, 0.934, 0.9522], "root_drift": 0.302, "root_usage": [0.188, 0.675, 0.115, 0.021], "root_ppl4": 2.484, "branch_drift": [0.294, 0.451, 0.31, 0.195]}}
|
| 45 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 3, "member": 0, "fresh": false, "bpb": 2.5721, "content": {"root_fid": 0.4719, "branch_fid": [0.9341, 0.9467, 0.9401, 0.9504], "root_drift": 0.1634, "root_usage": [0.301, 0.352, 0.214, 0.133], "root_ppl4": 3.77, "branch_drift": [0.394, 0.281, 0.285, 0.267]}}
|
| 46 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 3, "member": 1, "fresh": false, "bpb": 2.5821, "content": {"root_fid": 0.4717, "branch_fid": [0.9339, 0.9469, 0.9401, 0.9503], "root_drift": 0.1632, "root_usage": [0.295, 0.363, 0.216, 0.126], "root_ppl4": 3.743, "branch_drift": [0.394, 0.281, 0.285, 0.267]}}
|
| 47 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 3, "member": 2, "fresh": false, "bpb": 2.5872, "content": {"root_fid": 0.47, "branch_fid": [0.934, 0.9464, 0.9397, 0.9504], "root_drift": 0.1634, "root_usage": [0.293, 0.356, 0.218, 0.133], "root_ppl4": 3.773, "branch_drift": [0.394, 0.282, 0.285, 0.267]}}
|
| 48 |
+
{"exp": "15", "lineage": "tree_weights", "tseed": 0, "gen": 3, "member": 3, "fresh": true, "bpb": 2.6506, "content": {"root_fid": 0.4679, "branch_fid": [0.9027, 0.9044, 0.9224, 0.9192], "root_drift": 0.0969, "root_usage": [0.184, 0.503, 0.268, 0.045], "root_ppl4": 3.158, "branch_drift": [0.26, 0.331, 0.313, 0.345]}}
|
| 49 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 0, "member": 0, "fresh": false, "bpb": 2.6307, "content": {"root_fid": 0.4804, "branch_fid": [0.9218, 0.9276, 0.9287, 0.9395], "root_drift": 0.1437, "root_usage": [0.398, 0.406, 0.104, 0.092], "root_ppl4": 3.278, "branch_drift": [0.416, 0.318, 0.293, 0.292]}}
|
| 50 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 0, "member": 1, "fresh": false, "bpb": 2.6418, "content": {"root_fid": 0.4641, "branch_fid": [0.9338, 0.941, 0.9024, 0.9413], "root_drift": 0.1385, "root_usage": [0.374, 0.253, 0.312, 0.061], "root_ppl4": 3.49, "branch_drift": [0.341, 0.28, 0.416, 0.31]}}
|
| 51 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 0, "member": 2, "fresh": false, "bpb": 2.6517, "content": {"root_fid": 0.4861, "branch_fid": [0.9568, 0.9284, 0.9058, 0.9425], "root_drift": 0.0766, "root_usage": [0.005, 0.126, 0.834, 0.035], "root_ppl4": 1.742, "branch_drift": [0.204, 0.375, 0.323, 0.274]}}
|
| 52 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 0, "member": 3, "fresh": false, "bpb": 2.6486, "content": {"root_fid": 0.4824, "branch_fid": [0.9413, 0.9517, 0.9017, 0.9207], "root_drift": 0.1234, "root_usage": [0.301, 0.025, 0.19, 0.485], "root_ppl4": 3.061, "branch_drift": [0.354, 0.236, 0.329, 0.336]}}
|
| 53 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 1, "member": 0, "fresh": false, "bpb": 2.5901, "content": {"root_fid": 0.4789, "branch_fid": [0.9541, 0.9546, 0.9451, 0.9631], "root_drift": 0.0607, "root_usage": [0.414, 0.336, 0.084, 0.166], "root_ppl4": 3.449, "branch_drift": [0.13, 0.111, 0.106, 0.124]}}
|
| 54 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 1, "member": 1, "fresh": false, "bpb": 2.6014, "content": {"root_fid": 0.4799, "branch_fid": [0.9535, 0.9547, 0.9444, 0.9624], "root_drift": 0.0539, "root_usage": [0.411, 0.347, 0.09, 0.153], "root_ppl4": 3.443, "branch_drift": [0.131, 0.111, 0.103, 0.12]}}
|
| 55 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 1, "member": 2, "fresh": false, "bpb": 2.606, "content": {"root_fid": 0.4783, "branch_fid": [0.9542, 0.9541, 0.9444, 0.9629], "root_drift": 0.0556, "root_usage": [0.423, 0.332, 0.084, 0.161], "root_ppl4": 3.428, "branch_drift": [0.13, 0.111, 0.102, 0.12]}}
|
| 56 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 1, "member": 3, "fresh": true, "bpb": 2.6897, "content": {"root_fid": 0.4652, "branch_fid": [0.9132, 0.9563, 0.9428, 0.9073], "root_drift": 0.2214, "root_usage": [0.217, 0.01, 0.037, 0.736], "root_ppl4": 2.064, "branch_drift": [0.423, 0.252, 0.248, 0.332]}}
|
| 57 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 2, "member": 0, "fresh": false, "bpb": 2.5815, "content": {"root_fid": 0.4777, "branch_fid": [0.9548, 0.9546, 0.9451, 0.9627], "root_drift": 0.0042, "root_usage": [0.411, 0.338, 0.086, 0.165], "root_ppl4": 3.459, "branch_drift": [0.005, 0.005, 0.006, 0.005]}}
|
| 58 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 2, "member": 1, "fresh": false, "bpb": 2.6007, "content": {"root_fid": 0.4772, "branch_fid": [0.9545, 0.9545, 0.9446, 0.9624], "root_drift": 0.0041, "root_usage": [0.411, 0.34, 0.084, 0.165], "root_ppl4": 3.448, "branch_drift": [0.005, 0.005, 0.006, 0.005]}}
|
| 59 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 2, "member": 2, "fresh": false, "bpb": 2.5979, "content": {"root_fid": 0.4776, "branch_fid": [0.9548, 0.9544, 0.9447, 0.9627], "root_drift": 0.004, "root_usage": [0.408, 0.341, 0.087, 0.164], "root_ppl4": 3.46, "branch_drift": [0.005, 0.004, 0.006, 0.005]}}
|
| 60 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 2, "member": 3, "fresh": true, "bpb": 2.6507, "content": {"root_fid": 0.4669, "branch_fid": [0.9357, 0.9107, 0.9348, 0.9522], "root_drift": 0.3004, "root_usage": [0.184, 0.673, 0.119, 0.025], "root_ppl4": 2.516, "branch_drift": [0.293, 0.452, 0.309, 0.195]}}
|
| 61 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 3, "member": 0, "fresh": false, "bpb": 2.575, "content": {"root_fid": 0.4767, "branch_fid": [0.9549, 0.9541, 0.9448, 0.9623], "root_drift": 0.0029, "root_usage": [0.411, 0.335, 0.087, 0.167], "root_ppl4": 3.465, "branch_drift": [0.003, 0.003, 0.005, 0.004]}}
|
| 62 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 3, "member": 1, "fresh": false, "bpb": 2.5855, "content": {"root_fid": 0.4781, "branch_fid": [0.955, 0.9541, 0.9447, 0.9623], "root_drift": 0.0035, "root_usage": [0.414, 0.338, 0.083, 0.165], "root_ppl4": 3.439, "branch_drift": [0.003, 0.003, 0.005, 0.004]}}
|
| 63 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 3, "member": 2, "fresh": false, "bpb": 2.5915, "content": {"root_fid": 0.4765, "branch_fid": [0.9547, 0.9543, 0.9445, 0.9627], "root_drift": 0.0024, "root_usage": [0.405, 0.335, 0.089, 0.172], "root_ppl4": 3.49, "branch_drift": [0.003, 0.003, 0.005, 0.004]}}
|
| 64 |
+
{"exp": "15", "lineage": "tree_full", "tseed": 0, "gen": 3, "member": 3, "fresh": true, "bpb": 2.6512, "content": {"root_fid": 0.4685, "branch_fid": [0.9029, 0.9062, 0.9225, 0.9219], "root_drift": 0.095, "root_usage": [0.186, 0.496, 0.277, 0.041], "root_ppl4": 3.147, "branch_drift": [0.26, 0.33, 0.312, 0.346]}}
|
| 65 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 0, "member": 0, "fresh": false, "bpb": 2.5073, "content": {"book_fid": 0.9214, "drift": 0.18}}
|
| 66 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 0, "member": 1, "fresh": false, "bpb": 2.4472, "content": {"book_fid": 0.9244, "drift": 0.183}}
|
| 67 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 0, "member": 2, "fresh": false, "bpb": 2.473, "content": {"book_fid": 0.9296, "drift": 0.1964}}
|
| 68 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 0, "member": 3, "fresh": false, "bpb": 2.5014, "content": {"book_fid": 0.9392, "drift": 0.1777}}
|
| 69 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 1, "member": 0, "fresh": false, "bpb": 2.4823, "content": {"book_fid": 0.9347, "drift": 0.2046}}
|
| 70 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 1, "member": 1, "fresh": false, "bpb": 2.4448, "content": {"book_fid": 0.927, "drift": 0.1658}}
|
| 71 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 1, "member": 2, "fresh": false, "bpb": 2.5182, "content": {"book_fid": 0.9273, "drift": 0.1822}}
|
| 72 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 1, "member": 3, "fresh": true, "bpb": 2.474, "content": {"book_fid": 0.9102, "drift": 0.1995}}
|
| 73 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 2, "member": 0, "fresh": false, "bpb": 2.4365, "content": {"book_fid": 0.9279, "drift": 0.1765}}
|
| 74 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 2, "member": 1, "fresh": false, "bpb": 2.5354, "content": {"book_fid": 0.9299, "drift": 0.1962}}
|
| 75 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 2, "member": 2, "fresh": false, "bpb": 2.5406, "content": {"book_fid": 0.9189, "drift": 0.1976}}
|
| 76 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 2, "member": 3, "fresh": true, "bpb": 2.4844, "content": {"book_fid": 0.916, "drift": 0.2046}}
|
| 77 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 3, "member": 0, "fresh": false, "bpb": 2.4152, "content": {"book_fid": 0.9314, "drift": 0.1654}}
|
| 78 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 3, "member": 1, "fresh": false, "bpb": 2.4916, "content": {"book_fid": 0.9355, "drift": 0.1693}}
|
| 79 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 3, "member": 2, "fresh": false, "bpb": 2.5254, "content": {"book_fid": 0.9176, "drift": 0.1907}}
|
| 80 |
+
{"exp": "15", "lineage": "sign_none", "tseed": 0, "gen": 3, "member": 3, "fresh": true, "bpb": 2.4981, "content": {"book_fid": 0.9351, "drift": 0.1865}}
|
exp015_ch/results/results.json
ADDED
|
@@ -0,0 +1,107 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"best_of_gen": {
|
| 3 |
+
"sign_weights": [
|
| 4 |
+
2.4542,
|
| 5 |
+
2.4149,
|
| 6 |
+
2.411,
|
| 7 |
+
2.4098
|
| 8 |
+
],
|
| 9 |
+
"sign_full": [
|
| 10 |
+
2.4518,
|
| 11 |
+
2.412,
|
| 12 |
+
2.4083,
|
| 13 |
+
2.4081
|
| 14 |
+
],
|
| 15 |
+
"tree_weights": [
|
| 16 |
+
2.6228,
|
| 17 |
+
2.5889,
|
| 18 |
+
2.579,
|
| 19 |
+
2.5721
|
| 20 |
+
],
|
| 21 |
+
"tree_full": [
|
| 22 |
+
2.6307,
|
| 23 |
+
2.5901,
|
| 24 |
+
2.5815,
|
| 25 |
+
2.575
|
| 26 |
+
],
|
| 27 |
+
"sign_none": [
|
| 28 |
+
2.4472,
|
| 29 |
+
2.4448,
|
| 30 |
+
2.4365,
|
| 31 |
+
2.4152
|
| 32 |
+
]
|
| 33 |
+
},
|
| 34 |
+
"g0_founder_gap_sign": 0.0024,
|
| 35 |
+
"post_divergence_margins_sign": [
|
| 36 |
+
0.0029,
|
| 37 |
+
0.0027,
|
| 38 |
+
0.0017
|
| 39 |
+
],
|
| 40 |
+
"champion_drift": {
|
| 41 |
+
"sign_weights": [
|
| 42 |
+
0.1757,
|
| 43 |
+
0.196,
|
| 44 |
+
0.1997,
|
| 45 |
+
0.2028
|
| 46 |
+
],
|
| 47 |
+
"sign_full": [
|
| 48 |
+
0.1749,
|
| 49 |
+
0.1116,
|
| 50 |
+
0.0113,
|
| 51 |
+
0.0088
|
| 52 |
+
],
|
| 53 |
+
"tree_weights": [
|
| 54 |
+
0.1393,
|
| 55 |
+
0.162,
|
| 56 |
+
0.1626,
|
| 57 |
+
0.1634
|
| 58 |
+
],
|
| 59 |
+
"tree_full": [
|
| 60 |
+
0.1437,
|
| 61 |
+
0.0607,
|
| 62 |
+
0.0042,
|
| 63 |
+
0.0029
|
| 64 |
+
],
|
| 65 |
+
"sign_none": [
|
| 66 |
+
0.183,
|
| 67 |
+
0.1658,
|
| 68 |
+
0.1765,
|
| 69 |
+
0.1654
|
| 70 |
+
]
|
| 71 |
+
},
|
| 72 |
+
"champion_fid": {
|
| 73 |
+
"sign_weights": [
|
| 74 |
+
0.9423,
|
| 75 |
+
0.954,
|
| 76 |
+
0.9542,
|
| 77 |
+
0.9537
|
| 78 |
+
],
|
| 79 |
+
"sign_full": [
|
| 80 |
+
0.9421,
|
| 81 |
+
0.9554,
|
| 82 |
+
0.9557,
|
| 83 |
+
0.9557
|
| 84 |
+
],
|
| 85 |
+
"sign_none": [
|
| 86 |
+
0.9244,
|
| 87 |
+
0.927,
|
| 88 |
+
0.9279,
|
| 89 |
+
0.9314
|
| 90 |
+
]
|
| 91 |
+
},
|
| 92 |
+
"tree_root_ppl4": {
|
| 93 |
+
"tree_weights": [
|
| 94 |
+
3.284,
|
| 95 |
+
3.758,
|
| 96 |
+
3.759,
|
| 97 |
+
3.77
|
| 98 |
+
],
|
| 99 |
+
"tree_full": [
|
| 100 |
+
3.278,
|
| 101 |
+
3.449,
|
| 102 |
+
3.459,
|
| 103 |
+
3.465
|
| 104 |
+
]
|
| 105 |
+
},
|
| 106 |
+
"n_rows": 80
|
| 107 |
+
}
|