geolip-aleph-qwen

The AlephLM generation stake: composing the certified components of the GeoLIP aleph program into a utilizable generation structure on a real pretrained substrate. Successor to geolip-aleph-differentiation (exp011–exp021, now on hold), whose measurement program produced the design rules this repo spends. Experiment numbering restarts at 1.

The architecture (every component certified upstream)

component	what it is	certified by
Substrate	Qwen2.5-0.5B, frozen	the strongest prior result in the program is the frozen-trunk retrofit: GPT-2 124M ppl 38.65 → 26.53 through aleph relays, beating param-matched adapters 2/2 seeds (exp013)
Aleph relay stack	multi-slot M̂ read after each block, near-zero gated residual (gate init −3.0)	exp013 (gates grow ~3× — the trunk opts in; no init stabilization needed), exp020 (no cost in depth)
Discrete interface	the sign-code tap: winner half-axis + orientation per slot — the aleph logit as a cacheable, transmissible conditioning surface	sign ≥ soft in 12/12 pretrained-substrate cells (exp013); task parity (exp012)
Persistent registry	write-time-frozen aleph key encoder + external KV store — fact memory that later training cannot erase, by construction	exp021: addresses drift with the live trunk (72–91% key scrambling; key-path freezing changes nothing) — the frozen keyer is the requirement the data forced
Content pipe	weak→strong logit distillation for filling students/adapters	exp019: zero-to-negative distillation tax; better than ground truth on structured content 2/2 seeds

Design rules inherited from the closed line: pure Adam wd=0 on geometric paths; decoders read M̂ never M; slot-parallel reconstructive consumption (the collapse cure); CV is a readout; drift-check before any freeze; ≥2 seeds before any claim; every package ships standalone with a self-asserting results builder.

Experiment line

• exp001_relay — the relay retrofit at 0.5B ✅: frozen 17.798 → relay 14.09 / matched-MLP 13.93 (−21% ppl at <0.6% trainable, 2 seeds, replication <0.02). The GPT-2 ppl ordering inverts (substrate-scoped) while the gate mechanism replicates (relay gates grow 0.047→0.081, MLP gates shrink) — the gate-vs-ppl dissociation, certified at two substrates.
• exp002_refine — refining the relay ✅: the aleph-addressed patchwork consumer wins (13.997 mean, 2 seeds, first relay under 14); width/strobe saturated; coverage beats concentration (−0.48); the wide MLP diverges 1/2 seeds vs 0% across all relay variants; ordering budget-stable at 6k steps (2 seeds).
• exp003_instruct — instruct tasks + register differentiation ✅: hallucination reduction quantified (specialist grounding 0.979/0.958 vs frozen 0.542 — >10× fewer failures); register capture found (a single-task specialist answers every register in its own; perfect JSON, 0.0 task-validity); multi-task resolves it (all registers ≥0.875); the register-differentiation hypothesis confirmed on the sign-code surface (~2× deep-layer separation vs a wikitext-trained stack, 2 seeds) — the gauge only the aleph relay exposes.
• exp004_composite — composite construction gate ✅ (two acts, both ledgered): combination-scene targets from the fused datasets (qwen-deepfashion-fused, qwen-synth-characters-fused). Act 1: packed-window training teaches the ~3k-token composite register flawlessly (json_error 0.0 vs frozen 0.833) but never teaches stopping (truncated 1.000; still 0.938 with the cap lifted — true rows average 4.3k tokens, so 1024-token windows almost never contain an ending). Act 2: row-aligned SFT fixes it completely — structural_ok 1.000 (16/16), every failure bucket 0.0. Termination is a property of sample semantics, not the adapter stack. Honest caveats: entity over-emission 1.67×, grounding-lite 0.062 — construction solved, faithfulness is the next target. Ships the long-sequence memory rider (gradient checkpointing + chunked CE; 66 GB spill → 8.8 GB peak).
• exp005_taskmoe — multi-anchor task-MoE ✅ (with its own capacity control): the routed composite of three frozen register specialists + one trainable anchor serves all registers (0.875/1.0/1.0) under dense signed dispatch — but the control decides attribution: random frozen stacks match (0.979/0.979/1.0), so at register distance the specialists were passengers (the monolith-capacity trainable anchor carries; specialists may mildly interfere). No routing collapse (usage_ppl 3.7–3.9 of 4, 4/4 paths).
• exp006_story — story anchors, 4 request methods ✅: continuation/instruct/JSON registers all 1.0 (frozen 0.67/0.83/0.83), but the checkable keyword constraint stays at 0.083 — unmoved from frozen: format learning ≠ constraint compliance (teacher forcing makes ignorable instructions unlearned). Strong register separation (0.46 @L8).
• exp007_math — math anchors, 4 ask-methods ✅: format locks in 4/4 (frozen nl format 0.312), no correctness tax, and held-out f(x) evaluation improves 0.688 → 0.938. With exp006 this yields the predictability principle: the LM loss teaches exactly what target-token predictability demands — answers force using the question; story keywords are ignorable.
• exp008_tokendiff — CIFAR-10 discrete token diffusion through a gated adapter (feasibility gate): the frozen trunk + 5.5M adapter learns a genuine class-conditional discrete denoiser — beats the identity baseline at every corruption level (0.527 vs 0.252 at t=0.75; 0.378 vs 0.002 at pure noise). Greedy iterative decoding mode-collapses to the modal (all-dark) image — the ledgered v1 sampler lesson; stochastic-sampling amendment included.
• exp009_family — THE FAMILY ✅ (the flagship): six anchors — five frozen specialists (3 registers, story, math) + one trainable — under one dense signed dispatch serve 11 sub-competences at ~specialist level through a single routed model. Domain-level dispatch specialization without a selector (story prompts: 0.82 usage on the story anchor) and surgical modular decoupling both directions (story-off kills story, math stays perfect; math-off degrades math, story untouched). With exp005's control: the two-regime dispatch law — anchors specialize-and-carry at domain distance, blend into redundancy at register distance, and the sign-code register probe's separation predicts the regime in advance.

Each experiment lands in expNNN_*/ with code, README, self-asserting results builder, ledger, and checkpoints, reproducible from inside its folder.

Relation to prior work

The upstream repo holds the laws this build rests on: the consumption law, the address-bottleneck's scope, the projective-codebook law, the KD regime map, heredity-as-robustness, the memorization/generalization inverse law, and the frozen-keyer requirement — each with ledgers, controls, and replication seeds. This repo is where they compound.

License

MIT. AbstractPhil + Claude Fable 5.