exp003 + exp007 shipped: sigma is the only live register axis; AMoE v1 monolith falsifier fires under uniform pressure (machinery green) — next: structural band assignment

README.md

@@ -72,10 +72,11 @@ resolution) so gaps live in a narrow band — the paired design is load-bearing.
 | exp000b_natural | natural-paradigm wall: format specialization is a double dissociation; SDXL core tops the NL wall | **shipped (candidate, s0)** |
 | exp001_sd15_relay | relay-all16 vs matched LoRA-r32 vs frozen: **relay beats both; LoRA lands below frozen; post-train toggle bit-exact** | **shipped (candidate, s0)** |
 | exp002_sd15_addrcond | addr-cond 4-arm causal: **GUIDEPOST with a pulse** — inert beside full text (real≈deranged), but the address ALONE steers (+0.029); redesign = complementarity | **shipped (candidate, s0)** |
-| exp003_sigma_registers | sign-code separations: what is a diffusion "register"? (+prompt77 register) | running |
+| exp003_sigma_registers | register probe: **sigma is the ONLY live axis** — caption/type/cond all compress to ≤0; ordering prereg MISS with design consequences | **shipped (candidate, s0)** |
 | exp004_anima_relay | relays on the 2B DiT via diffusion-pipe (bf16 per dtype law) | staged |
 | exp005_sdxl_tree4a | the SDXL capacity battery (guidepost/scaffold/skeleton) | designed |
 | exp006_sd15core_relay | CORE EPRED certification: **relay beats frozen (−2.5%) and matched LoRA 2-for-2; grounding gained (+0.036) where LoRA traded it; gates GROW on the core** | **shipped (candidate, s0)** |
+| exp007_amoe_lora | AMoE v1 vs monolith falsifier: **falsifier fires — uniform pressure ⇒ flat usage, dispatch pays nothing (Tree-1b law on diffusion); machinery green; next = structural band assignment** | **shipped (candidate, s0)** |
 
 Code for each experiment ships in its folder with `results.json` from the run
 ledger. Substrate modules in `substrate/`.

exp003_sigma_registers/README.md

@@ -0,0 +1,42 @@
+# exp003_sigma_registers — what is a diffusion "register"? (read-only probe)
+
+**Question.** Does the two-regime predictor (sign-code separation reads the
+dispatch regime before training) transfer to diffusion states, and which
+candidate register axes actually separate?
+
+**Design.** Deterministic same-seed-init relays on the frozen Lune ckpt-2500
+(zero-init ⇒ the forward is the frozen model's, bit-exact); codes read at 4
+depth-spread sites from the relay's addressing cosines (winner half-axis id +
+orientation — read-only argmax, gauge only). Gauge: inter − intra Hamming with
+the intra diagonal EXCLUDED (the r2 restart — NOT comparable to qwen-line
+numbers). Axes: caption register (prompt / json_prompt / vit_json_prompt /
+**prompt77** — aligned, same 24 rows), sigma band (4 shift-warped centers,
+aligned), cond vs uncond (aligned), subject buckets + generator_type groups
+(domain-style, unaligned, documented).
+
+**Results** (`results.json`, deep site = site12):
+
+| axis | separation | reading |
+|---|---|---|
+| **sigma_band** | **+0.0108 (only positive; grows with depth)** | the live axis |
+| cond_uncond | −0.0187 | compresses |
+| caption_register | −0.0251 (negative at every band × site) | image content dominates; format invisible |
+| generator_type (n=8×4) | ≈ 0 (−0.004..+0.001) | even scene TYPE doesn't separate |
+| subject_bucket | honestly skipped | dataset max ≈6 rows/subject < gauge floor 8 |
+
+**Verdict.** Preregistered ordering MISS (predicted subject > caption > sigma
+> cond/uncond; measured sigma > cond/uncond > caption). On frozen-UNet block
+outputs, **content compresses everything conditioning-side to zero-or-negative
+separation, and sigma/state is the only axis the code surface sees** — the
+multimodal-compression precedent taken to its limit on a third substrate
+class. Design consequence (already acted on): dispatch registers for diffusion
+MoE should be keyed on state/sigma (exp007 wired sigma into its dispatch key),
+and imposed structurally (step-gated band assignment) rather than discovered;
+subject-level registers need richer data (the fused datasets) and possibly
+cond-pathway read sites rather than block outputs.
+
+**Caveats.** Init-time codes (P1-probe precedent) on one substrate; n=24
+aligned rows; negative separations partly reflect the (i,i) same-input
+cross-register pairs (documented gauge property — near-identical registers
+read ≈ −intra/n). GPU cost ≈ 4 min total across three runs (two crashes were
+data-shape asserts, both converted to skip-not-crash; all axes final).

exp003_sigma_registers/dexp003_sigma_probe.py

@@ -0,0 +1,302 @@
+"""dexp003_sigma_probe.py — exp003: sign-code separations on diffusion states.
+
+THE TIER-C GATEKEEPER (read-only, no training): does the two-regime predictor
+transfer to diffusion, and what is a diffusion "register"? Codes are read from
+DETERMINISTIC same-seed-init relays (P1-probe precedent — init-time probing is
+valid; relays are zero-init so the forward is the frozen model's bit-exact).
+
+Register axes probed (prereg ordering, charter 2026-07-16:
+subject-bucket > caption-register > sigma-band > cond/uncond):
+  caption_register — SAME 24 rows rendered as {prompt NL, json_prompt,
+                     vit_json_prompt} (aligned inputs, the gauge's home case).
+  sigma_band       — same rows + cond at 4 shift-warped band centers
+                     (aligned; the conditioning axis text LMs lack).
+  cond_uncond      — json_prompt cond vs zeroed cond (aligned).
+  subject_bucket   — top-4 dominant-subject buckets (n=12 each, content
+                     differs by construction — UNALIGNED, documented; the
+                     domain-style axis).
+Sites: 4 depth-spread relay sites of the 16; codes read at the relay INPUT
+(block output). Gauge: separation_no_diag (intra diagonal EXCLUDED — the r2
+restart; NOT comparable to qwen-line numbers).
+
+Pod: python3 pod2/dexp003_sigma_probe.py --run
+"""
+from __future__ import annotations
+
+import io
+import json
+import os
+import sys
+from collections import Counter, defaultdict
+
+sys.path[:0] = ["pod2", "."]
+
+import torch
+
+from pod_ledger import ledger_run, note
+from aleph_diffusion_core import sign_codes, separation_no_diag
+from d1_lune_sampler import encode_clip_225
+from d1_substrate import attach_relays
+
+DATASET = "AbstractPhil/synthetic-object-relations-json"
+SD_BASE = "stable-diffusion-v1-5/stable-diffusion-v1-5"
+SD_REPO = "AbstractPhil/sd15-flow-lune-json-prompt"
+SD_SUB = "checkpoint-00002500/unet"
+VAE_SCALE, SHIFT, SEED = 0.18215, 2.5, 0
+# N_PER_BUCKET 12 was unreachable on this dataset (240 rows -> ~150 dominant
+# subjects, max bucket 6 — single-object synthetic renders); 4 is the honest
+# ceiling, variance noted in results. generator_type added as a second
+# domain-style axis (bigger groups exist there).
+N_ALIGNED, N_BUCKET_ROWS, N_PER_BUCKET = 24, 240, 4
+PROBE_SITES = (0, 5, 8, 12)          # depth spread of the 16 (asserted names)
+BAND_CENTERS = (0.875, 0.625, 0.375, 0.125)
+CAPTION_COLS = ("prompt", "json_prompt", "vit_json_prompt")
+# natural-paradigm tweak (Phil 2026-07-16): the NL register is ALSO probed at
+# the standard 77-token encode ("prompt77") alongside the 225-chunk encodes
+NL77_REGISTER = "prompt77"
+OUT_DIR = ("/workspace/data/dexp003" if os.path.isdir("/workspace")
+           else os.path.join(os.environ.get("GEOLIP_DATA", "./data"),
+                             "dexp003"))
+
+
+def load_tail_rows(n):
+    from huggingface_hub import HfApi, hf_hub_download
+    import pyarrow.parquet as pq
+    api = HfApi()
+    files = sorted(f for f in api.list_repo_files(DATASET, repo_type="dataset")
+                   if f.endswith(".parquet"))
+    path = hf_hub_download(DATASET, files[-1], repo_type="dataset")
+    tbl = pq.read_table(path)
+    rows = tbl.slice(max(0, tbl.num_rows - n), n).to_pylist()
+    out = []
+    for r in rows:
+        img = r["image"]
+        row = {"image_bytes": img.get("bytes") if isinstance(img, dict)
+               else img,
+               "generator_type": str(r.get("generator_type", "?"))}
+        for c in CAPTION_COLS:
+            v = r.get(c)
+            row[c] = v if isinstance(v, str) else json.dumps(v)
+        out.append(row)
+    return out
+
+
+def dominant_subject(jp: str) -> str:
+    try:
+        subs = json.loads(jp).get("subjects") or []
+        s = subs[0]
+        name = s.get("name") if isinstance(s, dict) else str(s)
+        return (name or "?").strip().lower().split()[-1]
+    except Exception:
+        return "?"
+
+
+class SiteCapture:
+    """Forward hooks on the wrapped blocks: stash relay-INPUT hidden states."""
+
+    def __init__(self, unet, relays, names):
+        self.stash = {}
+        self.handles = []
+        self.sites = {}
+        from aleph_diffusion_core import BlockWithRelay
+        wrapped = [(n, m) for n, m in unet.named_modules()
+                   if isinstance(m, BlockWithRelay)]
+        assert len(wrapped) == len(relays)
+        for si in PROBE_SITES:
+            name, mod = wrapped[si]
+            self.sites[si] = (name, relays[si])
+
+            def mk(si):
+                def hook(m, inp, out):
+                    h = out[0] if isinstance(out, tuple) else out
+                    self.stash[si] = h.detach()
+                return hook
+            self.handles.append(mod.block.register_forward_hook(mk(si)))
+
+    def codes(self):
+        return {si: sign_codes(self.sites[si][1], h).cpu()
+                for si, h in self.stash.items()}
+
+    def remove(self):
+        for h in self.handles:
+            h.remove()
+
+
+def run(device="cuda"):
+    os.makedirs(OUT_DIR, exist_ok=True)
+    import numpy as np
+    from PIL import Image
+    from diffusers import UNet2DConditionModel, AutoencoderKL
+    from transformers import CLIPTextModel, CLIPTokenizer
+
+    torch.manual_seed(SEED)                       # deterministic relay init
+    unet = UNet2DConditionModel.from_pretrained(
+        SD_REPO, subfolder=SD_SUB, torch_dtype=torch.float32).to(device)
+    unet.requires_grad_(False)
+    unet.eval()
+    relays, names = attach_relays(unet)
+    for r in relays:
+        r.assert_zero_init()                      # forward stays frozen-exact
+    cap = SiteCapture(unet, relays, names)
+    vae = AutoencoderKL.from_pretrained(
+        SD_BASE, subfolder="vae", torch_dtype=torch.float32).to(device).eval()
+    tok = CLIPTokenizer.from_pretrained(SD_BASE, subfolder="tokenizer")
+    te = CLIPTextModel.from_pretrained(
+        SD_BASE, subfolder="text_encoder",
+        torch_dtype=torch.float32).to(device).eval()
+
+    @torch.no_grad()
+    def latents_of(rows):
+        px = torch.stack([
+            torch.from_numpy(np.asarray(
+                Image.open(io.BytesIO(r["image_bytes"])).convert("RGB")
+                .resize((512, 512)))).float().permute(2, 0, 1) / 127.5 - 1.0
+            for r in rows]).to(device)
+        g = torch.Generator(device=device).manual_seed(SEED)
+        return vae.encode(px).latent_dist.sample(generator=g) * VAE_SCALE
+
+    @torch.no_grad()
+    def probe(lat, ehs, sigma):
+        g = torch.Generator(device=device).manual_seed(SEED + 99)
+        noise = torch.randn(lat.shape, generator=g, device=device)
+        x = noise * sigma + lat * (1 - sigma)
+        t = torch.full((lat.shape[0],), sigma * 1000.0, device=device)
+        unet(x, t, ehs, return_dict=False)
+        return cap.codes()
+
+    results = {"gauge": "separation_no_diag (intra diagonal excluded; "
+                        "NOT comparable to qwen-line numbers)",
+               "sites": {si: cap.sites[si][0] for si in PROBE_SITES},
+               "band_centers": BAND_CENTERS, "seed": SEED, "axes": {}}
+
+    with ledger_run("dexp003 register probe", budget_h=1.0) as h:
+        rows = load_tail_rows(N_ALIGNED)
+        lat = latents_of(rows)
+        ehs_by_col = {}
+        with torch.no_grad():
+            for c in CAPTION_COLS:
+                ehs_by_col[c] = encode_clip_225(
+                    [r[c] for r in rows], tok, te, device)
+            # natural-paradigm register: standard 77-token encode of the NL col
+            ids = tok([r["prompt"] for r in rows], padding="max_length",
+                      max_length=tok.model_max_length, truncation=True,
+                      return_tensors="pt").input_ids.to(device)
+            ehs_by_col[NL77_REGISTER] = te(ids)[0]
+
+        # axis 1: caption register (aligned), per band x site
+        regs = list(CAPTION_COLS) + [NL77_REGISTER]
+        ax = {}
+        for b, s in enumerate(BAND_CENTERS):
+            codes = {c: probe(lat, ehs_by_col[c], s) for c in regs}
+            for si in PROBE_SITES:
+                ax[f"band{b}_site{si}"] = round(separation_no_diag(
+                    {c: codes[c][si] for c in regs}), 4)
+        results["axes"]["caption_register"] = ax
+        print(f"[exp003] caption_register: {ax}", flush=True)
+
+        # axis 2: sigma band (aligned; json_prompt cond), per site
+        codes_by_band = {f"s{b}": probe(lat, ehs_by_col["json_prompt"], s)
+                         for b, s in enumerate(BAND_CENTERS)}
+        ax = {f"site{si}": round(separation_no_diag(
+            {k: v[si] for k, v in codes_by_band.items()}), 4)
+            for si in PROBE_SITES}
+        results["axes"]["sigma_band"] = ax
+        print(f"[exp003] sigma_band: {ax}", flush=True)
+
+        # axis 3: cond vs uncond (aligned), mid band, per site
+        cu = {"cond": probe(lat, ehs_by_col["json_prompt"], 0.625),
+              "uncond": probe(lat, torch.zeros_like(
+                  ehs_by_col["json_prompt"]), 0.625)}
+        ax = {f"site{si}": round(separation_no_diag(
+            {k: v[si] for k, v in cu.items()}), 4) for si in PROBE_SITES}
+        results["axes"]["cond_uncond"] = ax
+        print(f"[exp003] cond_uncond: {ax}", flush=True)
+
+        # axis 4: domain-style groups (UNALIGNED, documented): subject buckets
+        # (n small on this dataset — variance flag) + generator_type groups
+        pool = load_tail_rows(N_BUCKET_ROWS)
+
+        def group_axis(name, keyfn, min_n):
+            groups = defaultdict(list)
+            for r in pool:
+                groups[keyfn(r)].append(r)
+            top = [s for s, c in Counter(
+                {k: len(v) for k, v in groups.items()}).most_common(6)
+                if len(groups[s]) >= min_n and s != "?"][:4]
+            if len(top) < 2:
+                results["axes"][name] = {
+                    "skipped": f"only {len(top)} groups >= {min_n}",
+                    "sizes": dict(Counter(
+                        {k: len(v) for k, v in groups.items()}))}
+                print(f"[exp003] {name}: SKIPPED (group sizes too small)",
+                      flush=True)
+                return
+            codes = {}
+            for s in top:
+                rs = groups[s][:min_n]
+                l = latents_of(rs)
+                with torch.no_grad():
+                    e = encode_clip_225([r["json_prompt"] for r in rs], tok,
+                                        te, device)
+                codes[s] = probe(l, e, 0.625)
+            try:
+                ax = {f"site{si}": round(separation_no_diag(
+                    {k: v[si] for k, v in codes.items()}), 4)
+                    for si in PROBE_SITES}
+            except AssertionError as e:
+                # gauge n-floor (>=8): axis honestly unmeasurable at this n —
+                # record, never crash (this dataset's subjects max out at ~6)
+                results["axes"][name] = {"groups": top, "n_per_group": min_n,
+                                         "skipped": f"gauge n-floor: {e}"}
+                print(f"[exp003] {name}: SKIPPED ({e})", flush=True)
+                return
+            results["axes"][name] = {"groups": top, "n_per_group": min_n, **ax}
+            print(f"[exp003] {name} ({top}): {ax}", flush=True)
+
+        group_axis("subject_bucket",
+                   lambda r: dominant_subject(r["json_prompt"]), N_PER_BUCKET)
+        group_axis("generator_type", lambda r: r["generator_type"], 8)
+
+        # prereg ordering check at the deepest probed site
+        deep = f"site{PROBE_SITES[-1]}"
+        order = {
+            "caption_register": results["axes"]["caption_register"]
+            [f"band1_{deep}"],
+            "sigma_band": results["axes"]["sigma_band"][deep],
+            "cond_uncond": results["axes"]["cond_uncond"][deep],
+        }
+        sb = results["axes"].get("subject_bucket", {})
+        if deep in sb:
+            order["subject_bucket"] = sb[deep]
+        ranked = sorted(order, key=order.get, reverse=True)
+        results["prereg"] = {
+            "predicted_order": ["subject_bucket", "caption_register",
+                                "sigma_band", "cond_uncond"],
+            "measured_order": ranked,
+            "measured_values_at_deep_site": order,
+            "order_hit": ranked[:len(order)] == [
+                k for k in ["subject_bucket", "caption_register",
+                            "sigma_band", "cond_uncond"] if k in order],
+        }
+        h["verdict"] = f"order {'HIT' if results['prereg']['order_hit'] else 'MISS'}: {ranked}"
+
+    cap.remove()
+    with open(os.path.join(OUT_DIR, "results.json"), "w") as f:
+        json.dump(results, f, indent=2)
+    note(f"dexp003: {json.dumps(results['prereg'])}")
+    print(json.dumps(results["prereg"], indent=2))
+    return results
+
+
+def smoke():
+    assert dominant_subject('{"subjects":[{"name":"a red car"}]}') == "car"
+    assert dominant_subject('{"subjects":["tall man"]}') == "man"
+    assert dominant_subject("not json") == "?"
+    print("dexp003 smoke PASSED (parse; GPU run is pod work)")
+
+
+if __name__ == "__main__":
+    if "--run" in sys.argv:
+        run()
+    else:
+        smoke()

exp003_sigma_registers/results.json

@@ -0,0 +1,90 @@
+{
+  "gauge": "separation_no_diag (intra diagonal excluded; NOT comparable to qwen-line numbers)",
+  "sites": {
+    "0": "down_blocks.0.attentions.0.transformer_blocks.0",
+    "5": "down_blocks.2.attentions.1.transformer_blocks.0",
+    "8": "up_blocks.1.attentions.2.transformer_blocks.0",
+    "12": "up_blocks.3.attentions.0.transformer_blocks.0"
+  },
+  "band_centers": [
+    0.875,
+    0.625,
+    0.375,
+    0.125
+  ],
+  "seed": 0,
+  "axes": {
+    "caption_register": {
+      "band0_site0": -0.0272,
+      "band0_site5": -0.0094,
+      "band0_site8": -0.0106,
+      "band0_site12": -0.0185,
+      "band1_site0": -0.0312,
+      "band1_site5": -0.017,
+      "band1_site8": -0.0208,
+      "band1_site12": -0.0251,
+      "band2_site0": -0.035,
+      "band2_site5": -0.0234,
+      "band2_site8": -0.028,
+      "band2_site12": -0.0316,
+      "band3_site0": -0.0368,
+      "band3_site5": -0.0274,
+      "band3_site8": -0.0316,
+      "band3_site12": -0.0348
+    },
+    "sigma_band": {
+      "site0": 0.0039,
+      "site5": -0.0,
+      "site8": -0.0004,
+      "site12": 0.0108
+    },
+    "cond_uncond": {
+      "site0": -0.0269,
+      "site5": -0.0122,
+      "site8": -0.013,
+      "site12": -0.0187
+    },
+    "subject_bucket": {
+      "groups": [
+        "garlic",
+        "cucumber",
+        "potato",
+        "banana"
+      ],
+      "n_per_group": 4,
+      "skipped": "gauge n-floor: too few aligned inputs"
+    },
+    "generator_type": {
+      "groups": [
+        "produce_pair",
+        "two_on_surface",
+        "object_on_surface",
+        "item_in_container"
+      ],
+      "n_per_group": 8,
+      "site0": -0.0041,
+      "site5": -0.0001,
+      "site8": 0.0008,
+      "site12": -0.0009
+    }
+  },
+  "prereg": {
+    "predicted_order": [
+      "subject_bucket",
+      "caption_register",
+      "sigma_band",
+      "cond_uncond"
+    ],
+    "measured_order": [
+      "sigma_band",
+      "cond_uncond",
+      "caption_register"
+    ],
+    "measured_values_at_deep_site": {
+      "caption_register": -0.0251,
+      "sigma_band": 0.0108,
+      "cond_uncond": -0.0187
+    },
+    "order_hit": false
+  }
+}

exp007_amoe_lora/README.md

@@ -0,0 +1,49 @@
+# exp007_amoe_lora — AMoE-LoRA v1: the first dispatch test (CANDIDATE, s0)
+
+**Question.** Does a bank of experts under dense signed aleph dispatch (keyed
+on hidden state + sigma) beat a parameter-matched monolith on the certified
+core-epred substrate? First MoE-structure experiment of the line.
+
+**Structure per site (16 sites).** A=4 zero-init rank-8 low-rank residual
+experts; dispatch = dense SIGNED aleph coefficients per token (sinh/cosh over
+A oriented axes — no softmax gate, no top-k, no balance loss; usage read-only)
+with sigma Fourier features added to the dispatch key (the exp003 finding:
+sigma is the live register axis). Combine: `x + Σ σ(gate_k)·w_k(x,σ)·Δ_k(x)`.
+Falsifier: one rank-32 delta per site, same placement, no dispatch (the
+matched-monolith control). 3000 steps each, batch 16, pure Adam wd=0.
+
+**Results** (`results.json`):
+
+| arm | trainable | val eps-MSE | Δ vs frozen |
+|---|---|---|---|
+| frozen | 0 | 0.124788 | — |
+| amoe A4r8 | 899,392 | 0.122015 | −2.2% |
+| **monolith r32** | 811,216 | **0.121777** | **−2.4%** |
+
+| prereg | outcome |
+|---|---|
+| P1 bank ≥ monolith | **MISS** (monolith wins by 0.0002 with 11% fewer params) |
+| P2 sigma-band usage differentiation | **MISS** — spread 0.0009; usage ≈ [0.25, 0.17, 0.33, 0.24] in EVERY band |
+| P3 toggle bit-exact post-train | HIT (≡ frozen to full precision) |
+| P4 no dead experts | HIT (all four alive, 0.17–0.33) |
+
+**Verdict (the falsifier did its job).** Under UNIFORM training pressure — one
+dataset, one objective, one register — dispatch pays nothing over a single
+matched-width delta, and the sigma-keyed dispatch never uses its sigma
+freedom. This reproduces the text line's Tree-1b/exp009 law on a diffusion
+trunk: *specialization requires differentiated pressure; the aggregation
+channel alone doesn't create it.* The experts blended (uniform usage) exactly
+as the two-regime law predicts for register-distance-zero mixtures.
+
+**What it buys the program.** The machinery is sound (parity, liveness,
+signed dense dispatch all green) and per-param the bank units are efficient
+(−2.2% at 0.9M vs the relay's −2.5% at 3.2M). The next MoE test must IMPOSE
+structure rather than hope dispatch discovers it in uniform data — the
+designed follow-on is the step-gated controller: experts ASSIGNED to sigma
+bands with per-band loss masking (coarse-to-fine), judged against this same
+monolith. Subject-bucketed expert assignment is the second axis (needs the
+richer fused datasets; this synthetic set's subject buckets max out at n≈6).
+
+**Caveats.** Single seed; 3k steps; A=4/r=8 single scale point; param match
+1.109 (bank heavier — disclosed); one substrate. Cost ≈ 3.0 GPU-h.
+Checkpoints: `amoe_core_s0.pt`, `mono_core_s0.pt`.

exp007_amoe_lora/amoe_core_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37928cf2d2ce85b4ca38d160c428e2fc118aa884dc882238631716a1764a7c45
+size 3681565

exp007_amoe_lora/dexp007_amoe_lora.py

@@ -0,0 +1,408 @@
+"""dexp007_amoe_lora.py — exp007: AMoE-LoRA v1 — the first dispatch test.
+
+THE FIRST MoE-STRUCTURE EXPERIMENT of the line (Phil: the MoE is the weekend
+goal; hit it only as fast as results yield). GATED: launches only after
+exp006 certifies the anchor unit on the epred core (env DEXP7_SUBSTRATE=core)
+— else holds, or runs on the exp001-certified lune trunk (=lune).
+
+Structure per site (16 BasicTransformerBlocks), doctrine-compliant
+(progression-plan Part II.2 — no softmax gate, no top-k, no balance loss):
+  experts   — A=4 zero-init low-rank residual deltas on the block output:
+              down Linear(d, r=8, bias=False) -> up Linear(r=8, d) zero W+b.
+  dispatch  — DENSE SIGNED aleph coefficients: key = normalize(key_proj(
+              x_token) + sig_proj(fourier(sigma))) in D=4; codebook K=A axes;
+              w_k = sinh(u_k)/sum cosh(u_j) per token — signed, no selection
+              event; usage logged READ-ONLY.
+  combine   — x + sum_k sigmoid(gate_k) * w_k * expert_k(x); gates init -3.0.
+  toggle    — enabled=False returns x (bit-exact law preserved).
+Sigma enters the DISPATCH KEY (exp003 finding: sigma/state is the live
+register axis; caption registers compress).
+
+Arms (shared cache, paired val, pure Adam wd=0, dtype matched to trunk):
+  frozen | amoe (A=4, r=8) | monolith LoRA-shape rank A*r=32 on the same
+  placement (THE FALSIFIER — matched total params, no dispatch; if it matches
+  the bank everywhere, dispatch pays nothing at this scale).
+
+Prereg (charter): P1 amoe >= monolith on paired val; P2 dispatch usage
+DIFFERENTIATES by sigma band (variance of per-band mean |w| exceeds a
+shuffled-band null x3); P3 toggle bit-exact post-train; P4 no dead experts
+(every expert's mean |w| > 0.02 of total). Honest negative: amoe ~ monolith
+=> the aggregation channel (Law 1) needs rework before any bigger MoE.
+
+Pod: DEXP7_SUBSTRATE=core|lune bash pod2/run_exp007.sh   (time-boxed: steps
+sized so the run NEVER crosses the 8am PT halt)
+"""
+from __future__ import annotations
+
+import io
+import json
+import math
+import os
+import sys
+import time
+
+sys.path[:0] = ["pod2", "."]
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from pod_ledger import ledger_run, note, burn_down
+from aleph_diffusion_core import derangement, sigma_band, SIGMA_BANDS
+from d1_substrate import enumerate_sd15_sites, MEM_FRACTION
+
+SUBSTRATE = os.environ.get("DEXP7_SUBSTRATE", "core")
+SD_BASE = "stable-diffusion-v1-5/stable-diffusion-v1-5"
+LUNE_REPO = "AbstractPhil/sd15-flow-lune-json-prompt"
+LUNE_SUB = "checkpoint-00002500/unet"
+N_TRAIN, N_VAL = 4096, 256
+A_EXPERTS, RANK = 4, 8
+BATCH = int(os.environ.get("DEXP7_BATCH", "16"))
+STEPS = int(os.environ.get("DEXP7_STEPS", "3000"))
+LR, CFG_DROPOUT, SEED, SHIFT, VAE_SCALE = 1e-3, 0.1, 0, 2.5, 0.18215
+DATA_DIR = ("/workspace/data/dexp007" if os.path.isdir("/workspace")
+            else os.path.join(os.environ.get("GEOLIP_DATA", "./data"),
+                              "dexp007"))
+CKPT_DIR = ("/workspace/ckpts2/dexp007" if os.path.isdir("/workspace")
+            else DATA_DIR)
+
+
+def fourier_sigma(s, dim=8):
+    """s in [0,1] (or t/1000) -> small Fourier features [B, dim]."""
+    freqs = torch.pow(2.0, torch.arange(dim // 2, device=s.device))
+    ang = 2 * math.pi * s[:, None] * freqs[None]
+    return torch.cat([torch.sin(ang), torch.cos(ang)], dim=-1)
+
+
+class AmoeBank(nn.Module):
+    """A experts + dense signed aleph dispatch, one bank per site."""
+
+    def __init__(self, d: int, A: int = A_EXPERTS, r: int = RANK,
+                 D: int = 4, tau: float = 0.1):
+        super().__init__()
+        self.d, self.A, self.r, self.tau = d, A, r, tau
+        self.down = nn.ModuleList(nn.Linear(d, r, bias=False)
+                                  for _ in range(A))
+        self.up = nn.ModuleList(nn.Linear(r, d) for _ in range(A))
+        for dn, up in zip(self.down, self.up):
+            nn.init.orthogonal_(dn.weight)
+            nn.init.zeros_(up.weight)               # zero-init out: weight
+            nn.init.zeros_(up.bias)                 # AND bias (leak law)
+        self.key_proj = nn.Linear(d, D, bias=False)
+        nn.init.orthogonal_(self.key_proj.weight)
+        self.sig_proj = nn.Linear(8, D, bias=False)
+        nn.init.orthogonal_(self.sig_proj.weight)
+        self.codebook = nn.Parameter(
+            F.normalize(torch.randn(A, D), dim=-1))
+        self.gates = nn.Parameter(torch.full((A,), -3.0))
+        self.enabled = True
+        self.last_usage = None                      # read-only telemetry
+
+    def assert_zero_init(self):
+        for up in self.up:
+            assert up.weight.abs().max().item() == 0.0
+            assert up.bias.abs().max().item() == 0.0
+
+    def dispatch(self, x, sig_feat):
+        """Dense signed coefficients per token: (..., A). No selection event."""
+        sig = self.sig_proj(sig_feat)                 # (B, D)
+        sig = sig.view(sig.shape[0], *([1] * (x.ndim - 2)), sig.shape[-1])
+        key = self.key_proj(x) + sig                  # rank-aware broadcast
+        Ax = F.normalize(self.codebook, dim=-1)
+        u = (F.normalize(key, dim=-1) @ Ax.transpose(-1, -2)) / self.tau
+        m = u.abs().amax(dim=-1, keepdim=True)
+        ep, en = torch.exp(u - m), torch.exp(-u - m)
+        return (ep - en) / (ep + en).sum(dim=-1, keepdim=True)
+
+    def forward(self, x, sig_feat):
+        if not self.enabled:
+            return x
+        w = self.dispatch(x, sig_feat)               # (B, T, A) signed
+        with torch.no_grad():
+            self.last_usage = w.abs().mean(dim=(0, 1)).detach().cpu()
+        delta = 0
+        g = torch.sigmoid(self.gates)
+        for k in range(self.A):
+            delta = delta + g[k] * w[..., k:k + 1] * self.up[k](
+                self.down[k](x))
+        return x + delta
+
+
+class BlockWithBank(nn.Module):
+    """Wrap: bank reads block output + the batch sigma stashed on self."""
+
+    def __init__(self, block, bank):
+        super().__init__()
+        self.block = block
+        self.bank = bank
+        self.sig_feat = None                         # set per step by the bed
+
+    def forward(self, *args, **kwargs):
+        out = self.block(*args, **kwargs)
+        h = out[0] if isinstance(out, tuple) else out
+        h = self.bank(h, self.sig_feat)
+        return (h,) + out[1:] if isinstance(out, tuple) else h
+
+
+class MonoLora(nn.Module):
+    """The falsifier: one rank-A*r delta, same placement, no dispatch."""
+
+    def __init__(self, d: int, r: int = A_EXPERTS * RANK):
+        super().__init__()
+        self.down = nn.Linear(d, r, bias=False)
+        nn.init.orthogonal_(self.down.weight)
+        self.up = nn.Linear(r, d)
+        nn.init.zeros_(self.up.weight)
+        nn.init.zeros_(self.up.bias)
+        self.gate = nn.Parameter(torch.tensor(-3.0))
+        self.enabled = True
+
+    def forward(self, x, sig_feat=None):
+        if not self.enabled:
+            return x
+        return x + torch.sigmoid(self.gate) * self.up(self.down(x))
+
+
+def load_unet(device):
+    from diffusers import UNet2DConditionModel
+    if SUBSTRATE == "lune":
+        unet = UNet2DConditionModel.from_pretrained(
+            LUNE_REPO, subfolder=LUNE_SUB, torch_dtype=torch.float32)
+    else:
+        unet = UNet2DConditionModel.from_pretrained(
+            SD_BASE, subfolder="unet", torch_dtype=torch.float32)
+    unet = unet.to(device)
+    unet.requires_grad_(False)
+    unet.eval()
+    unet.enable_gradient_checkpointing()
+    return unet
+
+
+def attach_banks(unet, mk):
+    sites = enumerate_sd15_sites(unet)
+    assert len(sites) == 16
+    mods, wraps = nn.ModuleList(), []
+    for name, block, d in sites:
+        p0 = next(block.parameters())
+        m = mk(d).to(device=p0.device, dtype=p0.dtype)
+        w = BlockWithBank(block, m)
+        parent = unet
+        parts = name.split(".")
+        for p in parts[:-1]:
+            parent = getattr(parent, p) if not p.isdigit() else parent[int(p)]
+        if parts[-1].isdigit():
+            parent[int(parts[-1])] = w
+        else:
+            setattr(parent, parts[-1], w)
+        mods.append(m)
+        wraps.append(w)
+    return mods, wraps
+
+
+def get_cache(device):
+    """Reuse the substrate-matching cache (dexp001 for lune / dexp006 core)."""
+    src = "/workspace/data/dexp001" if SUBSTRATE == "lune" \
+        else "/workspace/data/dexp006"
+    f = os.path.join(src, "cache.pt")
+    assert os.path.exists(f), f"needs {f} (run the substrate's exp first)"
+    return torch.load(f, map_location="cpu", weights_only=True)
+
+
+def sigma_of(cache_has_t, gen, bsz, device):
+    if cache_has_t:                                   # epred core: t in 0..999
+        t = torch.randint(0, 1000, (bsz,), generator=gen, device=device)
+        return t, t.float() / 1000.0
+    s = torch.rand(bsz, generator=gen, device=device)
+    s = (SHIFT * s) / (1 + (SHIFT - 1) * s)
+    return s * 1000, s
+
+
+def run(device="cuda"):
+    torch.cuda.set_per_process_memory_fraction(MEM_FRACTION, 0)
+    os.makedirs(CKPT_DIR, exist_ok=True)
+    os.makedirs(DATA_DIR, exist_ok=True)
+    cache = get_cache(device)
+    epred = SUBSTRATE != "lune"
+    if epred:
+        from dexp006_sd15core_relay import make_schedule, add_noise
+        acp = make_schedule(device)
+
+    def loss_of(unet, wraps, lat, ehs, gen):
+        bsz = lat.shape[0]
+        drop = torch.rand(bsz, generator=gen, device=device) < CFG_DROPOUT
+        ehs = ehs.clone()
+        ehs[drop] = 0
+        t, s01 = sigma_of(epred, gen, bsz, device)
+        sf = fourier_sigma(s01).to(lat.dtype)
+        for w in wraps:
+            w.sig_feat = sf
+        noise = torch.randn(lat.shape, generator=gen, device=device)
+        if epred:
+            noisy = add_noise(lat, noise, t.long(), acp)
+            target = noise
+        else:
+            s4 = s01[:, None, None, None]
+            noisy = noise * s4 + lat * (1 - s4)
+            target = noise - lat
+        pred = unet(noisy, t, ehs, return_dict=False)[0]
+        return F.mse_loss(pred, target)
+
+    @torch.no_grad()
+    def val(unet, wraps):
+        tot, band_usage = [], {i: [] for i in range(len(SIGMA_BANDS))}
+        for i in range(0, N_VAL, 32):
+            lat = cache["val_lat"][i:i + 32].to(device)
+            ehs = cache["val_ehs"][i:i + 32].to(device)
+            noise = cache["val_noise"][i:i + 32].to(device)
+            if epred:
+                t = cache["val_t"][i:i + 32].to(device)
+                s01 = t.float() / 1000.0
+                noisy, target = add_noise(lat, noise, t, acp), noise
+            else:
+                s = cache["val_sigma"][i:i + 32].to(device)
+                s01 = s
+                s4 = s[:, None, None, None]
+                noisy, target = noise * s4 + lat * (1 - s4), noise - lat
+                t = s * 1000
+            sf = fourier_sigma(s01).to(lat.dtype)
+            for w in wraps:
+                w.sig_feat = sf
+            pred = unet(noisy, t, ehs, return_dict=False)[0]
+            tot += ((pred - target) ** 2).mean(dim=(1, 2, 3)).tolist()
+            for w in wraps:
+                u = getattr(w.bank, "last_usage", None)
+                if u is not None:
+                    for j, sv in enumerate(s01.tolist()):
+                        band_usage[sigma_band(sv)].append(u.tolist())
+        usage = {f"band{k}":
+                 [round(sum(col) / len(col), 4) for col in zip(*v)]
+                 if v else None for k, v in band_usage.items()}
+        return sum(tot) / len(tot), usage
+
+    def train(unet, wraps, params, label):
+        opt = torch.optim.Adam(params, lr=LR, weight_decay=0.0)
+        gen = torch.Generator(device=device).manual_seed(SEED + 42)
+        idx = torch.Generator().manual_seed(SEED + 7)
+        torch.cuda.reset_peak_memory_stats()
+        t0 = time.time()
+        for step in range(1, STEPS + 1):
+            sel = torch.randint(0, N_TRAIN, (BATCH,), generator=idx)
+            loss = loss_of(unet, wraps, cache["lat"][sel].to(device),
+                           cache["ehs"][sel].to(device), gen)
+            loss.backward()
+            opt.step()
+            opt.zero_grad(set_to_none=True)
+            if step == 50 or step % 500 == 0:
+                print(f"[{label}] step {step}: loss {loss.item():.4f} | "
+                      f"{(time.time() - t0) / step:.2f}s/step | peak "
+                      f"{torch.cuda.max_memory_allocated() / 2**30:.1f}GB",
+                      flush=True)
+        return round((time.time() - t0) / STEPS, 3)
+
+    results = {"config": {"substrate": SUBSTRATE, "A": A_EXPERTS, "r": RANK,
+                          "steps": STEPS, "batch": BATCH, "lr": LR,
+                          "seed": SEED, "epred": epred}}
+
+    with ledger_run(f"dexp007 frozen ({SUBSTRATE})", budget_h=0.2) as h:
+        unet = load_unet(device)
+        v, _ = val(unet, [])
+        results["frozen"] = {"val": v}
+        del unet
+        torch.cuda.empty_cache()
+        h["verdict"] = f"val {v:.5f}"
+
+    with ledger_run("dexp007 amoe A4r8", budget_h=2.5) as h:
+        unet = load_unet(device)
+        banks, wraps = attach_banks(unet, lambda d: AmoeBank(d))
+        for b in banks:
+            b.assert_zero_init()
+        n_params = sum(p.numel() for p in banks.parameters())
+        print(f"[amoe] 16 banks, {n_params:,} trainable", flush=True)
+        spd = train(unet, wraps, banks.parameters(), "amoe")
+        v_on, usage = val(unet, wraps)
+        for b in banks:
+            b.enabled = False
+        v_off, _ = val(unet, wraps)
+        d = abs(v_off - results["frozen"]["val"])
+        assert d < 1e-9, f"toggle parity broken: {d}"
+        for b in banks:
+            b.enabled = True
+        torch.save({"banks": [b.state_dict() for b in banks]},
+                   os.path.join(CKPT_DIR, f"amoe_{SUBSTRATE}_s0.pt"))
+        # P2: sigma-band usage differentiation vs shuffled-band null
+        per_band = [u for u in usage.values() if u]
+        band_var = 0.0
+        if len(per_band) >= 2:
+            cols = list(zip(*per_band))
+            band_var = sum(
+                (max(c) - min(c)) for c in cols) / len(cols)
+        results["amoe"] = {"n_params": n_params, "val": v_on,
+                           "val_toggled_off": v_off, "s_per_step": spd,
+                           "usage_by_band": usage,
+                           "band_usage_spread": round(band_var, 5),
+                           "gates": [round(torch.sigmoid(g).item(), 4)
+                                     for g in banks[8].gates]}
+        del unet, banks
+        torch.cuda.empty_cache()
+        h["verdict"] = f"val {v_on:.5f} spread {band_var:.4f}"
+
+    with ledger_run("dexp007 monolith r32", budget_h=2.5) as h:
+        unet = load_unet(device)
+        monos, wraps = attach_banks(unet, lambda d: MonoLora(d))
+        n_params = sum(p.numel() for p in monos.parameters())
+        print(f"[mono] 16 sites, {n_params:,} trainable", flush=True)
+        spd = train(unet, wraps, monos.parameters(), "mono")
+        v, _ = val(unet, wraps)
+        torch.save({"monos": [m.state_dict() for m in monos]},
+                   os.path.join(CKPT_DIR, f"mono_{SUBSTRATE}_s0.pt"))
+        results["monolith"] = {"n_params": n_params, "val": v,
+                               "s_per_step": spd}
+        del unet, monos
+        torch.cuda.empty_cache()
+        h["verdict"] = f"val {v:.5f}"
+
+    av, mv = results["amoe"]["val"], results["monolith"]["val"]
+    results["verdict"] = {
+        "amoe_vs_monolith": "amoe" if av <= mv else "monolith",
+        "amoe_beats_frozen": av < results["frozen"]["val"],
+        "param_ratio": round(results["amoe"]["n_params"]
+                             / results["monolith"]["n_params"], 3),
+        "band_usage_spread": results["amoe"]["band_usage_spread"],
+        "note": "P1 amoe>=monolith; P2 sigma-band usage differentiation; "
+                "1-seed CANDIDATE; first MoE-structure test of the line",
+    }
+    with open(os.path.join(DATA_DIR, "results.json"), "w") as f:
+        json.dump(results, f, indent=2)
+    note(f"dexp007 verdict: {json.dumps(results['verdict'])}")
+    print(json.dumps(results["verdict"], indent=2))
+    burn_down()
+    return results
+
+
+def smoke():
+    b = AmoeBank(320)
+    b.assert_zero_init()
+    x = torch.randn(2, 9, 320)
+    sf = fourier_sigma(torch.tensor([0.3, 0.8]))
+    assert torch.equal(b(x, sf), x), "zero-init must be exact"
+    b.enabled = False
+    assert b(x, sf) is x
+    b.enabled = True
+    w = b.dispatch(x, sf)
+    assert w.shape == (2, 9, A_EXPERTS)
+    assert (w.abs().sum(-1) > 0).all(), "dense signed dispatch must be alive"
+    nn.init.normal_(b.up[0].weight, std=0.02)
+    assert not torch.equal(b(x, sf), x)
+    m = MonoLora(320)
+    assert torch.equal(m(x), x)
+    ba = sum(p.numel() for p in AmoeBank(320).parameters())
+    mo = sum(p.numel() for p in MonoLora(320).parameters())
+    print(f"dexp007 smoke PASSED (bank zero-init/toggle/dispatch; params "
+          f"bank {ba:,} vs mono {mo:,} @d=320)")
+
+
+if __name__ == "__main__":
+    if "--run" in sys.argv:
+        run()
+    else:
+        smoke()

exp007_amoe_lora/mono_core_s0.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45d6562a978296fa9348786f83319d11196e21ead6ab3d0540562a79d2328d17
+size 3265793

exp007_amoe_lora/results.json

@@ -0,0 +1,66 @@
+{
+  "config": {
+    "substrate": "core",
+    "A": 4,
+    "r": 8,
+    "steps": 3000,
+    "batch": 16,
+    "lr": 0.001,
+    "seed": 0,
+    "epred": true
+  },
+  "frozen": {
+    "val": 0.1247876692286809
+  },
+  "amoe": {
+    "n_params": 899392,
+    "val": 0.12201545690095372,
+    "val_toggled_off": 0.1247876692286809,
+    "s_per_step": 1.722,
+    "usage_by_band": {
+      "band0": [
+        0.2544,
+        0.1704,
+        0.3321,
+        0.2424
+      ],
+      "band1": [
+        0.2549,
+        0.1706,
+        0.3316,
+        0.2421
+      ],
+      "band2": [
+        0.254,
+        0.1705,
+        0.3323,
+        0.2423
+      ],
+      "band3": [
+        0.2543,
+        0.1696,
+        0.333,
+        0.2424
+      ]
+    },
+    "band_usage_spread": 0.0009,
+    "gates": [
+      0.064,
+      0.0523,
+      0.0644,
+      0.0609
+    ]
+  },
+  "monolith": {
+    "n_params": 811216,
+    "val": 0.12177696034905239,
+    "s_per_step": 1.647
+  },
+  "verdict": {
+    "amoe_vs_monolith": "monolith",
+    "amoe_beats_frozen": true,
+    "param_ratio": 1.109,
+    "band_usage_spread": 0.0009,
+    "note": "P1 amoe>=monolith; P2 sigma-band usage differentiation; 1-seed CANDIDATE; first MoE-structure test of the line"
+  }
+}