anima_closeout v14: full resume rebuild — every arm now resolves LOCAL -> HUB -> train off one ARM_SHIP registry that drives shipping AND restoring, so a trained arm is never retrained and a fresh runtime costs a ~29MB download per arm. Restored arms carry their eval_curve.json (no tfevents exist), smokes skip when a full arm already proves the same thing, CELL 6 rebuilds the held-out cache itself (1-step cache_eval over the eval TOML) and resolves its own checkpoints, CELL 7 reads results from disk. exp017's surgical verdict gained an effect-size guard: the shipped mb3 gates sit at sigmoid(-3.6..-7.8), so a lesion ratio on a near-inert stack is reported VACUOUS, not PASS. Verified against the live hub: real restore of relay_s0/s1 + mb3, no training launched, newest_save refuses restored dirs, ship-skip stateless; builder now enforces cross-cell name resolution over 3 run orders.

colab/anima_closeout.ipynb

@@ -29,9 +29,27 @@
     "| **exp017** | multiband3 on the Anima DiT \u2014 first live run of the fork's mode | the band mechanism has never run on a DiT |\n",
     "| **the fix** | `bf16_master_weights = true` + declared `seed` | exp004's 28 gates were **quantization-frozen**: plain Adam stepped bf16 leaves directly; ~4.5e-4 updates vs 0.0078 half-ULP at \u22123.0 rounded to no-ops (proven from the salvaged Adam state) |\n",
     "\n",
-    "Run cells top to bottom. Everything is idempotent: after a disconnect,\n",
-    "re-run **CELL 0**, then re-run the interrupted run cell (checkpoints every\n",
-    "30 min; pass `resume=True` to `run_arm`).\n",
+    "## Nothing is ever trained twice\n",
+    "\n",
+    "Every arm resolves **LOCAL \u2192 HUB \u2192 train**:\n",
+    "\n",
+    "1. a final-step checkpoint on this disk is reused as-is;\n",
+    "2. else the arm is pulled back from its shipped hub package (a fresh\n",
+    "   runtime after a pod cull costs a ~60 MB download per arm, not a\n",
+    "   training run) \u2014 its eval curve rides along, since a restored arm has\n",
+    "   no tfevents;\n",
+    "3. only a genuinely missing arm trains.\n",
+    "\n",
+    "CELL 2 prints the inventory (`local` / `hub` / `MISSING`) before anything\n",
+    "spends. The smokes are skipped when a full arm already proves the same\n",
+    "thing, CELL 6 rebuilds the held-out cache by itself if every arm was\n",
+    "restored, and CELLs 6\u20137 resolve their own checkpoints \u2014 so you can run\n",
+    "**0 \u2192 1 \u2192 2 \u2192 6 \u2192 7** in a brand-new runtime and get results without a\n",
+    "single training step.\n",
+    "\n",
+    "Run cells top to bottom. After a disconnect, re-run **CELL 0**, then the\n",
+    "interrupted cell (checkpoints land every 30 min; `run_arm(tag,\n",
+    "resume=True)` continues a half-finished run instead of restarting it).\n",
     "\n",
     "**Needs:** Colab secret `HF_TOKEN` (notebook access ON) with write access\n",
     "to `AbstractPhil/geolip-aleph-diffusion`.\n",
@@ -410,11 +428,12 @@
     "# -- training TOMLs per arm -----------------------------------------------\n",
     "_AP = WORK / \"models\" / \"anima\" / \"split_files\"\n",
     "\n",
-    "def _arm_toml(tag, seed, steps, mode=\"relay\", lora=False):\n",
+    "def _arm_toml(tag, seed, steps, mode=\"relay\", lora=False,\n",
+    "              dataset=\"dataset_train.toml\", evals=True):\n",
     "    out = RUNS / tag\n",
     "    L = [f\"# {tag} \u2014 generated by anima_closeout.ipynb\",\n",
     "         f\"output_dir = '{out}'\",\n",
-    "         f\"dataset = '{TOMLS}/dataset_train.toml'\",\n",
+    "         f\"dataset = '{TOMLS}/{dataset}'\",\n",
     "         f\"seed = {seed}\",\n",
     "         \"bf16_master_weights = true   # the exp004 gate-freeze fix\",\n",
     "         \"epochs = 1000\",\n",
@@ -427,12 +446,15 @@
     "         \"checkpoint_every_n_minutes = 30\",\n",
     "         \"activation_checkpointing = false   # 96GB rider\",\n",
     "         \"save_dtype = 'bfloat16'\",\n",
-    "         \"caching_batch_size = 8\",\n",
+    "         \"caching_batch_size = 8\"]\n",
+    "    if evals:\n",
+    "        L += [\n",
     "         f\"eval_datasets = [{{name = 'heldout', config = '{TOMLS}/dataset_eval.toml'}}]\",\n",
     "         \"eval_every_n_steps = 1000\",\n",
     "         \"eval_before_first_step = true\",\n",
     "         \"eval_micro_batch_size_per_gpu = 8\",\n",
-    "         \"eval_gradient_accumulation_steps = 1\",\n",
+    "         \"eval_gradient_accumulation_steps = 1\"]\n",
+    "    L += [\n",
     "         \"\", \"[model]\", \"type = 'anima'\",\n",
     "         f\"transformer_path = '{_AP}/diffusion_models/anima-base-v1.0.safetensors'\",\n",
     "         f\"llm_path = '{_AP}/text_encoders/qwen_3_06b_base.safetensors'\",\n",
@@ -472,6 +494,24 @@
     "    \"e017_mb3_smoke\": _arm_toml(\"e017_mb3_smoke\", 0, 200, mode=\"multiband3\"),\n",
     "    \"e017_mb3_s0\":    _arm_toml(\"e017_mb3_s0\", 0, 10000, mode=\"multiband3\"),\n",
     "    \"e017_mb3_s1\":    _arm_toml(\"e017_mb3_s1\", 1, 10000, mode=\"multiband3\"),\n",
+    "    # cache-only: trains 1 throwaway step on the HELD-OUT set purely to\n",
+    "    # materialize dp_cache/heldout (latents + text embeds) for the val bed\n",
+    "    # on a runtime where every arm was restored and nothing else trains.\n",
+    "    # Same dataset TOML -> same cache root/fingerprints as the real runs.\n",
+    "    \"cache_eval\":     _arm_toml(\"cache_eval\", 0, 1,\n",
+    "                                dataset=\"dataset_eval.toml\", evals=False),\n",
+    "}\n",
+    "\n",
+    "# THE registry: tag -> (hub package, arm folder). Drives BOTH shipping and\n",
+    "# restoring, so a trained arm and its hub copy can never disagree about\n",
+    "# where it lives. Arms absent here (the smokes) are local-only by design.\n",
+    "ARM_SHIP = {\n",
+    "    \"e004b_relay_s0\": (\"exp004b_anima_relay\", \"relay_s0\"),\n",
+    "    \"e004b_relay_s1\": (\"exp004b_anima_relay\", \"relay_s1\"),\n",
+    "    \"e004b_lora_s0\":  (\"exp004b_anima_relay\", \"lora_s0\"),\n",
+    "    \"e004b_lora_s1\":  (\"exp004b_anima_relay\", \"lora_s1\"),\n",
+    "    \"e017_mb3_s0\":    (\"exp017_anima_multiband\", \"mb3_s0\"),\n",
+    "    \"e017_mb3_s1\":    (\"exp017_anima_multiband\", \"mb3_s1\"),\n",
     "}\n",
     "print(\"TOMLs written:\", \", \".join(ARMS))\n"
    ]
@@ -487,7 +527,7 @@
     "# flight (gates MUST move \u2014 the exact exp004 failure), prove eval wiring\n",
     "# (eval_before_first_step fires at step 0), build the latent cache\n",
     "# (~0.5h, reused by every later arm).\n",
-    "import glob, io, json, os, subprocess, time\n",
+    "import glob, io, json, os, shutil, subprocess, time\n",
     "from pathlib import Path\n",
     "import torch\n",
     "\n",
@@ -516,18 +556,89 @@
     "        return proc.wait()\n",
     "\n",
     "def _arm_max_steps(tag):\n",
-    "    import tomllib\n",
+    "    try:\n",
+    "        import tomllib\n",
+    "    except ImportError:\n",
+    "        import tomli as tomllib\n",
     "    return tomllib.loads(ARMS[tag].read_text())[\"max_steps\"]\n",
     "\n",
-    "def run_arm(tag, resume=False):\n",
-    "    # IDEMPOTENT: an arm whose FINAL-step save already exists is never\n",
-    "    # retrained \u2014 after any crash the campaign resumes from local files,\n",
-    "    # skipping straight past every finished arm.\n",
+    "# \u2500\u2500 arm resolution: LOCAL -> HUB -> train \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\n",
+    "# A trained arm is NEVER retrained. Local checkpoints win; otherwise a\n",
+    "# previously shipped arm is pulled back from the hub (so a fresh runtime\n",
+    "# after a pod cull costs a download, not a training run); only a genuinely\n",
+    "# missing arm trains. ARM_SOURCE records which path each arm took.\n",
+    "WEIGHT_FILES = (\"aleph_relay.pt\", \"adapter_model.safetensors\")\n",
+    "ARM_SOURCE = globals().get(\"ARM_SOURCE\", {})\n",
+    "\n",
+    "def _curve_cache(tag):\n",
+    "    d = WORK / \"eval_curves\"\n",
+    "    d.mkdir(exist_ok=True)\n",
+    "    return d / f\"{tag}.json\"\n",
+    "\n",
+    "def _hub_files(refresh=False):\n",
+    "    if refresh or \"HUB_FILES\" not in ANIMA:\n",
+    "        from huggingface_hub import HfApi\n",
+    "        ANIMA[\"HUB_FILES\"] = sorted(HfApi().list_repo_files(HUB_REPO))\n",
+    "    return ANIMA[\"HUB_FILES\"]\n",
+    "\n",
+    "def local_save(tag):\n",
+    "    \"\"\"The arm's FINAL-step save on this disk, or None.\"\"\"\n",
     "    done = sorted(\n",
     "        glob.glob(str(RUNS / tag / \"*\" / f\"step{_arm_max_steps(tag)}\")))\n",
-    "    if done:\n",
-    "        print(f\"[{tag}] already trained \u2014 reusing {done[-1]}\")\n",
-    "        return Path(done[-1])\n",
+    "    return Path(done[-1]) if done else None\n",
+    "\n",
+    "def restore_arm(tag):\n",
+    "    \"\"\"Pull a shipped arm back from the hub into a local save dir, or None.\n",
+    "    Also restores its eval curve, which no longer exists as tfevents.\"\"\"\n",
+    "    if tag not in ARM_SHIP:\n",
+    "        return None\n",
+    "    from huggingface_hub import hf_hub_download\n",
+    "    pkg, arm = ARM_SHIP[tag]\n",
+    "    prefix = f\"{pkg}/{arm}/\"\n",
+    "    files = [f for f in _hub_files() if f.startswith(prefix)]\n",
+    "    if not any(f.rsplit(\"/\", 1)[-1] in WEIGHT_FILES for f in files):\n",
+    "        return None\n",
+    "    dest = RUNS / tag / \"from_hub\" / f\"step{_arm_max_steps(tag)}\"\n",
+    "    dest.mkdir(parents=True, exist_ok=True)\n",
+    "    for f in files:\n",
+    "        name = f.rsplit(\"/\", 1)[-1]\n",
+    "        # skip the log and the DERIVED duplicates of the .pt (canonical +\n",
+    "        # fork-format safetensors): nothing downstream reads them, and they\n",
+    "        # would triple the restore for every arm. They stay on the hub.\n",
+    "        if name == \"log_tail.txt\" or name.endswith(\".canonical.safetensors\") \\\n",
+    "           or name == \"aleph_relay.safetensors\":\n",
+    "            continue\n",
+    "        p = Path(hf_hub_download(HUB_REPO, f))\n",
+    "        if name == \"eval_curve.json\":\n",
+    "            _curve_cache(tag).write_text(p.read_text())\n",
+    "        else:\n",
+    "            shutil.copy(p, dest / name)\n",
+    "    print(f\"[{tag}] restored from hub: {prefix} -> {dest}\")\n",
+    "    return dest\n",
+    "\n",
+    "def arm_save(tag, allow_hub=True):\n",
+    "    \"\"\"Resolve a finished arm WITHOUT ever training. None if unavailable.\"\"\"\n",
+    "    p = local_save(tag)\n",
+    "    if p is not None:\n",
+    "        ARM_SOURCE.setdefault(tag, \"local\")\n",
+    "        return p\n",
+    "    if allow_hub:\n",
+    "        p = restore_arm(tag)\n",
+    "        if p is not None:\n",
+    "            ARM_SOURCE[tag] = \"hub\"\n",
+    "            return p\n",
+    "    return None\n",
+    "\n",
+    "def arm_available(tag):\n",
+    "    return arm_save(tag) is not None\n",
+    "\n",
+    "def run_arm(tag, resume=False, force=False):\n",
+    "    if not force:\n",
+    "        p = arm_save(tag)\n",
+    "        if p is not None:\n",
+    "            print(f\"[{tag}] NOT retraining \u2014 reusing \"\n",
+    "                  f\"{ARM_SOURCE[tag]} checkpoint {p}\")\n",
+    "            return p\n",
     "    log = RUNS / f\"{tag}.log\"\n",
     "    flags = \" --resume_from_checkpoint\" if resume else \"\"\n",
     "    t0 = time.time()\n",
@@ -537,18 +648,27 @@
     "    print(f\"[{tag}] rc={rc} wall={(time.time() - t0) / 60:.1f} min\")\n",
     "    if rc != 0:\n",
     "        _die(tag, log, f\"train exited rc={rc}\")\n",
+    "    ARM_SOURCE[tag] = \"trained\"\n",
     "    try:\n",
     "        return newest_save(tag)      # rc=0 but no run/save dir is ALSO death\n",
     "    except AssertionError as e:\n",
     "        _die(tag, log, str(e))\n",
     "\n",
-    "def latest_run_dir(tag):\n",
-    "    run_dirs = sorted(glob.glob(str(RUNS / tag / \"*\")))\n",
-    "    assert run_dirs, f\"no run dir for {tag}\"\n",
-    "    return Path(run_dirs[-1])\n",
+    "def latest_run_dir(tag, with_events=False, trained_only=False):\n",
+    "    dirs = sorted(glob.glob(str(RUNS / tag / \"*\")))\n",
+    "    if with_events or trained_only:\n",
+    "        # 'from_hub' is a restored checkpoint, not a run: no tfevents, and\n",
+    "        # it must never be mistaken for what a training launch just wrote\n",
+    "        dirs = [d for d in dirs if os.path.basename(d) != \"from_hub\"]\n",
+    "    if with_events:\n",
+    "        dirs = [d for d in dirs if glob.glob(\n",
+    "            os.path.join(d, \"**\", \"events.out.tfevents*\"), recursive=True)]\n",
+    "    assert dirs, (f\"no run dir for {tag}\"\n",
+    "                  + (\" containing tfevents\" if with_events else \"\"))\n",
+    "    return Path(dirs[-1])\n",
     "\n",
     "def newest_save(tag):\n",
-    "    rd = latest_run_dir(tag)\n",
+    "    rd = latest_run_dir(tag, trained_only=True)\n",
     "    saves = sorted(glob.glob(str(rd / \"epoch*\")) +\n",
     "                   glob.glob(str(rd / \"step*\")), key=os.path.getmtime)\n",
     "    assert saves, f\"{tag}: run wrote no save dir\"\n",
@@ -566,12 +686,23 @@
     "        out.extend(torch.as_tensor(g).float().flatten().tolist())\n",
     "    return out\n",
     "\n",
-    "def eval_curve(tag):\n",
-    "    \"\"\"The fork's deterministic eval scalars, parsed from tfevents.\"\"\"\n",
+    "def eval_curve(tag, required=True):\n",
+    "    \"\"\"The fork's deterministic eval scalars. Prefers the cached/restored\n",
+    "    JSON (a hub-restored arm has no tfevents at all), else parses this\n",
+    "    runtime's tfevents and caches the result. None when unavailable and\n",
+    "    not required.\"\"\"\n",
+    "    cached = _curve_cache(tag)\n",
+    "    if cached.exists():\n",
+    "        return json.loads(cached.read_text())\n",
+    "    try:\n",
+    "        rd = latest_run_dir(tag, with_events=True)\n",
+    "    except AssertionError:\n",
+    "        if required:\n",
+    "            raise\n",
+    "        return None\n",
     "    from tensorboard.backend.event_processing.event_accumulator import \\\n",
     "        EventAccumulator\n",
-    "    acc = EventAccumulator(str(latest_run_dir(tag)),\n",
-    "                           size_guidance={\"scalars\": 0})\n",
+    "    acc = EventAccumulator(str(rd), size_guidance={\"scalars\": 0})\n",
     "    acc.Reload()\n",
     "    out = {}\n",
     "    for t in acc.Tags()[\"scalars\"]:\n",
@@ -579,8 +710,26 @@
     "            out[t] = [(e.step, e.value) for e in acc.Scalars(t)]\n",
     "    assert \"heldout/loss\" in out, \\\n",
     "        f\"{tag}: no heldout/loss in tfevents ({acc.Tags()['scalars']})\"\n",
+    "    cached.write_text(json.dumps(out))\n",
     "    return out\n",
     "\n",
+    "def final_loss(tag):\n",
+    "    \"\"\"Last held-out loss of an arm's fork eval curve, or None.\"\"\"\n",
+    "    c = eval_curve(tag, required=False)\n",
+    "    return c[\"heldout/loss\"][-1][1] if c else None\n",
+    "\n",
+    "def ship_tag(tag, save_dir):\n",
+    "    \"\"\"Ship an arm under its registry identity \u2014 skipped when the arm CAME\n",
+    "    from the hub (re-uploading a restored copy is pure waste). The dir name\n",
+    "    is checked too, so the skip survives a kernel restart.\"\"\"\n",
+    "    if ARM_SOURCE.get(tag) == \"hub\" or Path(save_dir).parent.name == \"from_hub\":\n",
+    "        print(f\"[ship] {tag}: skipped (restored from hub, already there)\")\n",
+    "        return\n",
+    "    pkg, arm = ARM_SHIP[tag]\n",
+    "    curve = eval_curve(tag, required=False)\n",
+    "    ship_arm(pkg, arm, save_dir,\n",
+    "             extra=({f\"{pkg}/{arm}/eval_curve.json\": curve} if curve else None))\n",
+    "\n",
     "def ship_arm(package, arm, save_dir, extra=None):\n",
     "    \"\"\"Ship-on-completion: ckpt (+canonical safetensors) + TOML + log tail\n",
     "    into the hub package immediately.\"\"\"\n",
@@ -617,30 +766,49 @@
     "        operations=[CommitOperationAdd(path_in_repo=r, path_or_fileobj=s)\n",
     "                    for r, s in ops],\n",
     "        commit_message=f\"{package}/{arm}: ship-on-completion from Colab\")\n",
+    "    ANIMA.pop(\"HUB_FILES\", None)        # listing is now stale\n",
     "    print(f\"[ship] {package}/{arm}: {len(ops)} file(s) -> {HUB_REPO}\")\n",
     "\n",
-    "# -- SMOKE ----------------------------------------------------------------\n",
+    "# -- campaign inventory ---------------------------------------------------\n",
     "# refuse to launch on stale dataset TOMLs from an earlier notebook version:\n",
     "# without distinct dp_cache paths, train+eval share one cache root and\n",
     "# corrupt each other (the ArrowIndexError crash / silent eval-swap hazard)\n",
     "for _dt in (\"dataset_train.toml\", \"dataset_eval.toml\"):\n",
     "    assert \"dp_cache\" in (TOMLS / _dt).read_text(), (\n",
-    "        f\"{_dt} lacks the distinct cache path \u2014 RE-RUN CELL 1 (v8+) in \"\n",
-    "        \"THIS kernel before any arm; stale TOMLs corrupt the caches\")\n",
-    "\n",
-    "_save = run_arm(\"smoke\")\n",
-    "_g = gate_values(_save)\n",
-    "_moved = [x for x in _g if abs(x + 3.0) > 1e-3]\n",
-    "print(f\"G-GATE: {len(_moved)}/{len(_g)} gates moved off -3.0 \"\n",
-    "      f\"(range {min(_g):+.4f} .. {max(_g):+.4f})\")\n",
-    "assert _moved, (\n",
-    "    \"G-GATE FAILED: gates did not move \u2014 fp32 masters are not effective \"\n",
-    "    \"here. STOP. The designed fallback is the gate delta-reparam \"\n",
-    "    \"(gate = -3 + near-zero delta); do not burn full arms.\")\n",
-    "_c = eval_curve(\"smoke\")\n",
-    "print(f\"eval wiring OK: {len(_c)} heldout tags, \"\n",
-    "      f\"step-0 loss {_c['heldout/loss'][0][1]:.5f}\")\n",
-    "print(\"SMOKE: ALL GREEN \u2014 masters proven, eval proven, cache built\")\n"
+    "        f\"{_dt} lacks the distinct cache path \u2014 RE-RUN CELL 1 in THIS \"\n",
+    "        \"kernel before any arm; stale TOMLs corrupt the caches\")\n",
+    "\n",
+    "INVENTORY = {}\n",
+    "for _t in ARM_SHIP:                    # resolves + materializes each arm\n",
+    "    INVENTORY[_t] = ARM_SOURCE[_t] if arm_save(_t) else \"MISSING\"\n",
+    "print(\"campaign inventory:\")\n",
+    "for _t, _w in INVENTORY.items():\n",
+    "    print(f\"  {_t:<18} {_w}\")\n",
+    "_todo = [t for t, w in INVENTORY.items() if w == \"MISSING\"]\n",
+    "print(f\"arms needing training: {_todo or 'NONE \u2014 everything is available'}\")\n",
+    "\n",
+    "# -- SMOKE ----------------------------------------------------------------\n",
+    "# Its jobs: prove gates move under fp32 masters (exp004's exact failure),\n",
+    "# prove eval wiring, build the latent cache. All three are SUPERSEDED once\n",
+    "# a full aleph arm exists \u2014 so it only runs when one is actually needed.\n",
+    "_ALEPH = (\"e004b_relay_s0\", \"e004b_relay_s1\", \"e017_mb3_s0\", \"e017_mb3_s1\")\n",
+    "if any(INVENTORY[t] != \"MISSING\" for t in _ALEPH):\n",
+    "    print(\"SMOKE SKIPPED \u2014 a full aleph arm is already available, which \"\n",
+    "          \"proves the gate fix at 10k steps (a stronger result than 200).\")\n",
+    "else:\n",
+    "    _save = run_arm(\"smoke\")\n",
+    "    _g = gate_values(_save)\n",
+    "    _moved = [x for x in _g if abs(x + 3.0) > 1e-3]\n",
+    "    print(f\"G-GATE: {len(_moved)}/{len(_g)} gates moved off -3.0 \"\n",
+    "          f\"(range {min(_g):+.4f} .. {max(_g):+.4f})\")\n",
+    "    assert _moved, (\n",
+    "        \"G-GATE FAILED: gates did not move \u2014 fp32 masters are not effective \"\n",
+    "        \"here. STOP. The designed fallback is the gate delta-reparam \"\n",
+    "        \"(gate = -3 + near-zero delta); do not burn full arms.\")\n",
+    "    _c = eval_curve(\"smoke\")\n",
+    "    print(f\"eval wiring OK: {len(_c)} heldout tags, \"\n",
+    "          f\"step-0 loss {_c['heldout/loss'][0][1]:.5f}\")\n",
+    "    print(\"SMOKE: ALL GREEN \u2014 masters proven, eval proven, cache built\")\n"
    ]
   },
   {
@@ -667,44 +835,39 @@
    "outputs": [],
    "source": [
     "# \u2550\u2550\u2550 CELL 3 \u2014 exp004b: relay s0, relay s1, LoRA control \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n",
+    "# Every arm resolves LOCAL -> HUB -> train, so re-running this cell after a\n",
+    "# crash (or on a fresh runtime) costs downloads, not GPU-hours.\n",
     "relay_saves = {}\n",
     "for _tag in (\"e004b_relay_s0\", \"e004b_relay_s1\"):\n",
-    "    _save = run_arm(_tag)                  # run_arm(_tag, resume=True) after\n",
-    "    _g = gate_values(_save)                # a disconnect\n",
+    "    _save = run_arm(_tag)                  # run_arm(_tag, resume=True) to\n",
+    "    _g = gate_values(_save)                # continue a half-done run\n",
     "    _mv = [x for x in _g if abs(x + 3.0) > 1e-3]\n",
     "    print(f\"[{_tag}] P3 gates moved: {len(_mv)}/{len(_g)} \"\n",
-    "          f\"(range {min(_g):+.4f}..{max(_g):+.4f})\")\n",
+    "          f\"(range {min(_g):+.4f}..{max(_g):+.4f})  [{ARM_SOURCE[_tag]}]\")\n",
     "    assert _mv, f\"{_tag}: P3 FAILED \u2014 gates frozen at full scale\"\n",
-    "    _arm = _tag.replace(\"e004b_\", \"\")\n",
-    "    ship_arm(\"exp004b_anima_relay\", _arm, _save,\n",
-    "             extra={f\"exp004b_anima_relay/{_arm}/eval_curve.json\":\n",
-    "                    eval_curve(_tag)})\n",
+    "    ship_tag(_tag, _save)\n",
     "    relay_saves[_tag] = str(_save)\n",
     "\n",
     "lora_saves = {\"e004b_lora_s0\": str(run_arm(\"e004b_lora_s0\"))}\n",
-    "ship_arm(\"exp004b_anima_relay\", \"lora_s0\",\n",
-    "         Path(lora_saves[\"e004b_lora_s0\"]),\n",
-    "         extra={\"exp004b_anima_relay/lora_s0/eval_curve.json\":\n",
-    "                eval_curve(\"e004b_lora_s0\")})\n",
-    "\n",
-    "def _final(tag):\n",
-    "    return eval_curve(tag)[\"heldout/loss\"][-1][1]\n",
-    "\n",
-    "_r0, _r1 = _final(\"e004b_relay_s0\"), _final(\"e004b_relay_s1\")\n",
-    "_l0 = _final(\"e004b_lora_s0\")\n",
-    "_spread = abs(_r0 - _r1)\n",
-    "_gap = abs(min(_r0, _r1) - _l0)\n",
-    "print(f\"relay s0 {_r0:.5f} | s1 {_r1:.5f} (spread {_spread:.5f}) | \"\n",
-    "      f\"lora s0 {_l0:.5f} (gap {_gap:.5f})\")\n",
-    "if _gap <= 2 * max(_spread, 1e-6):\n",
-    "    print(\"GATE: gap within 2x seed spread -> LoRA s1 REQUIRED\")\n",
-    "    lora_saves[\"e004b_lora_s1\"] = str(run_arm(\"e004b_lora_s1\"))\n",
-    "    ship_arm(\"exp004b_anima_relay\", \"lora_s1\",\n",
-    "             Path(lora_saves[\"e004b_lora_s1\"]),\n",
-    "             extra={\"exp004b_anima_relay/lora_s1/eval_curve.json\":\n",
-    "                    eval_curve(\"e004b_lora_s1\")})\n",
+    "ship_tag(\"e004b_lora_s0\", Path(lora_saves[\"e004b_lora_s0\"]))\n",
+    "\n",
+    "_r0, _r1 = final_loss(\"e004b_relay_s0\"), final_loss(\"e004b_relay_s1\")\n",
+    "_l0 = final_loss(\"e004b_lora_s0\")\n",
+    "if None in (_r0, _r1, _l0):\n",
+    "    print(\"results gate SKIPPED \u2014 missing eval curve(s) for \"\n",
+    "          f\"{[t for t, v in zip(('relay_s0','relay_s1','lora_s0'), (_r0,_r1,_l0)) if v is None]}\"\n",
+    "          \" (arms restored without one); LoRA s1 decision deferred to CELL 6\")\n",
     "else:\n",
-    "    print(\"GATE: ordering decisive at s0 -> LoRA s1 skipped (disclosed)\")\n"
+    "    _spread = abs(_r0 - _r1)\n",
+    "    _gap = abs(min(_r0, _r1) - _l0)\n",
+    "    print(f\"relay s0 {_r0:.5f} | s1 {_r1:.5f} (spread {_spread:.5f}) | \"\n",
+    "          f\"lora s0 {_l0:.5f} (gap {_gap:.5f})\")\n",
+    "    if _gap <= 2 * max(_spread, 1e-6):\n",
+    "        print(\"GATE: gap within 2x seed spread -> LoRA s1 REQUIRED\")\n",
+    "        lora_saves[\"e004b_lora_s1\"] = str(run_arm(\"e004b_lora_s1\"))\n",
+    "        ship_tag(\"e004b_lora_s1\", Path(lora_saves[\"e004b_lora_s1\"]))\n",
+    "    else:\n",
+    "        print(\"GATE: ordering decisive at s0 -> LoRA s1 skipped (disclosed)\")\n"
    ]
   },
   {
@@ -731,24 +894,30 @@
     "# \u2550\u2550\u2550 CELL 4 \u2014 exp017: mb3 smoke, then both seeds \u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\u2550\n",
     "from amoe.io.checkpoint import load_diffusion_anchor\n",
     "\n",
-    "_save = run_arm(\"e017_mb3_smoke\")\n",
-    "_ck = load_diffusion_anchor(str(Path(_save) / \"aleph_relay.pt\"))\n",
-    "assert _ck.kind == \"multiband3\", \\\n",
-    "    f\"mb3 smoke saved kind={_ck.kind} \u2014 the mode did not engage\"\n",
-    "_g = gate_values(_save)\n",
-    "_mv = [x for x in _g if abs(x + 3.0) > 1e-3]\n",
-    "print(f\"mb3 smoke: kind OK, {_ck.n_sites} sites, gates moved \"\n",
-    "      f\"{len(_mv)}/{len(_g)}\")\n",
-    "assert _mv, \"mb3 smoke: gates frozen \u2014 same STOP rule as the G-gate\"\n",
+    "# the smoke exists to prove the mode ENGAGES before spending two seeds on\n",
+    "# it; a real mb3 arm already in hand proves the same thing more strongly\n",
+    "if all(arm_available(t) for t in (\"e017_mb3_s0\", \"e017_mb3_s1\")):\n",
+    "    print(\"mb3 smoke SKIPPED \u2014 both mb3 seed arms are already available\")\n",
+    "else:\n",
+    "    _save = run_arm(\"e017_mb3_smoke\")\n",
+    "    _ck = load_diffusion_anchor(str(Path(_save) / \"aleph_relay.pt\"))\n",
+    "    assert _ck.kind == \"multiband3\", \\\n",
+    "        f\"mb3 smoke saved kind={_ck.kind} \u2014 the mode did not engage\"\n",
+    "    _g = gate_values(_save)\n",
+    "    _mv = [x for x in _g if abs(x + 3.0) > 1e-3]\n",
+    "    print(f\"mb3 smoke: kind OK, {_ck.n_sites} sites, gates moved \"\n",
+    "          f\"{len(_mv)}/{len(_g)}\")\n",
+    "    assert _mv, \"mb3 smoke: gates frozen \u2014 same STOP rule as the G-gate\"\n",
     "\n",
     "mb3_saves = {}\n",
     "for _tag in (\"e017_mb3_s0\", \"e017_mb3_s1\"):\n",
     "    _save = run_arm(_tag)\n",
-    "    _arm = _tag.replace(\"e017_\", \"\")\n",
-    "    ship_arm(\"exp017_anima_multiband\", _arm, _save,\n",
-    "             extra={f\"exp017_anima_multiband/{_arm}/eval_curve.json\":\n",
-    "                    eval_curve(_tag)})\n",
+    "    _ck = load_diffusion_anchor(str(Path(_save) / \"aleph_relay.pt\"))\n",
+    "    assert _ck.kind == \"multiband3\", \\\n",
+    "        f\"{_tag}: kind={_ck.kind} \u2014 not a multiband3 stack\"\n",
+    "    ship_tag(_tag, _save)\n",
     "    mb3_saves[_tag] = str(_save)\n",
+    "    print(f\"[{_tag}] OK ({ARM_SOURCE[_tag]}), {_ck.n_sites} sites\")\n",
     "print(\"exp017 arms complete:\", list(mb3_saves))\n"
    ]
   },
@@ -979,12 +1148,35 @@
     "# the fork caches under {path}/cache/{model_name} with model_name 'anima'\n",
     "# (cosmos_predict2.py:254 renames the pipeline when llm_path is a file)\n",
     "EVAL_CACHE = str(WORK / \"dp_cache\" / \"heldout\" / \"cache\" / \"anima\")\n",
+    "\n",
+    "def ensure_eval_cache():\n",
+    "    \"\"\"The bed reads the fork's OWN eval cache. On a runtime where every\n",
+    "    arm was restored from the hub, nothing has trained, so that cache does\n",
+    "    not exist \u2014 rebuild it with a 1-step throwaway run over the held-out\n",
+    "    TOML (same TOML -> same root, same fingerprints; ~1 min, 512 rows).\n",
+    "    The throwaway checkpoint is never read by anything.\"\"\"\n",
+    "    if sorted(glob.glob(str(Path(EVAL_CACHE) / \"cache_*\"))):\n",
+    "        return\n",
+    "    print(\"eval cache absent (all arms restored) \u2014 building it now\")\n",
+    "    run_arm(\"cache_eval\", force=True)\n",
+    "    assert sorted(glob.glob(str(Path(EVAL_CACHE) / \"cache_*\"))), \\\n",
+    "        f\"cache_eval ran but produced nothing under {EVAL_CACHE}\"\n",
+    "\n",
+    "ensure_eval_cache()\n",
     "_buckets = sorted(glob.glob(str(Path(EVAL_CACHE) / \"cache_*\")))\n",
-    "assert _buckets, (f\"no cache_* bucket dirs under {EVAL_CACHE} \u2014 the arms \"\n",
-    "                  \"must have run (they build the eval cache) before the bed\")\n",
+    "assert _buckets, f\"no cache_* bucket dirs under {EVAL_CACHE}\"\n",
     "print(\"eval cache:\", EVAL_CACHE,\n",
     "      \"| buckets:\", [Path(b).name for b in _buckets])\n",
     "\n",
+    "# resolve every arm independently of CELLs 3/4 having run in THIS kernel\n",
+    "relay_saves = {t: str(arm_save(t)) for t in\n",
+    "               (\"e004b_relay_s0\", \"e004b_relay_s1\")}\n",
+    "mb3_saves = {t: str(arm_save(t)) for t in (\"e017_mb3_s0\", \"e017_mb3_s1\")}\n",
+    "assert all(relay_saves.values()) and all(mb3_saves.values()), \\\n",
+    "    \"run CELLs 3 and 4 first \u2014 some arms are neither local nor on the hub\"\n",
+    "print(\"arms resolved:\", {t: ARM_SOURCE[t] for t in\n",
+    "                         list(relay_saves) + list(mb3_saves)})\n",
+    "\n",
     "_full = tomllib.loads(ARMS[\"e004b_relay_s0\"].read_text())\n",
     "_model = {k: v for k, v in _full[\"model\"].items()\n",
     "          if not k.startswith(\"aleph_\") and not k.endswith(\"_lr\")}\n",
@@ -1024,30 +1216,36 @@
     "    \"mb3_s1\": {\"kind\": \"multiband3\", \"ckpt\": _pt(mb3_saves, \"e017_mb3_s1\")},\n",
     "}}, \"results_exp017\")\n",
     "\n",
-    "def _final(tag):\n",
-    "    return eval_curve(tag)[\"heldout/loss\"][-1][1]\n",
-    "\n",
+    "_curves = {t: eval_curve(t, required=False) for t in\n",
+    "           (\"e004b_relay_s0\", \"e004b_relay_s1\",\n",
+    "            \"e004b_lora_s0\", \"e004b_lora_s1\")}\n",
     "RES4 = {\"bed\": r4,\n",
-    "        \"fork_eval\": {t: eval_curve(t) for t in\n",
-    "                      (\"e004b_relay_s0\", \"e004b_relay_s1\", \"e004b_lora_s0\")},\n",
+    "        \"fork_eval\": {t: c for t, c in _curves.items() if c},\n",
+    "        \"arm_source\": {t: ARM_SOURCE.get(t) for t in ARM_SHIP},\n",
     "        \"split_manifest\": json.loads((DATA / \"split_manifest.json\")\n",
     "                                     .read_text())}\n",
-    "_rl = min(_final(\"e004b_relay_s0\"), _final(\"e004b_relay_s1\"))\n",
+    "_rl = min(final_loss(\"e004b_relay_s0\"), final_loss(\"e004b_relay_s1\"))\n",
+    "_l0 = final_loss(\"e004b_lora_s0\")\n",
     "_curve0 = RES4[\"fork_eval\"][\"e004b_relay_s0\"][\"heldout/loss\"]\n",
     "_at3k = min(v for s, v in _curve0 if s <= 3000)\n",
     "RES4[\"verdict\"] = {\n",
-    "    \"P1_relay_vs_lora\": \"relay\" if _rl <= _final(\"e004b_lora_s0\") else \"lora\",\n",
+    "    # bed loss is the paired instrument; the fork curve decides P1/P4\n",
+    "    \"P1_relay_vs_lora\": (\"relay\" if _rl <= _l0 else \"lora\")\n",
+    "                        if _l0 is not None else \"UNDECIDED (no LoRA control)\",\n",
+    "    \"P1_numbers\": {\"relay_best\": _rl, \"lora_s0\": _l0},\n",
     "    \"P2_toggle_bit_exact\": True,        # the bed asserted, else it raised\n",
     "    \"P3_gates_moved\": True,             # per-arm asserts in CELL 3\n",
     "    \"P4_3k_was_a_floor\": _curve0[-1][1] < _at3k,\n",
     "}\n",
     "(WORK / \"results_exp004b_full.json\").write_text(json.dumps(RES4, indent=1))\n",
-    "ship_arm(\"exp004b_anima_relay\", \"results\", newest_save(\"e004b_relay_s0\"),\n",
+    "ship_arm(\"exp004b_anima_relay\", \"results\",\n",
+    "         Path(relay_saves[\"e004b_relay_s0\"]),\n",
     "         extra={\"exp004b_anima_relay/results.json\": RES4})\n",
     "print(\"exp004b verdict:\", RES4[\"verdict\"])\n",
     "\n",
     "_q = [str(x) for x in r17[\"quantiles\"]]\n",
     "_allon = r17[\"arms\"][\"mb3_s0\"][\"loss_by_quantile\"]\n",
+    "_off = r17[\"arms\"][\"mb3_s0_toggled_off\"][\"loss_by_quantile\"]\n",
     "_bandq = {0: [\"0.1\", \"0.3\"], 1: [\"0.5\", \"0.7\"], 2: [\"0.9\"]}\n",
     "_surgical = {}\n",
     "for _b, _own_q in _bandq.items():\n",
@@ -1057,16 +1255,34 @@
     "    _cross = sum(_les[x] - _allon[x] for x in _cq) / len(_cq)\n",
     "    _surgical[_b] = {\"own_damage\": _own, \"cross_damage\": _cross,\n",
     "                     \"ratio\": (_own / _cross) if _cross > 0 else None}\n",
-    "RES17 = {\"bed\": r17, \"surgical\": _surgical,\n",
-    "         \"P1_surgical_10x\": all(\n",
-    "             (s[\"ratio\"] is None and s[\"own_damage\"] > 0) or\n",
+    "# EFFECT SIZE GUARD. The bed is bit-deterministic, so any lesion delta is\n",
+    "# \"real\" \u2014 but a stack whose gates trained shut barely moves the loss at\n",
+    "# all, and then a beautiful ratio is measuring nothing. The shipped mb3\n",
+    "# gates sit at sigmoid(-3.6..-7.8) = 0.027..0.0004, so this is the live\n",
+    "# risk, not a hypothetical: report the adapter's own effect and refuse to\n",
+    "# call a lesion surgical when the stack it lesions is inert.\n",
+    "_effect = sum(_off[x] - _allon[x] for x in _q) / len(_q)\n",
+    "_own_mean = sum(s[\"own_damage\"] for s in _surgical.values()) / len(_surgical)\n",
+    "_vacuous = abs(_effect) < 1e-6 or abs(_own_mean) < 0.01 * abs(_effect)\n",
+    "_clean = all((s[\"ratio\"] is None and s[\"own_damage\"] > 0) or\n",
     "             (s[\"ratio\"] is not None and s[\"ratio\"] >= 10)\n",
-    "             for s in _surgical.values()),\n",
+    "             for s in _surgical.values())\n",
+    "RES17 = {\"bed\": r17, \"surgical\": _surgical,\n",
+    "         \"adapter_effect_mean\": _effect,      # toggled_off - all_on\n",
+    "         \"own_damage_mean\": _own_mean,\n",
+    "         \"P1_surgical_10x\": bool(_clean and not _vacuous),\n",
+    "         \"P1_verdict\": (\"VACUOUS \u2014 the stack is near-inert (gates trained \"\n",
+    "                        \"shut); lesion ratios measure nothing\"\n",
+    "                        if _vacuous else\n",
+    "                        (\"SURGICAL\" if _clean else \"NOT SURGICAL\")),\n",
     "         \"P2_toggle_bit_exact\": True}\n",
     "(WORK / \"results_exp017_full.json\").write_text(json.dumps(RES17, indent=1))\n",
-    "ship_arm(\"exp017_anima_multiband\", \"results\", newest_save(\"e017_mb3_s0\"),\n",
+    "ship_arm(\"exp017_anima_multiband\", \"results\",\n",
+    "         Path(mb3_saves[\"e017_mb3_s0\"]),\n",
     "         extra={\"exp017_anima_multiband/results.json\": RES17})\n",
-    "print(\"exp017 surgical:\", json.dumps(_surgical, indent=1))\n"
+    "print(\"exp017 surgical:\", json.dumps(_surgical, indent=1))\n",
+    "print(f\"exp017 P1: {RES17['P1_verdict']} | adapter effect \"\n",
+    "      f\"{_effect:+.3e} | mean own-band damage {_own_mean:+.3e}\")\n"
    ]
   },
   {
@@ -1102,6 +1318,15 @@
     "\n",
     "_BANDS = [\"fidelity/detail (LOW noise)\", \"continuity/semantics (MID)\",\n",
     "          \"diversity/structure (HIGH noise)\"]\n",
+    "# survive a kernel that ran CELL 6 in an earlier session\n",
+    "RES4 = globals().get(\"RES4\") or json.loads(\n",
+    "    (WORK / \"results_exp004b_full.json\").read_text())\n",
+    "RES17 = globals().get(\"RES17\") or json.loads(\n",
+    "    (WORK / \"results_exp017_full.json\").read_text())\n",
+    "relay_saves = globals().get(\"relay_saves\") or {\n",
+    "    t: str(arm_save(t)) for t in (\"e004b_relay_s0\", \"e004b_relay_s1\")}\n",
+    "mb3_saves = globals().get(\"mb3_saves\") or {\n",
+    "    t: str(arm_save(t)) for t in (\"e017_mb3_s0\", \"e017_mb3_s1\")}\n",
     "_v = RES4[\"verdict\"]\n",
     "_ops = []\n",
     "for _seed, _tag in ((0, \"e004b_relay_s0\"), (1, \"e004b_relay_s1\")):\n",
@@ -1122,8 +1347,10 @@
     "        f\"Anima 2B DiT Multiband v1 (seed {_seed}) \u2014 NON-COMMERCIAL\",\n",
     "        \"Three sigma-band experts on the Anima DiT, step-gated \u2014 the \"\n",
     "        \"first multiband stack on a DiT-class trunk.\",\n",
-    "        f\"exp017: P1 surgical(>=10x)={RES17['P1_surgical_10x']}; lesion \"\n",
-    "        \"numbers in exp017_anima_multiband/results.json.\",\n",
+    "        f\"exp017: P1 {RES17['P1_verdict']} (surgical>=10x=\"\n",
+    "        f\"{RES17['P1_surgical_10x']}, adapter effect \"\n",
+    "        f\"{RES17['adapter_effect_mean']:+.2e}); lesion numbers in \"\n",
+    "        \"exp017_anima_multiband/results.json.\",\n",
     "        _seed,\n",
     "        [\"Non-commercial (derived from NC weights).\",\n",
     "         \"Needs an Anima checkpoint; band gating needs a step-gated \"\n",