anima_closeout v14: full resume rebuild — every arm now resolves LOCAL -> HUB -> train off one ARM_SHIP registry that drives shipping AND restoring, so a trained arm is never retrained and a fresh runtime costs a ~29MB download per arm. Restored arms carry their eval_curve.json (no tfevents exist), smokes skip when a full arm already proves the same thing, CELL 6 rebuilds the held-out cache itself (1-step cache_eval over the eval TOML) and resolves its own checkpoints, CELL 7 reads results from disk. exp017's surgical verdict gained an effect-size guard: the shipped mb3 gates sit at sigmoid(-3.6..-7.8), so a lesion ratio on a near-inert stack is reported VACUOUS, not PASS. Verified against the live hub: real restore of relay_s0/s1 + mb3, no training launched, newest_save refuses restored dirs, ship-skip stateless; builder now enforces cross-cell name resolution over 3 run orders.
anima_closeout v13: stale-CELL-6 report (v6-era DATA glob found no eval cache — replaced in v7 by the deterministic dp_cache/heldout/cache/anima root) + a real catch from the live logs: anima's TE cache dir is text_embeddings_1, not the hardcoded _0 — the val bed now DISCOVERS the text_embeddings_* dir per bucket (uncond excluded, exactly one asserted).
anima_closeout v12: smoke + both relay arms TRAINED; the LoRA control died in peft's torchao dispatcher (Colab preinstalls torchao 0.10.0, peft raises wanting >=0.16; nothing here uses torchao -> CELL 0 uninstalls it, probe then returns False and dispatch falls to standard Linear). run_arm is now IDEMPOTENT: an arm with its final-step save present is reused, never retrained — the campaign resumes from local files after any crash.
anima_closeout v11: first live run of bf16_master_weights crashed at the scheduler — ds_config['bf16'] made DeepSpeed wrap the client Adam (FP16_UnfusedOptimizer), which torch ConstantLR rejects. Fork 812449a: client-side MasterWeightsAdam (fp32 masters, still a torch Adam; engine stays on the exp004-proven no-wrapping path). Proven in pure torch incl. the exact exp004 freeze repro: -3.0 bf16 gate frozen under plain Adam, moves under MasterWeightsAdam. CELL 0 asserts the new mechanism. Caches from the 28-min build are all banked — next launch goes straight to training.
anima_closeout v10: latent caching died on its FIRST image read — the fork's ParquetImageReader whole-file column read cannot deliver >2GiB nested (struct) columns (ArrowNotImplementedError), and the 8192-row split monolith crosses that line. Fixed in the fork (8ed4b6b: per-row-group reads assembled into a ChunkedArray; proven against a 1200x2MB struct parquet reproducing the exact error). CELL 0 asserts the fixed reader pre-spend. No CELL 1 re-run needed: the monolith is readable as-is.
anima_closeout v9: the ArrowIndexError smoke crash was the shared-cache collision detonating — CELL 1 had last run as v6 (same ipykernel_5091 across all three crashes) so the dataset TOMLs on disk lacked path=, and eval's grouped-metadata save clobbered train's mmapped files (same keys, same filenames, shared local_parquet root) right before train's latents map read through them (writer died at the first 1000-row flush). v9: CELL 1 also rmtree's the poisoned default root; CELL 2 refuses to launch any arm if the dataset TOMLs lack dp_cache. RUNBOOK: re-run CELL 1 in the live kernel, then CELL 2.
anima_closeout v8: the un-masked smoke crash was ModuleNotFoundError: hyvideo — utils/patches.py imported hyvideo.text_encoder unconditionally at module level while CELL 0 (correctly) initializes only the ComfyUI submodule. Fixed in the fork (5f37138: guarded import, patch skipped when absent; verified by execution both ways). CELL 0 now asserts the guard is present before any spend.
anima_closeout v7: (1) rc no longer masked — the deepspeed|tee pipeline returned TEE's exit code, so the smoke's ~24s death printed rc=0; Python is now the tee and any failure (rc!=0 OR no run dir) dumps the log tail into the cell. (2) CRITICAL: train+eval dataset TOMLs shared the fork's default parquet cache root ('local_parquet' constant) — eval would clobber train's metadata/latents and the loop would silently train on the 512 eval rows; now explicit distinct path= roots under dp_cache/. (3) width_column/height_column (the real fork keys; image_*_column variants were silently ignored). (4) val bed reads the fork's REAL cache layout: per-bucket paired latents/text_embeddings_0 caches. Source-verified against the fork + exp004's as-run TOMLs by 4-agent audit.
anima_closeout v6: CELL 1's shard loop variable was `sh`, clobbering CELL 0's sh() shell helper in the shared notebook namespace -> CELL 2 smoke crashed with 'str' object is not callable. Renamed + builder now AST-checks every cell for cross-cell function-name clobbering (proven to flag the exact bug).
CELL 1 v5: properly columnar two-phase split. Phase A thin scan (id/captions/audit) with pyarrow.compute masks — exp011 gate, dexp011:154 caption coalesce using the forks own CAPTION_SENTINELS — stop at target; phase B heavy write filtered by is_in(id, split sets), image bytes never leave Arrow. Row-by-row loop removed. Both phases executed green against a real shard in 0.5s.
CELL 1 rewritten as an INCREMENTAL materializer: the fused set is ~160GB and the campaign needs ~9k rows — shards visited in sha256(name) order one at a time, gated+deduped+slimmed into the pool, source deleted per shard, hard stop at TRAIN_TARGET+EVAL_N (8192+512). Download bounded to ~10GB instead of the corpus. Split logic executed green against a real shard.
Caption key corrected per the campaign record: qwen-deepfashion-fused stays THE dataset (r2 exp011-013 design of record); caption = caption_joycaption with prompt_fused fallback (dexp011:154), materialized as a derived column; exp011 audit gate + id dedup in the split. The earlier ft1 retarget was wrong — the fix was the key, not the dataset.
Fix dataset target: exp004 trained on diffusion-pretrain-set-ft1 data/deepfashion (has caption_vlm_json), not qwen-deepfashion-fused; shard listing filtered to the config (the repo holds 582 parquet across configs). Caught by CELL 0 refusing before spend — as designed.
