anima_closeout v11: first live run of bf16_master_weights crashed at the scheduler — ds_config['bf16'] made DeepSpeed wrap the client Adam (FP16_UnfusedOptimizer), which torch ConstantLR rejects. Fork 812449a: client-side MasterWeightsAdam (fp32 masters, still a torch Adam; engine stays on the exp004-proven no-wrapping path). Proven in pure torch incl. the exact exp004 freeze repro: -3.0 bf16 gate frozen under plain Adam, moves under MasterWeightsAdam. CELL 0 asserts the new mechanism. Caches from the 28-min build are all banked — next launch goes straight to training.
anima_closeout v10: latent caching died on its FIRST image read — the fork's ParquetImageReader whole-file column read cannot deliver >2GiB nested (struct) columns (ArrowNotImplementedError), and the 8192-row split monolith crosses that line. Fixed in the fork (8ed4b6b: per-row-group reads assembled into a ChunkedArray; proven against a 1200x2MB struct parquet reproducing the exact error). CELL 0 asserts the fixed reader pre-spend. No CELL 1 re-run needed: the monolith is readable as-is.
anima_closeout v9: the ArrowIndexError smoke crash was the shared-cache collision detonating — CELL 1 had last run as v6 (same ipykernel_5091 across all three crashes) so the dataset TOMLs on disk lacked path=, and eval's grouped-metadata save clobbered train's mmapped files (same keys, same filenames, shared local_parquet root) right before train's latents map read through them (writer died at the first 1000-row flush). v9: CELL 1 also rmtree's the poisoned default root; CELL 2 refuses to launch any arm if the dataset TOMLs lack dp_cache. RUNBOOK: re-run CELL 1 in the live kernel, then CELL 2.
anima_closeout v8: the un-masked smoke crash was ModuleNotFoundError: hyvideo — utils/patches.py imported hyvideo.text_encoder unconditionally at module level while CELL 0 (correctly) initializes only the ComfyUI submodule. Fixed in the fork (5f37138: guarded import, patch skipped when absent; verified by execution both ways). CELL 0 now asserts the guard is present before any spend.
anima_closeout v7: (1) rc no longer masked — the deepspeed|tee pipeline returned TEE's exit code, so the smoke's ~24s death printed rc=0; Python is now the tee and any failure (rc!=0 OR no run dir) dumps the log tail into the cell. (2) CRITICAL: train+eval dataset TOMLs shared the fork's default parquet cache root ('local_parquet' constant) — eval would clobber train's metadata/latents and the loop would silently train on the 512 eval rows; now explicit distinct path= roots under dp_cache/. (3) width_column/height_column (the real fork keys; image_*_column variants were silently ignored). (4) val bed reads the fork's REAL cache layout: per-bucket paired latents/text_embeddings_0 caches. Source-verified against the fork + exp004's as-run TOMLs by 4-agent audit.
anima_closeout v6: CELL 1's shard loop variable was `sh`, clobbering CELL 0's sh() shell helper in the shared notebook namespace -> CELL 2 smoke crashed with 'str' object is not callable. Renamed + builder now AST-checks every cell for cross-cell function-name clobbering (proven to flag the exact bug).
CELL 1 v5: properly columnar two-phase split. Phase A thin scan (id/captions/audit) with pyarrow.compute masks — exp011 gate, dexp011:154 caption coalesce using the forks own CAPTION_SENTINELS — stop at target; phase B heavy write filtered by is_in(id, split sets), image bytes never leave Arrow. Row-by-row loop removed. Both phases executed green against a real shard in 0.5s.
CELL 1 rewritten as an INCREMENTAL materializer: the fused set is ~160GB and the campaign needs ~9k rows — shards visited in sha256(name) order one at a time, gated+deduped+slimmed into the pool, source deleted per shard, hard stop at TRAIN_TARGET+EVAL_N (8192+512). Download bounded to ~10GB instead of the corpus. Split logic executed green against a real shard.
Caption key corrected per the campaign record: qwen-deepfashion-fused stays THE dataset (r2 exp011-013 design of record); caption = caption_joycaption with prompt_fused fallback (dexp011:154), materialized as a derived column; exp011 audit gate + id dedup in the split. The earlier ft1 retarget was wrong — the fix was the key, not the dataset.
Fix dataset target: exp004 trained on diffusion-pretrain-set-ft1 data/deepfashion (has caption_vlm_json), not qwen-deepfashion-fused; shard listing filtered to the config (the repo holds 582 parquet across configs). Caught by CELL 0 refusing before spend — as designed.
Anima adapter is ComfyUI-loadable: the node pack now supports the MiniTrainDIT family (verified live on anima_baseV10, 28 sites, toggle-off bit-exact) — caveat corrected in the catalog and in the file metadata
Backfill the missing safetensors companions (dispatch banks, monoliths, cond adapter) into their original experiment packages — every .pt stack now has a verified companion
amoe_loras/: every stack converted to safetensors (38 raw, bitwise-verified) + 36 function-first production artifacts carrying their own usage cards; catalog + SCHEMA. Closes the 15 missing companions (banks/monos/cond) the burndown claim overstated.
Burndown record completion: every s1 replication ckpt, every monolith/control falsifier, exp011 fused stacks, exp012/013 blob+uniform stacks, exp014/015 key banks, exp004 fork run saves + LoRA control (salvaged from the pod before window close)
exp015 shipped: address-as-key falsified with corrected instruments (zero routing excess, zero match advantage) - the dispatch-key form closes; TE+AMOE joint training is the open form
exp013 shipped: CONDITIONING HYPOTHESIS CONFIRMED - blob coupling ~200x more effective on flow than eps (prereg 5x bar); blob work migrates to flow/v-pred substrates
exp003 + exp007 shipped: sigma is the only live register axis; AMoE v1 monolith falsifier fires under uniform pressure (machinery green) — next: structural band assignment
