anima_closeout v20: the bed now RUNS — frozen 0.110923, frozen_repeat identical (determinism holds), relay_s0_toggled_off bit-identical to frozen (TOGGLE LAW passes). Only the ENABLED relay crashed: attach_aleph_relays casts dtype but does not place on device (the trainer attaches on CPU then DeepSpeed moves the engine), and the bed had already moved the trunk to cuda, so fresh relay params sat on CPU -> 'mat2 is on cpu' at the first relay matmul. Fix: tf.to(DEV) after each attach (idempotent for placed params); verified a fresh attribute-assigned submodule auto-registers and .to() follows it.
anima_closeout v19: the bed's NaN was AUTOCAST_DTYPE. utils/common.py defines it as None and train.py sets it at :287 (common.AUTOCAST_DTYPE = config['model']['dtype']) BEFORE importing any model module — models/*.py do `from utils.common import AUTOCAST_DTYPE` and bake the value into their @torch.autocast decorators at class-definition time. The bed never set it, so every layer ran under autocast(dtype=None), which resolves to FP16 on cuda (verified: fast_dtype == float16), and the bf16-trained 2B DiT overflowed to NaN — the 'DETERMINISM FAILED' assert was just nan != nan. Bed now sets bf16 immediately after importing utils.common with an assert that no model module is loaded yet, and eval_arm raises a real numerics error on any non-finite loss instead of misreporting determinism.
anima_closeout v18: the bed was batch-1, which would leave a 96GB card ~99% idle and burn ~1.5 GPU-hours across both beds. Rows are now grouped by tensor shape (3 AR buckets; prompts are always 512 tokens) and run in batches of VB_BATCH (default 16, env-tunable). The noise bank is still drawn per row in the original seeded order, so the (row, noise, t) triples are unchanged; loss is reduced PER SAMPLE and accumulated in fp64 so batch composition cannot shift a total. Proven: VB_BATCH=1 reproduces the row-by-row reference EXACTLY (0.0), and 4/16/64 agree to <1e-8 (float32 reduction-order noise); every row covered exactly once, no batch mixes shapes.
anima_closeout v17: the bed called transformer.forward() directly, which is NOT the trainer's path — concat_padding_mask is on so forward()'s padding_mask=None default dies in prepare_embedded_sequence, and it also skips the llm_adapter and the per-block multiband window dispatch. The bed now runs pipe.to_layers() (InitialLayer -> LLMAdapterLayer -> 28x TransformerLayer -> FinalLayer) with the exact input tuple prepare_inputs builds, so it inherits the zero padding mask, the adapter call and the band dispatch instead of re-implementing any of them. The image_spec join from v16 is confirmed working on the live cache: 512 unique specs, 184+189+139 rows joined.
anima_closeout v16.1: correct the layout comment to the real bucket_suffix spellings (ar_frames_{ar:.3f}_{f} with an underscore vs cache_{ar:.3f}x{w}x{h}x{f})
anima_closeout v16: stop deriving cache directory names. v15 built the AR dir as ar_frames_{ar}x{f} but bucket_suffix (dataset.py:60-70) joins 2-tuples with an UNDERSCORE (ar_frames_0.630_1) and 4-tuples with 'x' (cache_0.630x400x640x1). The bed now WALKS the cache root, discovering every 'latents' and 'text_embeddings_*' dir (uncond_* excluded by prefix), and joins on image_spec, which is globally unique — so no directory-naming convention can break it again. Verified on a synthetic cache built by the fork's own ParquetCache with names from the fork's own bucket_suffix: 3 AR/size bucket pairs, shuffled latents, uncond + iteration_order decoys -> 24/24 rows correctly paired, each image exactly once.
anima_closeout v15: the val bed was reading the eval cache wrong in TWO ways. (1) text embeddings live in the AR-bucket dir (ar_frames_{ar}x{f}, dataset.py:485), not the size-bucket dir (cache_{ar}x{w}x{h}x{f}, :247) — hence 'found []'. (2) far worse: SizeBucketDataset SHUFFLES its metadata (:241-242) before caching latents while TE is cached unshuffled one level up, so pairing by row order marries each latent to someone else's caption and still yields plausible numbers. The bed now joins on image_spec — the key the fork itself joins on (:312) — and iterates in sorted spec order for run-to-run reproducibility. PROVEN on a synthetic cache written by the fork's OWN ParquetCache with the real shuffle: 24/24 rows correctly paired, while the old order-based read mismatched 23/24.
anima_closeout v14: full resume rebuild — every arm now resolves LOCAL -> HUB -> train off one ARM_SHIP registry that drives shipping AND restoring, so a trained arm is never retrained and a fresh runtime costs a ~29MB download per arm. Restored arms carry their eval_curve.json (no tfevents exist), smokes skip when a full arm already proves the same thing, CELL 6 rebuilds the held-out cache itself (1-step cache_eval over the eval TOML) and resolves its own checkpoints, CELL 7 reads results from disk. exp017's surgical verdict gained an effect-size guard: the shipped mb3 gates sit at sigmoid(-3.6..-7.8), so a lesion ratio on a near-inert stack is reported VACUOUS, not PASS. Verified against the live hub: real restore of relay_s0/s1 + mb3, no training launched, newest_save refuses restored dirs, ship-skip stateless; builder now enforces cross-cell name resolution over 3 run orders.
anima_closeout v13: stale-CELL-6 report (v6-era DATA glob found no eval cache — replaced in v7 by the deterministic dp_cache/heldout/cache/anima root) + a real catch from the live logs: anima's TE cache dir is text_embeddings_1, not the hardcoded _0 — the val bed now DISCOVERS the text_embeddings_* dir per bucket (uncond excluded, exactly one asserted).
anima_closeout v12: smoke + both relay arms TRAINED; the LoRA control died in peft's torchao dispatcher (Colab preinstalls torchao 0.10.0, peft raises wanting >=0.16; nothing here uses torchao -> CELL 0 uninstalls it, probe then returns False and dispatch falls to standard Linear). run_arm is now IDEMPOTENT: an arm with its final-step save present is reused, never retrained — the campaign resumes from local files after any crash.
anima_closeout v11: first live run of bf16_master_weights crashed at the scheduler — ds_config['bf16'] made DeepSpeed wrap the client Adam (FP16_UnfusedOptimizer), which torch ConstantLR rejects. Fork 812449a: client-side MasterWeightsAdam (fp32 masters, still a torch Adam; engine stays on the exp004-proven no-wrapping path). Proven in pure torch incl. the exact exp004 freeze repro: -3.0 bf16 gate frozen under plain Adam, moves under MasterWeightsAdam. CELL 0 asserts the new mechanism. Caches from the 28-min build are all banked — next launch goes straight to training.
anima_closeout v10: latent caching died on its FIRST image read — the fork's ParquetImageReader whole-file column read cannot deliver >2GiB nested (struct) columns (ArrowNotImplementedError), and the 8192-row split monolith crosses that line. Fixed in the fork (8ed4b6b: per-row-group reads assembled into a ChunkedArray; proven against a 1200x2MB struct parquet reproducing the exact error). CELL 0 asserts the fixed reader pre-spend. No CELL 1 re-run needed: the monolith is readable as-is.
anima_closeout v9: the ArrowIndexError smoke crash was the shared-cache collision detonating — CELL 1 had last run as v6 (same ipykernel_5091 across all three crashes) so the dataset TOMLs on disk lacked path=, and eval's grouped-metadata save clobbered train's mmapped files (same keys, same filenames, shared local_parquet root) right before train's latents map read through them (writer died at the first 1000-row flush). v9: CELL 1 also rmtree's the poisoned default root; CELL 2 refuses to launch any arm if the dataset TOMLs lack dp_cache. RUNBOOK: re-run CELL 1 in the live kernel, then CELL 2.
anima_closeout v8: the un-masked smoke crash was ModuleNotFoundError: hyvideo — utils/patches.py imported hyvideo.text_encoder unconditionally at module level while CELL 0 (correctly) initializes only the ComfyUI submodule. Fixed in the fork (5f37138: guarded import, patch skipped when absent; verified by execution both ways). CELL 0 now asserts the guard is present before any spend.
anima_closeout v7: (1) rc no longer masked — the deepspeed|tee pipeline returned TEE's exit code, so the smoke's ~24s death printed rc=0; Python is now the tee and any failure (rc!=0 OR no run dir) dumps the log tail into the cell. (2) CRITICAL: train+eval dataset TOMLs shared the fork's default parquet cache root ('local_parquet' constant) — eval would clobber train's metadata/latents and the loop would silently train on the 512 eval rows; now explicit distinct path= roots under dp_cache/. (3) width_column/height_column (the real fork keys; image_*_column variants were silently ignored). (4) val bed reads the fork's REAL cache layout: per-bucket paired latents/text_embeddings_0 caches. Source-verified against the fork + exp004's as-run TOMLs by 4-agent audit.
anima_closeout v6: CELL 1's shard loop variable was `sh`, clobbering CELL 0's sh() shell helper in the shared notebook namespace -> CELL 2 smoke crashed with 'str' object is not callable. Renamed + builder now AST-checks every cell for cross-cell function-name clobbering (proven to flag the exact bug).
CELL 1 v5: properly columnar two-phase split. Phase A thin scan (id/captions/audit) with pyarrow.compute masks — exp011 gate, dexp011:154 caption coalesce using the forks own CAPTION_SENTINELS — stop at target; phase B heavy write filtered by is_in(id, split sets), image bytes never leave Arrow. Row-by-row loop removed. Both phases executed green against a real shard in 0.5s.
CELL 1 rewritten as an INCREMENTAL materializer: the fused set is ~160GB and the campaign needs ~9k rows — shards visited in sha256(name) order one at a time, gated+deduped+slimmed into the pool, source deleted per shard, hard stop at TRAIN_TARGET+EVAL_N (8192+512). Download bounded to ~10GB instead of the corpus. Split logic executed green against a real shard.
Caption key corrected per the campaign record: qwen-deepfashion-fused stays THE dataset (r2 exp011-013 design of record); caption = caption_joycaption with prompt_fused fallback (dexp011:154), materialized as a derived column; exp011 audit gate + id dedup in the split. The earlier ft1 retarget was wrong — the fix was the key, not the dataset.
Fix dataset target: exp004 trained on diffusion-pretrain-set-ft1 data/deepfashion (has caption_vlm_json), not qwen-deepfashion-fused; shard listing filtered to the config (the repo holds 582 parquet across configs). Caught by CELL 0 refusing before spend — as designed.
Anima adapter is ComfyUI-loadable: the node pack now supports the MiniTrainDIT family (verified live on anima_baseV10, 28 sites, toggle-off bit-exact) — caveat corrected in the catalog and in the file metadata
Backfill the missing safetensors companions (dispatch banks, monoliths, cond adapter) into their original experiment packages — every .pt stack now has a verified companion
amoe_loras/: every stack converted to safetensors (38 raw, bitwise-verified) + 36 function-first production artifacts carrying their own usage cards; catalog + SCHEMA. Closes the 15 missing companions (banks/monos/cond) the burndown claim overstated.
Burndown record completion: every s1 replication ckpt, every monolith/control falsifier, exp011 fused stacks, exp012/013 blob+uniform stacks, exp014/015 key banks, exp004 fork run saves + LoRA control (salvaged from the pod before window close)
exp015 shipped: address-as-key falsified with corrected instruments (zero routing excess, zero match advantage) - the dispatch-key form closes; TE+AMOE joint training is the open form
exp013 shipped: CONDITIONING HYPOTHESIS CONFIRMED - blob coupling ~200x more effective on flow than eps (prereg 5x bar); blob work migrates to flow/v-pred substrates
