# mv-sam3d-for-6d-v2 — SSFlow implementation (corrected design) Seed-only **partial → full sparse-structure rectified flow** in SAM-3D's SS latent space (branch `ssflow`; spec: `../../../PLAN/PLAN.md` + `../../../PLAN/DATA_FORMAT.md`, with the USER-CORRECTED model design below superseding the plan's "strip cross-attn/condition" step). > **See `README_MVSAM3D_TRAINING.md`** for the current top-level doc: the full > module map (including the stage-2 **SlatFlowModel** / `train_slat_flow.py` > stack, spec `../../../PLAN/SLAT_GEN_PLAN.md`, which this file predates), > feature→file:line index, run commands, config-key lists, the measured > batch-size/ETA guidance, and the verified smoke evidence for BOTH training > loops. Stage-2-only features (the `enable_soft_mask`/`enable_plucker` > split, the `data.max_num_voxels` filter, TRUE batched multi-object > inference) are documented there. This file remains the SSFlow design record. ## Corrected design (supersedes commit ecf48ee's backbone stripping) **The stage-1 flow transformer is ARCHITECTURALLY IDENTICAL to the pretrained SAM-3D `ss_generator`** (`SparseStructureFlowTdfyWrapper` exactly as built by ss_generator.yaml). The earlier pass wrongly removed cross-attention, its pre-norm (norm2) and the condition pathway — that is REVERTED (`modulated.py` + `mot_sparse_structure_flow.py` restored pristine, and the DINO/pointmap/fuser embedders under `sam3d_objects/model/backbone/dit/` restored). * **KEPT** (warm-started from ss_generator.ckpt): the shape stream, every block's self-attn + **cross_attn (+ norm2)** + FFN + adaLN, `t_embedder`, the shortcut `d_embedder` (present but unused: `d=None`; frozen so DDP has no unused trainable params), and the **external condition embedder** (DINO-L/14-reg ×2 + PointPatchEmbed `EmbedderFuser`, frozen — exactly ss_generator.yaml's `module.condition_embedder.backbone`). * **DROPPED**: only the 6D-POSE latent streams (`6drotation_normalized`/`translation`/`scale`/`translation_scale`) and their targets/metrics — mv-sam3d-for-6d ADD-ONS, not part of the original SS generator. `latent_mapping` = `{"shape"}`, `latent_share_transformer={}`. The single-stream MOT self-attention guard (`if other_modalities:` in `mm_scale_dot_product_attention`; `mode != "default"` raises) stays — with one stream Path-2 would `torch.cat([])`. * **The ONLY training change vs the pretrained generator**: x0 = `z1_partial` (ss_encoder of the partial occupancy) instead of Gaussian noise: ``` x0 = z_partial = ss_encoder(PARTIAL 64^3 occupancy from backprojected multi-view depth; canonical [-0.5,0.5]^3; visible-only, noise='zeros') x1 = z_full = ss_encoder(FULL 64^3 surface occupancy of the mesh) z_t = (1-t)·x0 + t·x1 (sigma_min = 0) target v = x1 - x0, loss = MSE(v_pred, v) # v-prediction ``` This is EXACTLY the seed-only inference forcing (batch_appforce z1fwd), now trained. Latents are RAW posterior means — no mean/std normalization anywhere. * **CONDITIONING**: the pathway is kept. Geometry-only runs (no renders yet) pass a NULL/zero context `[B,1,1024]` — through qkv-bias cross-attention an all-zero context is token-count-invariant, so this equals the pretrained `force_zeros_cond` classifier-free branch (`cond * 0`). Real image/pointmap conditioning goes through `SSFlowModel.encode_condition(**modalities)` (kwargs `image`/`rgb_image`/`mask`/`rgb_image_mask`/`pointmap`/ `rgb_pointmap`, per ss_generator.yaml) once the **trellis500k renders** land — the render→condition DATA wiring is deliberately NOT built yet. ## Model (`mvsam3d/model/ssflow_model.py`, class `SSFlowModel`) * Backbone kwargs mirror ss_generator.yaml minus the pose latents: `cond_channels=1024`, `condition_embedder=None` INSIDE the wrapper (identity — the EmbedderFuser lives OUTSIDE, its tokens are the positional `cond` arg, exactly as pretrained), `force_zeros_cond=True`, `is_shortcut_model=True`, 24 blocks, 16 heads, mlp_ratio 4, ape, `qk_rms_norm=True`, `qk_rms_norm_cross=False`, time_scale 1000; `freeze_shared_parameters=False` (finetune everything trainable). * `forward(batch)` = `flow_step`: `batch["z_partial"]`, `batch["z_full"]` `[B,4096,8]`; optional `batch["cond"]` `[B,N,1024]` (absent → null/zero condition); t ~ logit-normal(−1, 1) per sample (Q3, pretraining-matched; `uniform` = the data→data ablation); returns `{"loss", "loss/shape"}`. * `sample(z_partial, num_steps=25, cond=None)` — SAM-3D-EXACT seeded inference (commit 0f5342f: the hand-rolled Euler loop was replaced by a thin wrapper, verified bit-identical before the swap): z0 = partial latent VERBATIM injected via the `seeded_noise` contextmanager into the REAL `FlowMatching` sampler (`generate → generate_iter → _prepare_t → Euler solve_iter`), with the deployed SS inference config (25 steps, `rescale_t=3` t-warp, CFG strength 7 on interval [0,500] — a mathematical no-op while the condition is null — fp16 autocast, d=0 no_shortcut); decode chain identical to `inference_pipeline.py:812-833`: permute/view → standalone `ss_decoder` (NEVER the TRELLIS-identical decoder inside ss_generator.ckpt) → logits `[B,1,64,64,64]` → `coords_original = argwhere(ss>0)[:,[0,2,3,4]].int()` → **`coords` = `downsample_sparse_structure(prune_sparse_structure(·, dist=1))` — the PRIMARY output (SAM-3D pipeline contract: pruned surface shell + int32-guard downsample). Use `coords` for IoU eval and downstream; `coords_original` is debug-only.** * Warm-start (`load_pretrained_backbone`) — **~100% of the model's tensors**: * backbone ← `_base_models.generator.reverse_fn.backbone.*`: **493/493** own tensors (0 left at init, 0 mismatched), incl. all `cross_attn.shape.*`, `norm2.shape.*`, `d_embedder.*`; the ckpt's other 452 tensors are the dropped 6D-pose-stream weights. * condition_embedder ← `_base_models.condition_embedder.*`: **722/722** (exact bijection with the ckpt's prefix). * Self-check (2026-08-31): flow_step loss+backward with null cond (cross-attn gradients flow); zero-context length invariance ≤ 2e-5; `sample()` returns int32 `(N,4)` coords ⊆ coords_original. ## SS VAE (`mvsam3d/model/ss_vae.py`) Unchanged loaders (strict, 0 missing/unexpected) + helpers: `ss_encode(occ) → [B,8,16,16,16]` (frozen cached encoder, no_grad — the train-step partial encode) and `grid_latent_to_tokens` (→ `[B,4096,8]`). Checkpoints resolve via `mvsam3d/paths.py` (`MIGRATOR_CHECKPOINTS/hf`); `migrator/checkpoints/hf` is a symlink to the downloaded SAM-3D set (the working ss_encoder.ckpt, 119 MB — the /data hf copy is 0-byte, do not use). ## Data * On-disk: DATA_FORMAT.md layout `//objects///{meta.json, geometry/voxels.npz, geometry/ss_latent_sam3d.npz, views/views.npz}`; `meta.json` written LAST is the completion marker (idempotent re-runs). * `mvsam3d/data/ssflow_dataset.py` — `TrellisSSFlowDataset`: * OWN views.npz loader: per-view `fx=K[i,0,0], fy=K[i,1,1], cx=K[i,0,2], cy=K[i,1,2]`, depth `uint16 mm / 1000`, OpenCV c2w. Res-agnostic (Q5: future trellis500k renders at 512; existing toys4k=768 / omni3d=800). * Per `__getitem__` (this IS the augmentation): random view subset **|S| ~ U{1..8} (Q4), clamped per object to its view count** → partial occupancy by the EXACT batch_appforce recipe (depth>0 mask, `binary_erosion(iterations=2)`, pinhole backproject, keep |xyz| ≤ 0.512, clip to [-0.5, 0.5−1e−9], voxelize 64, union, visible-only = noise='zeros') → returns `occ_partial [1,64,64,64]` + `z_full [4096,8]` + `occ_full` (for IoU). `exclude_names` drops given object ids. * `z_partial` is encoded in the TRAIN STEP by the frozen `ss_encode`, never precomputed. Deterministic per-(seed, epoch, idx) RNG. * **Val set (Q7)**: `data.val_roots` = explicit validation roots — 100 held-out toys4k objects (`smoke_data/val_toys4k100`, disjoint from toys4k200 via `--exclude_ids_from_root`) + all local omni3d samples (`smoke_data/val_omni3d`, front+side renders from `/lp-dev/jonghoon/mv-mesh/exp_faithfulness/omni3d/renders`). When set, val = ALL objects under val_roots and train = roots MINUS val ids; empty → legacy hash split. * `tools/preprocess_ssflow_toys4k.py` — DATA_FORMAT adapter (toys4k AND omni3d: auto-detects available `front/side/oside/back` views; `--source`, `--exclude_ids_from_root`). Gates: (a) full-occ encoder/decoder round-trip IoU ≥ ~0.9; (b) partial ⊂ full containment. * `tools/gen_x1_official.py` + `slat_official/` — the ONLY producer of the SLAT training target x1 (USER MANDATE: exact official algorithm, no reimplementation): vendored byte-identical TRELLIS dataset_toolkits (Blender 150-view render → open3d 64³ voxelize → all-voxel no-occlusion DINO mean → `encoder(feats, sample_posterior=False)`) with the single mandated deviation of SAM3D's `slat_encoder.ckpt`, emitting `slat/slat_official_sam3d.npz` (RAW SAM3D-space latent on ALL active voxels). The former on-the-fly visible-mean x1 was deleted; `encode_visible_latent` survives only as the inference-time forced x0 seed. ## Training (`mvsam3d/train/train_ss_encoder.py` + `configs/train_ssflow.yaml`) Rectified-flow v-pred loop: AdamW (trainable = backbone minus d_embedder; condition embedder + d_embedder frozen; `optim.weight_decay` config-driven, default 0.0), bf16 autocast, warmup→cosine, grad-accum, DDP via the vendored `stage1_flow_lora.{distributed,checkpointing,train_utils}` (single-GPU transparently skips the process group). Per step: move batch → `z_partial = grid_latent_to_tokens(ss_encode(occ_partial))` (fp32, no_grad, outside autocast) → `model({"z_partial", "z_full"})` (null cond) → backward/clip/step (`train_ss_encoder.py:304-308`). Two optional, config-driven, default-OFF additions (shared with the SlatFlow loop): * **EMA** (`ema.{enable,rate,eval_with_ema}`, default `enable: false`) — `mvsam3d/train/ema.py` `EmaShadow`: fp32 shadow of ONLY the trainable (`requires_grad`) params, `ema = rate*ema + (1-rate)*param` after every `optimizer.step()` (`train_ss_encoder.py:336-337`); with `eval_with_ema` the in-loop validation runs under the EMA weights (bit-exact restore after, `:354-357`). Persisted as a `step_XXXX_ema.pt` sidecar next to the `_optimizer.pt` sidecar (the vendored `checkpointing.py` is untouched) and auto-restored on `resume_from` (`:260-268`, `:380-381`). * **Gradient checkpointing** (`model.use_checkpoint`, default false) — threaded through the `SSFlowModel` ctor (`ssflow_model.py:145`) to the backbone's native per-block `use_checkpoint` (`:207`): activations of the 24 DiT blocks are recomputed in backward for a large peak-memory cut at ~modest step-time cost. NOTE: checkpoints include the frozen condition-embedder tensors (~1.2 GB fp32 extra) — keep them under /lp-dev. ``` cd export MIGRATOR_CHECKPOINTS=$PWD/migrator/checkpoints PYTHONPATH=. python mvsam3d/train/train_ss_encoder.py \ --config configs/train_ssflow.yaml --set train.steps=2000 # multi-GPU: torchrun --nproc_per_node=N ... (GPUs 0,1,4,5,6,7 only on this box) ``` ## Validation (`mvsam3d/train/validate.py`) Q7 val set (val_roots), per pass (fixed seed, training RNG saved/restored): `decoded_iou` — seeded `sample()` → **`coords` (the PRIMARY pruned+downsampled pipeline output) scattered to 64³** vs the full surface-occupancy grid; `partial_iou` (input-copy floor: decoded > partial ⇒ the model completes, not copies); held-out flow `loss`/`loss/shape`. ## Smoke plan (PLAN.md §6; runs = Smoke phase) 1. toys4k200 preprocess + gates — DONE; val set (100 toys4k + omni3d) — DONE. 2. Unit self-check — DONE (warm-start 493/493 + 722/722, null-cond forward, coords-primary sample). 3. Overfit: 8 objects, batch 8, lr 1e-4, bf16, warm, null cond — DONE (2026-09-01, `.debug/train_ssflow_overfit8_e2e.log`): train v-MSE 0.0953 → 0.0001 over 600 steps (~950×); held-out decoded_iou 0.6441 > partial_iou floor 0.5483 (n=16). PASS. 4. Generalization sniff: toys4k200 train vs val_roots; val v-MSE down, val decoded IoU > partial floor. 5. Then point `data.roots` at the trellis500k subset (Q8, download-agent scope) as it lands; wire render conditioning through `encode_condition`. ## Removed (dead code, PLAN.md §4) — updated `mvsam3d/model/{condition,flow_model}.py`, `mvsam3d/data/{dexycb_dataset, adapters,base_dataset,integrated,dexycb_calib}.py`, `mvsam3d/infer/`, `mvsam3d/train/train.py`, `sam3d_objects/geometry/` (pose-only rot6d/ssi), verify/diagnose/smoke tools, `build_canonical_latents_multi.py`, DexYCB/ integrated configs + `train.yaml`, stale docs, all stale sbatch scripts. RESTORED (corrected design): `sam3d_objects/model/backbone/dit/` embedders, pristine `modulated.py` + `mot_sparse_structure_flow.py`. Kept per spec: `mm_latent.py`, `ss_vae.py`, the Euler solver, `timestep_embedder.py`, `stage1_flow_lora` training utils, `tools/build_canonical_latents.py`, `canonical.py::voxelize_canonical`. ## 2026-09-02 — DINO seed-path memory-format fix + in-training appearance-forcing validation * `MultiViewSlatConditionEmbedder.dino_prenorm` now runs DINO **per view on an NCHW-contiguous tensor**. Root cause found by the step-0 parity harness: the dataset builds `crops` with `np.stack` over transposed HWC arrays, which yields channels-last strides; DINOv2's patch-embed conv then selects different cuDNN kernels (~1% token deviation vs the reference `batch_appforce_sam3d.dino_features`, max|d| 1.5–3.4 on a 185 scale), and a V-view batched forward adds ~7e-4. Per-view + `.contiguous()` is bit-exact against the reference dumps. This affects the training x0 seed and the non-anchor condition tokens identically (both come from `dino_prenorm`). * In-training validation: `mvsam3d/train/val_appforce.py` (trained stage-2 on cached reference stage-1 coords/base; seed + reinject schedules; per-rank decode + `evaluate_appforce.py` in the reference env via `tools/val_stage1_ref.py`; rank-0 merge -> `val_af///` in wandb), hooked into `train_slat_flow.py` (`val_appforce:` config block, step 0 + every 5000). Parity harness: `tools/parity_val_appforce.sh`. ## 2026-09-02 — TRUE multi-object BATCHED stage-2 (SLAT flow) TRAINING Stage-2 training was bs=1-per-forward (dense `[1,N,8]` over one coord set); the effective batch came only from `train.grad_accum_steps`. Now the verified batched-INFERENCE layout (`sample_batched`, `tools/selfcheck_batched_slat.py`) is reused for TRAINING, so an effective batch of 12 = 4 GPUs x `train.batch_objects=3` with **no gradient accumulation**. * `mvsam3d/model/slatflow_model.py` * `forward` dispatches on the presence of a per-object `layout` key (`collate_batched` batch -> batched step, `collate_single` -> legacy step). * `_object_train_pieces` — per-object cond (fuser anchor + non-anchor DINO tokens + Plücker) / x0 forced-appearance seed with fresh noise on the invisible rows / normalized official x1 — the SAME construction `flow_step` performs. * `flow_step_batched` — B objects concatenated into ONE sparse coord set with the batch-index column (TRELLIS `SLat.collate_fn`), conditions padded to a common L and block-diagonally masked via `CrossAttnBiasHolder.set_batch` (pad columns -inf; each object's soft mask on its own rows/columns), ONE `gen.reverse_fn` call. **t is per object** (`[B]`, one `gen._generate_t` draw each): the backbone's `t_embedder(t)` yields `[B,C]` which `sparse_batch_broadcast` expands per object over the sparse layout — so no extra plumbing is needed for the timestep. Because the dense tensor is `[1,T_total,8]` (no batch axis) the interpolant is applied with a PER-TOKEN t (t expanded over each layout slice) using the generator's own equations and `gen.sigma_min`: `x_t=(1-(1-σ)t)x0+t·x1`, `v=x1-(1-σ)x0`. Loss = **mean over objects of each object's own per-token MSE** (per-object mean, NOT token-count weighted) so the objective is identical to averaging B bs=1 losses. DDP guard: `plucker_fc` is kept in the graph (×0.0) when NO object in the batch has a non-anchor view. * `mvsam3d/train/train_slat_flow.py` — new `train.batch_objects` (default 1 = previous behaviour). >1 switches the loader to `batch_size=batch_objects` + `collate_batched` and requires `grad_accum_steps == 1` (hard error otherwise); the printed `global_batch` is now `batch_objects × grad_accum × world_size`. Validation stays bs=1 (`collate_single` + `flow_step`). * `tools/selfcheck_batched_train.py` — equivalence + memory self-check: with x0 noise and t pinned, batched loss == mean of the B single losses, and peak memory / grad finiteness for `use_checkpoint` off/on. ## 2026-09-03 — BATCHED sampling + PARALLEL decode/metrics for `val_appforce` The in-training appearance-forcing validation was one object per 25-step ODE solve and ONE decode process per rank. Both are now parallel; the outputs, file layout and wandb keys (`val_af///`) are unchanged. * `mvsam3d/train/val_appforce.py` * `_stage2_prepare` — the per-object part of `stage2` up to the solve (DINO prenorm -> condition -> `encode_visible_latent` -> seed tensor), in the same op order as the single path. Stays per object (it is cheap); the batching target is the 25-step flow. * `_stage2_solve_chunk` — ONE batched solve over a chunk via `SlatFlowModel.sample_batched` (the verified contract of `tools/selfcheck_batched_slat.py`: coords concatenated with the batch-index column, seeds `torch.cat` into `[1,T_total,8]`, conditions padded to a common L and block-diagonally masked through `CrossAttnBiasHolder.set_batch`). **`reinject` batched pin:** object *i*'s visible rows live at `offset_i + local_row` (offset_i = Σ N of the earlier objects in the chunk = exactly the layout `sample_batched` builds); `eps_fixed` / `z_forced` are those same rows of the object's OWN base noise / `z_norm`, concatenated in the same order, so the single-path formula `x[vis] = (1-(1-σ)t_next)·eps_fixed + t_next·z_forced` applies verbatim to the gathered global row set after every Euler step. `seed` is applied per object on its own base before the concatenation. * `stage2_batched(model, items, bases, schedule, batch_size=6)` — chunks the object list (sorted by voxel count so a chunk's padding waste is small) and returns one `stage2`-shaped result dict per input item, in input order. * `sample_jobs(..., batch_size=6, single=False)` — batched by default; `single=True` / `VAL_AF_SINGLE=1` / `batch_size<=1` restore the original path (`_sample_jobs_single`, kept verbatim) for A/B. A batched failure (e.g. OOM on a chunk of large objects) falls back to the single path for that chunk instead of losing the objects. * `decode_and_eval_rank(..., decode_procs=2, threads=8)` — `decode_procs` reference-env decode processes per rank run CONCURRENTLY, sharded with the decode mode's own `--shard i --nshards n` (disjoint `.npz`, disjoint GLB names, so the shared `glb/` dir and the per-rank hardlink dir are race-free); the 4 `evaluate_appforce.py` processes stay concurrent. `_ref_env` now sets OMP/MKL/OpenBLAS/NumExpr thread caps EXPLICITLY (the trainer already exports `OMP_NUM_THREADS`, so `setdefault` was a no-op): 4 ranks x 4 concurrent eval procs x 8 threads = 128 of ~255 cores. * config `val_appforce:` — new `batch_size` (6), `single` (false), `decode_procs` (2), `ref_threads` (8) in `configs/train_slatflow.yaml`, `configs/train_slatflow_prod.yaml`, `configs/_smoke_val_e2e.yaml`. * `tools/verify_batched_val.py` — offline verification against the SINGLE-path step-5000 dumps: reloads the step-5000 ckpt + EMA sidecar, reruns the single path (the run floor), then the batched path at B ∈ {1,4,6,8} for both schedules, reporting per object max|d| / mean|d| vs the reference and vs the single re-run, coords equality, and s/object timing. PASS iff `max|batched-ref| <= max(3·max|single_rerun-ref|, 0.05·feat_scale)`. `--parallel-eval-check` additionally runs the new parallel decode+eval on those objects into a temp dir and diffs the `results.json` metrics against the step-5000 `eval_rank*/results.json` rows.