jhoonjwa's picture
Add slat_dev snapshot 2026-11-11: stage-2 code with test-time soft-mask bias on/off (BIAS_INFERENCE.md)
ffee162 verified
|
Raw History Blame Contribute Delete
20.6 kB

mv-sam3d-for-6d-v2 — SSFlow implementation (corrected design)

Seed-only partial → full sparse-structure rectified flow in SAM-3D's SS latent space (branch ssflow; spec: ../../../PLAN/PLAN.md + ../../../PLAN/DATA_FORMAT.md, with the USER-CORRECTED model design below superseding the plan's "strip cross-attn/condition" step).

See README_MVSAM3D_TRAINING.md for the current top-level doc: the full module map (including the stage-2 SlatFlowModel / train_slat_flow.py stack, spec ../../../PLAN/SLAT_GEN_PLAN.md, which this file predates), feature→file:line index, run commands, config-key lists, the measured batch-size/ETA guidance, and the verified smoke evidence for BOTH training loops. Stage-2-only features (the enable_soft_mask/enable_plucker split, the data.max_num_voxels filter, TRUE batched multi-object inference) are documented there. This file remains the SSFlow design record.

Corrected design (supersedes commit ecf48ee's backbone stripping)

The stage-1 flow transformer is ARCHITECTURALLY IDENTICAL to the pretrained SAM-3D ss_generator (SparseStructureFlowTdfyWrapper exactly as built by ss_generator.yaml). The earlier pass wrongly removed cross-attention, its pre-norm (norm2) and the condition pathway — that is REVERTED (modulated.py + mot_sparse_structure_flow.py restored pristine, and the DINO/pointmap/fuser embedders under sam3d_objects/model/backbone/dit/ restored).

  • KEPT (warm-started from ss_generator.ckpt): the shape stream, every block's self-attn + cross_attn (+ norm2) + FFN + adaLN, t_embedder, the shortcut d_embedder (present but unused: d=None; frozen so DDP has no unused trainable params), and the external condition embedder (DINO-L/14-reg ×2 + PointPatchEmbed EmbedderFuser, frozen — exactly ss_generator.yaml's module.condition_embedder.backbone).
  • DROPPED: only the 6D-POSE latent streams (6drotation_normalized/translation/scale/translation_scale) and their targets/metrics — mv-sam3d-for-6d ADD-ONS, not part of the original SS generator. latent_mapping = {"shape"}, latent_share_transformer={}. The single-stream MOT self-attention guard (if other_modalities: in mm_scale_dot_product_attention; mode != "default" raises) stays — with one stream Path-2 would torch.cat([]).
  • The ONLY training change vs the pretrained generator: x0 = z1_partial (ss_encoder of the partial occupancy) instead of Gaussian noise:
x0 = z_partial = ss_encoder(PARTIAL 64^3 occupancy from backprojected multi-view
                            depth; canonical [-0.5,0.5]^3; visible-only, noise='zeros')
x1 = z_full    = ss_encoder(FULL 64^3 surface occupancy of the mesh)
z_t = (1-t)·x0 + t·x1          (sigma_min = 0)
target v = x1 - x0,  loss = MSE(v_pred, v)          # v-prediction

This is EXACTLY the seed-only inference forcing (batch_appforce z1fwd), now trained. Latents are RAW posterior means — no mean/std normalization anywhere.

  • CONDITIONING: the pathway is kept. Geometry-only runs (no renders yet) pass a NULL/zero context [B,1,1024] — through qkv-bias cross-attention an all-zero context is token-count-invariant, so this equals the pretrained force_zeros_cond classifier-free branch (cond * 0). Real image/pointmap conditioning goes through SSFlowModel.encode_condition(**modalities) (kwargs image/rgb_image/mask/rgb_image_mask/pointmap/ rgb_pointmap, per ss_generator.yaml) once the trellis500k renders land — the render→condition DATA wiring is deliberately NOT built yet.

Model (mvsam3d/model/ssflow_model.py, class SSFlowModel)

  • Backbone kwargs mirror ss_generator.yaml minus the pose latents: cond_channels=1024, condition_embedder=None INSIDE the wrapper (identity — the EmbedderFuser lives OUTSIDE, its tokens are the positional cond arg, exactly as pretrained), force_zeros_cond=True, is_shortcut_model=True, 24 blocks, 16 heads, mlp_ratio 4, ape, qk_rms_norm=True, qk_rms_norm_cross=False, time_scale 1000; freeze_shared_parameters=False (finetune everything trainable).
  • forward(batch) = flow_step: batch["z_partial"], batch["z_full"] [B,4096,8]; optional batch["cond"] [B,N,1024] (absent → null/zero condition); t ~ logit-normal(−1, 1) per sample (Q3, pretraining-matched; uniform = the data→data ablation); returns {"loss", "loss/shape"}.
  • sample(z_partial, num_steps=25, cond=None) — SAM-3D-EXACT seeded inference (commit 0f5342f: the hand-rolled Euler loop was replaced by a thin wrapper, verified bit-identical before the swap): z0 = partial latent VERBATIM injected via the seeded_noise contextmanager into the REAL FlowMatching sampler (generate → generate_iter → _prepare_t → Euler solve_iter), with the deployed SS inference config (25 steps, rescale_t=3 t-warp, CFG strength 7 on interval [0,500] — a mathematical no-op while the condition is null — fp16 autocast, d=0 no_shortcut); decode chain identical to inference_pipeline.py:812-833: permute/view → standalone ss_decoder (NEVER the TRELLIS-identical decoder inside ss_generator.ckpt) → logits [B,1,64,64,64] → coords_original = argwhere(ss>0)[:,[0,2,3,4]].int() → coords = downsample_sparse_structure(prune_sparse_structure(·, dist=1)) — the PRIMARY output (SAM-3D pipeline contract: pruned surface shell + int32-guard downsample). Use coords for IoU eval and downstream; coords_original is debug-only.
  • Warm-start (load_pretrained_backbone) — ~100% of the model's tensors:
    • backbone ← _base_models.generator.reverse_fn.backbone.*: 493/493 own tensors (0 left at init, 0 mismatched), incl. all cross_attn.shape.*, norm2.shape.*, d_embedder.*; the ckpt's other 452 tensors are the dropped 6D-pose-stream weights.
    • condition_embedder ← _base_models.condition_embedder.*: 722/722 (exact bijection with the ckpt's prefix).
  • Self-check (2026-08-31): flow_step loss+backward with null cond (cross-attn gradients flow); zero-context length invariance ≤ 2e-5; sample() returns int32 (N,4) coords ⊆ coords_original.

SS VAE (mvsam3d/model/ss_vae.py)

Unchanged loaders (strict, 0 missing/unexpected) + helpers: ss_encode(occ) → [B,8,16,16,16] (frozen cached encoder, no_grad — the train-step partial encode) and grid_latent_to_tokens (→ [B,4096,8]). Checkpoints resolve via mvsam3d/paths.py (MIGRATOR_CHECKPOINTS/hf); migrator/checkpoints/hf is a symlink to the downloaded SAM-3D set (the working ss_encoder.ckpt, 119 MB — the /data hf copy is 0-byte, do not use).

Data

  • On-disk: DATA_FORMAT.md layout <root>/<SOURCE>/objects/<id[:2]>/<id>/{meta.json, geometry/voxels.npz, geometry/ss_latent_sam3d.npz, views/views.npz}; meta.json written LAST is the completion marker (idempotent re-runs).
  • mvsam3d/data/ssflow_dataset.py — TrellisSSFlowDataset:
    • OWN views.npz loader: per-view fx=K[i,0,0], fy=K[i,1,1], cx=K[i,0,2], cy=K[i,1,2], depth uint16 mm / 1000, OpenCV c2w. Res-agnostic (Q5: future trellis500k renders at 512; existing toys4k=768 / omni3d=800).
    • Per __getitem__ (this IS the augmentation): random view subset |S| ~ U{1..8} (Q4), clamped per object to its view count → partial occupancy by the EXACT batch_appforce recipe (depth>0 mask, binary_erosion(iterations=2), pinhole backproject, keep |xyz| ≤ 0.512, clip to [-0.5, 0.5−1e−9], voxelize 64, union, visible-only = noise='zeros') → returns occ_partial [1,64,64,64] + z_full [4096,8]
      • occ_full (for IoU). exclude_names drops given object ids.
    • z_partial is encoded in the TRAIN STEP by the frozen ss_encode, never precomputed. Deterministic per-(seed, epoch, idx) RNG.
  • Val set (Q7): data.val_roots = explicit validation roots — 100 held-out toys4k objects (smoke_data/val_toys4k100, disjoint from toys4k200 via --exclude_ids_from_root) + all local omni3d samples (smoke_data/val_omni3d, front+side renders from /lp-dev/jonghoon/mv-mesh/exp_faithfulness/omni3d/renders). When set, val = ALL objects under val_roots and train = roots MINUS val ids; empty → legacy hash split.
  • tools/preprocess_ssflow_toys4k.py — DATA_FORMAT adapter (toys4k AND omni3d: auto-detects available front/side/oside/back views; --source, --exclude_ids_from_root). Gates: (a) full-occ encoder/decoder round-trip IoU ≥ ~0.9; (b) partial ⊂ full containment.
  • tools/gen_x1_official.py + slat_official/ — the ONLY producer of the SLAT training target x1 (USER MANDATE: exact official algorithm, no reimplementation): vendored byte-identical TRELLIS dataset_toolkits (Blender 150-view render → open3d 64³ voxelize → all-voxel no-occlusion DINO mean → encoder(feats, sample_posterior=False)) with the single mandated deviation of SAM3D's slat_encoder.ckpt, emitting slat/slat_official_sam3d.npz (RAW SAM3D-space latent on ALL active voxels). The former on-the-fly visible-mean x1 was deleted; encode_visible_latent survives only as the inference-time forced x0 seed.

Training (mvsam3d/train/train_ss_encoder.py + configs/train_ssflow.yaml)

Rectified-flow v-pred loop: AdamW (trainable = backbone minus d_embedder; condition embedder + d_embedder frozen; optim.weight_decay config-driven, default 0.0), bf16 autocast, warmup→cosine, grad-accum, DDP via the vendored stage1_flow_lora.{distributed,checkpointing,train_utils} (single-GPU transparently skips the process group). Per step: move batch → z_partial = grid_latent_to_tokens(ss_encode(occ_partial)) (fp32, no_grad, outside autocast) → model({"z_partial", "z_full"}) (null cond) → backward/clip/step (train_ss_encoder.py:304-308).

Two optional, config-driven, default-OFF additions (shared with the SlatFlow loop):

  • EMA (ema.{enable,rate,eval_with_ema}, default enable: false) — mvsam3d/train/ema.py EmaShadow: fp32 shadow of ONLY the trainable (requires_grad) params, ema = rate*ema + (1-rate)*param after every optimizer.step() (train_ss_encoder.py:336-337); with eval_with_ema the in-loop validation runs under the EMA weights (bit-exact restore after, :354-357). Persisted as a step_XXXX_ema.pt sidecar next to the _optimizer.pt sidecar (the vendored checkpointing.py is untouched) and auto-restored on resume_from (:260-268, :380-381).
  • Gradient checkpointing (model.use_checkpoint, default false) — threaded through the SSFlowModel ctor (ssflow_model.py:145) to the backbone's native per-block use_checkpoint (:207): activations of the 24 DiT blocks are recomputed in backward for a large peak-memory cut at ~modest step-time cost.

NOTE: checkpoints include the frozen condition-embedder tensors (~1.2 GB fp32 extra) — keep them under /lp-dev.

cd <repo>
export MIGRATOR_CHECKPOINTS=$PWD/migrator/checkpoints
PYTHONPATH=. python mvsam3d/train/train_ss_encoder.py \
    --config configs/train_ssflow.yaml --set train.steps=2000
# multi-GPU: torchrun --nproc_per_node=N ... (GPUs 0,1,4,5,6,7 only on this box)

Validation (mvsam3d/train/validate.py)

Q7 val set (val_roots), per pass (fixed seed, training RNG saved/restored): decoded_iou — seeded sample() → coords (the PRIMARY pruned+downsampled pipeline output) scattered to 64³ vs the full surface-occupancy grid; partial_iou (input-copy floor: decoded > partial ⇒ the model completes, not copies); held-out flow loss/loss/shape.

Smoke plan (PLAN.md §6; runs = Smoke phase)

  1. toys4k200 preprocess + gates — DONE; val set (100 toys4k + omni3d) — DONE.
  2. Unit self-check — DONE (warm-start 493/493 + 722/722, null-cond forward, coords-primary sample).
  3. Overfit: 8 objects, batch 8, lr 1e-4, bf16, warm, null cond — DONE (2026-09-01, .debug/train_ssflow_overfit8_e2e.log): train v-MSE 0.0953 → 0.0001 over 600 steps (~950×); held-out decoded_iou 0.6441 > partial_iou floor 0.5483 (n=16). PASS.
  4. Generalization sniff: toys4k200 train vs val_roots; val v-MSE down, val decoded IoU > partial floor.
  5. Then point data.roots at the trellis500k subset (Q8, download-agent scope) as it lands; wire render conditioning through encode_condition.

Removed (dead code, PLAN.md §4) — updated

mvsam3d/model/{condition,flow_model}.py, mvsam3d/data/{dexycb_dataset, adapters,base_dataset,integrated,dexycb_calib}.py, mvsam3d/infer/, mvsam3d/train/train.py, sam3d_objects/geometry/ (pose-only rot6d/ssi), verify/diagnose/smoke tools, build_canonical_latents_multi.py, DexYCB/ integrated configs + train.yaml, stale docs, all stale sbatch scripts. RESTORED (corrected design): sam3d_objects/model/backbone/dit/ embedders, pristine modulated.py + mot_sparse_structure_flow.py. Kept per spec: mm_latent.py, ss_vae.py, the Euler solver, timestep_embedder.py, stage1_flow_lora training utils, tools/build_canonical_latents.py, canonical.py::voxelize_canonical.

2026-09-02 — DINO seed-path memory-format fix + in-training appearance-forcing validation

  • MultiViewSlatConditionEmbedder.dino_prenorm now runs DINO per view on an NCHW-contiguous tensor. Root cause found by the step-0 parity harness: the dataset builds crops with np.stack over transposed HWC arrays, which yields channels-last strides; DINOv2's patch-embed conv then selects different cuDNN kernels (~1% token deviation vs the reference batch_appforce_sam3d.dino_features, max|d| 1.5–3.4 on a 185 scale), and a V-view batched forward adds ~7e-4. Per-view + .contiguous() is bit-exact against the reference dumps. This affects the training x0 seed and the non-anchor condition tokens identically (both come from dino_prenorm).
  • In-training validation: mvsam3d/train/val_appforce.py (trained stage-2 on cached reference stage-1 coords/base; seed + reinject schedules; per-rank decode + evaluate_appforce.py in the reference env via tools/val_stage1_ref.py; rank-0 merge -> val_af/<ds>/<schedule>/<metric> in wandb), hooked into train_slat_flow.py (val_appforce: config block, step 0 + every 5000). Parity harness: tools/parity_val_appforce.sh.

2026-09-02 — TRUE multi-object BATCHED stage-2 (SLAT flow) TRAINING

Stage-2 training was bs=1-per-forward (dense [1,N,8] over one coord set); the effective batch came only from train.grad_accum_steps. Now the verified batched-INFERENCE layout (sample_batched, tools/selfcheck_batched_slat.py) is reused for TRAINING, so an effective batch of 12 = 4 GPUs x train.batch_objects=3 with no gradient accumulation.

  • mvsam3d/model/slatflow_model.py
    • forward dispatches on the presence of a per-object layout key (collate_batched batch -> batched step, collate_single -> legacy step).
    • _object_train_pieces — per-object cond (fuser anchor + non-anchor DINO tokens + Plücker) / x0 forced-appearance seed with fresh noise on the invisible rows / normalized official x1 — the SAME construction flow_step performs.
    • flow_step_batched — B objects concatenated into ONE sparse coord set with the batch-index column (TRELLIS SLat.collate_fn), conditions padded to a common L and block-diagonally masked via CrossAttnBiasHolder.set_batch (pad columns -inf; each object's soft mask on its own rows/columns), ONE gen.reverse_fn call. t is per object ([B], one gen._generate_t draw each): the backbone's t_embedder(t) yields [B,C] which sparse_batch_broadcast expands per object over the sparse layout — so no extra plumbing is needed for the timestep. Because the dense tensor is [1,T_total,8] (no batch axis) the interpolant is applied with a PER-TOKEN t (t expanded over each layout slice) using the generator's own equations and gen.sigma_min: x_t=(1-(1-σ)t)x0+t·x1, v=x1-(1-σ)x0. Loss = mean over objects of each object's own per-token MSE (per-object mean, NOT token-count weighted) so the objective is identical to averaging B bs=1 losses. DDP guard: plucker_fc is kept in the graph (×0.0) when NO object in the batch has a non-anchor view.
  • mvsam3d/train/train_slat_flow.py — new train.batch_objects (default 1 = previous behaviour). >1 switches the loader to batch_size=batch_objects + collate_batched and requires grad_accum_steps == 1 (hard error otherwise); the printed global_batch is now batch_objects × grad_accum × world_size. Validation stays bs=1 (collate_single + flow_step).
  • tools/selfcheck_batched_train.py — equivalence + memory self-check: with x0 noise and t pinned, batched loss == mean of the B single losses, and peak memory / grad finiteness for use_checkpoint off/on.

2026-09-03 — BATCHED sampling + PARALLEL decode/metrics for val_appforce

The in-training appearance-forcing validation was one object per 25-step ODE solve and ONE decode process per rank. Both are now parallel; the outputs, file layout and wandb keys (val_af/<ds>/<schedule>/<metric>) are unchanged.

  • mvsam3d/train/val_appforce.py
    • _stage2_prepare — the per-object part of stage2 up to the solve (DINO prenorm -> condition -> encode_visible_latent -> seed tensor), in the same op order as the single path. Stays per object (it is cheap); the batching target is the 25-step flow.
    • _stage2_solve_chunk — ONE batched solve over a chunk via SlatFlowModel.sample_batched (the verified contract of tools/selfcheck_batched_slat.py: coords concatenated with the batch-index column, seeds torch.cat into [1,T_total,8], conditions padded to a common L and block-diagonally masked through CrossAttnBiasHolder.set_batch). reinject batched pin: object i's visible rows live at offset_i + local_row (offset_i = Σ N of the earlier objects in the chunk = exactly the layout sample_batched builds); eps_fixed / z_forced are those same rows of the object's OWN base noise / z_norm, concatenated in the same order, so the single-path formula x[vis] = (1-(1-σ)t_next)·eps_fixed + t_next·z_forced applies verbatim to the gathered global row set after every Euler step. seed is applied per object on its own base before the concatenation.
    • stage2_batched(model, items, bases, schedule, batch_size=6) — chunks the object list (sorted by voxel count so a chunk's padding waste is small) and returns one stage2-shaped result dict per input item, in input order.
    • sample_jobs(..., batch_size=6, single=False) — batched by default; single=True / VAL_AF_SINGLE=1 / batch_size<=1 restore the original path (_sample_jobs_single, kept verbatim) for A/B. A batched failure (e.g. OOM on a chunk of large objects) falls back to the single path for that chunk instead of losing the objects.
    • decode_and_eval_rank(..., decode_procs=2, threads=8) — decode_procs reference-env decode processes per rank run CONCURRENTLY, sharded with the decode mode's own --shard i --nshards n (disjoint <obj>.npz, disjoint GLB names, so the shared glb/ dir and the per-rank hardlink dir are race-free); the 4 evaluate_appforce.py processes stay concurrent. _ref_env now sets OMP/MKL/OpenBLAS/NumExpr thread caps EXPLICITLY (the trainer already exports OMP_NUM_THREADS, so setdefault was a no-op): 4 ranks x 4 concurrent eval procs x 8 threads = 128 of ~255 cores.
  • config val_appforce: — new batch_size (6), single (false), decode_procs (2), ref_threads (8) in configs/train_slatflow.yaml, configs/train_slatflow_prod.yaml, configs/_smoke_val_e2e.yaml.
  • tools/verify_batched_val.py — offline verification against the SINGLE-path step-5000 dumps: reloads the step-5000 ckpt + EMA sidecar, reruns the single path (the run floor), then the batched path at B ∈ {1,4,6,8} for both schedules, reporting per object max|d| / mean|d| vs the reference and vs the single re-run, coords equality, and s/object timing. PASS iff max|batched-ref| <= max(3·max|single_rerun-ref|, 0.05·feat_scale). --parallel-eval-check additionally runs the new parallel decode+eval on those objects into a temp dir and diffs the results.json metrics against the step-5000 eval_rank*/results.json rows.