jhoonjwa's picture
Add slat_dev snapshot 2026-11-11: stage-2 code with test-time soft-mask bias on/off (BIAS_INFERENCE.md)
ffee162 verified
|
Raw History Blame Contribute Delete
20.6 kB
# mv-sam3d-for-6d-v2 — SSFlow implementation (corrected design)
Seed-only **partial → full sparse-structure rectified flow** in SAM-3D's SS
latent space (branch `ssflow`; spec: `../../../PLAN/PLAN.md` +
`../../../PLAN/DATA_FORMAT.md`, with the USER-CORRECTED model design below
superseding the plan's "strip cross-attn/condition" step).
> **See `README_MVSAM3D_TRAINING.md`** for the current top-level doc: the full
> module map (including the stage-2 **SlatFlowModel** / `train_slat_flow.py`
> stack, spec `../../../PLAN/SLAT_GEN_PLAN.md`, which this file predates),
> feature→file:line index, run commands, config-key lists, the measured
> batch-size/ETA guidance, and the verified smoke evidence for BOTH training
> loops. Stage-2-only features (the `enable_soft_mask`/`enable_plucker`
> split, the `data.max_num_voxels` filter, TRUE batched multi-object
> inference) are documented there. This file remains the SSFlow design record.
## Corrected design (supersedes commit ecf48ee's backbone stripping)
**The stage-1 flow transformer is ARCHITECTURALLY IDENTICAL to the pretrained
SAM-3D `ss_generator`** (`SparseStructureFlowTdfyWrapper` exactly as built by
ss_generator.yaml). The earlier pass wrongly removed cross-attention, its
pre-norm (norm2) and the condition pathway — that is REVERTED
(`modulated.py` + `mot_sparse_structure_flow.py` restored pristine, and the
DINO/pointmap/fuser embedders under `sam3d_objects/model/backbone/dit/`
restored).
* **KEPT** (warm-started from ss_generator.ckpt): the shape stream, every
block's self-attn + **cross_attn (+ norm2)** + FFN + adaLN, `t_embedder`,
the shortcut `d_embedder` (present but unused: `d=None`; frozen so DDP has
no unused trainable params), and the **external condition embedder**
(DINO-L/14-reg ×2 + PointPatchEmbed `EmbedderFuser`, frozen — exactly
ss_generator.yaml's `module.condition_embedder.backbone`).
* **DROPPED**: only the 6D-POSE latent streams
(`6drotation_normalized`/`translation`/`scale`/`translation_scale`) and
their targets/metrics — mv-sam3d-for-6d ADD-ONS, not part of the original
SS generator. `latent_mapping` = `{"shape"}`, `latent_share_transformer={}`.
The single-stream MOT self-attention guard
(`if other_modalities:` in `mm_scale_dot_product_attention`; `mode !=
"default"` raises) stays — with one stream Path-2 would `torch.cat([])`.
* **The ONLY training change vs the pretrained generator**: x0 = `z1_partial`
(ss_encoder of the partial occupancy) instead of Gaussian noise:
```
x0 = z_partial = ss_encoder(PARTIAL 64^3 occupancy from backprojected multi-view
depth; canonical [-0.5,0.5]^3; visible-only, noise='zeros')
x1 = z_full = ss_encoder(FULL 64^3 surface occupancy of the mesh)
z_t = (1-t)·x0 + t·x1 (sigma_min = 0)
target v = x1 - x0, loss = MSE(v_pred, v) # v-prediction
```
This is EXACTLY the seed-only inference forcing (batch_appforce z1fwd), now
trained. Latents are RAW posterior means — no mean/std normalization anywhere.
* **CONDITIONING**: the pathway is kept. Geometry-only runs (no renders yet)
pass a NULL/zero context `[B,1,1024]` — through qkv-bias cross-attention an
all-zero context is token-count-invariant, so this equals the pretrained
`force_zeros_cond` classifier-free branch (`cond * 0`). Real image/pointmap
conditioning goes through `SSFlowModel.encode_condition(**modalities)`
(kwargs `image`/`rgb_image`/`mask`/`rgb_image_mask`/`pointmap`/
`rgb_pointmap`, per ss_generator.yaml) once the **trellis500k renders**
land — the render→condition DATA wiring is deliberately NOT built yet.
## Model (`mvsam3d/model/ssflow_model.py`, class `SSFlowModel`)
* Backbone kwargs mirror ss_generator.yaml minus the pose latents:
`cond_channels=1024`, `condition_embedder=None` INSIDE the wrapper
(identity — the EmbedderFuser lives OUTSIDE, its tokens are the positional
`cond` arg, exactly as pretrained), `force_zeros_cond=True`,
`is_shortcut_model=True`, 24 blocks, 16 heads, mlp_ratio 4, ape,
`qk_rms_norm=True`, `qk_rms_norm_cross=False`, time_scale 1000;
`freeze_shared_parameters=False` (finetune everything trainable).
* `forward(batch)` = `flow_step`: `batch["z_partial"]`, `batch["z_full"]`
`[B,4096,8]`; optional `batch["cond"]` `[B,N,1024]` (absent → null/zero
condition); t ~ logit-normal(−1, 1) per sample (Q3, pretraining-matched;
`uniform` = the data→data ablation); returns `{"loss", "loss/shape"}`.
* `sample(z_partial, num_steps=25, cond=None)` — SAM-3D-EXACT seeded inference
(commit 0f5342f: the hand-rolled Euler loop was replaced by a thin wrapper,
verified bit-identical before the swap): z0 = partial latent VERBATIM
injected via the `seeded_noise` contextmanager into the REAL `FlowMatching`
sampler (`generate → generate_iter → _prepare_t → Euler solve_iter`), with
the deployed SS inference config (25 steps, `rescale_t=3` t-warp, CFG
strength 7 on interval [0,500] — a mathematical no-op while the condition is
null — fp16 autocast, d=0 no_shortcut); decode chain identical to
`inference_pipeline.py:812-833`: permute/view → standalone `ss_decoder`
(NEVER the TRELLIS-identical decoder inside ss_generator.ckpt) → logits
`[B,1,64,64,64]` → `coords_original = argwhere(ss>0)[:,[0,2,3,4]].int()` →
**`coords` = `downsample_sparse_structure(prune_sparse_structure(·,
dist=1))` — the PRIMARY output (SAM-3D pipeline contract: pruned surface
shell + int32-guard downsample). Use `coords` for IoU eval and downstream;
`coords_original` is debug-only.**
* Warm-start (`load_pretrained_backbone`) — **~100% of the model's tensors**:
* backbone ← `_base_models.generator.reverse_fn.backbone.*`:
**493/493** own tensors (0 left at init, 0 mismatched), incl. all
`cross_attn.shape.*`, `norm2.shape.*`, `d_embedder.*`; the ckpt's other
452 tensors are the dropped 6D-pose-stream weights.
* condition_embedder ← `_base_models.condition_embedder.*`: **722/722**
(exact bijection with the ckpt's prefix).
* Self-check (2026-08-31): flow_step loss+backward with null cond (cross-attn
gradients flow); zero-context length invariance ≤ 2e-5; `sample()` returns
int32 `(N,4)` coords ⊆ coords_original.
## SS VAE (`mvsam3d/model/ss_vae.py`)
Unchanged loaders (strict, 0 missing/unexpected) + helpers:
`ss_encode(occ) → [B,8,16,16,16]` (frozen cached encoder, no_grad — the
train-step partial encode) and `grid_latent_to_tokens` (→ `[B,4096,8]`).
Checkpoints resolve via `mvsam3d/paths.py` (`MIGRATOR_CHECKPOINTS/hf`);
`migrator/checkpoints/hf` is a symlink to the downloaded SAM-3D set (the
working ss_encoder.ckpt, 119 MB — the /data hf copy is 0-byte, do not use).
## Data
* On-disk: DATA_FORMAT.md layout
`<root>/<SOURCE>/objects/<id[:2]>/<id>/{meta.json, geometry/voxels.npz,
geometry/ss_latent_sam3d.npz, views/views.npz}`; `meta.json` written LAST is
the completion marker (idempotent re-runs).
* `mvsam3d/data/ssflow_dataset.py` — `TrellisSSFlowDataset`:
* OWN views.npz loader: per-view `fx=K[i,0,0], fy=K[i,1,1], cx=K[i,0,2],
cy=K[i,1,2]`, depth `uint16 mm / 1000`, OpenCV c2w. Res-agnostic (Q5:
future trellis500k renders at 512; existing toys4k=768 / omni3d=800).
* Per `__getitem__` (this IS the augmentation): random view subset
**|S| ~ U{1..8} (Q4), clamped per object to its view count** → partial
occupancy by the EXACT batch_appforce recipe (depth>0 mask,
`binary_erosion(iterations=2)`, pinhole backproject, keep |xyz| ≤ 0.512,
clip to [-0.5, 0.5−1e−9], voxelize 64, union, visible-only =
noise='zeros') → returns `occ_partial [1,64,64,64]` + `z_full [4096,8]`
+ `occ_full` (for IoU). `exclude_names` drops given object ids.
* `z_partial` is encoded in the TRAIN STEP by the frozen `ss_encode`, never
precomputed. Deterministic per-(seed, epoch, idx) RNG.
* **Val set (Q7)**: `data.val_roots` = explicit validation roots — 100
held-out toys4k objects (`smoke_data/val_toys4k100`, disjoint from
toys4k200 via `--exclude_ids_from_root`) + all local omni3d samples
(`smoke_data/val_omni3d`, front+side renders from
`/lp-dev/jonghoon/mv-mesh/exp_faithfulness/omni3d/renders`). When set, val
= ALL objects under val_roots and train = roots MINUS val ids; empty →
legacy hash split.
* `tools/preprocess_ssflow_toys4k.py` — DATA_FORMAT adapter (toys4k AND
omni3d: auto-detects available `front/side/oside/back` views; `--source`,
`--exclude_ids_from_root`). Gates: (a) full-occ encoder/decoder round-trip
IoU ≥ ~0.9; (b) partial ⊂ full containment.
* `tools/gen_x1_official.py` + `slat_official/` — the ONLY producer of the
SLAT training target x1 (USER MANDATE: exact official algorithm, no
reimplementation): vendored byte-identical TRELLIS dataset_toolkits
(Blender 150-view render → open3d 64³ voxelize → all-voxel no-occlusion
DINO mean → `encoder(feats, sample_posterior=False)`) with the single
mandated deviation of SAM3D's `slat_encoder.ckpt`, emitting
`slat/slat_official_sam3d.npz` (RAW SAM3D-space latent on ALL active
voxels). The former on-the-fly visible-mean x1 was deleted;
`encode_visible_latent` survives only as the inference-time forced x0 seed.
## Training (`mvsam3d/train/train_ss_encoder.py` + `configs/train_ssflow.yaml`)
Rectified-flow v-pred loop: AdamW (trainable = backbone minus d_embedder;
condition embedder + d_embedder frozen; `optim.weight_decay` config-driven,
default 0.0), bf16 autocast, warmup→cosine, grad-accum, DDP via the vendored
`stage1_flow_lora.{distributed,checkpointing,train_utils}` (single-GPU
transparently skips the process group). Per step: move batch → `z_partial =
grid_latent_to_tokens(ss_encode(occ_partial))` (fp32, no_grad, outside
autocast) → `model({"z_partial", "z_full"})` (null cond) → backward/clip/step
(`train_ss_encoder.py:304-308`).
Two optional, config-driven, default-OFF additions (shared with the SlatFlow
loop):
* **EMA** (`ema.{enable,rate,eval_with_ema}`, default `enable: false`) —
`mvsam3d/train/ema.py` `EmaShadow`: fp32 shadow of ONLY the trainable
(`requires_grad`) params, `ema = rate*ema + (1-rate)*param` after every
`optimizer.step()` (`train_ss_encoder.py:336-337`); with `eval_with_ema`
the in-loop validation runs under the EMA weights (bit-exact restore after,
`:354-357`). Persisted as a `step_XXXX_ema.pt` sidecar next to the
`_optimizer.pt` sidecar (the vendored `checkpointing.py` is untouched) and
auto-restored on `resume_from` (`:260-268`, `:380-381`).
* **Gradient checkpointing** (`model.use_checkpoint`, default false) —
threaded through the `SSFlowModel` ctor (`ssflow_model.py:145`) to the
backbone's native per-block `use_checkpoint` (`:207`): activations of the
24 DiT blocks are recomputed in backward for a large peak-memory cut at
~modest step-time cost.
NOTE: checkpoints include the frozen condition-embedder tensors (~1.2 GB fp32
extra) — keep them under /lp-dev.
```
cd <repo>
export MIGRATOR_CHECKPOINTS=$PWD/migrator/checkpoints
PYTHONPATH=. python mvsam3d/train/train_ss_encoder.py \
--config configs/train_ssflow.yaml --set train.steps=2000
# multi-GPU: torchrun --nproc_per_node=N ... (GPUs 0,1,4,5,6,7 only on this box)
```
## Validation (`mvsam3d/train/validate.py`)
Q7 val set (val_roots), per pass (fixed seed, training RNG saved/restored):
`decoded_iou` — seeded `sample()` → **`coords` (the PRIMARY pruned+downsampled
pipeline output) scattered to 64³** vs the full surface-occupancy grid;
`partial_iou` (input-copy floor: decoded > partial ⇒ the model completes, not
copies); held-out flow `loss`/`loss/shape`.
## Smoke plan (PLAN.md §6; runs = Smoke phase)
1. toys4k200 preprocess + gates — DONE; val set (100 toys4k + omni3d) — DONE.
2. Unit self-check — DONE (warm-start 493/493 + 722/722, null-cond forward,
coords-primary sample).
3. Overfit: 8 objects, batch 8, lr 1e-4, bf16, warm, null cond — DONE
(2026-09-01, `.debug/train_ssflow_overfit8_e2e.log`): train v-MSE
0.0953 → 0.0001 over 600 steps (~950×); held-out decoded_iou 0.6441 >
partial_iou floor 0.5483 (n=16). PASS.
4. Generalization sniff: toys4k200 train vs val_roots; val v-MSE down, val
decoded IoU > partial floor.
5. Then point `data.roots` at the trellis500k subset (Q8, download-agent
scope) as it lands; wire render conditioning through `encode_condition`.
## Removed (dead code, PLAN.md §4) — updated
`mvsam3d/model/{condition,flow_model}.py`, `mvsam3d/data/{dexycb_dataset,
adapters,base_dataset,integrated,dexycb_calib}.py`, `mvsam3d/infer/`,
`mvsam3d/train/train.py`, `sam3d_objects/geometry/` (pose-only rot6d/ssi),
verify/diagnose/smoke tools, `build_canonical_latents_multi.py`, DexYCB/
integrated configs + `train.yaml`, stale docs, all stale sbatch scripts.
RESTORED (corrected design): `sam3d_objects/model/backbone/dit/` embedders,
pristine `modulated.py` + `mot_sparse_structure_flow.py`. Kept per spec:
`mm_latent.py`, `ss_vae.py`, the Euler solver, `timestep_embedder.py`,
`stage1_flow_lora` training utils, `tools/build_canonical_latents.py`,
`canonical.py::voxelize_canonical`.
## 2026-09-02 — DINO seed-path memory-format fix + in-training appearance-forcing validation
* `MultiViewSlatConditionEmbedder.dino_prenorm` now runs DINO **per view on an
NCHW-contiguous tensor**. Root cause found by the step-0 parity harness:
the dataset builds `crops` with `np.stack` over transposed HWC arrays, which
yields channels-last strides; DINOv2's patch-embed conv then selects
different cuDNN kernels (~1% token deviation vs the reference
`batch_appforce_sam3d.dino_features`, max|d| 1.5–3.4 on a 185 scale), and a
V-view batched forward adds ~7e-4. Per-view + `.contiguous()` is bit-exact
against the reference dumps. This affects the training x0 seed and the
non-anchor condition tokens identically (both come from `dino_prenorm`).
* In-training validation: `mvsam3d/train/val_appforce.py` (trained stage-2 on
cached reference stage-1 coords/base; seed + reinject schedules; per-rank
decode + `evaluate_appforce.py` in the reference env via
`tools/val_stage1_ref.py`; rank-0 merge -> `val_af/<ds>/<schedule>/<metric>`
in wandb), hooked into `train_slat_flow.py` (`val_appforce:` config block,
step 0 + every 5000). Parity harness: `tools/parity_val_appforce.sh`.
## 2026-09-02 — TRUE multi-object BATCHED stage-2 (SLAT flow) TRAINING
Stage-2 training was bs=1-per-forward (dense `[1,N,8]` over one coord set);
the effective batch came only from `train.grad_accum_steps`. Now the verified
batched-INFERENCE layout (`sample_batched`, `tools/selfcheck_batched_slat.py`)
is reused for TRAINING, so an effective batch of 12 = 4 GPUs x
`train.batch_objects=3` with **no gradient accumulation**.
* `mvsam3d/model/slatflow_model.py`
* `forward` dispatches on the presence of a per-object `layout` key
(`collate_batched` batch -> batched step, `collate_single` -> legacy step).
* `_object_train_pieces` — per-object cond (fuser anchor + non-anchor DINO
tokens + Plücker) / x0 forced-appearance seed with fresh noise on the
invisible rows / normalized official x1 — the SAME construction
`flow_step` performs.
* `flow_step_batched` — B objects concatenated into ONE sparse coord set
with the batch-index column (TRELLIS `SLat.collate_fn`), conditions padded
to a common L and block-diagonally masked via
`CrossAttnBiasHolder.set_batch` (pad columns -inf; each object's soft mask
on its own rows/columns), ONE `gen.reverse_fn` call. **t is per object**
(`[B]`, one `gen._generate_t` draw each): the backbone's `t_embedder(t)`
yields `[B,C]` which `sparse_batch_broadcast` expands per object over the
sparse layout — so no extra plumbing is needed for the timestep. Because
the dense tensor is `[1,T_total,8]` (no batch axis) the interpolant is
applied with a PER-TOKEN t (t expanded over each layout slice) using the
generator's own equations and `gen.sigma_min`:
`x_t=(1-(1-σ)t)x0+t·x1`, `v=x1-(1-σ)x0`.
Loss = **mean over objects of each object's own per-token MSE**
(per-object mean, NOT token-count weighted) so the objective is identical
to averaging B bs=1 losses. DDP guard: `plucker_fc` is kept in the graph
(×0.0) when NO object in the batch has a non-anchor view.
* `mvsam3d/train/train_slat_flow.py` — new `train.batch_objects` (default 1 =
previous behaviour). >1 switches the loader to `batch_size=batch_objects` +
`collate_batched` and requires `grad_accum_steps == 1` (hard error
otherwise); the printed `global_batch` is now
`batch_objects × grad_accum × world_size`. Validation stays bs=1
(`collate_single` + `flow_step`).
* `tools/selfcheck_batched_train.py` — equivalence + memory self-check:
with x0 noise and t pinned, batched loss == mean of the B single losses, and
peak memory / grad finiteness for `use_checkpoint` off/on.
## 2026-09-03 — BATCHED sampling + PARALLEL decode/metrics for `val_appforce`
The in-training appearance-forcing validation was one object per 25-step ODE
solve and ONE decode process per rank. Both are now parallel; the outputs,
file layout and wandb keys (`val_af/<ds>/<schedule>/<metric>`) are unchanged.
* `mvsam3d/train/val_appforce.py`
* `_stage2_prepare` — the per-object part of `stage2` up to the solve (DINO
prenorm -> condition -> `encode_visible_latent` -> seed tensor), in the
same op order as the single path. Stays per object (it is cheap); the
batching target is the 25-step flow.
* `_stage2_solve_chunk` — ONE batched solve over a chunk via
`SlatFlowModel.sample_batched` (the verified contract of
`tools/selfcheck_batched_slat.py`: coords concatenated with the batch-index
column, seeds `torch.cat` into `[1,T_total,8]`, conditions padded to a
common L and block-diagonally masked through
`CrossAttnBiasHolder.set_batch`). **`reinject` batched pin:** object *i*'s
visible rows live at `offset_i + local_row` (offset_i = Σ N of the earlier
objects in the chunk = exactly the layout `sample_batched` builds);
`eps_fixed` / `z_forced` are those same rows of the object's OWN base noise
/ `z_norm`, concatenated in the same order, so the single-path formula
`x[vis] = (1-(1-σ)t_next)·eps_fixed + t_next·z_forced` applies verbatim to
the gathered global row set after every Euler step. `seed` is applied
per object on its own base before the concatenation.
* `stage2_batched(model, items, bases, schedule, batch_size=6)` — chunks the
object list (sorted by voxel count so a chunk's padding waste is small) and
returns one `stage2`-shaped result dict per input item, in input order.
* `sample_jobs(..., batch_size=6, single=False)` — batched by default;
`single=True` / `VAL_AF_SINGLE=1` / `batch_size<=1` restore the original
path (`_sample_jobs_single`, kept verbatim) for A/B. A batched failure
(e.g. OOM on a chunk of large objects) falls back to the single path for
that chunk instead of losing the objects.
* `decode_and_eval_rank(..., decode_procs=2, threads=8)` — `decode_procs`
reference-env decode processes per rank run CONCURRENTLY, sharded with the
decode mode's own `--shard i --nshards n` (disjoint `<obj>.npz`, disjoint
GLB names, so the shared `glb/` dir and the per-rank hardlink dir are
race-free); the 4 `evaluate_appforce.py` processes stay concurrent.
`_ref_env` now sets OMP/MKL/OpenBLAS/NumExpr thread caps EXPLICITLY (the
trainer already exports `OMP_NUM_THREADS`, so `setdefault` was a no-op):
4 ranks x 4 concurrent eval procs x 8 threads = 128 of ~255 cores.
* config `val_appforce:` — new `batch_size` (6), `single` (false),
`decode_procs` (2), `ref_threads` (8) in `configs/train_slatflow.yaml`,
`configs/train_slatflow_prod.yaml`, `configs/_smoke_val_e2e.yaml`.
* `tools/verify_batched_val.py` — offline verification against the SINGLE-path
step-5000 dumps: reloads the step-5000 ckpt + EMA sidecar, reruns the single
path (the run floor), then the batched path at B ∈ {1,4,6,8} for both
schedules, reporting per object max|d| / mean|d| vs the reference and vs the
single re-run, coords equality, and s/object timing. PASS iff
`max|batched-ref| <= max(3·max|single_rerun-ref|, 0.05·feat_scale)`.
`--parallel-eval-check` additionally runs the new parallel decode+eval on
those objects into a temp dir and diffs the `results.json` metrics against
the step-5000 `eval_rank*/results.json` rows.