Download migrator/code/slat_dev_20261111/IMPLEMENTATION.md from jhoonjwa/bert_simpson: direct link, hf CLI and curl.
- Browser
- Download file 20.6 kB
-
https://huggingface.co/jhoonjwa/bert_simpson/resolve/main/migrator/code/slat_dev_20261111/IMPLEMENTATION.md
- Command line
-
hf download hf://jhoonjwa/bert_simpson/migrator/code/slat_dev_20261111/IMPLEMENTATION.md
-
curl -L -o IMPLEMENTATION.md https://huggingface.co/jhoonjwa/bert_simpson/resolve/main/migrator/code/slat_dev_20261111/IMPLEMENTATION.md
mv-sam3d-for-6d-v2 — SSFlow implementation (corrected design)
Seed-only partial → full sparse-structure rectified flow in SAM-3D's SS
latent space (branch ssflow; spec: ../../../PLAN/PLAN.md +
../../../PLAN/DATA_FORMAT.md, with the USER-CORRECTED model design below
superseding the plan's "strip cross-attn/condition" step).
See
README_MVSAM3D_TRAINING.mdfor the current top-level doc: the full module map (including the stage-2 SlatFlowModel /train_slat_flow.pystack, spec../../../PLAN/SLAT_GEN_PLAN.md, which this file predates), feature→file:line index, run commands, config-key lists, the measured batch-size/ETA guidance, and the verified smoke evidence for BOTH training loops. Stage-2-only features (theenable_soft_mask/enable_pluckersplit, thedata.max_num_voxelsfilter, TRUE batched multi-object inference) are documented there. This file remains the SSFlow design record.
Corrected design (supersedes commit ecf48ee's backbone stripping)
The stage-1 flow transformer is ARCHITECTURALLY IDENTICAL to the pretrained
SAM-3D ss_generator (SparseStructureFlowTdfyWrapper exactly as built by
ss_generator.yaml). The earlier pass wrongly removed cross-attention, its
pre-norm (norm2) and the condition pathway — that is REVERTED
(modulated.py + mot_sparse_structure_flow.py restored pristine, and the
DINO/pointmap/fuser embedders under sam3d_objects/model/backbone/dit/
restored).
- KEPT (warm-started from ss_generator.ckpt): the shape stream, every
block's self-attn + cross_attn (+ norm2) + FFN + adaLN,
t_embedder, the shortcutd_embedder(present but unused:d=None; frozen so DDP has no unused trainable params), and the external condition embedder (DINO-L/14-reg ×2 + PointPatchEmbedEmbedderFuser, frozen — exactly ss_generator.yaml'smodule.condition_embedder.backbone). - DROPPED: only the 6D-POSE latent streams
(
6drotation_normalized/translation/scale/translation_scale) and their targets/metrics — mv-sam3d-for-6d ADD-ONS, not part of the original SS generator.latent_mapping={"shape"},latent_share_transformer={}. The single-stream MOT self-attention guard (if other_modalities:inmm_scale_dot_product_attention;mode != "default"raises) stays — with one stream Path-2 wouldtorch.cat([]). - The ONLY training change vs the pretrained generator: x0 =
z1_partial(ss_encoder of the partial occupancy) instead of Gaussian noise:
x0 = z_partial = ss_encoder(PARTIAL 64^3 occupancy from backprojected multi-view
depth; canonical [-0.5,0.5]^3; visible-only, noise='zeros')
x1 = z_full = ss_encoder(FULL 64^3 surface occupancy of the mesh)
z_t = (1-t)·x0 + t·x1 (sigma_min = 0)
target v = x1 - x0, loss = MSE(v_pred, v) # v-prediction
This is EXACTLY the seed-only inference forcing (batch_appforce z1fwd), now trained. Latents are RAW posterior means — no mean/std normalization anywhere.
- CONDITIONING: the pathway is kept. Geometry-only runs (no renders yet)
pass a NULL/zero context
[B,1,1024]— through qkv-bias cross-attention an all-zero context is token-count-invariant, so this equals the pretrainedforce_zeros_condclassifier-free branch (cond * 0). Real image/pointmap conditioning goes throughSSFlowModel.encode_condition(**modalities)(kwargsimage/rgb_image/mask/rgb_image_mask/pointmap/rgb_pointmap, per ss_generator.yaml) once the trellis500k renders land — the render→condition DATA wiring is deliberately NOT built yet.
Model (mvsam3d/model/ssflow_model.py, class SSFlowModel)
- Backbone kwargs mirror ss_generator.yaml minus the pose latents:
cond_channels=1024,condition_embedder=NoneINSIDE the wrapper (identity — the EmbedderFuser lives OUTSIDE, its tokens are the positionalcondarg, exactly as pretrained),force_zeros_cond=True,is_shortcut_model=True, 24 blocks, 16 heads, mlp_ratio 4, ape,qk_rms_norm=True,qk_rms_norm_cross=False, time_scale 1000;freeze_shared_parameters=False(finetune everything trainable). forward(batch)=flow_step:batch["z_partial"],batch["z_full"][B,4096,8]; optionalbatch["cond"][B,N,1024](absent → null/zero condition); t ~ logit-normal(−1, 1) per sample (Q3, pretraining-matched;uniform= the data→data ablation); returns{"loss", "loss/shape"}.sample(z_partial, num_steps=25, cond=None)— SAM-3D-EXACT seeded inference (commit 0f5342f: the hand-rolled Euler loop was replaced by a thin wrapper, verified bit-identical before the swap): z0 = partial latent VERBATIM injected via theseeded_noisecontextmanager into the REALFlowMatchingsampler (generate → generate_iter → _prepare_t → Euler solve_iter), with the deployed SS inference config (25 steps,rescale_t=3t-warp, CFG strength 7 on interval [0,500] — a mathematical no-op while the condition is null — fp16 autocast, d=0 no_shortcut); decode chain identical toinference_pipeline.py:812-833: permute/view → standaloness_decoder(NEVER the TRELLIS-identical decoder inside ss_generator.ckpt) → logits[B,1,64,64,64]→coords_original = argwhere(ss>0)[:,[0,2,3,4]].int()→coords=downsample_sparse_structure(prune_sparse_structure(·, dist=1))— the PRIMARY output (SAM-3D pipeline contract: pruned surface shell + int32-guard downsample). Usecoordsfor IoU eval and downstream;coords_originalis debug-only.- Warm-start (
load_pretrained_backbone) — ~100% of the model's tensors:- backbone ←
_base_models.generator.reverse_fn.backbone.*: 493/493 own tensors (0 left at init, 0 mismatched), incl. allcross_attn.shape.*,norm2.shape.*,d_embedder.*; the ckpt's other 452 tensors are the dropped 6D-pose-stream weights. - condition_embedder ←
_base_models.condition_embedder.*: 722/722 (exact bijection with the ckpt's prefix).
- backbone ←
- Self-check (2026-08-31): flow_step loss+backward with null cond (cross-attn
gradients flow); zero-context length invariance ≤ 2e-5;
sample()returns int32(N,4)coords ⊆ coords_original.
SS VAE (mvsam3d/model/ss_vae.py)
Unchanged loaders (strict, 0 missing/unexpected) + helpers:
ss_encode(occ) → [B,8,16,16,16] (frozen cached encoder, no_grad — the
train-step partial encode) and grid_latent_to_tokens (→ [B,4096,8]).
Checkpoints resolve via mvsam3d/paths.py (MIGRATOR_CHECKPOINTS/hf);
migrator/checkpoints/hf is a symlink to the downloaded SAM-3D set (the
working ss_encoder.ckpt, 119 MB — the /data hf copy is 0-byte, do not use).
Data
- On-disk: DATA_FORMAT.md layout
<root>/<SOURCE>/objects/<id[:2]>/<id>/{meta.json, geometry/voxels.npz, geometry/ss_latent_sam3d.npz, views/views.npz};meta.jsonwritten LAST is the completion marker (idempotent re-runs). mvsam3d/data/ssflow_dataset.py—TrellisSSFlowDataset:- OWN views.npz loader: per-view
fx=K[i,0,0], fy=K[i,1,1], cx=K[i,0,2], cy=K[i,1,2], depthuint16 mm / 1000, OpenCV c2w. Res-agnostic (Q5: future trellis500k renders at 512; existing toys4k=768 / omni3d=800). - Per
__getitem__(this IS the augmentation): random view subset |S| ~ U{1..8} (Q4), clamped per object to its view count → partial occupancy by the EXACT batch_appforce recipe (depth>0 mask,binary_erosion(iterations=2), pinhole backproject, keep |xyz| ≤ 0.512, clip to [-0.5, 0.5−1e−9], voxelize 64, union, visible-only = noise='zeros') → returnsocc_partial [1,64,64,64]+z_full [4096,8]occ_full(for IoU).exclude_namesdrops given object ids.
z_partialis encoded in the TRAIN STEP by the frozenss_encode, never precomputed. Deterministic per-(seed, epoch, idx) RNG.
- OWN views.npz loader: per-view
- Val set (Q7):
data.val_roots= explicit validation roots — 100 held-out toys4k objects (smoke_data/val_toys4k100, disjoint from toys4k200 via--exclude_ids_from_root) + all local omni3d samples (smoke_data/val_omni3d, front+side renders from/lp-dev/jonghoon/mv-mesh/exp_faithfulness/omni3d/renders). When set, val = ALL objects under val_roots and train = roots MINUS val ids; empty → legacy hash split. tools/preprocess_ssflow_toys4k.py— DATA_FORMAT adapter (toys4k AND omni3d: auto-detects availablefront/side/oside/backviews;--source,--exclude_ids_from_root). Gates: (a) full-occ encoder/decoder round-trip IoU ≥ ~0.9; (b) partial ⊂ full containment.tools/gen_x1_official.py+slat_official/— the ONLY producer of the SLAT training target x1 (USER MANDATE: exact official algorithm, no reimplementation): vendored byte-identical TRELLIS dataset_toolkits (Blender 150-view render → open3d 64³ voxelize → all-voxel no-occlusion DINO mean →encoder(feats, sample_posterior=False)) with the single mandated deviation of SAM3D'sslat_encoder.ckpt, emittingslat/slat_official_sam3d.npz(RAW SAM3D-space latent on ALL active voxels). The former on-the-fly visible-mean x1 was deleted;encode_visible_latentsurvives only as the inference-time forced x0 seed.
Training (mvsam3d/train/train_ss_encoder.py + configs/train_ssflow.yaml)
Rectified-flow v-pred loop: AdamW (trainable = backbone minus d_embedder;
condition embedder + d_embedder frozen; optim.weight_decay config-driven,
default 0.0), bf16 autocast, warmup→cosine, grad-accum, DDP via the vendored
stage1_flow_lora.{distributed,checkpointing,train_utils} (single-GPU
transparently skips the process group). Per step: move batch → z_partial = grid_latent_to_tokens(ss_encode(occ_partial)) (fp32, no_grad, outside
autocast) → model({"z_partial", "z_full"}) (null cond) → backward/clip/step
(train_ss_encoder.py:304-308).
Two optional, config-driven, default-OFF additions (shared with the SlatFlow loop):
- EMA (
ema.{enable,rate,eval_with_ema}, defaultenable: false) —mvsam3d/train/ema.pyEmaShadow: fp32 shadow of ONLY the trainable (requires_grad) params,ema = rate*ema + (1-rate)*paramafter everyoptimizer.step()(train_ss_encoder.py:336-337); witheval_with_emathe in-loop validation runs under the EMA weights (bit-exact restore after,:354-357). Persisted as astep_XXXX_ema.ptsidecar next to the_optimizer.ptsidecar (the vendoredcheckpointing.pyis untouched) and auto-restored onresume_from(:260-268,:380-381). - Gradient checkpointing (
model.use_checkpoint, default false) — threaded through theSSFlowModelctor (ssflow_model.py:145) to the backbone's native per-blockuse_checkpoint(:207): activations of the 24 DiT blocks are recomputed in backward for a large peak-memory cut at ~modest step-time cost.
NOTE: checkpoints include the frozen condition-embedder tensors (~1.2 GB fp32 extra) — keep them under /lp-dev.
cd <repo>
export MIGRATOR_CHECKPOINTS=$PWD/migrator/checkpoints
PYTHONPATH=. python mvsam3d/train/train_ss_encoder.py \
--config configs/train_ssflow.yaml --set train.steps=2000
# multi-GPU: torchrun --nproc_per_node=N ... (GPUs 0,1,4,5,6,7 only on this box)
Validation (mvsam3d/train/validate.py)
Q7 val set (val_roots), per pass (fixed seed, training RNG saved/restored):
decoded_iou — seeded sample() → coords (the PRIMARY pruned+downsampled
pipeline output) scattered to 64³ vs the full surface-occupancy grid;
partial_iou (input-copy floor: decoded > partial ⇒ the model completes, not
copies); held-out flow loss/loss/shape.
Smoke plan (PLAN.md §6; runs = Smoke phase)
- toys4k200 preprocess + gates — DONE; val set (100 toys4k + omni3d) — DONE.
- Unit self-check — DONE (warm-start 493/493 + 722/722, null-cond forward, coords-primary sample).
- Overfit: 8 objects, batch 8, lr 1e-4, bf16, warm, null cond — DONE
(2026-09-01,
.debug/train_ssflow_overfit8_e2e.log): train v-MSE 0.0953 → 0.0001 over 600 steps (~950×); held-out decoded_iou 0.6441 > partial_iou floor 0.5483 (n=16). PASS. - Generalization sniff: toys4k200 train vs val_roots; val v-MSE down, val decoded IoU > partial floor.
- Then point
data.rootsat the trellis500k subset (Q8, download-agent scope) as it lands; wire render conditioning throughencode_condition.
Removed (dead code, PLAN.md §4) — updated
mvsam3d/model/{condition,flow_model}.py, mvsam3d/data/{dexycb_dataset, adapters,base_dataset,integrated,dexycb_calib}.py, mvsam3d/infer/,
mvsam3d/train/train.py, sam3d_objects/geometry/ (pose-only rot6d/ssi),
verify/diagnose/smoke tools, build_canonical_latents_multi.py, DexYCB/
integrated configs + train.yaml, stale docs, all stale sbatch scripts.
RESTORED (corrected design): sam3d_objects/model/backbone/dit/ embedders,
pristine modulated.py + mot_sparse_structure_flow.py. Kept per spec:
mm_latent.py, ss_vae.py, the Euler solver, timestep_embedder.py,
stage1_flow_lora training utils, tools/build_canonical_latents.py,
canonical.py::voxelize_canonical.
2026-09-02 — DINO seed-path memory-format fix + in-training appearance-forcing validation
MultiViewSlatConditionEmbedder.dino_prenormnow runs DINO per view on an NCHW-contiguous tensor. Root cause found by the step-0 parity harness: the dataset buildscropswithnp.stackover transposed HWC arrays, which yields channels-last strides; DINOv2's patch-embed conv then selects different cuDNN kernels (~1% token deviation vs the referencebatch_appforce_sam3d.dino_features, max|d| 1.5–3.4 on a 185 scale), and a V-view batched forward adds ~7e-4. Per-view +.contiguous()is bit-exact against the reference dumps. This affects the training x0 seed and the non-anchor condition tokens identically (both come fromdino_prenorm).- In-training validation:
mvsam3d/train/val_appforce.py(trained stage-2 on cached reference stage-1 coords/base; seed + reinject schedules; per-rank decode +evaluate_appforce.pyin the reference env viatools/val_stage1_ref.py; rank-0 merge ->val_af/<ds>/<schedule>/<metric>in wandb), hooked intotrain_slat_flow.py(val_appforce:config block, step 0 + every 5000). Parity harness:tools/parity_val_appforce.sh.
2026-09-02 — TRUE multi-object BATCHED stage-2 (SLAT flow) TRAINING
Stage-2 training was bs=1-per-forward (dense [1,N,8] over one coord set);
the effective batch came only from train.grad_accum_steps. Now the verified
batched-INFERENCE layout (sample_batched, tools/selfcheck_batched_slat.py)
is reused for TRAINING, so an effective batch of 12 = 4 GPUs x
train.batch_objects=3 with no gradient accumulation.
mvsam3d/model/slatflow_model.pyforwarddispatches on the presence of a per-objectlayoutkey (collate_batchedbatch -> batched step,collate_single-> legacy step)._object_train_pieces— per-object cond (fuser anchor + non-anchor DINO tokens + Plücker) / x0 forced-appearance seed with fresh noise on the invisible rows / normalized official x1 — the SAME constructionflow_stepperforms.flow_step_batched— B objects concatenated into ONE sparse coord set with the batch-index column (TRELLISSLat.collate_fn), conditions padded to a common L and block-diagonally masked viaCrossAttnBiasHolder.set_batch(pad columns -inf; each object's soft mask on its own rows/columns), ONEgen.reverse_fncall. t is per object ([B], onegen._generate_tdraw each): the backbone'st_embedder(t)yields[B,C]whichsparse_batch_broadcastexpands per object over the sparse layout — so no extra plumbing is needed for the timestep. Because the dense tensor is[1,T_total,8](no batch axis) the interpolant is applied with a PER-TOKEN t (t expanded over each layout slice) using the generator's own equations andgen.sigma_min:x_t=(1-(1-σ)t)x0+t·x1,v=x1-(1-σ)x0. Loss = mean over objects of each object's own per-token MSE (per-object mean, NOT token-count weighted) so the objective is identical to averaging B bs=1 losses. DDP guard:plucker_fcis kept in the graph (×0.0) when NO object in the batch has a non-anchor view.
mvsam3d/train/train_slat_flow.py— newtrain.batch_objects(default 1 = previous behaviour). >1 switches the loader tobatch_size=batch_objects+collate_batchedand requiresgrad_accum_steps == 1(hard error otherwise); the printedglobal_batchis nowbatch_objects × grad_accum × world_size. Validation stays bs=1 (collate_single+flow_step).tools/selfcheck_batched_train.py— equivalence + memory self-check: with x0 noise and t pinned, batched loss == mean of the B single losses, and peak memory / grad finiteness foruse_checkpointoff/on.
2026-09-03 — BATCHED sampling + PARALLEL decode/metrics for val_appforce
The in-training appearance-forcing validation was one object per 25-step ODE
solve and ONE decode process per rank. Both are now parallel; the outputs,
file layout and wandb keys (val_af/<ds>/<schedule>/<metric>) are unchanged.
mvsam3d/train/val_appforce.py_stage2_prepare— the per-object part ofstage2up to the solve (DINO prenorm -> condition ->encode_visible_latent-> seed tensor), in the same op order as the single path. Stays per object (it is cheap); the batching target is the 25-step flow._stage2_solve_chunk— ONE batched solve over a chunk viaSlatFlowModel.sample_batched(the verified contract oftools/selfcheck_batched_slat.py: coords concatenated with the batch-index column, seedstorch.catinto[1,T_total,8], conditions padded to a common L and block-diagonally masked throughCrossAttnBiasHolder.set_batch).reinjectbatched pin: object i's visible rows live atoffset_i + local_row(offset_i = Σ N of the earlier objects in the chunk = exactly the layoutsample_batchedbuilds);eps_fixed/z_forcedare those same rows of the object's OWN base noise /z_norm, concatenated in the same order, so the single-path formulax[vis] = (1-(1-σ)t_next)·eps_fixed + t_next·z_forcedapplies verbatim to the gathered global row set after every Euler step.seedis applied per object on its own base before the concatenation.stage2_batched(model, items, bases, schedule, batch_size=6)— chunks the object list (sorted by voxel count so a chunk's padding waste is small) and returns onestage2-shaped result dict per input item, in input order.sample_jobs(..., batch_size=6, single=False)— batched by default;single=True/VAL_AF_SINGLE=1/batch_size<=1restore the original path (_sample_jobs_single, kept verbatim) for A/B. A batched failure (e.g. OOM on a chunk of large objects) falls back to the single path for that chunk instead of losing the objects.decode_and_eval_rank(..., decode_procs=2, threads=8)—decode_procsreference-env decode processes per rank run CONCURRENTLY, sharded with the decode mode's own--shard i --nshards n(disjoint<obj>.npz, disjoint GLB names, so the sharedglb/dir and the per-rank hardlink dir are race-free); the 4evaluate_appforce.pyprocesses stay concurrent._ref_envnow sets OMP/MKL/OpenBLAS/NumExpr thread caps EXPLICITLY (the trainer already exportsOMP_NUM_THREADS, sosetdefaultwas a no-op): 4 ranks x 4 concurrent eval procs x 8 threads = 128 of ~255 cores.
- config
val_appforce:— newbatch_size(6),single(false),decode_procs(2),ref_threads(8) inconfigs/train_slatflow.yaml,configs/train_slatflow_prod.yaml,configs/_smoke_val_e2e.yaml. tools/verify_batched_val.py— offline verification against the SINGLE-path step-5000 dumps: reloads the step-5000 ckpt + EMA sidecar, reruns the single path (the run floor), then the batched path at B ∈ {1,4,6,8} for both schedules, reporting per object max|d| / mean|d| vs the reference and vs the single re-run, coords equality, and s/object timing. PASS iffmax|batched-ref| <= max(3·max|single_rerun-ref|, 0.05·feat_scale).--parallel-eval-checkadditionally runs the new parallel decode+eval on those objects into a temp dir and diffs theresults.jsonmetrics against the step-5000eval_rank*/results.jsonrows.