|
Download migrator/code/slat_dev_20261111/IMPLEMENTATION.md from jhoonjwa/bert_simpson: direct link, hf CLI and curl.
- Browser
- Download file 20.6 kB
-
https://huggingface.co/jhoonjwa/bert_simpson/resolve/main/migrator/code/slat_dev_20261111/IMPLEMENTATION.md
- Command line
-
hf download hf://jhoonjwa/bert_simpson/migrator/code/slat_dev_20261111/IMPLEMENTATION.md
-
curl -L -o IMPLEMENTATION.md https://huggingface.co/jhoonjwa/bert_simpson/resolve/main/migrator/code/slat_dev_20261111/IMPLEMENTATION.md
20.6 kB
| # mv-sam3d-for-6d-v2 — SSFlow implementation (corrected design) | |
| Seed-only **partial → full sparse-structure rectified flow** in SAM-3D's SS | |
| latent space (branch `ssflow`; spec: `../../../PLAN/PLAN.md` + | |
| `../../../PLAN/DATA_FORMAT.md`, with the USER-CORRECTED model design below | |
| superseding the plan's "strip cross-attn/condition" step). | |
| > **See `README_MVSAM3D_TRAINING.md`** for the current top-level doc: the full | |
| > module map (including the stage-2 **SlatFlowModel** / `train_slat_flow.py` | |
| > stack, spec `../../../PLAN/SLAT_GEN_PLAN.md`, which this file predates), | |
| > feature→file:line index, run commands, config-key lists, the measured | |
| > batch-size/ETA guidance, and the verified smoke evidence for BOTH training | |
| > loops. Stage-2-only features (the `enable_soft_mask`/`enable_plucker` | |
| > split, the `data.max_num_voxels` filter, TRUE batched multi-object | |
| > inference) are documented there. This file remains the SSFlow design record. | |
| ## Corrected design (supersedes commit ecf48ee's backbone stripping) | |
| **The stage-1 flow transformer is ARCHITECTURALLY IDENTICAL to the pretrained | |
| SAM-3D `ss_generator`** (`SparseStructureFlowTdfyWrapper` exactly as built by | |
| ss_generator.yaml). The earlier pass wrongly removed cross-attention, its | |
| pre-norm (norm2) and the condition pathway — that is REVERTED | |
| (`modulated.py` + `mot_sparse_structure_flow.py` restored pristine, and the | |
| DINO/pointmap/fuser embedders under `sam3d_objects/model/backbone/dit/` | |
| restored). | |
| * **KEPT** (warm-started from ss_generator.ckpt): the shape stream, every | |
| block's self-attn + **cross_attn (+ norm2)** + FFN + adaLN, `t_embedder`, | |
| the shortcut `d_embedder` (present but unused: `d=None`; frozen so DDP has | |
| no unused trainable params), and the **external condition embedder** | |
| (DINO-L/14-reg ×2 + PointPatchEmbed `EmbedderFuser`, frozen — exactly | |
| ss_generator.yaml's `module.condition_embedder.backbone`). | |
| * **DROPPED**: only the 6D-POSE latent streams | |
| (`6drotation_normalized`/`translation`/`scale`/`translation_scale`) and | |
| their targets/metrics — mv-sam3d-for-6d ADD-ONS, not part of the original | |
| SS generator. `latent_mapping` = `{"shape"}`, `latent_share_transformer={}`. | |
| The single-stream MOT self-attention guard | |
| (`if other_modalities:` in `mm_scale_dot_product_attention`; `mode != | |
| "default"` raises) stays — with one stream Path-2 would `torch.cat([])`. | |
| * **The ONLY training change vs the pretrained generator**: x0 = `z1_partial` | |
| (ss_encoder of the partial occupancy) instead of Gaussian noise: | |
| ``` | |
| x0 = z_partial = ss_encoder(PARTIAL 64^3 occupancy from backprojected multi-view | |
| depth; canonical [-0.5,0.5]^3; visible-only, noise='zeros') | |
| x1 = z_full = ss_encoder(FULL 64^3 surface occupancy of the mesh) | |
| z_t = (1-t)·x0 + t·x1 (sigma_min = 0) | |
| target v = x1 - x0, loss = MSE(v_pred, v) # v-prediction | |
| ``` | |
| This is EXACTLY the seed-only inference forcing (batch_appforce z1fwd), now | |
| trained. Latents are RAW posterior means — no mean/std normalization anywhere. | |
| * **CONDITIONING**: the pathway is kept. Geometry-only runs (no renders yet) | |
| pass a NULL/zero context `[B,1,1024]` — through qkv-bias cross-attention an | |
| all-zero context is token-count-invariant, so this equals the pretrained | |
| `force_zeros_cond` classifier-free branch (`cond * 0`). Real image/pointmap | |
| conditioning goes through `SSFlowModel.encode_condition(**modalities)` | |
| (kwargs `image`/`rgb_image`/`mask`/`rgb_image_mask`/`pointmap`/ | |
| `rgb_pointmap`, per ss_generator.yaml) once the **trellis500k renders** | |
| land — the render→condition DATA wiring is deliberately NOT built yet. | |
| ## Model (`mvsam3d/model/ssflow_model.py`, class `SSFlowModel`) | |
| * Backbone kwargs mirror ss_generator.yaml minus the pose latents: | |
| `cond_channels=1024`, `condition_embedder=None` INSIDE the wrapper | |
| (identity — the EmbedderFuser lives OUTSIDE, its tokens are the positional | |
| `cond` arg, exactly as pretrained), `force_zeros_cond=True`, | |
| `is_shortcut_model=True`, 24 blocks, 16 heads, mlp_ratio 4, ape, | |
| `qk_rms_norm=True`, `qk_rms_norm_cross=False`, time_scale 1000; | |
| `freeze_shared_parameters=False` (finetune everything trainable). | |
| * `forward(batch)` = `flow_step`: `batch["z_partial"]`, `batch["z_full"]` | |
| `[B,4096,8]`; optional `batch["cond"]` `[B,N,1024]` (absent → null/zero | |
| condition); t ~ logit-normal(−1, 1) per sample (Q3, pretraining-matched; | |
| `uniform` = the data→data ablation); returns `{"loss", "loss/shape"}`. | |
| * `sample(z_partial, num_steps=25, cond=None)` — SAM-3D-EXACT seeded inference | |
| (commit 0f5342f: the hand-rolled Euler loop was replaced by a thin wrapper, | |
| verified bit-identical before the swap): z0 = partial latent VERBATIM | |
| injected via the `seeded_noise` contextmanager into the REAL `FlowMatching` | |
| sampler (`generate → generate_iter → _prepare_t → Euler solve_iter`), with | |
| the deployed SS inference config (25 steps, `rescale_t=3` t-warp, CFG | |
| strength 7 on interval [0,500] — a mathematical no-op while the condition is | |
| null — fp16 autocast, d=0 no_shortcut); decode chain identical to | |
| `inference_pipeline.py:812-833`: permute/view → standalone `ss_decoder` | |
| (NEVER the TRELLIS-identical decoder inside ss_generator.ckpt) → logits | |
| `[B,1,64,64,64]` → `coords_original = argwhere(ss>0)[:,[0,2,3,4]].int()` → | |
| **`coords` = `downsample_sparse_structure(prune_sparse_structure(·, | |
| dist=1))` — the PRIMARY output (SAM-3D pipeline contract: pruned surface | |
| shell + int32-guard downsample). Use `coords` for IoU eval and downstream; | |
| `coords_original` is debug-only.** | |
| * Warm-start (`load_pretrained_backbone`) — **~100% of the model's tensors**: | |
| * backbone ← `_base_models.generator.reverse_fn.backbone.*`: | |
| **493/493** own tensors (0 left at init, 0 mismatched), incl. all | |
| `cross_attn.shape.*`, `norm2.shape.*`, `d_embedder.*`; the ckpt's other | |
| 452 tensors are the dropped 6D-pose-stream weights. | |
| * condition_embedder ← `_base_models.condition_embedder.*`: **722/722** | |
| (exact bijection with the ckpt's prefix). | |
| * Self-check (2026-08-31): flow_step loss+backward with null cond (cross-attn | |
| gradients flow); zero-context length invariance ≤ 2e-5; `sample()` returns | |
| int32 `(N,4)` coords ⊆ coords_original. | |
| ## SS VAE (`mvsam3d/model/ss_vae.py`) | |
| Unchanged loaders (strict, 0 missing/unexpected) + helpers: | |
| `ss_encode(occ) → [B,8,16,16,16]` (frozen cached encoder, no_grad — the | |
| train-step partial encode) and `grid_latent_to_tokens` (→ `[B,4096,8]`). | |
| Checkpoints resolve via `mvsam3d/paths.py` (`MIGRATOR_CHECKPOINTS/hf`); | |
| `migrator/checkpoints/hf` is a symlink to the downloaded SAM-3D set (the | |
| working ss_encoder.ckpt, 119 MB — the /data hf copy is 0-byte, do not use). | |
| ## Data | |
| * On-disk: DATA_FORMAT.md layout | |
| `<root>/<SOURCE>/objects/<id[:2]>/<id>/{meta.json, geometry/voxels.npz, | |
| geometry/ss_latent_sam3d.npz, views/views.npz}`; `meta.json` written LAST is | |
| the completion marker (idempotent re-runs). | |
| * `mvsam3d/data/ssflow_dataset.py` — `TrellisSSFlowDataset`: | |
| * OWN views.npz loader: per-view `fx=K[i,0,0], fy=K[i,1,1], cx=K[i,0,2], | |
| cy=K[i,1,2]`, depth `uint16 mm / 1000`, OpenCV c2w. Res-agnostic (Q5: | |
| future trellis500k renders at 512; existing toys4k=768 / omni3d=800). | |
| * Per `__getitem__` (this IS the augmentation): random view subset | |
| **|S| ~ U{1..8} (Q4), clamped per object to its view count** → partial | |
| occupancy by the EXACT batch_appforce recipe (depth>0 mask, | |
| `binary_erosion(iterations=2)`, pinhole backproject, keep |xyz| ≤ 0.512, | |
| clip to [-0.5, 0.5−1e−9], voxelize 64, union, visible-only = | |
| noise='zeros') → returns `occ_partial [1,64,64,64]` + `z_full [4096,8]` | |
| + `occ_full` (for IoU). `exclude_names` drops given object ids. | |
| * `z_partial` is encoded in the TRAIN STEP by the frozen `ss_encode`, never | |
| precomputed. Deterministic per-(seed, epoch, idx) RNG. | |
| * **Val set (Q7)**: `data.val_roots` = explicit validation roots — 100 | |
| held-out toys4k objects (`smoke_data/val_toys4k100`, disjoint from | |
| toys4k200 via `--exclude_ids_from_root`) + all local omni3d samples | |
| (`smoke_data/val_omni3d`, front+side renders from | |
| `/lp-dev/jonghoon/mv-mesh/exp_faithfulness/omni3d/renders`). When set, val | |
| = ALL objects under val_roots and train = roots MINUS val ids; empty → | |
| legacy hash split. | |
| * `tools/preprocess_ssflow_toys4k.py` — DATA_FORMAT adapter (toys4k AND | |
| omni3d: auto-detects available `front/side/oside/back` views; `--source`, | |
| `--exclude_ids_from_root`). Gates: (a) full-occ encoder/decoder round-trip | |
| IoU ≥ ~0.9; (b) partial ⊂ full containment. | |
| * `tools/gen_x1_official.py` + `slat_official/` — the ONLY producer of the | |
| SLAT training target x1 (USER MANDATE: exact official algorithm, no | |
| reimplementation): vendored byte-identical TRELLIS dataset_toolkits | |
| (Blender 150-view render → open3d 64³ voxelize → all-voxel no-occlusion | |
| DINO mean → `encoder(feats, sample_posterior=False)`) with the single | |
| mandated deviation of SAM3D's `slat_encoder.ckpt`, emitting | |
| `slat/slat_official_sam3d.npz` (RAW SAM3D-space latent on ALL active | |
| voxels). The former on-the-fly visible-mean x1 was deleted; | |
| `encode_visible_latent` survives only as the inference-time forced x0 seed. | |
| ## Training (`mvsam3d/train/train_ss_encoder.py` + `configs/train_ssflow.yaml`) | |
| Rectified-flow v-pred loop: AdamW (trainable = backbone minus d_embedder; | |
| condition embedder + d_embedder frozen; `optim.weight_decay` config-driven, | |
| default 0.0), bf16 autocast, warmup→cosine, grad-accum, DDP via the vendored | |
| `stage1_flow_lora.{distributed,checkpointing,train_utils}` (single-GPU | |
| transparently skips the process group). Per step: move batch → `z_partial = | |
| grid_latent_to_tokens(ss_encode(occ_partial))` (fp32, no_grad, outside | |
| autocast) → `model({"z_partial", "z_full"})` (null cond) → backward/clip/step | |
| (`train_ss_encoder.py:304-308`). | |
| Two optional, config-driven, default-OFF additions (shared with the SlatFlow | |
| loop): | |
| * **EMA** (`ema.{enable,rate,eval_with_ema}`, default `enable: false`) — | |
| `mvsam3d/train/ema.py` `EmaShadow`: fp32 shadow of ONLY the trainable | |
| (`requires_grad`) params, `ema = rate*ema + (1-rate)*param` after every | |
| `optimizer.step()` (`train_ss_encoder.py:336-337`); with `eval_with_ema` | |
| the in-loop validation runs under the EMA weights (bit-exact restore after, | |
| `:354-357`). Persisted as a `step_XXXX_ema.pt` sidecar next to the | |
| `_optimizer.pt` sidecar (the vendored `checkpointing.py` is untouched) and | |
| auto-restored on `resume_from` (`:260-268`, `:380-381`). | |
| * **Gradient checkpointing** (`model.use_checkpoint`, default false) — | |
| threaded through the `SSFlowModel` ctor (`ssflow_model.py:145`) to the | |
| backbone's native per-block `use_checkpoint` (`:207`): activations of the | |
| 24 DiT blocks are recomputed in backward for a large peak-memory cut at | |
| ~modest step-time cost. | |
| NOTE: checkpoints include the frozen condition-embedder tensors (~1.2 GB fp32 | |
| extra) — keep them under /lp-dev. | |
| ``` | |
| cd <repo> | |
| export MIGRATOR_CHECKPOINTS=$PWD/migrator/checkpoints | |
| PYTHONPATH=. python mvsam3d/train/train_ss_encoder.py \ | |
| --config configs/train_ssflow.yaml --set train.steps=2000 | |
| # multi-GPU: torchrun --nproc_per_node=N ... (GPUs 0,1,4,5,6,7 only on this box) | |
| ``` | |
| ## Validation (`mvsam3d/train/validate.py`) | |
| Q7 val set (val_roots), per pass (fixed seed, training RNG saved/restored): | |
| `decoded_iou` — seeded `sample()` → **`coords` (the PRIMARY pruned+downsampled | |
| pipeline output) scattered to 64³** vs the full surface-occupancy grid; | |
| `partial_iou` (input-copy floor: decoded > partial ⇒ the model completes, not | |
| copies); held-out flow `loss`/`loss/shape`. | |
| ## Smoke plan (PLAN.md §6; runs = Smoke phase) | |
| 1. toys4k200 preprocess + gates — DONE; val set (100 toys4k + omni3d) — DONE. | |
| 2. Unit self-check — DONE (warm-start 493/493 + 722/722, null-cond forward, | |
| coords-primary sample). | |
| 3. Overfit: 8 objects, batch 8, lr 1e-4, bf16, warm, null cond — DONE | |
| (2026-09-01, `.debug/train_ssflow_overfit8_e2e.log`): train v-MSE | |
| 0.0953 → 0.0001 over 600 steps (~950×); held-out decoded_iou 0.6441 > | |
| partial_iou floor 0.5483 (n=16). PASS. | |
| 4. Generalization sniff: toys4k200 train vs val_roots; val v-MSE down, val | |
| decoded IoU > partial floor. | |
| 5. Then point `data.roots` at the trellis500k subset (Q8, download-agent | |
| scope) as it lands; wire render conditioning through `encode_condition`. | |
| ## Removed (dead code, PLAN.md §4) — updated | |
| `mvsam3d/model/{condition,flow_model}.py`, `mvsam3d/data/{dexycb_dataset, | |
| adapters,base_dataset,integrated,dexycb_calib}.py`, `mvsam3d/infer/`, | |
| `mvsam3d/train/train.py`, `sam3d_objects/geometry/` (pose-only rot6d/ssi), | |
| verify/diagnose/smoke tools, `build_canonical_latents_multi.py`, DexYCB/ | |
| integrated configs + `train.yaml`, stale docs, all stale sbatch scripts. | |
| RESTORED (corrected design): `sam3d_objects/model/backbone/dit/` embedders, | |
| pristine `modulated.py` + `mot_sparse_structure_flow.py`. Kept per spec: | |
| `mm_latent.py`, `ss_vae.py`, the Euler solver, `timestep_embedder.py`, | |
| `stage1_flow_lora` training utils, `tools/build_canonical_latents.py`, | |
| `canonical.py::voxelize_canonical`. | |
| ## 2026-09-02 — DINO seed-path memory-format fix + in-training appearance-forcing validation | |
| * `MultiViewSlatConditionEmbedder.dino_prenorm` now runs DINO **per view on an | |
| NCHW-contiguous tensor**. Root cause found by the step-0 parity harness: | |
| the dataset builds `crops` with `np.stack` over transposed HWC arrays, which | |
| yields channels-last strides; DINOv2's patch-embed conv then selects | |
| different cuDNN kernels (~1% token deviation vs the reference | |
| `batch_appforce_sam3d.dino_features`, max|d| 1.5–3.4 on a 185 scale), and a | |
| V-view batched forward adds ~7e-4. Per-view + `.contiguous()` is bit-exact | |
| against the reference dumps. This affects the training x0 seed and the | |
| non-anchor condition tokens identically (both come from `dino_prenorm`). | |
| * In-training validation: `mvsam3d/train/val_appforce.py` (trained stage-2 on | |
| cached reference stage-1 coords/base; seed + reinject schedules; per-rank | |
| decode + `evaluate_appforce.py` in the reference env via | |
| `tools/val_stage1_ref.py`; rank-0 merge -> `val_af/<ds>/<schedule>/<metric>` | |
| in wandb), hooked into `train_slat_flow.py` (`val_appforce:` config block, | |
| step 0 + every 5000). Parity harness: `tools/parity_val_appforce.sh`. | |
| ## 2026-09-02 — TRUE multi-object BATCHED stage-2 (SLAT flow) TRAINING | |
| Stage-2 training was bs=1-per-forward (dense `[1,N,8]` over one coord set); | |
| the effective batch came only from `train.grad_accum_steps`. Now the verified | |
| batched-INFERENCE layout (`sample_batched`, `tools/selfcheck_batched_slat.py`) | |
| is reused for TRAINING, so an effective batch of 12 = 4 GPUs x | |
| `train.batch_objects=3` with **no gradient accumulation**. | |
| * `mvsam3d/model/slatflow_model.py` | |
| * `forward` dispatches on the presence of a per-object `layout` key | |
| (`collate_batched` batch -> batched step, `collate_single` -> legacy step). | |
| * `_object_train_pieces` — per-object cond (fuser anchor + non-anchor DINO | |
| tokens + Plücker) / x0 forced-appearance seed with fresh noise on the | |
| invisible rows / normalized official x1 — the SAME construction | |
| `flow_step` performs. | |
| * `flow_step_batched` — B objects concatenated into ONE sparse coord set | |
| with the batch-index column (TRELLIS `SLat.collate_fn`), conditions padded | |
| to a common L and block-diagonally masked via | |
| `CrossAttnBiasHolder.set_batch` (pad columns -inf; each object's soft mask | |
| on its own rows/columns), ONE `gen.reverse_fn` call. **t is per object** | |
| (`[B]`, one `gen._generate_t` draw each): the backbone's `t_embedder(t)` | |
| yields `[B,C]` which `sparse_batch_broadcast` expands per object over the | |
| sparse layout — so no extra plumbing is needed for the timestep. Because | |
| the dense tensor is `[1,T_total,8]` (no batch axis) the interpolant is | |
| applied with a PER-TOKEN t (t expanded over each layout slice) using the | |
| generator's own equations and `gen.sigma_min`: | |
| `x_t=(1-(1-σ)t)x0+t·x1`, `v=x1-(1-σ)x0`. | |
| Loss = **mean over objects of each object's own per-token MSE** | |
| (per-object mean, NOT token-count weighted) so the objective is identical | |
| to averaging B bs=1 losses. DDP guard: `plucker_fc` is kept in the graph | |
| (×0.0) when NO object in the batch has a non-anchor view. | |
| * `mvsam3d/train/train_slat_flow.py` — new `train.batch_objects` (default 1 = | |
| previous behaviour). >1 switches the loader to `batch_size=batch_objects` + | |
| `collate_batched` and requires `grad_accum_steps == 1` (hard error | |
| otherwise); the printed `global_batch` is now | |
| `batch_objects × grad_accum × world_size`. Validation stays bs=1 | |
| (`collate_single` + `flow_step`). | |
| * `tools/selfcheck_batched_train.py` — equivalence + memory self-check: | |
| with x0 noise and t pinned, batched loss == mean of the B single losses, and | |
| peak memory / grad finiteness for `use_checkpoint` off/on. | |
| ## 2026-09-03 — BATCHED sampling + PARALLEL decode/metrics for `val_appforce` | |
| The in-training appearance-forcing validation was one object per 25-step ODE | |
| solve and ONE decode process per rank. Both are now parallel; the outputs, | |
| file layout and wandb keys (`val_af/<ds>/<schedule>/<metric>`) are unchanged. | |
| * `mvsam3d/train/val_appforce.py` | |
| * `_stage2_prepare` — the per-object part of `stage2` up to the solve (DINO | |
| prenorm -> condition -> `encode_visible_latent` -> seed tensor), in the | |
| same op order as the single path. Stays per object (it is cheap); the | |
| batching target is the 25-step flow. | |
| * `_stage2_solve_chunk` — ONE batched solve over a chunk via | |
| `SlatFlowModel.sample_batched` (the verified contract of | |
| `tools/selfcheck_batched_slat.py`: coords concatenated with the batch-index | |
| column, seeds `torch.cat` into `[1,T_total,8]`, conditions padded to a | |
| common L and block-diagonally masked through | |
| `CrossAttnBiasHolder.set_batch`). **`reinject` batched pin:** object *i*'s | |
| visible rows live at `offset_i + local_row` (offset_i = Σ N of the earlier | |
| objects in the chunk = exactly the layout `sample_batched` builds); | |
| `eps_fixed` / `z_forced` are those same rows of the object's OWN base noise | |
| / `z_norm`, concatenated in the same order, so the single-path formula | |
| `x[vis] = (1-(1-σ)t_next)·eps_fixed + t_next·z_forced` applies verbatim to | |
| the gathered global row set after every Euler step. `seed` is applied | |
| per object on its own base before the concatenation. | |
| * `stage2_batched(model, items, bases, schedule, batch_size=6)` — chunks the | |
| object list (sorted by voxel count so a chunk's padding waste is small) and | |
| returns one `stage2`-shaped result dict per input item, in input order. | |
| * `sample_jobs(..., batch_size=6, single=False)` — batched by default; | |
| `single=True` / `VAL_AF_SINGLE=1` / `batch_size<=1` restore the original | |
| path (`_sample_jobs_single`, kept verbatim) for A/B. A batched failure | |
| (e.g. OOM on a chunk of large objects) falls back to the single path for | |
| that chunk instead of losing the objects. | |
| * `decode_and_eval_rank(..., decode_procs=2, threads=8)` — `decode_procs` | |
| reference-env decode processes per rank run CONCURRENTLY, sharded with the | |
| decode mode's own `--shard i --nshards n` (disjoint `<obj>.npz`, disjoint | |
| GLB names, so the shared `glb/` dir and the per-rank hardlink dir are | |
| race-free); the 4 `evaluate_appforce.py` processes stay concurrent. | |
| `_ref_env` now sets OMP/MKL/OpenBLAS/NumExpr thread caps EXPLICITLY (the | |
| trainer already exports `OMP_NUM_THREADS`, so `setdefault` was a no-op): | |
| 4 ranks x 4 concurrent eval procs x 8 threads = 128 of ~255 cores. | |
| * config `val_appforce:` — new `batch_size` (6), `single` (false), | |
| `decode_procs` (2), `ref_threads` (8) in `configs/train_slatflow.yaml`, | |
| `configs/train_slatflow_prod.yaml`, `configs/_smoke_val_e2e.yaml`. | |
| * `tools/verify_batched_val.py` — offline verification against the SINGLE-path | |
| step-5000 dumps: reloads the step-5000 ckpt + EMA sidecar, reruns the single | |
| path (the run floor), then the batched path at B ∈ {1,4,6,8} for both | |
| schedules, reporting per object max|d| / mean|d| vs the reference and vs the | |
| single re-run, coords equality, and s/object timing. PASS iff | |
| `max|batched-ref| <= max(3·max|single_rerun-ref|, 0.05·feat_scale)`. | |
| `--parallel-eval-check` additionally runs the new parallel decode+eval on | |
| those objects into a temp dir and diffs the `results.json` metrics against | |
| the step-5000 `eval_rank*/results.json` rows. | |