Reproducing MapVGGT

Feed-forward driving novel-view synthesis: VGGT-Omega (1B) backbone → per-pixel metric depth → world-space 3D Gaussians (positions from depth + known poses) → gsplat render → 2D UNet refine. MapGS components (HD-map-anchored tokens, scene-graph dynamics, map-aware losses) are included but were measured neutral on PSNR; the gains are the VGGT backbone + gentle finetuning, plus the UNet refine.

License: non-commercial research only — this is a derivative of VGGT-Omega (Meta FAIR Noncommercial Research License; see LICENSE). Do not use commercially. Trained on Waymo (no data redistribution) + Argoverse 2 (CC BY-NC-SA 4.0).

1. Environment

pip install torch gsplat safetensors imageio transformers huggingface_hub einops
# VGGT-Omega code + base weights (obtain from the FAIR-licensed source):
git clone https://github.com/facebookresearch/vggt-omega   # -> $VGGT_OMEGA_REPO
# download vggt_omega_1b_512.pt                              # -> $MAPVGGT_VGGT_CKPT
export VGGT_OMEGA_REPO=/path/to/vggt-omega
export MAPVGGT_VGGT_CKPT=/path/to/vggt_omega_1b_512.pt

The base VGGT-Omega weights are not redistributed here; the released checkpoints already contain a finetuned copy of them (model.vggt.*).

2. Data (regenerate from your own licensed copies)

Every source converts to a unified per-clip format (meta.pt + images/<f>_<v>.jpg):

# Waymo v1.4.3 perception tfrecords -> unified clips (front 3 cams, 20 frames @10fps)
python -c "from mapgs.data.convert.waymo_to_unified import convert_waymo; from mapgs.config import load_config; \
           convert_waymo('/path/to/waymo_tfrecords', 'data/unified/waymo', load_config(), store_longside=640, image_ext='jpg')"
# Argoverse 2 sensor logs -> unified clips
python -c "from mapgs.data.convert.av2_to_unified import convert_argoverse2; from mapgs.config import load_config; \
           convert_argoverse2('/path/to/av2_raw', 'data/unified/av2', load_config())"

Clips are named train_<segid>_<idx>; the trainer splits by segment (segid) to keep train/val scene-disjoint (no leakage). mapgs/data/convert/reconvert_av2_fix.py adds optional AV2 lens undistortion.

3. Train

# main model: VGGT backbone finetune + per-pixel head + UNet render-refine
PYTHONPATH=$VGGT_OMEGA_REPO python -m scripts.train_mapvggt_refine \
    --roots data/unified/waymo --iters 4000 --val-segs 40 --eval-every 500 \
    --out runs/mapvggt_refine.safetensors
# backbone-only / +map+dyn ablations:
PYTHONPATH=$VGGT_OMEGA_REPO python -m scripts.train_mapvggt_full --finetune-backbone --no-map --no-dyn   # base
PYTHONPATH=$VGGT_OMEGA_REPO python -m scripts.train_mapvggt_full --finetune-backbone                     # +map+dyn

Notes: backbone finetuned at lr 1e-5, UNet 3e-4; the held-out PSNR peaks ~step 1000 on the ~1157-clip set (data-limited) — keep the best checkpoint. gsplat+bf16 → reproducible at the seed/config level, not bit-exact.

4. Evaluate a checkpoint

PYTHONPATH=$VGGT_OMEGA_REPO python -m scripts.eval_mapvggt \
    --ckpt checkpoints/mapvggt_refine_best.safetensors --roots data/unified/waymo --val-segs 40
# -> held-out-SCENE val (n=40): PSNR 22.67±3.76  SSIM 0.689

5. Results (held-out-SCENE, segment-disjoint Waymo, 40 scenes, 256×448, n_in=8)

checkpoint	PSNR	SSIM
`abl_base_best` (VGGT + finetune)	21.7	0.66
`abl_full_best` (+ map tokens + dynamics)	21.7	0.66 (neutral)
`mapvggt_refine_best` (+ UNet refine)	22.67	0.689

Honest scope: ~5 dB below published feed-forward driving SOTA (DGGT 27.4 / PointForward 28.5, different protocols). Ablation shows the gain is the backbone + finetune + UNet refine; HD-map tokens, dynamics, resolution, color-fusion, uncertainty-covariance, and a skybox were all measured ~neutral (the image-space UNet subsumes them). Binding constraint: training data scale.

Checkpoint format

mapvggt_refine_best.safetensors: keys model.* (MapVGGT incl. finetuned model.vggt.* + model.head.*) and unet.* (RefineUNet). Load via scripts/eval_mapvggt.load_mapvggt_refine.
abl_base_best / abl_full_best: full-model state (MapVGGT, with/without map+dyn heads); load with MapVGGT(...).load_state_dict(..., strict=False).