# Reproducing MapVGGT

Feed-forward driving novel-view synthesis: **VGGT-Omega (1B)** backbone → per-pixel metric
depth → world-space 3D Gaussians (positions from depth + known poses) → `gsplat` render →
2D **UNet refine**. MapGS components (HD-map-anchored tokens, scene-graph dynamics, map-aware
losses) are included but were measured neutral on PSNR; the gains are the VGGT backbone +
gentle finetuning, plus the UNet refine.

> **License: non-commercial research only** — this is a derivative of VGGT-Omega (Meta FAIR
> Noncommercial Research License; see `LICENSE`). Do not use commercially. Trained on Waymo
> (no data redistribution) + Argoverse 2 (CC BY-NC-SA 4.0).

## 1. Environment
```bash
pip install torch gsplat safetensors imageio transformers huggingface_hub einops
# VGGT-Omega code + base weights (obtain from the FAIR-licensed source):
git clone https://github.com/facebookresearch/vggt-omega   # -> $VGGT_OMEGA_REPO
# download vggt_omega_1b_512.pt                              # -> $MAPVGGT_VGGT_CKPT
export VGGT_OMEGA_REPO=/path/to/vggt-omega
export MAPVGGT_VGGT_CKPT=/path/to/vggt_omega_1b_512.pt
```
The base VGGT-Omega weights are **not** redistributed here; the released checkpoints already
contain a finetuned copy of them (`model.vggt.*`).

## 2. Data (regenerate from your own licensed copies)
Every source converts to a unified per-clip format (`meta.pt` + `images/<f>_<v>.jpg`):
```bash
# Waymo v1.4.3 perception tfrecords -> unified clips (front 3 cams, 20 frames @10fps)
python -c "from mapgs.data.convert.waymo_to_unified import convert_waymo; from mapgs.config import load_config; \
           convert_waymo('/path/to/waymo_tfrecords', 'data/unified/waymo', load_config(), store_longside=640, image_ext='jpg')"
# Argoverse 2 sensor logs -> unified clips
python -c "from mapgs.data.convert.av2_to_unified import convert_argoverse2; from mapgs.config import load_config; \
           convert_argoverse2('/path/to/av2_raw', 'data/unified/av2', load_config())"
```
Clips are named `train_<segid>_<idx>`; the trainer splits **by segment** (`segid`) to keep
train/val scene-disjoint (no leakage). `mapgs/data/convert/reconvert_av2_fix.py` adds optional
AV2 lens undistortion.

## 3. Train
```bash
# main model: VGGT backbone finetune + per-pixel head + UNet render-refine
PYTHONPATH=$VGGT_OMEGA_REPO python -m scripts.train_mapvggt_refine \
    --roots data/unified/waymo --iters 4000 --val-segs 40 --eval-every 500 \
    --out runs/mapvggt_refine.safetensors
# backbone-only / +map+dyn ablations:
PYTHONPATH=$VGGT_OMEGA_REPO python -m scripts.train_mapvggt_full --finetune-backbone --no-map --no-dyn   # base
PYTHONPATH=$VGGT_OMEGA_REPO python -m scripts.train_mapvggt_full --finetune-backbone                     # +map+dyn
```
Notes: backbone finetuned at lr 1e-5, UNet 3e-4; the held-out PSNR peaks ~step 1000 on the
~1157-clip set (data-limited) — keep the best checkpoint. gsplat+bf16 → reproducible at the
seed/config level, not bit-exact.

## 4. Evaluate a checkpoint
```bash
PYTHONPATH=$VGGT_OMEGA_REPO python -m scripts.eval_mapvggt \
    --ckpt checkpoints/mapvggt_refine_best.safetensors --roots data/unified/waymo --val-segs 40
# -> held-out-SCENE val (n=40): PSNR 22.67±3.76  SSIM 0.689
```

## 5. Results (held-out-SCENE, segment-disjoint Waymo, 40 scenes, 256×448, n_in=8)
| checkpoint | PSNR | SSIM |
|---|---|---|
| `abl_base_best` (VGGT + finetune) | 21.7 | 0.66 |
| `abl_full_best` (+ map tokens + dynamics) | 21.7 | 0.66 (neutral) |
| `mapvggt_refine_best` (+ UNet refine) | **22.67** | **0.689** |

Honest scope: ~5 dB below published feed-forward driving SOTA (DGGT 27.4 / PointForward 28.5,
different protocols). Ablation shows the gain is the backbone + finetune + UNet refine; HD-map
tokens, dynamics, resolution, color-fusion, uncertainty-covariance, and a skybox were all
measured ~neutral (the image-space UNet subsumes them). Binding constraint: training data scale.

## Checkpoint format
- `mapvggt_refine_best.safetensors`: keys `model.*` (MapVGGT incl. finetuned `model.vggt.*` + `model.head.*`)
  and `unet.*` (RefineUNet). Load via `scripts/eval_mapvggt.load_mapvggt_refine`.
- `abl_base_best` / `abl_full_best`: full-model state (`MapVGGT`, with/without map+dyn heads); load with
  `MapVGGT(...).load_state_dict(..., strict=False)`.