| --- |
| license: other |
| license_name: fair-noncommercial-research-derivative |
| license_link: https://github.com/facebookresearch/vggt-omega/blob/main/LICENSE |
| tags: [3d-gaussian-splatting, novel-view-synthesis, autonomous-driving, vggt, feed-forward] |
| --- |
| |
| # MapVGGT — Map-grounded feed-forward 3DGS on a VGGT-Omega backbone (PRIVATE) |
|
|
| > **PRIVATE research artifact.** Non-commercial, research-only. See **License** below before any use. |
|
|
| MapVGGT is a feed-forward novel-view-synthesis model for driving scenes: |
| **VGGT-Omega (1B)** predicts per-pixel **metric depth**; each input pixel is lifted to a |
| world-space 3D Gaussian (positions from depth + **known** camera poses); a per-pixel head |
| predicts opacity/scale/rotation; the union is rendered with `gsplat`; a small 2D **UNet |
| refines** the rendered image. MapGS components (HD-map–anchored tokens, scene-graph |
| dynamics, map-depth / free-space losses) are included but — see results — found neutral. |
|
|
| ## Honest results (held-out-SCENE, segment-disjoint Waymo, 40 distinct scenes, 256×448, n_in=8) |
| |
| | model | PSNR | SSIM | notes | |
| |---|---|---|---| |
| | VGGT-Omega + gentle finetune (backbone) | 21.7 | 0.66 | `abl_base_best` | |
| | + MAGT map tokens + scene-graph dynamics | 21.7 | 0.66 | `abl_full_best` — **neutral** (ablation) | |
| | **+ UNet render-refine** | **22.67** | **0.689** | `mapvggt_refine_best` — **headline** | |
| |
| **Be candid about scope.** This is a **research/system artifact, not SOTA**: ~22.7 dB is |
| **~5 dB below** published feed-forward driving NVS (DGGT 27.4, PointForward 28.5, on |
| different protocols). Established by clean ablation: the entire gain over a generic |
| backbone is **VGGT-Omega + gentle backbone finetuning**; the single extra lever that |
| moved the metric is the **UNet refine (+0.85 dB)**. HD-map tokens, scene-graph dynamics, |
| higher resolution, multi-view color fusion, uncertainty-shaped covariance, and a skybox |
| were all **measured neutral** on this metric (the image-space UNet subsumes them). The |
| binding constraint is data scale (1/3 Waymo, ~1157 clips; overfits ~step 1000). Per-clip |
| PSNR anti-correlates with view-extrapolation distance (r=-0.57): the model is strong on |
| slow/overlapping scenes, weak on fast ego-motion / disocclusion. |
| |
| ## Contents |
| - `mapvggt/` — model (`model.py`), heads (`heads.py`: MAGT map tokens, scene-graph dynamics), |
| `refine.py` (RefineUNet). `crosscolor.py` / `uncertainty.py` are **experimental, validated |
| negative** (kept for the record; not used in training). |
| - `mapgs/` — data pipeline (unified clip format, Waymo/AV2 converters), HD-map, losses, metrics. |
| - `scripts/` — `train_mapvggt_refine.py` (main trainer), `train_mapvggt_full.py` (map+dyn), |
| `eval_mapvggt.py` (canonical loader + held-out eval), data-restore utilities. |
| - `checkpoints/` — `mapvggt_refine_best.safetensors` (headline 22.67), `abl_base_best`, |
| `abl_full_best`. **Each ~4.6 GB and embeds the finetuned VGGT-Omega 1B backbone** (keys |
| `model.vggt.*`, `model.head.*`, `unet.*` for the refine ckpt). |
|
|
| ## NOT included (by design) |
| - **Base VGGT-Omega weights** (`vggt_omega_1b_512.pt`) — obtain from its FAIR-licensed source; |
| set `MAPVGGT_VGGT_CKPT`. (Our refine ckpt already contains a finetuned copy of these weights.) |
| - **Training data** — Waymo Open clips (its license **forbids redistribution**) and AV2 clips |
| (regenerate with `mapgs/data/convert/*` from your own licensed copies). |
| - Vendored clones (`_vggt_omega_repo`, `_tokengs_repo`); clone yourself and set `VGGT_OMEGA_REPO`. |
|
|
| ## Usage |
| ```bash |
| export VGGT_OMEGA_REPO=/path/to/vggt-omega # facebookresearch/vggt-omega clone |
| export MAPVGGT_VGGT_CKPT=/path/to/vggt_omega_1b_512.pt # base weights (FAIR-licensed) |
| # eval the released checkpoint on a segment-disjoint Waymo val split: |
| python -m scripts.eval_mapvggt --ckpt checkpoints/mapvggt_refine_best.safetensors \ |
| --roots /path/to/data/unified/waymo |
| ``` |
| The refine checkpoint round-trips to 22.67±3.76 / 0.689 via `scripts/eval_mapvggt.py`. |
|
|
| ## License & provenance (read before use) |
| - **Derivative of VGGT-Omega (Meta FAIR), under the FAIR Noncommercial Research License.** |
| The checkpoints contain finetuned VGGT-Omega weights → they inherit FAIR terms: |
| **non-commercial, research-only; do not redistribute.** This repo is **PRIVATE** for that reason. |
| ⚠️ Commercial use (incl. by a commercial org) is **not permitted** under FAIR terms. |
| - Lineage: **TokenGS** (NVIDIA, research-only) — earlier backbone, code under Apache-2.0; |
| **Depth-Anything-V2** (Apache-2.0); **PointForward** (scene-graph dynamics formulation). |
| - Training data: **Waymo Open Dataset** (subject to Waymo terms, no redistribution) and |
| **Argoverse 2** (CC BY-NC-SA 4.0). MapGS code is the authors' own. |
|
|
| *Reproducibility note:* gsplat + bf16 make runs reproducible at the seed/config level, not bit-exact. |
|
|
