README.md

---
title: Map Det3D
emoji: 🌍
colorFrom: red
colorTo: indigo
sdk: gradio
sdk_version: 5.49.1
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0
short_description: Streaming multi-view 3D object detection and tracking
models:
  - RoyYang0714/Map-Det3D
  - facebook/map-anything
---

# Map-Det3D

Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection
from Streaming Inputs.

The demo streams the posed RGB frames of a scene through the model one at a
time and lifts every detection into a single metric world frame, so the boxes
accumulate into one consistent 3D layout with persistent track IDs. Results are
shown in an embedded [Rerun](https://rerun.io) viewer.

- 🌟 [GitHub Repository](https://github.com/cvg/Map-Det3D)
- 📄 [Paper](https://arxiv.org/abs/2608.12179)

## Running it locally

```bash
pip install -r requirements.txt
python app.py
```

## Notes on the deployment

- The Space needs **ZeroGPU** hardware. `@spaces.GPU` requests GPU time that
  scales with the number of frames, up to the 300 s ceiling.
- `sdk_version` is pinned to a Gradio 5 release because `gradio_rerun==0.26.0`
  is a Gradio 5 custom component. It has to stay on the same version as the
  `rerun-sdk` the model code logs with, so bumping one means bumping all three.
- `vis4d_cuda_ops/` shadows the package of the same name, because pip cannot
  install the real one here: it has no wheels, and pip compiles it in an
  isolated build environment that resolves its own newest torch while the Space
  caps the runtime torch lower, so the extension fails to import with
  `undefined symbol: c10::NotImplementedError`. The local package compiles only
  the upstream CPU sources, against the torch the Space runs, and routes
  deformable attention through the grid-sample implementation already in
  `mapdet3d`. Detections and tracks come out identical; see its docstring.
- `mapdet3d/` is vendored from the research repository so the Space does not
  depend on a published package.
- The model weights are pulled from
  [`RoyYang0714/Map-Det3D`](https://huggingface.co/RoyYang0714/Map-Det3D) at
  start-up, which also pulls
  [`facebook/map-anything`](https://huggingface.co/facebook/map-anything)
  (CC-BY-NC-4.0) as the geometry backbone.