README.md

metadata

title: Map Det3D
emoji: 🌍
colorFrom: red
colorTo: indigo
sdk: gradio
sdk_version: 5.49.1
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0
short_description: Streaming multi-view 3D object detection and tracking
models:
  - RoyYang0714/Map-Det3D
  - facebook/map-anything

Map-Det3D

Metric Feed-Forward 3D Reconstruction Prior for Multi-view 3D Object Detection from Streaming Inputs.

The demo streams the posed RGB frames of a scene through the model one at a time and lifts every detection into a single metric world frame, so the boxes accumulate into one consistent 3D layout with persistent track IDs. Results are shown in an embedded Rerun viewer.

• 🌟 GitHub Repository
• 📄 Paper

Running it locally

pip install -r requirements.txt
python app.py

Notes on the deployment

• The Space needs ZeroGPU hardware. @spaces.GPU requests GPU time that scales with the number of frames, up to the 300 s ceiling.
• sdk_version is pinned to a Gradio 5 release because gradio_rerun==0.26.0 is a Gradio 5 custom component. It has to stay on the same version as the rerun-sdk the model code logs with, so bumping one means bumping all three.
• vis4d_cuda_ops/ shadows the package of the same name, because pip cannot install the real one here: it has no wheels, and pip compiles it in an isolated build environment that resolves its own newest torch while the Space caps the runtime torch lower, so the extension fails to import with undefined symbol: c10::NotImplementedError. The local package compiles only the upstream CPU sources, against the torch the Space runs, and routes deformable attention through the grid-sample implementation already in mapdet3d. Detections and tracks come out identical; see its docstring.
• mapdet3d/ is vendored from the research repository so the Space does not depend on a published package.
• The model weights are pulled from RoyYang0714/Map-Det3D at start-up, which also pulls facebook/map-anything (CC-BY-NC-4.0) as the geometry backbone.