diff --git a/.gitattributes b/.gitattributes index 290a3e06262be72abe5883b8116de7ebf95af208..7960a7533cae6fd3d7a12ba0c4061aded3b5b388 100644 --- a/.gitattributes +++ b/.gitattributes @@ -7,3 +7,4 @@ *.gif filter=lfs diff=lfs merge=lfs -text *.glb filter=lfs diff=lfs merge=lfs -text *.onnx filter=lfs diff=lfs merge=lfs -text +*.whl filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md index af89276f73d64971a633176d63cf4ef1c0fa8227..f9b07807ade724feebdeaf6738cd9222ccab773e 100644 --- a/README.md +++ b/README.md @@ -55,14 +55,13 @@ The `@spaces.GPU` decorator is a no-op off-Spaces, so it runs on CPU or a local GPU unchanged. > **ZeroGPU note:** ZeroGPU runs on Blackwell GPUs that need `torch>=2.8`, but -> the official `dvlt` package hard-pins `torch==2.5.1`. This repo therefore -> **vendors** `dvlt` under [`packages/dvlt`](packages/dvlt) — a clone of -> [`nv-tlabs/dvlt`](https://github.com/nv-tlabs/dvlt) @ `265fe61` with only its -> `torch`/`torchvision` pins relaxed. HF installs `requirements.txt` *before* -> copying the repo, so that subdirectory is pip-installed via git against this -> Space's own repo (`git+…/spaces/blanchon/dvlt@main#subdirectory=packages/dvlt`). -> The Python source is unmodified; see -> [`packages/dvlt/README.md`](packages/dvlt/README.md) for the exact edits. +> the official `dvlt` package hard-pins `torch==2.5.1`. This repo **vendors** +> `dvlt` under [`packages/dvlt`](packages/dvlt): `repo/` is +> [`nv-tlabs/dvlt`](https://github.com/nv-tlabs/dvlt) @ `265fe61` with a 3-line +> pin relaxation (`torch`/`torchvision`/`pydantic`), and `wheels/` holds the +> prebuilt wheel (`build.sh`). It's installed from that wheel via the Space's +> own HTTPS `resolve` URL — not git or a local path, both of which fail on HF's +> Gradio build. See [`packages/dvlt/README.md`](packages/dvlt/README.md). ## Credits diff --git a/app.py b/app.py index 9659dad4c54d02f5c26fffd5917e3e1aca605159..10728ce996c472cb0bd6b42b07ea8599052a6a24 100644 --- a/app.py +++ b/app.py @@ -4,8 +4,8 @@ Déjà View (DVLT) — Looping Transformers for Multi-View 3D Reconstruction. ZeroGPU-ready: GPU work is wrapped in `@spaces.GPU` (see `dvlt_demo.model`), and falls back to CPU when no GPU / `spaces` package is available. -`dvlt_demo` sits next to this file (importable directly), and `dvlt` is -installed as a path dependency from `packages/dvlt` — no sys.path juggling. +`dvlt` is installed from the vendored wheel in `packages/dvlt/wheels/` (see +`requirements.txt`); `dvlt_demo` sits next to this file. Nothing touches sys.path. """ from dvlt_demo.ui import build_demo diff --git a/packages/dvlt/README.md b/packages/dvlt/README.md index 9ed1d072f34b6689769f9c0ffd9010624a13c043..69b65d18334942d7e4411e6e833e7ffdf3ad7045 100644 --- a/packages/dvlt/README.md +++ b/packages/dvlt/README.md @@ -1,26 +1,28 @@ -# dvlt — vendored clone (edited) +# Vendored `dvlt` -This is a clone of the official **Déjà View (DVLT)** package, -[`nv-tlabs/dvlt`](https://github.com/nv-tlabs/dvlt) @ commit `265fe61`, vendored -into the [`blanchon/dvlt`](https://huggingface.co/spaces/blanchon/dvlt) Space. -It's pip-installed from this subdirectory via git -(`git+…/spaces/blanchon/dvlt@main#subdirectory=packages/dvlt`), because HF -installs `requirements.txt` before copying the repo, so a bare local path can't -be used. Locally, `uv` installs it from the path (`tool.uv.sources`). +- **`repo/`** — a verbatim copy of the upstream **Déjà View (DVLT)** repository, + [`nv-tlabs/dvlt`](https://github.com/nv-tlabs/dvlt) @ commit `265fe61`. The + **only** edit is in `repo/pyproject.toml`: the `torch`/`torchvision` pins were + relaxed from `==` to `>=` (ZeroGPU runs on Blackwell GPUs that need + `torch>=2.8`, and the `torchvision==0.20.1` pin transitively forced + `torch==2.5.1`). Everything else is unmodified. +- **`build.sh`** — builds the wheel from `repo/` into `wheels/`. +- **`wheels/`** — the prebuilt wheel the Space installs. -## What was edited +## Why a vendored wheel instead of `pip install dvlt` -Only `pyproject.toml` — the Python source under `src/dvlt/` is **unmodified**: +The Space (Gradio SDK + ZeroGPU) can't install this any other way: -- `torch==2.5.1` → `torch>=2.5` — ZeroGPU runs on Blackwell GPUs (sm_120) which - need `torch>=2.8`; the exact pin made the package uninstallable there. -- `torchvision==0.20.1` → `torchvision>=0.20` — must track the installed torch. -- `huggingface-hub==1.16.4` → `huggingface-hub>=0.30` — so it co-resolves with - the Gradio/Spaces stack the Space pulls in. +- a local path dep (`./packages/dvlt/repo`) isn't available when HF runs + `pip install -r requirements.txt` — it installs requirements *before* copying + the repo; +- `pip install git+https://huggingface.co/...` fails — pip clones with + `--filter=blob:none`, and HF's git backend can't serve that partial fetch in + the build sandbox. -Edited 2026-06-01 by Julien Blanchon (with Claude) for the demo Space only. -For anything else, use the upstream repository. +So the wheel is committed here and installed via the Space's own HTTPS +`resolve` URL (a plain download — no git, no `sys.path`). See `../../requirements.txt`. Upstream code is Apache-2.0; the `nvidia/dvlt` weights are under the NVIDIA -non-commercial research license. See `LICENSE`, `LICENSES/`, and -`THIRD_PARTY_LICENSES.md`. +non-commercial research license (see `repo/LICENSE`, `repo/LICENSES/`, +`repo/THIRD_PARTY_LICENSES.md`). diff --git a/packages/dvlt/build.sh b/packages/dvlt/build.sh new file mode 100755 index 0000000000000000000000000000000000000000..0142dbc0923f3d0900dbc437317607c32e3f44e0 --- /dev/null +++ b/packages/dvlt/build.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# Build the vendored dvlt wheel from repo/ into wheels/. +# +# The Space installs dvlt from the resulting wheel via its HTTPS resolve URL +# (see ../../requirements.txt) — not from git or a local path, both of which +# fail on Hugging Face's Gradio build (see ../../README.md). Re-run this after +# editing repo/ and commit the refreshed wheel. +set -euo pipefail + +HERE="$(cd "$(dirname "$0")" && pwd)" +rm -rf "$HERE/wheels" +mkdir -p "$HERE/wheels" + +if command -v uv >/dev/null 2>&1; then + uv build --wheel "$HERE/repo" --out-dir "$HERE/wheels" +else + python -m build --wheel --outdir "$HERE/wheels" "$HERE/repo" +fi + +# Keep repo/ a pristine copy of upstream — drop in-tree build artifacts. +rm -rf "$HERE/repo/build" "$HERE/repo"/src/*.egg-info + +echo "Built:" +ls -1 "$HERE/wheels" diff --git a/packages/dvlt/pyproject.toml b/packages/dvlt/pyproject.toml deleted file mode 100644 index 0b7ff5b066d93291bf79f13a22ecebdeeec93259..0000000000000000000000000000000000000000 --- a/packages/dvlt/pyproject.toml +++ /dev/null @@ -1,72 +0,0 @@ -[project] -name = "dvlt" -version = "0.0.1" -description = "Déjà View: Looping Transformers for Multi-View 3D Reconstruction" -readme = "README.md" -requires-python = ">= 3.10" -license = { file = "LICENSE" } -authors = [ - {name = "Tobias Fischer", email = "tobiasfischer623@gmail.com"}, - {name = "Alessandro Burzio", email = "aburzio@nvidia.com"}, - {name = "Haithem Turki", email = "hturki@nvidia.com"}, -] -classifiers = [ - "Development Status :: 3 - Alpha", - "Intended Audience :: Science/Research", - "License :: OSI Approved :: Apache Software License", - "Operating System :: POSIX :: Linux", - "Programming Language :: Python :: 3", - "Programming Language :: Python :: 3.11", - "Topic :: Scientific/Engineering :: Artificial Intelligence", - "Topic :: Scientific/Engineering :: Image Processing", -] -dependencies = [ - # core - "accelerate==1.8.1", - "h5py==3.16.0", - "huggingface-hub>=0.30", # EDIT: was ==1.16.4 — relaxed so it co-resolves with gradio/spaces on the Space - "hydra-core==1.3.2", - "numpy==2.4.4", - "omegaconf==2.3.0", - "opencv-python==4.11.0.86", - "Pillow==11.2.1", - "pydantic==2.12.4", - "safetensors==0.5.3", - "scipy==1.16.0", - "torch>=2.5", # EDIT: was ==2.5.1 — ZeroGPU (Blackwell) ships torch>=2.8 - "torchvision>=0.20", # EDIT: was ==0.20.1 — must track the installed torch - "tqdm==4.67.1", - # logging + console - "rich==14.3.3", - "tensorboard==2.19.0", - "wandb==0.25.1", - # visualization (transitively required by callbacks/__init__) - "matplotlib==3.10.3", - "plotly==6.2.0", - "rerun-sdk==0.24.1", - # 3D geometry (ICP, voxel downsampling) - "open3d==0.19.0", -] - -[project.optional-dependencies] -demos = [ - "gradio==6.1.0", - "onnxruntime-gpu==1.25.0", - "requests==2.32.4", - "trimesh==4.8.3", -] -dev = [ - "black==25.1.0", - "pre-commit==4.2.0", - "pytest==8.4.1", - "ruff==0.12.1", -] -all = ["dvlt[demos,dev]"] - -[build-system] -requires = ["setuptools>=61.0", "wheel"] -build-backend = "setuptools.build_meta" - -[tool.setuptools.packages.find] -where = ["src"] -include = ["dvlt*"] diff --git a/packages/dvlt/repo/.gitattributes b/packages/dvlt/repo/.gitattributes new file mode 100644 index 0000000000000000000000000000000000000000..887a2c18f01df0784f690be728f22c1f9d1fa5c4 --- /dev/null +++ b/packages/dvlt/repo/.gitattributes @@ -0,0 +1,2 @@ +# SCM syntax highlighting & preventing 3-way merges +pixi.lock merge=binary linguist-language=YAML linguist-generated=true diff --git a/packages/dvlt/repo/.gitignore b/packages/dvlt/repo/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..c7429ce5a1266d19fd6eaf0f67f3849977317c9f --- /dev/null +++ b/packages/dvlt/repo/.gitignore @@ -0,0 +1,201 @@ +.hydra/ +output/ +ckpt/ +dependency/ +# Byte-compiled / optimized / DLL files +__pycache__/ +**/__pycache__/ +*.py[cod] +*$py.class + +# C extensions +*.so + +# Distribution / packaging +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +pip-wheel-metadata/ +share/python-wheels/ +*.egg-info/ +.installed.cfg +*.egg +MANIFEST + +# PyInstaller +# Usually these files are written by a python script from a template +# before PyInstaller builds the exe, so as to inject date/other infos into it. +*.manifest +*.spec + +# Installer logs +pip-log.txt +pip-delete-this-directory.txt + +# Unit test / coverage reports +htmlcov/ +.tox/ +.nox/ +.coverage +.coverage.* +.cache +nosetests.xml +coverage.xml +*.cover +*.py,cover +.hypothesis/ +.pytest_cache/ +cover/ + +# Translations +*.mo +*.pot + +# Django stuff: +*.log +local_settings.py +db.sqlite3 +db.sqlite3-journal + +# Flask stuff: +instance/ +.webassets-cache + +# Scrapy stuff: +.scrapy + +# Sphinx documentation +docs/_build/ + +# PyBuilder +target/ + +# Jupyter Notebook +.ipynb_checkpoints + +# IPython +profile_default/ +ipython_config.py + +# pyenv +.python-version + +# pipenv +# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. +# However, in case of collaboration, if having platform-specific dependencies or dependencies +# having no cross-platform support, pipenv may install dependencies that don't work, or not +# install all needed dependencies. +#Pipfile.lock + +# PEP 582; used by e.g. github.com/David-OConnor/pyflow +__pypackages__/ + +# Celery stuff +celerybeat-schedule +celerybeat.pid + +# SageMath parsed files +*.sage.py + +# Environments +.env +.venv +env/ +venv/ +ENV/ +env.bak/ +venv.bak/ + +# Spyder project settings +.spyderproject +.spyproject + +# Rope project settings +.ropeproject + +# mkdocs documentation +/site + +# mypy +.mypy_cache/ +.dmypy.json +dmypy.json + +# Pyre type checker +.pyre/ + +# pytype static type analyzer +.pytype/ + +# Profiling data +.prof + +# Folder specific to your needs +**/tmp/ +**/outputs/skyseg.onnx +skyseg.onnx + +# pixi environments +.pixi +*.egg-info + +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +build/ +develop-eggs/ +dist/ +downloads/ +eggs/ +.eggs/ +lib/ +lib64/ +parts/ +sdist/ +var/ +wheels/ +*.egg-info/ +.installed.cfg +*.egg + +# Jupyter Notebook +.ipynb_checkpoints + +# Virtual Environment +venv/ +env/ +ENV/ + +# IDE +.idea/ +.vscode/ +.cursor/ +.ruff_cache/ +*.swp +*.swo + +# DNRM specific +/datasets/ +outputs/ +third_party/ +visualization/ +checkpoints/ + +# wandb +wandb/ + +# local configs +local.yaml diff --git a/packages/dvlt/repo/.pre-commit-config.yaml b/packages/dvlt/repo/.pre-commit-config.yaml new file mode 100644 index 0000000000000000000000000000000000000000..7984ccdbe1279b95fca2b15d95ef0a2d579afa62 --- /dev/null +++ b/packages/dvlt/repo/.pre-commit-config.yaml @@ -0,0 +1,32 @@ +repos: +- repo: https://github.com/pre-commit/pre-commit-hooks + rev: v4.5.0 + hooks: + - id: trailing-whitespace + - id: end-of-file-fixer + - id: check-yaml + - id: check-added-large-files + +- repo: https://github.com/psf/black + rev: 25.1.0 + hooks: + - id: black + language_version: python3.12 + files: ^(src/|tests/) + +- repo: https://github.com/astral-sh/ruff-pre-commit + rev: v0.11.4 + hooks: + - id: ruff + args: [--fix, --exit-non-zero-on-fix] + files: ^(src/|tests/) + +- repo: local + hooks: + - id: pytest-cpu + name: pytest-cpu-tests + entry: python tests/run_tests.py -c cpu + language: system + pass_filenames: false + always_run: true + stages: [pre-commit] diff --git a/packages/dvlt/repo/CONTRIBUTING.md b/packages/dvlt/repo/CONTRIBUTING.md new file mode 100644 index 0000000000000000000000000000000000000000..470697a14e63c0fe414f3e944bd684cf09e29880 --- /dev/null +++ b/packages/dvlt/repo/CONTRIBUTING.md @@ -0,0 +1,68 @@ +# Contributing to DVLT + +We appreciate your interest in contributing to DVLT! This document provides +guidelines and information for contributors. + +For development setup, code style, testing, and design overview, see +[docs/CONTRIB.md](docs/CONTRIB.md). + +## Signing Your Work + +We require that all contributors "sign-off" on their commits. This certifies +that the contribution is your original work, or you have rights to submit it +under the same license, or a compatible license. + +Any contribution which contains commits that are not Signed-Off will not be +accepted. + +To sign off on a commit you simply use the `--signoff` (or `-s`) option when +committing your changes: + +```bash +$ git commit -s -m "Add cool feature." +``` + +This will append the following to your commit message: + +``` +Signed-off-by: Your Name +``` + +## Full text of the DCO + +``` + Developer Certificate of Origin + Version 1.1 + + Copyright (C) 2004, 2006 The Linux Foundation and its contributors. + + Everyone is permitted to copy and distribute verbatim copies of this + license document, but changing it is not allowed. + + + Developer's Certificate of Origin 1.1 + + By making a contribution to this project, I certify that: + + (a) The contribution was created in whole or in part by me and I + have the right to submit it under the open source license + indicated in the file; or + + (b) The contribution is based upon previous work that, to the best + of my knowledge, is covered under an appropriate open source + license and I have the right under that license to submit that + work with modifications, whether created in whole or in part + by me, under the same open source license (unless I am + permitted to submit under a different license), as indicated + in the file; or + + (c) The contribution was provided directly to me by some other + person who certified (a), (b) or (c) and I have not modified + it. + + (d) I understand and agree that this project and the contribution + are public and that a record of the contribution (including all + personal information I submit with it, including my sign-off) is + maintained indefinitely and may be redistributed consistent with + this project or the open source license(s) involved. +``` diff --git a/packages/dvlt/LICENSE b/packages/dvlt/repo/LICENSE similarity index 100% rename from packages/dvlt/LICENSE rename to packages/dvlt/repo/LICENSE diff --git a/packages/dvlt/LICENSES/NVIDIA-LICENSE.txt b/packages/dvlt/repo/LICENSES/NVIDIA-LICENSE.txt similarity index 100% rename from packages/dvlt/LICENSES/NVIDIA-LICENSE.txt rename to packages/dvlt/repo/LICENSES/NVIDIA-LICENSE.txt diff --git a/packages/dvlt/LICENSES/VGGT-LICENSE.txt b/packages/dvlt/repo/LICENSES/VGGT-LICENSE.txt similarity index 100% rename from packages/dvlt/LICENSES/VGGT-LICENSE.txt rename to packages/dvlt/repo/LICENSES/VGGT-LICENSE.txt diff --git a/packages/dvlt/repo/README.md b/packages/dvlt/repo/README.md new file mode 100644 index 0000000000000000000000000000000000000000..daeed62e393b0689f31ae95fd30f78005ad1dcfa --- /dev/null +++ b/packages/dvlt/repo/README.md @@ -0,0 +1,262 @@ +
+

Déjà View: Looping Transformers for Multi-View 3D Reconstruction

+ +Project Page +arXiv +Hugging Face + +**[NVIDIA](https://www.nvidia.com/)**     **[University of Modena and Reggio Emilia](https://www.unimore.it/it)**     **[University of Toronto](https://www.utoronto.ca/)**     **[ETH Zurich](https://ethz.ch/)** + +[Alessandro Burzio*](https://research.nvidia.com/labs/dvl/author/alessandro-burzio/), [Tobias Fischer*](https://tobiasfshr.github.io/), [Sven Elflein](https://selflein.github.io/), [Qunjie Zhou](https://research.nvidia.com/labs/dvl/author/qunjie-zhou/), [Riccardo de Lutio](https://riccardodelutio.github.io/), [Jiawei Ren](https://jiawei-ren.github.io/), [Jiahui Huang](https://huangjh-pub.github.io/), [Shengyu Huang](https://shengyuh.github.io/), [Marc Pollefeys](https://people.inf.ethz.ch/marc.pollefeys/), [Laura Leal-Taixé](https://research.nvidia.com/labs/dvl/author/laura-leal-taixe/), [Zan Gojcic+](https://zgojcic.github.io/), [Haithem Turki+](https://haithemturki.com/) +
+ +

+ Déjà View demo +

+ +## Overview + +DéjàView (DVLT) is a recurrent transformer for multi-view 3D reconstruction. It +loops a shared block of frame/global attention with discrete depth indexing, +producing per-pixel rays, depth, confidence, and camera poses from an unordered +set of images. Trained once, the number of refinement steps `K` becomes an +inference-time compute knob, matching or outperforming substantially larger +feed-forward baselines at a fraction of their parameters. + +This repository contains: + +- The DVLT model + four configurable ablations (vanilla, decoupled blocks, + no `s_out` token, no depth-scaling). +- Evaluation wrappers for five baselines: VGGT, VGGT-Omega, Depth-Anything-3, + MapAnything, and Pi3. Each wrapper imports the upstream package (installed + separately — see [INSTALL.md](docs/INSTALL.md)). +- A training stack built on `accelerate` + Hydra, with optional W&B logging. +- A Stage-2 fine-tune recipe for the depth-conv head. +- Rerun-based visualization tools. + +## Release status + +- [x] Inference code +- [x] Model weights +- [x] Evaluation code + - [x] eval datasets preprocess and loaders +- [x] Training code + - [x] ScanNet++ training dataset loader + - [ ] other training dataset loaders + +## Quickstart + +### Install + +See [docs/INSTALL.md](docs/INSTALL.md). The short version: + +```bash +conda create -n dvlt python=3.12 && conda activate dvlt +conda install pytorch=2.5.1 torchvision pytorch-cuda=12.4 -c pytorch -c nvidia -c conda-forge +pip install -e .[all] +``` + +### Quick setup + +Quick example script: + +```python +import torch +from accelerate import Accelerator + +from dvlt.model.dvlt.model import DVLT +from dvlt.util.preprocess import load_sequence, preprocess_images + +checkpoint_path = "nvidia/dvlt" # local dir, HTTPS URL, or HF Hub repo id +# load_sequence accepts a directory, a single video, or an explicit list of files. +input_path = "path/to/scene_dir" +# Or: input_path = "path/to/clip.mp4" +# Or: from glob import glob; input_path = sorted(glob("path/to/scene_dir/*.png")) + +accelerator = Accelerator(mixed_precision="bf16") + +model = DVLT(img_size=504) +model.load_pretrained(checkpoint_path, strict=True) +model.setup_test(accelerator) + +_, frames = load_sequence(input_path) +batch = preprocess_images(frames, img_size=504, patch_size=14, device=accelerator.device) + +with torch.no_grad(), accelerator.autocast(): + predictions = model.predict(batch, accelerator) + +cameras = predictions["cameras"][0] # Cameras object with shape [S] +extrinsics_c2w = cameras.camera_to_worlds # (S, 3, 4) — OpenCV convention [R | t] +intrinsics = cameras.get_intrinsics_matrices() # (S, 3, 3) + +depths = predictions["depths"][0] # (S, H, W) +world_points = predictions["world_points"][0] # (S, H, W, 3) +``` + +### Train + +```bash +# Single-GPU +python -m dvlt.scripts.train --config-name dvlt-large data=scannetpp + +# Multi-GPU (4 GPUs) +accelerate launch --num-processes 4 -m dvlt.scripts.train --config-name dvlt-large data=scannetpp + +# Resume +python -m dvlt.scripts.train \ + --config-dir=outputs/ \ + --config-name=config.yaml \ + trainer.resume_from_checkpoint=latest +``` + +### Evaluate + +`benchmark_lite` (DTU, ETH3D, 7Scenes) is a convenience benchmark over the +datasets that don't require heavy preprocessing; the full `benchmark` adds +[ScanNet++](src/dvlt/scripts/preprocess/preprocess_scannetpp.md) and +[NuScenes](src/dvlt/scripts/preprocess/preprocess_nuscenes.md). + +```bash +python -m dvlt.scripts.test --config-name dvlt data=benchmark +# multi-GPU: accelerate launch --num-processes -m dvlt.scripts.test --config-name dvlt data=benchmark +python -m dvlt.scripts.test --config-name dvlt data=benchmark_lite +# multi-GPU: accelerate launch --num-processes -m dvlt.scripts.test --config-name dvlt data=benchmark_lite +``` + +DVLT reference results on the full `benchmark`: + +| Dataset | Pose AUC@3 | Pose AUC@30 | Depth inlier@3% | Depth AbsRel | +|---|---|---|---|---| +| DTU | 0.8319 | 0.9880 | 0.9706 | 0.0093 | +| ETH3D | 0.6604 | 0.9536 | 0.7717 | 0.0267 | +| 7Scenes | 0.1393 | 0.8172 | 0.7437 | 0.0349 | +| ScanNet++ | 0.7941 | 0.9803 | 0.9239 | 0.0167 | +| NuScenes | 0.4340 | 0.8534 | 0.5853 | 0.0673 | + + +### Interactive demo + +Browser UI for uploading images / video and exploring the predicted 3D point +cloud, depth maps and camera trajectory. The dropdown switches between DVLT +and the baseline wrappers (VGGT, VGGT-Omega, DA3, Pi3, MapAnything); +each baseline requires its upstream package installed (see +[docs/INSTALL.md](docs/INSTALL.md)). + +```bash +# Launch on http://localhost:7860 (DVLT preselected) +python -m dvlt.scripts.gradio_demo +``` + +The same script also has a headless **offline mode** that skips Gradio +and writes a `.glb` + `.rrd` per (sequence, model) under +`demo_outputs//`. `--input` accepts a directory of images, a +single image, or a video file (mp4/mov/gif/...), and may be repeated to +process multiple sequences in one go; `--models` is a comma-separated list of +config names from the curated registry (or `all`). + +```bash +# Run two models on two sequences (one image dir, one video) +python -m dvlt.scripts.gradio_demo --offline \ + --input /path/to/scene_dir \ + --input /path/to/clip.mp4 \ + --models dvlt + +# Run every registered model on one sequence +python -m dvlt.scripts.gradio_demo --offline --input /path/to/scene_dir --models all +``` + +## Configuration + +DVLT uses [Hydra](https://hydra.cc) for configuration. Top-level experiment +configs live in `src/dvlt/config/experiments/`: + +| Config | Description | +|---|---| +| `dvlt-large` | Stage-1 recipe (large model, full training schedule, linear depth head). | +| `dvlt-large-ablation` | Vanilla ablation parent — toggle decoupled blocks, no-`s_out`, no-depthscale via overrides. | +| `dvlt-large-ablation-decoupled` | Fully decoupled blocks (`recurrence_mode=none`, no looping): a distinct block per step, fixed 16 steps. | +| `dvlt-large-depthconv-stage2` | Stage-2 depth-conv head fine-tune (matches the released checkpoint and the model's default `depth_head_type="conv"`). | +| `dvlt` | Inference-only alias for the released stage-2 checkpoint. | +| `vggt`, `vggt_omega`, `da3-{base,large,giant}`, `pi3`, `pi3x`, `mapanything` | Eval-only baseline wrappers. Require the upstream package installed (see [INSTALL.md](docs/INSTALL.md)). | + +### User configuration (data paths) + +Per-user settings (most importantly, the dataset root) live in +`src/dvlt/config/experiments/user/`. Copy `default.yaml` to `local.yaml`, +edit `data_root`, and select it via `user=local`: + +```bash +python -m dvlt.scripts.train --config-name dvlt-large data=scannetpp user=local +``` + +`user.data_root` can also be overridden inline or via the `DVLT_DATA_ROOT` +environment variable. + +### Selecting datasets + +Pick a single curated dataset config: + +```bash +python -m dvlt.scripts.train --config-name dvlt-large data=scannetpp +python -m dvlt.scripts.train --config-name dvlt-large data=mixed_all +``` + +## Tab completion + +For scripts using the `@cli` decorator (train, test, visualize): + +```bash +eval "$(python -m dvlt.scripts.train -sc install)" +# later, to remove: +eval "$(python -m dvlt.scripts.train -sc uninstall)" +``` + +## Documentation + +- [docs/INSTALL.md](docs/INSTALL.md) — environment setup + baseline installs +- [docs/data/DATA.md](docs/data/DATA.md) — data pipeline overview + how to + add a new dataset parser +- [docs/CONTRIB.md](docs/CONTRIB.md) — dev setup, code style, tests +- [docs/TESTING.md](docs/TESTING.md) — full test-runner documentation + +## Acknowledgments + +We are also grateful to several other open-source repositories that we drew inspiration from or built upon during the development of our pipeline: +- [VGGT](https://github.com/facebookresearch/vggt) +- [Pi3](https://github.com/yyfz/Pi3) +- [CUT3R](https://github.com/CUT3R/CUT3R) +- [MapAnything](https://github.com/facebookresearch/map-anything) +- [Depth-Anything-3](https://github.com/bytedance-seed/depth-anything-3) + +## Citation + +If you find this work useful, please cite: + +```bibtex +@article{burzio2026dejaview, + title = {D\'ej\`a View: Looping Transformers for Multi-View 3D Reconstruction}, + author = {Burzio, Alessandro and Fischer, Tobias and Elflein, Sven and Zhou, Qunjie and de Lutio, Riccardo and Ren, Jiawei and Huang, Jiahui and Huang, Shengyu and Pollefeys, Marc and Leal-Taix{\'e}, Laura and Gojcic, Zan and Turki, Haithem}, + journal = {arXiv preprint arXiv:2605.30215}, + year = {2026} +} +``` + +## License + attribution + +The DVLT **code** is released under the **Apache License, Version 2.0** — see +[LICENSE](LICENSE). The **model weights** (the `nvidia/dvlt` checkpoint) are +released under the **NVIDIA License** — non-commercial, research-and-evaluation +use only; see [LICENSES/NVIDIA-LICENSE.txt](LICENSES/NVIDIA-LICENSE.txt). + +Portions of the codebase are adapted from third-party open-source projects +(DINOv2, PyTorch3D, MoGe, AnyCalib, MultiNeRF, Depth-Anything-3, VGGT). Each +adapted file carries the upstream copyright + license notice in its header; +see [THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md) for the full attribution +map and full upstream license texts. The VGGT-derived files are distributed +under the VGGT License; see [LICENSES/VGGT-LICENSE.txt](LICENSES/VGGT-LICENSE.txt). + +The baseline evaluation wrappers in `src/dvlt/model/{vggt,vggt_omega,da3,mapanything,pi3}/` +import (do not vendor) their respective upstream packages, each of which is +governed by its own license — see +[THIRD_PARTY_LICENSES.md](THIRD_PARTY_LICENSES.md) §"Upstream packages used +for evaluation". diff --git a/packages/dvlt/THIRD_PARTY_LICENSES.md b/packages/dvlt/repo/THIRD_PARTY_LICENSES.md similarity index 100% rename from packages/dvlt/THIRD_PARTY_LICENSES.md rename to packages/dvlt/repo/THIRD_PARTY_LICENSES.md diff --git a/packages/dvlt/repo/assets/nvidia-hq-dvlt.gif b/packages/dvlt/repo/assets/nvidia-hq-dvlt.gif new file mode 100644 index 0000000000000000000000000000000000000000..e4bc2b18399696a762574c626eab5eaeb2938572 --- /dev/null +++ b/packages/dvlt/repo/assets/nvidia-hq-dvlt.gif @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f8c6586e158bcb682c86a9a22aff2ca53cf2546587a6a22eefec134fc0efd3f0 +size 13417487 diff --git a/packages/dvlt/repo/docs/CONTRIB.md b/packages/dvlt/repo/docs/CONTRIB.md new file mode 100644 index 0000000000000000000000000000000000000000..62129f5ab4593d6e851b7a3b6ace6cd0c1e90b1e --- /dev/null +++ b/packages/dvlt/repo/docs/CONTRIB.md @@ -0,0 +1,97 @@ +# Contribution guidelines + +If you contribute to the codebase, please follow these steps + + +## Development Setup + +If you're contributing to the codebase, we recommend installing the development dependencies and setting up pre-commit hooks: + +```bash +# Install development dependencies +pip install -e .[dev] + +# Install pre-commit hooks +pip install pre-commit +pre-commit install +``` + +### Code Style + +This project uses: +- Black for code formatting +- Ruff for linting and import sorting +- Google-style docstrings + +You can manually run the formatters: +```bash +# Format code with Black +black src tests + +# Run Ruff linter and auto-fix Ruff violations where possible +ruff check --fix src tests +``` + +### Running Tests + +The project has a comprehensive test suite with smart hardware detection. For quick development feedback: + +```bash +# Fast CPU-only tests (recommended for development) +python tests/run_tests.py --category cpu + +# All tests appropriate for your hardware +python tests/run_tests.py + +# Run specific test categories +python tests/run_tests.py --category cpu # CPU-only tests (fast, no GPU required) +python tests/run_tests.py --category gpu # GPU tests (single GPU, not integration) +python tests/run_tests.py --category integration # Integration tests (requires GPU for performance) +python tests/run_tests.py --category distributed # Distributed tests (requires accelerate launch) +python tests/run_tests.py --category slow # Slow/comprehensive tests + +# List all available test categories +python tests/run_tests.py --list +``` + +For direct pytest usage: +```bash +# Run all tests (NOTE: Cannot run distributed tests) +pytest + +# Run specific test file +pytest tests/path/to/test_file.py +``` + +See [docs/TESTING.md](TESTING.md) for comprehensive testing documentation. + +The pre-commit hooks automatically run fast tests before each commit to ensure code quality. + +## Sanity Check Mode + +For quick development testing of training functionality, use the built-in sanity check mode (with your desired config): + +```bash +# Instead of long training runs for testing +python -m dvlt.scripts.train --config-name dvlt-large.yaml trainer.sanity_check=True +``` + +This mode: +- **Tests all key functions**: train_step, test_step, log_train, log_test, model saving +- **Uses temporary directories**: No persistent output clutter +- **Mock loggers**: Prints logged keys/types instead of creating wandb runs +- **Fast execution**: Runs 11 training steps + 2 validation batches (~1-2 minutes) +- **Auto cleanup**: Removes temporary files when complete + +Automatically overrides settings to: +- `max_train_steps=11`, `validation_steps=11`, `validation_batches=2` +- `log_every_n_steps=1` + + +## Design + +Check [DATA.md](docs/data/DATA.md) for an overview of the data pipeline flow. We explain there: + +- How to add new parsers for arbitrary new datasets +- How to maintain and extend the data preprocessing code +- What batch format the model will be fed during training and evaluation diff --git a/packages/dvlt/repo/docs/INSTALL.md b/packages/dvlt/repo/docs/INSTALL.md new file mode 100644 index 0000000000000000000000000000000000000000..43b6b91c50d0d7bdb12efe08eec06f0dfbedd6bd --- /dev/null +++ b/packages/dvlt/repo/docs/INSTALL.md @@ -0,0 +1,74 @@ +# Installation + +Tested with **Python 3.11**, **PyTorch 2.5.1**, and **CUDA 12.4** on Linux. + +```bash +conda create -n dvlt python=3.11 && conda activate dvlt +conda install pytorch=2.5.1 torchvision pytorch-cuda=12.4 -c pytorch -c nvidia -c conda-forge + +git clone .git dvlt && cd dvlt +pip install -e .[all] +``` + +Extras (pick what you need): `pip install -e .[demos|dev|all]` + +- `demos` — Gradio demos (`gradio`, `onnxruntime-gpu`, `trimesh`) +- `dev` — `black`, `ruff`, `pre-commit`, `pytest` +- `all` — both + +## Baselines (optional) + +The baseline wrappers in `src/dvlt/model/{vggt,vggt_omega,da3,mapanything,pi3}/` +import the upstream packages at runtime. Install only the ones you plan to +evaluate: + +```bash +# VGGT — code: commercial-use-friendly; checkpoints: original +# is non-commercial, VGGT-1B-Commercial allows commercial use. See upstream LICENSE. +pip install git+https://github.com/facebookresearch/vggt.git --no-deps + +# VGGT-Omega — FAIR Noncommercial Research License (non-commercial / research-only; +# applies to both code and checkpoints). See upstream LICENSE. +pip install git+https://github.com/facebookresearch/vggt-omega.git --no-deps + +# Depth-Anything-3 — Apache-2.0 +pip install git+https://github.com/ByteDance-Seed/Depth-Anything-3.git --no-deps + +# MapAnything — Apache-2.0 +pip install git+https://github.com/facebookresearch/map-anything.git --no-deps +pip install uniception==0.1.6 + +# Pi3 / Pi3X — BSD-3-Clause (clone into third_party/Pi3 first) +git clone https://github.com//Pi3.git third_party/Pi3 +pip install -e third_party/Pi3 --no-deps +``` + +See [THIRD_PARTY_LICENSES.md](../THIRD_PARTY_LICENSES.md) §"Upstream packages +used for evaluation" for license details — verify each upstream's terms suit +your use case before installing. + +## Data backend (`dataverse`) + +DVLT's dataset parsers depend on the `dataverse` package. + +```bash +git clone .git third_party/dataverse +pip install -e third_party/dataverse +``` + +Some datasets need extra dependencies — install the matching extras as needed, +for example: + +```bash +pip install -e 'third_party/dataverse[kubric]' +``` + +The Gradio demos and `dvlt.scripts.visualize` work without `dataverse`; the +full training stack does not. + +## Sanity check + +```bash +python -c "import dvlt; print('dvlt OK')" +pytest -q tests/data/datasets/ +``` diff --git a/packages/dvlt/repo/docs/TESTING.md b/packages/dvlt/repo/docs/TESTING.md new file mode 100644 index 0000000000000000000000000000000000000000..e48cfed6fa60daca8e336c8e0e19a6ddfd42e610 --- /dev/null +++ b/packages/dvlt/repo/docs/TESTING.md @@ -0,0 +1,134 @@ +# Testing Guide + +## 🚀 Quick Start + +```bash +# Run all appropriate tests for your hardware (default) +python tests/run_tests.py + +# Run specific categories +python tests/run_tests.py -c cpu # CPU tests only +python tests/run_tests.py -c gpu # GPU tests (requires CUDA) +python tests/run_tests.py -c integration # Integration tests (requires GPU) +python tests/run_tests.py -c distributed # Distributed tests (requires multiple GPUs) + +# See all available categories +python tests/run_tests.py --list +``` + +## 📋 Test Categories + +The system uses **explicit pytest markers** to categorize tests: + +### Available Categories + +| Category | Command | Marker Required | Hardware | Description | +|----------|---------|----------------|----------|-------------| +| **cpu** | `pytest tests/ -m "not gpu and not distributed and not slow and not integration"` | None | Any CPU | Fast CPU-only tests | +| **gpu** | `pytest tests/ -m "gpu and not distributed and not integration"` | `@pytest.mark.gpu` | NVIDIA GPU | GPU tests (excluding integration) | +| **integration** | `pytest tests/ -m "integration and not distributed"` | `@pytest.mark.integration` | NVIDIA GPU | End-to-end tests | +| **slow** | `pytest tests/ -m "slow"` | `@pytest.mark.slow` | Varies | Time-intensive tests | +| **distributed** | `accelerate launch --num_processes 2 -m pytest tests/ -m "distributed"` | `@pytest.mark.distributed` | 2+ GPUs | Multi-GPU tests | + +### Default Behavior (Intelligent Hardware Detection) + +When no category is specified, the system automatically runs appropriate tests: +- **CPU only**: Runs CPU tests +- **Single GPU**: Runs CPU + GPU + Integration tests +- **Multiple GPUs**: Runs All tests including Distributed + +## 🎯 Test Markers + +### Required Markers for Non-CPU Tests + +```python +# CPU test (no marker needed - default) +def test_utility_function(): + """Fast CPU-only test.""" + pass + +# GPU test +@pytest.mark.gpu +def test_device_computation(): + """Test requiring GPU acceleration.""" + pass + +# Integration test (requires GPU for performance) +@pytest.mark.integration +def test_training_pipeline(): + """End-to-end integration test.""" + pass + +# Distributed test +@pytest.mark.distributed +def test_multi_process_sync(): + """Test requiring multiple GPUs with accelerate launch.""" + pass + +# Slow test +@pytest.mark.slow +def test_comprehensive_scenario(): + """Time-intensive comprehensive test.""" + pass +``` + +## 🚀 Development Workflow + +**Recommended testing order:** +1. **Development**: `python tests/run_tests.py -c cpu` (fast feedback) +2. **Feature Testing**: `python tests/run_tests.py -c gpu` (when GPU available) +3. **Integration**: `python tests/run_tests.py -c integration` (end-to-end validation) +4. **Full Testing**: `python tests/run_tests.py` (all appropriate tests) + +## 📝 Adding New Tests + +**Choose appropriate marker:** +- No marker = CPU test (default) +- `@pytest.mark.gpu` = GPU required +- `@pytest.mark.integration` = End-to-end test (GPU needed for performance) +- `@pytest.mark.distributed` = Multi-GPU test +- `@pytest.mark.slow` = Time-intensive test + +**Verify categorization:** +```bash +# Test your specific function +pytest tests/path/to/test_file.py::test_function_name -v + +# Run the expected category +python tests/run_tests.py -c [expected_category] +``` + +## 📊 Example Test Structure + +```python +import pytest +import torch +from src.dvlt.model.base import BaseModel + +# CPU test (no marker) +def test_model_initialization(): + model = BaseModel() + assert model is not None + +# GPU test +@pytest.mark.gpu +def test_model_cuda_operations(): + if not torch.cuda.is_available(): + pytest.skip("CUDA not available") + model = BaseModel().cuda() + x = torch.randn(1, 3, 224, 224).cuda() + output = model(x) + assert output.device.type == 'cuda' + +# Integration test +@pytest.mark.integration +def test_training_integration(): + # ... comprehensive training test + pass + +# Distributed test +@pytest.mark.distributed +def test_multi_gpu_training(): + # ... distributed training test + pass +``` diff --git a/packages/dvlt/repo/docs/data/DATA.md b/packages/dvlt/repo/docs/data/DATA.md new file mode 100644 index 0000000000000000000000000000000000000000..4bd5b1ce295ab2f1cb5d14571b1cac907760bdab --- /dev/null +++ b/packages/dvlt/repo/docs/data/DATA.md @@ -0,0 +1,138 @@ +# Dataset Preparation + +All dataset paths are resolved from a single config variable, `user.data_root`. +It defaults to `/datasets`, and can be overridden with the CLI +flag `user.data_root=/path/to/data`. The per-dataset configs under +[`src/dvlt/config/experiments/data/`](../../src/dvlt/config/experiments/data/) +read from `${user.data_root}/{train,test}/`. + +Supported datasets and per-dataset preparation: + +| Dataset | Use | Preparation | Config | +|---|---|---|---| +| DTU | eval | download only (Spann3R mvsnet, see below) | [`dtu.yaml`](../../src/dvlt/config/experiments/data/test_datasets/dtu.yaml) | +| ETH3D | eval | [`preprocess_eth3d.md`](../../src/dvlt/scripts/preprocess/preprocess_eth3d.md) | [`eth3d.yaml`](../../src/dvlt/config/experiments/data/test_datasets/eth3d.yaml) | +| 7Scenes | eval | [`preprocess_7scenes.md`](../../src/dvlt/scripts/preprocess/preprocess_7scenes.md) | [`sevenscenes.yaml`](../../src/dvlt/config/experiments/data/test_datasets/sevenscenes.yaml) | +| NuScenes | eval | [`preprocess_nuscenes.md`](../../src/dvlt/scripts/preprocess/preprocess_nuscenes.md) | [`nuscenes.yaml`](../../src/dvlt/config/experiments/data/test_datasets/nuscenes.yaml) | +| ScanNet++ | train + eval | [`preprocess_scannetpp.md`](../../src/dvlt/scripts/preprocess/preprocess_scannetpp.md) | [`scannetpp.yaml`](../../src/dvlt/config/experiments/data/test_datasets/scannetpp.yaml) | + +Preprocessing scripts live under +[`src/dvlt/scripts/preprocess/`](../../src/dvlt/scripts/preprocess/) and are run +as modules, e.g. `python -m dvlt.scripts.preprocess..