Video-depth PointDiT-B, update 32500, EMA2

video-depth-pointdit/README.md

@@ -23,8 +23,8 @@ are not expected to close that gap.
 | `rows/*.jsonl` | the recorded evaluation rows (raw and EMA2 weights; video and single-frame calls) |
 | `manifest.json`, `manifest.sha256` | file names, sizes, sha256 (`manifest.sha256` is `sha256sum -c` input), code commit, update count |
 | `infer_video_depth.py` | depth for your own video or frames: npy, colour-mapped PNGs, MP4 (see "Run inference on your own video") |
-| `reproduce_evaluation.sh` | the exact evaluation commands, with the folder as the run directory |
-| `load_point.py` | builds the model through the repository's builder and loads the EMA weights |
+| `reproduce_evaluation.sh` | the evaluation commands, taking every external path by flag (see Quick start) |
+| `load_point.py` | builds the model through the repository's builder and loads the EMA weights (reads only this folder) |
 
 There is no optimizer state and no raw (non-EMA) weight file here. The evaluator needs only the EMA2 artifact.
 The frozen DINOv3 encoder is not included (see Licence).
@@ -53,41 +53,91 @@ The frozen DINOv3 encoder is not included (see Licence).
   prediction), `--base-steps 1`, on the logit-normal quantile grid the model was trained with (its native sampler is
   four steps, noise scale 2.0).
 
-## How to load and evaluate (Civo cluster)
+## Quick start without Civo
 
-The repository is `git@github.com:NIRVANALAN/vgg_forecast_4d.git`. The recorded rows came from commit `f7b6a03b`.
-A full re-score of the three benchmarks with `reproduce_evaluation.sh` from a downloaded copy of this folder is recorded in
-the repository's hand-off note (`doc/handoff/video_depth_pointdit_checkpoint_and_mot_integration.md`); what was checked
-before this upload is stated under "Run inference on your own video" below.
-The commands need the Civo paths: the released PointDiT-B 512 checkpoint (`model.point.checkpoint` in the config), the
-frozen DINOv3 weights (`model.dino.weights`) and the Sintel / Bonn / KITTI data. Outside Civo, edit those paths in
-`train/resolved_resume_29500.yaml`; the evaluator checks their sha256 against the config.
+Everything below uses only this folder, the project repository on GitHub, two public checkouts and files you request or
+download yourself. No path of the Civo cluster is on any command line.
+
+**1. Code.** `git clone git@github.com:NIRVANALAN/vgg_forecast_4d.git` (you need access to the repository) and check out
+`main`. Loading and inference need commit `2310eec5` or later. The evaluation flag `--data-root` (used below) is on the
+branch `docs/video-depth-pointdit-off-civo` until its pull request is merged; check that branch out for the evaluation.
+Run every command from the clone root with `export PYTHONPATH=$PWD/src`.
+
+**2. Environment.** The scripts were run in Python 3.12.13 with torch 2.9.0+cu130 (CUDA 13.0), numpy 2.4.6, omegaconf
+2.3.1, hydra-core 1.3.4, einops 0.8.2, safetensors 0.8.0, opencv-python 4.11.0.86, imageio 2.37.4 with imageio-ffmpeg
+0.6.0, scipy 1.18.0, pillow 12.3.0, matplotlib 3.11.1, accelerate 1.14.0, timm 1.0.28, transformers 5.14.1. The recorded
+rows used FlashAttention 2.8.3 (`fa2`); the scripts default to PyTorch's `sdpa` attention, which needs no extra
+package (pass `--attention-backend fa2` where it is offered if you installed flash-attn). A GPU is needed for evaluation
+and recommended for inference; loading runs on a CPU.
+
+**3. This folder.** `hf download yslan/dev-pointdit-video-depth --include "video-depth-pointdit/*" --local-dir <dir>`; the
+folder is `<dir>/video-depth-pointdit`, called `$FOLDER` below. `cd $FOLDER && sha256sum -c manifest.sha256` checks it.
+
+**4. DINOv3 ViT-B/16 (not included).** Request it at `https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/`
+(Meta's licence). The file is `dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth`, sha256
+`73cec8be7427c8655ceced13ce62f6e20a1fa90d1b4d4a550df17a1144081a7c` (the scripts refuse another file). The code is
+`git clone https://github.com/facebookresearch/dinov3` (tested at `6876159a11b4df116f30f667f8c9888617df0751`). Both go
+to the scripts by flag: `--dino-weights <file> --dinov3-repo <checkout>`.
+
+**5. Load (CPU, about 15 s).**
+
+```bash
+python $FOLDER/load_point.py $FOLDER
+# temporal_pointdit window_valid_mean_depth 32500 0.9996 140204928
+```
+
+**6. Inference on your video (section below).**
+
+```bash
+python $FOLDER/infer_video_depth.py $FOLDER --input clip.mp4 --out out/ \
+  --dino-weights <dinov3 file> --dinov3-repo <dinov3 checkout>
+```
+
+**7. Evaluation (one GPU).** Needs the benchmark data in the layout of the repository's evaluator
+(`monocular_depth.cohort_pairs`; the layout and the 21 frames per sequence are written out in the repository's
+`doc/handoff/video_depth_and_camera_pose_evaluation_for_external_models.md`, section 3.1) and a checkout of
+`https://github.com/NIRVANALAN/STream3R` at `7bdca8cf7ec206ff8b08c0941ad8286d72087d4f`, whose depth readers the evaluator
+uses (`STREAM3R_ROOT`):
 
 ```bash
-git clone git@github.com:NIRVANALAN/vgg_forecast_4d.git && cd vgg_forecast_4d && git checkout 2310eec5
-source /vgg/yushi/miniconda3/etc/profile.d/conda.sh && conda activate th29py312
-export PYTHONPATH=$PWD/src
-FOLDER=<this folder>            # the downloaded video-depth-pointdit/
-python $FOLDER/load_point.py $FOLDER                   # builds the model, loads EMA2 (CPU, about 15 s)
-bash $FOLDER/reproduce_evaluation.sh $FOLDER $PWD/out  # one GPU; checks manifest.sha256 first
+export STREAM3R_ROOT=<STream3R checkout>
+bash $FOLDER/reproduce_evaluation.sh $FOLDER out/ --dino-weights <dinov3 file> --dinov3-repo <dinov3 checkout> \
+  --data-root <dir holding sintel/ bonn/ kitti/> [--datasets "sintel bonn kitti"]
 ```
 
-`reproduce_evaluation.sh` runs `eval/temporal_point/checkpoint_evaluation.py --task video_depth` per dataset with
-the aspect-preserving depth-fit protocol, EMA2 weights, seed 0, `--base-steps 1`, `--depth-state-cap 1.984375`.
+It checks `manifest.sha256`, writes a temporary run directory (this folder's config with the three external paths
+replaced; the shipped config is not edited) and runs `eval/temporal_point/checkpoint_evaluation.py --task video_depth`
+per dataset with the aspect-preserving depth-fit protocol, EMA2, seed 0, `--base-steps 1`, `--depth-state-cap 1.984375`.
+
+**What reads what.** The released PointDiT-B 512 checkpoint is never read: the scripts set `model.point.weights: random`,
+which builds the same architecture, and all 216 tensors then come from the EMA2 file. The model after loading EMA2 is
+bit-identical to one built over the released checkpoint (checked: 216 of 216 tensors equal). The recorded training
+config keeps its Civo paths as history. Which keys still name one:
+
+| Config key (`train/resolved_resume_29500.yaml`) | Read by load / inference / evaluation? | How to override |
+|---|---|---|
+| `model.point.checkpoint` (the released PointDiT-B 512 file) | no: skipped (see above) | nothing to do |
+| `model.dino.weights` (DINOv3 file), `model.dino.repo` (DINOv3 checkout) | yes, by inference and evaluation | `--dino-weights`, `--dinov3-repo` |
+| the evaluation data root (not in the config: `monocular_depth.DATA_ROOT`) | yes, by evaluation | `--data-root` |
+| STream3R checkout (not in the config: default server path) | yes, by evaluation | environment variable `STREAM3R_ROOT` |
+| `data.track4d.sources[*].kwargs.ROOT`, `metric_scales_csv`, `index_cache_dir`, `camera_exclusion_manifest` roots | no: training data, not read by these scripts | none needed |
+| `train.output_dir`, `train.resume_from` | no | none needed |
+
+The last two rows are from reading the code (the evaluator reads only `data.track4d.frames`, the model, DINO and schedule
+sections); the off-Civo check below ran on a machine where those directories still exist, so it does not prove it.
 
 ## Run inference on your own video
 
 `infer_video_depth.py` takes a video file or a folder of frames and writes per-frame depth. It builds the model through
 the repository's builder (`build_point_model`), loads the EMA2 weights, computes the frozen DINOv3 ViT-B/16 features of the
 frames and samples with the repository's sampler exactly as the evaluation does: one Euler step from Gaussian noise,
-the contracted depth state capped at 1.984375, bf16 autocast. Run it from a clone of the repository, in the
-`th29py312` environment, on one GPU (a CPU works but takes minutes per frame set):
+the contracted depth state capped at 1.984375, bf16 autocast. Run it from a clone of the repository (steps 1 to 4
+of the Quick start), on one GPU (a CPU works but takes minutes per frame set):
 
 ```bash
-cd vgg_forecast_4d && git checkout 2310eec5 && export PYTHONPATH=$PWD/src
-source /vgg/yushi/miniconda3/etc/profile.d/conda.sh && conda activate th29py312
-python $FOLDER/infer_video_depth.py $FOLDER --input clip.mp4 --out out/            # first 21 frames
-python $FOLDER/infer_video_depth.py $FOLDER --input frames_dir/ --out out/ --max-frames 25 --stride 2
+D="--dino-weights <dinov3 file> --dinov3-repo <dinov3 checkout>"
+python $FOLDER/infer_video_depth.py $FOLDER --input clip.mp4 --out out/ $D            # first 21 frames
+python $FOLDER/infer_video_depth.py $FOLDER --input frames_dir/ --out out/ --max-frames 25 --stride 2 $D
 ```
 
 - **Frames:** the first `--max-frames` (default 21, at most 25) frames at `--stride`, in one call. Each is resized, whole
@@ -106,17 +156,23 @@ python $FOLDER/infer_video_depth.py $FOLDER --input frames_dir/ --out out/ --max
   known scale, or fit one scale and shift per clip against a reference, as the benchmark does.
 - **Seed.** The sampler starts from Gaussian noise: `--seed` fixes it. `--seed-key 0:<dataset>:<sequence>` uses the
   benchmark's noise key, which reproduces a benchmark row's call exactly.
-- **Weights it needs:** the DINOv3 ViT-B/16 file named under Licence (`--dino-weights` and `--dinov3-repo` override the
-  config's Civo paths), and by default the released PointDiT-B 512 checkpoint named in the config; add
-  `--random-init-point` to skip that file (every tensor then comes from the EMA2 file).
-
-**Checked:** on one H100 (2026-10-07, repository commit `2310eec5`) the script was run on the first 21 frames of Bonn
-`balloon2` and Sintel `alley_2` with the benchmark noise key (`--seed-key 0:<dataset>:<sequence>`). After one scale and
-shift per sequence in depth against the ground truth (the registered protocol), its depth scores AbsRel 0.0970 on
-`balloon2` (recorded row: 0.0970) and 0.2377 on `alley_2` (recorded: 0.2368); delta<1.25 0.9480 and 0.6414 (recorded
-0.9480 and 0.6420). The recorded rows came from an A100; the small differences are GPU and kernel numerics. The
-`--hfov-degrees` output and the MP4 ran on the same job. `--random-init-point` was checked on a CPU: a model built without
-the released checkpoint holds exactly the 216 EMA2 tensors.
+- **Weights it needs:** only this folder and the DINOv3 ViT-B/16 file and checkout (`--dino-weights`, `--dinov3-repo`,
+  both required). `--attention-backend` is `sdpa` by default; `fa2` is the recorded run's.
+
+**Checked.** (1) Before the off-Civo flags were added, on one H100 (2026-10-07, repository commit `2310eec5`), the script
+run on the first 21 frames of Bonn `balloon2` and Sintel `alley_2` with the benchmark noise key
+(`--seed-key 0:<dataset>:<sequence>`) scores, after one scale and shift per sequence in depth against the ground truth
+(the registered protocol), AbsRel 0.0970 on `balloon2` (recorded row: 0.0970) and 0.2377 on `alley_2` (recorded: 0.2368);
+delta<1.25 0.9480 and 0.6414 (recorded 0.9480 and 0.6420). The recorded rows came from an A100; the small differences are
+GPU and kernel numerics. (2) With the flags of this README, on one H100 (repository branch `docs/video-depth-pointdit-off-civo`),
+every external input copied to node-local `/tmp` and passed by flag (this folder, the DINOv3 file and checkout, a STream3R
+checkout, the Bonn frames), with `TRACK4D_ROOT` and `VGGT_OMEGA_ROOT` pointing at nonexistent paths: `load_point.py` printed
+`temporal_pointdit window_valid_mean_depth 32500 0.9996 140204928`; `infer_video_depth.py` ran on a folder of 21 Bonn frames
+and on an mp4 made from them (448 x 592 raster, `sdpa` attention); and `reproduce_evaluation.sh --datasets bonn` re-scored
+Bonn at AbsRel 0.060529 against the recorded 0.060454 (difference +7.4e-5; `sdpa` here, `fa2` in the recorded row).
+(3) A model built without the released checkpoint holds exactly the same 216 tensors as one built over it, after loading
+EMA2 (all equal, bit for bit). A full three-set re-score from a downloaded copy of this folder with the earlier scripts
+(`fa2`, Civo paths) gave Sintel 0.381766, Bonn 0.060523, KITTI 0.090556 against the recorded 0.381446, 0.060454, 0.090497.
 
 ## Recorded results
 
@@ -168,6 +224,4 @@ is high (0.84 and 0.97 on those two). KITTI's near third (closer than about 10 m
 - DINOv3 (Meta) is under the DINOv3 licence and is NOT redistributed here. Request it at
   `https://ai.meta.com/resources/models-and-libraries/dinov3-downloads/`. The model needs the ViT-B/16 LVD-1689M
   weights (`dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth`, sha256
-  `73cec8be7427c8655ceced13ce62f6e20a1fa90d1b4d4a550df17a1144081a7c`). On Civo they are at
-  `/vgg/yushi/Repo/baselines/dinov3/weights/dinov3_vitb16.pth` (the config names a symlink to it under
-  `/scratch/shared/engs2907/pointdit_official/pretrained/dinov3/`).
+  `73cec8be7427c8655ceced13ce62f6e20a1fa90d1b4d4a550df17a1144081a7c`).

video-depth-pointdit/infer_video_depth.py

@@ -4,8 +4,11 @@
     python infer_video_depth.py <this folder> --input clip.mp4 --out out_dir
     python infer_video_depth.py <this folder> --input frames_dir/ --out out_dir --max-frames 21 --stride 2
 
-Run it from the root of a clone of NIRVANALAN/vgg_forecast_4d (commit 2310eec5 or later) with PYTHONPATH=<clone>/src,
-in the th29py312 environment, on one GPU (a CPU works, slowly). It builds the model through the repository's builder
+Run it from a clone of github.com/NIRVANALAN/vgg_forecast_4d (commit 2310eec5 or later) with PYTHONPATH=<clone>/src,
+on one GPU (a CPU works, slowly). Every external input is a flag: the DINOv3 ViT-B/16 weights (--dino-weights) and a
+checkout of github.com/facebookresearch/dinov3 (--dinov3-repo). It reads no other file outside this folder and your
+input: the released PointDiT-B checkpoint that the training config names is not read (every tensor comes from the EMA2
+file; the model is bit-identical to one built over it). It builds the model through the repository's builder
 (`build_point_model`), loads the EMA2 weights (`_load_weights`), computes the frozen DINOv3 ViT-B/16 features of the
 frames, and samples with the repository's sampler (`sample_window_self_maps`: the shared Euler driver, one step from
 Gaussian noise, the contracted depth state capped at 1.984375, bf16 autocast) exactly as the evaluator does.
@@ -80,11 +83,11 @@ def main() -> int:
     parser.add_argument("--seed", type=int, default=0, help="seed of the Gaussian noise the sampler starts from")
     parser.add_argument("--seed-key", help="the evaluator's noise key '<seed>:<dataset>:<sequence>' (reproduction)")
     parser.add_argument("--hfov-degrees", type=float, help="horizontal field of view; writes depth_frame0_gauge.npy")
-    parser.add_argument("--dino-weights", type=Path, help="DINOv3 ViT-B/16 weights (default: the path in the config)")
-    parser.add_argument("--dinov3-repo", type=Path, help="DINOv3 source checkout (default: the path in the config)")
-    parser.add_argument("--attention-backend", help="Point attention (default: the recorded fa2 on a GPU, sdpa on a CPU)")
-    parser.add_argument("--random-init-point", action="store_true",
-                        help="do not read the released PointDiT-B 512 checkpoint: every tensor comes from the EMA2 file")
+    parser.add_argument("--dino-weights", type=Path, required=True,
+                        help="dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth (sha256 73cec8be..., checked against the config)")
+    parser.add_argument("--dinov3-repo", type=Path, required=True, help="a checkout of github.com/facebookresearch/dinov3")
+    parser.add_argument("--attention-backend", default="sdpa",
+                        help="Point attention: sdpa (default, needs no extra package) or fa2 (the recorded run's, needs flash-attn)")
     parser.add_argument("--no-mp4", action="store_true")
     args = parser.parse_args()
     if not 1 <= args.max_frames <= MAX_FRAMES or args.stride < 1:
@@ -104,13 +107,10 @@ def main() -> int:
     device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
     run, ckpt = args.folder / "train", args.folder / "train" / "ckpt_32500"
     cfg = resolve_run_config(run, ckpt)["cfg"]
-    cfg.model.point.attention_backend = args.attention_backend or ("fa2" if device.type == "cuda" else "sdpa")
-    if args.random_init_point:
-        cfg.model.point.weights = "random"
-    if args.dino_weights:
-        cfg.model.dino.weights = str(args.dino_weights)
-    if args.dinov3_repo:
-        cfg.model.dino.repo = str(args.dinov3_repo)
+    cfg.model.point.attention_backend = args.attention_backend
+    cfg.model.point.weights = "random"           # model.point.checkpoint (the released checkpoint) is not read
+    cfg.model.dino.weights = str(args.dino_weights)
+    cfg.model.dino.repo = str(args.dinov3_repo)
 
     bgr = read_frames(args.input, args.max_frames, args.stride)
     height, width = bgr[0].shape[:2]

video-depth-pointdit/load_point.py

@@ -1,22 +1,28 @@
-"""Build the video-depth PointDiT-B from the Hugging Face folder and load its EMA weights (CPU check)."""
+"""Build the video-depth PointDiT-B through the repository's builder and load its EMA2 weights (CPU, no Civo file).
+
+    python load_point.py <this folder>
+
+Run from a clone of github.com/NIRVANALAN/vgg_forecast_4d with PYTHONPATH=<clone>/src. Nothing outside this folder is
+read: the released PointDiT-B 512 checkpoint that the training config names is NOT read (`model.point.weights: random`
+builds the same architecture, and every one of the 216 tensors then comes from the EMA2 file; the model after the load is
+bit-identical to one built over the released checkpoint). The frozen DINOv3 encoder is not needed to load.
+"""
 import sys
 from pathlib import Path
 
-import torch
-
 from vggforecast.evaluation.checkpoint_loading import _load_weights, resolve_run_config
 from vggforecast.training.model_loading import build_point_model
 
-FOLDER = Path(sys.argv[1])                    # .../video-depth-pointdit
+FOLDER = Path(sys.argv[1])
 RUN, CKPT = FOLDER / "train", FOLDER / "train" / "ckpt_32500"
 
 cfg = resolve_run_config(RUN, CKPT)["cfg"]   # resolved_resume_29500.yaml + launch_resume_29500.json
-cfg.model.point.attention_backend = "sdpa"   # the recorded run used fa2 (GPU); sdpa runs anywhere
-model, core, provenance = build_point_model(cfg)          # PointDiT-B over the released 512 checkpoint
+cfg.model.point.weights = "random"           # do not read model.point.checkpoint
+cfg.model.point.attention_backend = "sdpa"   # the run used fa2 on a GPU; sdpa needs no extra package
+model, core, provenance = build_point_model(cfg)
 base_hw = tuple(int(v) * core.trunk.patch for v in core.trunk.position_source_grid)
-loaded = _load_weights(model, core, CKPT, "ema2", base_hw)  # every tensor of the model comes from ema2
+loaded = _load_weights(model, core, CKPT, "ema2", base_hw)  # every tensor comes from ema2
 model.eval()
-print("architecture", model.architecture, "stage", core.stage if hasattr(core, "stage") else None)
-print("target_centering", core.target_centering, "parameters", sum(p.numel() for p in model.parameters()))
-print("loaded", {k: v for k, v in loaded.items() if k in ("weights", "completed_updates", "ema_decay")})
-print("point expert", type(model.mot.point).__name__, "blocks", len(core.trunk.blocks))
+print(model.architecture, core.target_centering, loaded["completed_updates"], loaded["ema_decay"],
+      sum(p.numel() for p in model.parameters()))
+# temporal_pointdit window_valid_mean_depth 32500 0.9996 140204928

video-depth-pointdit/manifest.json

@@ -21,23 +21,23 @@
   "files": [
     {
       "path": "README.md",
-      "bytes": 13715,
-      "sha256": "30a3803b864b20b96c6bbc1ec8f4c3bfccd78370c944f4951dd149004f262b8a"
+      "bytes": 17774,
+      "sha256": "5062b47c21ad5247032a7b2af694b16646704bf60461ef48086830c0e34cfc64"
     },
     {
       "path": "infer_video_depth.py",
-      "bytes": 10596,
-      "sha256": "d8f4e2ee11f66b9a1b6d69a5eb1dbf8821241b340170e6c1d9d54bdc42877305"
+      "bytes": 10836,
+      "sha256": "05e428ba854c366ec5a76706ca26199fc506efe1381abc2961e395357aea4185"
     },
     {
       "path": "load_point.py",
-      "bytes": 1351,
-      "sha256": "2e12b1461a95d28c7fb6326d0549c59865c3cab24772efdd416d496ec46d870a"
+      "bytes": 1646,
+      "sha256": "a53ae802629e7b16df4b84e7df89cc91b8c46d8a561a9b275f38a4bf094c0e69"
     },
     {
       "path": "reproduce_evaluation.sh",
-      "bytes": 1784,
-      "sha256": "726305ffb62dc37a75dfc15c1709e1b1646423acd269156e096c3b9da421d1f0"
+      "bytes": 3492,
+      "sha256": "e319b66268356bdd1f183c5658b619098ed04a1abeb4b342a51d8d042eb5311c"
     },
     {
       "path": "rows/single_frame_test_video_depth_anything_aspect_preserving_depth_fit_bonn.jsonl",

video-depth-pointdit/manifest.sha256

@@ -1,7 +1,7 @@
-30a3803b864b20b96c6bbc1ec8f4c3bfccd78370c944f4951dd149004f262b8a  README.md
-d8f4e2ee11f66b9a1b6d69a5eb1dbf8821241b340170e6c1d9d54bdc42877305  infer_video_depth.py
-2e12b1461a95d28c7fb6326d0549c59865c3cab24772efdd416d496ec46d870a  load_point.py
-726305ffb62dc37a75dfc15c1709e1b1646423acd269156e096c3b9da421d1f0  reproduce_evaluation.sh
+5062b47c21ad5247032a7b2af694b16646704bf60461ef48086830c0e34cfc64  README.md
+05e428ba854c366ec5a76706ca26199fc506efe1381abc2961e395357aea4185  infer_video_depth.py
+a53ae802629e7b16df4b84e7df89cc91b8c46d8a561a9b275f38a4bf094c0e69  load_point.py
+e319b66268356bdd1f183c5658b619098ed04a1abeb4b342a51d8d042eb5311c  reproduce_evaluation.sh
 2ef4877a1486026e9c8904b5e481a6f953b3fbf2694c5b441bbbb54bf41d1c58  rows/single_frame_test_video_depth_anything_aspect_preserving_depth_fit_bonn.jsonl
 ebd65468808940716fb759f5b7f6f8c8892b1c65947d759eccd4e3a0544c5b5a  rows/single_frame_test_video_depth_anything_aspect_preserving_depth_fit_kitti.jsonl
 1940d1b65e64dc545002c415a139aff185f2afcf990a7f29fd41d5ecf3e2323d  rows/single_frame_test_video_depth_anything_aspect_preserving_depth_fit_sintel.jsonl

video-depth-pointdit/reproduce_evaluation.sh

@@ -1,24 +1,55 @@
 #!/usr/bin/env bash
 # Re-scores this checkpoint (EMA2, update 32,500) on Sintel / Bonn / KITTI with the registered native-aspect depth-fit
 # protocol, one 21-frame call per sequence, one step from Gaussian noise. Run from the root of a clone of
-# NIRVANALAN/vgg_forecast_4d (commit 2310eec5 or later) on a Civo GPU node, in the th29py312 environment:
+# github.com/NIRVANALAN/vgg_forecast_4d (commit 2310eec5 or later), with its requirements installed, on one GPU:
 #
-#   bash reproduce_evaluation.sh <downloaded video-depth-pointdit folder> <output dir>
+#   export STREAM3R_ROOT=<checkout of github.com/NVlabs/STream3R at 7bdca8cf7ec206ff8b08c0941ad8286d72087d4f>
+#   bash reproduce_evaluation.sh <downloaded video-depth-pointdit folder> <output dir> \
+#        --dino-weights <dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth> --dinov3-repo <dinov3 checkout> \
+#        --data-root <dir holding sintel/ bonn/ kitti/>  [--attention-backend sdpa|fa2] [--datasets "sintel bonn kitti"]
 #
-# Needs the Civo data and the frozen weights named in train/resolved_resume_29500.yaml (model.point.checkpoint,
-# model.dino.weights). Rows are written to <output dir>/test_<protocol>_<dataset>.jsonl and compare with rows/.
+# No Civo path is needed: the config's DINOv3 paths are replaced by the flags in a temporary copy of the run
+# directory (the training config is not edited), and the released PointDiT-B checkpoint is not read (every tensor comes
+# from the EMA2 file). Rows go to <output dir>/test_<protocol>_<dataset>.jsonl and compare with rows/.
 set -uo pipefail
-(( $# == 2 )) || { echo "usage: $0 <folder> <out_dir>" >&2; exit 2; }
-FOLDER=$(realpath "$1"); OUT=$(realpath -m "$2"); mkdir -p "$OUT"
+usage() { sed -n '2,12p' "$0" >&2; exit 2; }
+(( $# >= 2 )) || usage
+FOLDER=$(realpath "$1"); OUT=$(realpath -m "$2"); shift 2
+DINO_WEIGHTS=; DINO_REPO=; DATA_ROOT=; ATTENTION=sdpa; DATASETS="sintel bonn kitti"
+while (( $# )); do
+  case "$1" in
+    --dino-weights) DINO_WEIGHTS=$(realpath "$2"); shift 2;;
+    --dinov3-repo) DINO_REPO=$(realpath "$2"); shift 2;;
+    --data-root) DATA_ROOT=$(realpath "$2"); shift 2;;
+    --attention-backend) ATTENTION=$2; shift 2;;
+    --datasets) DATASETS=$2; shift 2;;
+    *) usage;;
+  esac
+done
+[[ -n $DINO_WEIGHTS && -n $DINO_REPO && -n $DATA_ROOT && -n ${STREAM3R_ROOT:-} ]] || { echo "missing flag or STREAM3R_ROOT" >&2; usage; }
 export PYTHONPATH="$PWD/src" WANDB_MODE=disabled
-export TRACK4D_ROOT=/scratch/shared/engs2907/logs/vgg_forecast_4d/tools/track4d_dev_93b3649d
-export STREAM3R_ROOT=/vgg/yushi/Repo/baselines/STream3R VGGT_OMEGA_ROOT=/vgg/yushi/Repo/baselines/vggt-omega
-(cd "$FOLDER" && sha256sum -c manifest.sha256) || exit 3     # the files are the ones in the manifest
-RUN=$FOLDER/train; CK=$RUN/ckpt_32500; PROTO=video_depth_anything_aspect_preserving_depth_fit
+(cd "$FOLDER" && sha256sum -c manifest.sha256 >/dev/null) || { echo "the folder does not match manifest.sha256" >&2; exit 3; }
+mkdir -p "$OUT"
+# A temporary run directory: the folder's config with the three external paths replaced, the folder's checkpoint.
+RUN=$(mktemp -d "$OUT/run.XXXXXX")
+python - "$FOLDER" "$RUN" "$DINO_WEIGHTS" "$DINO_REPO" "$ATTENTION" <<'PY'
+import sys
+from pathlib import Path
+from omegaconf import OmegaConf
+folder, run, weights, repo, attention = (Path(sys.argv[1]), Path(sys.argv[2]), *sys.argv[3:])
+cfg = OmegaConf.load(folder / "train" / "resolved_resume_29500.yaml")
+cfg.model.point.weights = "random"                 # the released checkpoint is not read
+cfg.model.point.attention_backend = attention
+cfg.model.dino.weights, cfg.model.dino.repo = weights, repo
+OmegaConf.save(cfg, run / "resolved_resume_29500.yaml")
+(run / "launch_resume_29500.json").write_bytes((folder / "train" / "launch_resume_29500.json").read_bytes())
+(run / "ckpt_32500").symlink_to(folder / "train" / "ckpt_32500")
+PY
+CK=$RUN/ckpt_32500; PROTO=video_depth_anything_aspect_preserving_depth_fit
 DECODE=$(python -B eval/temporal_point/video_depth_decode_arguments.py --run-dir "$RUN" --checkpoint "$CK" \
   --centred-decode normalized_centred_depth) || exit 4
-for ds in sintel bonn kitti; do
+for ds in $DATASETS; do
   python -B eval/temporal_point/checkpoint_evaluation.py --task video_depth --dataset $ds --run-dir "$RUN" \
-    --checkpoint "$CK" --sequence-frames window $DECODE --video-depth-protocol $PROTO --weights ema2 --seeds 0 \
-    --base-steps 1 --depth-state-cap 1.984375 --out "$OUT/test_${PROTO}_${ds}.jsonl" || exit 5
+    --checkpoint "$CK" --data-root "$DATA_ROOT" --sequence-frames window $DECODE --video-depth-protocol $PROTO \
+    --weights ema2 --seeds 0 --base-steps 1 --depth-state-cap 1.984375 --out "$OUT/test_${PROTO}_${ds}.jsonl" || exit 5
 done