Depth Estimation
Safetensors
vitl16
lc700x commited on
Commit
36184d1
·
1 Parent(s): a4169c4

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +643 -2
README.md CHANGED
@@ -1,3 +1,644 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
1
+ <div align="center">
2
+
3
+ <h1>
4
+ <img src="https://raw.githubusercontent.com/Tarikul-Islam-Anik/Animated-Fluent-Emojis/master/Emojis/Objects/Telescope.png" alt="Telescope" width="40" height="40" />
5
+ InfiniDepth: Arbitrary-Resolution and Fine-Grained Depth Estimation with Neural Implicit Fields
6
+ </h1>
7
+
8
+ <div align="center">
9
+ <a href="https://zju3dv.github.io/InfiniDepth/">
10
+ <img src="https://img.shields.io/badge/Project-Page-red?logo=googlechrome&logoColor=red">
11
+ </a>
12
+ <a href="https://arxiv.org/abs/2601.03252">
13
+ <img src="https://img.shields.io/badge/arXiv-Paper-blue?logo=arxiv&logoColor=blue">
14
+ </a>
15
+ <a href="https://huggingface.co/spaces/ritianyu/InfiniDepth">
16
+ <img src="https://img.shields.io/badge/HuggingFace-Demo-yellow?logo=huggingface&logoColor=yellow">
17
+ </a>
18
+ <a href="https://huggingface.co/datasets/ritianyu/game_4k_data">
19
+ <img src="https://img.shields.io/badge/HuggingFace-Dataset-orange?logo=huggingface&logoColor=orange">
20
+ </a>
21
+ <a href="assets/wechat.jpg">
22
+ <img src="https://img.shields.io/badge/微信-WeChat-green?logo=wechat&logoColor=green">
23
+ </a>
24
+ </div>
25
+
26
+ <p align="center">
27
+ <a href="https://ritianyu.github.io/">Hao Yu*</a> •
28
+ <a href="https://haotongl.github.io/">Haotong Lin*</a> •
29
+ <a href="https://github.com/PLUS-WAVE">Jiawei Wang*</a> •
30
+ <a href="https://github.com/Dustbin-Li">Jiaxin Li</a> •
31
+ <a href="https://wangyida.github.io/">Yida Wang</a> •
32
+ <a href="#">Xueyang Zhang</a> •
33
+ <a href="https://ywang-zju.github.io/">Yue Wang</a> <br>
34
+ <a href="https://www.xzhou.me/">Xiaowei Zhou</a> •
35
+ <a href="https://csse.szu.edu.cn/staff/ruizhenhu/">Ruizhen Hu</a> •
36
+ <a href="https://pengsida.net/">Sida Peng</a>
37
+ </p>
38
+
39
+ </div>
40
+
41
+ <div align="center">
42
+
43
+ <img src="assets/demo.gif" alt="InfiniDepth Demo" width="90%" />
44
+
45
+ </div>
46
+
47
+ ## 📢 News
48
+ > **[2026-04]** 🎉 Training and evaluation code of InfiniDepth (RGB Only & Depth Sensor Augmentation) is available now!
49
+
50
+ > **[2026-03]** 🎉 Inference code of InfiniDepth (RGB Only & Depth Sensor Augmentation) is available now!
51
+
52
+ > **[2026-02]** 🎉 InfiniDepth has been accepted to CVPR 2026! Code coming soon!
53
+
54
+
55
+ ## ✨ What can InfiniDepth do?
56
+ InfiniDepth supports three practical capabilities for single-image 3D perception and reconstruction:
57
+
58
+ | Capability | Input | Output |
59
+ | --- | --- | --- |
60
+ | Monocular & Arbitrary-Resolution Depth Estimation | RGB Image | Arbitrary-Resolution Depth Map |
61
+ | Monocular View Synthesis | RGB Image | 3D Gaussian Splatting (3DGS) |
62
+ | Depth Sensor Augmentation (Monocular Metric Depth Estimation) | RGB Image + Depth Sensor | Metric Depth + 3D Gaussian Splatting (3DGS) |
63
+
64
+ ## ⚙️ Installation
65
+
66
+ Please see [INSTALL.md](INSTALL.md) for manual installation.
67
+
68
+ ## 🤗 Hugging Face Space Demo
69
+
70
+ If you want to test InfiniDepth before running local CLI inference, start with the hosted demo:
71
+
72
+ - Hugging Face Space: https://huggingface.co/spaces/ritianyu/InfiniDepth
73
+
74
+ This repo also includes a Gradio Space entrypoint at `app.py`:
75
+
76
+ - Input: RGB image (required), depth map (optional)
77
+ - Task Switch: `Depth` / `3DGS`
78
+ - Model Switch: `InfiniDepth` / `InfiniDepth_DepthSensor`
79
+
80
+ ### Local run
81
+
82
+ ```bash
83
+ python app.py
84
+ ```
85
+
86
+ ### Notes
87
+
88
+ - In this demo, `InfiniDepth_DepthSensor` requires a depth map input; RGB-only inference should use `InfiniDepth`.
89
+ - Supported depth formats in the demo upload: `.png`, `.npy`, `.npz`, `.h5`, `.hdf5`, `.exr`.
90
+
91
+
92
+ ## 🚀 Inference
93
+
94
+ ### Quick Command Index
95
+
96
+ | If you want ... | Recommended command |
97
+ | --- | --- |
98
+ | Relative Depth from Single RGB Image | `bash example_scripts/infer_depth/courtyard_infinidepth.sh` |
99
+ | 3D Gaussian from Single RGB Image | `bash example_scripts/infer_gs/courtyard_infinidepth_gs.sh` |
100
+ | Metric Depth from RGB + Depth Sensor | `bash example_scripts/infer_depth/eth3d_infinidepth_depthsensor.sh` |
101
+ | 3D Gaussian from RGB + Depth Sensor | `bash example_scripts/infer_gs/eth3d_infinidepth_depthsensor_gs.sh` |
102
+ | Multi-View / Video Depth + Global Point Cloud | `bash example_scripts/infer_depth/waymo_multi_view_infinidepth.sh` |
103
+
104
+ <details>
105
+ <summary><strong> 1. Relative Depth from Single RGB Image</strong> (<code>inference_depth.py</code>)</summary>
106
+
107
+ Use this when you want a relative depth map from a single RGB image and, optionally, a point cloud export.
108
+
109
+ **Required input**
110
+
111
+ - `RGB image`
112
+
113
+ **Required checkpoints**
114
+
115
+ - `checkpoints/depth/infinidepth.ckpt`
116
+ - `checkpoints/moge-2-vitl-normal/model.pt` recover metric scale for point cloud export
117
+
118
+ **Optional checkpoint**
119
+
120
+ - `checkpoints/sky/skyseg.onnx` additional sky filtering
121
+
122
+ **Recommended command**
123
+
124
+ ```bash
125
+ python inference_depth.py \
126
+ --input_image_path=example_data/image/courtyard.jpg \
127
+ --model_type=InfiniDepth \
128
+ --depth_model_path=checkpoints/depth/infinidepth.ckpt \
129
+ --output_resolution_mode=upsample \
130
+ --upsample_ratio=2
131
+ ```
132
+
133
+ Replace `example_data/image/courtyard.jpg` with your own image path.
134
+
135
+ **For the example above, outputs are written to**
136
+
137
+ - `example_data/pred_depth/` for the colorized depth map
138
+ - `example_data/pred_pcd/` for the exported point cloud when `--save_pcd=True`
139
+
140
+ **Example scripts**
141
+
142
+ ```bash
143
+ bash example_scripts/infer_depth/courtyard_infinidepth.sh
144
+ bash example_scripts/infer_depth/camera_infinidepth.sh
145
+ bash example_scripts/infer_depth/eth3d_infinidepth.sh
146
+ bash example_scripts/infer_depth/waymo_infinidepth.sh
147
+ ```
148
+
149
+ **Most useful options**
150
+
151
+ | Argument | What it controls |
152
+ | --- | --- |
153
+ | `--output_resolution_mode` | Choose `upsample`, `original`, or `specific`. |
154
+ | `--upsample_ratio` | Used when `output_resolution_mode=upsample`. |
155
+ | `--output_size` | Explicit output size `(H,W)` when `output_resolution_mode=specific`. |
156
+ | `--save_pcd` | Export a point cloud alongside the depth map. |
157
+ | `--fx_org --fy_org --cx_org --cy_org` | Camera intrinsics in the original image resolution. |
158
+
159
+ </details>
160
+
161
+ <details>
162
+ <summary><strong>2. 3D Gaussian + Novel-View Video from Single RGB Image</strong> (<code>inference_gs.py</code>)</summary>
163
+
164
+ Use this when you want a 3D Gaussian export from a single RGB image and an optional novel-view video.
165
+
166
+ **Required input**
167
+
168
+ - `RGB image`
169
+
170
+ **Required checkpoints**
171
+
172
+ - `checkpoints/depth/infinidepth.ckpt`
173
+ - `checkpoints/gs/infinidepth_gs.ckpt`
174
+ - `checkpoints/moge-2-vitl-normal/model.pt` recover metric scale for 3D Gaussian export
175
+
176
+ **Optional checkpoint**
177
+
178
+ - `checkpoints/sky/skyseg.onnx` additional sky filtering
179
+
180
+ **Recommended command**
181
+
182
+ ```bash
183
+ python inference_gs.py \
184
+ --input_image_path=example_data/image/courtyard.jpg \
185
+ --model_type=InfiniDepth \
186
+ --depth_model_path=checkpoints/depth/infinidepth.ckpt \
187
+ --gs_model_path=checkpoints/gs/infinidepth_gs.ckpt
188
+ ```
189
+
190
+ Replace `example_data/image/courtyard.jpg` with your own image path.
191
+
192
+ **For the example above, outputs are written to**
193
+
194
+ - `example_data/pred_gs/InfiniDepth_courtyard_gaussians.ply`
195
+ - `example_data/pred_gs/InfiniDepth_courtyard_novel_orbit.mp4`
196
+
197
+ If `--render_size` is omitted, the novel-view video is rendered at the original input image resolution.
198
+
199
+ **Example scripts**
200
+
201
+ ```bash
202
+ bash example_scripts/infer_gs/courtyard_infinidepth_gs.sh
203
+ bash example_scripts/infer_gs/camera_infinidepth_gs.sh
204
+ bash example_scripts/infer_gs/fruit_infinidepth_gs.sh
205
+ bash example_scripts/infer_gs/eth3d_infinidepth_gs.sh
206
+ ```
207
+
208
+ **Most useful options**
209
+
210
+ | Argument | What it controls |
211
+ | --- | --- |
212
+ | `--render_novel_video` | Turn novel-view rendering on or off. |
213
+ | `--render_size` | Output video resolution `(H,W)`. |
214
+ | `--novel_trajectory` | Camera motion type: `orbit` or `swing`. |
215
+ | `--sample_point_num` | Number of sampled points used for gaussian construction. |
216
+ | `--enable_skyseg_model` | Enable sky masking before gaussian sampling. |
217
+ | `--sample_sky_mask_dilate_px` | Dilate the sky mask before filtering. |
218
+
219
+ > The exported `.ply` files can be visualized in 3D viewers such as [SuperSplat](https://superspl.at/).
220
+
221
+ </details>
222
+
223
+ <details>
224
+ <summary><strong>3. Depth Sensor Augmentation (Metric Depth and 3D Gaussian from RGB + Depth Sensor)</strong></summary>
225
+
226
+ Use this mode when you have an RGB image plus metric depth from a depth sensor.
227
+
228
+ **Required inputs**
229
+
230
+ - `RGB image`
231
+ - `Sparse depth` in `.png`, `.npy`, `.npz`, `.h5`, `.hdf5`, or `.exr`
232
+
233
+ **Required checkpoints**
234
+
235
+ - `checkpoints/depth/infinidepth_depthsensor.ckpt`
236
+ - `checkpoints/moge-2-vitl-normal/model.pt`
237
+ - `checkpoints/gs/infinidepth_depthsensor_gs.ckpt`
238
+
239
+ **Required flags**
240
+
241
+ - `--model_type=InfiniDepth_DepthSensor`
242
+ - `--input_depth_path=...`
243
+
244
+
245
+ **Metric Depth Inference Command**
246
+
247
+ ```bash
248
+ python inference_depth.py \
249
+ --input_image_path=example_data/image/eth3d_office.png \
250
+ --input_depth_path=example_data/depth/eth3d_office.npz \
251
+ --model_type=InfiniDepth_DepthSensor \
252
+ --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
253
+ --fx_org=866.39 \
254
+ --fy_org=866.04 \
255
+ --cx_org=791.5 \
256
+ --cy_org=523.81 \
257
+ --output_resolution_mode=upsample \
258
+ --upsample_ratio=1
259
+ ```
260
+
261
+ **3D Gaussian Inference Command**
262
+
263
+ ```bash
264
+ python inference_gs.py \
265
+ --input_image_path=example_data/image/eth3d_office.png \
266
+ --input_depth_path=example_data/depth/eth3d_office.npz \
267
+ --model_type=InfiniDepth_DepthSensor \
268
+ --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
269
+ --gs_model_path=checkpoints/gs/infinidepth_depthsensor_gs.ckpt \
270
+ --fx_org=866.39 \
271
+ --fy_org=866.04 \
272
+ --cx_org=791.5 \
273
+ --cy_org=523.81
274
+ ```
275
+
276
+ **Example scripts**
277
+
278
+ ```bash
279
+ bash example_scripts/infer_depth/eth3d_infinidepth_depthsensor.sh
280
+ bash example_scripts/infer_depth/waymo_infinidepth_depthsensor.sh
281
+ bash example_scripts/infer_gs/eth3d_infinidepth_depthsensor_gs.sh
282
+ bash example_scripts/infer_gs/waymo_infinidepth_depthsensor_gs.sh
283
+ ```
284
+
285
+ **Most useful options**
286
+
287
+ | Argument | What it controls |
288
+ | --- | --- |
289
+ | `--fx_org --fy_org --cx_org --cy_org` | Strongly recommended when you know the sensor intrinsics. |
290
+ | `--output_resolution_mode` | Output behavior for `inference_depth.py`. |
291
+ | `--render_size` | Video resolution for `inference_gs.py`. |
292
+ | `--output_ply_dir` | Custom output directory for gaussian export. |
293
+
294
+ </details>
295
+
296
+ <details>
297
+ <summary><strong>4. Multi-View / Video Depth + Global Point Cloud</strong> (<code>inference_multi_view_depth.py</code>)</summary>
298
+
299
+ Use this when you want sequence-level depth inference from an RGB image folder or video, plus per-frame aligned point clouds and one merged global point cloud. By default the script runs DA3 once on the whole sequence, then aligns each InfiniDepth depth map to the corresponding DA3 depth map before export. When you already know the camera intrinsics and extrinsics, you can instead provide them directly and skip DA3 entirely.
300
+
301
+ **Required inputs**
302
+
303
+ - `RGB image directory`, `single RGB image`, or `video`
304
+ - `Sparse depth` directory / single file / depth video when `--model_type=InfiniDepth_DepthSensor`
305
+
306
+ **Required checkpoints / dependencies**
307
+
308
+ - `checkpoints/depth/infinidepth.ckpt` for RGB-only inference
309
+ - `checkpoints/depth/infinidepth_depthsensor.ckpt` for RGB + depth sensor inference
310
+ - `checkpoints/moge-2-vitl-normal/model.pt` recover metric scale for RGB-only frame inference
311
+ - `depth-anything-3` installed in the current environment when using the default DA3-based sequence mode; default DA3 model is `depth-anything/DA3-LARGE-1.1`
312
+
313
+ **Optional checkpoint**
314
+
315
+ - `checkpoints/sky/skyseg.onnx` additional sky filtering
316
+
317
+ **RGB-Only Multi-View / Video Command**
318
+
319
+ ```bash
320
+ python inference_multi_view_depth.py \
321
+ --input_path=example_data/multi-view/waymo/image \
322
+ --model_type=InfiniDepth \
323
+ --depth_model_path=checkpoints/depth/infinidepth.ckpt \
324
+ ```
325
+
326
+ **RGB + Depth Sensor Multi-View / Video Command**
327
+
328
+ ```bash
329
+ python inference_multi_view_depth.py \
330
+ --input_path=example_data/multi-view/waymo/image \
331
+ --input_depth_path=example_data/multi-view/waymo/depth \
332
+ --model_type=InfiniDepth_DepthSensor \
333
+ --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
334
+ ```
335
+
336
+ For video input, replace `--input_path` with a video file. When `--model_type=InfiniDepth_DepthSensor`, `--input_depth_path` can also be a depth video and must contain the same number of frames as the RGB input.
337
+
338
+ **Explicit Camera-Parameter Multi-View Command**
339
+
340
+ ```bash
341
+ python inference_multi_view_depth.py \
342
+ --input_path=example_data/multi-view/waymo/image \
343
+ --camera_intrinsics_dir=/path/to/intrinsics \
344
+ --camera_extrinsics_dir=/path/to/extrinsics \
345
+ --model_type=InfiniDepth \
346
+ --depth_model_path=checkpoints/depth/infinidepth.ckpt \
347
+ ```
348
+
349
+ The explicit camera mode expects Waymo-style text files under `intrinsics/` and `extrinsics/`. Files are sorted lexicographically and matched one-to-one against the sorted RGB image list, so the number of camera files must exactly match the number of images. In this mode the script skips DA3 loading, DA3 cache export, DA3 RANSAC conditioning, and DA3 post scale alignment. This mode currently supports image inputs only, not video.
350
+
351
+ **For the RGB-only example above, outputs are written to**
352
+
353
+ - `example_data/multi-view/waymo/pred_sequence/image/frames/depth/` for aligned raw depth maps
354
+ - `example_data/multi-view/waymo/pred_sequence/image/frames/depth_vis/` for colorized depth maps
355
+ - `example_data/multi-view/waymo/pred_sequence/image/frames/pcd/` for per-frame aligned point clouds
356
+ - `example_data/multi-view/waymo/pred_sequence/image/frames/meta/` for per-frame camera and alignment metadata
357
+ - `example_data/multi-view/waymo/pred_sequence/image/da3/sequence_pose.npz` for cached DA3 predictions
358
+ - `example_data/multi-view/waymo/pred_sequence/image/merged/sequence_merged.ply` for the merged global point cloud
359
+
360
+ **Example scripts**
361
+
362
+ ```bash
363
+ bash example_scripts/infer_depth/waymo_multi_view_infinidepth.sh
364
+ bash example_scripts/infer_depth/waymo_multi_view_infinidepth_depthsensor.sh
365
+ bash example_scripts/infer_depth/waymo_multi_view_infinidepth_explicit_camera.sh
366
+ ```
367
+
368
+ **Most useful options**
369
+
370
+ | Argument | What it controls |
371
+ | --- | --- |
372
+ | `--input_path` | RGB image directory, single image, or video path. |
373
+ | `--input_depth_path` | Depth directory, single depth file, or depth video; required for `InfiniDepth_DepthSensor`. |
374
+ | `--camera_intrinsics_dir --camera_extrinsics_dir` | Enable explicit camera mode from sorted Waymo-style txt directories. Image inputs only; file counts must match the RGB frame count. |
375
+ | `--input_mode` | Force `images` or `video` instead of auto detection. |
376
+ | `--align_to_da3_depth` | Align each InfiniDepth depth map to the corresponding DA3 depth map before export. Ignored in explicit camera mode. |
377
+ | `--save_frame_pcd` | Save one aligned point cloud per frame. |
378
+ | `--save_merged_pcd` | Save the merged global point cloud across the whole sequence. |
379
+ | `--da3_scale_align_conf_threshold` | Minimum DA3 confidence used during per-frame scale estimation. |
380
+ | `--output_root` | Override the default `pred_sequence/<sequence_name>/` output directory. |
381
+
382
+ </details>
383
+
384
+ <details>
385
+ <summary><strong>5. Common Argument Conventions</strong></summary>
386
+
387
+ | Argument | Used in | Description |
388
+ | --- | --- | --- |
389
+ | `--input_image_path` | depth + gs | Path to the input RGB image. |
390
+ | `--input_path` | multi-view | Path to an RGB image directory, single image, or video. |
391
+ | `--input_depth_path` | depth + gs + multi-view | Optional metric depth prompt; required for `InfiniDepth_DepthSensor`. In multi-view mode this can be a depth directory, single depth file, or depth video. |
392
+ | `--camera_intrinsics_dir --camera_extrinsics_dir` | multi-view | Optional sequence camera parameter directories. When both are set, multi-view inference skips DA3 and uses the provided sorted txt files directly. |
393
+ | `--model_type` | depth + gs + multi-view | `InfiniDepth` for RGB-only, `InfiniDepth_DepthSensor` for RGB + sparse depth. |
394
+ | `--depth_model_path` | depth + gs | Path to the depth checkpoint. |
395
+ | `--gs_model_path` | gs only | Path to the gaussian predictor checkpoint. |
396
+ | `--moge2_pretrained` | depth + gs | MoGe-2 checkpoint used when `--input_depth_path` is missing. |
397
+ | `--fx_org --fy_org --cx_org --cy_org` | depth + gs | Camera intrinsics in original image resolution. Missing values fall back to MoGe-2 estimates or image-size defaults. |
398
+ | `--input_size` | depth + gs | Network input size `(H,W)` used during inference. |
399
+ | `--enable_skyseg_model` | depth + gs + multi-view | Enable sky masking before depth or gaussian sampling. |
400
+ | `--sky_model_ckpt_path` | depth + gs | Path to the sky segmentation ONNX checkpoint. |
401
+
402
+ **Depth output modes**
403
+
404
+ - `--output_resolution_mode=upsample`: output size = `input_size * upsample_ratio`
405
+ - `--output_resolution_mode=original`: output size = original input image size
406
+ - `--output_resolution_mode=specific`: output size = `output_size`
407
+
408
+ **Default output directories**
409
+
410
+ | Script | Default directory |
411
+ | --- | --- |
412
+ | `inference_depth.py` depth images | `pred_depth/` next to your input data folder |
413
+ | `inference_depth.py` point clouds | `pred_pcd/` next to your input data folder |
414
+ | `inference_gs.py` gaussians and videos | `pred_gs/` next to your input data folder |
415
+ | `inference_multi_view_depth.py` sequence outputs | `pred_sequence/<sequence_name>/` next to your input data folder |
416
+
417
+ </details>
418
+
419
+ ## 🏋️ Training and Validation
420
+
421
+ The repo also provides `main.py` training and validation entrypoints for InfiniDepth and InfiniDepth_DepthSensor.
422
+
423
+ First, prepare the training/validation data and the pretrained weight as described in [DATA.md](DATA.md).
424
+
425
+ Before running any command, export the environment variables required by `main.py`:
426
+
427
+ ```bash
428
+ export workspace=/path/to/your/experiments
429
+ export commonspace=/path/to/your/common_space
430
+ ```
431
+
432
+ - `workspace` stores experiment outputs under `outputs/<task>/<exp_name>/`
433
+ - `commonspace` stores datasets and pretrained weights shared across experiments
434
+
435
+
436
+ ### Quick Command Index
437
+
438
+ | If you want ... | Recommended command |
439
+ | --- | --- |
440
+ | Fine-tune from an existing checkpoint | Add `ckpt_path=...` to the training command |
441
+ | Train from scratch | Omit `ckpt_path` and use a fresh `exp_name` |
442
+ | Validate on the mixed real-data benchmark | Run `main.py` with `--entry val` |
443
+
444
+ <details>
445
+ <summary><strong>1. Fine-Tuning from an Existing Checkpoint</strong> (<code>main.py</code>, default <code>train_net</code> entry)</summary>
446
+
447
+ Use this when you want to initialize training from an existing InfiniDepth checkpoint. The training config referenced below uses Hypersim as the training set and runs validation on a mixed real-data benchmark at the end of each epoch.
448
+
449
+ **Required environment**
450
+
451
+ - `workspace=/path/to/your/experiments`
452
+ - `commonspace=/path/to/your/common_space`
453
+
454
+ **RGB-Only Fine-Tuning Command**
455
+
456
+ ```bash
457
+ python3 main.py \
458
+ --cfg_file training/exp_configs/exps/infinidepth.yaml \
459
+ --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
460
+ ckpt_path=checkpoints/depth/infinidepth.ckpt \
461
+ exp_name=finetune_infinidepth_on_hypersim \
462
+ model.compute_abs_metric=True \
463
+ model.save_orig_pred=True \
464
+ model.save_metrics=True \
465
+ pl_trainer.devices=8
466
+ ```
467
+
468
+ **RGB + Depth Sensor Fine-Tuning Command**
469
+
470
+ ```bash
471
+ python3 main.py \
472
+ --cfg_file training/exp_configs/exps/infinidepth_depthsensor.yaml \
473
+ --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
474
+ ckpt_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
475
+ exp_name=finetune_infinidepth_depthsensor_on_hypersim \
476
+ model.compute_abs_metric=True \
477
+ model.save_orig_pred=True \
478
+ model.save_metrics=True \
479
+ pl_trainer.devices=8
480
+ ```
481
+
482
+ **Equivalent launch scripts**
483
+
484
+ ```bash
485
+ bash launch_scripts/train/infinidepth.sh
486
+ bash launch_scripts/train/infinidepth_depthsensor.sh
487
+ ```
488
+
489
+ **Outputs**
490
+
491
+ - `${workspace}/outputs/${task}/${exp_name}/checkpoints/` for saved checkpoints
492
+ - `${workspace}/outputs/${task}/${exp_name}/tb/` for TensorBoard logs
493
+
494
+ **Notes**
495
+
496
+ - `ckpt_path` must point to an existing checkpoint.
497
+ - The training data config is `training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml`.
498
+ - If the same `exp_name` already has checkpoints in `${workspace}/outputs/${task}/${exp_name}/checkpoints/`, training will resume from the latest saved checkpoint in that directory.
499
+
500
+ </details>
501
+
502
+ <details>
503
+ <summary><strong>2. Train from Scratch</strong> (<code>main.py</code>, no <code>ckpt_path</code>)</summary>
504
+
505
+ Use this when you want to start training without loading an InfiniDepth `.ckpt`. In this mode, do not pass `ckpt_path`. The model will still load the DINOv3 backbone from `${commonspace}/pretrained_models/dinov3/`. You need to download the DINOv3 weights yourself and place them there before running.
506
+
507
+ **RGB-Only Training-from-Scratch Command**
508
+
509
+ ```bash
510
+ python3 main.py \
511
+ --cfg_file training/exp_configs/exps/infinidepth.yaml \
512
+ --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
513
+ exp_name=train_infinidepth_from_scratch \
514
+ model.compute_abs_metric=True \
515
+ model.save_orig_pred=True \
516
+ model.save_metrics=True \
517
+ pl_trainer.devices=2
518
+ ```
519
+
520
+ **RGB + Depth Sensor Training-from-Scratch Command**
521
+
522
+ ```bash
523
+ python3 main.py \
524
+ --cfg_file training/exp_configs/exps/infinidepth_depthsensor.yaml \
525
+ --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
526
+ exp_name=train_infinidepth_depthsensor_from_scratch \
527
+ model.compute_abs_metric=True \
528
+ model.save_orig_pred=True \
529
+ model.save_metrics=True \
530
+ pl_trainer.devices=2
531
+ ```
532
+
533
+ **Notes**
534
+
535
+ - Use a new `exp_name` for a clean scratch run.
536
+ - If you intentionally want to reuse an old `exp_name`, set `resume_training=False` to prevent automatic resume. Be careful: when `resume_training=False`, the code will delete the old output directory before training.
537
+
538
+ </details>
539
+
540
+ <details>
541
+ <summary><strong>3. Validation from a Checkpoint</strong> (<code>main.py</code> with <code>--entry val</code>)</summary>
542
+
543
+ Use this when you want to run validation metrics on the mixed real-data benchmark defined in `training/exp_configs/components/data/test/infinidepth_mix_data.yaml`.
544
+
545
+ **RGB-Only Validation Command**
546
+
547
+ ```bash
548
+ python3 main.py \
549
+ --cfg_file training/exp_configs/exps/infinidepth.yaml \
550
+ --include training/exp_configs/components/data/test/infinidepth_mix_data.yaml \
551
+ --entry val \
552
+ ckpt_path=checkpoints/depth/infinidepth.ckpt \
553
+ exp_name=eval_infinidepth \
554
+ model.compute_abs_metric=True \
555
+ model.save_orig_pred=True \
556
+ model.save_metrics=True \
557
+ pl_trainer.devices=2
558
+ ```
559
+
560
+ **RGB + Depth Sensor Validation Command**
561
+
562
+ ```bash
563
+ python3 main.py \
564
+ --cfg_file training/exp_configs/exps/infinidepth_depthsensor.yaml \
565
+ --include training/exp_configs/components/data/test/infinidepth_mix_data.yaml \
566
+ --entry val \
567
+ ckpt_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
568
+ exp_name=eval_infinidepth_depthsensor \
569
+ model.compute_abs_metric=True \
570
+ model.save_orig_pred=True \
571
+ model.save_metrics=True \
572
+ pl_trainer.devices=2
573
+ ```
574
+
575
+ **Reference launch scripts**
576
+
577
+ ```bash
578
+ bash launch_scripts/eval/infinidepth.sh
579
+ bash launch_scripts/eval/infinidepth_depthsensor.sh
580
+ ```
581
+
582
+ **Validation datasets**
583
+
584
+ - Kitti val split
585
+ - ETH3D val split
586
+ - NYU val split
587
+ - ScanNet val split
588
+ - DIODE indoor val split
589
+ - Synth4K (CyberPunk, DeadIsland, SpiderMan2, SpiderManMM, WatchDogLegion)
590
+
591
+ These datasets are read from `${commonspace}/datasets/` using the meta files referenced in `training/exp_configs/components/data/test/infinidepth_mix_data.yaml`.
592
+
593
+ **Outputs**
594
+
595
+ - `${workspace}/outputs/${task}/${exp_name}/val_metrics/` for validation logs
596
+ - `${workspace}/outputs/${task}/${exp_name}/default/metrics/metrics.json` for aggregated validation metrics
597
+ - `${workspace}/outputs/${task}/${exp_name}/default/metrics/all_scenes.csv` when `model.save_metrics=True`
598
+
599
+ </details>
600
+
601
+ <details>
602
+ <summary><strong>4. Common Overrides</strong></summary>
603
+
604
+ | Argument | What it controls |
605
+ | --- | --- |
606
+ | `ckpt_path` | Initialization or evaluation checkpoint path. Omit it for training from scratch. |
607
+ | `exp_name` | Experiment name used to build `${workspace}/outputs/${task}/${exp_name}`. |
608
+ | `pl_trainer.devices` | Number of GPUs used by PyTorch Lightning. |
609
+ | `model.compute_abs_metric` | Enable absolute-metric evaluation during training or validation. |
610
+ | `model.save_orig_pred` | Save original prediction outputs alongside logs and metrics. |
611
+ | `model.save_metrics` | Save metric files for later inspection. |
612
+ | `--entry val` | Switch `main.py` from the default training entry to the validation entry. |
613
+ | `--include` | Merge an extra data config, such as `training/exp_configs/components/data/test/infinidepth_mix_data.yaml`. |
614
+
615
+ </details>
616
+
617
+ ## 🙏 Acknowledgments
618
+
619
+ We thank <a href="https://yuanhongyu.xyz/" target="_blank">Yuanhong Yu</a>, <a href="https://gangweix.github.io/" target="_blank">Gangwei Xu</a>, <a href="https://github.com/ghy0324" target="_blank">Haoyu Guo</a> and <a href="https://hugoycj.github.io/" target="_blank">Chongjie Ye</a> for their insightful discussions and valuable suggestions, and <a href="https://zhenx.me/" target="_blank">Zhen Xu</a> for his dedicated efforts in curating the synthetic data.
620
+
621
+
622
+ ## 📖 Citation
623
+
624
+ If you find InfiniDepth useful in your research, please consider citing:
625
+
626
+ ```bibtex
627
+ @article{yu2026infinidepth,
628
+ title={InfiniDepth: Arbitrary-Resolution and Fine-Grained Depth Estimation with Neural Implicit Fields},
629
+ author={Hao Yu, Haotong Lin, Jiawei Wang, Jiaxin Li, Yida Wang, Xueyang Zhang, Yue Wang, Xiaowei Zhou, Ruizhen Hu and Sida Peng},
630
+ booktitle={arXiv preprint},
631
+ year={2026}
632
+ }
633
+ ```
634
  ---
635
+
636
+ <div align="center">
637
+
638
+ <img src="https://raw.githubusercontent.com/Tarikul-Islam-Anik/Animated-Fluent-Emojis/master/Emojis/Hand%20gestures/Folded%20Hands%20Light%20Skin%20Tone.png" alt="Thanks" width="25" height="25" />
639
+
640
+ **Thank you for your interest in InfiniDepth!**
641
+
642
+ <sub>⭐ Star this repo if you find it interesting!</sub>
643
+
644
+ </div>