markomih amael-apple patricejaton commited on
Commit
25c2928
·
0 Parent(s):

Co-authored-by: amael-apple <amael-apple@users.noreply.huggingface.co>
Co-authored-by: patricejaton <patricejaton@users.noreply.huggingface.co>

.gitattributes ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/teaser.jpg filter=lfs diff=lfs merge=lfs -text
37
+ assets/hero.mp4 filter=lfs diff=lfs merge=lfs -text
38
+ assets/hero-poster.jpg filter=lfs diff=lfs merge=lfs -text
39
+ assets/tradeoff.png filter=lfs diff=lfs merge=lfs -text
40
+ assets/method.png filter=lfs diff=lfs merge=lfs -text
41
+ assets/tradeoff-light.png filter=lfs diff=lfs merge=lfs -text
42
+ assets/tradeoff-dark.png filter=lfs diff=lfs merge=lfs -text
43
+ assets/method-light.png filter=lfs diff=lfs merge=lfs -text
44
+ assets/method-dark.png filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Disclaimer: IMPORTANT: This Apple Machine Learning Research Model is
2
+ specifically developed and released by Apple Inc. ("Apple") for the sole purpose
3
+ of scientific research of artificial intelligence and machine-learning
4
+ technology. “Apple Machine Learning Research Model” means the model, including
5
+ but not limited to algorithms, formulas, trained model weights, parameters,
6
+ configurations, checkpoints, and any related materials (including
7
+ documentation).
8
+
9
+ This Apple Machine Learning Research Model is provided to You by
10
+ Apple in consideration of your agreement to the following terms, and your use,
11
+ modification, creation of Model Derivatives, and or redistribution of the Apple
12
+ Machine Learning Research Model constitutes acceptance of this Agreement. If You
13
+ do not agree with these terms, please do not use, modify, create Model
14
+ Derivatives of, or distribute this Apple Machine Learning Research Model or
15
+ Model Derivatives.
16
+
17
+ * License Scope: In consideration of your agreement to abide by the following
18
+ terms, and subject to these terms, Apple hereby grants you a personal,
19
+ non-exclusive, worldwide, non-transferable, royalty-free, revocable, and
20
+ limited license, to use, copy, modify, distribute, and create Model
21
+ Derivatives (defined below) of the Apple Machine Learning Research Model
22
+ exclusively for Research Purposes. You agree that any Model Derivatives You
23
+ may create or that may be created for You will be limited to Research Purposes
24
+ as well. “Research Purposes” means non-commercial scientific research and
25
+ academic development activities, such as experimentation, analysis, testing
26
+ conducted by You with the sole intent to advance scientific knowledge and
27
+ research. “Research Purposes” does not include any commercial exploitation,
28
+ product development or use in any commercial product or service.
29
+
30
+ * Distribution of Apple Machine Learning Research Model and Model Derivatives:
31
+ If you choose to redistribute Apple Machine Learning Research Model or its
32
+ Model Derivatives, you must provide a copy of this Agreement to such third
33
+ party, and ensure that the following attribution notice be provided: “Apple
34
+ Machine Learning Research Model is licensed under the Apple Machine Learning
35
+ Research Model License Agreement.” Additionally, all Model Derivatives must
36
+ clearly be identified as such, including disclosure of modifications and
37
+ changes made to the Apple Machine Learning Research Model. The name,
38
+ trademarks, service marks or logos of Apple may not be used to endorse or
39
+ promote Model Derivatives or the relationship between You and Apple. “Model
40
+ Derivatives” means any models or any other artifacts created by modifications,
41
+ improvements, adaptations, alterations to the architecture, algorithm or
42
+ training processes of the Apple Machine Learning Research Model, or by any
43
+ retraining, fine-tuning of the Apple Machine Learning Research Model.
44
+
45
+ * No Other License: Except as expressly stated in this notice, no other rights
46
+ or licenses, express or implied, are granted by Apple herein, including but
47
+ not limited to any patent, trademark, and similar intellectual property rights
48
+ worldwide that may be infringed by the Apple Machine Learning Research Model,
49
+ the Model Derivatives or by other works in which the Apple Machine Learning
50
+ Research Model may be incorporated.
51
+
52
+ * Compliance with Laws: Your use of Apple Machine Learning Research Model must
53
+ be in compliance with all applicable laws and regulations.
54
+
55
+ * Term and Termination: The term of this Agreement will begin upon your
56
+ acceptance of this Agreement or use of the Apple Machine Learning Research
57
+ Model and will continue until terminated in accordance with the following
58
+ terms. Apple may terminate this Agreement at any time if You are in breach of
59
+ any term or condition of this Agreement. Upon termination of this Agreement,
60
+ You must cease to use all Apple Machine Learning Research Models and Model
61
+ Derivatives and permanently delete any copy thereof. Sections 3, 6 and 7 will
62
+ survive termination.
63
+
64
+ * Disclaimer and Limitation of Liability: This Apple Machine Learning Research
65
+ Model and any outputs generated by the Apple Machine Learning Research Model
66
+ are provided on an “AS IS” basis. APPLE MAKES NO WARRANTIES, EXPRESS OR
67
+ IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF
68
+ NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE,
69
+ REGARDING THE APPLE MACHINE LEARNING RESEARCH MODEL OR OUTPUTS GENERATED BY
70
+ THE APPLE MACHINE LEARNING RESEARCH MODEL. You are solely responsible for
71
+ determining the appropriateness of using or redistributing the Apple Machine
72
+ Learning Research Model and any outputs of the Apple Machine Learning Research
73
+ Model and assume any risks associated with Your use of the Apple Machine
74
+ Learning Research Model and any output and results. IN NO EVENT SHALL APPLE BE
75
+ LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
76
+ IN ANY WAY OUT OF THE USE, REPRODUCTION, MODIFICATION AND/OR DISTRIBUTION OF
77
+ THE APPLE MACHINE LEARNING RESEARCH MODEL AND ANY OUTPUTS OF THE APPLE MACHINE
78
+ LEARNING RESEARCH MODEL, HOWEVER CAUSED AND WHETHER UNDER THEORY OF CONTRACT,
79
+ TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, EVEN IF APPLE HAS
80
+ BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
81
+
82
+ * Governing Law: This Agreement will be governed by and construed under the laws
83
+ of the State of California without regard to its choice of law principles. The
84
+ Convention on Contracts for the International Sale of Goods shall not apply to
85
+ the Agreement except that the arbitration clause and any arbitration hereunder
86
+ shall be governed by the Federal Arbitration Act, Chapters 1 and 2.
87
+
88
+ Copyright (C) 2026 Apple Inc. All Rights Reserved.
README.md ADDED
@@ -0,0 +1,134 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: apple-amlr
4
+ license_link: https://github.com/apple-aiml-research/ml-depth-pro-2/blob/main/LICENSE_MODEL
5
+ pipeline_tag: depth-estimation
6
+ library_name: depth-pro-2
7
+ tags:
8
+ - depth-estimation
9
+ - metric-depth
10
+ - monocular-depth
11
+ - video-depth
12
+ ---
13
+
14
+ <h1 align="center">Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth</h1>
15
+
16
+ <p align="center">
17
+ <a href="https://github.com/apple-aiml-research/ml-depth-pro-2"><img src="https://img.shields.io/badge/GitHub-181717?style=flat-square&amp;logo=github&amp;logoColor=white" alt="GitHub"/></a>
18
+ <a href="https://arxiv.org/abs/TODO"><img src="https://img.shields.io/badge/Technical%20Report-B31B1B?style=flat-square&amp;logo=arxiv&amp;logoColor=white" alt="Technical Report"/></a>
19
+ <a href="https://apple-aiml-research.github.io/ml-depth-pro-2/"><img src="https://img.shields.io/badge/Project%20Page-666666?style=flat-square&amp;logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJ3aGl0ZSIgc3Ryb2tlLXdpZHRoPSIyIj48Y2lyY2xlIGN4PSIxMiIgY3k9IjEyIiByPSIxMCIvPjxwYXRoIGQ9Ik0yIDEyaDIwTTEyIDJhMTUgMTUgMCAwIDEgMCAyME0xMiAyYTE1IDE1IDAgMCAwIDAgMjAiLz48L3N2Zz4%3D" alt="Project Page"/></a>
20
+ </p>
21
+ <!-- Hidden until the Space is public:
22
+ <h3 align="center">
23
+ 🚀 <a href="https://huggingface.co/spaces/apple/DepthPro2">Try the Interactive Demo →</a>
24
+ </h3>
25
+ -->
26
+
27
+ Depth Pro 2 estimates metric depth for every pixel from an ordinary camera: a single photo, a
28
+ whole video, or a live feed, at the input's full resolution. Its depth is sharp at object edges
29
+ and steady from frame to frame.
30
+
31
+ <video src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/hero.mp4" poster="https://huggingface.co/apple/DepthPro2/resolve/main/assets/hero-poster.jpg" autoplay muted loop playsinline width="100%"></video>
32
+
33
+ ## Models
34
+
35
+ Each size is its own repository with its weights. Larger sizes give sharper, steadier depth;
36
+ smaller ones run faster.
37
+
38
+ | Model | Repository | Parameters | Download size |
39
+ |:---:|:---:|:---:|:---:|
40
+ | S | [apple/DepthPro2-S](https://huggingface.co/apple/DepthPro2-S) | 24M | 97 MB |
41
+ | B | [apple/DepthPro2-B](https://huggingface.co/apple/DepthPro2-B) | 97M | 386 MB |
42
+ | L | [apple/DepthPro2-L](https://huggingface.co/apple/DepthPro2-L) | 334M | 1.3 GB |
43
+ | H+ | [apple/DepthPro2-H](https://huggingface.co/apple/DepthPro2-H) | 872M | 3.5 GB |
44
+
45
+ All sizes share a focal length estimator,
46
+ [apple/DepthPro2-focal](https://huggingface.co/apple/DepthPro2-focal) (23.5M parameters, 94 MB),
47
+ which loads with each size and predicts the focal length when it is unknown.
48
+
49
+ ## Quick start
50
+
51
+ ```bash
52
+ pip install "git+https://github.com/apple-aiml-research/ml-depth-pro-2"
53
+ ```
54
+
55
+ ```python
56
+ from functools import partial
57
+
58
+ import torch
59
+ from depth_pro_2 import DepthPro2, load_rgb, upright
60
+ from depth_pro_2.model import to_normalized_tensor
61
+
62
+ # apple/DepthPro2-S, -B, -L or -H; bfloat16 is how the models run on a GPU.
63
+ model = DepthPro2.from_pretrained("apple/DepthPro2-H", precision=torch.bfloat16)
64
+ model = model.to("cuda")
65
+ transform = partial(to_normalized_tensor, device=torch.device("cuda"))
66
+ ```
67
+
68
+ ### Image
69
+
70
+ ```python
71
+ # f_px is the focal length recorded in the photo, or None.
72
+ image, f_px = load_rgb("photo.jpg")
73
+ depth = model.infer(transform(image), f_px=f_px)["depth"] # meters, (H, W)
74
+ ```
75
+
76
+ ### Video
77
+
78
+ ```python
79
+ import av
80
+
81
+ # 16 frames at a time: one prediction per frame, in order.
82
+ video = av.open("video.mp4")
83
+ frames = (transform(upright(f)) for f in video.decode(video=0))
84
+ for prediction in model.infer_video(frames):
85
+ depth = prediction["depth"] # meters, (H, W)
86
+ ```
87
+
88
+ Depth is infinite where the scene is infinitely far, such as the sky. The
89
+ [code](https://github.com/apple-aiml-research/ml-depth-pro-2) has the command line and a live-feed
90
+ example.
91
+
92
+ ## Results
93
+
94
+ ### Sharpness and temporal consistency
95
+
96
+ <img class="block dark:hidden" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/tradeoff-light.png" alt="Boundary sharpness against temporal consistency for Depth Pro 2 and prior single-frame and multi-frame methods"/>
97
+ <img class="hidden dark:block" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/tradeoff-dark.png" alt="Boundary sharpness against temporal consistency for Depth Pro 2 and prior single-frame and multi-frame methods"/>
98
+
99
+ Depth Pro 2 outperforms prior work operating on a single frame or multiple frames on sharpness
100
+ (Boundary F1 score) and temporal consistency (Jitter). Higher F1 and lower jitter are better, so the
101
+ top right is best. Results averaged over Sintel and Spring datasets.
102
+
103
+ ### Photos
104
+
105
+ ![Depth Pro 2 on photos: the inputs on top, their predicted depth below](assets/teaser.jpg)
106
+
107
+ Top row: the input photos. Bottom row: Depth Pro 2's depth, red near and blue far.
108
+
109
+ ## Method
110
+
111
+ <a href="https://huggingface.co/apple/DepthPro2/blob/main/assets/method-light.png"><img class="block dark:hidden" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/method-light.png" alt="Depth Pro 2's architecture: a multi-view transformer over the input frames, then a DPT decoder per frame"/></a>
112
+ <a href="https://huggingface.co/apple/DepthPro2/blob/main/assets/method-dark.png"><img class="hidden dark:block" src="https://huggingface.co/apple/DepthPro2/resolve/main/assets/method-dark.png" alt="Depth Pro 2's architecture: a multi-view transformer over the input frames, then a DPT decoder per frame"/></a>
113
+
114
+ A multi-view transformer processes one or more frames jointly, and a DPT decoder predicts an inverse
115
+ depth map for each frame. ① After the first third of its blocks, the transformer alternates
116
+ attention within each frame and attention across frames. ② The decoder assembles tokens from
117
+ several blocks into feature maps at 1/32 to 1/4 of the input resolution.
118
+
119
+ ## License
120
+
121
+ The weights are released under the [Apple Machine Learning Research Model license](https://github.com/apple-aiml-research/ml-depth-pro-2/blob/main/LICENSE_MODEL), for
122
+ research use only: non-commercial scientific research and academic development.
123
+
124
+ ## Citation
125
+
126
+ ```bibtex
127
+ @article{DepthPro2:2026:arxiv,
128
+ title = {Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth},
129
+ author = {Peiyun Hu and Marko Mihajlovic and Lyndon Duong and Lahav Lipson and Wei Dong and Ama\"{e}l Delaunoy and Stephan R. Richter and Vladlen Koltun},
130
+ journal = {arXiv preprint arXiv:TODO},
131
+ year = {2026},
132
+ url = {https://arxiv.org/abs/TODO},
133
+ }
134
+ ```
assets/hero-poster.jpg ADDED

Git LFS Details

  • SHA256: a84b478e458997f7edca179eb73ae6179fd93d4613fb17ef51552791ad57e683
  • Pointer size: 131 Bytes
  • Size of remote file: 259 kB
assets/hero.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:de9b022998c774145f15d9c1a307f092ea89178e274241144dbbd6b06cf38003
3
+ size 19092201
assets/method-dark.png ADDED

Git LFS Details

  • SHA256: e90bc63b52226c731ac4e30998c6bd801a90404cea87d4b19b1d635d4ac1f05d
  • Pointer size: 132 Bytes
  • Size of remote file: 1.41 MB
assets/method-light.png ADDED

Git LFS Details

  • SHA256: b14f1172ea43f7d1472a1ea3f66d5ca35213f3edbbc6f226da21d4042d0163df
  • Pointer size: 132 Bytes
  • Size of remote file: 1.41 MB
assets/teaser.jpg ADDED

Git LFS Details

  • SHA256: 5749bd76be8ab26451142fbb70d2bb09fbf45132bee132d6b28e65074b20ad50
  • Pointer size: 132 Bytes
  • Size of remote file: 2.59 MB
assets/tradeoff-dark.png ADDED

Git LFS Details

  • SHA256: 23c7e2ddc00e9e9ce414e230dad8a48661a770efa9443ba3f0e03be5bdf93754
  • Pointer size: 131 Bytes
  • Size of remote file: 145 kB
assets/tradeoff-light.png ADDED

Git LFS Details

  • SHA256: c7ab808cfc784d61aa45b3e532b2f107e2609cc8e723cd15d035f9b6bc1a769c
  • Pointer size: 131 Bytes
  • Size of remote file: 145 kB