Commit ·
32875bd
0
Parent(s):
init
Browse filesCo-authored-by: amael-apple <amael-apple@users.noreply.huggingface.co>
Co-authored-by: patricejaton <patricejaton@users.noreply.huggingface.co>
- .gitattributes +35 -0
- LICENSE +88 -0
- README.md +85 -0
- config.json +23 -0
- model.safetensors +3 -0
.gitattributes
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
LICENSE
ADDED
|
@@ -0,0 +1,88 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Disclaimer: IMPORTANT: This Apple Machine Learning Research Model is
|
| 2 |
+
specifically developed and released by Apple Inc. ("Apple") for the sole purpose
|
| 3 |
+
of scientific research of artificial intelligence and machine-learning
|
| 4 |
+
technology. “Apple Machine Learning Research Model” means the model, including
|
| 5 |
+
but not limited to algorithms, formulas, trained model weights, parameters,
|
| 6 |
+
configurations, checkpoints, and any related materials (including
|
| 7 |
+
documentation).
|
| 8 |
+
|
| 9 |
+
This Apple Machine Learning Research Model is provided to You by
|
| 10 |
+
Apple in consideration of your agreement to the following terms, and your use,
|
| 11 |
+
modification, creation of Model Derivatives, and or redistribution of the Apple
|
| 12 |
+
Machine Learning Research Model constitutes acceptance of this Agreement. If You
|
| 13 |
+
do not agree with these terms, please do not use, modify, create Model
|
| 14 |
+
Derivatives of, or distribute this Apple Machine Learning Research Model or
|
| 15 |
+
Model Derivatives.
|
| 16 |
+
|
| 17 |
+
* License Scope: In consideration of your agreement to abide by the following
|
| 18 |
+
terms, and subject to these terms, Apple hereby grants you a personal,
|
| 19 |
+
non-exclusive, worldwide, non-transferable, royalty-free, revocable, and
|
| 20 |
+
limited license, to use, copy, modify, distribute, and create Model
|
| 21 |
+
Derivatives (defined below) of the Apple Machine Learning Research Model
|
| 22 |
+
exclusively for Research Purposes. You agree that any Model Derivatives You
|
| 23 |
+
may create or that may be created for You will be limited to Research Purposes
|
| 24 |
+
as well. “Research Purposes” means non-commercial scientific research and
|
| 25 |
+
academic development activities, such as experimentation, analysis, testing
|
| 26 |
+
conducted by You with the sole intent to advance scientific knowledge and
|
| 27 |
+
research. “Research Purposes” does not include any commercial exploitation,
|
| 28 |
+
product development or use in any commercial product or service.
|
| 29 |
+
|
| 30 |
+
* Distribution of Apple Machine Learning Research Model and Model Derivatives:
|
| 31 |
+
If you choose to redistribute Apple Machine Learning Research Model or its
|
| 32 |
+
Model Derivatives, you must provide a copy of this Agreement to such third
|
| 33 |
+
party, and ensure that the following attribution notice be provided: “Apple
|
| 34 |
+
Machine Learning Research Model is licensed under the Apple Machine Learning
|
| 35 |
+
Research Model License Agreement.” Additionally, all Model Derivatives must
|
| 36 |
+
clearly be identified as such, including disclosure of modifications and
|
| 37 |
+
changes made to the Apple Machine Learning Research Model. The name,
|
| 38 |
+
trademarks, service marks or logos of Apple may not be used to endorse or
|
| 39 |
+
promote Model Derivatives or the relationship between You and Apple. “Model
|
| 40 |
+
Derivatives” means any models or any other artifacts created by modifications,
|
| 41 |
+
improvements, adaptations, alterations to the architecture, algorithm or
|
| 42 |
+
training processes of the Apple Machine Learning Research Model, or by any
|
| 43 |
+
retraining, fine-tuning of the Apple Machine Learning Research Model.
|
| 44 |
+
|
| 45 |
+
* No Other License: Except as expressly stated in this notice, no other rights
|
| 46 |
+
or licenses, express or implied, are granted by Apple herein, including but
|
| 47 |
+
not limited to any patent, trademark, and similar intellectual property rights
|
| 48 |
+
worldwide that may be infringed by the Apple Machine Learning Research Model,
|
| 49 |
+
the Model Derivatives or by other works in which the Apple Machine Learning
|
| 50 |
+
Research Model may be incorporated.
|
| 51 |
+
|
| 52 |
+
* Compliance with Laws: Your use of Apple Machine Learning Research Model must
|
| 53 |
+
be in compliance with all applicable laws and regulations.
|
| 54 |
+
|
| 55 |
+
* Term and Termination: The term of this Agreement will begin upon your
|
| 56 |
+
acceptance of this Agreement or use of the Apple Machine Learning Research
|
| 57 |
+
Model and will continue until terminated in accordance with the following
|
| 58 |
+
terms. Apple may terminate this Agreement at any time if You are in breach of
|
| 59 |
+
any term or condition of this Agreement. Upon termination of this Agreement,
|
| 60 |
+
You must cease to use all Apple Machine Learning Research Models and Model
|
| 61 |
+
Derivatives and permanently delete any copy thereof. Sections 3, 6 and 7 will
|
| 62 |
+
survive termination.
|
| 63 |
+
|
| 64 |
+
* Disclaimer and Limitation of Liability: This Apple Machine Learning Research
|
| 65 |
+
Model and any outputs generated by the Apple Machine Learning Research Model
|
| 66 |
+
are provided on an “AS IS” basis. APPLE MAKES NO WARRANTIES, EXPRESS OR
|
| 67 |
+
IMPLIED, INCLUDING WITHOUT LIMITATION THE IMPLIED WARRANTIES OF
|
| 68 |
+
NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE,
|
| 69 |
+
REGARDING THE APPLE MACHINE LEARNING RESEARCH MODEL OR OUTPUTS GENERATED BY
|
| 70 |
+
THE APPLE MACHINE LEARNING RESEARCH MODEL. You are solely responsible for
|
| 71 |
+
determining the appropriateness of using or redistributing the Apple Machine
|
| 72 |
+
Learning Research Model and any outputs of the Apple Machine Learning Research
|
| 73 |
+
Model and assume any risks associated with Your use of the Apple Machine
|
| 74 |
+
Learning Research Model and any output and results. IN NO EVENT SHALL APPLE BE
|
| 75 |
+
LIABLE FOR ANY SPECIAL, INDIRECT, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
|
| 76 |
+
IN ANY WAY OUT OF THE USE, REPRODUCTION, MODIFICATION AND/OR DISTRIBUTION OF
|
| 77 |
+
THE APPLE MACHINE LEARNING RESEARCH MODEL AND ANY OUTPUTS OF THE APPLE MACHINE
|
| 78 |
+
LEARNING RESEARCH MODEL, HOWEVER CAUSED AND WHETHER UNDER THEORY OF CONTRACT,
|
| 79 |
+
TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY OR OTHERWISE, EVEN IF APPLE HAS
|
| 80 |
+
BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
| 81 |
+
|
| 82 |
+
* Governing Law: This Agreement will be governed by and construed under the laws
|
| 83 |
+
of the State of California without regard to its choice of law principles. The
|
| 84 |
+
Convention on Contracts for the International Sale of Goods shall not apply to
|
| 85 |
+
the Agreement except that the arbitration clause and any arbitration hereunder
|
| 86 |
+
shall be governed by the Federal Arbitration Act, Chapters 1 and 2.
|
| 87 |
+
|
| 88 |
+
Copyright (C) 2026 Apple Inc. All Rights Reserved.
|
README.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: apple-amlr
|
| 4 |
+
license_link: https://github.com/apple-aiml-research/ml-depth-pro-2/blob/main/LICENSE_MODEL
|
| 5 |
+
pipeline_tag: depth-estimation
|
| 6 |
+
library_name: depth-pro-2
|
| 7 |
+
tags:
|
| 8 |
+
- depth-estimation
|
| 9 |
+
- metric-depth
|
| 10 |
+
- monocular-depth
|
| 11 |
+
- video-depth
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
<h1 align="center">Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth</h1>
|
| 15 |
+
|
| 16 |
+
<p align="center">
|
| 17 |
+
<a href="https://github.com/apple-aiml-research/ml-depth-pro-2"><img src="https://img.shields.io/badge/GitHub-181717?style=flat-square&logo=github&logoColor=white" alt="GitHub"/></a>
|
| 18 |
+
<a href="https://arxiv.org/abs/TODO"><img src="https://img.shields.io/badge/Technical%20Report-B31B1B?style=flat-square&logo=arxiv&logoColor=white" alt="Technical Report"/></a>
|
| 19 |
+
<a href="https://apple-aiml-research.github.io/ml-depth-pro-2/"><img src="https://img.shields.io/badge/Project%20Page-666666?style=flat-square&logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJ3aGl0ZSIgc3Ryb2tlLXdpZHRoPSIyIj48Y2lyY2xlIGN4PSIxMiIgY3k9IjEyIiByPSIxMCIvPjxwYXRoIGQ9Ik0yIDEyaDIwTTEyIDJhMTUgMTUgMCAwIDEgMCAyME0xMiAyYTE1IDE1IDAgMCAwIDAgMjAiLz48L3N2Zz4%3D" alt="Project Page"/></a>
|
| 20 |
+
<a href="https://huggingface.co/apple/DepthPro2"><img src="https://img.shields.io/badge/All%20Sizes-FFD21E?style=flat-square&logo=huggingface&logoColor=black" alt="All Sizes"/></a>
|
| 21 |
+
</p>
|
| 22 |
+
<h3 align="center">Model size S · 24M parameters</h3>
|
| 23 |
+
<!-- Hidden until the Space is public:
|
| 24 |
+
<h3 align="center">
|
| 25 |
+
🚀 <a href="https://huggingface.co/spaces/apple/DepthPro2">Try the Interactive Demo →</a>
|
| 26 |
+
</h3>
|
| 27 |
+
-->
|
| 28 |
+
|
| 29 |
+
| B200 | VGA | 720p | 1080p | 4K |
|
| 30 |
+
|:---|:---:|:---:|:---:|:---:|
|
| 31 |
+
| Throughput (fps) | 347.0 | 143.9 | 44.8 | 3.6 |
|
| 32 |
+
| Peak memory (GiB) | 1.9 | 3.8 | 8.0 | 31.9 |
|
| 33 |
+
|
| 34 |
+
<details>
|
| 35 |
+
<summary>Benchmark details</summary>
|
| 36 |
+
|
| 37 |
+
- **GPU:** NVIDIA B200, in bfloat16 and PyTorch eager mode (no compilation)
|
| 38 |
+
- **Input:** a 32-frame video clip, processed in one call at its native resolution
|
| 39 |
+
- **Resolutions:** VGA is 640 × 480; 720p, 1080p and 4K are 16:9
|
| 40 |
+
- **Throughput:** frames per second, averaged over the clip
|
| 41 |
+
- **Memory:** the peak GPU memory PyTorch reserves
|
| 42 |
+
- **Included:** the [focal length estimator](https://huggingface.co/apple/DepthPro2-focal), a separate 23.5M-parameter model that loads with the depth model
|
| 43 |
+
|
| 44 |
+
</details>
|
| 45 |
+
|
| 46 |
+
## How to use
|
| 47 |
+
|
| 48 |
+
```bash
|
| 49 |
+
pip install "git+https://github.com/apple-aiml-research/ml-depth-pro-2"
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
```python
|
| 53 |
+
from functools import partial
|
| 54 |
+
|
| 55 |
+
import torch
|
| 56 |
+
from depth_pro_2 import DepthPro2, load_rgb
|
| 57 |
+
from depth_pro_2.model import to_normalized_tensor
|
| 58 |
+
|
| 59 |
+
model = DepthPro2.from_pretrained("apple/DepthPro2-S", precision=torch.bfloat16)
|
| 60 |
+
model = model.to("cuda")
|
| 61 |
+
transform = partial(to_normalized_tensor, device=torch.device("cuda"))
|
| 62 |
+
|
| 63 |
+
image, f_px = load_rgb("photo.jpg") # f_px: EXIF focal length, or None
|
| 64 |
+
depth = model.infer(transform(image), f_px=f_px)["depth"] # meters, (H, W)
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
Videos, the other sizes and the command line are on the
|
| 68 |
+
[Depth Pro 2 page](https://huggingface.co/apple/DepthPro2).
|
| 69 |
+
|
| 70 |
+
## License
|
| 71 |
+
|
| 72 |
+
The weights are released under the [Apple Machine Learning Research Model license](https://github.com/apple-aiml-research/ml-depth-pro-2/blob/main/LICENSE_MODEL), for
|
| 73 |
+
research use only: non-commercial scientific research and academic development.
|
| 74 |
+
|
| 75 |
+
## Citation
|
| 76 |
+
|
| 77 |
+
```bibtex
|
| 78 |
+
@article{DepthPro2:2026:arxiv,
|
| 79 |
+
title = {Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth},
|
| 80 |
+
author = {Peiyun Hu and Marko Mihajlovic and Lyndon Duong and Lahav Lipson and Wei Dong and Ama\"{e}l Delaunoy and Stephan R. Richter and Vladlen Koltun},
|
| 81 |
+
journal = {arXiv preprint arXiv:TODO},
|
| 82 |
+
year = {2026},
|
| 83 |
+
url = {https://arxiv.org/abs/TODO},
|
| 84 |
+
}
|
| 85 |
+
```
|
config.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"backbone_preset": "vit_small_patch16_dinov3",
|
| 3 |
+
"cross_frame_blocks": [
|
| 4 |
+
5,
|
| 5 |
+
7,
|
| 6 |
+
9,
|
| 7 |
+
11
|
| 8 |
+
],
|
| 9 |
+
"dim_decoder": 64,
|
| 10 |
+
"dims_reassemble": [
|
| 11 |
+
48,
|
| 12 |
+
96,
|
| 13 |
+
192,
|
| 14 |
+
384
|
| 15 |
+
],
|
| 16 |
+
"fov": false,
|
| 17 |
+
"reassembled_blocks": [
|
| 18 |
+
2,
|
| 19 |
+
5,
|
| 20 |
+
8,
|
| 21 |
+
11
|
| 22 |
+
]
|
| 23 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:583c4d630779fd231f969c738733624963ad34bd3c29f9c64088257e3436f97c
|
| 3 |
+
size 97264508
|