Add ExecuTorch WebGPU program for depth-anything-v2-small
Browse files
NOTICE
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
This repository contains a modified version of depth-anything/Depth-Anything-V2-Small-hf.
|
| 2 |
|
| 3 |
Modifications by Meta Platforms, Inc.: conversion to the ExecuTorch program format (.pte) for the
|
| 4 |
-
ExecuTorch WebGPU backend
|
| 5 |
|
| 6 |
The original work and these modifications are distributed under the license in LICENSE.
|
|
|
|
| 1 |
This repository contains a modified version of depth-anything/Depth-Anything-V2-Small-hf.
|
| 2 |
|
| 3 |
Modifications by Meta Platforms, Inc.: conversion to the ExecuTorch program format (.pte) for the
|
| 4 |
+
ExecuTorch WebGPU backend.
|
| 5 |
|
| 6 |
The original work and these modifications are distributed under the license in LICENSE.
|
README.md
CHANGED
|
@@ -2,9 +2,9 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
- depth-anything/Depth-Anything-V2-Small-hf
|
| 5 |
-
base_model_relation: quantized
|
| 6 |
library_name: executorch
|
| 7 |
pipeline_tag: depth-estimation
|
|
|
|
| 8 |
tags:
|
| 9 |
- executorch
|
| 10 |
- webgpu
|
|
@@ -15,8 +15,7 @@ tags:
|
|
| 15 |
|
| 16 |
This repository contains an [ExecuTorch](https://github.com/pytorch/executorch) conversion of
|
| 17 |
[depth-anything/Depth-Anything-V2-Small-hf](https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf) for the ExecuTorch **WebGPU** backend, so the model runs entirely in a web browser on
|
| 18 |
-
the visitor's GPU. It powers the corresponding demo in the
|
| 19 |
-
ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)).
|
| 20 |
|
| 21 |
- **Base model:** [depth-anything/Depth-Anything-V2-Small-hf](https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf)
|
| 22 |
- **Format:** ExecuTorch `.pte` programs delegated to the WebGPU backend
|
|
@@ -25,6 +24,32 @@ ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executor
|
|
| 25 |
|
| 26 |
> **Note:** This is the Small variant, which is licensed Apache-2.0 (the larger Depth Anything V2 variants use a different license).
|
| 27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
## Files
|
| 29 |
|
| 30 |
| File | Size | SHA-256 |
|
|
@@ -35,10 +60,13 @@ Full checksums are in `manifest.json`. The playground verifies them after downlo
|
|
| 35 |
|
| 36 |
## Running it
|
| 37 |
|
| 38 |
-
These programs
|
| 39 |
-
([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu))
|
| 40 |
-
|
| 41 |
-
|
|
|
|
|
|
|
|
|
|
| 42 |
|
| 43 |
## Provenance
|
| 44 |
|
|
@@ -47,8 +75,23 @@ which downloads these files once, caches them in the browser, and runs everythin
|
|
| 47 |
- **Reproduce:** `python recipes/depth-anything-v2-small.py --out out/depth-anything-v2-small` (recipe in `export/`)
|
| 48 |
- Re-running the recipe produces byte-identical files.
|
| 49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
## License
|
| 51 |
|
| 52 |
This is a derivative of the upstream model and is distributed under the same terms
|
| 53 |
(`apache-2.0`); see `LICENSE`.
|
| 54 |
-
Modifications (conversion to the ExecuTorch format
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
- depth-anything/Depth-Anything-V2-Small-hf
|
|
|
|
| 5 |
library_name: executorch
|
| 6 |
pipeline_tag: depth-estimation
|
| 7 |
+
inference: false
|
| 8 |
tags:
|
| 9 |
- executorch
|
| 10 |
- webgpu
|
|
|
|
| 15 |
|
| 16 |
This repository contains an [ExecuTorch](https://github.com/pytorch/executorch) conversion of
|
| 17 |
[depth-anything/Depth-Anything-V2-Small-hf](https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf) for the ExecuTorch **WebGPU** backend, so the model runs entirely in a web browser on
|
| 18 |
+
the visitor's GPU. It powers the corresponding demo in the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)).
|
|
|
|
| 19 |
|
| 20 |
- **Base model:** [depth-anything/Depth-Anything-V2-Small-hf](https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf)
|
| 21 |
- **Format:** ExecuTorch `.pte` programs delegated to the WebGPU backend
|
|
|
|
| 24 |
|
| 25 |
> **Note:** This is the Small variant, which is licensed Apache-2.0 (the larger Depth Anything V2 variants use a different license).
|
| 26 |
|
| 27 |
+
## About the original model
|
| 28 |
+
|
| 29 |
+
The Small variant of Depth Anything V2, a monocular depth estimation model trained on about 595K synthetic labeled images and 62M+ real unlabeled images.
|
| 30 |
+
|
| 31 |
+
- **Developed by:** Lihe Yang et al. (HKU and TikTok)
|
| 32 |
+
- **Paper:** [Depth Anything V2](https://arxiv.org/abs/2406.09414)
|
| 33 |
+
- 24.8M parameters
|
| 34 |
+
- Predicts relative depth from a single image
|
| 35 |
+
|
| 36 |
+
For training data, evaluation results and full details, see the original model card: [depth-anything/Depth-Anything-V2-Small-hf](https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf).
|
| 37 |
+
|
| 38 |
+
## Intended use and limitations
|
| 39 |
+
|
| 40 |
+
Relative depth maps for photos and camera frames.
|
| 41 |
+
|
| 42 |
+
Outputs relative (not metric) depth. Accuracy drops on reflective or transparent surfaces and unusual scenes.
|
| 43 |
+
|
| 44 |
+
## What changed in this conversion
|
| 45 |
+
|
| 46 |
+
- Fixed 224x224 input (upstream processes larger images)
|
| 47 |
+
|
| 48 |
+
## Inputs and outputs
|
| 49 |
+
|
| 50 |
+
- Input `[1, 3, 224, 224]` float32 RGB, ImageNet-normalized: `(x / 255 - mean) / std`, mean 0.485/0.456/0.406, std 0.229/0.224/0.225
|
| 51 |
+
- Output `[1, 224, 224]` float32 relative inverse depth (larger means closer)
|
| 52 |
+
|
| 53 |
## Files
|
| 54 |
|
| 55 |
| File | Size | SHA-256 |
|
|
|
|
| 60 |
|
| 61 |
## Running it
|
| 62 |
|
| 63 |
+
These programs were exported with the ExecuTorch WebGPU backend
|
| 64 |
+
([backends/webgpu](https://github.com/pytorch/executorch/tree/main/backends/webgpu)) and validated with the
|
| 65 |
+
WebAssembly runtime build used by the ExecuTorch WebGPU Playground (launching soon on [executorch.ai](https://executorch.ai)). That runtime build is not published yet; until it is,
|
| 66 |
+
the playground is the supported way to run them. It downloads these files once, caches them in the browser,
|
| 67 |
+
and runs everything locally.
|
| 68 |
+
|
| 69 |
+
Requirements: a browser with WebGPU enabled.
|
| 70 |
|
| 71 |
## Provenance
|
| 72 |
|
|
|
|
| 75 |
- **Reproduce:** `python recipes/depth-anything-v2-small.py --out out/depth-anything-v2-small` (recipe in `export/`)
|
| 76 |
- Re-running the recipe produces byte-identical files.
|
| 77 |
|
| 78 |
+
## Citation
|
| 79 |
+
|
| 80 |
+
If you use this model, please cite the original work:
|
| 81 |
+
|
| 82 |
+
```bibtex
|
| 83 |
+
@misc{yang2024depth,
|
| 84 |
+
title={Depth Anything V2},
|
| 85 |
+
author={Lihe Yang and Bingyi Kang and Zilong Huang and Zhen Zhao and Xiaogang Xu and Jiashi Feng and Hengshuang Zhao},
|
| 86 |
+
year={2024},
|
| 87 |
+
eprint={2406.09414},
|
| 88 |
+
archivePrefix={arXiv},
|
| 89 |
+
primaryClass={cs.CV}
|
| 90 |
+
}
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
## License
|
| 94 |
|
| 95 |
This is a derivative of the upstream model and is distributed under the same terms
|
| 96 |
(`apache-2.0`); see `LICENSE`.
|
| 97 |
+
Modifications (conversion to the ExecuTorch format) by Meta Platforms, Inc.
|