Hugging Face's logo

qualcomm
/
OWL-ViT

Object Detection
PyTorch
foundation
android
Model card Files
xet
 Community
qaihm-bot commited on
Commit
d8da5d5
·
verified ·
1 Parent(s): 620ff80

v0.60.0

Browse files

See https://github.com/qualcomm/ai-hub-models/releases/v0.60.0 for changelog.

Files changed (3) hide show
  1. LICENSE +1 -0
  2. README.md +138 -0
  3. release_assets.json +38 -0
LICENSE ADDED
@@ -0,0 +1 @@
 
 
1
+ The license of the original trained model can be found at https://github.com/google-research/scenic/blob/main/LICENSE.
README.md ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: pytorch
3
+ license: other
4
+ tags:
5
+ - foundation
6
+ - android
7
+ pipeline_tag: object-detection
8
+
9
+ ---
10
+
11
+ ![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/web-assets/model_demo.png)
12
+
13
+ # OWL-ViT: Optimized for Qualcomm Devices
14
+
15
+ OWL-ViT (Open-World Localization with Vision Transformers) is an open-vocabulary object detector that uses a CLIP-based ViT-B/32 backbone. Given an image and one or more free-form text queries, the model predicts bounding boxes and confidence scores for each query.
16
+
17
+ This is based on the implementation of OWL-ViT found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).
18
+ This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.60.0/src/qai_hub_models/models/owl_vit) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).
19
+
20
+ Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.
21
+
22
+ ## Getting Started
23
+ There are two ways to deploy this model on your device:
24
+
25
+ ### Option 1: Download Pre-Exported Models
26
+
27
+ Below are pre-exported model assets ready for deployment.
28
+
29
+ | Runtime | Precision | Chipset | SDK Versions | Download |
30
+ |---|---|---|---|---|
31
+ | ONNX | w8a16 | Universal | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.60.0/owl_vit-onnx-w8a16.zip)
32
+ | QNN_DLC | float | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.60.0/owl_vit-qnn_dlc-float.zip)
33
+ | QNN_DLC | w8a16 | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.60.0/owl_vit-qnn_dlc-w8a16.zip)
34
+ | TFLITE | float | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.60.0/owl_vit-tflite-float.zip)
35
+
36
+ For more device-specific assets and performance metrics, visit **[OWL-ViT on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/owl_vit)**.
37
+
38
+
39
+ ### Option 2: Export with Custom Configurations
40
+
41
+ Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.60.0/src/qai_hub_models/models/owl_vit) Python library to compile and export the model with your own:
42
+ - Custom weights (e.g., fine-tuned checkpoints)
43
+ - Custom input shapes
44
+ - Target device and runtime configurations
45
+
46
+ This option is ideal if you need to customize the model beyond the default configuration provided here.
47
+
48
+ See our repository for [OWL-ViT on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.60.0/src/qai_hub_models/models/owl_vit) for usage instructions.
49
+
50
+ ## Model Details
51
+
52
+ **Model Type:** Model_use_case.object_detection
53
+
54
+ **Model Stats:**
55
+ - Input image resolution: 768x768
56
+ - Model checkpoint: google/owlvit-base-patch32
57
+ - Model size (float): 613 MB
58
+ - Number of parameters: 153M
59
+
60
+ ## Performance Summary
61
+ | Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
62
+ |---|---|---|---|---|---|---
63
+ | OWL-ViT | ONNX | w8a16 | Snapdragon® X2 Elite | 8.32 ms | 3 - 3 MB | NPU
64
+ | OWL-ViT | ONNX | w8a16 | Snapdragon® X Elite | 20.932 ms | 174 - 174 MB | NPU
65
+ | OWL-ViT | ONNX | w8a16 | Snapdragon® 8 Gen 3 Mobile | 14.312 ms | 0 - 740 MB | NPU
66
+ | OWL-ViT | ONNX | w8a16 | Snapdragon® 8 Gen 1 Mobile | 35.695 ms | 4 - 762 MB | NPU
67
+ | OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS6490 | 128.395 ms | 3 - 9 MB | NPU
68
+ | OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 19.154 ms | 1 - 8 MB | NPU
69
+ | OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 20.312 ms | 0 - 192 MB | NPU
70
+ | OWL-ViT | ONNX | w8a16 | Qualcomm® QCS8450 | 35.695 ms | 4 - 762 MB | NPU
71
+ | OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 24.091 ms | 2 - 8 MB | NPU
72
+ | OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 20.932 ms | 174 - 174 MB | NPU
73
+ | OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-6690 | 214.248 ms | 0 - 783 MB | NPU
74
+ | OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-7790 | 34.955 ms | 6 - 582 MB | NPU
75
+ | OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 11.447 ms | 0 - 663 MB | NPU
76
+ | OWL-ViT | ONNX | w8a16 | Snapdragon® 8 Elite Mobile | 11.447 ms | 0 - 663 MB | NPU
77
+ | OWL-ViT | ONNX | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 7.735 ms | 0 - 635 MB | NPU
78
+ | OWL-ViT | ONNX | w8a16 | Snapdragon® 7 Gen 4 Mobile | 34.955 ms | 6 - 582 MB | NPU
79
+ | OWL-ViT | QNN_DLC | float | Snapdragon® X2 Elite | 19.285 ms | 7 - 7 MB | NPU
80
+ | OWL-ViT | QNN_DLC | float | Snapdragon® X Elite | 45.544 ms | 7 - 7 MB | NPU
81
+ | OWL-ViT | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 33.335 ms | 4 - 659 MB | NPU
82
+ | OWL-ViT | QNN_DLC | float | Snapdragon® 8 Gen 1 Mobile | 63.06 ms | 7 - 645 MB | NPU
83
+ | OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-8275 | 53.98 ms | 7 - 16 MB | NPU
84
+ | OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-8275 | 135.024 ms | 2 - 485 MB | NPU
85
+ | OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 45.526 ms | 7 - 810 MB | NPU
86
+ | OWL-ViT | QNN_DLC | float | Qualcomm® SA8775P | 52.015 ms | 2 - 484 MB | NPU
87
+ | OWL-ViT | QNN_DLC | float | Qualcomm® SA8650P | 52.015 ms | 2 - 484 MB | NPU
88
+ | OWL-ViT | QNN_DLC | float | Qualcomm® SA8255P | 52.015 ms | 2 - 484 MB | NPU
89
+ | OWL-ViT | QNN_DLC | float | Qualcomm® QCS8450 | 63.06 ms | 7 - 645 MB | NPU
90
+ | OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-9075 | 51.802 ms | 9 - 17 MB | NPU
91
+ | OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-X7181 | 45.544 ms | 7 - 7 MB | NPU
92
+ | OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ Q-8750 | 26.973 ms | 0 - 472 MB | NPU
93
+ | OWL-ViT | QNN_DLC | float | Qualcomm® SA7255P | 135.024 ms | 2 - 485 MB | NPU
94
+ | OWL-ViT | QNN_DLC | float | Qualcomm® SA8295P | 60.616 ms | 1 - 462 MB | NPU
95
+ | OWL-ViT | QNN_DLC | float | Snapdragon® 8 Elite Mobile | 26.973 ms | 0 - 472 MB | NPU
96
+ | OWL-ViT | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 20.271 ms | 7 - 487 MB | NPU
97
+ | OWL-ViT | QNN_DLC | w8a16 | Snapdragon® X2 Elite | 23.468 ms | 3 - 3 MB | NPU
98
+ | OWL-ViT | QNN_DLC | w8a16 | Snapdragon® X Elite | 86.383 ms | 3 - 3 MB | NPU
99
+ | OWL-ViT | QNN_DLC | w8a16 | Snapdragon® 8 Gen 3 Mobile | 80.117 ms | 4 - 989 MB | NPU
100
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 94.872 ms | 3 - 9 MB | NPU
101
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 145.08 ms | 1 - 749 MB | NPU
102
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 96.977 ms | 3 - 5 MB | NPU
103
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® SA8775P | 96.013 ms | 0 - 747 MB | NPU
104
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® SA8650P | 96.013 ms | 0 - 747 MB | NPU
105
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® SA8255P | 96.013 ms | 0 - 747 MB | NPU
106
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 93.754 ms | 3 - 8 MB | NPU
107
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 86.383 ms | 3 - 3 MB | NPU
108
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 68.137 ms | 3 - 740 MB | NPU
109
+ | OWL-ViT | QNN_DLC | w8a16 | Qualcomm® SA7255P | 145.08 ms | 1 - 749 MB | NPU
110
+ | OWL-ViT | QNN_DLC | w8a16 | Snapdragon® 8 Elite Mobile | 68.137 ms | 3 - 740 MB | NPU
111
+ | OWL-ViT | QNN_DLC | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 21.851 ms | 3 - 771 MB | NPU
112
+ | OWL-ViT | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 23.339 ms | 0 - 637 MB | NPU
113
+ | OWL-ViT | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 48.576 ms | 0 - 634 MB | NPU
114
+ | OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ IQ-8275 | 41.61 ms | 0 - 311 MB | NPU
115
+ | OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ IQ-8275 | 116.202 ms | 0 - 485 MB | NPU
116
+ | OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 32.764 ms | 0 - 4 MB | NPU
117
+ | OWL-ViT | TFLITE | float | Qualcomm® SA8775P | 39.287 ms | 0 - 471 MB | NPU
118
+ | OWL-ViT | TFLITE | float | Qualcomm® SA8650P | 39.287 ms | 0 - 471 MB | NPU
119
+ | OWL-ViT | TFLITE | float | Qualcomm® SA8255P | 39.287 ms | 0 - 471 MB | NPU
120
+ | OWL-ViT | TFLITE | float | Qualcomm® QCS8450 | 48.576 ms | 0 - 634 MB | NPU
121
+ | OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 44.952 ms | 0 - 310 MB | NPU
122
+ | OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 17.856 ms | 0 - 479 MB | NPU
123
+ | OWL-ViT | TFLITE | float | Qualcomm® SA7255P | 116.202 ms | 0 - 485 MB | NPU
124
+ | OWL-ViT | TFLITE | float | Qualcomm® SA8295P | 48.103 ms | 0 - 460 MB | NPU
125
+ | OWL-ViT | TFLITE | float | Snapdragon® 8 Elite Mobile | 17.856 ms | 0 - 479 MB | NPU
126
+ | OWL-ViT | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 12.474 ms | 0 - 496 MB | NPU
127
+
128
+ ## License
129
+ * The license for the original implementation of OWL-ViT can be found
130
+ [here](https://github.com/google-research/scenic/blob/main/LICENSE).
131
+
132
+ ## References
133
+ * [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230)
134
+ * [Source Model Implementation](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit)
135
+
136
+ ## Community
137
+ * Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
138
+ * For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).
release_assets.json ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "version": "0.60.0",
3
+ "precisions": {
4
+ "float": {
5
+ "universal_assets": {
6
+ "qnn_dlc": {
7
+ "tool_versions": {
8
+ "qairt": "2.45.0.260326154327"
9
+ },
10
+ "download_url": "https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.60.0/owl_vit-qnn_dlc-float.zip"
11
+ },
12
+ "tflite": {
13
+ "tool_versions": {
14
+ "qairt": "2.45.0.260326154327"
15
+ },
16
+ "download_url": "https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.60.0/owl_vit-tflite-float.zip"
17
+ }
18
+ }
19
+ },
20
+ "w8a16": {
21
+ "universal_assets": {
22
+ "onnx": {
23
+ "tool_versions": {
24
+ "qairt": "2.45.0.260326154327",
25
+ "onnx_runtime": "1.27.1"
26
+ },
27
+ "download_url": "https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.60.0/owl_vit-onnx-w8a16.zip"
28
+ },
29
+ "qnn_dlc": {
30
+ "tool_versions": {
31
+ "qairt": "2.45.0.260326154327"
32
+ },
33
+ "download_url": "https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.60.0/owl_vit-qnn_dlc-w8a16.zip"
34
+ }
35
+ }
36
+ }
37
+ }
38
+ }