README.md

---
library_name: pytorch
license: apache-2.0
tags:
- foundation
- android
pipeline_tag: object-detection

---

![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/web-assets/model_demo.png)

# OWL-ViT: Optimized for Qualcomm Devices

OWL-ViT (Open-World Localization with Vision Transformers) is an open-vocabulary object detector that uses a CLIP-based ViT-B/32 backbone. Given an image and one or more free-form text queries, the model predicts bounding boxes and confidence scores for each query.

This is based on the implementation of OWL-ViT found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).
This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.61.0/src/qai_hub_models/models/owl_vit) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).

Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.

## Getting Started
There are two ways to deploy this model on your device:

### Option 1: Download Pre-Exported Models

Below are pre-exported model assets ready for deployment.

| Runtime | Precision | Chipset | SDK Versions | Download |
|---|---|---|---|---|
| ONNX | w8a16 | Universal | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.61.0/owl_vit-onnx-w8a16.zip)
| QNN_DLC | float | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.61.0/owl_vit-qnn_dlc-float.zip)
| QNN_DLC | w8a16 | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.61.0/owl_vit-qnn_dlc-w8a16.zip)
| TFLITE | float | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.61.0/owl_vit-tflite-float.zip)

For more device-specific assets and performance metrics, visit **[OWL-ViT on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/owl_vit)**.


### Option 2: Export with Custom Configurations

Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.61.0/src/qai_hub_models/models/owl_vit) Python library to compile and export the model with your own:
- Custom weights (e.g., fine-tuned checkpoints)
- Custom input shapes
- Target device and runtime configurations

This option is ideal if you need to customize the model beyond the default configuration provided here.

See our repository for [OWL-ViT on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.61.0/src/qai_hub_models/models/owl_vit) for usage instructions.

## Model Details

**Model Type:** Model_use_case.object_detection

**Model Stats:**
- Input image resolution: 768x768
- Model checkpoint: google/owlvit-base-patch32
- Model size (float): 613 MB
- Number of parameters: 153M

## Performance Summary
| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
|---|---|---|---|---|---|---
| OWL-ViT | ONNX | w8a16 | Snapdragon® X2 Elite | 8.322 ms | 3 - 3 MB | NPU
| OWL-ViT | ONNX | w8a16 | Snapdragon® X Elite | 21.246 ms | 174 - 174 MB | NPU
| OWL-ViT | ONNX | w8a16 | Snapdragon® 8 Gen 3 Mobile | 14.26 ms | 4 - 744 MB | NPU
| OWL-ViT | ONNX | w8a16 | Snapdragon® 8 Gen 1 Mobile | 35.229 ms | 4 - 762 MB | NPU
| OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS6490 | 128.381 ms | 3 - 9 MB | NPU
| OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 19.264 ms | 1 - 8 MB | NPU
| OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 20.329 ms | 0 - 192 MB | NPU
| OWL-ViT | ONNX | w8a16 | Qualcomm® QCS8450 | 35.229 ms | 4 - 762 MB | NPU
| OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 20.619 ms | 0 - 6 MB | NPU
| OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 21.246 ms | 174 - 174 MB | NPU
| OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-6690 | 198.344 ms | 6 - 791 MB | NPU
| OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-7790 | 34.762 ms | 6 - 583 MB | NPU
| OWL-ViT | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 11.459 ms | 1 - 665 MB | NPU
| OWL-ViT | ONNX | w8a16 | Snapdragon® 8 Elite Mobile | 11.459 ms | 1 - 665 MB | NPU
| OWL-ViT | ONNX | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 7.733 ms | 0 - 635 MB | NPU
| OWL-ViT | ONNX | w8a16 | Snapdragon® 7 Gen 4 Mobile | 34.762 ms | 6 - 583 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® X2 Elite | 19.448 ms | 7 - 7 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® X Elite | 45.551 ms | 7 - 7 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 33.388 ms | 7 - 664 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® 8 Gen 1 Mobile | 62.97 ms | 7 - 643 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-8275 | 54.253 ms | 7 - 16 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-8275 | 135.045 ms | 3 - 485 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 45.535 ms | 7 - 10 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® SA8775P | 51.997 ms | 2 - 483 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® SA8650P | 51.997 ms | 2 - 483 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® SA8255P | 51.997 ms | 2 - 483 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® QCS8450 | 62.97 ms | 7 - 643 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-9075 | 52.66 ms | 7 - 15 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-X7181 | 45.551 ms | 7 - 7 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ Q-8750 | 27.135 ms | 0 - 471 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® SA7255P | 135.045 ms | 3 - 485 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® SA8295P | 60.462 ms | 0 - 462 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® 8 Elite Mobile | 27.135 ms | 0 - 471 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 20.246 ms | 7 - 487 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Snapdragon® X2 Elite | 23.457 ms | 3 - 3 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Snapdragon® X Elite | 86.563 ms | 3 - 3 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Snapdragon® 8 Gen 3 Mobile | 80.149 ms | 4 - 989 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 94.251 ms | 1 - 7 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 145.296 ms | 1 - 749 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 98.204 ms | 4 - 5 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® SA8775P | 95.854 ms | 1 - 749 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® SA8650P | 95.854 ms | 1 - 749 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® SA8255P | 95.854 ms | 1 - 749 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 93.666 ms | 3 - 8 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 86.563 ms | 3 - 3 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 69.654 ms | 3 - 740 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Qualcomm® SA7255P | 145.296 ms | 1 - 749 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Snapdragon® 8 Elite Mobile | 69.654 ms | 3 - 740 MB | NPU
| OWL-ViT | QNN_DLC | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 21.867 ms | 3 - 771 MB | NPU
| OWL-ViT | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 23.55 ms | 0 - 642 MB | NPU
| OWL-ViT | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 48.493 ms | 0 - 634 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ IQ-8275 | 41.844 ms | 0 - 311 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ IQ-8275 | 116.255 ms | 0 - 485 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 32.292 ms | 0 - 4 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® SA8775P | 39.311 ms | 0 - 471 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® SA8650P | 39.311 ms | 0 - 471 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® SA8255P | 39.311 ms | 0 - 471 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® QCS8450 | 48.493 ms | 0 - 634 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 39.987 ms | 0 - 310 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 17.785 ms | 0 - 474 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® SA7255P | 116.255 ms | 0 - 485 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® SA8295P | 48.014 ms | 0 - 460 MB | NPU
| OWL-ViT | TFLITE | float | Snapdragon® 8 Elite Mobile | 17.785 ms | 0 - 474 MB | NPU
| OWL-ViT | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 12.465 ms | 0 - 497 MB | NPU

## License
* The license for the original implementation of OWL-ViT can be found
  [here](https://github.com/google-research/scenic/blob/main/LICENSE).

## References
* [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230)
* [Source Model Implementation](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit)

## Community
* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).