| --- |
| library_name: pytorch |
| license: apache-2.0 |
| tags: |
| - foundation |
| - amd |
| - rocm |
| - zero-shot-image-classification |
| pipeline_tag: zero-shot-image-classification |
| --- |
| |
|  |
|
|
| # SigLIP: Optimized for AMD ROCm |
|
|
| SigLIP is a vision-language embedding model evaluated here for zero-shot image classification on CIFAR-10. This repository packages inference for zero-shot image classification using **PyTorch** (CPU/GPU) and **vLLM** (GPU pooling runner), exported and validated for **AMD ROCm** so it runs efficiently on AMD GPUs and CPUs. |
|
|
| This is based on the implementation of SigLIP found [here](https://huggingface.co/google/siglip-base-patch16-224). |
| This repository contains configurations and scripts optimized for **AMD® ROCm™** platforms. You can use the [siglip AMD scripts](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/siglip) to reproduce results or export with custom configurations. More details on model performance can be found [here](#accuracy-pipeline). |
|
|
| --- |
|
|
| ## Task Overview |
|
|
| **Task:** Zero-shot image classification |
|
|
| **Dataset:** CIFAR-10 test split (10,000 images, 10 classes) |
|
|
| **Output metrics:** Accuracy (%), image throughput (img/s) |
|
|
| > **Model variants:** Default is **base-224** (`google/siglip-base-patch16-224`). Override with `MODEL_VARIANT=base-256|base-384|large-256|large-384|so400m-224|so400m-384`. |
| |
| > **vLLM note:** vLLM's SigLIP integration handles one modality per request (text batch, then one image per request); the client combines embeddings manually — the same approach as the upstream evaluation scripts. |
| |
| --- |
| |
| ## AMD ROCm Optimization |
| |
| This model export has been adapted and validated for **AMD Instinct™ / Radeon™ GPUs** running **ROCm**, as well as AMD CPUs. Key points: |
| |
| - Validated backends: **PyTorch** (native ROCm HIP kernels, CPU and GPU) and **vLLM** (ROCm-enabled, built from source, pooling runner on GPU). |
| - No code changes required versus the upstream SigLIP implementation — only environment/runtime configuration differs. |
| - CPU fallback path supported via PyTorch for environments without a ROCm-capable GPU. |
| |
| | Runtime | Precision | Backend | Hardware | Notes | |
| |---|---|---|---|---| |
| | PyTorch | FP32 / FP16 / BF16 | HIP (ROCm) | AMD CPU | — | |
| | PyTorch | FP32 / FP16 / BF16 | HIP (ROCm) | AMD Instinct™ / Radeon™ GPU | — | |
| | vLLM | FP32 / FP16 / BF16 | ROCm server (pooling runner) | AMD Instinct™ / Radeon™ GPU | One modality per request; CPU client scores embeddings | |
| |
| --- |
| |
| ## Getting Started |
| |
| For setup instructions, evaluation scripts, and custom configuration options, see the [siglip on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/siglip). |
| |
| --- |
| |
| ## Model Details |
| |
| **Model Type:** Vision-language embedding model (zero-shot image classification) |
| |
| **Base Model:** `google/siglip-base-patch16-224` |
| |
| **Model Stats:** |
| - Model variant: base-224 (default) — base-256 / base-384 / large-256 / large-384 / so400m-224 / so400m-384 also supported |
| - Precision tested: FP32, FP16, BF16 (CPU, GPU PyTorch, GPU vLLM) |
| |
| --- |
| |
| ## Accuracy Pipeline |
| |
| Higher accuracy means a larger fraction of images are assigned the correct CIFAR-10 label — 100% is perfect, 10% is random chance for this 10-way task. Values above ~90% on CIFAR-10 with SigLIP-base are typical; larger variants (large, so400m) often reach ~96–97%. |
| |
| ### Metrics Explained |
| |
| | Metric | Description | |
| |--------|-------------| |
| | Accuracy (%) | Fraction of test images where the highest-scoring class prompt matches the ground-truth label. Primary quality metric; FP16/BF16 should track FP32 within a few tenths of a percent. | |
| | Image throughput (img/s) | Images encoded and scored per second during the timed loop (text embeddings computed once upfront and excluded from the timer). Higher is faster; PyTorch-direct GPU is typically faster than per-image vLLM HTTP calls. | |
| |
| ### Accuracy Results |
| |
| **Full Dataset Evaluation (CIFAR-10 test)** — siglip-base-patch16-224: |
| |
| <!-- accuracy-table-start --> |
| | Device | Backend | Precision | Accuracy (%) | Throughput (img/s) | |
| |--------|---------|-----------|--------------|--------------------| |
| | CPU | PyTorch | BF16 | 92.57 | 40.84 | |
| | CPU | PyTorch | FP16 | 92.55 | 5.19 | |
| | CPU | PyTorch | FP32 | 92.54 | 18.35 | |
| | GPU | PyTorch | BF16 | 92.59 | 100.16 | |
| | GPU | PyTorch | FP16 | 92.56 | 100.75 | |
| | GPU | PyTorch | FP32 | 92.54 | 35.04 | |
| | GPU | vLLM | BF16 | 92.54 | 77.91 | |
| | GPU | vLLM | FP16 | 92.55 | 75.78 | |
| | GPU | vLLM | FP32 | 92.52 | 28.12 | |
| <!-- accuracy-table-end --> |
| |
| --- |
| |
| ## Dig Deeper |
| |
| Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples? |
| |
| 📂 **[View the full project on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/siglip)** |
| |
| The GitHub repository includes: |
| - Setup and prerequisites for ROCm environments |
| - Scripts for both PyTorch and vLLM runners |
| - Additional model variants and datasets |
| - Benchmarking and reproduction instructions |
| |
