File size: 5,614 Bytes
da65901 9f1f4b1 9d15774 9f1f4b1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 | ---
license: mit
library_name: pytorch
tags:
- foundation
- amd
- rocm
- clip
- zero-shot-image-classification
pipeline_tag: zero-shot-image-classification
---
# CLIP: Optimized for AMD ROCm
CLIP (Contrastive Language-Image Pre-training) performs zero-shot image classification by comparing image embeddings against text prompt embeddings. This repository packages CLIP evaluation on CIFAR-10 using **PyTorch** (CPU or GPU via OpenAI CLIP) and **vLLM** (GPU server with pooling runner, CPU HTTP client), exported and validated for **AMD ROCm** so it runs efficiently on AMD GPUs and CPUs.
This is based on the implementation of CLIP found [here](https://github.com/openai/CLIP).
This repository contains configurations and scripts optimized for **AMD® ROCm™** platforms. You can use the [<repo-name> AMD scripts](<github-link>) to reproduce results or export with custom configurations. More details on model performance can be found [here](#performance-summary).
---
## Task Overview
**Task:** Zero-shot image classification
**Dataset:** CIFAR-10 test split (10,000 images, 10 classes)
**Output metrics:** Zero-shot accuracy (%)
> **Quick Start:** Before running this example, complete the main repository setup — see **[Prerequisites](<github-link>#prerequisites)**, **[Configure System Paths](<github-link>#2-configure-system-paths)**, and **[Virtual Environments](<github-link>#3-virtual-environments)** in the [main README](<github-link>).
> **Model variants:** Default is **base32** (`openai/clip-vit-base-patch32` / `ViT-B/32`). Override with `MODEL_VARIANT=base16|large14|large14-336`.
> **vLLM note:** vLLM's CLIP backend embeds **one modality per request** — text prompts and images are sent in separate API calls, then cosine similarity is computed client-side (same approach as the original evaluation scripts).
---
## AMD ROCm Optimization
This model export has been adapted and validated for **AMD Instinct™ / Radeon™ GPUs** running **ROCm**, as well as AMD CPUs. Key points:
- Exported/tested with ROCm `<rocm-version>` and PyTorch ROCm build `<torch-rocm-version>`.
- Validated backends: **PyTorch** (native ROCm HIP kernels) and **vLLM** (ROCm-enabled server build).
- No code changes required versus the upstream OpenAI CLIP implementation — only environment/runtime configuration differs.
- CPU fallback path supported for environments without a ROCm-capable GPU.
| Runtime | Precision | Backend | Hardware | Notes |
|---|---|---|---|---|
| PyTorch | fp32/fp16 | HIP (ROCm) | AMD Instinct / Radeon GPU | Native OpenAI CLIP inference |
| PyTorch | fp32 | CPU | AMD CPU (EPYC/Ryzen) | CPU-only fallback |
| vLLM | fp16 | ROCm server | AMD Instinct GPU | Pooling runner, image-only requests |
| vLLM (client) | — | HTTP | AMD CPU | Text-prompt requests, CPU client |
---
## Getting Started
### Option 1: Use Provided Scripts
Pre-configured evaluation scripts are available for direct use on ROCm hardware. See [Quick Start](#task-overview) above for setup steps.
### Option 2: Run with Custom Configuration
Use the scripts in [<repo-name> on GitHub](<github-link>) to run with your own:
- Custom model variant (`base16`, `large14`, `large14-336`, etc.)
- Custom dataset (beyond CIFAR-10)
- Target AMD GPU/CPU and runtime (PyTorch vs vLLM)
This option is ideal if you need to customize the evaluation beyond the default configuration provided here.
---
## Model Details
**Model Type:** Zero-shot image classification (contrastive image-text embedding)
**Base Model:** `openai/clip-vit-base-patch32` (ViT-B/32)
**Model Stats:**
- Model variant: base32 (default) — base16 / large14 / large14-336 also supported
- Image encoder: ViT-B/32
- Text encoder: Transformer (CLIP text tower)
- Input resolution: 224x224 (base variants), 336x336 (large14-336)
- Number of parameters: `<fill-in>`
- Precision tested: fp32, fp16
---
## Performance Summary
Higher zero-shot accuracy means more test images are assigned the correct CIFAR-10 class via CLIP's image–text similarity — 100% is perfect, 10% is chance level for 10 classes. Values above ~85% on CIFAR-10 with ViT-B/32 are typical for this benchmark.
### Metrics Explained
| Metric | Description |
|--------|-------------|
| Zero-shot accuracy (%) | Fraction of CIFAR-10 test images whose highest-scoring text prompt matches the ground-truth label after softmax over 10 class prompts. Primary accuracy metric; sensitive to both image and text embedding quality. |
### Accuracy Results
**Full Dataset Evaluation (CIFAR-10 test)** — filled from `evaluation_results/`; run `make metrics` to refresh:
<!-- accuracy-table-start -->
| Device | Backend | Precision | Variant | Accuracy (%) |
|--------|---------|-----------|---------|--------------|
| CPU | PyTorch | FP16 | clip-vit-base-patch32 | 88.79 |
| CPU | PyTorch | FP32 | clip-vit-base-patch32 | 88.80 |
| GPU | PyTorch | FP16 | clip-vit-base-patch32 | 88.75 |
| GPU | PyTorch | FP32 | clip-vit-base-patch32 | 88.80 |
| GPU | vLLM | FP16 | clip-vit-base-patch32 | 88.78 |
| GPU | vLLM | FP32 | clip-vit-base-patch32 | 88.80 |
<!-- accuracy-table-end -->
---
## Dig Deeper
Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?
📂 **[View the full project on GitHub](<github-link>)**
The GitHub repository includes:
- Setup and prerequisites for ROCm environments
- Scripts for both PyTorch and vLLM runners
- Additional model variants and datasets
- Benchmarking and reproduction instructions
---
|