|
|
--- |
|
|
library_name: onnx |
|
|
tags: |
|
|
- clip |
|
|
- multimodal |
|
|
- visual-search |
|
|
- zero-shot-classification |
|
|
- onnx |
|
|
- inference4j |
|
|
license: mit |
|
|
datasets: |
|
|
- openai/clip-training-data |
|
|
--- |
|
|
|
|
|
# CLIP ViT-B/32 — ONNX (Vision + Text Encoders) |
|
|
|
|
|
ONNX export of [openai/clip-vit-base-patch32](https://huggingface.co/openai/clip-vit-base-patch32) |
|
|
split into separate vision and text encoder models for independent use. |
|
|
|
|
|
Converted for use with [inference4j](https://github.com/inference4j/inference4j), |
|
|
an inference-only AI library for Java. |
|
|
|
|
|
## Usage with inference4j |
|
|
|
|
|
### Visual search (image-text similarity) |
|
|
|
|
|
```java |
|
|
try (ClipImageEncoder imageEncoder = ClipImageEncoder.builder().build(); |
|
|
ClipTextEncoder textEncoder = ClipTextEncoder.builder().build()) { |
|
|
|
|
|
float[] imageEmb = imageEncoder.encode(ImageIO.read(Path.of("photo.jpg").toFile())); |
|
|
float[] textEmb = textEncoder.encode("a photo of a cat"); |
|
|
|
|
|
float similarity = dot(imageEmb, textEmb); |
|
|
} |
|
|
``` |
|
|
|
|
|
### Zero-shot classification |
|
|
|
|
|
```java |
|
|
float[] imageEmb = imageEncoder.encode(photo); |
|
|
String[] labels = {"cat", "dog", "bird", "car"}; |
|
|
|
|
|
float bestScore = Float.NEGATIVE_INFINITY; |
|
|
String bestLabel = null; |
|
|
for (String label : labels) { |
|
|
float score = dot(imageEmb, textEncoder.encode("a photo of a " + label)); |
|
|
if (score > bestScore) { |
|
|
bestScore = score; |
|
|
bestLabel = label; |
|
|
} |
|
|
} |
|
|
``` |
|
|
|
|
|
## Files |
|
|
|
|
|
| File | Description | Size | |
|
|
|------|-------------|------| |
|
|
| `vision_model.onnx` | Vision encoder (ViT-B/32) | ~340 MB | |
|
|
| `text_model.onnx` | Text encoder (Transformer) | ~255 MB | |
|
|
| `vocab.json` | BPE vocabulary (49408 tokens) | ~1.6 MB | |
|
|
| `merges.txt` | BPE merge rules (48894 merges) | ~1.7 MB | |
|
|
|
|
|
## Model Details |
|
|
|
|
|
| Property | Value | |
|
|
|----------|-------| |
|
|
| Architecture | ViT-B/32 (vision) + Transformer (text) | |
|
|
| Embedding dim | 512 | |
|
|
| Max text length | 77 tokens | |
|
|
| Image input | `[batch, 3, 224, 224]` — CLIP-normalized | |
|
|
| Text input | `input_ids` + `attention_mask` `[batch, 77]` | |
|
|
| ONNX opset | 17 | |
|
|
|
|
|
## Preprocessing |
|
|
|
|
|
### Vision |
|
|
1. Resize to 224×224 (bicubic) |
|
|
2. CLIP normalization: mean=`[0.48145466, 0.4578275, 0.40821073]`, |
|
|
std=`[0.26862954, 0.26130258, 0.27577711]` |
|
|
3. NCHW layout: `[1, 3, 224, 224]` |
|
|
|
|
|
### Text |
|
|
1. Byte-level BPE tokenization using `vocab.json` + `merges.txt` |
|
|
2. Add `<|startoftext|>` (49406) and `<|endoftext|>` (49407) |
|
|
3. Pad/truncate to 77 tokens |
|
|
|
|
|
## Original Paper |
|
|
|
|
|
> Radford, A., Kim, J. W., Hallacy, C., et al. (2021). |
|
|
> Learning Transferable Visual Models From Natural Language Supervision. |
|
|
> ICML 2021. [arXiv:2103.00020](https://arxiv.org/abs/2103.00020) |
|
|
|
|
|
## License |
|
|
|
|
|
The original CLIP model is released under the MIT License by OpenAI. |
|
|
|
