File size: 4,103 Bytes
d9063ba f7e7769 d9063ba f7e7769 d9063ba f7e7769 d9063ba f7e7769 d9063ba f7e7769 d9063ba f7e7769 d9063ba f7e7769 d9063ba f7e7769 d9063ba f7e7769 d9063ba f7e7769 d9063ba f7e7769 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
pipeline_tag: zero-shot-image-classification
license: mit
base_model: openai/clip-vit-base-patch32
library_name: kerasformers
tags:
- keras
- kerasformers
- clip
- zero-shot-image-classification
- vision
- arxiv:2103.00020
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/kerasformers/clip-6a6a9c7bfdc6c38dcb984c24) for all versions of CLIP.***
# Run CLIP with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/KerasFormers) [](https://imvision12.github.io/KerasFormers/clip/) [](https://huggingface.co/collections/kerasformers/clip-6a6a9c7bfdc6c38dcb984c24)
# kerasformers/clip_vit_base_32
Paper: [Learning Transferable Visual Models From Natural Language Supervision (arXiv:2103.00020)](https://arxiv.org/abs/2103.00020) · [HF Papers](https://huggingface.co/papers/2103.00020)
CLIP (Contrastive Language-Image Pre-training) is a vision + text dual-encoder trained on (image, caption) pairs with a contrastive loss. Both encoders project to a shared embedding space for zero-shot classification, retrieval, and embeddings.
For more details on the model, please go to the upstream [model card](https://huggingface.co/openai/clip-vit-base-patch32).
Pure-**Keras 3** conversion of [`openai/clip-vit-base-patch32`](https://huggingface.co/openai/clip-vit-base-patch32) for [kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is a **zero-shot image-text** checkpoint (`CLIPZeroShotClassify`): pass image(s) and text prompts at inference time.
## ✨ Quick start
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from kerasformers.models.clip import (
CLIPProcessor,
CLIPZeroShotClassify,
)
processor = CLIPProcessor.from_weights("kerasformers/clip_vit_base_32")
model = CLIPZeroShotClassify.from_weights("kerasformers/clip_vit_base_32")
labels = [
"a photo of a cat",
"a photo of a dog",
"a photo of a car",
"a photo of a living room",
]
inputs = processor(text=labels, image_paths="your_image.jpg")
output = model(
{
"images": inputs["images"],
"token_ids": inputs["input_ids"],
"padding_mask": inputs["attention_mask"],
}
)
print(output["image_logits"].shape)
```
Load any CLIP variant the same way with `from_weights("kerasformers/<variant>")`:
| Variant | Hub | Notes |
|---|---|---|
| `clip_vit_base_16` | [`kerasformers/clip_vit_base_16`](https://huggingface.co/kerasformers/clip_vit_base_16) | OpenAI |
| `clip_vit_base_32` | [`kerasformers/clip_vit_base_32`](https://huggingface.co/kerasformers/clip_vit_base_32) | OpenAI |
| `clip_vit_large_14` | [`kerasformers/clip_vit_large_14`](https://huggingface.co/kerasformers/clip_vit_large_14) | OpenAI |
| `clip_vit_large_14_336` | [`kerasformers/clip_vit_large_14_336`](https://huggingface.co/kerasformers/clip_vit_large_14_336) | OpenAI |
| `clip_vit_g_14` | [`kerasformers/clip_vit_g_14`](https://huggingface.co/kerasformers/clip_vit_g_14) | LAION |
| `clip_vit_bigg_14` | [`kerasformers/clip_vit_bigg_14`](https://huggingface.co/kerasformers/clip_vit_bigg_14) | LAION |
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
- Prefer `Processor.from_weights(...)` so image size and tokenizer match the variant.
- Map processor `input_ids` / `attention_mask` to model `token_ids` / `padding_mask`.
- OpenAI variants use `quick_gelu`; LAION g/G use `gelu`.
- See [CLIP docs](https://imvision12.github.io/KerasFormers/clip/) and [Loading Weights](https://imvision12.github.io/KerasFormers/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `CLIPZeroShotClassify.from_weights("hf:openai/clip-vit-base-patch32")`.
## Special Thanks
A huge thank you to the OpenAI CLIP and LAION authors for creating and releasing these models.
License: MIT.
|