README.md

---
pipeline_tag: zero-shot-image-classification
license: mit
base_model: openai/clip-vit-base-patch16
library_name: kerasformers
tags:
- keras
- kerasformers
- clip
- zero-shot-image-classification
- vision
- arxiv:2103.00020
- pytorch
- jax
- tf
---

## ***See [our collection](https://huggingface.co/collections/kerasformers/clip-6a6a9c7bfdc6c38dcb984c24) for all versions of CLIP.***

# Run CLIP with Keras 3: JAX, PyTorch, or TensorFlow

[![GitHub](https://img.shields.io/badge/GitHub-KerasFormers-black?logo=github)](https://github.com/IMvision12/KerasFormers) [![Docs](https://img.shields.io/badge/Docs-CLIP-blue)](https://imvision12.github.io/KerasFormers/clip/) [![Collection](https://img.shields.io/badge/HF-CLIP%20collection-yellow)](https://huggingface.co/collections/kerasformers/clip-6a6a9c7bfdc6c38dcb984c24)

# kerasformers/clip_vit_base_16

Paper: [Learning Transferable Visual Models From Natural Language Supervision (arXiv:2103.00020)](https://arxiv.org/abs/2103.00020) · [HF Papers](https://huggingface.co/papers/2103.00020)

CLIP (Contrastive Language-Image Pre-training) is a vision + text dual-encoder trained on (image, caption) pairs with a contrastive loss. Both encoders project to a shared embedding space for zero-shot classification, retrieval, and embeddings.

For more details on the model, please go to the upstream [model card](https://huggingface.co/openai/clip-vit-base-patch16).

Pure-**Keras 3** conversion of [`openai/clip-vit-base-patch16`](https://huggingface.co/openai/clip-vit-base-patch16) for [kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on **TensorFlow / Torch / JAX**.

This is a **zero-shot image-text** checkpoint (`CLIPZeroShotClassify`): pass image(s) and text prompts at inference time.

## ✨ Quick start

```python
import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from kerasformers.models.clip import (
    CLIPProcessor,
    CLIPZeroShotClassify,
)

processor = CLIPProcessor.from_weights("kerasformers/clip_vit_base_16")
model = CLIPZeroShotClassify.from_weights("kerasformers/clip_vit_base_16")

labels = [
    "a photo of a cat",
    "a photo of a dog",
    "a photo of a car",
    "a photo of a living room",
]
inputs = processor(text=labels, image_paths="your_image.jpg")
output = model(
    {
        "images": inputs["images"],
        "token_ids": inputs["input_ids"],
        "padding_mask": inputs["attention_mask"],
    }
)
print(output["image_logits"].shape)
```

Load any CLIP variant the same way with `from_weights("kerasformers/<variant>")`:

| Variant | Hub | Notes |
|---|---|---|
| `clip_vit_base_16` | [`kerasformers/clip_vit_base_16`](https://huggingface.co/kerasformers/clip_vit_base_16) | OpenAI |
| `clip_vit_base_32` | [`kerasformers/clip_vit_base_32`](https://huggingface.co/kerasformers/clip_vit_base_32) | OpenAI |
| `clip_vit_large_14` | [`kerasformers/clip_vit_large_14`](https://huggingface.co/kerasformers/clip_vit_large_14) | OpenAI |
| `clip_vit_large_14_336` | [`kerasformers/clip_vit_large_14_336`](https://huggingface.co/kerasformers/clip_vit_large_14_336) | OpenAI |
| `clip_vit_g_14` | [`kerasformers/clip_vit_g_14`](https://huggingface.co/kerasformers/clip_vit_g_14) | LAION |
| `clip_vit_bigg_14` | [`kerasformers/clip_vit_bigg_14`](https://huggingface.co/kerasformers/clip_vit_bigg_14) | LAION |

## Tips

- Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
- Prefer `Processor.from_weights(...)` so image size and tokenizer match the variant.
- Map processor `input_ids` / `attention_mask` to model `token_ids` / `padding_mask`.
- OpenAI variants use `quick_gelu`; LAION g/G use `gelu`.
- See [CLIP docs](https://imvision12.github.io/KerasFormers/clip/) and [Loading Weights](https://imvision12.github.io/KerasFormers/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `CLIPZeroShotClassify.from_weights("hf:openai/clip-vit-base-patch16")`.

## Special Thanks

A huge thank you to the OpenAI CLIP and LAION authors for creating and releasing these models.

License: MIT.