README.md

---
pipeline_tag: zero-shot-image-classification
license: apache-2.0
base_model: google/tipsv2-so400m14
library_name: kerasformers
tags:
- keras
- kerasformers
- tipsv2
- zero-shot-image-classification
- vision
- arxiv:2604.12012
- pytorch
- jax
- tf
---

## ***See [our collection](https://huggingface.co/collections/kerasformers/tipsv2-6a8a3f36af77204954a49fb4) for all versions of TIPSv2.***

# Run TIPSv2 with Keras 3: JAX, PyTorch, or TensorFlow

[![GitHub](https://img.shields.io/badge/GitHub-KerasFormers-black?logo=github)](https://github.com/IMvision12/KerasFormers) [![Collection](https://img.shields.io/badge/HF-TIPSv2%20collection-yellow)](https://huggingface.co/collections/kerasformers/tipsv2-6a8a3f36af77204954a49fb4)

# kerasformers/tipsv2-so400m14

Paper: [TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment (arXiv:2604.12012)](https://huggingface.co/papers/2604.12012)

TIPSv2 (Google DeepMind) is a CLIP/SigLIP-style dual encoder: a DINOv2-style ViT vision tower with register tokens plus a bidirectional text tower, aligned with a temperature-scaled contrastive objective.

For more details on the model, please go to the upstream [model card](https://huggingface.co/google/tipsv2-so400m14).

Pure-**Keras 3** conversion of [`google/tipsv2-so400m14`](https://huggingface.co/google/tipsv2-so400m14) for [kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on **TensorFlow / Torch / JAX**. The full model and both towers load from this single repo.

## ✨ Quick start (zero-shot)

```python
import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from PIL import Image
import numpy as np
import keras
from kerasformers.models.tipsv2 import Tipsv2Model, Tipsv2Processor

model = Tipsv2Model.from_weights("kerasformers/tipsv2-so400m14")
processor = Tipsv2Processor.from_weights("kerasformers/tipsv2-so400m14")

image = Image.open("your_image.jpg").convert("RGB")
texts = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
inputs = processor(text=texts, images=np.array(image))
out = model(inputs)
probs = keras.ops.softmax(out["logits_per_image"], axis=-1)
print(keras.ops.convert_to_numpy(probs)[0])
```

Towers only:

```python
from kerasformers.models.tipsv2 import Tipsv2VisionModel, Tipsv2TextModel
vision = Tipsv2VisionModel.from_weights("kerasformers/tipsv2-so400m14")
text = Tipsv2TextModel.from_weights("kerasformers/tipsv2-so400m14")
```

All TIPSv2 variants load the same way with `from_weights("kerasformers/<variant>")`:

| Variant | Hub |
|---|---|
| `tipsv2-b14` | [`kerasformers/tipsv2-b14`](https://huggingface.co/kerasformers/tipsv2-b14) |
| `tipsv2-l14` | [`kerasformers/tipsv2-l14`](https://huggingface.co/kerasformers/tipsv2-l14) |
| `tipsv2-so400m14` | [`kerasformers/tipsv2-so400m14`](https://huggingface.co/kerasformers/tipsv2-so400m14) |
| `tipsv2-g14` | [`kerasformers/tipsv2-g14`](https://huggingface.co/kerasformers/tipsv2-g14) |

## Tips

- Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
- The image processor rescales to `[0, 1]` (no mean/std normalization); input resolution is 448.
- Upstream checkpoints: `Tipsv2Model.from_weights("hf:google/tipsv2-so400m14")`.

## Special Thanks

A huge thank you to the TIPSv2 authors (Google DeepMind) and the HF community.

License: Apache-2.0 (matches the upstream `google/tipsv2-so400m14` checkpoint).