File size: 3,355 Bytes
30d034f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | ---
pipeline_tag: zero-shot-image-classification
license: apache-2.0
base_model: google/tipsv2-b14
library_name: kerasformers
tags:
- keras
- kerasformers
- tipsv2
- zero-shot-image-classification
- vision
- arxiv:2604.12012
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/kerasformers/tipsv2-6a8a3f36af77204954a49fb4) for all versions of TIPSv2.***
# Run TIPSv2 with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/KerasFormers) [](https://huggingface.co/collections/kerasformers/tipsv2-6a8a3f36af77204954a49fb4)
# kerasformers/tipsv2-b14
Paper: [TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment (arXiv:2604.12012)](https://huggingface.co/papers/2604.12012)
TIPSv2 (Google DeepMind) is a CLIP/SigLIP-style dual encoder: a DINOv2-style ViT vision tower with register tokens plus a bidirectional text tower, aligned with a temperature-scaled contrastive objective.
For more details on the model, please go to the upstream [model card](https://huggingface.co/google/tipsv2-b14).
Pure-**Keras 3** conversion of [`google/tipsv2-b14`](https://huggingface.co/google/tipsv2-b14) for [kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on **TensorFlow / Torch / JAX**. The full model and both towers load from this single repo.
## ✨ Quick start (zero-shot)
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from PIL import Image
import numpy as np
import keras
from kerasformers.models.tipsv2 import Tipsv2Model, Tipsv2Processor
model = Tipsv2Model.from_weights("kerasformers/tipsv2-b14")
processor = Tipsv2Processor.from_weights("kerasformers/tipsv2-b14")
image = Image.open("your_image.jpg").convert("RGB")
texts = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
inputs = processor(text=texts, images=np.array(image))
out = model(inputs)
probs = keras.ops.softmax(out["logits_per_image"], axis=-1)
print(keras.ops.convert_to_numpy(probs)[0])
```
Towers only:
```python
from kerasformers.models.tipsv2 import Tipsv2VisionModel, Tipsv2TextModel
vision = Tipsv2VisionModel.from_weights("kerasformers/tipsv2-b14")
text = Tipsv2TextModel.from_weights("kerasformers/tipsv2-b14")
```
All TIPSv2 variants load the same way with `from_weights("kerasformers/<variant>")`:
| Variant | Hub |
|---|---|
| `tipsv2-b14` | [`kerasformers/tipsv2-b14`](https://huggingface.co/kerasformers/tipsv2-b14) |
| `tipsv2-l14` | [`kerasformers/tipsv2-l14`](https://huggingface.co/kerasformers/tipsv2-l14) |
| `tipsv2-so400m14` | [`kerasformers/tipsv2-so400m14`](https://huggingface.co/kerasformers/tipsv2-so400m14) |
| `tipsv2-g14` | [`kerasformers/tipsv2-g14`](https://huggingface.co/kerasformers/tipsv2-g14) |
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
- The image processor rescales to `[0, 1]` (no mean/std normalization); input resolution is 448.
- Upstream checkpoints: `Tipsv2Model.from_weights("hf:google/tipsv2-b14")`.
## Special Thanks
A huge thank you to the TIPSv2 authors (Google DeepMind) and the HF community.
License: Apache-2.0 (matches the upstream `google/tipsv2-b14` checkpoint).
|