README.md

---
license: apache-2.0
tags:
- vision
- image-text
- contrastive-learning
- zero-shot
- feature-extraction
- arxiv:2410.16512
library_name: transformers
pipeline_tag: zero-shot-image-classification
---

# TIPS — g/14 low-res (v1)

TIPS (Text-Image Pre-training with Spatial awareness, ICLR 2025) is a family of contrastive vision-language models that produce spatially rich image features aligned with text embeddings. This is the original (v1) g/14 low-res release with 1.1B vision params and 389M text params, converted from the [official checkpoints](https://github.com/google-deepmind/tips).

| Variant | Vision params | Text params | Embed dim | Resolution |
|---------|---------------|-------------|-----------|------------|
| [S/14](https://huggingface.co/google/tipsv1-s14) | 22M | 34M | 384 | 448 |
| [B/14](https://huggingface.co/google/tipsv1-b14) | 86M | 110M | 768 | 448 |
| [L/14](https://huggingface.co/google/tipsv1-l14) | 304M | 184M | 1024 | 448 |
| [So400m/14](https://huggingface.co/google/tipsv1-so400m14) | 413M | 448M | 1152 | 448 |
| [g/14](https://huggingface.co/google/tipsv1-g14) | 1.1B | 389M | 1536 | 448 |
| [g/14 low-res](https://huggingface.co/google/tipsv1-g14-lowres) | 1.1B | 389M | 1536 | 224 |

## Usage

```bash
pip install transformers torch torchvision sentencepiece scikit-learn requests
```

### Load the model

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("google/tipsv1-g14-lowres", trust_remote_code=True)
model.eval()
```

### Encode images

Images should be tensors in `[0, 1]` range (just `ToTensor()`, no ImageNet normalization).

```python
import requests
from PIL import Image
from torchvision import transforms

url = "https://huggingface.co/spaces/google/TIPSv2/resolve/main/examples/zeroseg/pascal_context_00049_image.png"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
transform = transforms.Compose([transforms.Resize((224, 224)), transforms.ToTensor()])
pixel_values = transform(image).unsqueeze(0)

out = model.encode_image(pixel_values)
print(out.cls_token.shape)     # (1, 1, 1536) — global image embedding
print(out.patch_tokens.shape)  # (1, 256, 1536) — per-patch spatial features
```

The second CLS token (`out.register_tokens`) was trained on synthetic captions; the first (`out.cls_token`) on web alt-text, and is the one aligned with the text tower.

### Encode text

```python
text_emb = model.encode_text(["a photo of a bus", "a photo of a dog"])
print(text_emb.shape)  # (2, 1536) — one embedding per query
```

### Zero-shot classification

```python
import torch.nn.functional as F

classes = ["bus", "car", "dog", "cat"]
cls = F.normalize(out.cls_token[:, 0, :], dim=-1)
text_emb = F.normalize(model.encode_text(classes), dim=-1)
similarity = cls @ text_emb.T
print(classes[similarity.argmax()])  # predicted class
```

### Visualize spatial features

```python
import numpy as np
from sklearn.decomposition import PCA

feat = out.patch_tokens[0].detach().cpu().numpy()
rgb = PCA(n_components=3, whiten=True).fit_transform(feat).reshape(16, 16, 3)
rgb = 1 / (1 + np.exp(-2.0 * rgb))  # sigmoid for [0, 1] range with good contrast
```

## Model details

- ViT-g/14 low-res vision encoder (40 layers, patch size 14, two CLS tokens) + 12-layer transformer text encoder
- Native resolution 224; other patch-multiple resolutions work via positional-embedding interpolation
- Preprocessing: images to `[0, 1]`, no normalization; SentencePiece tokenizer, lowercased, max 64 tokens

## License

Apache 2.0

## Citation

```bibtex
@inproceedings{maninis2025tips,
  title     = {{TIPS: Text-Image Pretraining with Spatial Awareness}},
  author    = {Maninis, Kevis-Kokitsi and Chen, Kaifeng and Ghosh, Soham and Karpur, Arjun and Chen, Koert and Xia, Ye and Cao, Bingyi and Salz, Daniel and Han, Guangxing and Dlabal, Jan and Gnanapragasam, Dan and Seyedhosseini, Mojtaba and Zhou, Howard and Araujo, Andre},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2025},
  url       = {https://arxiv.org/abs/2410.16512}
}
```