README.md

metadata

license: apache-2.0
tags:
  - vision
  - image-text
  - contrastive-learning
  - zero-shot
  - feature-extraction
  - arxiv:2410.16512
library_name: transformers
pipeline_tag: zero-shot-image-classification

TIPS — g/14 low-res (v1)

TIPS (Text-Image Pre-training with Spatial awareness, ICLR 2025) is a family of contrastive vision-language models that produce spatially rich image features aligned with text embeddings. This is the original (v1) g/14 low-res release with 1.1B vision params and 389M text params, converted from the official checkpoints.

Variant	Vision params	Text params	Embed dim	Resolution
S/14	22M	34M	384	448
B/14	86M	110M	768	448
L/14	304M	184M	1024	448
So400m/14	413M	448M	1152	448
g/14	1.1B	389M	1536	448
g/14 low-res	1.1B	389M	1536	224

Usage

pip install transformers torch torchvision sentencepiece scikit-learn requests

Load the model

from transformers import AutoModel

model = AutoModel.from_pretrained("google/tipsv1-g14-lowres", trust_remote_code=True)
model.eval()

Encode images

Images should be tensors in [0, 1] range (just ToTensor(), no ImageNet normalization).

import requests
from PIL import Image
from torchvision import transforms

url = "https://huggingface.co/spaces/google/TIPSv2/resolve/main/examples/zeroseg/pascal_context_00049_image.png"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
transform = transforms.Compose([transforms.Resize((224, 224)), transforms.ToTensor()])
pixel_values = transform(image).unsqueeze(0)

out = model.encode_image(pixel_values)
print(out.cls_token.shape)     # (1, 1, 1536) — global image embedding
print(out.patch_tokens.shape)  # (1, 256, 1536) — per-patch spatial features

The second CLS token (out.register_tokens) was trained on synthetic captions; the first (out.cls_token) on web alt-text, and is the one aligned with the text tower.

Encode text

text_emb = model.encode_text(["a photo of a bus", "a photo of a dog"])
print(text_emb.shape)  # (2, 1536) — one embedding per query

Zero-shot classification

import torch.nn.functional as F

classes = ["bus", "car", "dog", "cat"]
cls = F.normalize(out.cls_token[:, 0, :], dim=-1)
text_emb = F.normalize(model.encode_text(classes), dim=-1)
similarity = cls @ text_emb.T
print(classes[similarity.argmax()])  # predicted class

Visualize spatial features

import numpy as np
from sklearn.decomposition import PCA

feat = out.patch_tokens[0].detach().cpu().numpy()
rgb = PCA(n_components=3, whiten=True).fit_transform(feat).reshape(16, 16, 3)
rgb = 1 / (1 + np.exp(-2.0 * rgb))  # sigmoid for [0, 1] range with good contrast

Model details

• ViT-g/14 low-res vision encoder (40 layers, patch size 14, two CLS tokens) + 12-layer transformer text encoder
• Native resolution 224; other patch-multiple resolutions work via positional-embedding interpolation
• Preprocessing: images to [0, 1], no normalization; SentencePiece tokenizer, lowercased, max 64 tokens

License

Apache 2.0

Citation

@inproceedings{maninis2025tips,
  title     = {{TIPS: Text-Image Pretraining with Spatial Awareness}},
  author    = {Maninis, Kevis-Kokitsi and Chen, Kaifeng and Ghosh, Soham and Karpur, Arjun and Chen, Koert and Xia, Ye and Cao, Bingyi and Salz, Daniel and Han, Guangxing and Dlabal, Jan and Gnanapragasam, Dan and Seyedhosseini, Mojtaba and Zhou, Howard and Araujo, Andre},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2025},
  url       = {https://arxiv.org/abs/2410.16512}
}