Armaggheddon/tips-v2-b14-onnx

TIPSv2-B/14 — ONNX Export

Dual-encoder vision-language model (86M vision / 110M text). 448×448 images → 768-dim embeddings. Original: google-deepmind/tips.
Exported with tips-onnx — see the repo for custom exports (other precisions, fixed sizes) and TensorRT engine builds.

Available files

File	Precision	Size	Backend
vision_encoder_fp32.onnx	FP32	329 MB	CPU / CUDA / TRT
text_encoder_fp32.onnx	FP32	418 MB	CPU / CUDA / TRT
vision_encoder_fp16.onnx	FP16	165 MB	CPU / CUDA / TRT
text_encoder_fp16.onnx	FP16	209 MB	CPU / CUDA / TRT
vision_encoder_int8_dynamic.onnx	INT8 (CPU)	83 MB	CPU
text_encoder_int8_dynamic.onnx	INT8 (CPU)	105 MB	CPU

Fixed-size 448×448 exports for TensorRT: | vision_encoder_448_fp32.onnx | FP32 | 330 MB | TRT | | vision_encoder_448_fp16.onnx | FP16 | 165 MB | TRT |

The _448 exports skip the dynamic position-encoding interpolation path, producing a fixed-shape graph that trtexec can parse.

Calibration

INT8 Q/DQ quantization (tools/export.py --precision int8_qdq) needs calibration data in the vision/text form of the encoder inputs, from any source — at least 64 samples per encoder (the minimum suggested by NVIDIA ModelOpt; this project's calibration used 500). The development data was built from lmms-lab/COCO-Caption (500 images + captions).

Rebuild calibration data with tools/make_calibration.py (synthetic, structural testing only) or from your own dataset in the same format: checkpoints/calib_vision.npy ((N, 3, 448, 448) float32 images) and checkpoints/calib_text.npz (token_ids / padding_mask, (N, 64) int64).

Input specification

Vision encoder: image — (B, 3, H, W) float32, pixels in [0, 1].

H, W must be multiples of 14 (patch size): 224, 238, ..., 336, ..., 448, 518, ...

Text encoder: token_ids (B, 64) int64, padding_mask (B, 64) int64 (0=valid, 1=pad).

Usage

from huggingface_hub import hf_hub_download
import onnxruntime as ort, numpy as np
from PIL import Image

# Vision
path = hf_hub_download("Armaggheddon/tips-v2-b14-onnx", "vision_encoder_fp32.onnx")
sess = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
img = np.array(Image.open("photo.jpg").convert("RGB").resize((448,448)), dtype=np.float32) / 255.0
cls1, cls2, patches = sess.run(None, {"image": img.transpose(2,0,1)[None]})

# Text (needs tokenizer from original TIPS repo)
path = hf_hub_download("Armaggheddon/tips-v2-b14-onnx", "text_encoder_fp32.onnx")
sess = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
# token_ids, padding = tokenizer.tokenize(["a cat on a couch"])
emb = sess.run(None, {"token_ids": token_ids, "padding_mask": padding})[0]

FP32 models: the .onnx file references an external .onnx.data companion — download both files from the repo. See the repo's example_inference.py for a download helper that handles this automatically.

Evaluation

Numerical accuracy vs PyTorch FP32 baseline (ONNX Runtime CPU, batch=1):

Precision	Vision cosine	Text cosine	Cross-modal Δ
FP32	1.000000	1.000000	2.2×10⁻⁸
FP16	1.000000	1.000000	6.2×10⁻⁵
INT8 dyn	0.987054	0.998132	3.2×10⁻³

Performance

GPU latency at batch=1, 448×448 vision, RTX 3070 Ti (lower is better):

Encoder	Precision	PT CUDA	ORT CUDA	TRT
Vision	FP32	27.6 ms	23.8 ms	17.6 ms
Vision	FP16	27.6 ms	14.3 ms	4.6 ms
Text	FP32	14.4 ms	7.4 ms	—
Text	FP16	16.5 ms	5.7 ms	0.79 ms

TensorRT deployment

Fixed spatial dimensions required. Use the _448 exports for vision:

# Build vision engine (FP16)
trtexec --onnx=onnx/B/vision_encoder_448_fp16.onnx \
    --minShapes=image:1x3x448x448 --optShapes=image:1x3x448x448 --maxShapes=image:1x3x448x448 \
    --saveEngine=onnx/B/vision_encoder_fp16.engine

# Build text engine (FP16)
trtexec --onnx=onnx/B/text_encoder_fp16.onnx \
    --minShapes=token_ids:1x64,padding_mask:1x64 \
    --optShapes=token_ids:1x64,padding_mask:1x64 \
    --maxShapes=token_ids:1x64,padding_mask:1x64 \
    --saveEngine=onnx/B/text_encoder_fp16.engine

Citation

@InProceedings{tips_v2_paper,
    Title={{TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment}},
    Author={Cao, Bingyi and Chen, Koert and Maninis, Kevis-Kokitsi and Chen, Kaifeng and Karpur, Arjun and Xia, Ye and Dua, Sahil and Dabral, Tanmaya and Han, Guangxing and Han, Bohyung and Ainslie, Joshua and Bewley, Alex and Jacob, Mithun and Wagner, Ren\'e and Ramos, Washington and Choromanski, Krzysztof and Seyedhosseini, Mojtaba and Zhou, Howard and Araujo, Andr\'e},
    Booktitle={CVPR},
    year={2026},
}