antonlnz/siglip2-so400m-image-coreml

SigLIP2-so400m image encoder, Core ML

The image tower of SigLIP2-so400m converted to Core ML (fp16), so it can run with hardware acceleration on Macs. The weights are Google's, untouched — only the format changed.

Converted from open_clip, model ViT-SO400M-14-SigLIP2-378, pretrained tag webli.

This is the image tower only. For image↔text search you also need the matching text tower, which is not in this repository.

Interface

Input	image, an image of 378 × 378
Output	embedding, 1152 dimensions, float16, already L2-normalized
Size	815 MB
Minimum target	macOS 14

Two things are easy to get wrong, and neither one fails loudly — you just get worse search results:

The input is 378, not 384. The original model is named patch14-384, but its patch embedding is a 14×14 convolution with stride 14: 27 patches fit, and the last 6 pixels of each dimension are dropped. So the model effectively looks at the top-left 378×378 region of a 384-resized image, which is also what it saw during training. If you resize your image to 378 directly instead of resizing to 384 and cropping to 378, cosine similarity against the original model drops from 0.998 to 0.970.

The output is float16. Reading the MLMultiArray buffer as float32 gives you a plausible-looking vector that has nothing to do with the real embedding — we measured 0.02 cosine similarity against the reference. Check dataType rather than assuming.

Normalization is baked into the model (scale 2/255, bias −1, i.e. SigLIP2's mapping to [−1, 1]), so pass raw pixels; do not normalize them yourself.

Fidelity, and which compute unit to use

Measured on an M2 Pro against the fp16 ONNX export of the same model (onnx-community/siglip2-so400m-patch14-384-ONNX), over 134 frames drawn from 30 different videos, with identical preprocessing on both sides. "Top-10 overlap" is how much the ranking of those 134 frames agrees with the ONNX ranking across 30 real search queries — which is what a user actually notices.

Compute units	ms/image	Mean cosine	Min	Below 0.99	Top-10 overlap
.cpuAndGPU	224	0.9999	0.9991	0/134	98 %
.cpuOnly	353	0.9990	0.9817	2/134	96 %
.cpuAndNeuralEngine	184	0.9946	0.9442	14/134	82 %

Use .cpuAndGPU. The Neural Engine is the fastest of the three, but at 400M parameters its fp16 arithmetic drifts far enough to reorder search results, and .all will pick it. The GPU is 20 % slower and reproduces the original almost exactly.

For reference, the same ONNX model on CPU runs at 1395 ms/image, so the GPU path is about 6× faster.

If you are converting a smaller SigLIP2 (the 86M-parameter base model, for instance), this does not apply: there the Neural Engine is both the fastest and lossless, and .all is the right choice. The lesson is to measure the distribution and the ranking, not the mean cosine — the mean hid this.

How it was converted

uv run convert_to_coreml.py ViT-SO400M-14-SigLIP2-378

Using the script published in batmac/ViT-B-16-SigLIP2-Image-CoreML: it loads the model through open_clip, wraps it so the output comes out L2-normalized, traces it with torch.jit.trace, and converts with coremltools (minimum_deployment_target=macOS14).

Note that the script's own --verify step fails on this model: it builds a fixed 224×224 test image, and Core ML rejects any size other than 378. Verify against a reference implementation instead.

License

Apache 2.0, inherited from the original model. This is a derivative work; the only modifications are the format (PyTorch → Core ML) and the precision (fp16).