README.md

metadata

license: apache-2.0
base_model: google/siglip2-base-patch16-256
pipeline_tag: feature-extraction
tags:
  - coreml
  - siglip2
  - siglip
  - clip
  - image-text
  - embeddings
  - vision
  - apple-silicon

SigLIP2 base-patch16-256 — Core ML

Core ML (.mlpackage) conversion of google/siglip2-base-patch16-256 for on-device text and image embedding on Apple platforms. The image and text towers ship as separate models; both produce 768-dimensional, L2-normalized embeddings in a shared space, so image↔text similarity is a single dot product.

Converted and maintained by nodevorg for Alarian's semantic frame search. Not affiliated with Google.

Files

file	size	sha256
ImageEncoder.mlpackage.zip	163 MiB	406938456c8f8e91f01632cb74131fdfb993064d8d0bc8f6fbbd142709f1ba66
TextEncoder.int8emb.mlpackage.zip — recommended	323 MiB	11471872101a6ca88dd51ed9de668b728ae9d09c0f853aeac8a6cc96f1c109d6
TextEncoder.mlpackage.zip	506 MiB	4f31f38a1729a0044f0ffc95a940dc1a6a2fdcd5fcf78a03cbb662bada6cbafb
tokenizer.json	33 MiB	cb9140fae3ac5122c972d37adf83e1248471a38147ad76f8215c8872c6fd8322
tokenizer_config.json	—	14afe629fe4959b9e0d51e1852b8d9f7ad074f90a1a7125a4fcdd17f06e78fc8
special_tokens_map.json	—	baec30ea10906f16adb8c18af7a34023002c1746542612b8b41c9f09e1351351

TextEncoder.int8emb is a drop-in replacement for the fp16 TextEncoder — identical input/output names, shapes, and dtypes — with the token-embedding table quantized to int8 (36% smaller download, worst-case parity 0.9996; details below). Use the full-fp16 text encoder only if you need maximum precision.

The tokenizer files are copied verbatim from the source checkpoint (Gemma BPE, 256k vocabulary).

Model interface

ImageEncoder

• Input — pixel_values: Core ML ImageType, RGB, 256×256. Preprocessing is baked into the model (scale=1/127.5, bias=[-1,-1,-1], matching the checkpoint's SiglipImageProcessor configuration): resize to 256×256 and pass raw [0, 255] RGB — no caller-side normalization. Resize with an aspect-ratio squash rather than a center crop; that matches training.
• Output — image_embedding: 768-dim float16, L2-normalized.

TextEncoder (both variants)

• Input — input_ids: 1×64 int32 token ids. Tokenize with the included tokenizer using padding="max_length", max_length=64, lowercased input (do_lower_case=true). There is deliberately no attention_mask input: SigLIP's text tower trains with unmasked bidirectional attention over the fixed 64-token sequence and pools the last sequence position, so a mask would deviate from training.
• Output — text_embedding: 768-dim float16, L2-normalized.

Because both outputs are normalized in-graph, cosine similarity is dot(image_embedding, text_embedding).

Usage (Swift)

import CoreML

let image = try MLModel(contentsOf: compiledImageEncoderURL)
let text  = try MLModel(contentsOf: compiledTextEncoderURL)

// Image: 256×256 RGB CVPixelBuffer, raw [0, 255] values
let imageOut = try image.prediction(
    from: MLDictionaryFeatureProvider(dictionary: ["pixel_values": pixelBuffer]))
let imageEmbedding = imageOut.featureValue(for: "image_embedding")!.multiArrayValue!

// Text: 64 int32 token ids (e.g. via swift-transformers' Tokenizers module,
// loading the tokenizer.json from this repo), lowercased, padded to length 64
let ids = try MLMultiArray(shape: [1, 64], dataType: .int32) // fill with token ids
let textOut = try text.prediction(
    from: MLDictionaryFeatureProvider(dictionary: ["input_ids": ids]))
let textEmbedding = textOut.featureValue(for: "text_embedding")!.multiArrayValue!

// Similarity = dot product (embeddings are already L2-normalized)

Architecture note

The source checkpoint's config.json declares model_type: siglip, not siglip2. At this fixed-resolution, non-NaFlex tier, the SigLIP2-trained weights are architecturally identical to SigLIP v1 — transformers.AutoModel resolves the checkpoint to SiglipModel, and that is what was converted. Relative to SigLIP v1, only the weights and the tokenizer (Gemma BPE 256k, vs. 32k SentencePiece) differ.

Most of the text encoder's size is the 256k-row token-embedding table (256000 × 768 × 2 bytes ≈ 375 MiB at fp16), which is why quantizing that single tensor recovers most of the possible size savings.

Quantized text encoder

TextEncoder.int8emb.mlpackage is produced by post-training quantization of the converted fp16 model via coremltools.optimize.coreml, applying OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8", granularity="per_channel") to only the text_model.embeddings.token_embedding weight (op-name-targeted); attention and MLP weights remain fp16. The embedding table is a gather weight — each forward pass reads one row per token, so quantization error does not compound through a matmul chain.

A whole-model int8 variant was also evaluated: ≈270 MiB unzipped (vs. ≈352 MiB for embedding-only) but with worst-case parity of 0.9964 vs. 0.9996. The embedding-only variant was published as the better size/precision trade-off for retrieval workloads, where score margins matter.

Parity vs. PyTorch reference

Cosine similarity between the PyTorch fp32 reference (transformers) and the converted Core ML models, on a fixture of 4 images (solid red/green/blue, synthetic sunset gradient) × 4 texts. Acceptance gate: > 0.99 per item, with unchanged text↔image rankings.

fp16 encoders

item	cosine
image: solid_red	0.9999988
image: solid_green	0.9999992
image: solid_blue	0.9999992
image: gradient_sky	0.9999987
text: "a red square"	0.9999995
text: "a photo of a sunset over the ocean"	0.9999997
text: "a solid blue color"	0.9999995
text: "a green field"	0.9999996

Worst case 0.9999987. Text↔image rankings match the PyTorch reference exactly across all 4×4 pairs.

int8-embedding text encoder

text	cosine
"a red square"	0.99978
"a photo of a sunset over the ocean"	0.99976
"a solid blue color"	0.99962
"a green field"	0.99975

Worst case 0.99962; rankings identical to both the PyTorch reference and the fp16 model.

Conversion details

Converted from the Hugging Face transformers reference implementation. Each tower was traced independently (torch.jit.trace on thin wrappers around SiglipModel.vision_model / .text_model) and converted with:

• convert_to="mlprogram", compute_precision=FLOAT16
• minimum_deployment_target: macOS 15

Pinned toolchain: torch==2.7.0, transformers==5.14.1, coremltools==9.0, numpy==2.3.5. Note that numpy>=2.4.0 breaks the coremltools==9.0 PyTorch frontend for models using torch.nn.MultiheadAttention (SigLIP's vision pooling head) — see apple/coremltools#2633; pin numpy<2.4.0 until the fix ships in a release.

License

Apache-2.0, inherited from the base model. Free for commercial and closed-source use.