README.md

---
license: apache-2.0
base_model: google/siglip2-base-patch16-256
pipeline_tag: feature-extraction
tags:
  - coreml
  - siglip2
  - siglip
  - clip
  - image-text
  - embeddings
  - vision
  - apple-silicon
---

# SigLIP2 base-patch16-256 — Core ML

Core ML (`.mlpackage`) conversion of [`google/siglip2-base-patch16-256`](https://huggingface.co/google/siglip2-base-patch16-256) for on-device text and image embedding on Apple platforms. The image and text towers ship as separate models; both produce 768-dimensional, L2-normalized embeddings in a shared space, so image↔text similarity is a single dot product.

Converted and maintained by [nodevorg](https://huggingface.co/nodevorg) for [Alarian](https://alarian.ai)'s semantic frame search. Not affiliated with Google.

## Files

| file | size | sha256 |
|---|---|---|
| `ImageEncoder.mlpackage.zip` | 163 MiB | `406938456c8f8e91f01632cb74131fdfb993064d8d0bc8f6fbbd142709f1ba66` |
| `TextEncoder.int8emb.mlpackage.zip` — **recommended** | 323 MiB | `11471872101a6ca88dd51ed9de668b728ae9d09c0f853aeac8a6cc96f1c109d6` |
| `TextEncoder.mlpackage.zip` | 506 MiB | `4f31f38a1729a0044f0ffc95a940dc1a6a2fdcd5fcf78a03cbb662bada6cbafb` |
| `tokenizer.json` | 33 MiB | `cb9140fae3ac5122c972d37adf83e1248471a38147ad76f8215c8872c6fd8322` |
| `tokenizer_config.json` | — | `14afe629fe4959b9e0d51e1852b8d9f7ad074f90a1a7125a4fcdd17f06e78fc8` |
| `special_tokens_map.json` | — | `baec30ea10906f16adb8c18af7a34023002c1746542612b8b41c9f09e1351351` |

`TextEncoder.int8emb` is a drop-in replacement for the fp16 `TextEncoder` — identical input/output names, shapes, and dtypes — with the token-embedding table quantized to int8 (36% smaller download, worst-case parity 0.9996; details below). Use the full-fp16 text encoder only if you need maximum precision.

The tokenizer files are copied verbatim from the source checkpoint (Gemma BPE, 256k vocabulary).

## Model interface

### ImageEncoder

- **Input** — `pixel_values`: Core ML `ImageType`, RGB, 256×256. Preprocessing is baked into the model (`scale=1/127.5`, `bias=[-1,-1,-1]`, matching the checkpoint's `SiglipImageProcessor` configuration): resize to 256×256 and pass raw `[0, 255]` RGB — no caller-side normalization. Resize with an aspect-ratio squash rather than a center crop; that matches training.
- **Output** — `image_embedding`: 768-dim float16, L2-normalized.

### TextEncoder (both variants)

- **Input** — `input_ids`: 1×64 int32 token ids. Tokenize with the included tokenizer using `padding="max_length"`, `max_length=64`, lowercased input (`do_lower_case=true`). There is deliberately **no `attention_mask` input**: SigLIP's text tower trains with unmasked bidirectional attention over the fixed 64-token sequence and pools the last sequence position, so a mask would deviate from training.
- **Output** — `text_embedding`: 768-dim float16, L2-normalized.

Because both outputs are normalized in-graph, cosine similarity is `dot(image_embedding, text_embedding)`.

## Usage (Swift)

```swift
import CoreML

let image = try MLModel(contentsOf: compiledImageEncoderURL)
let text  = try MLModel(contentsOf: compiledTextEncoderURL)

// Image: 256×256 RGB CVPixelBuffer, raw [0, 255] values
let imageOut = try image.prediction(
    from: MLDictionaryFeatureProvider(dictionary: ["pixel_values": pixelBuffer]))
let imageEmbedding = imageOut.featureValue(for: "image_embedding")!.multiArrayValue!

// Text: 64 int32 token ids (e.g. via swift-transformers' Tokenizers module,
// loading the tokenizer.json from this repo), lowercased, padded to length 64
let ids = try MLMultiArray(shape: [1, 64], dataType: .int32) // fill with token ids
let textOut = try text.prediction(
    from: MLDictionaryFeatureProvider(dictionary: ["input_ids": ids]))
let textEmbedding = textOut.featureValue(for: "text_embedding")!.multiArrayValue!

// Similarity = dot product (embeddings are already L2-normalized)
```

## Architecture note

The source checkpoint's `config.json` declares `model_type: siglip`, not `siglip2`. At this fixed-resolution, non-NaFlex tier, the SigLIP2-trained weights are architecturally identical to SigLIP v1 — `transformers.AutoModel` resolves the checkpoint to `SiglipModel`, and that is what was converted. Relative to SigLIP v1, only the weights and the tokenizer (Gemma BPE 256k, vs. 32k SentencePiece) differ.

Most of the text encoder's size is the 256k-row token-embedding table (256000 × 768 × 2 bytes ≈ 375 MiB at fp16), which is why quantizing that single tensor recovers most of the possible size savings.

## Quantized text encoder

`TextEncoder.int8emb.mlpackage` is produced by post-training quantization of the converted fp16 model via `coremltools.optimize.coreml`, applying `OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8", granularity="per_channel")` to **only** the `text_model.embeddings.token_embedding` weight (op-name-targeted); attention and MLP weights remain fp16. The embedding table is a gather weight — each forward pass reads one row per token, so quantization error does not compound through a matmul chain.

A whole-model int8 variant was also evaluated: ≈270 MiB unzipped (vs. ≈352 MiB for embedding-only) but with worst-case parity of 0.9964 vs. 0.9996. The embedding-only variant was published as the better size/precision trade-off for retrieval workloads, where score margins matter.

## Parity vs. PyTorch reference

Cosine similarity between the PyTorch fp32 reference (`transformers`) and the converted Core ML models, on a fixture of 4 images (solid red/green/blue, synthetic sunset gradient) × 4 texts. Acceptance gate: > 0.99 per item, with unchanged text↔image rankings.

### fp16 encoders

| item | cosine |
|---|---|
| image: solid_red | 0.9999988 |
| image: solid_green | 0.9999992 |
| image: solid_blue | 0.9999992 |
| image: gradient_sky | 0.9999987 |
| text: "a red square" | 0.9999995 |
| text: "a photo of a sunset over the ocean" | 0.9999997 |
| text: "a solid blue color" | 0.9999995 |
| text: "a green field" | 0.9999996 |

Worst case **0.9999987**. Text↔image rankings match the PyTorch reference exactly across all 4×4 pairs.

### int8-embedding text encoder

| text | cosine |
|---|---|
| "a red square" | 0.99978 |
| "a photo of a sunset over the ocean" | 0.99976 |
| "a solid blue color" | 0.99962 |
| "a green field" | 0.99975 |

Worst case **0.99962**; rankings identical to both the PyTorch reference and the fp16 model.

## Conversion details

Converted from the Hugging Face `transformers` reference implementation. Each tower was traced independently (`torch.jit.trace` on thin wrappers around `SiglipModel.vision_model` / `.text_model`) and converted with:

- `convert_to="mlprogram"`, `compute_precision=FLOAT16`
- `minimum_deployment_target`: macOS 15

Pinned toolchain: `torch==2.7.0`, `transformers==5.14.1`, `coremltools==9.0`, `numpy==2.3.5`. Note that `numpy>=2.4.0` breaks the `coremltools==9.0` PyTorch frontend for models using `torch.nn.MultiheadAttention` (SigLIP's vision pooling head) — see [apple/coremltools#2633](https://github.com/apple/coremltools/issues/2633); pin `numpy<2.4.0` until the fix ships in a release.

## License

Apache-2.0, inherited from the base model. Free for commercial and closed-source use.