EmbeddingGemma 2 Core ML v1.0.0
8036f0b verified 6 days ago
21.7 kB
---
license: apache-2.0
base_model: google/embeddinggemma-2
base_model_relation: quantized
library_name: coreml
pipeline_tag: feature-extraction
tags: [coreml, embeddings, multimodal-embedding, image-text, sentence-similarity, ios, macos, neural-engine]
---
<!-- Rendered by stages/s7_publish (card.py + model_card.md) from the stage reports. Do not edit on the Hub. -->

# EmbeddingGemma 2 for Core ML

This is [`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2) (Google DeepMind's text and image embedding model, one
shared 768-d space) converted to **multifunction Core ML models that run on the Apple Neural Engine**, one per image
budget. Every stage of the conversion was checked against the original PyTorch model. It is an unofficial conversion. It is not made or
endorsed by Google.

- **Base model:** `google/embeddinggemma-2` at revision `914f7f89142e33e77833254d9c9b90c3cef7303b` (`model.safetensors` sha256 `197a32965d4b1105faf060417baa899e193fb73cd401f42ec9295234d5553d79`).
- **Modalities:** text and images. Audio and video are not converted.
- **Variant shipped:** `int8_table4`: int8 per-channel conv weights, int4 per-block (32) table; tables and norms FP16.
- **Layout:** one folder per image budget, `b70/` and `b280/`. Each holds the text functions and that budget's image
  functions; a device uses one budget only, so it downloads one folder's archive.
- **Size:** apps download one file, `b<budget>/EmbeddingGemma2.aar` (an LZFSE Apple Archive of the package and the tokenizer:
  343.2 MB for budget 70, 352.9 MB for budget 280; the s4 gate is 360 MB for the
  larger), described by `b<budget>/manifest.json` (size, SHA-256 and image budget). Unpacked it is 435.0 /
  454.9 MB; the package alone is 402.8 / 422.8 MB. The same files are also in each
  folder unpacked, each listed with its size and SHA-256 in `b<budget>/files.json`.
- **Runs on:** iOS 18+ and macOS 15+ (the model uses `ios18` ops). It is built for `.cpuAndNeuralEngine`.
- **Swift package:** [`EmbeddingGemma`](https://github.com/Delavalom/embeddinggemma-coreml) (download, verification,
  tokenizer, image preprocessing, plus an on-disk vector index). See "Use from Swift" below.

## License and terms

- **The weights in this repo are a modified version of `google/embeddinggemma-2`, under the Apache License, Version 2.0.**
  This is the license the upstream model card declares at the pinned revision (`license: apache-2.0`, linking
  Google's [Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license), which is the Apache 2.0 text).
  The original model is by Google DeepMind. See `LICENSE` (the full Apache 2.0 text) and `NOTICE`.
- **Changes made (Apache 2.0, section 4(b)):** the original weights were re-implemented in PyTorch, converted to Core
  ML, cast to FP16, and compressed (int8 per-channel convolution weights, int4 per-block embedding table). The
  vectors are close to the original's but not identical (see "Accuracy").
- **Upstream use policy, passed through:** the upstream model card says *"Deployments must adhere to the
  [Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy)."* That requirement comes from
  the upstream model, and it applies to this conversion too. The upstream card also says developers are responsible
  for application-level safeguards (retrieval filtering, fairness testing) suited to their use case. Read
  Google's [model card](https://huggingface.co/google/embeddinggemma-2) for its intended uses, limitations and ethical
  considerations. They all apply here.
- The tokenizer files (`tokenizer/tokenizer.json`, `tokenizer/tokenizer_config.json`) are unmodified copies from
  the upstream revision above.
- The conversion code and the Swift package are Apache-2.0 too (the GitHub repo's `LICENSE` and `NOTICE`).

## Functions

One `.mlpackage` per image budget, `b70/EmbeddingGemma2.mlpackage` and `b280/EmbeddingGemma2.mlpackage`, each with
nine functions with fixed shapes: the four text functions (the same in both) and the five image functions of its
budget. Both were split from one 14-function package, and every function gives the same output bytes as there.
Every function returns `embedding`, float32 `[1, 768]`, L2-normalized.

| function | input | tokens |
|---|---|---|
| `embed_text_16`, `_32`, `_64`, `_128` | `input_ids` int32 `[1, L]`, `attention_mask` int32 `[1, L]` (1 = token, 0 = padding) | L = 16 / 32 / 64 / 128 |
| `embed_image_70_{tall,portrait,square,landscape,wide}` (in `b70/`) | `pixels` `[1, 3, H, W]`, RGB in 0-1 | 60-70 image tokens |
| `embed_image_280_{tall,portrait,square,landscape,wide}` (in `b280/`) | `pixels` `[1, 3, H, W]`, RGB in 0-1 | 256-280 image tokens |

Image shapes (H x W). The bucket comes from the aspect ratio r = width / height: tall r < 0.6, portrait r < 0.83,
square r <= 1.2, landscape r <= 1.67, wide above that.

| budget | tall | portrait | square | landscape | wide |
|---|---|---|---|---|---|
| 70 | 576 x 240 | 480 x 336 | 384 x 384 | 336 x 480 | 240 x 576 |
| 280 | 1152 x 528 | 960 x 672 | 768 x 768 | 672 x 960 | 528 x 1152 |

- **Text** is `<bos>` + tokens(prompt + text) + `<eos>`, padded with `<pad>` (id 0) to the smallest length that
  fits. Longer inputs are cut to 128 tokens, keeping `<bos>` and `<eos>` (the upstream model allows up to 8K
  tokens). Use the upstream prompts: `task: search result | query: ` for queries, `title: none | text: ` for
  documents (also `task: classification | query: ` and `task: clustering | query: `).
- **Images** are resized straight to the bucket size (no crop or padding). Use 8-bit sRGB, antialiased bicubic
  resampling, and scale by 1/255 with no mean or std normalization, as the reference processor does. The Swift
  package does this.
- **Matryoshka:** you can truncate the 768-d output to 512, 256 or 128 dims and re-normalize. The Swift package
  defaults to 256. All accuracy numbers below are at 768 dims.

## Accuracy against the original model

Measured by the pipeline on 200 hand-written texts (120 of them search queries) and 200 images (CC0 and public
domain photos from Wikimedia Commons, plus synthetic screenshots, memes and receipts; 14 categories), at both image
budgets. Cosine values are the **minimum** over all inputs, not the mean.

| stage | what is compared | min cosine, text | min cosine, image |
|---|---|---|---|
| s1 | our PyTorch rewrite (FP32) vs the official `sentence-transformers` model | 1 - 1.4e-12 | 1 - 1.0e-08 |
| s2 | Core ML FP32 vs the official model | 1 - 1.2e-12 | 1 - 1.9e-08 |
| s3 | Core ML FP16 on the Neural Engine vs the official model (at least 99.86% of ops on the Neural Engine) | 0.99996 | 0.99404 (1st percentile 0.99848) |
| s4 | **shipped `int8_table4`** on `CPU_AND_NE` vs the official model | **0.99791** | **0.99133** (1st percentile 0.99539) |
| s5 | the Swift package end to end on a Mac (tokenizer + preprocessing + model) vs the official model on the same decoded pixels; vs Python Core ML on identical inputs: 1 - 1.0e-15 | 0.99017 (text and image) | |
Retrieval with the shipped variant (the 120 queries ranking the 200 images, budget 70 / 280):
| metric | budget 70 | budget 280 |
|---|---|---|
| text-to-image recall@10 vs the FP reference on the same bucketed images | 0.962 | 0.966 |
| 5th-percentile Spearman of the top 50, same reference | 0.966 | 0.972 |
| text-to-image recall@10 vs the exact-aspect FP model (recorded, not gated) | 0.844 | 0.874 |
For 98.3% of queries (worst budget), the exact-aspect model's top-1 image is in the shipped model's top
10. Image-to-image precision@10 by category label is at most 0.012 below the exact-aspect FP model.
Swift tokenization matches the reference ids for every fixture text (`True`). The 99th-percentile pixel
difference against the reference preprocessing is 0.000000 (0-1 units).
**Caveats:**
- **The retrieval gates use a bucketed FP reference.** The upstream processor resizes each image to its own
  aspect-preserving size. This model has five fixed shapes per budget, so even with no compression, the bucketed
  vector differs from the exact-aspect one (min cosine 0.916 at budget 70, 0.893 at 280).
  That is why recall@10 against the exact-aspect model is lower than recall@10 against the bucketed reference. The
  compression and FP16 gates (min cosine >= 0.99, recall@10 >= 0.95, Spearman >= 0.9) are therefore measured
  against the FP model on the same bucketed images. Bucketing gets its own task-level gates (top-1 in top 10
  >= 0.95, category precision@10 drop <= 0.03). Measured by category labels, bucketing costs no retrieval
  quality. Details: ADR-0007 in the GitHub repo.
- **Two images fall below the FP16 bar on the Neural Engine.** In FP16 on the Neural Engine, fixture images i141
  and i161 reach 0.99404 / 0.99405 cosine. The other 398 are at 0.998 or above. On the GPU these two images pass,
  but the GPU is 2.8x slower at budget 70. They change the top 10 of only a few queries: with only those two
  vectors swapped in, recall@10 stays at 0.9975 (budget 70) and 0.9992 (budget 280). They were accepted at task
  level, and images stay on the Neural Engine. The lowest image cosines of the shipped variant are i161 0.99133 (`embed_image_280_landscape`), i141 0.99322 (`embed_image_70_wide`), i165 0.99336 (`embed_image_280_portrait`).
  Details: ADR-0006.
- The test set is small: 200 texts, 200 images. Multilingual text, code and long documents were not tested
  separately. For quality on standard benchmarks, see Google's model card. This conversion only claims closeness
  to the original.
## Speed and memory
**Mac** (s4, `CPU_AND_NE`, median of 20 runs after warm-up). These were measured on the build Mac; the v1.0.0
build Mac is an M1 Pro with macOS 26. `embed_text_16` takes 4.4 ms and `embed_image_70_square` takes
43.8 ms. In every function, at least 99.81% of the ops are placed on the Neural Engine (`MLComputePlan`, minimum
over the 14 functions, measured on the package the two budget packages were split from).

**iPhone** (s6, PhotoSearchDemo's benchmark: each iPhone runs its own budget, `ImageBudget.auto`, 280 on A17 Pro
and newer and 70 otherwise: the newest 1,000 photos at 70 or 200 at 280, and 20 queries; times cover preprocessing +
prediction):

| device | iOS | ms per image, 70 (median) | ms per image, 280 (median) | ms per query (median) | peak memory, MB |
|---|---|---|---|---|---|
| iPhone14,3 | 27.0.1 | 64.0 | not run | 6.7 | 630 |

Worst device: 64.0 ms per image at 70, not measured (no A17 Pro or newer iPhone ran s6 yet) at 280, 6.7 ms per query, peak 630 MB (the
gate is 800 MB; the target for budget 70 is 60 ms, which is recorded, not gated).

**Storage:** the Swift package downloads the archive of the device's image budget, `b<budget>/EmbeddingGemma2.aar`
(343.2 MB at 70, 352.9 MB at 280, resumable), checks its size and SHA-256 against
`b<budget>/manifest.json`, unpacks it (435.0 / 454.9 MB) with the AppleArchive framework and
compiles the package. It needs free space for the archive, the unpacked files and the compiled model while it
installs; afterwards it keeps the compiled model and the tokenizer. The s4 gate measures the larger archive,
352.9 MB (limit 360 MB).

## Use from Swift

```swift
// Package.swift: .package(url: "https://github.com/Delavalom/embeddinggemma-coreml", from: "1.0.0")
import EmbeddingGemma
import EmbeddingIndex
let model = try await EmbeddingGemma.load(.huggingFace(revision: EmbeddingGemma.defaultRevision))
let photo = try await model.embed(image: cgImage, orientation: .up)  // [Float], 256-d by default, normalized
let query = try await model.embed(query: "receipts from last week")
```

The package pins this repo (`delavalom/embeddinggemma-2-coreml`) at a commit and the SHA-256 of each budget's `manifest.json`, which pins its
archive's size and SHA-256. `budget: .auto` picks 280 tokens on A17 Pro and newer iPhones and on Macs, and 70 on
other devices, and loads that budget's folder (`b280/` or `b70/`) only.

## Use from Python (coremltools, macOS 15+)

```python
import numpy as np, coremltools as ct
from tokenizers import Tokenizer
from torchvision.transforms.functional import pil_to_tensor, resize, InterpolationMode
from PIL import Image
BUDGET = 280  # the folder you downloaded: b280/ or b70/; its text functions are the same in both
PKG = f"b{BUDGET}/EmbeddingGemma2.mlpackage"
tok = Tokenizer.from_file(f"b{BUDGET}/tokenizer/tokenizer.json")
# self-check: the layout the model was traced with is <bos> ... <eos>
assert tok.encode("task: search result | query: screenshots of tweets").ids == \
    [2, 8071, 236787, 3927, 1354, 1109, 7609, 236787, 75490, 529, 46331, 1]
def fn(name):
    return ct.models.MLModel(PKG, function_name=name, compute_units=ct.ComputeUnit.CPU_AND_NE)
def embed_text(text, prompt="task: search result | query: "):
    ids = tok.encode(prompt + text).ids
    ids = ids if len(ids) <= 128 else ids[:127] + [ids[-1]]
    n = next(L for L in (16, 32, 64, 128) if L >= len(ids))
    x = np.zeros((1, n), np.int32); x[0, :len(ids)] = ids
    m = np.zeros((1, n), np.int32); m[0, :len(ids)] = 1
    return fn(f"embed_text_{n}").predict({"input_ids": x, "attention_mask": m})["embedding"][0]
SHAPES = {70: {"tall": (576, 240), "portrait": (480, 336), "square": (384, 384), "landscape": (336, 480),
               "wide": (240, 576)},
          280: {"tall": (1152, 528), "portrait": (960, 672), "square": (768, 768), "landscape": (672, 960),
                "wide": (528, 1152)}}
def bucket(r):  # r = width / height
    return ("tall" if r < 0.6 else "portrait" if r < 0.83 else "square" if r <= 1.2
            else "landscape" if r <= 1.67 else "wide")
def embed_image(path):
    img = Image.open(path).convert("RGB")  # like the reference: 8-bit RGB, ICC profile ignored
    name = bucket(img.width / img.height)
    px = resize(pil_to_tensor(img), list(SHAPES[BUDGET][name]), InterpolationMode.BICUBIC, antialias=True)
    px = px.float() / 255
    return fn(f"embed_image_{BUDGET}_{name}").predict({"pixels": px[None].numpy()})["embedding"][0]
def truncate(v, dims=256):  # Matryoshka
    v = v[:dims]; return v / np.linalg.norm(v)
```

The model is compiled each time `MLModel` is created. To make many calls, keep one instance per function. The
Python path is not gated; the gated path is the Swift package (s5).

## Reproducibility

The model was built by an 8-stage pipeline in the GitHub repo. Each stage writes `artifacts/<stage>/report.json`,
which is checked against `gates.toml`. A stage counts only if its report passed and is current. Each report
records `git_sha` and `code_hash` (sha256 over the stage's code, the lock file and its gate table), and
`inputs_sha256` chains it to the reports it read. The reports behind this release:

| stage | status | git_sha | code_hash |
|---|---|---|---|
| s0_reference | passed | `59b9bf9863cb` | `9a6a53c2abe21038` |
| s1_rewrite | passed | `7645eff4d951` | `e98a54bc6bbc6abf` |
| s2_coreml_fp32 | passed | `82b02a4da1ec` | `21bb94d9d33d0230` |
| s3_coreml_ane | passed | `8e33af79825f` | `edb80276335be2a4` |
| s4_compress | passed | `b9064e9abb2b` | `e68c9ca0822062d0` |
| s5_swift | passed | `90679438068a` | `464d01a178d66eb5` |
| s6_device | passed | `d5b2e6edc33e` | `70b253e29d57b0ce` |
Reference environment: transformers 5.20.2, sentence-transformers 6.2.0; conversion:
coremltools 9.0. To rebuild: `just setup`, then `just stage 0` through `just stage 5` (`just stage 6` needs
an iPhone). Numbers measured on the CPU and the Neural Engine can
vary slightly from run to run.
# Results
Generated by `python -m scripts.gen_results` from `artifacts/*/report.json` and `docs/results/*.json`. Do not edit by hand.

Model: `google/embeddinggemma-2` at revision `914f7f89142e33e77833254d9c9b90c3cef7303b`.

## Gates

| stage | status | metric | value | gate | ok |
|---|---|---|---|---|---|
| s0_reference | passed | license_apache | true | == true | yes |
| s0_reference | passed | key_inventory_written | true | == true | yes |
| s0_reference | passed | deterministic | true | == true | yes |
| s1_rewrite | passed | strict_load | true | == true | yes |
| s1_rewrite | passed | min_cosine_text | 1 - 1.4e-12 | >= 0.9999 | yes |
| s1_rewrite | passed | min_cosine_image | 1 - 1e-08 | >= 0.9999 | yes |
| s2_coreml_fp32 | passed | min_cosine_text | 1 - 1.2e-12 | >= 0.9999 | yes |
| s2_coreml_fp32 | passed | min_cosine_image | 1 - 1.9e-08 | >= 0.9999 | yes |
| s3_coreml_ane | passed | min_cosine_text | 0.999965 | >= 0.995 | yes |
| s3_coreml_ane | passed | min_cosine_image | 0.994037 | >= 0.99 | yes |
| s3_coreml_ane | passed | min_p1_cosine_image | 0.998482 | >= 0.995 | yes |
| s3_coreml_ane | passed | min_recall_at_10 | 0.9675 | >= 0.95 | yes |
| s3_coreml_ane | passed | min_p5_spearman_top50 | 0.987025 | >= 0.9 | yes |
| s3_coreml_ane | passed | min_ane_fraction | 0.99864 | >= 0.95 | yes |
| s4_compress | passed | min_cosine | 0.991329 | >= 0.99 | yes |
| s4_compress | passed | min_recall_at_10 | 0.961667 | >= 0.95 | yes |
| s4_compress | passed | min_p5_spearman_top50 | 0.966043 | >= 0.9 | yes |
| s4_compress | passed | min_t2i_top1_in_top10 | 0.983333 | >= 0.95 | yes |
| s4_compress | passed | max_i2i_label_p10_drop | 0.012 | <= 0.03 | yes |
| s4_compress | passed | max_download_mb | 352.911 | <= 360 | yes |
| s5_swift | passed | token_ids_identical | true | == true | yes |
| s5_swift | passed | max_pixel_abs_diff | 0 | <= 0.003922 | yes |
| s5_swift | passed | min_cosine_runtime | 1 - 1e-15 | >= 0.9999 | yes |
| s5_swift | passed | min_cosine | 0.990173 | >= 0.99 | yes |
| s5_swift | passed | min_recall_at_10 | 0.9575 | >= 0.95 | yes |
| s5_swift | passed | min_p5_spearman_top50 | 0.96449 | >= 0.9 | yes |
| s5_swift | passed | min_t2i_top1_in_top10 | 0.983333 | >= 0.95 | yes |
| s5_swift | passed | max_i2i_label_p10_drop | 0.0065 | <= 0.03 | yes |
| s6_device | passed | max_peak_rss_mb | 630.065 | <= 800 | yes |
| s6_device | passed | max_cold_compile_peak_mb | 1517.66 | <= 2100 | yes |
| s6_device | passed | min_devices | 1 | >= 1 | yes |
| s7_publish | not run | - | - | - | - |

## s4 variants

Chosen: `int8_table4` (winner: `int8_table4`). Variants are tried smallest first; the first that passes every gate wins, so later ones may not have run. Download limit: 360 MB.

| variant | download MB | package MB | min cosine | recall@10 | p5 Spearman top-50 | exact-aspect top-1 in top 10 | i2i p@10 drop | min ANE fraction | recall@10 vs exact-aspect | passes | description |
|---|---|---|---|---|---|---|---|---|---|---|---|
| int8_table4 | 358.669 | 452.4 | 0.991329 | 0.961667 | 0.966043 | 0.983333 | 0.012 | 0.998131 | 0.844167 | all | int8 per-channel conv weights, int4 per-block (32) table; tables and norms FP16 |
Shipped per image budget (download MB = the device archive; `max_download_mb` is the larger): budget 70 `EmbeddingGemma2-b70.aar` 343.178 MB (9 functions), budget 280 `EmbeddingGemma2-b280.aar` 352.911 MB (9 functions). Same outputs as the 14-function package (CPU_ONLY, bitwise): True.

## s5 Swift parity

Pixels (0-1 units; PNG and synthetic are gated, JPEG includes the decoder and is recorded only, ADR-0004):

| inputs | p99 | max | bit-exact |
|---|---|---|---|
| png | 0 | 0 | 60/60 |
| synthetic | 0 | 0 | 20/20 |
| jpeg | 0.00784314 | 0.121569 | 0/340 |

| comparison | min cosine | recall@10 (worst budget) | p5 Spearman top-50 |
|---|---|---|---|
| Swift vs Python Core ML, same inputs (gated) | 1 - 1e-15 | - | - |
| Swift vs FP reference on Swift's pixels (gated) | 0.990173 | 0.9575 | 0.96449 |
| Swift vs s0 (recorded) | 0.971191 | 0.946667 | 0.952845 |
| FP on Swift's pixels vs s0: the decoder (recorded) | 0.971614 | 0.968333 | 0.969143 |

Against s0 (recorded only; on JPEGs it includes the decoder, ADR-0008):

| vectors | n | min cosine | mean cosine |
|---|---|---|---|
| text query | 120 | 0.998874 | 0.99951 |
| text document | 80 | 0.997913 | 0.999262 |
| image 70_jpeg | 170 | 0.978874 | 0.999002 |
| image 70_png | 30 | 0.995402 | 0.99898 |
| image 280_jpeg | 170 | 0.971191 | 0.998763 |
| image 280_png | 30 | 0.999682 | 0.999764 |

## Device benchmarks (s6)

| source | device | status | key metrics |
|---|---|---|---|
| artifacts/s6_device/report.json | iPhone14,3 | passed | max_peak_rss_mb=630.065, max_cold_compile_peak_mb=1517.66, min_devices=1 |
| docs/results/s6_iPhone14,3.json | iPhone14,3 | passed | max_peak_rss_mb=630.065, max_cold_compile_peak_mb=1517.66, min_devices=1 |

## Limitations

- Text and images only. Text is limited to 128 tokens; images are limited to budgets of 70 or 280 tokens and five
  fixed aspect buckets each.
- Results are close to the original model, not bit-identical (see "Accuracy"). Do not mix vectors from this model
  with vectors from the PyTorch model or another conversion in the same index. The Swift index stores the model
  revision for that reason.
- Intel Macs have no Neural Engine; they run on the CPU/GPU and are slower. The fastest settings are on the Neural
  Engine with `.cpuAndNeuralEngine`.
- The first load compiles the model on the device, which takes a while. The Swift package caches the compiled
  model.
- Everything in the upstream card's "Limitations" and "Ethical Considerations and Risks" applies.

## Citation

Please cite the original model as Google DeepMind's model card asks
([`google/embeddinggemma-2`](https://huggingface.co/google/embeddinggemma-2)). For this conversion:

```bibtex
@software{embeddinggemma_coreml,
  title  = {EmbeddingGemma 2 for Core ML},
  author = {Delavalom},
  year   = {2026},
  url    = {https://huggingface.co/delavalom/embeddinggemma-2-coreml},
  note   = {Core ML conversion of google/embeddinggemma-2}
}
```