EmbeddingGemma 2 (text + vision) – Core ML, int8 weights

An int8 weight-quantized version of the text and vision packages from FluidInference/embeddinggemma-2-coreml (revision 3f978ddf92cac9aa07eb2dcf6eb36cf450f3dba4), which converts google/embeddinggemma-2 to Core ML. The download drops from about 880 MB to about 470 MB. The audio package is not included.

The interface matches the fp16 conversion. Only the token table format changes; see the upstream README and config.json for the full pipeline:

File Contents
EmbeddingGemma2Text.mlpackage functions embed_32 … embed_512, pack_256; int8 weights
EmbeddingGemma2Vision.mlpackage functions vision_70, vision_140, vision_280; int8 weights
embeddings.i8 token table, int8, [262144, 512], row-major
embeddings_scale.f16 one fp16 scale per table row: row[r] = embeddings.i8[r] * scale[r]
position_embeddings.f16 vision position table, unchanged
tokenizer.json unchanged

The input embeddings are row * sqrt(512), the same as upstream.

Quantization

  • Models. Linear symmetric int8 per output channel (coremltools 9.0 OpLinearQuantizerConfig, weight threshold 2048). Activations stay fp16. The public PTQ API rejects multifunction packages, so scripts/quantize.py runs the compression pass on the MIL program directly. It then deduplicates weights shared between functions, so the functions keep sharing one weight.bin.
  • Token table. bf16 to int8 with a symmetric fp16 scale per row (scripts/table_int8.py). Mean relative row error is 0.76%.

Evaluation

Tested on a Mac with Apple silicon, compute units CPU + Neural Engine. The test set is 150 COCO Karpathy test images, retrieved by their first caption, plus 532 short multilingual queries. Vectors are truncated to 256-d (Matryoshka).

R@1 R@5 R@10 cosine to fp16, images (min / mean) cosine to fp16, queries (min / mean)
fp16 (upstream) 0.873 0.980 0.993 – –
int8 (this repo) 0.867 0.980 0.993 0.9989 / 0.9997 0.9991 / 0.9999

For 96.8% of the queries, the top-1 image is the same as with fp16. Vectors are close to fp16 but not identical, so do not mix them in one index.

We also tried 4-bit variants and did not publish them:

  • int4 per-block packages fail to load on the Neural Engine for some functions.
  • 4-bit palettization lowers the query cosine to fp16 to 0.87 at the minimum.

License

Apache 2.0, the same as the upstream conversion. EmbeddingGemma is subject to the Gemma Terms of Use and the Gemma Prohibited Use Policy.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for lorelaiassistant/embeddinggemma-2-coreml-int8

Quantized
(1)
this model