EmbeddingGemma 2 (text + vision) β Core ML, int8 weights
An int8 weight-quantized version of the text and vision packages from
FluidInference/embeddinggemma-2-coreml
(revision 3f978ddf92cac9aa07eb2dcf6eb36cf450f3dba4), which converts
google/embeddinggemma-2 to Core ML.
The download drops from about 880 MB to about 470 MB. The audio package is not included.
The interface matches the fp16 conversion. Only the token table format changes; see the
upstream README and config.json for the full pipeline:
| File | Contents |
|---|---|
EmbeddingGemma2Text.mlpackage |
functions embed_32 β¦ embed_512, pack_256; int8 weights |
EmbeddingGemma2Vision.mlpackage |
functions vision_70, vision_140, vision_280; int8 weights |
embeddings.i8 |
token table, int8, [262144, 512], row-major |
embeddings_scale.f16 |
one fp16 scale per table row: row[r] = embeddings.i8[r] * scale[r] |
position_embeddings.f16 |
vision position table, unchanged |
tokenizer.json |
unchanged |
The input embeddings are row * sqrt(512), the same as upstream.
Quantization
- Models. Linear symmetric int8 per output channel (
coremltools9.0OpLinearQuantizerConfig, weight threshold 2048). Activations stay fp16. The public PTQ API rejects multifunction packages, soscripts/quantize.pyruns the compression pass on the MIL program directly. It then deduplicates weights shared between functions, so the functions keep sharing oneweight.bin. - Token table. bf16 to int8 with a symmetric fp16 scale per row
(
scripts/table_int8.py). Mean relative row error is 0.76%.
Evaluation
Tested on a Mac with Apple silicon, compute units CPU + Neural Engine. The test set is 150 COCO Karpathy test images, retrieved by their first caption, plus 532 short multilingual queries. Vectors are truncated to 256-d (Matryoshka).
| R@1 | R@5 | R@10 | cosine to fp16, images (min / mean) | cosine to fp16, queries (min / mean) | |
|---|---|---|---|---|---|
| fp16 (upstream) | 0.873 | 0.980 | 0.993 | β | β |
| int8 (this repo) | 0.867 | 0.980 | 0.993 | 0.9989 / 0.9997 | 0.9991 / 0.9999 |
For 96.8% of the queries, the top-1 image is the same as with fp16. Vectors are close to fp16 but not identical, so do not mix them in one index.
We also tried 4-bit variants and did not publish them:
- int4 per-block packages fail to load on the Neural Engine for some functions.
- 4-bit palettization lowers the query cosine to fp16 to 0.87 at the minimum.
License
Apache 2.0, the same as the upstream conversion. EmbeddingGemma is subject to the Gemma Terms of Use and the Gemma Prohibited Use Policy.
- Downloads last month
- -
Model tree for lorelaiassistant/embeddinggemma-2-coreml-int8
Base model
google/embeddinggemma-2