ewin-reg/embeddinggemma-2-int8-multimodal

EmbeddingGemma 2 int8: multimodal embedding model in 798 MiB

This is google/embeddinggemma-2 with the weights of the text, vision and audio towers packed to int8. It keeps the same API and the same 768-dimension embeddings. It is about 44% smaller on disk than the bf16 original (1,420 MiB). Nothing was trained: the weights are rounded once after training, with no calibration data.

It sits next to the smaller int4 release, ewin-reg/embeddinggemma-2-int4-multimodal (486 MiB). This int8 file is 64% bigger than that one, and it changes the top search result much less often.

In plain words

An embedding model turns text, pictures or sound into lists of numbers so a computer can search them by meaning. This is the same model as Google's, stored at a lower precision so the file is smaller.

• Searching pictures with text, and text with pictures, gave the same ranking quality as the original in our test (no measurable loss).
• The best answer was almost always the same as the original's: the top result changed for 0.8% of picture-to-text searches and 0.3% of text-to-picture searches.
• It is about 44% smaller, not tiny. If you need the smallest file, use the int4 release, which is smaller but changes the top result more often.
• Sound input was not tested on this file. If you rely on audio search, check it yourself first.

At a glance

	Original (bf16)	int4 release	This model (int8)
Weights on disk	1,420 MiB	486.2 MiB	797.9 MiB
Image-to-text top-1 changes vs original	none	15.3%	0.8%
Text-to-image top-1 changes vs original	none	3.1%	0.3%
Image-to-text top-10 overlap	1.000	0.916	0.992
Mean cosine similarity to original	1.000	0.9959	0.99995
Image and text retrieval, NDCG@10 loss	none	0.25%	about 0.00% (95% interval upper bound 0.10%)

All three rows after the first column were measured on the same 1,000 COCO images against the same fp32 reference, in one gate run. The original model's NDCG@10 on this test is 0.8234.

The memory saving at run time is much smaller than the disk saving, because Python, torch and the activations stay the same size. This card does not include a separate memory or speed measurement for the int8 package.

Quick start

You need transformers==5.19.0, sentence-transformers>=6.1 and accelerate.

import sys, torch
from huggingface_hub import snapshot_download

path = snapshot_download("ewin-reg/embeddinggemma-2-int8-multimodal")
sys.path.insert(0, path)
import qpack3

st = qpack3.load_st(path, dtype=torch.bfloat16, device="cuda")  # or "cpu"
q = st.encode(["task: search result | query: a red bicycle"])
d = st.encode(["title: none | text: a bicycle leaning on a wall"])
img = st.encode([pil_image])            # list of PIL images

Use the original model's prompts: queries start with task: search result | query: and documents with title: none | text: . The loader builds the model on empty weights and attaches the packed tensors, so it never allocates a full-precision copy. Standard from_pretrained does not load this package; use qpack3.py from this repository.

How it was made

• Post-training quantization only. No training, fine-tuning, distillation or calibration data.
• Every linear layer in the text, vision and audio towers, and the token embedding table, is stored as int8 with a scale for each group of 16 weights. Scales are stored as 8-bit values per row with fp16 row bounds.
• Kept in bf16: the patch input projection, the vision position table (cut to 2,520 rows), the vision embedding projection, norms, and the audio convolutions and embed_audio.
• quant_config.json lists every layer under int8_layers and sets audio_bits and embed_bits to 8. Its top-level bits: 4 and the text and vision description strings describe the int4 format family; for this package the int8_layers list covers all text and vision layers, so they are stored as int8.
• The package was built on a Kaggle CPU kernel and checked on a Kaggle T4 GPU.

Retrieval quality

Evaluation uses 1,000 COCO 2014 validation images from the Karpathy test split, five captions each, with the unquantized fp32 model as the reference. The same protocol was used for the int4 release. Reported: NDCG@10 and Recall@10 in both directions, with a paired bootstrap interval, plus top-1 changes.

Metric	Result for this model
Image and text NDCG@10 loss	about 0.00% (95% interval upper bound 0.10%)
Image-to-text top-1 changes	0.8%
Text-to-image top-1 changes	0.3%
Image-to-text top-10 overlap	0.992
Mean cosine to original	0.99995

How to read this: the ranking quality loss is below the 1% limit I set, and the top answer matches the original in 99.2% (picture to text) and 99.7% (text to picture) of searches. It is not an exact copy of the original.

Limits to know before you use it

• Audio was not evaluated for this package. The audio tower is stored as int8, but the gate run skipped audio. The earlier audio test also saturated (Recall@1 0.998), so it could not tell models apart.
• Retrieval was measured in fp32. A bf16 retrieval run on this package never completed (three Kaggle runs failed), so bf16 quality is unmeasured. The self-test cosine values against the reference are near 1.0 (about 0.997 to 1.002 in bf16 noise) but are not a retrieval measurement.
• Keep max_soft_tokens at 280 or lower (the default). The vision position table was cut to 2,520 rows, which is exact for 280 soft tokens and wrong above it.
• Evaluated on COCO image and caption retrieval only. Other languages, domains and document images are untested.
• No memory or speed benchmark was run for this package.

Bias, risks and safety

This model inherits the biases and risks of google/embeddinggemma-2. Quantization was not evaluated for changes in bias or harmful output. Embedding models can encode social biases into search results; check results before using them in high-stakes settings.

Environmental impact

Quantization and evaluation ran as short Kaggle CPU and T4 GPU jobs. No training was done, and no energy or emissions figures were measured.

Citation and acknowledgements

Quantized derivative of Google's EmbeddingGemma 2; see the base model card for citation details. Packaged by ewin-reg.