digital-creature/embeddinggemma-2-text-onnx-int8

EmbeddingGemma 2 — text tower, ONNX, dynamic int8 (CPU)

The text encoder of google/embeddinggemma-2 as a plain two-input ONNX graph, quantized for fast query embedding on CPU. Its vectors live in the same 768-d space as the full multimodal model, so text queries embedded here can be matched against documents (text, images, audio, video) embedded by the full model, for example through a hosted API.

What it is

• Source: the fp32 text graph onnx/model.onnx of onnx-community/embeddinggemma-2-ONNX.
• Text only. The source graph declares three required media inputs besides the text ones (image_features, video_features, audio_features, each [n, 512]). Here they are empty [0, 512] constants, so the graph takes only input_ids and attention_mask. Text outputs are unchanged: the fp32 version of this graph matched the hosted google/embeddinggemma-2 API at cosine 1.0000.
• Dynamic int8 on MatMul only (onnxruntime.quantization.quantize_dynamic, int8 weights). The token embedding table stays fp32: a lookup reads a few rows and needs no dequantization pass.
• Output sentence_embedding (float32, 768-d, pooled and normalized as in the source graph).
• Files: onnx/model.onnx + onnx/model.onnx.data, config.json, tokenizer.json, tokenizer_config.json. No safetensors.
• config.json additionally carries max_position_embeddings and hidden_size at the top level, which some loaders read there rather than in text_config. hidden_size is set to 768, the width of the output embedding; the transformer itself is 512 wide (text_config.hidden_size).

Measured (2026-10-16)

	fp32 text graph	this int8
Weights	1084 MB	674 MB
Query, ONNX Runtime, Apple M-series, 4 threads	135 ms	45 ms
Cosine to fp32 (6 queries in EN, DE, FR)	1	0.989–0.995

Prompts

EmbeddingGemma 2 expects task prompts on text; the graph adds none. Prefix queries with task: search result | query: and documents with title: {title} | text: {content} (title: none without a title). Images, audio and video take no prompt.

Licence

Apache License 2.0, as the source model. Modified from google/embeddinggemma-2 via onnx-community/embeddinggemma-2-ONNX: media inputs replaced with empty constants, MatMul weights quantized to int8, two keys added to config.json.