surus-ai/embeddinggemma-2-onnx

EmbeddingGemma-2 — ONNX (INT8 quantized)

Pre-exported ONNX version of google/embeddinggemma-2 (text-only, 768-d embeddings) for CPU inference without torch or sentence-transformers.

What's in this repo

File	Size	Description
model_q8.onnx	261 MB	INT8 dynamically quantized ONNX model
tokenizer.json	31 MB	HuggingFace fast tokenizer

Usage (Python)

import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

PROMPT = "task: search result | query: "

tok = Tokenizer.from_file("tokenizer.json")
tok.enable_padding()
tok.enable_truncation(max_length=8192)

sess = ort.InferenceSession("model_q8.onnx", providers=["CPUExecutionProvider"])

def embed_query(text: str) -> list[float]:
    encoded = tok.encode(PROMPT + text)
    ids = np.array([encoded.ids], dtype=np.int64)
    mask = np.array([encoded.attention_mask], dtype=np.int64)
    hidden = sess.run(["last_hidden_state"], {"input_ids": ids, "attention_mask": mask})[0]
    m = mask[:, :, np.newaxis].astype(np.float32)
    pooled = (hidden * m).sum(axis=1) / m.sum(axis=1)
    norm = np.linalg.norm(pooled, axis=1, keepdims=True)
    return [float(v) for v in (pooled / norm)[0]]

vec = embed_query("machine learning for NLP")
# len(vec) == 768

Dependencies

Only onnxruntime, tokenizers, and numpy. No torch.

Quality

Compared against the original torch + sentence-transformers pipeline:

• FP32 vs INT8 cosine similarity: 0.990–0.995
• Search ranking: identical top-5 results
• Latency (CPU, 4 cores): 24ms per query (INT8) vs 68ms (torch)

Export process

Exported with export_onnx.py from surus-lat/benchy-agent:

1. Load google/embeddinggemma-2 text encoder (vision/audio disabled)
2. Export to ONNX via legacy TorchScript trace (opset 17)
3. Dynamic INT8 quantization via onnxruntime.quantization