justinchuby/gemma-4-e2b-it-onnx

gemma-4-E2B-it — ONNX / OpenVINO

ONNX and OpenVINO exports of google/gemma-4-E2B-it, built with mobius and packaged for ONNX Runtime GenAI. INT4 variants are quantized with Olive.

Gemma 4 E2B is an any-to-any multimodal model, so every variant is a multi-component package with four ONNX models — decoder, embedding, vision_encoder, audio_encoder — plus a shared genai_config.json, tokenizer, and image / audio processor configs.

Variants

Each variant folder is <precision>/<target> (or a top-level folder for the OpenVINO packages). Pick the precision for your quality/size budget and the target for your runtime/EP.

Precisions

Precision	Folder prefix	Notes
FP16	f16/	Full-precision half.
BF16	bf16/	BFloat16 (no INT4).
INT4 · K-Quant (Q4_K_M)	Q4_K_M/	Olive K-Quant, block 32. Recommended INT4.
INT4 · NF4	NF4/	NF4 4-bit weights.
INT4 · OpenVINO	openvino-*/	K-Quant + MatMulNBitsToQDQ, compiled for OpenVINO.

Targets (execution-provider builds)

Target (subfolder)	Runtime / EP	Provider in genai_config
cuda	ONNX Runtime GenAI + CUDA	cuda (fused GroupQueryAttention)
default	ONNX Runtime GenAI (CPU / DML)	empty → portable ONNX
onnx-standard	any conformant ONNX runtime	empty → pure ai.onnx, no com.microsoft ops
webgpu	ONNX Runtime Web / WebGPU	webgpu

Full matrix (present in this repo)

Folder	Precision	Target	Size
f16/cuda	FP16	CUDA	~11.3 GB
f16/default	FP16	CPU / DML	~11.3 GB
f16/onnx-standard	FP16	portable ONNX	~11.3 GB
f16/webgpu	FP16	WebGPU	~11.3 GB
bf16/cuda	BF16	CUDA	~11.3 GB
bf16/default	BF16	CPU / DML	~11.3 GB
bf16/onnx-standard	BF16	portable ONNX	~11.3 GB
Q4_K_M/cuda	INT4 (K-Quant)	CUDA	~8.1 GB
Q4_K_M/default	INT4 (K-Quant)	CPU / DML	~8.1 GB
NF4/cuda	INT4 (NF4)	CUDA	~7.9 GB
NF4/default	INT4 (NF4)	CPU / DML	~7.9 GB
openvino-epcontext/	INT4	OpenVINO EP (ORT)	~7.4 GB
openvino-ir/	INT4	OpenVINO runtime	~7.4 GB

Usage — ONNX Runtime GenAI (the f16 / bf16 / Q4_K_M / NF4 variants)

pip install onnxruntime-genai        # CPU / WebGPU / DML
pip install onnxruntime-genai-cuda   # for the */cuda variants

import onnxruntime_genai as og
model = og.Model("Q4_K_M/cuda")   # or f16/default, NF4/cuda, ...
# ... standard ORT GenAI generation loop ...

OpenVINO

Both OpenVINO packages are INT4 and require OpenVINO ≥ 2026.3 (the release that ships the opset-24 RMSNormalization / RotaryEmbedding ONNX-frontend translators, openvinotoolkit/openvino#35545).

openvino-epcontext/ — compiled graph for ONNX Runtime ✅ verified

Each component's model.onnx is a single com.microsoft::EPContext node (source="OpenVINOExecutionProvider") that points at the compiled OpenVINO IR (model.xml / model.bin) next to it, so ONNX Runtime's OpenVINO EP loads the precompiled graph instead of recompiling.

Verified end-to-end: the decoder loads via the OpenVINO EP and a forward pass returns logits (1, seq, 262144) fp16 (all finite), using ONNX Runtime 1.28 built against OpenVINO 2026.3.

import onnxruntime as ort
sess = ort.InferenceSession(
    "openvino-epcontext/decoder/model.onnx",
    providers=[("OpenVINOExecutionProvider", {"device_type": "NPU"})],  # or GPU / CPU
)

openvino-ir/ — OpenVINO IR

Native OpenVINO IR (openvino_model.xml / .bin) per component:

import openvino as ov
decoder = ov.Core().compile_model("openvino-ir/decoder/openvino_model.xml", "NPU")

runtime-wheels/

The exact wheels used to verify the OpenVINO packages (Linux, cp312) — OpenVINO 2026.3 + ONNX Runtime 1.28 with the OpenVINO EP — since no official release ships OpenVINO ≥ 2026.3 yet:

pip install runtime-wheels/openvino-2026.3.0-*.whl \
            runtime-wheels/onnxruntime_openvino-1.28.0-*.whl

Build recipe

Built via microsoft/olive-recipes google-gemma-4-E2B-it: MobiusBuilder for the FP16 / BF16 exports; MobiusBuilder → OnnxKQuantQuantization for INT4; the OpenVINO packages additionally run MatMulNBitsToQDQ → OpenVINOConversion → OpenVINOEncapsulation.

License

Inherits the Gemma license.