thomasjvu/lisper-gemma4-e2b-audio-onnx-q2f16-experimental

Lisper Gemma 4 E2B Audio ONNX - Mixed q2f16 Experiment

This is an experimental WebGPU package for manual testing only.

It keeps the validated q4f16 package intact and changes only the decoder:

• decoder_model_merged_q2f16: trained Lisper decoder, 2-bit weight-only quantization with fp16 runtime tensors.
• embed_tokens_q4f16: trained Lisper embeddings, unchanged from the validated q4f16 package.
• audio_encoder_q4f16: official Gemma 4 E2B audio encoder, unchanged from the validated q4f16 package.
• vision_encoder_q4f16: official Gemma 4 E2B vision encoder, unchanged from the validated q4f16 package.

The embedding model is intentionally not q2f16. The current ONNX Runtime WebGPU GatherBlockQuantized kernel is hardcoded for 4-bit packing, so a q2 embedding package would be a likely runtime failure without a custom kernel or graph rewrite.

Expected App Config

VITE_LISPER_BROWSER_MODEL_ID=thomasjvu/lisper-gemma4-e2b-audio-onnx-q2f16-experimental
VITE_LISPER_BROWSER_DTYPE=q2f16-experimental

The app maps q2f16-experimental to a per-component dtype request:

• decoder_model_merged: q2f16
• embed_tokens: q4f16
• audio_encoder: q4f16
• vision_encoder: q4f16

Size

Local package payload is about 2.61 GB decimal / 2.43 GiB.

The validated q4f16 package is about 3.15 GB decimal / 2.93 GiB, so this experiment saves about 538 MB decimal by quantizing only the decoder to 2-bit.

Validation

See validation-summary.json.

All four ONNX sessions load with ONNX Runtime CPU. decoder_model_merged_q2f16.onnx passes onnx.checker; the audio and vision wrappers use Microsoft contrib ops, so generic ONNX checker reports missing contrib-op registrations even though ONNX Runtime loads them.