EmbeddingGemma 2 β€” text + vision, 4-bit (bitsandbytes NF4)

A smaller build of google/embeddinggemma-2 for memory-constrained GPUs.

This is an unofficial derivative, not a Google release.

Which model to use

Goal Model
Better quality (image embeddings closer to the original) H4RUming/embeddinggemma-2-text-vision-bnb-mixed-8bit-4bit β€” mixed 8-bit / 4-bit
Lowest VRAM This model β€” 4-bit

Use the mixed 8-bit model (vision encoder 8-bit + language model 4-bit) for better quality, and this 4-bit model to save VRAM. The two models produce the same text embeddings; they differ in image embeddings.

What was changed from the original

  1. Audio encoder removed (audio_tower, embed_audio, 304.8M parameters). Text and image inputs never pass through the audio encoder, so this step alone leaves text and image embeddings identical to the original (max difference 0).
  2. 4-bit quantization β€” bitsandbytes NF4, double quantization, compute dtype bfloat16. The Linear layers of both the language model and the vision encoder are 4-bit; the token embedding table stays in bfloat16.

No weights were retrained.

Original Audio removed Mixed 8-bit / 4-bit This model
GPU model memory 1.39 GiB 0.82 GiB 0.49 GiB 0.42 GiB

Measurements

Measured on a Jetson Orin Nano 8GB (JetPack 7.2.1, torch 2.15.0+cu132). The reference is the audio-removed model in bfloat16.

Fidelity to the reference (cosine similarity of embeddings for the same input)

This model (4-bit) Mixed 8-bit / 4-bit
Text (min) 0.975 0.975
Images β€” first pages of 24 real PDF documents (mean / min) 0.931 / 0.875 0.974 / 0.955
Retrieving a first-page image from the document title (1 of 24, reference 0.96) 0.92 0.96

Korean text retrieval (title + summary of 5,000 public support-program notices; 42 hand-written natural-language questions; 33 synonym pairs)

Audio removed (bf16) This model (4-bit)
Questions R@1 0.690 0.667
Questions R@10 0.952 0.976
Questions MRR 0.801 0.789
Synonym AUC 0.815 0.801
Throughput (batch 16) 29.7 docs/s 29.3 docs/s

The 95% bootstrap interval of the MRR difference is [βˆ’0.062, +0.089], which includes 0. The sample is small, so treat these as indicative.

Quantization does not make the model faster. The weights are expanded to bfloat16 at compute time; what shrinks is memory.

Limitations

  • Image embeddings are more sensitive to quantization than text embeddings. If image retrieval matters, use the mixed 8-bit model. Image retrieval quality was checked only on a small sample.
  • Audio input is not supported (ValueError). Video input was not tested.
  • Requires a CUDA GPU and bitsandbytes. It cannot be loaded on CPU.

Usage

pip install "sentence-transformers>=6.1" "transformers>=5.19" torchvision bitsandbytes accelerate
import torch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("H4RUming/embeddinggemma-2-text-vision-bnb-4bit", device="cuda")
model.max_seq_length = 2048          # the default is effectively unlimited, so set it yourself

q = model.encode("SME R&D support programs in artificial intelligence", prompt_name="SearchQuery")
d = model.encode(["title: none | text: Funding for R&D costs of AI-based products."])
print(model.similarity(q, d))

# Images arrive as float32, so wrap image encoding in autocast to keep dtypes consistent
from PIL import Image
with torch.autocast("cuda", dtype=torch.bfloat16):
    img = model.encode([Image.open("page.png")])

Query and document prefixes (prompt_name) and the title: {title} | text: {content} document format follow the original model card.

License

Apache License 2.0, same as the original. Original: Google, google/embeddinggemma-2. The changes are listed under "What was changed from the original".

Downloads last month
37
Safetensors
Model size
0.4B params
Tensor type
BF16
Β·
U8
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for H4RUming/embeddinggemma-2-text-vision-bnb-4bit

Quantized
(146)
this model