Instructions to use H4RUming/embeddinggemma-2-text-vision-bnb-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use H4RUming/embeddinggemma-2-text-vision-bnb-4bit with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("H4RUming/embeddinggemma-2-text-vision-bnb-4bit") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Notebooks
- Google Colab
- Kaggle
EmbeddingGemma 2 β text + vision, 4-bit (bitsandbytes NF4)
A smaller build of google/embeddinggemma-2 for memory-constrained GPUs.
This is an unofficial derivative, not a Google release.
Which model to use
| Goal | Model |
|---|---|
| Better quality (image embeddings closer to the original) | H4RUming/embeddinggemma-2-text-vision-bnb-mixed-8bit-4bit β mixed 8-bit / 4-bit |
| Lowest VRAM | This model β 4-bit |
Use the mixed 8-bit model (vision encoder 8-bit + language model 4-bit) for better quality, and this 4-bit model to save VRAM. The two models produce the same text embeddings; they differ in image embeddings.
What was changed from the original
- Audio encoder removed (
audio_tower,embed_audio, 304.8M parameters). Text and image inputs never pass through the audio encoder, so this step alone leaves text and image embeddings identical to the original (max difference 0). - 4-bit quantization β bitsandbytes NF4, double quantization, compute dtype
bfloat16. The Linear layers of both the language model and the vision encoder are 4-bit; the token embedding table stays inbfloat16.
No weights were retrained.
| Original | Audio removed | Mixed 8-bit / 4-bit | This model | |
|---|---|---|---|---|
| GPU model memory | 1.39 GiB | 0.82 GiB | 0.49 GiB | 0.42 GiB |
Measurements
Measured on a Jetson Orin Nano 8GB (JetPack 7.2.1, torch 2.15.0+cu132). The reference is the audio-removed model in bfloat16.
Fidelity to the reference (cosine similarity of embeddings for the same input)
| This model (4-bit) | Mixed 8-bit / 4-bit | |
|---|---|---|
| Text (min) | 0.975 | 0.975 |
| Images β first pages of 24 real PDF documents (mean / min) | 0.931 / 0.875 | 0.974 / 0.955 |
| Retrieving a first-page image from the document title (1 of 24, reference 0.96) | 0.92 | 0.96 |
Korean text retrieval (title + summary of 5,000 public support-program notices; 42 hand-written natural-language questions; 33 synonym pairs)
| Audio removed (bf16) | This model (4-bit) | |
|---|---|---|
| Questions R@1 | 0.690 | 0.667 |
| Questions R@10 | 0.952 | 0.976 |
| Questions MRR | 0.801 | 0.789 |
| Synonym AUC | 0.815 | 0.801 |
| Throughput (batch 16) | 29.7 docs/s | 29.3 docs/s |
The 95% bootstrap interval of the MRR difference is [β0.062, +0.089], which includes 0. The sample is small, so treat these as indicative.
Quantization does not make the model faster. The weights are expanded to bfloat16 at compute time; what shrinks is memory.
Limitations
- Image embeddings are more sensitive to quantization than text embeddings. If image retrieval matters, use the mixed 8-bit model. Image retrieval quality was checked only on a small sample.
- Audio input is not supported (
ValueError). Video input was not tested. - Requires a CUDA GPU and
bitsandbytes. It cannot be loaded on CPU.
Usage
pip install "sentence-transformers>=6.1" "transformers>=5.19" torchvision bitsandbytes accelerate
import torch
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("H4RUming/embeddinggemma-2-text-vision-bnb-4bit", device="cuda")
model.max_seq_length = 2048 # the default is effectively unlimited, so set it yourself
q = model.encode("SME R&D support programs in artificial intelligence", prompt_name="SearchQuery")
d = model.encode(["title: none | text: Funding for R&D costs of AI-based products."])
print(model.similarity(q, d))
# Images arrive as float32, so wrap image encoding in autocast to keep dtypes consistent
from PIL import Image
with torch.autocast("cuda", dtype=torch.bfloat16):
img = model.encode([Image.open("page.png")])
Query and document prefixes (prompt_name) and the title: {title} | text: {content} document format follow the original model card.
License
Apache License 2.0, same as the original. Original: Google, google/embeddinggemma-2. The changes are listed under "What was changed from the original".
- Downloads last month
- 37
Model tree for H4RUming/embeddinggemma-2-text-vision-bnb-4bit
Base model
google/embeddinggemma-2