gemma4-e4b-w4a16-it-v2

W4A16 weight-only quantized version of google/gemma-4-E4B-it, produced by Team Horizon using llm-compressor.

Weights are quantized to 4-bit integers; activations remain in FP16. The model retains full multimodal (vision + text) capability.

---

Key takeaways:

• +5.1% mean recovery over the FP16 base — the quantized model outperforms the original on these benchmarks.
• Image understanding improved from 0.68 → 0.78 (+14.7%), suggesting the calibration mix (TextVQA + DocVQA + UltraChat) benefited visual reasoning.
• ~18% energy reduction vs the base model at a fraction of the memory footprint.
• Document analysis score is within 3.5% of the base model.

---

Calibration Dataset

Quantization used 512 mixed-modal calibration samples:

Dataset	Samples	Purpose
lmms-lab/textvqa	256	Scene-text visual QA
lmms-lab/DocVQA	128	Document visual QA
HuggingFaceH4/ultrachat_200k	128	General instruction following

---

Usage

vLLM (recommended)

pip install vllm

from vllm import LLM, SamplingParams
from PIL import Image

llm = LLM(
    model="amir22010/gemma4-e4b-w4a16-it-v2",
    dtype="half",
    max_model_len=8192,
    gpu_memory_utilization=0.90,
    trust_remote_code=True,
)

image = Image.open("your_image.jpg")

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

outputs = llm.generate(
    {
        "prompt": "<|image|>\nDescribe this image in detail.",
        "multi_modal_data": {"image": image},
    },
    sampling_params,
)

print(outputs[0].outputs[0].text)

vLLM OpenAI-compatible server

vllm serve amir22010/gemma4-e4b-w4a16-it-v2 \
  --dtype half \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

Then query via the OpenAI client:

from openai import OpenAI
import base64

client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")

with open("your_image.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="amir22010/gemma4-e4b-w4a16-it-v2",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}},
                {"type": "text", "text": "What does this image show?"},
            ],
        }
    ],
    max_tokens=512,
)

print(response.choices[0].message.content)

Transformers

from transformers import AutoProcessor, AutoModelForCausalLM
from PIL import Image
import torch

model_id = "amir22010/gemma4-e4b-w4a16-it-v2"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

image = Image.open("your_image.jpg")
prompt = "<|image|>\nDescribe this image."

inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=512)

print(processor.decode(output[0], skip_special_tokens=True))

---

Quantization Details

Property	Value
Method	W4A16 (weight-only, RTN)
Targets	All Linear layers
Excluded	lm_head, .*audio.*
Tool	llm-compressor QuantizationModifier
Calibration samples	512
Max sequence length	2048
Base model dtype	auto (bfloat16)

---

Hardware Requirements

Configuration	VRAM
Minimum (inference only)	~10 GB
Recommended	16–24 GB
Multi-GPU	tensor_parallel_size: 1 (single GPU sufficient)

---

Limitations

• Quantization introduces minor accuracy degradation on document analysis tasks (~3.5% vs base).
• Energy consumption, while improved vs base, remains higher than heavily-compressed Round 1 competitors. Further PTQ or sparsity methods could close this gap.
• Audio modality layers are excluded from quantization and remain in base precision.

---

Citation

If you use this model, please cite the original Gemma 4 work:

@misc{gemma4_2025,
  title  = {Gemma 4: Open Models Based on Gemini Research and Technology},
  author = {Google DeepMind},
  year   = {2025},
  url    = {https://ai.google.dev/gemma}
}

---

License

This model is derived from google/gemma-4-E4B-it and is subject to the Gemma Terms of Use.