Instructions to use RedHatAI/gemma-4-26B-A4B-it-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/gemma-4-26B-A4B-it-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="RedHatAI/gemma-4-26B-A4B-it-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("RedHatAI/gemma-4-26B-A4B-it-NVFP4")
model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/gemma-4-26B-A4B-it-NVFP4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/gemma-4-26B-A4B-it-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/gemma-4-26B-A4B-it-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/gemma-4-26B-A4B-it-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4

SGLang

How to use RedHatAI/gemma-4-26B-A4B-it-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/gemma-4-26B-A4B-it-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/gemma-4-26B-A4B-it-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/gemma-4-26B-A4B-it-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/gemma-4-26B-A4B-it-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/gemma-4-26B-A4B-it-NVFP4 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4
```

gemma-4-26B-A4B-it-NVFP4

Model Overview

Model Architecture: Gemma4ForConditionalGeneration
- Input: Text / Image
- Output: Text
Model Optimizations:
- Weight quantization: FP4
- Activation quantization: FP4
Release Date: 2026-04-04
Version: 1.0
Model Developers: RedHatAI

This model is a quantized version of google/gemma-4-26B-A4B-it. It was evaluated on several tasks to assess its quality in comparison to the unquantized model.

Model Optimizations

This model was obtained by quantizing the weights and activations of google/gemma-4-26B-A4B-it to NVFP4 data type, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 4, reducing the disk size and GPU memory requirements by approximately 75%.

Weights are quantized to FP4 with a group size of 16, and activations are quantized to FP4 with local per-group scaling. Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor. Vision tower, embedding, output head, and MoE router layers are kept in their original precision.

Deployment

Use with vLLM

This model can be deployed using vLLM. For detailed instructions including multimodal inference, thinking mode, function calling, and benchmarking, see the Gemma 4 26B-A4B vLLM usage guide.

Start the vLLM server:

vllm serve RedHatAI/gemma-4-26B-A4B-it-NVFP4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --chat-template examples/tool_chat_template_gemma4.jinja \
  --limit-mm-per-prompt '{"image": 4, "audio": 1}'

Tip: For text-only workloads, pass --limit-mm-per-prompt '{"image": 0, "audio": 0}' to skip vision encoder memory allocation and free up GPU memory for a longer context window.

Send requests to the server:

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

model = "RedHatAI/gemma-4-26B-A4B-it-NVFP4"

messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)

generated_text = outputs.choices[0].message.content
print(generated_text)

Creation

This model was created by applying NVFP4 quantization with LLM Compressor, as presented in the code snippet below.

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import apply
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "google/gemma-4-26B-A4B-it"
SAVE_DIR = MODEL_ID.split("/")[1] + "-NVFP4"

model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

ds = load_dataset("mgoin/ultrachat_200k_s3", split="train_sft")
calibration_data = [ex["prompt"] for ex in ds.select(range(512))]

recipe = QuantizationModifier(
    targets=["Linear"],
    ignore=["lm_head", "re:.*embed.*", "re:.*router", "re:.*vision_tower.*"],
    scheme="NVFP4",
)

apply(model=model, tokenizer=tokenizer, recipe=recipe, calibration_data=calibration_data)
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

Evaluation

This model was evaluated on GSM8K Platinum, MMLU-Pro, IFEval, MATH-500, AIME 2025, GPQA Diamond, LiveCodeBench v6, and BFCLv4 (function calling) using lm-evaluation-harness, lighteval, and BFCL — all served with vLLM (OpenAI-compatible API). Accuracy results are reported both without and with thinking enabled; BFCLv4 was evaluated with thinking enabled.

Accuracy

Without thinking

Category	Benchmark	google/gemma-4-26B-A4B-it	RedHatAI/gemma-4-26B-A4B-it-NVFP4	Recovery
Instruction Following	IFEval (0-shot, prompt-level strict)	89.96	87.24	97.0%
Instruction Following	IFEval (0-shot, inst-level strict)	93.21	91.29	97.9%
Reasoning	GSM8K Platinum (0-shot, strict-match)	95.43	94.43	98.9%
	MMLU-Pro (0-shot, custom-extract)	83.47	81.65	97.8%
	MATH-500 (0-shot, pass@1)	84.80	87.60	103.3%
	AIME 2025 (0-shot, pass@1)	80.00	66.25	82.8%
	GPQA Diamond (0-shot, pass@1)	73.20	72.39	98.9%
Coding	LiveCodeBench v6 (0-shot, pass@1)	74.48	68.19	91.6%

With thinking

Category	Benchmark	google/gemma-4-26B-A4B-it	RedHatAI/gemma-4-26B-A4B-it-NVFP4	Recovery
Instruction Following	IFEval (0-shot, prompt-level strict)	94.39	94.15	99.7%
Instruction Following	IFEval (0-shot, inst-level strict)	96.16	95.84	99.7%
Reasoning	GSM8K Platinum (0-shot, strict-match)	95.95	95.31	99.3%
	MMLU-Pro (0-shot, custom-extract)	85.19	83.29	97.8%
	MATH-500 (0-shot, pass@1)	85.87	85.07	99.1%
	AIME 2025 (0-shot, pass@1)	88.75	81.25	91.5%
	GPQA Diamond (0-shot, pass@1)	80.81	77.61	96.0%
Coding	LiveCodeBench v6 (0-shot, pass@1)	77.90	72.76	93.4%
Tool Calling	BFCLv4 Overall	67.62%	62.96%	93.1%
	BFCLv4 Single Turn	83.85%	82.00%	97.8%
	BFCLv4 Multi-Turn	62.13%	62.13%	100.0%
	BFCLv4 Agentic	62.16%	50.33%	80.9%

Reproduction

The results were obtained using the following commands:

Each benchmark was run 3 times with different random seeds (1234, 2345, 3456) and the scores were averaged; AIME 2025 used 8 seeds.

vLLM server:

vllm serve RedHatAI/gemma-4-26B-A4B-it-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 69632 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --chat-template examples/tool_chat_template_gemma4.jinja \
  --limit-mm-per-prompt '{"image":0,"audio":0}' \
  --async-scheduling \
  --default-chat-template-kwargs '{"enable_thinking": true}'

Note: To reproduce the results without thinking, remove --default-chat-template-kwargs '{"enable_thinking": true}'. To run without tool calling, remove --enable-auto-tool-choice, --tool-call-parser gemma4, and --reasoning-parser gemma4.

GSM8K Platinum (lm-eval, 0-shot, 3 repetitions)

lm_eval --model local-chat-completions \
  --tasks gsm8k_platinum_cot_llama \
  --model_args "model=RedHatAI/gemma-4-26B-A4B-it-NVFP4,max_length=69632,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=1200" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_gsm8k_platinum.json \
  --seed 1234 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"

MMLU-Pro (lm-eval, 0-shot, 3 repetitions)

lm_eval --model local-chat-completions \
  --tasks mmlu_pro_chat \
  --model_args "model=RedHatAI/gemma-4-26B-A4B-it-NVFP4,max_length=69632,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=1200" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_mmlu_pro.json \
  --seed 1234 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"

IFEval (lm-eval, 0-shot, 3 repetitions)

lm_eval --model local-chat-completions \
  --tasks ifeval \
  --model_args "model=RedHatAI/gemma-4-26B-A4B-it-NVFP4,max_length=69632,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=32,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=1200" \
  --num_fewshot 0 \
  --apply_chat_template \
  --output_path results_ifeval.json \
  --seed 1234 \
  --gen_kwargs "do_sample=True,temperature=1.0,top_p=0.95,top_k=64,max_gen_toks=32000,seed=1234"

MATH-500, AIME 2025, GPQA Diamond, LiveCodeBench v6 (lighteval, 3 repetitions; 8 for AIME 2025)

litellm_config.yaml:

model_parameters:
  provider: hosted_vllm
  model_name: hosted_vllm/RedHatAI/gemma-4-26B-A4B-it-NVFP4
  base_url: http://0.0.0.0:8000/v1
  api_key: ''
  timeout: 3600
  concurrent_requests: 32
  generation_parameters:
    temperature: 1.0
    max_new_tokens: 65536
    top_p: 0.95
    top_k: 64
    seed: 1234

Run once per seed (changing seed in the config each time):

lighteval endpoint litellm litellm_config.yaml 'math_500|0' \
  --output-dir results/ --save-details

lighteval endpoint litellm litellm_config.yaml 'aime25|0' \
  --output-dir results/ --save-details

lighteval endpoint litellm litellm_config.yaml 'gpqa:diamond|0' \
  --output-dir results/ --save-details

lighteval endpoint litellm litellm_config.yaml 'lcb:codegeneration_v6|0' \
  --output-dir results/ --save-details

BFCLv4

BFCL requires the model to be registered in the leaderboard codebase before running evaluation.

Step 1 — Register the model in bfcl_eval/constants/model_config.py

Add the following entry to api_inference_model_map:

"gemma-4-26b-a4b-it-NVFP4": ModelConfig(
    model_name="gemma-4-26b-a4b-it-NVFP4",
    display_name="Gemma-4-26B-A4B-it-NVFP4 (FC)",
    url="https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4",
    org="Google",
    license="Apache 2.0",
    model_handler=OpenAICompletionsHandler,
    input_price=None,
    output_price=None,
    is_fc_model=True,
    underscore_to_dot=True,
),

Step 2 — Add the key to bfcl_eval/constants/supported_models.py

Add "gemma-4-26b-a4b-it-NVFP4" to the SUPPORTED_MODELS list.

Step 3 — Start the vLLM server (use the command at the top of this section; the --served-model-name flag ensures BFCL can find the model by its registered slug).

Step 4 — Generate responses and evaluate

bfcl generate --model gemma-4-26b-a4b-it-NVFP4 --test-category all
bfcl evaluate --model gemma-4-26b-a4b-it-NVFP4 --test-category all

Downloads last month: 1,146,947

Safetensors

Model size

15B params

Tensor type

F32

BF16

F8_E4M3

Model tree for RedHatAI/gemma-4-26B-A4B-it-NVFP4

Base model

google/gemma-4-26B-A4B

Finetuned

google/gemma-4-26B-A4B-it

Quantized

(301)

this model