keptn/gemma4-31b-w4a8

Gemma 4 31B-it · W4A8 · for radiance

google/gemma-4-31B-it in one 20 GiB file for the radiance inference engine on AMD RDNA4. The file reads text, images and video and includes Google's assistant head for speculative decoding.

File	gemma4-31b-w4a8.rad, 19.98 GiB
Needs	radiance 1.5.0 or newer; 1, 2 or 4 cards with 32 GB (tested on Radeon AI PRO R9700)
Format	weights int4, one scale per 128 columns (GPTQ, with SmoothQuant on q/k/v); activations int8 per row; lm_head int8
Speculation	Google's Gemma 4 assistant head, bf16, 3 draft tokens
Inputs	text · image · video
Context	262,144 tokens
Version	1.0 (2026-10-13)

Speed

Radeon AI PRO R9700 at 215 W, radiance 1.5.0, fp8 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

	1 card	2 cards	4 cards
decode, one stream	78.1	137.4	215.1
prefill at 24k tokens	1,540	2,777	3,737
4 streams, total	138	374	518

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens).

	mean KL	99th percentile KL	top token the same	perplexity ratio
this file	0.0517	0.721	94.6%	×1.056
keptn/gemma4-31b-i8g	0.00236	0.0311	98.8%	×1.004
keptn/gemma4-31b-w5	0.00803	0.107	97.8%	×1.010

Quick start

With the compose files from the radiance repository (set RADIANCE_MODELS, RADIANCE_API_KEY and the group ids in .env, see radiance's README):

hf download keptn/gemma4-31b-w4a8 gemma4-31b-w4a8.rad --local-dir ~/models
cp -r radiance/deploy/compose ~/radiance-compose && cd ~/radiance-compose
cp .env.example .env                    # RADIANCE_MODELS=~/models, RADIANCE_API_KEY, RENDER_GID, VIDEO_GID
docker compose -f gemma4-31b-w4a8.yaml up -d

Or directly (two cards):

docker run -d --init --name gemma4 --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e HSA_ENABLE_IPC_MODE_LEGACY=1 -e ROCR_VISIBLE_DEVICES=0,1 \
    stilldeadcode/radiance:1.5.0 --model /models/gemma4-31b-w4a8.rad --host 0.0.0.0 --port 8000 \
    --tp 2 --kv-cache-dtype fp8 --max-model-len 65536 --max-num-seqs 8 \
    --max-num-batched-tokens 2048 --num-speculative-tokens 3 --temp 1.0 --top-p 0.95 --top-k 64

	change
one card	ROCR_VISIBLE_DEVICES=0, drop --tp 2, --max-num-seqs 2
four cards	ROCR_VISIBLE_DEVICES=0,1,2,3, --tp 4
full context	--max-model-len 262144 --max-num-seqs 2
thinking	per request: "chat_template_kwargs": {"enable_thinking": true}

ROCR_VISIBLE_DEVICES picks the cards. Use the numbers of your cards (amd-smi list). Three cards do not work, because Gemma's 32 attention heads do not split into three.

Memory and context

KV pool at --max-model-len 262144, fp8 KV, radiance 1.5.0, from radiance's startup report. Tokens of context in total, shared by all sequences:

	2 sequences	4 sequences	8 sequences
1 card	270,448	248,848	205,632
2 cards	524,256	1,007,088	963,728
4 cards	524,144	1,048,416	2,097,040

On one card each sequence also keeps its own sliding-window state. So more sequences leave less for the shared pool.

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance w5pr-3a701182-b115. There is one measurement for each vision or audio encoder, made with gemma4-31b-w5.rad.

input	cases	same answer	mean KL	top token the same
images (DocVQA and AI2D, fixed selection)	20	16/20	0.471	66/70
video (MVBench, 5 clips, 4 frames each)	5	4/5	0.0272	31/32

How it was made

1. Start from the bf16 checkpoint google/gemma-4-31B-it.
2. Calibration set: 110 chats, 165,626 tokens (25 with thinking).
3. SmoothQuant on q/k/v (alpha 0.5). A few channels in front of attention carry large values. In int8 they cost all other channels their precision. SmoothQuant moves that range into the weights. The model still computes the same function. Script and scales: smoothing/.
4. GPTQ to int4 with one f16 scale per 128 columns: gemma4-31b-w4a8-lmi8.recipe, built with rad-convert.
5. lm_head int8 (round to nearest). Google's assistant head and the vision tower stay bf16.

python smoothing/smooth_checkpoint.py gemma-4-31B-it smoothing/smoothquant-stats.npz 0.5 gemma-4-31B-it-sq
rad-convert gemma-4-31B-it-sq --draft-model gemma-4-31B-it-assistant \
    --recipe gemma4-31b-w4a8-lmi8.recipe --set quantization_config.quant_method=compressed-tensors \
    --quant output.weight=rtn:codes=i8,group=128,scale=bf16,clamp=sym -o gemma4-31b-w4a8.rad

GPTQ needs statistics from the calibration set (calib= in the recipe).

Calibration: own corpus of 110 chats, 165,626 tokens (older than keptn/quant-calibration).

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

• radiance runs on AMD RDNA4 cards.

Credits

• Gemma 4 support in radiance: based on the initial work of sfjes (sfjes@vmdaemon.org) in PR #75.
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance).
• Base model, assistant head and vision tower: Google.
• Test sets: MT-Bench, DocVQA, AI2D.

License

Apache 2.0, the license of the base model. See LICENSE.