Gemma 4 12B-it 路 bf16 路 for radiance
google/gemma-4-12B-it as one 24.95 GiB file for the radiance inference engine on AMD RDNA4. The weights are as Google ships them, with Google's assistant head for speculative decoding.
| File | gemma4-12b-bf16.rad, 24.95 GiB |
| Needs | radiance 1.17.0 or newer; 1, 2 or 4 cards with 32 GB (tested on Radeon AI PRO R9700). Tested with radiance 2.0.0. |
| Format | weights bf16, as in the checkpoint |
| Speculation | Google's Gemma 4 assistant head, 3 draft tokens |
| Inputs | text 路 image 路 audio |
| Context | 262,144 tokens |
| Version | 1.0 (2026-10-30) |
Speed
Radeon AI PRO R9700 at 215 W, radiance gemma4-12b branch 75664cc0, bf16 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):
| 1 card | 2 cards | 4 cards | |
|---|---|---|---|
| decode, one stream | 62.7 | 113.3 | 182.8 |
| prefill at 24k tokens | 2,743 | 4,620 | 6,191 |
| 4 streams, total | 163 | 300 | 474 |
Quality
Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens).
| mean KL | 99th percentile KL | top token the same | perplexity ratio | |
|---|---|---|---|---|
| this file | 0.00026 | 0.00325 | 99.6% | 脳1.000 |
| keptn/gemma4-12b-i8g | 0.00416 | 0.0535 | 98.1% | 脳1.004 |
Quick start
Start the model on two cards. Save this as radiance.yaml:
server:
host: 0.0.0.0
port: 8000
models:
gemma4-12b-bf16:
model: /models/gemma4-12b-bf16.rad
tp: 2
p2p: on
kv-cache-dtype: bf16
max-model-len: 262144
max-num-seqs: 8
num-speculative-tokens: 3
max-num-batched-tokens: 2048
default-max-tokens: 32768
max-tokens-cap: 65536
temp: 1.0
top-p: 0.95
top-k: 64
chat-template-kwargs: {"enable_thinking":false}
The entry name gemma4-12b-bf16 is the model id in requests. Then run:
hf download keptn/gemma4-12b-bf16 gemma4-12b-bf16.rad config.json --local-dir ~/models
docker pull stilldeadcode/radiance:2.0.0
docker run -d --init --name gemma4 --network host --group-add video --group-add render \
--device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
-v ~/models:/models:ro -v $PWD/radiance.yaml:/etc/radiance/radiance.yaml:ro \
-e ROCR_VISIBLE_DEVICES=0,1 \
stilldeadcode/radiance:2.0.0 --config /etc/radiance/radiance.yaml
| change | |
|---|---|
| one card | ROCR_VISIBLE_DEVICES=0, tp: 1, remove p2p: on |
| four cards | ROCR_VISIBLE_DEVICES=0,1,2,3, tp: 4 |
| no drafter | num-speculative-tokens: 0 |
| thinking | per request: "chat_template_kwargs": {"enable_thinking": true} |
The sampling values are the ones used in the measured runs (temperature 1.0, top_p 0.95, top_k 64).
radiance 1.17.0 reads command-line flags, not this file. To use it, take the flags from this card's version history.
Memory and context
KV pool at --max-model-len 262144, bf16 KV, radiance gemma4-12b branch 75664cc0, from radiance's startup report. Tokens of context in total, shared by all sequences:
| 2 sequences | 4 sequences | 8 sequences | |
|---|---|---|---|
| 1 card | 451,184 | 408,752 | 322,928 |
| 2 cards | 524,336 | 1,048,624 | 1,184,112 |
| 4 cards | 524,336 | 1,048,624 | 1,610,544 |
Images, audio and video
The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance PR #338 final (67866b87). There is one measurement for each vision or audio encoder, made with gemma4-12b-bf16.rad.
| input | cases | same answer | mean KL | top token the same |
|---|---|---|---|---|
| images (DocVQA and AI2D, fixed selection) | 20 | 20/20 | 0.00184 | 149/149 |
| audio (LibriSpeech test-clean, 12 clips) | 12 | 11/12 | 0.00068 | 350/351 |
Word error rate on the audio clips: HF 6.68% 路 radiance 6.81%.
How it was made
- Downloaded
google/gemma-4-12B-it(revision707f0a3b) andgoogle/gemma-4-12B-it-assistant(revision3b0b0a4a). - Converted the checkpoint with the recipe
gemma4-12b-bf16: every weight stays bf16 as in the checkpoint. The position table of the vision tower is stored as two taps. - Added the assistant head as the drafter. The vision tower and the audio encoder stay as in the checkpoint.
- Checked that
rad-convertreads all 725 checkpoint tensors.
rad-convert gemma-4-12B-it --draft-model gemma-4-12B-it-assistant \
--set quantization_config.quant_method=compressed-tensors \
--recipe gemma4-12b-bf16.recipe -o gemma4-12b-bf16.rad
Calibration: no calibration data.
Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).
Limits
- radiance runs on AMD RDNA4 cards.
- The file needs radiance 1.17.0 or newer.
- One card holds the file (24.95 GiB) and the KV cache for 451,184 tokens at 2 sequences. Eight sequences at the full context need more cards.
- Google lists text, image and audio as inputs. Audio is tested on 12 LibriSpeech clips only.
Credits
- Base model, assistant head, vision tower and audio encoder: Google.
- radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance).
- Test sets: MT-Bench, DocVQA, AI2D, LibriSpeech.
License
Apache 2.0, the license of the base model. See LICENSE.
- Downloads last month
- 113