Gemma 4 12B-it 路 bf16 路 for radiance

google/gemma-4-12B-it as one 24.95 GiB file for the radiance inference engine on AMD RDNA4. The weights are as Google ships them, with Google's assistant head for speculative decoding.

File gemma4-12b-bf16.rad, 24.95 GiB
Needs radiance 1.17.0 or newer; 1, 2 or 4 cards with 32 GB (tested on Radeon AI PRO R9700). Tested with radiance 2.0.0.
Format weights bf16, as in the checkpoint
Speculation Google's Gemma 4 assistant head, 3 draft tokens
Inputs text 路 image 路 audio
Context 262,144 tokens
Version 1.0 (2026-10-30)

Speed

Radeon AI PRO R9700 at 215 W, radiance gemma4-12b branch 75664cc0, bf16 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

1 card 2 cards 4 cards
decode, one stream 62.7 113.3 182.8
prefill at 24k tokens 2,743 4,620 6,191
4 streams, total 163 300 474

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens).

mean KL 99th percentile KL top token the same perplexity ratio
this file 0.00026 0.00325 99.6% 脳1.000
keptn/gemma4-12b-i8g 0.00416 0.0535 98.1% 脳1.004

Quick start

Start the model on two cards. Save this as radiance.yaml:

server:
  host: 0.0.0.0
  port: 8000
models:
  gemma4-12b-bf16:
    model: /models/gemma4-12b-bf16.rad
    tp: 2
    p2p: on
    kv-cache-dtype: bf16
    max-model-len: 262144
    max-num-seqs: 8
    num-speculative-tokens: 3
    max-num-batched-tokens: 2048
    default-max-tokens: 32768
    max-tokens-cap: 65536
    temp: 1.0
    top-p: 0.95
    top-k: 64
    chat-template-kwargs: {"enable_thinking":false}

The entry name gemma4-12b-bf16 is the model id in requests. Then run:

hf download keptn/gemma4-12b-bf16 gemma4-12b-bf16.rad config.json --local-dir ~/models
docker pull stilldeadcode/radiance:2.0.0
docker run -d --init --name gemma4 --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -v $PWD/radiance.yaml:/etc/radiance/radiance.yaml:ro \
    -e ROCR_VISIBLE_DEVICES=0,1 \
    stilldeadcode/radiance:2.0.0 --config /etc/radiance/radiance.yaml
change
one card ROCR_VISIBLE_DEVICES=0, tp: 1, remove p2p: on
four cards ROCR_VISIBLE_DEVICES=0,1,2,3, tp: 4
no drafter num-speculative-tokens: 0
thinking per request: "chat_template_kwargs": {"enable_thinking": true}

The sampling values are the ones used in the measured runs (temperature 1.0, top_p 0.95, top_k 64).

radiance 1.17.0 reads command-line flags, not this file. To use it, take the flags from this card's version history.

Memory and context

KV pool at --max-model-len 262144, bf16 KV, radiance gemma4-12b branch 75664cc0, from radiance's startup report. Tokens of context in total, shared by all sequences:

2 sequences 4 sequences 8 sequences
1 card 451,184 408,752 322,928
2 cards 524,336 1,048,624 1,184,112
4 cards 524,336 1,048,624 1,610,544

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance PR #338 final (67866b87). There is one measurement for each vision or audio encoder, made with gemma4-12b-bf16.rad.

input cases same answer mean KL top token the same
images (DocVQA and AI2D, fixed selection) 20 20/20 0.00184 149/149
audio (LibriSpeech test-clean, 12 clips) 12 11/12 0.00068 350/351

Word error rate on the audio clips: HF 6.68% 路 radiance 6.81%.

How it was made

  1. Downloaded google/gemma-4-12B-it (revision 707f0a3b) and google/gemma-4-12B-it-assistant (revision 3b0b0a4a).
  2. Converted the checkpoint with the recipe gemma4-12b-bf16: every weight stays bf16 as in the checkpoint. The position table of the vision tower is stored as two taps.
  3. Added the assistant head as the drafter. The vision tower and the audio encoder stay as in the checkpoint.
  4. Checked that rad-convert reads all 725 checkpoint tensors.
rad-convert gemma-4-12B-it --draft-model gemma-4-12B-it-assistant \
    --set quantization_config.quant_method=compressed-tensors \
    --recipe gemma4-12b-bf16.recipe -o gemma4-12b-bf16.rad

Calibration: no calibration data.

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

  • radiance runs on AMD RDNA4 cards.
  • The file needs radiance 1.17.0 or newer.
  • One card holds the file (24.95 GiB) and the KV cache for 451,184 tokens at 2 sequences. Eight sequences at the full context need more cards.
  • Google lists text, image and audio as inputs. Audio is tested on 12 LibriSpeech clips only.

Credits

License

Apache 2.0, the license of the base model. See LICENSE.

Downloads last month
113
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for keptn/gemma4-12b-bf16

Finetuned
(310)
this model