Gemma 4 12B-it 路 int8 路 for radiance

google/gemma-4-12B-it as one 13.79 GiB file for the radiance inference engine on AMD RDNA4. The weights of the trunk are int8 with one scale per 128 weights, rounded to nearest.

File gemma4-12b-i8g.rad, 13.79 GiB
Needs radiance 1.17.0 or newer; 1, 2 or 4 cards with 32 GB (tested on Radeon AI PRO R9700). Tested with radiance 2.0.0.
Format weights int8, one bf16 scale per 128 weights; activations int8, one scale per token and 128 values
Speculation Google's Gemma 4 assistant head, 3 draft tokens
Inputs text 路 image 路 audio
Context 262,144 tokens
Version 1.0 (2026-10-30)

Speed

Radeon AI PRO R9700 at 215 W, radiance gemma4-12b branch 75664cc0, bf16 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

1 card 2 cards 4 cards
decode, one stream 104.6 174.7 250.0
prefill at 24k tokens 3,597 5,883 7,069
4 streams, total 306 506 714

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens).

mean KL 99th percentile KL top token the same perplexity ratio
this file 0.00416 0.0535 98.1% 脳1.004
keptn/gemma4-12b-bf16 0.00026 0.00325 99.6% 脳1.000

Quick start

Start the model on two cards. Save this as radiance.yaml:

server:
  host: 0.0.0.0
  port: 8000
models:
  gemma4-12b-i8g:
    model: /models/gemma4-12b-i8g.rad
    tp: 2
    p2p: on
    kv-cache-dtype: bf16
    max-model-len: 262144
    max-num-seqs: 8
    num-speculative-tokens: 3
    max-num-batched-tokens: 2048
    default-max-tokens: 32768
    max-tokens-cap: 65536
    temp: 1.0
    top-p: 0.95
    top-k: 64
    chat-template-kwargs: {"enable_thinking":false}

The entry name gemma4-12b-i8g is the model id in requests. Then run:

hf download keptn/gemma4-12b-i8g gemma4-12b-i8g.rad config.json --local-dir ~/models
docker pull stilldeadcode/radiance:2.0.0
docker run -d --init --name gemma4 --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -v $PWD/radiance.yaml:/etc/radiance/radiance.yaml:ro \
    -e ROCR_VISIBLE_DEVICES=0,1 \
    stilldeadcode/radiance:2.0.0 --config /etc/radiance/radiance.yaml
change
one card ROCR_VISIBLE_DEVICES=0, tp: 1, remove p2p: on
four cards ROCR_VISIBLE_DEVICES=0,1,2,3, tp: 4
no drafter num-speculative-tokens: 0
thinking per request: "chat_template_kwargs": {"enable_thinking": true}

The sampling values are the ones used in the measured runs (temperature 1.0, top_p 0.95, top_k 64).

radiance 1.17.0 reads command-line flags, not this file. To use it, take the flags from this card's version history.

Memory and context

KV pool at --max-model-len 262144, bf16 KV, radiance gemma4-12b branch 75664cc0, from radiance's startup report. Tokens of context in total, shared by all sequences:

2 sequences 4 sequences 8 sequences
1 card 524,336 1,048,624 1,048,112
2 cards 524,336 1,048,624 1,547,056
4 cards 524,336 1,048,624 1,792,432

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance PR #338 final (67866b87). There is one measurement for each vision or audio encoder, made with gemma4-12b-bf16.rad.

input cases same answer mean KL top token the same
images (DocVQA and AI2D, fixed selection) 20 20/20 0.00184 149/149
audio (LibriSpeech test-clean, 12 clips) 12 11/12 0.00068 350/351

Word error rate on the audio clips: HF 6.68% 路 radiance 6.81%.

How it was made

  1. Downloaded google/gemma-4-12B-it (revision 707f0a3b) and google/gemma-4-12B-it-assistant (revision 3b0b0a4a).
  2. Converted the checkpoint with the recipe gemma4-12b-i8g, reusing the tensors of the bf16 file. Rounded to nearest at int8, one bf16 scale per 128 weights: the attention, the feed-forward layers, the lm_head and the assistant head.
  3. Left the vision tower, the audio encoder and the norms as in the checkpoint.
  4. Checked that rad-convert reads all 725 checkpoint tensors.
rad-convert gemma-4-12B-it --draft-model gemma-4-12B-it-assistant \
    --set quantization_config.quant_method=compressed-tensors \
    --recipe gemma4-12b-i8g.recipe --reuse gemma4-12b-bf16.rad -o gemma4-12b-i8g.rad

Calibration: no calibration data.

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

  • radiance runs on AMD RDNA4 cards.
  • The file needs radiance 1.17.0 or newer.
  • The trunk is rounded to nearest with no calibration data.
  • Images and audio are measured with the bf16 file only. This file uses the same vision tower and audio encoder, but the comparison is not run on it.
  • Google lists text, image and audio as inputs. Audio is tested on 12 LibriSpeech clips only.

Credits

License

Apache 2.0, the license of the base model. See LICENSE.

Downloads last month
136
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for keptn/gemma4-12b-i8g

Finetuned
(310)
this model