| --- |
| license: apache-2.0 |
| license_link: https://ai.google.dev/gemma/docs/gemma_4_license |
| base_model: |
| - google/gemma-4-26B-A4B-it |
| - z-lab/gemma-4-26B-A4B-it-DFlash |
| pipeline_tag: text-generation |
| tags: |
| - radiance |
| - rocm |
| - amd |
| - rdna4 |
| - fp8 |
| - moe |
| - speculative-decoding |
| - dflash |
| --- |
| |
| # Gemma 4 26B-A4B-it + DFlash, FP8 experts — radiance container |
|
|
| [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) as a single `.rad` |
| container for the [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine (AMD RDNA4, |
| ROCm), with z-lab's [DFlash](https://huggingface.co/z-lab/gemma-4-26B-A4B-it-DFlash) block-diffusion |
| drafter merged in for speculative decoding. radiance's Gemma 4 26B-A4B support is by Perire. |
|
|
| | | | |
| |---|---| |
| | File | `gemma4-26b-a4b-dflash-fp8.rad` — 33.94 GiB | |
| | Weights | the routed experts' gate_up projections FP8 E4M3 per 128×128 block (bf16 scales); their down projections, the attention, the dense MLP, the router, the norms and the tied embedding bf16 as the checkpoint ships them | |
| | Speculator | DFlash, drafting 15 tokens a step. Its linears and its head are FP8 E4M3 per 128×128 block; the drafter only proposes, and the model verifies every token it emits | |
| | Context | 262,144 tokens (the checkpoint's trained length) | |
| | Modalities | text | |
| | Needs | radiance 1.5.0 or newer | |
| |
| ## Serve |
| |
| ```sh |
| radiance --model gemma4-26b-a4b-dflash-fp8.rad --tp 2 --max-model-len 65536 --kv-cache-dtype fp8 \ |
| --max-num-seqs 8 --max-num-batched-tokens 2048 --draft-window adaptive --host 0.0.0.0 --port 8000 |
| ``` |
| |
| The Docker image is `stilldeadcode/radiance`. The server speaks the OpenAI API (`/v1/chat/completions`, |
| `/v1/completions`), with tool calls and structured output. `--draft-window adaptive` lets the drafter's |
| window follow how much of it is accepted; without it every step drafts the container's 15 tokens. |
| |
| Measured on two Radeon AI PRO R9700 (gfx1201), one sequence: |
| |
| | | prose | code | |
| |---|---|---| |
| | fixed window (15) | 113.7 tok/s | 216.7 tok/s | |
| | `--draft-window adaptive` | 159.0 tok/s | 249.3 tok/s | |
| |
| An 11.5K-token prompt is prefilled in 1.0 s. |
| |
| ## How this file was made |
| |
| ```sh |
| rad-convert google/gemma-4-26B-A4B-it --draft-model z-lab/gemma-4-26B-A4B-it-DFlash \ |
| --recipe gemma4-26b-a4b-df.recipe -o gemma4-26b-a4b-dflash-fp8.rad |
| ``` |
| |
| The recipe (`gemma4-26b-a4b-df.recipe` in this repository) names what is quantised; everything it does |
| not name is the checkpoint's bf16: |
| |
| ``` |
| blk.*.ffn_gate_up_exps.*.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 |
| dflash.fc.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 |
| dflash.blk.*.attn_q.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 |
| dflash.blk.*.attn_k.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 |
| dflash.blk.*.attn_v.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 |
| dflash.blk.*.attn_output.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 |
| dflash.blk.*.ffn_gate_up.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 |
| dflash.blk.*.ffn_down.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 |
| dflash.draft_head.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 |
| ``` |
| |
