Gemma 4 31B-it 路 EXL3 路 for radiance

google/gemma-4-31B-it in four EXL3-V3 files for the radiance inference engine.

Experimental. These files need two radiance pull requests that are not merged yet: #401 (EXL3-V3, by Hero422) and #537 (the Gemma 4 plugin on top of it). Build radiance from the branch of #537. Speed numbers and a quick start follow.

Files

file size bits per weight
gemma4-31b-exl3-3.25bpw.rad 16.77 GiB 3.25 (mix)
gemma4-31b-exl3-4.07bpw.rad 19.57 GiB 4.07 (mix)
gemma4-31b-exl3-4.89bpw.rad 22.36 GiB 4.89 (mix)
gemma4-31b-exl3-6.0bpw.rad 26.15 GiB 6.00

Bits per weight is the average over the linear layers of the transformer blocks. In a mix, each of the 350 linear layers gets its own width, from 2.75 to 6.0 bits. A model fitted to measured KL chooses the widths. Every file also holds the int8 output head, the bf16 token embedding, the vision and audio towers and the MTP draft head.

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns (42,283 positions). The reference is the bf16 model in Hugging Face transformers. Lower KL is better.

file size mean KL top token the same
EXL3 3.25 bpw mix 16.77 GiB 0.0181 96.7%
EXL3 4.07 bpw mix 19.57 GiB 0.00598 98.0%
EXL3 4.89 bpw mix 22.36 GiB 0.00314 98.6%
EXL3 6.0 bpw 26.15 GiB 0.00226 98.7%
keptn/gemma4-31b-a4 19.95 GiB 0.0246 96.1%
keptn/gemma4-31b-w5 23.36 GiB 0.00792 97.8%
keptn/gemma4-31b-i8g 33.37 GiB 0.00236 98.8%

Download

Download one file and config.json:

hf download keptn/gemma4-31b-exl3 gemma4-31b-exl3-4.07bpw.rad config.json --local-dir ~/models
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for keptn/gemma4-31b-exl3

Finetuned
(389)
this model