Gemma 4 31B-it 路 EXL3 路 for radiance
google/gemma-4-31B-it in four EXL3-V3 files for the radiance inference engine.
Experimental. These files need two radiance pull requests that are not merged yet: #401 (EXL3-V3, by Hero422) and #537 (the Gemma 4 plugin on top of it). Build radiance from the branch of #537. Speed numbers and a quick start follow.
Files
| file | size | bits per weight |
|---|---|---|
gemma4-31b-exl3-3.25bpw.rad |
16.77 GiB | 3.25 (mix) |
gemma4-31b-exl3-4.07bpw.rad |
19.57 GiB | 4.07 (mix) |
gemma4-31b-exl3-4.89bpw.rad |
22.36 GiB | 4.89 (mix) |
gemma4-31b-exl3-6.0bpw.rad |
26.15 GiB | 6.00 |
Bits per weight is the average over the linear layers of the transformer blocks. In a mix, each of the 350 linear layers gets its own width, from 2.75 to 6.0 bits. A model fitted to measured KL chooses the widths. Every file also holds the int8 output head, the bf16 token embedding, the vision and audio towers and the MTP draft head.
Quality
Distance to the bf16 original on the 80 MT-Bench questions, both turns (42,283 positions). The reference is the bf16 model in Hugging Face transformers. Lower KL is better.
| file | size | mean KL | top token the same |
|---|---|---|---|
| EXL3 3.25 bpw mix | 16.77 GiB | 0.0181 | 96.7% |
| EXL3 4.07 bpw mix | 19.57 GiB | 0.00598 | 98.0% |
| EXL3 4.89 bpw mix | 22.36 GiB | 0.00314 | 98.6% |
| EXL3 6.0 bpw | 26.15 GiB | 0.00226 | 98.7% |
| keptn/gemma4-31b-a4 | 19.95 GiB | 0.0246 | 96.1% |
| keptn/gemma4-31b-w5 | 23.36 GiB | 0.00792 | 97.8% |
| keptn/gemma4-31b-i8g | 33.37 GiB | 0.00236 | 98.8% |
Download
Download one file and config.json:
hf download keptn/gemma4-31b-exl3 gemma4-31b-exl3-4.07bpw.rad config.json --local-dir ~/models
- Downloads last month
- -