keptn/nemotron-3.5-bolt-e4b-a4

NVIDIA Nemotron 3.5 Bolt E4B · 4-bit · for radiance

nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 in one 11.5 GiB file for the radiance inference engine on AMD RDNA4. The feed-forward layers are 4-bit (GPTQ), with NVIDIA's DFlash drafter, the bf16 vision tower and the audio encoder, on one, two or four cards.

• Fast: 301 tok/s BetterBench combined on one card (346 on two cards, 440 on four), 18% more than the int8 file
• Small: the weights take 4.1 GiB of the card
• Long context: 65,536 tokens; one card holds 8 sequences at full length
• Vision: images through the model's bf16 vision tower
• Audio: speech through the model's bf16 audio encoder; LibriSpeech test-clean WER 3.8%
• Thinking: on or off per request; the benchmarks below show both

For the smallest distance to the original, use the int8 file (KL divergence 0.010 against 0.113 here). It is slower on one stream and better on code.

File	nemotron-3.5-bolt-e4b-a4.rad, 11.50 GiB
Needs	radiance main, commit 248ecf5 or newer (4-bit for Gemma 4, PR #214), built from source; one card with 32 GB (tested on Radeon AI PRO R9700) and 8 GiB of free host memory. Two cards: PR #239. Four cards: PR #258. Audio: PR #257
Feed-forward	int4 with a zero point, one f16 scale per 128 weights (GPTQ); layers 19 to 23 int8
Attention	int8, one bf16 scale per 128 weights
Activations	int8 with one scale per token and 128 values; at decode an f16 copy for the int4 layers
lm_head	int8, one bf16 scale per 128 weights; bf16 activations
Speculation (DFlash)	NVIDIA's DFlash drafter, 5 layers, FP8 with one bf16 scale per 128×128 block, 7 draft tokens
Vision tower	the original, bf16, in pinned host memory
Audio encoder	the original, bf16, in pinned host memory
Context	65,536 tokens
Version	1.1 (2026-10-25): adds the audio encoder

Quick start

radiance 1.12.0 does not contain PR #214. It is in main from commit 248ecf5. Until the next release, build the image from main. docker/build.sh tags it radiance:latest:

git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance && docker/build.sh

Then:

hf download keptn/nemotron-3.5-bolt-e4b-a4 nemotron-3.5-bolt-e4b-a4.rad --local-dir ~/models
docker run -d --init --name nemotron-bolt --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e ROCR_VISIBLE_DEVICES=0 \
    radiance:latest --model /models/nemotron-3.5-bolt-e4b-a4.rad --host 0.0.0.0 --port 8000 \
    --kv-cache-dtype bf16 --max-model-len 65536 --max-num-seqs 8 --num-speculative-tokens 7 \
    --mm-max-patches 2520 --vision-tower-placement ram \
    --temp 1.0 --top-p 0.95 --top-k 64 --chat-template-kwargs '{"enable_thinking":false}'

	change
fp8 KV	--kv-cache-dtype fp8: half the KV memory, KL divergence 0.118 instead of 0.113
no drafter	--num-speculative-tokens 0
two cards	--tp 2 --p2p on, ROCR_VISIBLE_DEVICES=0,1; needs radiance with PR #239
four cards	--tp 4 --p2p on, ROCR_VISIBLE_DEVICES=0,1,2,3; needs radiance with PR #258
text only	--mm-max-patches 0 (no images and no audio), drop --vision-tower-placement
thinking	off by default, as in NVIDIA's template; per request "chat_template_kwargs": {"enable_thinking": true}

The drafter runs at 7 draft tokens only: it was trained on blocks of 8. Sampling is NVIDIA's (temperature 1.0, top_p 0.95, both thinking modes) and the checkpoint's top_k 64.

Quality

Distance to the bf16 original. KL divergence on wikitext-2 (test), llama.cpp method: 100 chunks of 512 tokens, the second half of each chunk scored, 25,500 tokens. Reference: the bf16 original in HF transformers, fp32 math.

	mean KL	median KL	99th percentile	top token the same	perplexity (reference: 117.18)
this file, bf16 KV	0.113	0.030	1.23	87.5%	122.74
this file, fp8 KV	0.118	0.032	1.30	87.3%	122.24
the same 4-bit recipe without GPTQ (not published)	0.146			85.8%	129.6
int8 file, bf16 KV	0.0103	0.0023	0.11	96.3%	118.83
bf16 file, bf16 KV	0.0015	0.00023	0.014	98.8%	117.56

On two and four cards (bf16 KV) the mean KL divergence of this file is 0.111 and 0.112.

Benchmarks, greedy. The thinking-off column is the one to compare with other cards.

benchmark	thinking off	thinking on	int8 file, off	int8 file, on
GSM8K (math, 5-shot)	83.0%	90.5%	82.1%	89.5%
IFEval (following instructions, strict)	85.6%	91.9%	86.0%	91.3%
LiveCodeBench v6 (175 tasks, pass@1)	38.3%	52.6%	46.9%	53.7%

• LiveCodeBench v6 by difficulty (off and on): easy 39/43 and 42/43, medium 19/52 and 31/52, hard 9/80 and 19/80. Without thinking, code loses most from the 4-bit weights; with thinking it is close to the int8 file.
• Thinking on uses output limits of 8,192 (GSM8K), 16,384 (IFEval) and 65,536 tokens minus the prompt (LiveCodeBench; the context has 65,536 tokens).
• Thinking off, 108 GSM8K answers stop at the task's limit of 256 tokens, 7 IFEval answers at 1,280 and no LiveCodeBench answer at 8,192. Thinking on, 1 IFEval answer stops at 16,384 tokens and no other answer stops. Answers that stop count as wrong.

Speed

Radeon AI PRO R9700 at 215 W, bf16 KV, 7 draft tokens, thinking off. One card: radiance main 248ecf5. Two cards: radiance main 248ecf5 + PR #239 + PR #238 (both now in main), --p2p on. Four cards: the same + PR #257 + PR #258, --p2p on.

BetterBench 0.6.1, full run: 8 categories, 20 runs each after 3 warm-up runs, temperature 0.7, one stream. Then prefill and parallel streams. Decode tok/s (median):

	1 card	2 cards	4 cards	int8 file, 1 card
combined	300.6	345.9	439.6	254.2
chat	252.6	291.9	333.3	180.0
code	344.7	390.6	489.9	289.7
file_edit	345.8	403.2	505.5	292.6
json	476.8	553.3	692.0	399.7
math	409.9	474.8	606.9	343.7
prose	153.7	177.5	224.9	124.7
reasoning	200.7	216.0	292.7	173.1
summarization	278.9	355.8	460.4	247.1
prefill at 24k tokens	8,486	12,287	12,577	8,469
prefill at 48k tokens	6,784	10,363	11,560	6,782
4 streams, total	591	874	1,122	672
8 streams, total	769	1,160	1,353	938

One stream is faster than with the int8 file, because each token reads fewer bytes of weights. With many streams the int8 file is faster: there the int4 layers spend more time on unpacking than they save on reading.

Two cards split the weights and the KV cache: +15% combined for this file. The bf16 file gains most (+56%), because it reads the most bytes of weights for each token. Four cards give +27% combined over two cards.

Vision

The model's vision tower (bf16, with the clamps the checkpoint stores) reads images in chat requests (image_url).

benchmark (lmms-eval 0.7.3, LMMs-Eval-Lite, greedy, thinking off)	result	int8 file
DocVQA val lite (500 questions)	78.5 (ANLS)	80.2
AI2D lite (500 questions)	75.0%	74.4%

One AI2D answer gives more than the letter and stops at the task's limit of 16 tokens. It counts as wrong.

Audio

The model's audio encoder (bf16) reads speech in chat requests (input_audio, WAV). It needs radiance with PR #257. A clip is cut at 30 s.

LibriSpeech test-clean, 2,620 utterances, greedy, thinking off, prompt "Transcribe this audio clip.", Whisper English text normalizer (as the Open ASR Leaderboard). One card, 8 requests at a time:

file	WER	RTFx
bf16	3.87%	64
int8	3.86%	126
this file	3.81%	117

Two cards (--tp 2 --p2p on), this file: WER 3.81%, RTFx 129. Four cards (--tp 4 --p2p on), the 12 clips below: KL 0.018, 4 of 12 with the same text.

Distance to the original. 12 LibriSpeech clips, compared with the bf16 original in HF transformers (fp32 math). KL divergence over the top 20 tokens, up to the first token that is different:

file	same text	mean KL
bf16	11/12	0.00055
int8	9/12	0.0015
this file	5/12	0.017

Most errors come from the model, not from the encoder: it translates Latin passages, completes quotes from memory and spells names differently. One answer of 2,620 (the same clip in all three files, on one and two cards) repeats spaces after an unknown name and stops at the limit of 160 tokens. It counts as wrong. NVIDIA reports no audio results for this model.

Memory

KV cache, --max-model-len 65536, from radiance's startup report:

	2 sequences	4 sequences	8 sequences
1 card, bf16 KV	131,072 tokens	262,160 tokens	524,304 tokens
1 card, fp8 KV	131,104 tokens	262,160 tokens	524,336 tokens

Tokens of context the KV pool holds in total, shared by all sequences. Every sequence can have the full context. Only 24 of the 42 layers hold KV: the last 18 layers read the cache of layers 22 and 23. With 8 sequences the cache takes 18.2 GiB (bf16 KV) or 9.1 GiB (fp8 KV) of the card. The weights take 4.1 GiB. The token tables (6.5 GiB), the vision tower and the audio encoder (0.9 GiB together) stay in pinned host memory. With two cards each card holds half of the KV cache (9.1 GiB for 8 sequences at bf16 KV). With four cards each card holds 6.6 GiB (8 sequences, bf16 KV).

How it was made

1. Downloaded nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 (revision 49843eed).
2. Converted NVIDIA's drafter from GGUF to safetensors with drafter/gguf_to_safetensors.py, as for the int8 file.
3. Measured the inputs of the int4 layers on a calibration set of 110 chats (165,286 tokens) with the bf16 model, and kept them as GPTQ statistics.
4. Built the container with GPTQ to int4 for the feed-forward of layers 0 to 18 and 24 to 41, then two rounds of coordinate descent. Attention, layers 19 to 23 and the lm_head are int8 (round to nearest). The token tables, the per-layer projections, the vision tower and the audio encoder stay bf16; the drafter goes to block FP8.

rad-convert NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 --recipe nemotron-3.5-bolt-e4b-a4.recipe \
    --draft-model dflash-bf16 --set quantization_config.quant_method=compressed-tensors \
    -o nemotron-3.5-bolt-e4b-a4.rad

GPTQ needs statistics from the calibration set (calib= in the recipe). The recipe is radiance's gemma4-e4b-bf16-a4-df.recipe with GPTQ in place of round to nearest for the int4 layers.

Limits

• radiance runs on AMD RDNA4 cards.
• The file needs radiance main from commit 248ecf5 or newer. radiance 1.12.0 and older do not have its 4-bit format.
• Two cards need PR #239 (in radiance main, not yet in a release). Four cards need PR #258.
• NVIDIA lists text as the only input of this model. The vision tower and the audio encoder are those of the checkpoint.
• Audio needs radiance with PR #257. Older radiance serves text and images from this file and answers audio requests with HTTP 501.
• The context is 65,536 tokens, the length NVIDIA trained.
• The wikitext-2 perplexity of the original is 117. Compare the KL divergence with the bf16 file (0.0015) and the int8 file (0.010), not with other cards.

Credits

• Model and drafter: NVIDIA, post-trained from Google's Gemma 4 E4B (Apache 2.0).
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance).
• Test sets: GSM8K, IFEval, LiveCodeBench, LMMs-Eval-Lite, LibriSpeech, wikitext-2.

License

OpenMDW License Agreement 1.1, the license of the base model. See LICENSE. The Gemma 4 base model is under the Apache License 2.0, see LICENSE.Apache-2.0. NOTICE names the sources and NVIDIA's terms.