| --- |
| base_model: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF |
| base_model_relation: quantized |
| library_name: gguf |
| tags: [gguf, llama.cpp, cpu, avx512-vnni, nemotron, mamba2, moe, vlm, multimodal] |
| license: other |
| license_name: openmdw-1.1 |
| license_link: https://openmdw.ai/license/1-1/ |
| --- |
| |
| # Nemotron 3.5 Bolt 74B-A9B (24 GB IQ2 GGUF) — CPU-optimized build |
|
|
| One model, one quant, one machine: `NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf` (unmodified file from |
| `nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF`) plus `mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf` |
| (unmodified vision projector) and a llama.cpp patch (base commit `de5122d`) tuned for an Intel Xeon Platinum 8559C. |
| The GGUF files are not changed; all changes are code. Both files are redistributed under the NVIDIA OpenMDW-1.1 license. |
|
|
| ## What is in this repo |
|
|
| | file | purpose | |
| |---|---| |
| | `NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf` | model (unchanged, 20.5 GiB) | |
| | `mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf` | vision encoder projector (unchanged) | |
| | `patches/nemotron-nvfp4-cpu-opt.patch`, `patches/BASE_COMMIT` | llama.cpp patch and base commit | |
| | `patches/optional-iq2xxs-vnni.patch` | experimental IQ2_XXS kernel, no end-to-end gain (apply after the main patch) | |
| | `patches/0002-nvfp4-f16-prefill.patch` | fp16 prefill path for NVFP4, **accepted**: pp512 40.0 -> 57.0 t/s (apply after the main patch) | |
| | `build.sh` | clone, patch, and build llama.cpp | |
| | `benchmarks/` | raw A/B throughput, KL-divergence log, multimodal check | |
| |
| ## What the patch changes |
| |
| The Nemotron GGUF is a Mamba-2 / Transformer / LatentMoE hybrid. Its size comes from three weight types: |
| IQ2_XXS (routed and shared experts, 16.3 GiB), **NVFP4** (all Mamba-2 `ssm_in` / `ssm_out` projections, 2.3 GiB), and Q8_0 |
| (attention, latent projections, output). Profiling the stock build on this CPU showed the NVFP4 projections as |
| **39% of decode op time and 67% of prefill op time**. Stock llama.cpp has only an AVX2 `vec_dot` for NVFP4 and no |
| batched kernel, so this patch adds: |
|
|
| 1. **AVX512-VNNI `vec_dot` for NVFP4 x Q8_0** (`ggml/src/ggml-cpu/arch/x86/quants.c`). One 64-weight block per |
| iteration. The E2M1 magnitude comes from a 16-byte shuffle, the sign from nibble bit 3 (applied to the q8 side with a |
| mask so `dpbusd` takes unsigned magnitudes). UE4M3 scales are decoded in SIMD with a bit move plus a multiply by 2^119 |
| (`0x7F` maps to 0, as in the reference). The 16 int32 lanes are scaled by one per-lane float vector, so no horizontal |
| reduction is needed inside the loop. |
| 2. **2 x 2 kernel (`nrc == 2`)** for prefill: two weight rows and two activation columns per call, so each weight block is |
| decoded once and used for both columns. Enabled with `.nrows = 2` for NVFP4 in the CPU type traits. Non-AVX512 builds |
| fall back to four `nrc == 1` calls, so the patch is safe on other CPUs. |
|
|
| Unit tests (random data, double-precision reference): relative error about 5e-7 for both kernels, and for the 2 x 2 path. |
|
|
| ## Results (16 cores: `taskset -c 0-15`, `-t 16 -fa on -lm mlock -b 1024 -ub 1024`, llama-bench) |
|
|
| Interleaved runs on a shared host (see `benchmarks/ab_results.csv`; 12-thread repeat in `ab_12threads.csv`: pp256 25.2 -> 36.3 t/s, tg64 9.51 -> 9.53 t/s): |
|
|
| | build | pp256 t/s (median, n=5) | tg64 t/s (mean, n=3) | |
| |---|---|---| |
| | stock llama.cpp `de5122d` | 30.9 | 10.2 | |
| | this patch | **45.1** | 9.1 (5.8 / 9.9 / 11.6) | |
| | speedup | **~1.46x** | no reliable decode gain measured | |
|
|
| - Prefill: consistent across all five rounds (stock 30.6–33.2, patch 43.0–45.4 t/s). |
| - Decode: the noise across rounds (5.8 to 11.7 t/s) is larger than any effect, so this patch does not claim a decode speedup. |
| The single-thread NVFP4 gemv is about 25% faster in a microbenchmark (4.3 vs 5.6 ns per 64-weight block), but the |
| end-to-end decode gain did not show up on this host because decode is memory-bandwidth bound (next section). |
|
|
| ## Experiment E1 — fp16 prefill path for NVFP4 (2026-10-29, accepted) |
|
|
| The 2x2 kernel still re-decodes each 4-bit weight block per pair of activation columns, which is arithmetic-bound |
| at large batches. `patches/0002-nvfp4-f16-prefill.patch` dequantizes every NVFP4 row once into an exact fp16 |
| shadow, converts the batch activations to fp16 in a scratch buffer, and runs a normal fp16 x fp16 -> fp32 GEMM |
| through llamafile `sgemm`. Decode (N = 1) keeps the packed 4-bit weights and is unchanged. |
|
|
| Interleaved A/B (`-t 12 -fa 1 -lm mlock`, 3 rounds x 2 reps, median; `benchmarks/exp_e1_f16_prefill.csv`): |
|
|
| | test | main patch | + E1 | speedup | |
| |---|---|---|---| |
| | pp64 | 31.9 | 40.4 | 1.27x | |
| | pp128 | 34.9 | 46.2 | 1.32x | |
| | pp256 | 37.9 | 52.9 | 1.39x | |
| | pp512 | 40.0 | 57.0 | 1.42x | |
| | tg64 | 10.3 | 10.9 | unchanged (noise) | |
|
|
| PPL 8.6881 +/- 0.66211 vs 8.7051 +/- 0.66329 (fp16 activation rounding; still more precise than the Q8_0 path). |
| Cost: ~8.8 GB extra RSS, allocated lazily on the first prefill of each NVFP4 tensor, so decode-only sessions do not |
| pay it. Details: `benchmarks/exp_e1_f16_prefill.md`. |
|
|
| ## Why decode does not speed up, and what does help (measured on this machine) |
|
|
| - **Decode is memory-bandwidth bound.** The NVFP4 `ssm_in` matvec moves about 43 MB in 0.53 ms (~80 GB/s) against a measured |
| ceiling of 120-150 GB/s, so a faster kernel cannot raise tok/s much. An AVX512-VNNI IQ2_XXS kernel (gather grid + sign bytes |
| rebuilt with `pdep` and parity, `patches/optional-iq2xxs-vnni.patch`) is correct (rel. error 5e-7) and ~10% faster on one |
| thread (4.4 vs 4.9 ns per 64 weights), but made no end-to-end difference (tg64 median 9.53 vs 9.51 t/s, 12 threads), so it |
| is **not** in the main patch. |
| - **Thread count under the CPU quota** (`benchmarks/thread_sweep.csv`, tg64): 16 threads got CFS-throttled and ranged 7.0-10.9 t/s; |
| 12 threads were never throttled and gave 9.6-10.3 t/s. Use `-t 12` for steadier decode; prefill is faster at 16 threads |
| (45 t/s vs 36 t/s at 12) when the quota allows. |
| - **Speculative decoding** (`benchmarks/spec_decoding.jsonl`, greedy, 200 tokens, `-t 12`; baseline 10.9 t/s): |
|
|
| | mode | code | prose | repeated text | |
| |---|---|---|---| |
| | none | 10.9 | 10.7-11.0 | 10.9 | |
| | `--spec-type draft-dflash` (LiLiCorr drafter, n-max 6) | 11.6 (78% accepted) | **5.5** (28% accepted) | 13.4 | |
| | `--spec-type ngram-mod` (no draft model) | 10.7 cold / 13.5 warm | 10.3 / 9.0 | 10.1 cold / 22.2 warm | |
|
|
| DFlash is not recommended on CPU: verifying 7 tokens through the MoE reads many more experts, so low-acceptance text gets |
| much slower. `ngram-mod` costs little on novel text and helps when output repeats earlier text (editing files, echoing tool |
| output). "Warm" means the same text had already been generated once in the session, so it is a best case. |
|
|
| - **Runtime knobs: nothing to gain** (`benchmarks/knobs_decode.csv`, tg64, 12 threads, 3 interleaved rounds, medians): |
| default `--poll 50` 10.37, `--poll 0` 10.16, `--poll 100` 10.39, `--cpu-strict 1` 9.96, `-lm none` 10.20 t/s. All differences are |
| inside the run-to-run noise (single outliers of ~6 t/s appeared once for `--cpu-strict 1` and `-lm none`). `--prio 2` did not run |
| in this container. Transparent hugepages are `madvise` system-wide and were not changed. |
| - **`-ub` for prefill** (`benchmarks/ubatch_prefill.csv`, pp512, 12 threads, 2 rounds): 128 -> 33-35 t/s, 256 -> 37, 512 -> 40, |
| 1024 -> 39-41 t/s. The default 512 is already the best choice. |
|
|
| - **Smaller `output.weight`** (`benchmarks/lm_head_quant.txt`): the Q8_0 lm_head (570 MB) is ~4.4% of decode op time. A standalone 12-thread |
| gemv takes 6.4-6.8 ms (Q8_0), 4.1-4.9 ms (Q6_K), 2.4 ms (Q4_K). That caps the decode gain at roughly 1-3%, below the measurement noise, and |
| costs logit accuracy, so no requantized GGUF was made. |
| |
| Suggested server flags on a quota-limited shared host: `-t 12 -fa on -lm mlock --spec-type ngram-mod`. |
| |
| ## Accuracy |
| |
| Against stock logits, wikitext-2 test split, 16 chunks x 512 tokens (`benchmarks/kld.log`): |
| |
| | comparison | mean KLD | same top-1 | |
| |---|---|---| |
| | this patch vs stock (`-fa on`) | 0.00270 ± 0.00009 | 97.75% | |
| | stock `-fa off` vs stock `-fa on` (noise floor) | 0.00257 ± 0.00010 | 97.40% | |
| |
| The patch's deviation is the same size as the stock flash-attention on/off difference. Stock PPL on this text is |
| 6.604 ± 0.264. Results are not bit-exact (float accumulation order differs). Short factual prompts were answered correctly (see the multimodal check); an explicit greedy-output comparison against stock was not run. |
| |
| ## Multimodal check |
| |
| `llama-server --mmproj ...` with the patched build correctly names all six icons in a test image, and text answers are |
| correct (`17*23 -> 391`, capital of France -> Paris). See `benchmarks/multimodal_check.md`. The image check is coarse: |
| colors are only answered as "teal". |
|
|
| ## Build and run |
|
|
| ``` |
| ./build.sh |
| taskset -c 0-15 llama.cpp/build/bin/llama-server -m NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \ |
| --mmproj mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf -t 16 -fa on -lm mlock --jinja -c 8192 --port 8080 |
| ``` |
|
|
| Needs about 22 GB of lockable memory for the model plus the projector. |
|
|
| ## Limitations / next steps |
|
|
| - Validated only on Xeon 8559C with 16 cores. |
| - IQ2_XXS expert matmul stays on the upstream AVX2 kernel in the main patch (see the optional patch above). |
| - Decode gain is not established (see above). Prefill is the measured win (main patch + E1: pp512 ~1.4x over main patch). |
| - E1 costs ~8.8 GB of RAM for the fp16 weight shadow. |
| |
