File size: 3,827 Bytes
45f1d0b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | # RecursiveSI/flint-nvfp4-gguf (flint-nvfp4.gguf, 22.9 GB) with the CPU patch
NVFP4 weights (485 tensors, 20.7 GiB) plus F32 per-tensor `.scale` tensors and BF16 token_embd. Only the NVFP4 file was
measured here; the Q4_1 and Q4_K_XL numbers in `README.md` / `q41-summary.md` were not re-run.
## What was added to the patch (`patches/flint-cpu-opt.patch`)
1. **AVX512-VNNI `vec_dot` for NVFP4 x Q8_0** (decode, `ggml/src/ggml-cpu/arch/x86/quants.c`). Weight codes are
`|w|` (0..12, unsigned operand of `vpdpbusd`), the sign of `w` is moved onto the int8 activations
(`_mm256_sign_epi8`). The 64-element block is split into 4 subblocks of 16 with per-subblock UE4M3 scales.
2. **Tiled VNNI GEMM for NVFP4** (prefill, `ggml/src/ggml-cpu/tiled/`). Same framework as the Q4_1 path. Weights are
stored as `LUT + 12` (unsigned, 0..24) with one f32 scale per 16-wide subblock (`sd16`). The bias is removed exactly
with the per-16 activation code sums (`bsums`): `sum(w*a) = sum((w+12)*a) - 12*sum(a)`. Activations use Q8_0 with
one scale per 32-block. `tiled_unpk_fp4_64` is an AVX2 unpacker (scalar fallback).
Enabled only in VNNI builds (`__AVX512VNNI__ && __AVX512VL__`).
Integer dot products are exact; only the float accumulation order changes.
## Results
Xeon Platinum 8559C, `taskset -c 0-15`, `-t 16 -fa 1 -b 1024 -ub 1024 -lm mmap+mlock -p 256,2048 -n 128 -r 3`.
Three interleaved rounds per build, median of the per-round means (tokens/s). Raw CSV: `bin_*_r{1,2,3}.csv`,
flat list: `summary.csv`.
| build | pp256 | pp2048 | tg128 |
|---|---|---|---|
| **before**: stock llama.cpp (de5122d87), no patch | 83.7 | 83.6 | 30.6 |
| patch (Q4_1 kernels), NVFP4 path unchanged | 82.3 | 84.4 | 31.8 |
| **after**: patch + NVFP4 VNNI vec_dot + tiled VNNI GEMM | **178.5** | **211.1** | 32.1 |
| gain | pp256 | pp2048 | tg128 |
|---|---|---|---|
| after vs stock | 2.13x | 2.52x | 1.05x (noise) |
| after vs patch-before | 2.17x | 2.50x | 1.01x (noise) |
Per-round values: pp256 after = 153.1 / 178.5 / 181.6, pp2048 after = 206.7 / 218.6 / 211.1, tg128 after = 32.1 / 29.8 / 32.4.
Decode (tg128) did not change measurably. The VNNI vec_dot was about 1 % faster in an earlier single-run check, which is within
noise. Per-op profiling (`GGML_PROF`) showed that prefill `MUL_MAT`/`MUL_MAT_ID` with 256 rows dominated the run time.
## Correctness
- `vec_dot` AVX512 vs generic reference (random blocks, n = 1920..12352): relative error <= 1.3e-6 (float ordering only).
- Dense `mul_mat`, tiled (M=64) vs vec_dot (M=1): relative L2 1.4e-7 .. 2.5e-7 for K in {1024, 2048, 4096, 8192} and N in {8, 16, 24, 100, 256, 512}
(covers ragged row counts and the narrow path). Versus f32 reference: about 3.7e-3 (same order as Q4_1 in the HANDOFF).
- wikitext-2 test, 16 chunks x 512, llama-perplexity:
| build | PPL |
|---|---|
| stock | 24.4165 +- 1.566 |
| patch + NVFP4 opt | 24.4135 +- 1.568 |
## Notes
- The host was shared and heavily loaded during these runs (load average 55..63). Single runs ranged widely
(stock tg128: 14.1 / 32.3 / 30.6), so use the medians. Absolute numbers are lower than the Q4_1 reference figures in
`q41-summary.md`, which were measured on a quieter host. Compare only within this table.
- After the patch, NVFP4 prefill (pp2048 211) is still below Q4_1 (pp2048 297 in `q41-summary.md`). Decode is about 32 t/s versus
55.7 t/s for Q4_1 on the reference host, which is not a fair comparison because the hosts differ.
- Reproduce: `bench_run.sh` runs three copies of llama-bench (stock, patch before, patch after) from the directories
set in `LD_LIBRARY_PATH`/paths in the script. Those copies live under `/tmp/opencode` and are not kept. Rebuild them from
`de5122d87` (stock) and from `build.sh` (patched). `test_nvfp4.c` and `tiled_test.c` are the kernel checks above.
|