| # RecursiveSI/flint-int4-gguf (flint-int4-q4_1.gguf) with this patch |
| |
| The Q4_1 file (24.2 GiB, every weight tensor Q4_1; token_embd and the router are F32) is not hosted here; download it from |
| [RecursiveSI/flint-int4-gguf](https://huggingface.co/RecursiveSI/flint-int4-gguf). The patch adds an AVX512-VNNI `vec_dot` for |
| Q4_1 x Q8_1 (the x86 path had AVX2 only) and puts Q4_1 into the tiled VNNI GEMM used for prefill. |
| |
| `llama-bench -t 16 -fa on -lm mlock -b 1024 -ub 1024`, 3 interleaved rounds, median t/s (raw: `q41-ab-interleaved.csv` for the |
| vec_dot kernel, `q41-tiled-ab-interleaved.csv` for the tiled GEMM), |
| Xeon Platinum 8559C, same host as the other benchmarks (shared, +-8 % per run). |
|
|
| | | pp256 | pp2048 | tg128 | |
| |---|---|---|---| |
| | stock llama.cpp (de5122d) | 182.7 | 188.7 | 48.3 | |
| | this patch, vec_dot only | 198.8 | 208.5 | 54.0 | |
| | this patch, tiled GEMM | **266.5** | **297.2** | **55.7** | |
| | gain vs stock | +46 % | +57 % | +15 % | |
| | gain vs vec_dot only | +34 % | +42 % | +3 % (noise) | |
|
|
| Decode is unchanged by the tiled path (single-row work stays below the tiled row threshold); the small tg difference is noise. |
|
|
| ## Tiled Q4_1 GEMM (prefill) |
| Q4_1 tiles carry no super-block scale: each 32-wide block of a 256-element slab has its own f32 `d` (scale) and `m` (min). |
| The tiled kernel gains `sd`/`sm` arrays (per-32-block f32, indexed per tile row) and a `Q41` template flag; activations use the |
| existing Q8_1 `vec_dot_type`, so no new quantizer was needed. Per 32-block the kernel computes |
| `d_a * (d_w * dot(q, code) + m_w * sum(code))`: the integer dot and integer code sum are exact and only the f32 accumulation |
| order differs from `vec_dot`. VNNI uses `vpdpbusd` with the 0..15 weight nibbles as the unsigned side. The driver scales the |
| k-block byte offset by `256/blck_size` (8 for Q4_1) instead of assuming one block per 256 elements. |
| |
| ## vec_dot kernel (decode) |
| Two 32-wide blocks per iteration. The two blocks' nibbles are packed into one ymm with memory-source `vinserti128` |
| (no shuffle-port work), low and high nibbles each go through one `vpdpbusd` against the matching Q8_1 halves, the four fp16 |
| header values of the two blocks are converted at once, and the per-block `d*dy` scale is spread with a single `vpermps`. |
| A 512-bit version that re-ordered nibbles with cross-lane shuffles was slower than stock (shuffle-port bound) and was dropped. |
| Prefetch distance `GGML_Q41_PF` (blocks, default 192; 192-320 equal in the cold-MoE microbenchmark). |
| Microbenchmarks (16 threads, K=2048): cache-resident mul_mat 183 -> 220 GB/s, cold MoE decode (`mul_mat_id`) 43 -> 57 GB/s. |
|
|
| ## Accuracy |
| wikitext-2 test, 32 x 512, tiled Q4_1 vs the vec_dot path of the same patch (logits baseline): |
|
|
| | | KLD | same top-1 | |
| |---|---|---| |
| | tiled Q4_1 vs vec_dot (`q41-tiled-kld.log`) | 0.0372 +- 0.0026 | 94.2 % | |
| | vec_dot patch vs stock | 0.0364 +- 0.0029 | 93.8 % | |
| | stock with `-fa off` (noise floor) | 0.0385 +- 0.0027 | 93.9 % | |
| |
| The tiled path is inside the patch's own noise floor. Graph-level `mul_mat` / `mul_mat_id` error versus a float reference is |
| identical for tiled and vec_dot (`5.29e-3` vs `5.29e-3` for T=1, `5.35e-3` vs `5.39e-3` for T=64; Q8_0 reference 5.35e-3). |
| Integer dot products are exact; only float accumulation order differs. |
|
|
| ## Versus the Q4_K_XL file (same patch, same host) |
| | file | pp256 | pp2048 | tg128 | |
| |---|---|---|---| |
| | flint-UD-Q4_K_XL (25.1 GB), patched | 296 | 307 | 42.8 | |
| | flint-int4-q4_1 (26.0 GB), patched | 266 | 297 | 55.7 | |
| |
| Q4_1 decodes about 30 % faster (0.625 B/weight and a simple layout) and now prefills within about 10 % of Q4_K_XL (the tiled |
| Q4_1 GEMM closed most of the earlier gap). PPL of the two files is similar (18.26 vs 18.36 stock). |
| |
