# RecursiveSI/flint-int4-gguf (flint-int4-q4_1.gguf) with this patch

The Q4_1 file (24.2 GiB, every weight tensor Q4_1; token_embd and the router are F32) is not hosted here; download it from
[RecursiveSI/flint-int4-gguf](https://huggingface.co/RecursiveSI/flint-int4-gguf). The patch adds an AVX512-VNNI `vec_dot` for
Q4_1 x Q8_1 (the x86 path had AVX2 only) and puts Q4_1 into the tiled VNNI GEMM used for prefill.

`llama-bench -t 16 -fa on -lm mlock -b 1024 -ub 1024`, 3 interleaved rounds, median t/s (raw: `q41-ab-interleaved.csv` for the
vec_dot kernel, `q41-tiled-ab-interleaved.csv` for the tiled GEMM),
Xeon Platinum 8559C, same host as the other benchmarks (shared, +-8 % per run).

| | pp256 | pp2048 | tg128 |
|---|---|---|---|
| stock llama.cpp (de5122d) | 182.7 | 188.7 | 48.3 |
| this patch, vec_dot only | 198.8 | 208.5 | 54.0 |
| this patch, tiled GEMM | **266.5** | **297.2** | **55.7** |
| gain vs stock | +46 % | +57 % | +15 % |
| gain vs vec_dot only | +34 % | +42 % | +3 % (noise) |

Decode is unchanged by the tiled path (single-row work stays below the tiled row threshold); the small tg difference is noise.

## Tiled Q4_1 GEMM (prefill)
Q4_1 tiles carry no super-block scale: each 32-wide block of a 256-element slab has its own f32 `d` (scale) and `m` (min).
The tiled kernel gains `sd`/`sm` arrays (per-32-block f32, indexed per tile row) and a `Q41` template flag; activations use the
existing Q8_1 `vec_dot_type`, so no new quantizer was needed. Per 32-block the kernel computes
`d_a * (d_w * dot(q, code) + m_w * sum(code))`: the integer dot and integer code sum are exact and only the f32 accumulation
order differs from `vec_dot`. VNNI uses `vpdpbusd` with the 0..15 weight nibbles as the unsigned side. The driver scales the
k-block byte offset by `256/blck_size` (8 for Q4_1) instead of assuming one block per 256 elements.

## vec_dot kernel (decode)
Two 32-wide blocks per iteration. The two blocks' nibbles are packed into one ymm with memory-source `vinserti128`
(no shuffle-port work), low and high nibbles each go through one `vpdpbusd` against the matching Q8_1 halves, the four fp16
header values of the two blocks are converted at once, and the per-block `d*dy` scale is spread with a single `vpermps`.
A 512-bit version that re-ordered nibbles with cross-lane shuffles was slower than stock (shuffle-port bound) and was dropped.
Prefetch distance `GGML_Q41_PF` (blocks, default 192; 192-320 equal in the cold-MoE microbenchmark).
Microbenchmarks (16 threads, K=2048): cache-resident mul_mat 183 -> 220 GB/s, cold MoE decode (`mul_mat_id`) 43 -> 57 GB/s.

## Accuracy
wikitext-2 test, 32 x 512, tiled Q4_1 vs the vec_dot path of the same patch (logits baseline):

| | KLD | same top-1 |
|---|---|---|
| tiled Q4_1 vs vec_dot (`q41-tiled-kld.log`) | 0.0372 +- 0.0026 | 94.2 % |
| vec_dot patch vs stock | 0.0364 +- 0.0029 | 93.8 % |
| stock with `-fa off` (noise floor) | 0.0385 +- 0.0027 | 93.9 % |

The tiled path is inside the patch's own noise floor. Graph-level `mul_mat` / `mul_mat_id` error versus a float reference is
identical for tiled and vec_dot (`5.29e-3` vs `5.29e-3` for T=1, `5.35e-3` vs `5.39e-3` for T=64; Q8_0 reference 5.35e-3).
Integer dot products are exact; only float accumulation order differs.

## Versus the Q4_K_XL file (same patch, same host)
| file | pp256 | pp2048 | tg128 |
|---|---|---|---|
| flint-UD-Q4_K_XL (25.1 GB), patched | 296 | 307 | 42.8 |
| flint-int4-q4_1 (26.0 GB), patched | 266 | 297 | 55.7 |

Q4_1 decodes about 30 % faster (0.625 B/weight and a simple layout) and now prefills within about 10 % of Q4_K_XL (the tiled
Q4_1 GEMM closed most of the earlier gap). PPL of the two files is similar (18.26 vs 18.36 stock).