# flint-UD-Q4_K_XL CPU optimization (Xeon Platinum 8559C, 16 threads)

`llama-bench -t 16 -fa on -lm mlock -b 1024 -ub 1024`, 3 interleaved rounds (stock / Qwen3.6 patch / this patch per round), median t/s.
Raw: `ab-interleaved-flint.csv`.

| variant | pp256 | pp2048 | tg128 |
|---|---|---|---|
| stock llama.cpp (de5122d) | 231.0 | 274.2 | 38.2 |
| Qwen3.6 CPU-OPT patch, unchanged | 264.4 | 307.8 | 42.1 |
| **this patch** | **295.8** | **306.7** | **42.8** |

Versus stock: pp256 1.28x, pp2048 1.12x, tg128 1.12x. Versus the Qwen patch alone, only pp256 (+12%) is clearly better;
decode (+2%) and pp2048 are within noise. Most of the gain comes from the Qwen3.6 patch's kernels also applying to flint.

## What is new (flint-specific)
flint-UD-Q4_K_XL is Q5_K 11.6 GiB + IQ4_XS 9.0 GiB + Q8_0 2.0 GiB (+ small Q4_K/Q6_K/IQ3_S).
1. IQ4_XS x Q8_K `vec_dot`: AVX512-VNNI (the x86 path had only AVX2). Unsigned LUT (+128) with `vpdpbusd`, bias removed
   through `bsums`, L2 prefetch (`GGML_IQ4XS_PF`, default 40). Decode MUL_MAT_ID iq4_xs reaches ~97 GB/s.
2. Q5_K x Q8_K `vec_dot`: 5th bit merged with `vptestmb` + masked add instead of shift/and/or.
   Bit-identical results to the previous VNNI version (KLD unchanged on Qwen3.6 as well).
Other contents are the Qwen3.6 patch (Q4_K/Q5_K/Q6_K VNNI, AMX tweaks, tiled MoE group 32). Its Gated-DeltaNet changes
do not apply to flint.

## Accuracy (wikitext-2 test, 32 x 512, vs stock logits)
| | KLD | same top-1 | PPL |
|---|---|---|---|
| stock vs stock (-t 16 vs -t 8) | 0.0000 | 100 % | 18.363 |
| Qwen3.6 patch unchanged | 0.0664 +- 0.0049 | 92.2 % | - |
| this patch | 0.0666 +- 0.0049 | ~92 % | **18.277** +- 0.787 |

PPL is not worse than stock (18.363 +- 0.795).

KLD is much larger than on Qwen3.6 (0.003-0.006), but **flint is numerically chaotic**: stock llama.cpp compared with itself
under a different numeric path gives the same level of KLD (control experiments, same baseline logits):

| stock, changed numeric path | KLD | same top-1 |
|---|---|---|
| `-fa off` | 0.0762 +- 0.0055 | 91.7 % |
| `-ub 128 -b 128` | 0.0700 +- 0.0051 | 92.0 % |
| this patch | 0.0666 +- 0.0049 | ~92 % |

So 0.067 is inside the noise floor of stock itself and does not indicate a defect in the kernels. Supporting evidence:
a graph-level test (`mul_mat` and `mul_mat_id`, K=2048, N=512, random weights) gives the same relative L2 error versus a float
reference for stock and this patch to 4 digits (Q4_K/Q5_K/Q6_K/IQ4_XS ~7e-3, Q8_0 ~5.4e-3, dominated by Q8_K activation quantization).
Bisect (`kld-bisect.log`, KLD with one VNNI kernel disabled): Q4_K 0.0670, Q5_K 0.0524, Q6_K 0.0685, IQ4_XS 0.0664;
removing all of `quants.c` gives 0.0162. Why tiny differences get amplified (e.g. top-6-of-256 expert routing ties) was not verified.
The earlier flint_cpu_opt README reported KLD 0.2058, which was the same effect measured on a different baseline.

## Profile (decode, 64 tokens)
MUL_MAT q8_0 41 % (attention, shared expert, lm_head; close to bandwidth bound), MUL_MAT_ID q5_K 22 %,
MUL_MAT q5_K 8 %, MUL_MAT_ID iq4_xs 8 %. Prefill: FLASH_ATTN_EXT 24 %, MUL_MAT_ID q5_K 21 %, q8_0 17 %, iq4_xs 15 %.
Prefetch distance, thread count and `--poll` sweeps showed no gain beyond noise.
Not done: flash-attention kernel work, Q8_0 decode kernel.