Add AVX512-VNNI Q4_1 vec_dot; benchmarks for RecursiveSI/flint-int4-gguf

README.md

@@ -13,3 +13,7 @@ Measured on Xeon Platinum 8559C, 16 threads (median of 3 interleaved rounds, vs
 pp256 231 -> 296 t/s (1.28x), pp2048 274 -> 307 (1.12x), tg128 38.2 -> 42.8 (1.12x).
 PPL 18.28 vs 18.36 stock (no degradation). KLD vs stock is 0.067 (top-1 ~92 %), which is the same as stock vs stock with a different numeric path (`-fa off` 0.076, `-ub 128` 0.070): flint is very sensitive to tiny numeric differences.
 Details, caveats and the KLD bisect: `benchmarks/summary.md`. Validated only on this CPU.
+
+## Also: RecursiveSI/flint-int4-gguf (Q4_1)
+The same patch adds an AVX512-VNNI Q4_1 kernel: on `flint-int4-q4_1.gguf` pp256 +9 %, pp2048 +10 %, tg128 +12 % versus stock
+(54 t/s decode, faster than the Q4_K_XL file's 43 t/s, but prefill is 209 vs 307 t/s). See `benchmarks/q41-summary.md`.

benchmarks/q41-ab-interleaved.csv

@@ -0,0 +1,18 @@
+stock,1,0,170.605076
+stock,1,0,187.385593
+stock,1,0,48.343508
+opt,1,0,186.811295
+opt,1,0,205.963108
+opt,1,0,53.775119
+stock,2,0,182.723342
+stock,2,0,194.848386
+stock,2,0,49.456506
+opt,2,0,198.750075
+opt,2,0,208.458004
+opt,2,0,53.959147
+stock,3,0,185.988560
+stock,3,0,188.651516
+stock,3,0,44.801554
+opt,3,0,215.583520
+opt,3,0,209.538459
+opt,3,0,54.916187

benchmarks/q41-kld.log

@@ -0,0 +1,11 @@
+== stock base
+1.42.826.729 I Final estimate: PPL = 18.2603 +/- 0.77991
+== opt vs stock
+chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
+Mean    KLD:   0.036404 ±   0.002873
+Same top p: 93.775 ± 0.267 %
+== stock -fa off vs stock (noise floor)
+chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
+Mean    KLD:   0.038469 ±   0.002729
+Same top p: 93.897 ± 0.265 %
+DONE

benchmarks/q41-summary.md

@@ -0,0 +1,42 @@
+# RecursiveSI/flint-int4-gguf (flint-int4-q4_1.gguf) with this patch
+
+The Q4_1 file (24.2 GiB, every weight tensor Q4_1; token_embd and the router are F32) is not hosted here; download it from
+[RecursiveSI/flint-int4-gguf](https://huggingface.co/RecursiveSI/flint-int4-gguf). The patch adds an AVX512-VNNI `vec_dot` for
+Q4_1 x Q8_1 (the x86 path had AVX2 only).
+
+`llama-bench -t 16 -fa on -lm mlock -b 1024 -ub 1024`, 3 interleaved rounds, median t/s (raw: `q41-ab-interleaved.csv`),
+Xeon Platinum 8559C, same host as the other benchmarks (shared, +-8 % per run).
+
+| | pp256 | pp2048 | tg128 |
+|---|---|---|---|
+| stock llama.cpp (de5122d) | 182.7 | 188.7 | 48.3 |
+| this patch | **198.8** | **208.5** | **54.0** |
+| gain | +9 % | +10 % | +12 % |
+
+## Kernel
+Two 32-wide blocks per iteration. The two blocks' nibbles are packed into one ymm with memory-source `vinserti128`
+(no shuffle-port work), low and high nibbles each go through one `vpdpbusd` against the matching Q8_1 halves, the four fp16
+header values of the two blocks are converted at once, and the per-block `d*dy` scale is spread with a single `vpermps`.
+A 512-bit version that re-ordered nibbles with cross-lane shuffles was slower than stock (shuffle-port bound) and was dropped.
+Prefetch distance `GGML_Q41_PF` (blocks, default 192; 192-320 equal in the cold-MoE microbenchmark).
+Microbenchmarks (16 threads, K=2048): cache-resident mul_mat 183 -> 220 GB/s, cold MoE decode (`mul_mat_id`) 43 -> 57 GB/s.
+
+## Accuracy (wikitext-2 test, 32 x 512, vs stock logits)
+| | KLD | same top-1 |
+|---|---|---|
+| this patch | 0.0364 +- 0.0029 | 93.8 % |
+| stock with `-fa off` (noise floor) | 0.0385 +- 0.0027 | 93.9 % |
+
+PPL 18.240 (patch) vs 18.260 (stock). Graph-level `mul_mat` / `mul_mat_id` error versus a float reference is identical to stock
+(5.3e-3 .. 5.4e-3). Integer dot products are exact; only float accumulation order differs.
+
+## Versus the Q4_K_XL file (same patch, same host)
+| file | pp256 | pp2048 | tg128 |
+|---|---|---|---|
+| flint-UD-Q4_K_XL (25.1 GB), patched | 296 | 307 | 42.8 |
+| flint-int4-q4_1 (26.0 GB), patched | 199 | 209 | 54.0 |
+
+Q4_1 decodes about 26 % faster (0.625 B/weight and a simple layout) but prefills about 32 % slower: K-quant weights go through the
+tiled VNNI GEMM, while Q4_1 still uses the per-row `vec_dot` path (60 % of prefill time is `MUL_MAT_ID q4_1`). Supporting Q4_1 in
+the tiled path would need per-32-block float scales in the tiled kernel (its tiles carry integer sub-block scales under one
+super-block scale) and was not done. PPL of the two files is similar (18.26 vs 18.36 stock).

patches/flint-cpu-opt.patch

@@ -172,10 +172,69 @@ index 9f3a744b5..d23e4d971 100644
  
          parallel_for_ggml(params, n_batch * NB, [&](int begin, int end) {
 diff --git a/ggml/src/ggml-cpu/arch/x86/quants.c b/ggml/src/ggml-cpu/arch/x86/quants.c
-index 21cd6312a..9b65eebb2 100644
+index 21cd6312a..07d4feb37 100644
 --- a/ggml/src/ggml-cpu/arch/x86/quants.c
 +++ b/ggml/src/ggml-cpu/arch/x86/quants.c
-@@ -2118,7 +2118,89 @@ void ggml_vec_dot_q4_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const voi
+@@ -941,6 +941,49 @@ void ggml_vec_dot_q4_1_q8_1(int n, float * GGML_RESTRICT s, size_t bs, const voi
+     __m256 acc = _mm256_setzero_ps();
+ 
+     float summs = 0;
++    float vtot  = 0.0f;
++
++#if defined(__AVX512F__) && defined(__AVX512BW__) && defined(__AVX512VL__) && defined(__AVX512DQ__) && defined(__AVX512VNNI__) && defined(__F16C__)
++    // AVX512-VNNI path: two 32-wide blocks per iteration. x codes are 0..15 (unsigned), y codes are int8, so
++    // vpdpbusd gives the exact int32 sums; per-block d*dy is applied per 8-lane half, m*sy is accumulated separately.
++    {
++        static int q41pf = -1;
++        if (q41pf < 0) { const char * e = getenv("GGML_Q41_PF"); q41pf = e ? atoi(e) : 192; }
++        const __m256i m4 = _mm256_set1_epi8(0x0F);
++        __m256 acc0 = _mm256_setzero_ps();
++        __m256 acc1 = _mm256_setzero_ps();
++        __m128 sm = _mm_setzero_ps();
++        const __m256i sidx = _mm256_setr_epi32(0,0,0,0, 2,2,2,2);
++        for (; ib + 3 < nb; ib += 4) {
++            _mm_prefetch((const char *)(x + ib + q41pf), _MM_HINT_T1);
++            _mm_prefetch((const char *)(x + ib + q41pf) + 64, _MM_HINT_T1);
++#define Q41_PAIR(ACC, I) { \
++            const block_q4_1 * xa = x + (I); const block_q4_1 * xb = x + (I) + 1; \
++            const block_q8_1 * ya = y + (I); const block_q8_1 * yb = y + (I) + 1; \
++            /* [block a | block b] in one ymm, built with memory-source vinserti128 (no shuffle-port work) */ \
++            const __m256i v   = _mm256_inserti128_si256(_mm256_castsi128_si256(_mm_loadu_si128((const __m128i *) xa->qs)), \
++                                                        _mm_loadu_si128((const __m128i *) xb->qs), 1); \
++            const __m256i lo  = _mm256_and_si256(v, m4); \
++            const __m256i hi  = _mm256_and_si256(_mm256_srli_epi16(v, 4), m4); \
++            const __m256i ylo = _mm256_inserti128_si256(_mm256_castsi128_si256(_mm_loadu_si128((const __m128i *) ya->qs)), \
++                                                        _mm_loadu_si128((const __m128i *) yb->qs), 1); \
++            const __m256i yhi = _mm256_inserti128_si256(_mm256_castsi128_si256(_mm_loadu_si128((const __m128i *) (ya->qs + 16))), \
++                                                        _mm_loadu_si128((const __m128i *) (yb->qs + 16)), 1); \
++            const __m256i dot = _mm256_dpbusd_epi32(_mm256_dpbusd_epi32(_mm256_setzero_si256(), lo, ylo), hi, yhi); \
++            /* headers: x = (d, m), y = (d, s) -> pp = (d0*dy0, m0*s0, d1*dy1, m1*s1) */ \
++            const __m128 hx = _mm_cvtph_ps(_mm_unpacklo_epi32(_mm_cvtsi32_si128(*(const int *) xa), _mm_cvtsi32_si128(*(const int *) xb))); \
++            const __m128 hy = _mm_cvtph_ps(_mm_unpacklo_epi32(_mm_cvtsi32_si128(*(const int *) ya), _mm_cvtsi32_si128(*(const int *) yb))); \
++            const __m128 pp = _mm_mul_ps(hx, hy); \
++            sm = _mm_add_ps(sm, pp); /* lanes 1 and 3 carry m*s */ \
++            const __m256 sc = _mm256_permutevar8x32_ps(_mm256_castps128_ps256(pp), sidx); \
++            ACC = _mm256_fmadd_ps(sc, _mm256_cvtepi32_ps(dot), ACC); }
++            Q41_PAIR(acc0, ib)
++            Q41_PAIR(acc1, ib + 2)
++#undef Q41_PAIR
++        }
++        vtot = hsum_float_8(_mm256_add_ps(acc0, acc1)) + _mm_cvtss_f32(_mm_shuffle_ps(sm, sm, 0x01)) + _mm_cvtss_f32(_mm_shuffle_ps(sm, sm, 0x03));
++    }
++#endif
+ 
+     // Main loop
+     for (; ib < nb; ++ib) {
+@@ -969,7 +1012,7 @@ void ggml_vec_dot_q4_1_q8_1(int n, float * GGML_RESTRICT s, size_t bs, const voi
+ #endif
+     }
+ 
+-    *s = hsum_float_8(acc) + summs;
++    *s = hsum_float_8(acc) + summs + vtot;
+ #else
+     UNUSED(nb);
+     UNUSED(x);
+@@ -2118,7 +2161,89 @@ void ggml_vec_dot_q4_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const voi
  
      uint32_t utmp[4];
  
@@ -266,7 +325,7 @@ index 21cd6312a..9b65eebb2 100644
  
      const __m256i m4 = _mm256_set1_epi8(0xF);
  
-@@ -2296,7 +2378,110 @@ void ggml_vec_dot_q5_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const voi
+@@ -2296,7 +2421,104 @@ void ggml_vec_dot_q5_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const voi
  
      uint32_t utmp[4];
  
@@ -284,12 +343,6 @@ index 21cd6312a..9b65eebb2 100644
 +    const __m512i sidx2 = K5_SCALE_IDX(4, 6);
 +    const __m512i sidx3 = K5_SCALE_IDX(5, 7);
 +    #undef K5_SCALE_IDX
-+    #define K5_CNT(a, b) _mm512_inserti64x4(_mm512_castsi256_si512(_mm256_set1_epi16(a)), _mm256_set1_epi16(b), 1)
-+    const __m512i cl0 = K5_CNT(4, 2);   // sub-blocks 0,2 : bit s -> bit 4 via left shift 4-s
-+    const __m512i cl1 = K5_CNT(3, 1);   // sub-blocks 1,3
-+    const __m512i cr2 = K5_CNT(0, 2);   // sub-blocks 4,6 : right shift s-4
-+    const __m512i cr3 = K5_CNT(1, 3);   // sub-blocks 5,7
-+    #undef K5_CNT
 +    // per-vector bit selectors: low 32 bytes test bit s, high 32 bytes test bit s+2 (sub-block pairs (0,2),(1,3),(4,6),(5,7))
 +    #define K5_BM(a, b) _mm512_inserti64x4(_mm512_castsi256_si512(_mm256_set1_epi8((char)(1 << (a)))), _mm256_set1_epi8((char)(1 << (b))), 1)
 +    const __m512i bm0 = K5_BM(0, 2);
@@ -378,7 +431,7 @@ index 21cd6312a..9b65eebb2 100644
  
      const __m256i m4 = _mm256_set1_epi8(0xF);
      const __m128i mzero = _mm_setzero_si128();
-@@ -2500,7 +2685,81 @@ void ggml_vec_dot_q6_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const voi
+@@ -2500,7 +2722,81 @@ void ggml_vec_dot_q6_K_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const voi
  
      const int nb = n / QK_K;
  
@@ -461,7 +514,7 @@ index 21cd6312a..9b65eebb2 100644
  
      const __m256i m3 = _mm256_set1_epi8(3);
      const __m256i m15 = _mm256_set1_epi8(15);
-@@ -4078,7 +4337,75 @@ void ggml_vec_dot_iq4_xs_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const v
+@@ -4078,7 +4374,75 @@ void ggml_vec_dot_iq4_xs_q8_K(int n, float * GGML_RESTRICT s, size_t bs, const v
  
      const int nb = n / QK_K;