KLD: add stock-vs-stock control showing flint noise floor; add graph accuracy test

README.md

@@ -11,5 +11,5 @@ taskset -c 0-15 llama.cpp/build/bin/llama-cli -m flint-UD-Q4_K_XL.gguf -t 16 -fa
 
 Measured on Xeon Platinum 8559C, 16 threads (median of 3 interleaved rounds, vs stock llama.cpp):
 pp256 231 -> 296 t/s (1.28x), pp2048 274 -> 307 (1.12x), tg128 38.2 -> 42.8 (1.12x).
-PPL 18.28 vs 18.36 stock (no degradation), but **KLD vs stock is 0.067** (top-1 agreement ~92 %), much larger than on Qwen3.6.
+PPL 18.28 vs 18.36 stock (no degradation). KLD vs stock is 0.067 (top-1 ~92 %), which is the same as stock vs stock with a different numeric path (`-fa off` 0.076, `-ub 128` 0.070): flint is very sensitive to tiny numeric differences.
 Details, caveats and the KLD bisect: `benchmarks/summary.md`. Validated only on this CPU.

benchmarks/graph_accuracy_test.cpp

@@ -0,0 +1,57 @@
+#include "ggml.h"
+#include "ggml-cpu.h"
+#include "ggml-backend.h"
+#include "ggml-alloc.h"
+#include <cstdio>
+#include <cmath>
+#include <vector>
+#include <random>
+#include <cstring>
+int main(){
+    ggml_backend_t be = ggml_backend_cpu_init(); ggml_backend_cpu_set_n_threads(be,16);
+    std::mt19937 g(7); std::normal_distribution<float> nd(0,1);
+    ggml_type types[]={GGML_TYPE_Q4_K,GGML_TYPE_Q5_K,GGML_TYPE_Q6_K,GGML_TYPE_IQ4_XS,GGML_TYPE_Q8_0};
+    const int K=2048,N=512,E=64,NU=6;
+    for(ggml_type t:types) for(int mode=0;mode<2;mode++) for(int T: {1,64}){
+        ggml_quantize_init(t);
+        size_t ctxsz=ggml_tensor_overhead()*16+ggml_graph_overhead()+(1<<20);
+        ggml_init_params ip={ctxsz,nullptr,true}; ggml_context*c=ggml_init(ip);
+        int ne=(mode==0)?1:E;
+        ggml_tensor*w=ggml_new_tensor_3d(c,t,K,N,ne);
+        ggml_tensor*x; ggml_tensor*ids=nullptr; ggml_tensor*y;
+        if(mode==0){ x=ggml_new_tensor_2d(c,GGML_TYPE_F32,K,T); y=ggml_mul_mat(c,ggml_view_2d(c,w,K,N,w->nb[1],0),x);} 
+        else { x=ggml_new_tensor_3d(c,GGML_TYPE_F32,K,NU,T); ids=ggml_new_tensor_2d(c,GGML_TYPE_I32,NU,T); y=ggml_mul_mat_id(c,w,x,ids);} 
+        ggml_cgraph*gr=ggml_new_graph(c); ggml_build_forward_expand(gr,y);
+        ggml_backend_buffer_t buf=ggml_backend_alloc_ctx_tensors(c,be);
+        // data
+        size_t rows=(size_t)N*ne; std::vector<float> wf((size_t)K*rows), im(K,1.f);
+        for(auto&v:wf)v=nd(g)*0.05f;
+        std::vector<char> wq(ggml_row_size(t,K)*rows);
+        ggml_quantize_chunk(t,wf.data(),wq.data(),0,rows,K,im.data());
+        ggml_backend_tensor_set(w,wq.data(),0,wq.size());
+        std::vector<float> xf((size_t)ggml_nelements(x)); for(auto&v:xf)v=nd(g);
+        ggml_backend_tensor_set(x,xf.data(),0,xf.size()*4);
+        std::vector<int32_t> idv; 
+        if(ids){ idv.resize((size_t)NU*T); for(int tt=0;tt<T;tt++){ for(int u=0;u<NU;u++) idv[tt*NU+u]=(tt*7+u*11+3)%E; } ggml_backend_tensor_set(ids,idv.data(),0,idv.size()*4);}        
+        ggml_backend_graph_compute(be,gr);
+        std::vector<float> out(ggml_nelements(y)); ggml_backend_tensor_get(y,out.data(),0,out.size()*4);
+        // reference: dequantize weights, float dot
+        std::vector<float> wd((size_t)K*rows);
+        auto tr=ggml_get_type_traits(t); tr->to_float(wq.data(),wd.data(),(int64_t)K*rows);
+        double num=0,den=0;
+        for(int tt=0;tt<T;tt++){
+            int nu=(mode==0)?1:NU;
+            for(int u=0;u<nu;u++){
+                int e=(mode==0)?0:idv[tt*NU+u];
+                const float* xv=&xf[(size_t)(mode==0? tt*K : (tt*NU+u)*K)];
+                for(int n=0;n<N;n++){
+                    double r=0; const float*wr=&wd[((size_t)e*N+n)*K]; for(int k=0;k<K;k++) r+=(double)wr[k]*xv[k];
+                    double o=out[mode==0? (size_t)tt*N+n : ((size_t)tt*NU+u)*N+n];
+                    num+=(o-r)*(o-r); den+=r*r;
+                }
+            }
+        }
+        printf("%-7s %s T=%-3d rel_L2_err=%.3e\n",ggml_type_name(t),mode?"mul_mat_id":"mul_mat   ",T,sqrt(num/den));
+        ggml_backend_buffer_free(buf); ggml_free(c);
+    }
+}

benchmarks/summary.md

@@ -28,12 +28,23 @@ do not apply to flint.
 | Qwen3.6 patch unchanged | 0.0664 +- 0.0049 | 92.2 % | - |
 | this patch | 0.0666 +- 0.0049 | ~92 % | **18.277** +- 0.787 |
 
-PPL is not worse than stock (18.363 +- 0.795), but KLD is far larger than on Qwen3.6 (0.003-0.006), so this patch is
-**not** numerically close to stock on flint. Bisect (`kld-bisect.log`, KLD when one VNNI kernel is disabled):
-Q4_K 0.0670, Q5_K 0.0524, Q6_K 0.0685, IQ4_XS 0.0664; removing all of `quants.c` gives 0.0162.
-So the Q4_K/Q5_K/Q6_K VNNI kernels carry most of the deviation and the new IQ4_XS kernel only ~0.0002.
-I did not find a defect in them (official `test-quantize-fns` dot-product errors equal stock); my unverified guess is that
-flint's 256-expert top-6 routing amplifies tiny numeric differences. The earlier flint_cpu_opt README reported KLD 0.2058.
+PPL is not worse than stock (18.363 +- 0.795).
+
+KLD is much larger than on Qwen3.6 (0.003-0.006), but **flint is numerically chaotic**: stock llama.cpp compared with itself
+under a different numeric path gives the same level of KLD (control experiments, same baseline logits):
+
+| stock, changed numeric path | KLD | same top-1 |
+|---|---|---|
+| `-fa off` | 0.0762 +- 0.0055 | 91.7 % |
+| `-ub 128 -b 128` | 0.0700 +- 0.0051 | 92.0 % |
+| this patch | 0.0666 +- 0.0049 | ~92 % |
+
+So 0.067 is inside the noise floor of stock itself and does not indicate a defect in the kernels. Supporting evidence:
+a graph-level test (`mul_mat` and `mul_mat_id`, K=2048, N=512, random weights) gives the same relative L2 error versus a float
+reference for stock and this patch to 4 digits (Q4_K/Q5_K/Q6_K/IQ4_XS ~7e-3, Q8_0 ~5.4e-3, dominated by Q8_K activation quantization).
+Bisect (`kld-bisect.log`, KLD with one VNNI kernel disabled): Q4_K 0.0670, Q5_K 0.0524, Q6_K 0.0685, IQ4_XS 0.0664;
+removing all of `quants.c` gives 0.0162. Why tiny differences get amplified (e.g. top-6-of-256 expert routing ties) was not verified.
+The earlier flint_cpu_opt README reported KLD 0.2058, which was the same effect measured on a different baseline.
 
 ## Profile (decode, 64 tokens)
 MUL_MAT q8_0 41 % (attention, shared expert, lm_head; close to bandwidth bound), MUL_MAT_ID q5_K 22 %,