HANDOFF: start Q4_1 optimization on this machine (baseline + plan)

HANDOFF.md

@@ -0,0 +1,73 @@
+# HANDOFF — flint Q4_1 CPU optimization (this machine)
+
+Updated: 2026-10-31 (UTC+1) — periodic checkpoint (uploaded every ~5 min while working).
+
+## Read this first
+
+This is the live handoff for continuing the Q4_1 optimization of
+`HelloSun/flint_cpu_opt`. The static docs are `README.md`,
+`benchmarks/q41-summary.md` (Q4_1) and `benchmarks/summary.md` (Q4_K_XL).
+
+## Machine (the one this work targets)
+
+- Intel Xeon Platinum 8559C, 2 sockets, 192 logical CPUs online.
+- **cgroup limit: 16 CPUs** (`cpu.max = 1600000/100000`), `nproc` = 16.
+- 2 TiB RAM host; memory guard: agent-started processes must stay < 80 GiB RSS.
+- Flags: AVX512-F/BW/VL/DQ, AVX512-VNNI, AVX512-FP16, AMX-INT8/AMX-BF16, F16C.
+- Same CPU model as the numbers in `machine.txt` (16 threads), but the host is
+  shared, so single runs vary ~±8 %.
+- Repo base: llama.cpp `de5122d879d00d2895c36be7e0cee94105aeeabd` with
+  `patches/flint-cpu-opt.patch` applied in `llama.cpp/`.
+
+## Current model/server
+
+- Model: `flint-int4-q4_1.gguf` (24.2 GiB, every weight Q4_1; token_embd +
+  router F32), downloaded from `RecursiveSI/flint-int4-gguf`.
+- `llama-server` running on `127.0.0.1:20000`, alias `flint-q4_1`,
+  `-t 16 -fa on --jinja -c 32768` (started by `build.sh`).
+- pi coding agent is configured to use it (`pi-local`).
+
+## Current Q4_1 kernel state (already in the patch)
+
+1. `ggml_vec_dot_q4_1_q8_1` (decode): AVX512-VNNI, 2 blocks per ymm, 4
+   blocks/iteration, prefetch `GGML_Q41_PF` (default 192 blocks, T1 hint).
+2. Tiled VNNI GEMM for Q4_1 prefill: per-32-block f32 `sd`/`sm`, `Q41`
+   template flag, Q8_1 activations.
+3. Reported on 8559C/16t: pp256 266, pp2048 297, tg128 55.7 t/s; stock
+   183/189/48.3. KLD inside stock's own noise floor.
+
+## Plan / next steps
+
+- [x] Read existing docs and machine context.
+- [ ] Build `llama-bench` + `llama-cli` (only `llama-server` was built).
+- [ ] Baseline on **this** machine: interleaved A/B, `taskset -c 0-15`,
+      `-t 16 -fa on -b 1024 -ub 1024`, Q4_1 file.
+- [ ] Sweep `GGML_Q41_PF` (e.g. 128/192/256/320) and thread counts on the
+      real model (not only microbench).
+- [ ] Try one decode-side improvement (memory-bound path) and/or an AMX
+      prefill variant; keep only if it beats noise.
+- [ ] A/B compare, record raw CSV/log under `benchmarks/`.
+- [ ] Update `README.md` + `benchmarks/q41-summary.md` only with measured,
+      reproducible numbers.
+- [ ] Restart `llama-server` with the final best settings.
+
+## How to reproduce
+
+```bash
+cd /root/flint_cpu_opt
+cmake --build llama.cpp/build -j16 --target llama-bench llama-cli
+taskset -c 0-15 llama.cpp/build/bin/llama-bench -m flint-int4-q4_1.gguf \
+  -t 16 -fa on -lm mlock -b 1024 -ub 1024 -p 256,2048 -n 128
+```
+
+## Notes / gotchas
+
+- Do **not** run 100 parallel Tor/open code probes here; memory guard kills
+  processes over 80 GiB / single child over 64 GiB.
+- The llama-server is serving pi; stop it before a clean benchmark, restart
+  afterwards (`bash build.sh` reuses an already-running server).
+- Host is shared: use medians of >= 3 interleaved rounds; differences under
+  ~8 % are not a win.
+- Kernel changes must keep integer dot products exact; only f32 accumulation
+  order may change. Check `benchmarks/graph_accuracy_test.cpp` and KLD if a
+  numeric path changes.