Add full report (moved from model card)

docs/REPORT.md

@@ -0,0 +1,175 @@
+# Full report (moved from the model card)
+
+Detailed build, accuracy, memory-tier, throughput and hot/cold-expert results. Raw logs are in `../results/`.
+
+
+# Flint EPQ for llama.cpp
+
+CPU expert paging for the Unsloth `flint-UD-Q4_K_XL.gguf` checkpoint. Routed gate/up/down expert weights stay on disk and are copied into a bounded RAM arena as needed. Flint's router, selected-expert weights, shared expert, attention and dense layers remain on the upstream llama.cpp graph.
+
+All numbers in this card come from one build: llama.cpp `970b51c` plus `flint-epq.patch` (sha256 `94559012e2ea65cad6ce2175927c19095d13256bb4ec8fb2fdcd163570e00385`). Raw logs are in `results/`.
+
+Configurations:
+
+- **native**: unmodified llama.cpp MoE path (`SDQ_PAGER=0`), the reference.
+- **v1**: expert paging with the per-expert custom op ("mode C"). `SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0`.
+- **v2**: v1 plus preload of all experts into the arena and row-parallel decode for gated experts. `SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1`, arena ≥ 28.6 GiB.
+- **v2 forced** (ablation): v2 with `SDQ_ROWPAR_MAX_MISS=100000`. Row-parallel runs even when some needed experts are not resident.
+
+## Source model
+
+- GGUF: [`unsloth/flint-GGUF`](https://huggingface.co/unsloth/flint-GGUF), `flint-UD-Q4_K_XL.gguf` (25,054,688,704 bytes)
+- Flint architecture: 49 layers, first 2 dense, followed by 47 MoE layers; 256 experts, top-6; 12,032 routed experts in total
+- Routed expert tensors: 21.47 GiB. Gate/up quantization types vary by layer (`IQ3_S` 1 layer, `IQ4_XS` 25, `Q4_K` 3, `Q5_K` 18); down uses `IQ4_XS` 16, `Q5_K` 29, `Q6_K` 1, `Q8_0` 1. Every type is covered by the pager's dot-product and activation-quantization tables.
+- Size per expert (gate + up + down): 2.44 MiB at most (2,555,904 bytes), 1.83 MiB on average. One expert is far smaller than any memory tier.
+
+## Test system
+
+| Component | Specification |
+|---|---|
+| CPU | Intel Xeon Platinum 8559C, 2 sockets × 48 cores, SMT enabled (192 host logical CPUs) |
+| CPU features | AVX2, AVX-512, VNNI, BF16; AMX present on the host but disabled in this build |
+| Container CPU quota | 16 CPUs (`cpu.max` = 1,600,000 / 100,000). All runs below used 16 threads |
+| Host memory | 2.0 TiB |
+| Container memory limit | 104,000,000,000 bytes (about 96.9 GiB), cgroup v2; no writable cgroup, so no kernel-enforced limit |
+| Container swap limit | 2,936,848,384 bytes (about 2.74 GiB); no test used swap |
+| GPU | None; CPU-only build and inference |
+| OS / kernel | Amazon Linux 2023 x86_64, Linux 6.18.48 |
+
+## Build
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp
+cd llama.cpp
+git checkout 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc
+curl -L 'https://huggingface.co/HelloSun/fint-epqv2/resolve/main/flint-epq.patch' -o flint-epq.patch
+git apply flint-epq.patch
+cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF \
+  -DGGML_NATIVE=OFF -DGGML_AVX2=ON -DGGML_AVX512=ON \
+  -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
+  -DGGML_AMX_INT8=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_BF16=OFF
+cmake --build build -j 8 --target llama-cli llama-server llama-completion llama-perplexity
+```
+
+Run commands. The model is `models/flint-UD-Q4_K_XL.gguf` (`hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir models`).
+
+```bash
+# native
+SDQ_PAGER=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
+
+# v1 with a 1 GiB expert arena (set the size in MiB with SDQ_ARENA_MB)
+SDQ_PAGER=1 SDQ_ARENA_MB=1024 SDQ_PREFETCH=0 SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
+
+# v2 (arena must hold all 12,032 experts: >= 28.6 GiB)
+SDQ_PAGER=1 SDQ_ARENA_MB=29400 SDQ_PREFETCH=0 SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
+```
+
+At exit the program prints `MoE op 路徑計數：row-parallel N 次，mode C M 次`, which shows which path each MoE op used.
+
+`bench.sh <GiB> <pager 0|1>` is the memory benchmark used below. Version is chosen with `SDQ_PRELOAD` and `SDQ_ROW_PARALLEL`, and `ARENA_MB` overrides the arena size.
+
+## Accuracy
+
+Reference: native llama.cpp on the same build. Text: the first 6,000 bytes of llama.cpp's `README.md` (`results/acc/corpus.txt`).
+
+| Test | Native | v1 | v2 |
+|---|---|---|---|
+| Greedy, 24 tokens, 8 threads | reference | identical to native | identical to native |
+| Greedy, 24 tokens, 16 threads | reference | differs from native | identical to v1 |
+| Prefill perplexity (`-ub 64`, 8 threads, 4 chunks) | 10.7790 ± 1.6026 | 10.8014 ± 1.6076 (+0.21 %) | 10.8014 ± 1.6076 |
+| Prefill KL vs native (mean / median / max) | — | 0.019120 / 0.004286 / 0.455 | 0.019120 / 0.004286 / 0.455 |
+| Prefill top-1 agreement vs native | — | 92.3 % ± 1.2 | 92.3 % ± 1.2 |
+| Decode KL vs native (`-ub 1`, 16 threads, 2 chunks) | reference (PPL 14.7355) | 0.044057; top-1 92.5 % | 0.044057; top-1 92.5 % |
+| Decode, v2 forced with 1 GiB arena, KL vs native | — | — | 0.044057; top-1 92.5 % |
+| Control: native `-ub 64` vs native `-ub 1` | — | — | KL 0.034464; top-1 90.2 % |
+| Decode, v2 vs v1 | — | — | KL 0.000001 (max 0.000038); top-1 100 % |
+
+Findings:
+
+- v2 matches v1 in prefill exactly. Prefill does not use row-parallel, so this shows the preload does not change results.
+- In decode, v2 and v1 differ by KL 1e-6 with 100 % top-1 agreement. Row-parallel is numerically equivalent to mode C but not bit-identical.
+- EPQ is not bit-identical to native. Float summation order differs, and 8-bit activation quantization amplifies that difference through the layers. The native `-ub 64` vs `-ub 1` control shows the same magnitude of divergence (KL 0.034, top-1 90 %). The EPQ deviation is in the range of reordering sensitivity for this checkpoint.
+- Greedy text identity is fragile. At 8 threads v1 and v2 match native for 24 tokens; at 16 threads both diverge from native by a near-tie argmax flip.
+
+## Memory tiers (enforced RSS watchdog)
+
+`bench.sh` samples process-tree RSS every 20 ms and kills the run if it exceeds the tier. This is an enforced sampled limit, not a kernel limit: RSS can exceed the threshold briefly between samples, and the count includes file-backed pages as reported by `/proc`.
+
+Workload: context 128, batch 32, ubatch 16, 8 generated tokens, 16 threads. Each row is the median of 3 runs. "killed" means the watchdog terminated all 3 runs (exit −9), with the peak RSS at the kill shown.
+
+| Tier | Native | v1 | v2 (as designed) | v2 forced |
+|---|---|---|---|---|
+| 2 GiB, arena 256 MiB | killed (peak 2.04–2.07 GiB) | killed (peak 2.00 GiB) | killed (peak 2.00 GiB) | killed (peak 2.00 GiB) |
+| 2 GiB, arena 128 MiB | — | pass, peak 1.887 GiB; prompt 6.2 t/s; gen 3.9 t/s | pass, peak 1.887 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 0 / mode C 516 | pass, peak 1.883 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 425 / mode C 91 |
+| 4 GiB, arena 1 GiB | killed (peak 4.09–4.34 GiB) | pass, 2.764 GiB; prompt 7.0; gen 4.6 | pass, 2.764 GiB; prompt 7.3; gen 4.6 | pass, 2.758 GiB; prompt 7.0; gen 4.6; RP 425 |
+| 8 GiB, arena 3 GiB | killed (peak 8.12–8.14 GiB) | pass, 4.764 GiB; prompt 5.5; gen 3.6 | pass, 4.764 GiB; prompt 8.7; gen 4.9 | pass, 4.757 GiB; prompt 9.2; gen 4.8; RP 425 |
+| 16 GiB, arena 10 GiB | killed (peak 16.01 GiB) | pass, 11.766 GiB; prompt 10.3; gen 5.4 | pass, 11.766 GiB; prompt 11.8; gen 5.5; RP 12 / mode C 504 | pass, 11.759 GiB; prompt 12.1; gen 5.3; RP 425 |
+| 32 GiB | pass, 24.286 GiB; prompt 58.1; gen 21.3 | pass, 19.765 GiB (arena 18 GiB, 7,561 of 12,032 experts); prompt 12.2; gen 5.5 | pass, 30.473 GiB (arena 28.7 GiB, all experts); prompt 31.6; gen 14.6; RP 425 / mode C 91 | pass, 19.759 GiB (arena 18 GiB); prompt 11.7; gen 5.5; RP 425 |
+
+Generation speeds are noisy. Some runs with identical code paths differ by 20–30 % (for example v1 at 8 GiB ranged 3.2–4.9 t/s). Treat differences under about 20 % as unresolved.
+
+What this shows:
+
+- **Native llama.cpp needs about 24.3 GiB.** It is killed at every tier below 32 GiB.
+- **The 2 GiB tier requires an arena of about 128 MiB.** A 256 MiB arena is killed at the threshold.
+- **v2 changes behavior only at 32 GiB with all experts resident** (425 row-parallel decode ops). Below that, the path is the same as v1 (16 GiB: 12 of 516 ops).
+- **At 32 GiB, v2 generates 14.6 t/s against 5.5 t/s for v1 (about 2.7×)**, and it uses 30.5 GiB of RAM.
+- **Forced row-parallel shows no reliable speed gain** at any tier. Its medians fall within the run-to-run spread of v1 and v2. Its output is unchanged.
+
+## Throughput
+
+Workload: context 512, batch 128, ubatch 32, 32 generated tokens, 16 threads, 3 runs each (median; range in brackets). No watchdog applied.
+
+| Configuration | Prompt | Generation | Peak RSS |
+|---|---:|---:|---:|
+| native (`SDQ_PAGER=0`) | 47.4 t/s | 19.4 t/s (17.9–21.0) | 24.31 GiB |
+| v1, arena 18 GiB | 12.7 t/s | 5.5 t/s (5.2–6.0) | 19.79 GiB |
+| v1, arena 28.7 GiB, no preload | 11.8 t/s | 5.3 t/s (5.3–5.6) | 30.51 GiB |
+| **v2**, arena 28.7 GiB, preload + row-parallel | **34.1 t/s** | **13.7 t/s** (13.4–13.8) | 30.50 GiB |
+
+v2 generates about 2.5× faster than v1 (13.7 vs 5.5 t/s) and about 70 % of native generation speed. The prompt phase is also faster in v2, probably because the preload removes SSD reads during prefill (not separately measured).
+
+## Hot and cold experts
+
+Routing is not uniform. The hot/cold code (profile-pinned experts) was tested to see whether a cache that keeps frequently used experts resident would help.
+
+- **Skew is real.** Per layer, the Gini coefficient of expert request counts is 0.38–0.62. The top 6 % of experts take 19–36 % of requests (uniform would be 6 %).
+- **A profile transfers across texts moderately.** The top 10 % of experts per layer, taken from a profiling text, cover 28 % of requests in a held-out text (uniform: 10 %). Per-layer rank correlation between the two texts has median 0.58.
+- **The working set is larger than small arenas.** One token uses 47 layers × 6 = 282 distinct (layer, expert) pairs, and an expert is typically reused only on a later token. Under LRU a cache of about 282 slots (≈ 0.7 GiB) or less gets almost no hits.
+- **Pinning a profiled hot set helps only in a narrow band.** Measured on the held-out text, 514 tokens by the pager's count, with pinning at 25 % of arena slots:
+
+| Arena | LRU hit rate | Profile-pinned hit rate | Difference |
+|---:|---:|---:|---:|
+| 128 MiB (52 slots) | 0 % | 0 % | 0 |
+| 256 MiB (105 slots) | 0 % | 0 % | 0 |
+| 1 GiB (420 slots) | 19.3 % | 21.5 % | +2.2 points |
+| 3 GiB (1,260 slots) | 40.3 % | 37.5 % | −2.8 points |
+
+Perplexity is identical (6.4793) for every LRU and pinned run, so caching changes speed only.
+
+A trace replay (`results/hot/sim.py`, 1,017 tokens, uniform slots) gives a similar picture: +3.2 points at 1 GiB and +0.7 at 3 GiB, and it predicts a small hit rate at 128–256 MiB that the real pager does not reach.
+
+Two problems in the earlier hot/cold code were fixed in this build. Hits were not counted in the profile (only misses were), and with no `SDQ_HOT_PIN` every profiled expert was pinned, which left no cold slot to evict. The default is now 25 % of arena slots, and pinning still requires a profile (`SDQ_PROFILE_IN`). Without one the cache is plain LRU.
+
+**Conclusion.** The hot/cold effect is real but modest, and it changes sign with arena size. The main limit is the per-token working set: below about 0.7 GiB, most experts are re-read from SSD on every token, which is why the 2–4 GiB tiers are slow. Pinning stays opt-in.
+
+Files: `results/hot/` (profile, scripts, route traces as `.gz`, stats, logs, analysis and simulation output).
+
+## Version coverage
+
+| Feature | Version | Status for Flint |
+|---|---|---|
+| Expert paging with per-expert custom op ("mode C") | v1 | Active in every run (prefill always uses it) |
+| Preload all experts into the arena | v2 | Active when the arena holds all 12,032 experts (≥ 28.6 GiB) |
+| Row-parallel decode for gated experts | v2 | Active when all needed experts are resident and the thread count exceeds the number of needed experts (decode, ≥ 7 threads). `SDQ_ROWPAR_MAX_MISS` relaxes the residency guard (ablation) |
+| Nemotron no-gate / ReLU² path | v1, v2 | Not used by Flint |
+
+The gated SiLU path routes through the pager. The v2 row-parallel path computes gate and up for the same row block and applies the same SwiGLU as mode C.
+
+Flint's upstream `build_moe_ffn` applies a power-of-two amax rescale to the down-projection input and undoes it on the output. The rescale is exact in F32 and Q8_K quantization is scale-invariant, so the paged path omits it. The accuracy results above, measured against native on the same build, are the test of this choice.
+
+## Attribution
+
+The pager is adapted from HelloSun's Nemotron EPQ [v1 and v2](https://huggingface.co/HelloSun/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF_EPQ).
+
+The GGUF weights remain hosted by Unsloth. This repository contains source, patch, scripts, logs and test notes only.