HelloSun
/

fint-epqv2

@@ -12,38 +12,11 @@ tags:
 # Flint EPQ for llama.cpp
-CPU expert paging for the Unsloth `flint-UD-Q4_K_XL.gguf` checkpoint. Routed gate/up/down expert weights stay on disk and are copied into a bounded RAM arena as needed. Flint's router, selected-expert weights, shared expert, attention and dense layers remain on the upstream llama.cpp graph.
-All numbers in this card come from one build: llama.cpp `970b51c` plus `flint-epq.patch` (sha256 `94559012e2ea65cad6ce2175927c19095d13256bb4ec8fb2fdcd163570e00385`). Raw logs are in `results/`.
-Configurations:
-- **native**: unmodified llama.cpp MoE path (`SDQ_PAGER=0`), the reference.
-- **v1**: expert paging with the per-expert custom op ("mode C"). `SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0`.
-- **v2**: v1 plus preload of all experts into the arena and row-parallel decode for gated experts. `SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1`, arena ≥ 28.6 GiB.
-- **v2 forced** (ablation): v2 with `SDQ_ROWPAR_MAX_MISS=100000`. Row-parallel runs even when some needed experts are not resident.
-## Source model
-- GGUF: [`unsloth/flint-GGUF`](https://huggingface.co/unsloth/flint-GGUF), `flint-UD-Q4_K_XL.gguf` (25,054,688,704 bytes)
-- Flint architecture: 49 layers, first 2 dense, followed by 47 MoE layers; 256 experts, top-6; 12,032 routed experts in total
-- Routed expert tensors: 21.47 GiB. Gate/up quantization types vary by layer (`IQ3_S` 1 layer, `IQ4_XS` 25, `Q4_K` 3, `Q5_K` 18); down uses `IQ4_XS` 16, `Q5_K` 29, `Q6_K` 1, `Q8_0` 1. Every type is covered by the pager's dot-product and activation-quantization tables.
-- Size per expert (gate + up + down): 2.44 MiB at most (2,555,904 bytes), 1.83 MiB on average. One expert is far smaller than any memory tier.
-## Test system
-| Component | Specification |
-|---|---|
-| CPU | Intel Xeon Platinum 8559C, 2 sockets × 48 cores, SMT enabled (192 host logical CPUs) |
-| CPU features | AVX2, AVX-512, VNNI, BF16; AMX present on the host but disabled in this build |
-| Container CPU quota | 16 CPUs (`cpu.max` = 1,600,000 / 100,000). All runs below used 16 threads |
-| Host memory | 2.0 TiB |
-| Container memory limit | 104,000,000,000 bytes (about 96.9 GiB), cgroup v2; no writable cgroup, so no kernel-enforced limit |
-| Container swap limit | 2,936,848,384 bytes (about 2.74 GiB); no test used swap |
-| GPU | None; CPU-only build and inference |
-| OS / kernel | Amazon Linux 2023 x86_64, Linux 6.18.48 |
-## Build
 ```bash
 git clone https://github.com/ggml-org/llama.cpp
@@ -51,132 +24,45 @@ cd llama.cpp
 git checkout 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc
 curl -L 'https://huggingface.co/HelloSun/fint-epqv2/resolve/main/flint-epq.patch' -o flint-epq.patch
 git apply flint-epq.patch
-cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF \
-  -DGGML_NATIVE=OFF -DGGML_AVX2=ON -DGGML_AVX512=ON \
-  -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
   -DGGML_AMX_INT8=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_BF16=OFF
-cmake --build build -j 8 --target llama-cli llama-server llama-completion llama-perplexity
-```
-Run commands. The model is `models/flint-UD-Q4_K_XL.gguf` (`hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir models`).
-```bash
-# native
-SDQ_PAGER=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
-# v1 with a 1 GiB expert arena (set the size in MiB with SDQ_ARENA_MB)
-SDQ_PAGER=1 SDQ_ARENA_MB=1024 SDQ_PREFETCH=0 SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
-# v2 (arena must hold all 12,032 experts: >= 28.6 GiB)
-SDQ_PAGER=1 SDQ_ARENA_MB=29400 SDQ_PREFETCH=0 SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
 ```
-At exit the program prints `MoE op 路徑計數：row-parallel N 次，mode C M 次`, which shows which path each MoE op used.
-`bench.sh <GiB> <pager 0|1>` is the memory benchmark used below. Version is chosen with `SDQ_PRELOAD` and `SDQ_ROW_PARALLEL`, and `ARENA_MB` overrides the arena size.
-## Accuracy
-Reference: native llama.cpp on the same build. Text: the first 6,000 bytes of llama.cpp's `README.md` (`results/acc/corpus.txt`).
-| Test | Native | v1 | v2 |
-|---|---|---|---|
-| Greedy, 24 tokens, 8 threads | reference | identical to native | identical to native |
-| Greedy, 24 tokens, 16 threads | reference | differs from native | identical to v1 |
-| Prefill perplexity (`-ub 64`, 8 threads, 4 chunks) | 10.7790 ± 1.6026 | 10.8014 ± 1.6076 (+0.21 %) | 10.8014 ± 1.6076 |
-| Prefill KL vs native (mean / median / max) | — | 0.019120 / 0.004286 / 0.455 | 0.019120 / 0.004286 / 0.455 |
-| Prefill top-1 agreement vs native | — | 92.3 % ± 1.2 | 92.3 % ± 1.2 |
-| Decode KL vs native (`-ub 1`, 16 threads, 2 chunks) | reference (PPL 14.7355) | 0.044057; top-1 92.5 % | 0.044057; top-1 92.5 % |
-| Decode, v2 forced with 1 GiB arena, KL vs native | — | — | 0.044057; top-1 92.5 % |
-| Control: native `-ub 64` vs native `-ub 1` | — | — | KL 0.034464; top-1 90.2 % |
-| Decode, v2 vs v1 | — | — | KL 0.000001 (max 0.000038); top-1 100 % |
-Findings:
-- v2 matches v1 in prefill exactly. Prefill does not use row-parallel, so this shows the preload does not change results.
-- In decode, v2 and v1 differ by KL 1e-6 with 100 % top-1 agreement. Row-parallel is numerically equivalent to mode C but not bit-identical.
-- EPQ is not bit-identical to native. Float summation order differs, and 8-bit activation quantization amplifies that difference through the layers. The native `-ub 64` vs `-ub 1` control shows the same magnitude of divergence (KL 0.034, top-1 90 %). The EPQ deviation is in the range of reordering sensitivity for this checkpoint.
-- Greedy text identity is fragile. At 8 threads v1 and v2 match native for 24 tokens; at 16 threads both diverge from native by a near-tie argmax flip.
-## Memory tiers (enforced RSS watchdog)
-`bench.sh` samples process-tree RSS every 20 ms and kills the run if it exceeds the tier. This is an enforced sampled limit, not a kernel limit: RSS can exceed the threshold briefly between samples, and the count includes file-backed pages as reported by `/proc`.
-Workload: context 128, batch 32, ubatch 16, 8 generated tokens, 16 threads. Each row is the median of 3 runs. "killed" means the watchdog terminated all 3 runs (exit −9), with the peak RSS at the kill shown.
-| Tier | Native | v1 | v2 (as designed) | v2 forced |
-|---|---|---|---|---|
-| 2 GiB, arena 256 MiB | killed (peak 2.04–2.07 GiB) | killed (peak 2.00 GiB) | killed (peak 2.00 GiB) | killed (peak 2.00 GiB) |
-| 2 GiB, arena 128 MiB | — | pass, peak 1.887 GiB; prompt 6.2 t/s; gen 3.9 t/s | pass, peak 1.887 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 0 / mode C 516 | pass, peak 1.883 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 425 / mode C 91 |
-| 4 GiB, arena 1 GiB | killed (peak 4.09–4.34 GiB) | pass, 2.764 GiB; prompt 7.0; gen 4.6 | pass, 2.764 GiB; prompt 7.3; gen 4.6 | pass, 2.758 GiB; prompt 7.0; gen 4.6; RP 425 |
-| 8 GiB, arena 3 GiB | killed (peak 8.12–8.14 GiB) | pass, 4.764 GiB; prompt 5.5; gen 3.6 | pass, 4.764 GiB; prompt 8.7; gen 4.9 | pass, 4.757 GiB; prompt 9.2; gen 4.8; RP 425 |
-| 16 GiB, arena 10 GiB | killed (peak 16.01 GiB) | pass, 11.766 GiB; prompt 10.3; gen 5.4 | pass, 11.766 GiB; prompt 11.8; gen 5.5; RP 12 / mode C 504 | pass, 11.759 GiB; prompt 12.1; gen 5.3; RP 425 |
-| 32 GiB | pass, 24.286 GiB; prompt 58.1; gen 21.3 | pass, 19.765 GiB (arena 18 GiB, 7,561 of 12,032 experts); prompt 12.2; gen 5.5 | pass, 30.473 GiB (arena 28.7 GiB, all experts); prompt 31.6; gen 14.6; RP 425 / mode C 91 | pass, 19.759 GiB (arena 18 GiB); prompt 11.7; gen 5.5; RP 425 |
-Generation speeds are noisy. Some runs with identical code paths differ by 20–30 % (for example v1 at 8 GiB ranged 3.2–4.9 t/s). Treat differences under about 20 % as unresolved.
-What this shows:
-- **Native llama.cpp needs about 24.3 GiB.** It is killed at every tier below 32 GiB.
-- **The 2 GiB tier requires an arena of about 128 MiB.** A 256 MiB arena is killed at the threshold.
-- **v2 changes behavior only at 32 GiB with all experts resident** (425 row-parallel decode ops). Below that, the path is the same as v1 (16 GiB: 12 of 516 ops).
-- **At 32 GiB, v2 generates 14.6 t/s against 5.5 t/s for v1 (about 2.7×)**, and it uses 30.5 GiB of RAM.
-- **Forced row-parallel shows no reliable speed gain** at any tier. Its medians fall within the run-to-run spread of v1 and v2. Its output is unchanged.
-## Throughput
-Workload: context 512, batch 128, ubatch 32, 32 generated tokens, 16 threads, 3 runs each (median; range in brackets). No watchdog applied.
-| Configuration | Prompt | Generation | Peak RSS |
-|---|---:|---:|---:|
-| native (`SDQ_PAGER=0`) | 47.4 t/s | 19.4 t/s (17.9–21.0) | 24.31 GiB |
-| v1, arena 18 GiB | 12.7 t/s | 5.5 t/s (5.2–6.0) | 19.79 GiB |
-| v1, arena 28.7 GiB, no preload | 11.8 t/s | 5.3 t/s (5.3–5.6) | 30.51 GiB |
-| **v2**, arena 28.7 GiB, preload + row-parallel | **34.1 t/s** | **13.7 t/s** (13.4–13.8) | 30.50 GiB |
-v2 generates about 2.5× faster than v1 (13.7 vs 5.5 t/s) and about 70 % of native generation speed. The prompt phase is also faster in v2, probably because the preload removes SSD reads during prefill (not separately measured).
-## Hot and cold experts
-Routing is not uniform. The hot/cold code (profile-pinned experts) was tested to see whether a cache that keeps frequently used experts resident would help.
-- **Skew is real.** Per layer, the Gini coefficient of expert request counts is 0.38–0.62. The top 6 % of experts take 19–36 % of requests (uniform would be 6 %).
-- **A profile transfers across texts moderately.** The top 10 % of experts per layer, taken from a profiling text, cover 28 % of requests in a held-out text (uniform: 10 %). Per-layer rank correlation between the two texts has median 0.58.
-- **The working set is larger than small arenas.** One token uses 47 layers × 6 = 282 distinct (layer, expert) pairs, and an expert is typically reused only on a later token. Under LRU a cache of about 282 slots (≈ 0.7 GiB) or less gets almost no hits.
-- **Pinning a profiled hot set helps only in a narrow band.** Measured on the held-out text, 514 tokens by the pager's count, with pinning at 25 % of arena slots:
-| Arena | LRU hit rate | Profile-pinned hit rate | Difference |
-|---:|---:|---:|---:|
-| 128 MiB (52 slots) | 0 % | 0 % | 0 |
-| 256 MiB (105 slots) | 0 % | 0 % | 0 |
-| 1 GiB (420 slots) | 19.3 % | 21.5 % | +2.2 points |
-| 3 GiB (1,260 slots) | 40.3 % | 37.5 % | −2.8 points |
-Perplexity is identical (6.4793) for every LRU and pinned run, so caching changes speed only.
-A trace replay (`results/hot/sim.py`, 1,017 tokens, uniform slots) gives a similar picture: +3.2 points at 1 GiB and +0.7 at 3 GiB, and it predicts a small hit rate at 128–256 MiB that the real pager does not reach.
-Two problems in the earlier hot/cold code were fixed in this build. Hits were not counted in the profile (only misses were), and with no `SDQ_HOT_PIN` every profiled expert was pinned, which left no cold slot to evict. The default is now 25 % of arena slots, and pinning still requires a profile (`SDQ_PROFILE_IN`). Without one the cache is plain LRU.
-**Conclusion.** The hot/cold effect is real but modest, and it changes sign with arena size. The main limit is the per-token working set: below about 0.7 GiB, most experts are re-read from SSD on every token, which is why the 2–4 GiB tiers are slow. Pinning stays opt-in.
-Files: `results/hot/` (profile, scripts, route traces as `.gz`, stats, logs, analysis and simulation output).
-## Version coverage
-| Feature | Version | Status for Flint |
-|---|---|---|
-| Expert paging with per-expert custom op ("mode C") | v1 | Active in every run (prefill always uses it) |
-| Preload all experts into the arena | v2 | Active when the arena holds all 12,032 experts (≥ 28.6 GiB) |
-| Row-parallel decode for gated experts | v2 | Active when all needed experts are resident and the thread count exceeds the number of needed experts (decode, ≥ 7 threads). `SDQ_ROWPAR_MAX_MISS` relaxes the residency guard (ablation) |
-| Nemotron no-gate / ReLU² path | v1, v2 | Not used by Flint |
-The gated SiLU path routes through the pager. The v2 row-parallel path computes gate and up for the same row block and applies the same SwiGLU as mode C.
-Flint's upstream `build_moe_ffn` applies a power-of-two amax rescale to the down-projection input and undoes it on the output. The rescale is exact in F32 and Q8_K quantization is scale-invariant, so the paged path omits it. The accuracy results above, measured against native on the same build, are the test of this choice.
-## Attribution
-The pager is adapted from HelloSun's Nemotron EPQ [v1 and v2](https://huggingface.co/HelloSun/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF_EPQ).
-The GGUF weights remain hosted by Unsloth. This repository contains source, patch, scripts, logs and test notes only.

 # Flint EPQ for llama.cpp
+CPU expert paging for `unsloth/flint-GGUF` (`flint-UD-Q4_K_XL.gguf`). Routed expert weights stay on disk and are loaded into a RAM arena on demand. Full results: [docs/REPORT.md](docs/REPORT.md).
+## Install
+Requires a CMake build of llama.cpp at commit `970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc` with `flint-epq.patch` applied (sha256 `94559012e2ea65cad6ce2175927c19095d13256bb4ec8fb2fdcd163570e00385`).
 ```bash
 git clone https://github.com/ggml-org/llama.cpp
 git checkout 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc
 curl -L 'https://huggingface.co/HelloSun/fint-epqv2/resolve/main/flint-epq.patch' -o flint-epq.patch
 git apply flint-epq.patch
+cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DGGML_NATIVE=OFF \
+  -DGGML_AVX2=ON -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
   -DGGML_AMX_INT8=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_BF16=OFF
+cmake --build build -j 8 --target llama-cli
+hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir models
 ```
+## Run
+Prompt used below: `Explain in one sentence why the sky is blue.`
+**native** (unmodified llama.cpp MoE path, needs about 24 GiB RAM):
+```bash
+SDQ_PAGER=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
+  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
+```
+**v1** (expert paging, mode C). Preload and row-parallel default to on, so turn them off explicitly. `SDQ_ARENA_MB` is the expert cache size in MiB:
+```bash
+SDQ_PAGER=1 SDQ_ARENA_MB=18432 SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0 \
+  build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
+  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
+```
+**v2** (default settings: preload and row-parallel on). The arena must hold all 12,032 experts, so use at least 28.6 GiB:
+```bash
+SDQ_PAGER=1 SDQ_ARENA_MB=29400 \
+  build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 \
+  -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
+```
+Use 16 threads (`-t 16`). Row-parallel needs more threads than the number of experts used per step (6), so at least 7.
+## Notes
+- Memory: the process needs about 1.8 GiB plus the arena. Under a 2 GiB limit a 128 MiB arena works (peak 1.89 GiB); 256 MiB does not.
+- Speed (32 GiB, 16 threads): native 19.4 t/s generation, v1 5.5 t/s, v2 13.7 t/s. Details and memory-tier tables are in the report.
+- Output matches native greedy decoding at 8 threads; at 16 threads it diverges after a few tokens (float reordering). See the report.