docs/REPORT.md

Full report (moved from the model card)

Detailed build, accuracy, memory-tier, throughput and hot/cold-expert results. Raw logs are in ../results/.

Flint EPQ for llama.cpp

CPU expert paging for the Unsloth flint-UD-Q4_K_XL.gguf checkpoint. Routed gate/up/down expert weights stay on disk and are copied into a bounded RAM arena as needed. Flint's router, selected-expert weights, shared expert, attention and dense layers remain on the upstream llama.cpp graph.

All numbers in this card come from one build: llama.cpp 970b51c plus flint-epq.patch (sha256 369ef9710e507f5f8caeae398382399b43dd9adeeb006dfd27ce5e84bc041db4). Raw logs are in results/.

Configurations:

• native: unmodified llama.cpp MoE path (SDQ_PAGER=0), the reference.
• v1: expert paging with the per-expert custom op ("mode C"). SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0.
• v2: v1 plus preload of all experts into the arena and row-parallel decode for gated experts. SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1, arena ≥ 28.6 GiB.
• v2 forced (ablation): v2 with SDQ_ROWPAR_MAX_MISS=100000. Row-parallel runs even when some needed experts are not resident.

Source model

• GGUF: unsloth/flint-GGUF, flint-UD-Q4_K_XL.gguf (25,054,688,704 bytes)
• Flint architecture: 49 layers, first 2 dense, followed by 47 MoE layers; 256 experts, top-6; 12,032 routed experts in total
• Routed expert tensors: 21.47 GiB. Gate/up quantization types vary by layer (IQ3_S 1 layer, IQ4_XS 25, Q4_K 3, Q5_K 18); down uses IQ4_XS 16, Q5_K 29, Q6_K 1, Q8_0 1. Every type is covered by the pager's dot-product and activation-quantization tables.
• Size per expert (gate + up + down): 2.44 MiB at most (2,555,904 bytes), 1.83 MiB on average. One expert is far smaller than any memory tier.

Test system

Component	Specification
CPU	Intel Xeon Platinum 8559C, 2 sockets × 48 cores, SMT enabled (192 host logical CPUs)
CPU features	AVX2, AVX-512, VNNI, BF16; AMX present on the host but disabled in this build
Container CPU quota	16 CPUs (cpu.max = 1,600,000 / 100,000). All runs below used 16 threads
Host memory	2.0 TiB
Container memory limit	104,000,000,000 bytes (about 96.9 GiB), cgroup v2; no writable cgroup, so no kernel-enforced limit
Container swap limit	2,936,848,384 bytes (about 2.74 GiB); no test used swap
GPU	None; CPU-only build and inference
OS / kernel	Amazon Linux 2023 x86_64, Linux 6.18.48

Build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc
curl -L 'https://huggingface.co/HelloSun/fint-epqv2/resolve/main/flint-epq.patch' -o flint-epq.patch
git apply flint-epq.patch
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF \
  -DGGML_NATIVE=OFF -DGGML_AVX2=ON -DGGML_AVX512=ON \
  -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
  -DGGML_AMX_INT8=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_BF16=OFF
cmake --build build -j 8 --target llama-cli llama-server llama-completion llama-perplexity

Run commands. The model is models/flint-UD-Q4_K_XL.gguf (hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir models).

# native
SDQ_PAGER=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

# v1 with a 1 GiB expert arena (set the size in MiB with SDQ_ARENA_MB)
SDQ_PAGER=1 SDQ_ARENA_MB=1024 SDQ_PREFETCH=0 SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

# v2 (arena must hold all 12,032 experts: >= 28.6 GiB)
SDQ_PAGER=1 SDQ_ARENA_MB=29400 SDQ_PREFETCH=0 SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

At exit the program prints MoE op 路徑計數：row-parallel N 次，mode C M 次, which shows which path each MoE op used.

bench.sh <GiB> <pager 0|1> is the memory benchmark used below. Version is chosen with SDQ_PRELOAD and SDQ_ROW_PARALLEL, and ARENA_MB overrides the arena size.

Accuracy

Reference: native llama.cpp on the same build. Text: the first 6,000 bytes of llama.cpp's README.md (results/acc/corpus.txt).

Test	Native	v1	v2
Greedy, 24 tokens, 8 threads	reference	identical to native	identical to native
Greedy, 24 tokens, 16 threads	reference	differs from native	identical to v1
Prefill perplexity (-ub 64, 8 threads, 4 chunks)	10.7790 ± 1.6026	10.8014 ± 1.6076 (+0.21 %)	10.8014 ± 1.6076
Prefill KL vs native (mean / median / max)	—	0.019120 / 0.004286 / 0.455	0.019120 / 0.004286 / 0.455
Prefill top-1 agreement vs native	—	92.3 % ± 1.2	92.3 % ± 1.2
Decode KL vs native (-ub 1, 16 threads, 2 chunks)	reference (PPL 14.7355)	0.044057; top-1 92.5 %	0.044057; top-1 92.5 %
Decode, v2 forced with 1 GiB arena, KL vs native	—	—	0.044057; top-1 92.5 %
Control: native -ub 64 vs native -ub 1	—	—	KL 0.034464; top-1 90.2 %
Decode, v2 vs v1	—	—	KL 0.000001 (max 0.000038); top-1 100 %

Findings:

• v2 matches v1 in prefill exactly. Prefill does not use row-parallel, so this shows the preload does not change results.
• In decode, v2 and v1 differ by KL 1e-6 with 100 % top-1 agreement. Row-parallel is numerically equivalent to mode C but not bit-identical.
• EPQ is not bit-identical to native. Float summation order differs, and 8-bit activation quantization amplifies that difference through the layers. The native -ub 64 vs -ub 1 control shows the same magnitude of divergence (KL 0.034, top-1 90 %). The EPQ deviation is in the range of reordering sensitivity for this checkpoint.
• Greedy text identity is fragile. At 8 threads v1 and v2 match native for 24 tokens; at 16 threads both diverge from native by a near-tie argmax flip.

Memory tiers (enforced RSS watchdog)

bench.sh samples process-tree RSS every 20 ms and kills the run if it exceeds the tier. This is an enforced sampled limit, not a kernel limit: RSS can exceed the threshold briefly between samples, and the count includes file-backed pages as reported by /proc.

Workload: context 128, batch 32, ubatch 16, 8 generated tokens, 16 threads. Each row is the median of 3 runs. "killed" means the watchdog terminated all 3 runs (exit −9), with the peak RSS at the kill shown.

Tier	Native	v1	v2 (as designed)	v2 forced
2 GiB, arena 256 MiB	killed (peak 2.04–2.07 GiB)	killed (peak 2.00 GiB)	killed (peak 2.00 GiB)	killed (peak 2.00 GiB)
2 GiB, arena 128 MiB	—	pass, peak 1.887 GiB; prompt 6.2 t/s; gen 3.9 t/s	pass, peak 1.887 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 0 / mode C 516	pass, peak 1.883 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 425 / mode C 91
4 GiB, arena 1 GiB	killed (peak 4.09–4.34 GiB)	pass, 2.764 GiB; prompt 7.0; gen 4.6	pass, 2.764 GiB; prompt 7.3; gen 4.6	pass, 2.758 GiB; prompt 7.0; gen 4.6; RP 425
8 GiB, arena 3 GiB	killed (peak 8.12–8.14 GiB)	pass, 4.764 GiB; prompt 5.5; gen 3.6	pass, 4.764 GiB; prompt 8.7; gen 4.9	pass, 4.757 GiB; prompt 9.2; gen 4.8; RP 425
16 GiB, arena 10 GiB	killed (peak 16.01 GiB)	pass, 11.766 GiB; prompt 10.3; gen 5.4	pass, 11.766 GiB; prompt 11.8; gen 5.5; RP 12 / mode C 504	pass, 11.759 GiB; prompt 12.1; gen 5.3; RP 425
32 GiB	pass, 24.286 GiB; prompt 58.1; gen 21.3	pass, 19.765 GiB (arena 18 GiB, 7,561 of 12,032 experts); prompt 12.2; gen 5.5	pass, 30.473 GiB (arena 28.7 GiB, all experts); prompt 31.6; gen 14.6; RP 425 / mode C 91	pass, 19.759 GiB (arena 18 GiB); prompt 11.7; gen 5.5; RP 425

Generation speeds are noisy. Some runs with identical code paths differ by 20–30 % (for example v1 at 8 GiB ranged 3.2–4.9 t/s). Treat differences under about 20 % as unresolved.

What this shows:

• Native llama.cpp needs about 24.3 GiB. It is killed at every tier below 32 GiB.
• The 2 GiB tier requires an arena of about 128 MiB. A 256 MiB arena is killed at the threshold.
• v2 changes behavior only at 32 GiB with all experts resident (425 row-parallel decode ops). Below that, the path is the same as v1 (16 GiB: 12 of 516 ops).
• At 32 GiB, v2 generates 14.6 t/s against 5.5 t/s for v1 (about 2.7×), and it uses 30.5 GiB of RAM.
• Forced row-parallel shows no reliable speed gain at any tier. Its medians fall within the run-to-run spread of v1 and v2. Its output is unchanged.

Throughput

Workload: context 512, batch 128, ubatch 32, 32 generated tokens, 16 threads, 3 runs each (median; range in brackets). No watchdog applied.

Configuration	Prompt	Generation	Peak RSS
native (SDQ_PAGER=0)	47.4 t/s	19.4 t/s (17.9–21.0)	24.31 GiB
v1, arena 18 GiB	12.7 t/s	5.5 t/s (5.2–6.0)	19.79 GiB
v1, arena 28.7 GiB, no preload	11.8 t/s	5.3 t/s (5.3–5.6)	30.51 GiB
v2, arena 28.7 GiB, preload + row-parallel	34.1 t/s	13.7 t/s (13.4–13.8)	30.50 GiB

v2 generates about 2.5× faster than v1 (13.7 vs 5.5 t/s) and about 70 % of native generation speed. The prompt phase is also faster in v2, probably because the preload removes SSD reads during prefill (not separately measured).

Hot and cold experts

Routing is not uniform. The hot/cold code (profile-pinned experts) was tested to see whether a cache that keeps frequently used experts resident would help.

• Skew is real. Per layer, the Gini coefficient of expert request counts is 0.38–0.62. The top 6 % of experts take 19–36 % of requests (uniform would be 6 %).
• A profile transfers across texts moderately. The top 10 % of experts per layer, taken from a profiling text, cover 28 % of requests in a held-out text (uniform: 10 %). Per-layer rank correlation between the two texts has median 0.58.
• The working set is larger than small arenas. One token uses 47 layers × 6 = 282 distinct (layer, expert) pairs, and an expert is typically reused only on a later token. Under LRU a cache of about 282 slots (≈ 0.7 GiB) or less gets almost no hits.
• Pinning a profiled hot set helps only in a narrow band. Measured on the held-out text, 514 tokens by the pager's count, with pinning at 25 % of arena slots:

Arena	LRU hit rate	Profile-pinned hit rate	Difference
128 MiB (52 slots)	0 %	0 %	0
256 MiB (105 slots)	0 %	0 %	0
1 GiB (420 slots)	19.3 %	21.5 %	+2.2 points
3 GiB (1,260 slots)	40.3 %	37.5 %	−2.8 points

Perplexity is identical (6.4793) for every LRU and pinned run, so caching changes speed only.

A trace replay (results/hot/sim.py, 1,017 tokens, uniform slots) gives a similar picture: +3.2 points at 1 GiB and +0.7 at 3 GiB, and it predicts a small hit rate at 128–256 MiB that the real pager does not reach.

Two problems in the earlier hot/cold code were fixed in this build. Hits were not counted in the profile (only misses were), and with no SDQ_HOT_PIN every profiled expert was pinned, which left no cold slot to evict. The default is now 25 % of arena slots, and pinning still requires a profile (SDQ_PROFILE_IN). Without one the cache is plain LRU.

Conclusion. The hot/cold effect is real but modest, and it changes sign with arena size. The main limit is the per-token working set: below about 0.7 GiB, most experts are re-read from SSD on every token, which is why the 2–4 GiB tiers are slow. Pinning stays opt-in.

Files: results/hot/ (profile, scripts, route traces as .gz, stats, logs, analysis and simulation output).

Version coverage

Feature	Version	Status for Flint
Expert paging with per-expert custom op ("mode C")	v1	Active in every run (prefill always uses it)
Preload all experts into the arena	v2	Active when the arena holds all 12,032 experts (≥ 28.6 GiB)
Row-parallel decode for gated experts	v2	Active when all needed experts are resident and the thread count exceeds the number of needed experts (decode, ≥ 7 threads). SDQ_ROWPAR_MAX_MISS relaxes the residency guard (ablation)
Nemotron no-gate / ReLU² path	v1, v2	Not used by Flint

The gated SiLU path routes through the pager. The v2 row-parallel path computes gate and up for the same row block and applies the same SwiGLU as mode C.

Flint's upstream build_moe_ffn applies a power-of-two amax rescale to the down-projection input and undoes it on the output. The rescale is exact in F32 and Q8_K quantization is scale-invariant, so the paged path omits it. The accuracy results above, measured against native on the same build, are the test of this choice.

Attribution

The pager is adapted from HelloSun's Nemotron EPQ v1 and v2.

The GGUF weights remain hosted by Unsloth. This repository contains source, patch, scripts, logs and test notes only.