Add auto arena mode (80% of available memory, capped at all experts); patch 369ef971

2873e19 verified 14 days ago

13.2 kB

# Full report (moved from the model card)

Detailed build, accuracy, memory-tier, throughput and hot/cold-expert results. Raw logs are in `../results/`.


# Flint EPQ for llama.cpp

CPU expert paging for the Unsloth `flint-UD-Q4_K_XL.gguf` checkpoint. Routed gate/up/down expert weights stay on disk and are copied into a bounded RAM arena as needed. Flint's router, selected-expert weights, shared expert, attention and dense layers remain on the upstream llama.cpp graph.

All numbers in this card come from one build: llama.cpp `970b51c` plus `flint-epq.patch` (sha256 `369ef9710e507f5f8caeae398382399b43dd9adeeb006dfd27ce5e84bc041db4`). Raw logs are in `results/`.

Configurations:

- **native**: unmodified llama.cpp MoE path (`SDQ_PAGER=0`), the reference.
- **v1**: expert paging with the per-expert custom op ("mode C"). `SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0`.
- **v2**: v1 plus preload of all experts into the arena and row-parallel decode for gated experts. `SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1`, arena ≥ 28.6 GiB.
- **v2 forced** (ablation): v2 with `SDQ_ROWPAR_MAX_MISS=100000`. Row-parallel runs even when some needed experts are not resident.

## Source model

- GGUF: [`unsloth/flint-GGUF`](https://huggingface.co/unsloth/flint-GGUF), `flint-UD-Q4_K_XL.gguf` (25,054,688,704 bytes)
- Flint architecture: 49 layers, first 2 dense, followed by 47 MoE layers; 256 experts, top-6; 12,032 routed experts in total
- Routed expert tensors: 21.47 GiB. Gate/up quantization types vary by layer (`IQ3_S` 1 layer, `IQ4_XS` 25, `Q4_K` 3, `Q5_K` 18); down uses `IQ4_XS` 16, `Q5_K` 29, `Q6_K` 1, `Q8_0` 1. Every type is covered by the pager's dot-product and activation-quantization tables.
- Size per expert (gate + up + down): 2.44 MiB at most (2,555,904 bytes), 1.83 MiB on average. One expert is far smaller than any memory tier.

## Test system

| Component | Specification |
|---|---|
| CPU | Intel Xeon Platinum 8559C, 2 sockets × 48 cores, SMT enabled (192 host logical CPUs) |
| CPU features | AVX2, AVX-512, VNNI, BF16; AMX present on the host but disabled in this build |
| Container CPU quota | 16 CPUs (`cpu.max` = 1,600,000 / 100,000). All runs below used 16 threads |
| Host memory | 2.0 TiB |
| Container memory limit | 104,000,000,000 bytes (about 96.9 GiB), cgroup v2; no writable cgroup, so no kernel-enforced limit |
| Container swap limit | 2,936,848,384 bytes (about 2.74 GiB); no test used swap |
| GPU | None; CPU-only build and inference |
| OS / kernel | Amazon Linux 2023 x86_64, Linux 6.18.48 |
## Build
```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc
curl -L 'https://huggingface.co/HelloSun/fint-epqv2/resolve/main/flint-epq.patch' -o flint-epq.patch
git apply flint-epq.patch
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF \
  -DGGML_NATIVE=OFF -DGGML_AVX2=ON -DGGML_AVX512=ON \
  -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
  -DGGML_AMX_INT8=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_BF16=OFF
cmake --build build -j 8 --target llama-cli llama-server llama-completion llama-perplexity
```
Run commands. The model is `models/flint-UD-Q4_K_XL.gguf` (`hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir models`).
```bash
# native
SDQ_PAGER=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

# v1 with a 1 GiB expert arena (set the size in MiB with SDQ_ARENA_MB)
SDQ_PAGER=1 SDQ_ARENA_MB=1024 SDQ_PREFETCH=0 SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
# v2 (arena must hold all 12,032 experts: >= 28.6 GiB)
SDQ_PAGER=1 SDQ_ARENA_MB=29400 SDQ_PREFETCH=0 SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
```
At exit the program prints `MoE op 路徑計數：row-parallel N 次，mode C M 次`, which shows which path each MoE op used.
`bench.sh <GiB> <pager 0|1>` is the memory benchmark used below. Version is chosen with `SDQ_PRELOAD` and `SDQ_ROW_PARALLEL`, and `ARENA_MB` overrides the arena size.
## Accuracy
Reference: native llama.cpp on the same build. Text: the first 6,000 bytes of llama.cpp's `README.md` (`results/acc/corpus.txt`).
| Test | Native | v1 | v2 |
|---|---|---|---|
| Greedy, 24 tokens, 8 threads | reference | identical to native | identical to native |
| Greedy, 24 tokens, 16 threads | reference | differs from native | identical to v1 |
| Prefill perplexity (`-ub 64`, 8 threads, 4 chunks) | 10.7790 ± 1.6026 | 10.8014 ± 1.6076 (+0.21 %) | 10.8014 ± 1.6076 |
| Prefill KL vs native (mean / median / max) | — | 0.019120 / 0.004286 / 0.455 | 0.019120 / 0.004286 / 0.455 |
| Prefill top-1 agreement vs native | — | 92.3 % ± 1.2 | 92.3 % ± 1.2 |
| Decode KL vs native (`-ub 1`, 16 threads, 2 chunks) | reference (PPL 14.7355) | 0.044057; top-1 92.5 % | 0.044057; top-1 92.5 % |
| Decode, v2 forced with 1 GiB arena, KL vs native | — | — | 0.044057; top-1 92.5 % |
| Control: native `-ub 64` vs native `-ub 1` | — | — | KL 0.034464; top-1 90.2 % |
| Decode, v2 vs v1 | — | — | KL 0.000001 (max 0.000038); top-1 100 % |
Findings:
- v2 matches v1 in prefill exactly. Prefill does not use row-parallel, so this shows the preload does not change results.
- In decode, v2 and v1 differ by KL 1e-6 with 100 % top-1 agreement. Row-parallel is numerically equivalent to mode C but not bit-identical.
- EPQ is not bit-identical to native. Float summation order differs, and 8-bit activation quantization amplifies that difference through the layers. The native `-ub 64` vs `-ub 1` control shows the same magnitude of divergence (KL 0.034, top-1 90 %). The EPQ deviation is in the range of reordering sensitivity for this checkpoint.
- Greedy text identity is fragile. At 8 threads v1 and v2 match native for 24 tokens; at 16 threads both diverge from native by a near-tie argmax flip.
## Memory tiers (enforced RSS watchdog)
`bench.sh` samples process-tree RSS every 20 ms and kills the run if it exceeds the tier. This is an enforced sampled limit, not a kernel limit: RSS can exceed the threshold briefly between samples, and the count includes file-backed pages as reported by `/proc`.
Workload: context 128, batch 32, ubatch 16, 8 generated tokens, 16 threads. Each row is the median of 3 runs. "killed" means the watchdog terminated all 3 runs (exit −9), with the peak RSS at the kill shown.
| Tier | Native | v1 | v2 (as designed) | v2 forced |
|---|---|---|---|---|
| 2 GiB, arena 256 MiB | killed (peak 2.04–2.07 GiB) | killed (peak 2.00 GiB) | killed (peak 2.00 GiB) | killed (peak 2.00 GiB) |
| 2 GiB, arena 128 MiB | — | pass, peak 1.887 GiB; prompt 6.2 t/s; gen 3.9 t/s | pass, peak 1.887 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 0 / mode C 516 | pass, peak 1.883 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 425 / mode C 91 |
| 4 GiB, arena 1 GiB | killed (peak 4.09–4.34 GiB) | pass, 2.764 GiB; prompt 7.0; gen 4.6 | pass, 2.764 GiB; prompt 7.3; gen 4.6 | pass, 2.758 GiB; prompt 7.0; gen 4.6; RP 425 |
| 8 GiB, arena 3 GiB | killed (peak 8.12–8.14 GiB) | pass, 4.764 GiB; prompt 5.5; gen 3.6 | pass, 4.764 GiB; prompt 8.7; gen 4.9 | pass, 4.757 GiB; prompt 9.2; gen 4.8; RP 425 |
| 16 GiB, arena 10 GiB | killed (peak 16.01 GiB) | pass, 11.766 GiB; prompt 10.3; gen 5.4 | pass, 11.766 GiB; prompt 11.8; gen 5.5; RP 12 / mode C 504 | pass, 11.759 GiB; prompt 12.1; gen 5.3; RP 425 |
| 32 GiB | pass, 24.286 GiB; prompt 58.1; gen 21.3 | pass, 19.765 GiB (arena 18 GiB, 7,561 of 12,032 experts); prompt 12.2; gen 5.5 | pass, 30.473 GiB (arena 28.7 GiB, all experts); prompt 31.6; gen 14.6; RP 425 / mode C 91 | pass, 19.759 GiB (arena 18 GiB); prompt 11.7; gen 5.5; RP 425 |
Generation speeds are noisy. Some runs with identical code paths differ by 20–30 % (for example v1 at 8 GiB ranged 3.2–4.9 t/s). Treat differences under about 20 % as unresolved.
What this shows:
- **Native llama.cpp needs about 24.3 GiB.** It is killed at every tier below 32 GiB.
- **The 2 GiB tier requires an arena of about 128 MiB.** A 256 MiB arena is killed at the threshold.
- **v2 changes behavior only at 32 GiB with all experts resident** (425 row-parallel decode ops). Below that, the path is the same as v1 (16 GiB: 12 of 516 ops).
- **At 32 GiB, v2 generates 14.6 t/s against 5.5 t/s for v1 (about 2.7×)**, and it uses 30.5 GiB of RAM.
- **Forced row-parallel shows no reliable speed gain** at any tier. Its medians fall within the run-to-run spread of v1 and v2. Its output is unchanged.
## Throughput
Workload: context 512, batch 128, ubatch 32, 32 generated tokens, 16 threads, 3 runs each (median; range in brackets). No watchdog applied.
| Configuration | Prompt | Generation | Peak RSS |
|---|---:|---:|---:|
| native (`SDQ_PAGER=0`) | 47.4 t/s | 19.4 t/s (17.9–21.0) | 24.31 GiB |
| v1, arena 18 GiB | 12.7 t/s | 5.5 t/s (5.2–6.0) | 19.79 GiB |
| v1, arena 28.7 GiB, no preload | 11.8 t/s | 5.3 t/s (5.3–5.6) | 30.51 GiB |
| **v2**, arena 28.7 GiB, preload + row-parallel | **34.1 t/s** | **13.7 t/s** (13.4–13.8) | 30.50 GiB |
v2 generates about 2.5× faster than v1 (13.7 vs 5.5 t/s) and about 70 % of native generation speed. The prompt phase is also faster in v2, probably because the preload removes SSD reads during prefill (not separately measured).
## Hot and cold experts
Routing is not uniform. The hot/cold code (profile-pinned experts) was tested to see whether a cache that keeps frequently used experts resident would help.
- **Skew is real.** Per layer, the Gini coefficient of expert request counts is 0.38–0.62. The top 6 % of experts take 19–36 % of requests (uniform would be 6 %).
- **A profile transfers across texts moderately.** The top 10 % of experts per layer, taken from a profiling text, cover 28 % of requests in a held-out text (uniform: 10 %). Per-layer rank correlation between the two texts has median 0.58.
- **The working set is larger than small arenas.** One token uses 47 layers × 6 = 282 distinct (layer, expert) pairs, and an expert is typically reused only on a later token. Under LRU a cache of about 282 slots (≈ 0.7 GiB) or less gets almost no hits.
- **Pinning a profiled hot set helps only in a narrow band.** Measured on the held-out text, 514 tokens by the pager's count, with pinning at 25 % of arena slots:
| Arena | LRU hit rate | Profile-pinned hit rate | Difference |
|---:|---:|---:|---:|
| 128 MiB (52 slots) | 0 % | 0 % | 0 |
| 256 MiB (105 slots) | 0 % | 0 % | 0 |
| 1 GiB (420 slots) | 19.3 % | 21.5 % | +2.2 points |
| 3 GiB (1,260 slots) | 40.3 % | 37.5 % | −2.8 points |
Perplexity is identical (6.4793) for every LRU and pinned run, so caching changes speed only.
A trace replay (`results/hot/sim.py`, 1,017 tokens, uniform slots) gives a similar picture: +3.2 points at 1 GiB and +0.7 at 3 GiB, and it predicts a small hit rate at 128–256 MiB that the real pager does not reach.
Two problems in the earlier hot/cold code were fixed in this build. Hits were not counted in the profile (only misses were), and with no `SDQ_HOT_PIN` every profiled expert was pinned, which left no cold slot to evict. The default is now 25 % of arena slots, and pinning still requires a profile (`SDQ_PROFILE_IN`). Without one the cache is plain LRU.
**Conclusion.** The hot/cold effect is real but modest, and it changes sign with arena size. The main limit is the per-token working set: below about 0.7 GiB, most experts are re-read from SSD on every token, which is why the 2–4 GiB tiers are slow. Pinning stays opt-in.
Files: `results/hot/` (profile, scripts, route traces as `.gz`, stats, logs, analysis and simulation output).
## Version coverage
| Feature | Version | Status for Flint |
|---|---|---|
| Expert paging with per-expert custom op ("mode C") | v1 | Active in every run (prefill always uses it) |
| Preload all experts into the arena | v2 | Active when the arena holds all 12,032 experts (≥ 28.6 GiB) |
| Row-parallel decode for gated experts | v2 | Active when all needed experts are resident and the thread count exceeds the number of needed experts (decode, ≥ 7 threads). `SDQ_ROWPAR_MAX_MISS` relaxes the residency guard (ablation) |
| Nemotron no-gate / ReLU² path | v1, v2 | Not used by Flint |
The gated SiLU path routes through the pager. The v2 row-parallel path computes gate and up for the same row block and applies the same SwiGLU as mode C.
Flint's upstream `build_moe_ffn` applies a power-of-two amax rescale to the down-projection input and undoes it on the output. The rescale is exact in F32 and Q8_K quantization is scale-invariant, so the paged path omits it. The accuracy results above, measured against native on the same build, are the test of this choice.
## Attribution
The pager is adapted from HelloSun's Nemotron EPQ [v1 and v2](https://huggingface.co/HelloSun/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF_EPQ).
The GGUF weights remain hosted by Unsloth. This repository contains source, patch, scripts, logs and test notes only.

	# Full report (moved from the model card)

	Detailed build, accuracy, memory-tier, throughput and hot/cold-expert results. Raw logs are in `../results/`.


	# Flint EPQ for llama.cpp

	CPU expert paging for the Unsloth `flint-UD-Q4_K_XL.gguf` checkpoint. Routed gate/up/down expert weights stay on disk and are copied into a bounded RAM arena as needed. Flint's router, selected-expert weights, shared expert, attention and dense layers remain on the upstream llama.cpp graph.

	All numbers in this card come from one build: llama.cpp `970b51c` plus `flint-epq.patch` (sha256 `369ef9710e507f5f8caeae398382399b43dd9adeeb006dfd27ce5e84bc041db4`). Raw logs are in `results/`.

	Configurations:

	- native: unmodified llama.cpp MoE path (`SDQ_PAGER=0`), the reference.
	- v1: expert paging with the per-expert custom op ("mode C"). `SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0`.
	- v2: v1 plus preload of all experts into the arena and row-parallel decode for gated experts. `SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1`, arena ≥ 28.6 GiB.
	- v2 forced (ablation): v2 with `SDQ_ROWPAR_MAX_MISS=100000`. Row-parallel runs even when some needed experts are not resident.

	## Source model

	- GGUF: [`unsloth/flint-GGUF`](https://huggingface.co/unsloth/flint-GGUF), `flint-UD-Q4_K_XL.gguf` (25,054,688,704 bytes)
	- Flint architecture: 49 layers, first 2 dense, followed by 47 MoE layers; 256 experts, top-6; 12,032 routed experts in total
	- Routed expert tensors: 21.47 GiB. Gate/up quantization types vary by layer (`IQ3_S` 1 layer, `IQ4_XS` 25, `Q4_K` 3, `Q5_K` 18); down uses `IQ4_XS` 16, `Q5_K` 29, `Q6_K` 1, `Q8_0` 1. Every type is covered by the pager's dot-product and activation-quantization tables.
	- Size per expert (gate + up + down): 2.44 MiB at most (2,555,904 bytes), 1.83 MiB on average. One expert is far smaller than any memory tier.

	## Test system

	\| Component \| Specification \|
	\|---\|---\|
	\| CPU \| Intel Xeon Platinum 8559C, 2 sockets × 48 cores, SMT enabled (192 host logical CPUs) \|
	\| CPU features \| AVX2, AVX-512, VNNI, BF16; AMX present on the host but disabled in this build \|
	\| Container CPU quota \| 16 CPUs (`cpu.max` = 1,600,000 / 100,000). All runs below used 16 threads \|
	\| Host memory \| 2.0 TiB \|
	\| Container memory limit \| 104,000,000,000 bytes (about 96.9 GiB), cgroup v2; no writable cgroup, so no kernel-enforced limit \|
	\| Container swap limit \| 2,936,848,384 bytes (about 2.74 GiB); no test used swap \|
	\| GPU \| None; CPU-only build and inference \|
	\| OS / kernel \| Amazon Linux 2023 x86_64, Linux 6.18.48 \|

	## Build

	```bash
	git clone https://github.com/ggml-org/llama.cpp
	cd llama.cpp
	git checkout 970b51ce7c9ba5fff0726c5e9994f1c2f0e4eedc
	curl -L 'https://huggingface.co/HelloSun/fint-epqv2/resolve/main/flint-epq.patch' -o flint-epq.patch
	git apply flint-epq.patch
	cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF \
	-DGGML_NATIVE=OFF -DGGML_AVX2=ON -DGGML_AVX512=ON \
	-DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON \
	-DGGML_AMX_INT8=OFF -DGGML_AMX_TILE=OFF -DGGML_AMX_BF16=OFF
	cmake --build build -j 8 --target llama-cli llama-server llama-completion llama-perplexity
	```

	Run commands. The model is `models/flint-UD-Q4_K_XL.gguf` (`hf download unsloth/flint-GGUF flint-UD-Q4_K_XL.gguf --local-dir models`).

	```bash
	# native
	SDQ_PAGER=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

	# v1 with a 1 GiB expert arena (set the size in MiB with SDQ_ARENA_MB)
	SDQ_PAGER=1 SDQ_ARENA_MB=1024 SDQ_PREFETCH=0 SDQ_PRELOAD=0 SDQ_ROW_PARALLEL=0 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io

	# v2 (arena must hold all 12,032 experts: >= 28.6 GiB)
	SDQ_PAGER=1 SDQ_ARENA_MB=29400 SDQ_PREFETCH=0 SDQ_PRELOAD=1 SDQ_ROW_PARALLEL=1 build/bin/llama-cli -m models/flint-UD-Q4_K_XL.gguf -c 512 -p 'Explain in one sentence why the sky is blue.' -n 32 -t 16 --temp 0 -st --simple-io
	```

	At exit the program prints `MoE op 路徑計數：row-parallel N 次，mode C M 次`, which shows which path each MoE op used.

	`bench.sh <GiB> <pager 0\|1>` is the memory benchmark used below. Version is chosen with `SDQ_PRELOAD` and `SDQ_ROW_PARALLEL`, and `ARENA_MB` overrides the arena size.

	## Accuracy

	Reference: native llama.cpp on the same build. Text: the first 6,000 bytes of llama.cpp's `README.md` (`results/acc/corpus.txt`).

	\| Test \| Native \| v1 \| v2 \|
	\|---\|---\|---\|---\|
	\| Greedy, 24 tokens, 8 threads \| reference \| identical to native \| identical to native \|
	\| Greedy, 24 tokens, 16 threads \| reference \| differs from native \| identical to v1 \|
	\| Prefill perplexity (`-ub 64`, 8 threads, 4 chunks) \| 10.7790 ± 1.6026 \| 10.8014 ± 1.6076 (+0.21 %) \| 10.8014 ± 1.6076 \|
	\| Prefill KL vs native (mean / median / max) \| — \| 0.019120 / 0.004286 / 0.455 \| 0.019120 / 0.004286 / 0.455 \|
	\| Prefill top-1 agreement vs native \| — \| 92.3 % ± 1.2 \| 92.3 % ± 1.2 \|
	\| Decode KL vs native (`-ub 1`, 16 threads, 2 chunks) \| reference (PPL 14.7355) \| 0.044057; top-1 92.5 % \| 0.044057; top-1 92.5 % \|
	\| Decode, v2 forced with 1 GiB arena, KL vs native \| — \| — \| 0.044057; top-1 92.5 % \|
	\| Control: native `-ub 64` vs native `-ub 1` \| — \| — \| KL 0.034464; top-1 90.2 % \|
	\| Decode, v2 vs v1 \| — \| — \| KL 0.000001 (max 0.000038); top-1 100 % \|

	Findings:

	- v2 matches v1 in prefill exactly. Prefill does not use row-parallel, so this shows the preload does not change results.
	- In decode, v2 and v1 differ by KL 1e-6 with 100 % top-1 agreement. Row-parallel is numerically equivalent to mode C but not bit-identical.
	- EPQ is not bit-identical to native. Float summation order differs, and 8-bit activation quantization amplifies that difference through the layers. The native `-ub 64` vs `-ub 1` control shows the same magnitude of divergence (KL 0.034, top-1 90 %). The EPQ deviation is in the range of reordering sensitivity for this checkpoint.
	- Greedy text identity is fragile. At 8 threads v1 and v2 match native for 24 tokens; at 16 threads both diverge from native by a near-tie argmax flip.

	## Memory tiers (enforced RSS watchdog)

	`bench.sh` samples process-tree RSS every 20 ms and kills the run if it exceeds the tier. This is an enforced sampled limit, not a kernel limit: RSS can exceed the threshold briefly between samples, and the count includes file-backed pages as reported by `/proc`.

	Workload: context 128, batch 32, ubatch 16, 8 generated tokens, 16 threads. Each row is the median of 3 runs. "killed" means the watchdog terminated all 3 runs (exit −9), with the peak RSS at the kill shown.

	\| Tier \| Native \| v1 \| v2 (as designed) \| v2 forced \|
	\|---\|---\|---\|---\|---\|
	\| 2 GiB, arena 256 MiB \| killed (peak 2.04–2.07 GiB) \| killed (peak 2.00 GiB) \| killed (peak 2.00 GiB) \| killed (peak 2.00 GiB) \|
	\| 2 GiB, arena 128 MiB \| — \| pass, peak 1.887 GiB; prompt 6.2 t/s; gen 3.9 t/s \| pass, peak 1.887 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 0 / mode C 516 \| pass, peak 1.883 GiB; prompt 6.1 t/s; gen 3.9 t/s; RP 425 / mode C 91 \|
	\| 4 GiB, arena 1 GiB \| killed (peak 4.09–4.34 GiB) \| pass, 2.764 GiB; prompt 7.0; gen 4.6 \| pass, 2.764 GiB; prompt 7.3; gen 4.6 \| pass, 2.758 GiB; prompt 7.0; gen 4.6; RP 425 \|
	\| 8 GiB, arena 3 GiB \| killed (peak 8.12–8.14 GiB) \| pass, 4.764 GiB; prompt 5.5; gen 3.6 \| pass, 4.764 GiB; prompt 8.7; gen 4.9 \| pass, 4.757 GiB; prompt 9.2; gen 4.8; RP 425 \|
	\| 16 GiB, arena 10 GiB \| killed (peak 16.01 GiB) \| pass, 11.766 GiB; prompt 10.3; gen 5.4 \| pass, 11.766 GiB; prompt 11.8; gen 5.5; RP 12 / mode C 504 \| pass, 11.759 GiB; prompt 12.1; gen 5.3; RP 425 \|
	\| 32 GiB \| pass, 24.286 GiB; prompt 58.1; gen 21.3 \| pass, 19.765 GiB (arena 18 GiB, 7,561 of 12,032 experts); prompt 12.2; gen 5.5 \| pass, 30.473 GiB (arena 28.7 GiB, all experts); prompt 31.6; gen 14.6; RP 425 / mode C 91 \| pass, 19.759 GiB (arena 18 GiB); prompt 11.7; gen 5.5; RP 425 \|

	Generation speeds are noisy. Some runs with identical code paths differ by 20–30 % (for example v1 at 8 GiB ranged 3.2–4.9 t/s). Treat differences under about 20 % as unresolved.

	What this shows:

	- Native llama.cpp needs about 24.3 GiB. It is killed at every tier below 32 GiB.
	- The 2 GiB tier requires an arena of about 128 MiB. A 256 MiB arena is killed at the threshold.
	- v2 changes behavior only at 32 GiB with all experts resident (425 row-parallel decode ops). Below that, the path is the same as v1 (16 GiB: 12 of 516 ops).
	- At 32 GiB, v2 generates 14.6 t/s against 5.5 t/s for v1 (about 2.7×), and it uses 30.5 GiB of RAM.
	- Forced row-parallel shows no reliable speed gain at any tier. Its medians fall within the run-to-run spread of v1 and v2. Its output is unchanged.

	## Throughput

	Workload: context 512, batch 128, ubatch 32, 32 generated tokens, 16 threads, 3 runs each (median; range in brackets). No watchdog applied.

	\| Configuration \| Prompt \| Generation \| Peak RSS \|
	\|---\|---:\|---:\|---:\|
	\| native (`SDQ_PAGER=0`) \| 47.4 t/s \| 19.4 t/s (17.9–21.0) \| 24.31 GiB \|
	\| v1, arena 18 GiB \| 12.7 t/s \| 5.5 t/s (5.2–6.0) \| 19.79 GiB \|
	\| v1, arena 28.7 GiB, no preload \| 11.8 t/s \| 5.3 t/s (5.3–5.6) \| 30.51 GiB \|
	\| v2, arena 28.7 GiB, preload + row-parallel \| 34.1 t/s \| 13.7 t/s (13.4–13.8) \| 30.50 GiB \|

	v2 generates about 2.5× faster than v1 (13.7 vs 5.5 t/s) and about 70 % of native generation speed. The prompt phase is also faster in v2, probably because the preload removes SSD reads during prefill (not separately measured).

	## Hot and cold experts

	Routing is not uniform. The hot/cold code (profile-pinned experts) was tested to see whether a cache that keeps frequently used experts resident would help.

	- Skew is real. Per layer, the Gini coefficient of expert request counts is 0.38–0.62. The top 6 % of experts take 19–36 % of requests (uniform would be 6 %).
	- A profile transfers across texts moderately. The top 10 % of experts per layer, taken from a profiling text, cover 28 % of requests in a held-out text (uniform: 10 %). Per-layer rank correlation between the two texts has median 0.58.
	- The working set is larger than small arenas. One token uses 47 layers × 6 = 282 distinct (layer, expert) pairs, and an expert is typically reused only on a later token. Under LRU a cache of about 282 slots (≈ 0.7 GiB) or less gets almost no hits.
	- Pinning a profiled hot set helps only in a narrow band. Measured on the held-out text, 514 tokens by the pager's count, with pinning at 25 % of arena slots:

	\| Arena \| LRU hit rate \| Profile-pinned hit rate \| Difference \|
	\|---:\|---:\|---:\|---:\|
	\| 128 MiB (52 slots) \| 0 % \| 0 % \| 0 \|
	\| 256 MiB (105 slots) \| 0 % \| 0 % \| 0 \|
	\| 1 GiB (420 slots) \| 19.3 % \| 21.5 % \| +2.2 points \|
	\| 3 GiB (1,260 slots) \| 40.3 % \| 37.5 % \| −2.8 points \|

	Perplexity is identical (6.4793) for every LRU and pinned run, so caching changes speed only.

	A trace replay (`results/hot/sim.py`, 1,017 tokens, uniform slots) gives a similar picture: +3.2 points at 1 GiB and +0.7 at 3 GiB, and it predicts a small hit rate at 128–256 MiB that the real pager does not reach.

	Two problems in the earlier hot/cold code were fixed in this build. Hits were not counted in the profile (only misses were), and with no `SDQ_HOT_PIN` every profiled expert was pinned, which left no cold slot to evict. The default is now 25 % of arena slots, and pinning still requires a profile (`SDQ_PROFILE_IN`). Without one the cache is plain LRU.

	Conclusion. The hot/cold effect is real but modest, and it changes sign with arena size. The main limit is the per-token working set: below about 0.7 GiB, most experts are re-read from SSD on every token, which is why the 2–4 GiB tiers are slow. Pinning stays opt-in.

	Files: `results/hot/` (profile, scripts, route traces as `.gz`, stats, logs, analysis and simulation output).

	## Version coverage

	\| Feature \| Version \| Status for Flint \|
	\|---\|---\|---\|
	\| Expert paging with per-expert custom op ("mode C") \| v1 \| Active in every run (prefill always uses it) \|
	\| Preload all experts into the arena \| v2 \| Active when the arena holds all 12,032 experts (≥ 28.6 GiB) \|
	\| Row-parallel decode for gated experts \| v2 \| Active when all needed experts are resident and the thread count exceeds the number of needed experts (decode, ≥ 7 threads). `SDQ_ROWPAR_MAX_MISS` relaxes the residency guard (ablation) \|
	\| Nemotron no-gate / ReLU² path \| v1, v2 \| Not used by Flint \|

	The gated SiLU path routes through the pager. The v2 row-parallel path computes gate and up for the same row block and applies the same SwiGLU as mode C.

	Flint's upstream `build_moe_ffn` applies a power-of-two amax rescale to the down-projection input and undoes it on the output. The rescale is exact in F32 and Q8_K quantization is scale-invariant, so the paged path omits it. The accuracy results above, measured against native on the same build, are the test of this choice.

	## Attribution

	The pager is adapted from HelloSun's Nemotron EPQ [v1 and v2](https://huggingface.co/HelloSun/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF_EPQ).

	The GGUF weights remain hosted by Unsloth. This repository contains source, patch, scripts, logs and test notes only.