HelloSun/LFM2.5-24B-A2B-Q4_K_M_EPQ

LFM2.5-24B-A2B Q4_K_M — EPQ expert paging experiment

This repository contains an experimental CPU inference path for paging inactive Mixture-of-Experts (MoE) weights from SSD into a bounded RAM arena. Model weights are not included. Download the upstream GGUF from LiquidAI/LFM2.5-24B-A2B-GGUF:

hf download LiquidAI/LFM2.5-24B-A2B-GGUF \
  LFM2.5-24B-A2B-Q4_K_M.gguf --local-dir models

The measurements and limits are documented in results/report.md. This is a reproduction of the experiment style in HelloSun/ling30flash-moe-expert-paging-q4, adapted to the LFM2.5 lfm2moe architecture.

Model facts

• GGUF: LFM2.5-24B-A2B-Q4_K_M.gguf (13.529 GiB; exact file size is recorded in results)
• 40 blocks, first 2 dense; 38 MoE layers
• 64 experts per MoE layer; top-4 selected per token
• Expert projections: gate/up Q4_K; down mixes Q4_K and Q6_K by layer
• Expert tensors: 12.89 GiB total; GGUF file: 13.529 GiB
• EPQ pages complete expert triplets; routing and weighting stay in llama.cpp

Memory and decode performance validation

Hard anonymous-memory ceilings were enforced with Linux prlimit --data; each level was run twice with the same prompt, context 2048, ubatch 128, 8 threads, greedy sampling, and 47 generated decode tokens. The table reports median speed and both individual repetitions. Anon HWM is the anonymous high-water mark checked against the hard ceiling. RSS also includes file-backed mappings and can therefore exceed the anonymous limit. Swap stayed at zero in every run.

Hard RAM limit	EPQ arena	Slots	Anon HWM	RSS HWM	Decode tok/s (runs; median)	Prefill tok/s (runs; median)	Expert cache hit	SSD MiB / decode token	Swap
4 GiB	1.58 GiB	299	2.52 GiB	3.15 GiB	3.38, 3.02; 3.20	12.3, 10.7; 11.5	28.86%	540.3	0
8 GiB	5.59 GiB	1,054	6.52 GiB	7.15 GiB	7.39, 5.03; 6.21	12.2, 10.5; 11.35	60.26%	207.9	0
16 GiB	12.89 GiB	2,432	13.83 GiB	14.45 GiB	13.78, 11.11; 12.45	13.3, 10.8; 12.05	77.49%	25.3	0
32 GiB	12.89 GiB	2,432	13.83 GiB	14.45 GiB	11.51, 14.21; 12.86	6.4, 8.0; 7.2	77.49%	25.3	0

The pager caps the arena at one slot per expert, so the 16 and 32 GiB settings both fit the entire 12.89 GiB expert set. Speed varied on this shared host; cache hit rate and SSD traffic show the more stable trend.

Original upstream llama.cpp comparison

The baseline is a clean, unpatched llama.cpp checkout at the same commit, e467e5b5e5ba43071d6151b3d881deb9300d000e, built Release with GGML_NATIVE=ON. It uses llama-cli (not the EPQ sdq-chat binary), but the model, prompt, system message, ctx=2048, batch/ubatch=128, 8 compute threads, greedy temperature 0, flash attention, cache types, and 48-token generation limit are held constant. llama-cli reports 48 generated tokens; the EPQ harness stops at EOG, so EPQ's strict run reports 47. The 16 GiB row is the direct hard-limit comparison.

Mode / anonymous hard ceiling	Decode tok/s (runs; median)	Prompt/prefill tok/s (runs; median)	Anon HWM	RSS HWM	Swap	Result
Original upstream llama.cpp / 8 GiB	—	—	—	—	0	FAIL: CPU repack allocation requests 10,368 MiB
EPQ / 8 GiB	7.39, 5.03; 6.21	12.2, 10.5; 11.35	6.52 GiB	7.15 GiB	0	PASS
Original upstream llama.cpp / 16 GiB	25.6, 30.3; 27.95	89.5, 94.9; 92.2	10.89 GiB	24.41 GiB	0	PASS
EPQ / 16 GiB	13.78, 11.11; 12.45	13.3, 10.8; 12.05	13.83 GiB	14.45 GiB	0	PASS

The upstream 16 GiB decode median is about 2.2× EPQ on this host. EPQ's trade-off is lower resident memory: about 14.45 GiB RSS at the 16 GiB setting, versus 24.41 GiB for upstream; at an 8 GiB anonymous ceiling the upstream backend cannot allocate its CPU repack buffer, while EPQ runs successfully. RSS can exceed the anonymous RLIMIT_DATA cap because file-backed mappings are not covered by that limit. Timings vary with shared-host load; see raw runs under results/evidence/upstream_*.

For a same-front-end RAM comparison, the patched build's sdq-chat --no-pager control was also run against EPQ sdq-chat, with the same prompt, context, threads, batch sizes, and 16 GiB hard limit:

sdq-chat mode / 16 GiB hard limit	Decode tok/s (runs; median)	Anon HWM	RSS HWM	Swap
Native full load (--no-pager)	22.71, 23.04, 18.87; 22.71	10.87 GiB	24.32 GiB	0
EPQ paging	13.78, 11.11; 12.45	13.83 GiB	14.45 GiB	0

The native sdq-chat control is not upstream code: it uses the unmodified llama.cpp MoE graph but shares the experiment CLI and measurement path. The clean-upstream llama-cli results above are the actual original-upstream baseline; both comparisons are supplied so front-end differences are visible.

Correctness validation

Check	Result
NumPy dequantization vs ggml Q4_K/Q6_K dequantizers	36/36 exact across layers 2, 5, 20, 39 and experts 0, 17, 63
Single expert L2/E17 (Q6_K down)	PASS; max relative error 0.0170; correlation 0.999866
Single expert L5/E17 (Q4_K down)	PASS; max relative error 0.0114; correlation 0.999910
Single expert L39/E17 (Q6_K down)	PASS; max relative error 0.0191; correlation 0.999823
Full routed MoE layer 2 (4 selected experts)	PASS; max relative error 0.02523; correlation 0.999835
Full routed MoE layer 5 (4 selected experts)	PASS; max relative error 0.00406; correlation 0.999854
Full routed MoE layer 39 (4 selected experts)	PASS; max relative error 0.00495; correlation 0.999961

The full-layer checks use the actual graph's routed expert IDs and weights, then recompute expert outputs from GGUF bytes in NumPy.

Expert I/O and page-cache validation

The GGUF page cache was evicted before measurement. While decoding, /proc I/O counters and whole-file mincore residency were sampled every 0.5 seconds.

Read path	Block-device read_bytes peak	GGUF page-cache peak	Process RssAnon peak	Process RssFile peak
EPQ default (O_DIRECT)	17.829 GiB	0.657 GiB	2.370 GiB	0.631 GiB
Buffered control, cache retained	7.922 GiB	7.930 GiB (8.035 GiB final)	2.370 GiB	0.632 GiB

The buffered control confirms the mincore measurement detects cached reads; the EPQ direct-I/O run keeps model-file residency low while reading experts. Raw sampling logs and per-run records are under results/.

Build and run

Tested against llama.cpp commit e467e5b5e5ba43071d6151b3d881deb9300d000e. The repository includes the llama.cpp integration patch and the sdq_* source files (legacy internal implementation names retained from the previous EPQ port).

export EPQ_MODEL="$PWD/models/LFM2.5-24B-A2B-Q4_K_M.gguf"
./serve.sh build
RAM_BUDGET_MB=8192 ./serve.sh run

The experimental server defaults to port 22222. RAM_BUDGET_MB is the total anonymous-memory budget used by the pager's sizing logic. To impose a hard Linux anonymous-memory ceiling as in the report, launch the binary with e.g.:

SDQ_RAM_BUDGET_MB=8192 SDQ_RESERVE_MB=2400 SDQ_PREFETCH=0 \
  prlimit --data=8589934592 --nofile=65536 \
  llama.cpp/build/bin/sdq-chat -m "$EPQ_MODEL" -c 2048 -b 128 -t 8 \
  -p 'Explain MoE expert paging.' -n 48 --temp 0 --no-interactive

Use SDQ_PAGER=0 with sdq-chat --no-pager for a full-load CPU control. GPU execution was unavailable in the measurement container.

For a clean checkout, place the downloaded model at models/ beside this directory (or set EPQ_MODEL to its path). verify_dequant.py requires a small ggml_deq helper; build it after building llama.cpp with:

g++ -O2 -o llama.cpp/build/bin/ggml_deq ggml_deq.cpp \
  -Illama.cpp/ggml/include -Illama.cpp/ggml/src \
  -Lllama.cpp/build/bin -lggml -lggml-base \
  -Wl,-rpath,"$PWD/llama.cpp/build/bin"

Verification and benchmarks

The report includes the strict prlimit --data sweep, native full-load baseline, NumPy/ggml dequantization equivalence checks, three real routed-layer comparisons, single-expert checks, and direct-I/O vs buffered page-cache evidence.

python3 verify_dequant.py
python3 verify_epq.py results/evidence/st_l2e17.bin \
  results/evidence/st_l5e17.bin results/evidence/st_l39e17.bin
python3 verify_moe_layer.py results/evidence/moe_layer.2 2

Model license and usage terms remain those of the upstream LiquidAI model.