Follow-up to #38: GLM-5.3-Flash on CPU only (2× Xeon 6530, AMX) — 21 tok/s decode, 25.7 with MTP, ~480 tok/s prefill, working prefix cache

In my previous post (#38) I ran the FP8 checkpoint
with KTransformers on 2×L40S + EPYC 9845. It worked, but in the end it had two drawbacks:

• single-stream decode was 18.3 tok/s, and that figure needed the deferred-experts approximation (15.6 tok/s exact);
• the radix/prefix cache was force-disabled and MTP was not available on sm_89. In multi-turn agent loops, every
  turn re-prefilled the whole history.

Since then I went the other way: no GPU at all, a dual-socket Intel box with AMX, and a heavily patched
llama.cpp. On my workloads it now beats the GPU setup on single-user decode, and the prefix cache works.

TL;DR

The GPU box is still ahead on raw prefill and on many concurrent users. For one user driving an agent, the CPU
box wins for two reasons:

• With a working prefix cache, a turn that adds 1K tokens to a 50K-token conversation only processes the new
  tokens, plus a few hundred from the last recurrent-state checkpoint: by my estimate a few seconds, instead of
  re-prefilling 50K tokens (~55 s at 900 tok/s).
• MTP adds another ~25 % on decode.

Hardware and software

• 2× Intel Xeon Gold 6530 (Emerald Rapids, 2 × 32 cores, AMX), 1 TB DDR5, ~530 GB/s measured, 2 NUMA nodes.
  No GPU.
• GLM-5.3-Flash converted to Q8_0 GGUF (318 GiB) from the official checkpoint.
• llama.cpp (commit fc9ce6b) plus about 45 patches. Stock llama.cpp only runs this model on CPU with --no-repack;
  otherwise it hits a scheduler assert.
• Server: llama-server -t 64 -b 2048 -ub 2048, plus --spec-type draft-mtp --spec-draft-n-max 2 for MTP.
  -ub 2048 costs nothing in decode.

What was optimized

Decode on this machine is memory-bandwidth bound (~18B active parameters per token at 8.5 bits). Prefill is
compute bound. Every CPU operation of the graph was profiled, then rewritten where it paid off.

Memory and NUMA (2 sockets)

• Weights are split by rows across the two NUMAthe rows stored in its own memory: no
  cross-socket weight traffic. The MoE decode kernel reaches ~440 GB/s of the ~530 GB/s the machine can do.
• The Kimi Delta Attention (KDA) recurrent state is also placed per node: each socket updates its own heads.
• Threads are pinned, and FTZ/DAZ is forced. Denormals in the delta-rule states made some kernels 2–10× slower.

AMX kernels

• MoE experts in decode:
  • AMX int8 tiles;
  • activation quantization spread over all thr
  • wider work items;
  • gate + up + clamped SwiGLU fused into one pass.
• MoE in prefill: AMX-BF16. Activations are converted to bf16 once and copied to each NUMA node; Q8_0 weights are
  dequantized to bf16 panels on the fly. 1.7× faster.
• Projections in prefill: AMX-BF16 GEMM.
• MLA absorbed products (attn_k_b / attn_v_b, 3D weights that stock runs at ~1 TFLOPS on a few threads):
  per-head AMX int8 tiles, 8× faster.
• MLA attention: a shared-KV (MQA-style) flash-attention kernel that visits only the unmasked keys. Its AMX-BF16
  core is used for prefill.

Synchronization

• A two-level tree barrier, one level per socket, replaces the flat spin barrier.
• Weights of the next nodes are prefetched while threads wait in a barrier.
• Many small ops are fused, which removes barriers per layer.

GLM-5.3-specific ops

• Hyper-connections (mHC):
  • HC pre-mix + RMSNorm + norm weight in a sin
  • a dedicated small-batch path;
  • in prefill, RMSNorm fused with the hc_mixes product, so the normalized rows are never written.
• KDA:
  • fused gated delta-rule recurrence for decod
  • chunked (WY) form for prefill: blocks of 16 tokens, small fp32 AVX-512 GEMMs, 4.8× faster;
  • tiled short convolution, 3.1× faster;
  • vectorized gates using a vectorized expf that is bit-exact with glibc, checked on all 2³² inputs.
• MoE: the weighted sum of experts and the shared-expert add are fused. The shared expert runs first, so its weights
  can be prefetched.

Results on one 2048-token prefill batch

• 7.9 s → about 4.25 s (MoE 2.50 → 1.49 s, delta-rule recurrence 651 → 137 ms, MLA k_b/v_b 807 → 99 ms).
• Element-wise ops and norms: 2–5× faster.

Everything else

• MTP speculative decoding with the built-in MTP head: n = 2 is the sweet spot.
• Loader: the AMX repacking of the expert tensors now runs in parallel, straight into its NUMA place. Server startup
  went from 2 min 09 to 17 s, with the GGUF on

How correctness was checked

• Decode: every decode-path patch was verified bit-identical to the previous build. Logits are compared
  byte for byte, so the final build decodes exactly like the first optimized one. Against stock llama.cpp, the
  difference is at the level of stock's own batch-size noise.
• Prefill: the paths that are not bit-identical (chunked delta rule, bf16 MoE, AMX attention) were accepted
  only below the engine's natural noise. The reference build compared with itself, prefill against token-by-token
  decode, gives KLD 0.0131. The final build coms KLD 0.0094, measured on 2,048
  positions at 2K context.
• Text: greedy generations on short promptse the same answers and content. The
  wording diverges only at near-tie tokens.

One lesson for anyone benchmarking kernels or quants on this model. The MoE routing (top-8 of 288 experts)
amplifies any rounding change. Even a pure fp32 summation-order change shows up as KLD ≈ 0.010 through routing
flips. On a short sample (2 × 512-token chunks,haotic: one configuration read 0.034
there, reproducibly, and 0.0105 on 2,048 positions. That short measurement had made me wrongly reject the AMX-BF16
attention at first.

My advice: use at least ~2K positions, and compare against the engine's own batch-size noise, not against zero.

SGLang on CPU?

I also checked SGLang 0.5.21. glm5_next is there, but the KDA, DSA indexer and mHC kernels are CUDA/TileLang
only, so there is no CPU attention backend. I ported it to CPU myself (KDA recurrence in C++, dense MLA instead of
DSA): 5.9 tok/s decode and 278 tok/s prefill. The patched llama.cpp is well ahead on both.

Caveats

• These patches target Intel AMX (Sapphire / Emerald / Granite Rapids) and a 2-socket NUMA layout. They are not
  upstream.
• Measured contexts: decode 19.0 tok/s at 4K and 17.8 tok/s at 16K. I have not benchmarked 100K+ contexts yet,
  which the L40S setup handled.
• For many concurrent users or huge one-shot prompts, the GPU setup from #38 remains faster.