GLM-5.3-Flash, streaming EXL3 packs of turboderp's self-calibrated quants, for mlx-serve
This is the repo's main card, dated 2026-11-19. "We" is OpensourceWTF (David Tai). Every measurement on it is ours, from one Apple M5 Max with 128 GB on 2026-11-19, unless it names turboderp or a README.
Summary
This repo holds four of turboderp's self-calibrated EXL3 quantizations of GLM-5.3-Flash, one in each branch. turboderp made them with exllamav3 and publishes them in turboderp/GLM-5.3-Flash-exl3. We repacked each one so that mlx-serve can stream the routed experts from the SSD of an M5-class Mac. We did not quantize again and did not change any weight byte, so please credit turboderp when you use them.
On one M5 Max with 128 GB, with greedy decoding and the default settings, the four branches decode at
21.10 to 50.40 tok/s at a 16K prompt, and at 16.40 to
36.45 tok/s at 256K. The default uses no int8 math. The opt-in int8 prefill (int8=4) makes prefill
about a third faster at 16K, but it writes different text. We suggest sc_3.10bpw for a 128 GB Mac.
To run a pack today, you must build mlx-serve main at c1e256ad or later and two plugins from source. No
mlx-serve release has the plugin hooks yet. The table shows our build, not yours. On your build, our measured memory
plan for the experts (the bank fit) does not apply, so the server keeps fewer experts in memory, and decode is likely
lower. We did not measure how much. You cannot make a fit for your own build yet, because we have not published the
tool that writes one. What you can run today vs what we measured lists each difference.
| Branch | Prefill 16K, default | Prefill 16K, int8=4 | Decode 16K | Prefill 256K, default | Prefill 256K, int8=4 | Decode 256K |
|---|---|---|---|---|---|---|
sc_4.10bpw |
1,174 | 1,536 | 21.10 | 970.5 | 1,219 | 16.40 |
sc_3.10bpw |
1,191 | 1,588 | 40.95 | 1,017 | 1,321 | 28.55 |
sc_2.60bpw |
1,194 | 1,588 | 46.90 | 1,022 | 1,334 | 34.30 |
sc_2.10bpw |
1,215 | 1,615 | 50.40 | 1,038 | 1,361 | 36.45 |
Each value is in tok/s, greedy, and the mean of two runs with max_tokens 1,024. Decode is the default's: int8=4
changes only prefill. At a 1K prompt, the time to the first token was 1.24 s
(sc_2.10bpw) to 6.38 s (sc_4.10bpw). Context sweep gives every arm and
size.
Branches
| Branch | bpw | Branch size | KLD (turboderp's) | Source and credit |
|---|---|---|---|---|
sc_4.10bpw |
4.10 | 167.85 GB | 0.0288 | turboderp's sc_4.10bpw |
sc_3.10bpw |
3.10 | 128.07 GB | 0.0574 | turboderp's sc_3.10bpw |
sc_2.60bpw |
2.60 | 109.04 GB | 0.0929 | turboderp's sc_2.60bpw |
sc_2.10bpw |
2.10 | 88.31 GB | 0.1570 | turboderp's sc_2.10bpw |
A branch size is the sum of the branch's files, in GB (10^9 bytes). bpw is turboderp's bits per weight, which leaves out the embeddings and the output head. KLD is turboderp's mean KL divergence against the original model (see Quality). We checked every branch file by file (name, size and sha256) against the pack folder that we uploaded.
Which branch to use
Use sc_3.10bpw on a 128 GB M5 Mac. At a 16K prompt it decodes at 40.95 tok/s (default greedy),
about twice the 21.10 tok/s of sc_4.10bpw. Its KLD is 0.0574, against 0.0929 for
sc_2.60bpw. A smaller branch has a higher KLD and decodes faster, because more of its experts stay in memory and it
reads less from the SSD. At 16K, decode kept 160 of the 288 experts of each layer in memory in
sc_4.10bpw, 221 in sc_3.10bpw, 264 in sc_2.60bpw and all 288 in
sc_2.10bpw. We did not test a Mac with less than 128 GB, so we make no suggestion for one.
Quality
turboderp measured the KL divergence of each quantization against the original model, with exllamav3 on his files. His test text was 28,154 input and 45,470 output tokens that the model wrote itself (his card calls it a self-generated, in-domain trace). His chart marks the measurement's noise floor at 0.01378. His numbers describe the weights in these packs, which hold his bytes in another file layout, but not mlx-serve's kernels. He does not publish perplexity for these branches.
The chart is turboderp's, linked from his card. EXL3-SC is his self-calibrated series, the source of these
branches. EXL3 is his plain series. We did not measure KL divergence through mlx-serve. The
turboexl3_glm53f README gives HumanEval pass@1 with sc_3.10bpw on an earlier combined build
of the plugins (2026-11-11 and 2026-11-12): 152 of 164 by default, and 152 of 164 at int8=4. We did not run HumanEval
on the sweep's build.
Related repositories
| Repo | What it holds |
|---|---|
| turboderp/GLM-5.3-Flash-exl3 | turboderp's EXL3 quantizations, one branch per bitrate. It is the source of every branch here. |
| zai-org/GLM-5.3-Flash | Z.ai's original model. |
OpenSourceWTF/mlx-serve-streaming-pack-converters, glm53f/ |
The conversion scripts that repack his branches into this layout, the pack format (LAYOUT.md) and the exact revisions that built these packs. |
| OpensourceWTF/GLM-5.3-Flash-streaming-repack-exl3-3.05bpw | Our earlier GLM-5.3-Flash pack, deprecated. Its card points here. Its trunk was FP8 and BF16 tensors that we made from another release, and its MTP experts were NVFP4, so turboderp's KLD does not apply to it. It is not a branch here. |
How a pack works
GLM-5.3-Flash is Z.ai's mixture-of-experts (MoE) model. Each of its 42 MoE layers has 288 routed experts (small feed-forward networks), and a router picks 8 of them for each token. The routed experts are most of each pack. The server keeps only part of them in memory and reads the others from the SSD when a token needs them.
A pack is one of turboderp's branches with its experts moved out of the safetensors shards into records. A record
holds one expert's gate, up and down projections, in turboderp's bytes. experts.bin holds the routed experts of MoE
layers 3 to 44. mtp-experts.bin holds the 288 experts of the MTP layer (layer 45), which proposes draft tokens
(multi-token prediction). Manifest files give the offset, size and sha256 of each record. Every other tensor stays in
safetensors shards with turboderp's names, dtypes and bytes. We call these tensors the trunk.
The trunk and the MTP experts stay in memory: 10.7 GB in sc_2.10bpw, 12.4 GB in sc_2.60bpw and sc_3.10bpw, and 14.2 GB in
sc_4.10bpw. The routed experts in memory sit in the bank: a set of memory slots, with the same number of slots in
each layer. When a token routes to an expert that is not in a slot, the server reads its record from the SSD into a
slot. More slots mean fewer SSD reads and faster decode.
Two mlx-serve plugins serve the packs. The quant plugin turboexl3_glm53f runs the EXL3 weights and streams the
experts. The model plugin glm53f runs GLM-5.3-Flash's forward with our speed changes. mlx-serve's built-in
glm5_next code plans the bank.
Every file in a branch
| File | What it is | Source |
|---|---|---|
experts.bin |
The routed experts of MoE layers 3 to 44: 42 layers x 288 records. | turboderp's bytes, in records |
expert-manifest-v2.json, expert-manifest.json |
The record table of experts.bin (offset, size and sha256 of each record, bits per weight of each layer), in the v2 and the older v1 form. Its parity.all_pass records the converter's checks. |
ours |
mtp-experts.bin, mtp-expert-manifest-v2.json, mtp-expert-manifest.json |
The 288 experts of the MTP layer and their record tables. | turboderp's bytes, in records |
model-000NN-of-000MM-nonexpert.safetensors |
The trunk: one shard for each of turboderp's shards NN, with every tensor of it that is not a routed or MTP expert. | turboderp's tensors, in his order |
model.safetensors.index.json |
The index of the trunk shards. | ours |
config.json |
turboderp's file, plus the expert_streaming block that names the manifests and the source commit. The converter also changed its indentation. |
turboderp's, with one block added |
quantization_config.json |
exllamav3's record of each tensor's bits. | turboderp's, unchanged |
tokenizer.json, tokenizer_config.json, chat_template.jinja, generation_config.json, processor_config.json, LICENSE |
The tokenizer, chat template and settings. Each branch carries turboderp's chat template for it. The sc_4.10bpw template (10,950 bytes) is newer than the other three (8,617 bytes). Our test prompts and the usage profiles used the sc_3.10bpw template. |
turboderp's, unchanged |
expert-usage-v1.json, expert-usage-decode-v1.json |
Expert-usage profiles. At load, the slots hold the experts that a profile ranks highest. See Gaps. | ours |
bank-fit/<id>.json (every branch) |
The bank fit, keyed to the build that it was measured on. See Memory. | ours |
TD-REPACK-RECEIPT.json |
The source repo, branch and commit, the bits per weight of each layer, and the sizes and sha256 of the two record files. | ours |
receipts/ |
The converter's check record for each source shard, its plan and state, and the source index and headers. | ours |
manifest.sha256 |
The sha256 of every file in the branch except README.md, .gitattributes, the sweep's assets/ and sweep/, and itself. |
ours |
assets/, sweep/ (every branch) |
The context sweep's charts and data. sc_3.10bpw's assets/family/ also holds this card's sweep charts, because main holds only this card and the license. |
ours |
README.md |
The branch's card. | ours |
.gitattributes |
Hugging Face's storage settings for the repo. | Hugging Face |
Default or int8 prefill
The default uses no int8 math, but it is not byte-identical to stock, which is mlx-serve's built-in glm5_next
forward. Most of its switches compute the same bytes as stock. Three prompt-pass switches of glm53f are
rounding-class: they add values in another order or round them once more. Each of the three passed a GPU bound test
in the plugin's tests. Each also passed an NLL floor test, with a shift below 0.001 nats a token, on the earlier
ddalcu/mlx-serve#1196 build. Because these three switches round differently, the
default text can differ from stock. For stock bytes, set GLM53F_KDA_PROMPT_PERCORE=0, GLM53F_KDA_PROMPT_CHUNK=0
and GLM53F_DSA_PROMPT_ONEP=0.
TURBOEXL3_GLM53F_PF_INT8=4 is opt-in and inexact. At level 4, the routed experts' prefill GEMMs and all 11 trunk
linears at prefill rows run on int8. In the sweep, its greedy text differed from the default greedy text at every
prompt size of every branch. int8=4 changes only prefill. Its decode and hit-rate values differ from the default's because it writes different text, and a different text gets a different MTP hit rate. Set it in the server's environment and restart the server. In the
mlx-serve app, a value saved on the plugin page replaces the environment variable.
TURBOEXL3_GLM53F_PF_INT8=4 <your mlx-serve command>
The Exactness sections of the glm53f README and the
turboexl3_glm53f README define each class, give the test results and list the int8 levels
1 to 4.
Memory: the bank and the bank fit
When the bank opens, glm5_next decides how many slots each layer gets in the prompt phase and in the decode phase.
It plans against the memory ceiling, a limit on the whole Mac's non-file memory, not on the server alone. It plans
in one of two ways.
- With a bank fit that matches the server. A bank fit holds memory peaks that we measured in one greedy request of
our standard 16K test: coding.json, a 16,387-token coding prompt, then 1,024 output tokens. With it,
glm5_nextplans each phase so that the Mac's non-file memory peaks about 1 GiB under the ceiling. - Otherwise, by the bills. A bill is mlx-serve's upper bound on the memory that a phase of one request uses. Because a bill is an upper bound, the bank gets fewer slots than the memory allows.
The fit's prompt is not the sweep's 16K prompt. The sweep's 16K point is a prompt of the code-ctx family, at 16,384 tokens.
Each branch holds its fit as bank-fit/<id>.json, measured again on the sweep's build. The file applies only when its
key equals the server's key in all five parts. The parts are the server build, the machine (chip and memory), the
quant plugin's binary, the model plugin's binary and the pack. It also applies only at its ceiling,
MLX_SERVE_GPU_CEILING_MB=112640. On a server and plugins that you build yourself, the key differs, so the bank fits
by the bills. A --ctx-size above the fit's measured request (17,411 tokens) makes the bank smaller.
The key, the context size, the fit's fields and the log lines
For a development build, the key's build part is a hash of the server's own executable and its MLX build. A release
build (-Dgit-sha) keys on its version, commit and MLX build instead. Each plugin part names the plugin binary's
sha256. So a fit measured on a release applies to every server of that release with the same chip, memory and plugin
binaries. A plugin binary matches only if it is byte-identical. scripts/install.sh builds each plugin on your Mac,
and in our experience two Zig builds of a plugin do not give the same sha256.
For a --ctx-size above the fit's evidence request (17,411 tokens), glm5_next takes the bills' growth from the
evidence length to --ctx-size out of the bank. The server command below uses --ctx-size 32768, and the context
sweep used the prompt size plus 1,024.
Field of sc_3.10bpw's bank-fit/3e1310430989be99.json |
Value | Meaning |
|---|---|---|
format |
mlx-serve-bank-fit-v3 |
The file format. mlx-serve refuses the earlier keyless format v2. |
key.build |
dev 9f21f261d5c6d871 fast kv16 |
Our development build of the server, ReleaseFast, KV cache at 16 bits. |
key.machine |
Apple M5 Max 128GB |
The chip and the memory. |
key.quant_plugin |
turboexl3_glm53f 0.4.0 + sha256 dcc55fb9621b... |
The quant plugin's name, version and binary. |
key.model_plugin |
glm53f 0.1.0 + sha256 9aacac0ede88... |
The model plugin's name, version and binary. |
key.pack |
49d531362a0d... |
A sha256 over the pack's config.json, its bank manifests and the size of experts.bin. |
ceiling_bytes |
118,111,600,640 | The ceiling that the file applies at (MLX_SERVE_GPU_CEILING_MB=112640). |
cell_tokens |
17,408 | mlx-serve's fixed reference request: a 16,384-token prompt and a 1,024-token reply. |
evidence_tokens |
17,411 | The measured request: the 16,387-token coding.json prompt and 1,024 output tokens. |
base_prompt_bytes |
36,864,676,160 | The Mac's peak non-file memory in the prompt phase, less the bank's bytes. |
base_decode_bytes |
26,917,935,808 | The same in the decode phase. |
rest_bytes |
8,247,050,240 | The non-file memory outside the server when the bank opened. |
The file's name is the first 16 hex digits of the key's sha256. Each branch card gives its own file's name and sha256.
At each load, glm5_next logs the server's key:
glm5_next: bank fit key id=<id> build=... | machine=... | quant_plugin=... | model_plugin=... | pack=.... It then
reads bank-fit/<id>.json for that key. If the pack has no such file, it reads a top-level bank-fit.json, if there
is one. If neither exists, it plans by the bills without another line. If it applies a file, it logs
glm5_next: the bank fits by bank-fit/<id>.json (...). If it refuses a file, it logs
glm5_next: <file> not applied (<reason>): the bank fits by the bills. The reasons are:
BankFitOtherBuild,BankFitOtherMachine,BankFitOtherQuantPlugin,BankFitOtherModelPlugin,BankFitOtherPack: that part of the key differs.BankFitNoHostKey: the server sets no key.BankFitUnkeyed: the file has the keyless formatmlx-serve-bank-fit-v2.BankFitV1Rest: the file has the first formatmlx-serve-bank-fit-v1.BankFitOtherCeiling: the server's ceiling is not the file's.BankFitOtherCell: the server's reference request is not the file's.BankFitFormat,BankFitInvalid: the file is not a v3 bank fit, or a field is missing.
The code is bankFit in src/sdk/bases/kda_hybrid/trunk.zig, parseBankFit and readBankFit in
src/sdk/quant/outside_model.zig, and the key in src/engines/mlx/plugin/fit_key.zig (mlx-serve c1e256ad). We
have not published the tool that writes the file.
How to download and run
Requirements
- A Mac with an M5-class GPU, which has neural accelerators (NAX). On any other GPU the plugin refuses to load the pack.
- macOS 26.2 or later, Xcode 26.2 or later with its Metal toolchain, and Homebrew, to build mlx-serve.
- mlx-serve built from source at
mainc1e256ador later. - The plugins
turboexl3_glm53fandglm53f, built from source against the kit of the same mlx-serve checkout. - Free disk for the branch, 88 to 168 GB. Keep the pack on the internal SSD.
- The Hugging Face CLI
hf(pip install -U huggingface_hub).
The software parts are not released yet. This is their state:
| Part | State on 2026-11-19 |
|---|---|
| mlx-serve hooks for the two plugins | ddalcu/mlx-serve#1229 and ddalcu/mlx-serve#1239 are merged into mlx-serve main (c1e256ad has both). No mlx-serve release has them: the latest release, v26.11.2, came before both. |
turboexl3_glm53f, the quant plugin |
CommunityServe/plugins#51, open, head f5ad3b9. |
glm53f, the model plugin |
CommunityServe/plugins#54, draft, head 82a4b15. It waits on CommunityServe/plugins#52, which moves the plugins repo's kit pin to mlx-serve a5e30916. You do not need CommunityServe/plugins#52 when MLX_SERVE_KIT points at your own kit, as in the commands below. |
A streaming pack serves text only, one request at a time, because mlx-serve does not use the vision tower on this path. The packs do not load in exllamav3, transformers, vLLM, SGLang or mlx-lm. For exllamav3, use turboderp's repository.
What you can run today vs what we measured
A build that you make today differs from the build that we measured in four ways:
| Part | What we measured | What you can build today |
|---|---|---|
| Server | mlx-serve a5e30916 plus the hooks of ddalcu/mlx-serve#1239 as they were before its merge | mlx-serve main c1e256ad or later. It holds ddalcu/mlx-serve#1239 as merged and nine other mlx-serve commits. We did not measure it. |
| Plugins | Earlier commits of the two PR branches with our patches. The served code is the PR heads' code, plus READ_AHEAD3 and ADOPT_PIECES on (see Measured stack). |
The PR heads f5ad3b9 and 82a4b15. On mlx-serve main, READ_AHEAD3 does not exist and ADOPT_PIECES is off. |
| Bank fit | Each branch's own fit applied. | Our fits do not apply, so the bank fits by the bills and gets fewer slots. We did not measure that speed. |
--ctx-size |
The prompt size plus 1,024. | 32768 in the command below. A larger value gives the bank fewer slots. With a matching fit, 32768 gives about the slots of the sweep's 32K row (sc_3.10bpw: 190 / 219 prompt / decode, against 193 / 221 at 16K). We did not measure it. |
Download and check a branch
Pass the branch name to --revision. This downloads sc_3.10bpw only:
hf download OpensourceWTF/GLM-5.3-Flash-streaming-repack-exl3 --revision sc_3.10bpw \
--local-dir ~/models/GLM-5.3-Flash-streaming-repack-exl3-sc_3.10bpw
In the pack folder, check every file against manifest.sha256. Each line must end with OK:
cd ~/models/GLM-5.3-Flash-streaming-repack-exl3-sc_3.10bpw
shasum -a 256 -c manifest.sha256
The converter's checks, and a spot check against turboderp's files
The converter's own checks are in the branch. These commands print True, True and the number of source shards
whose receipt passed (22 in sc_3.10bpw):
python3 -c "import json; print(json.load(open('expert-manifest-v2.json'))['parity']['all_pass'])"
python3 -c "import json; print(json.load(open('mtp-expert-manifest-v2.json'))['parity']['all_pass'])"
python3 -c "import glob, json; print(sum(json.load(open(f))['pass'] for f in glob.glob('receipts/model-*.json')))"
To compare the pack with turboderp's files on Hugging Face without the receipts, run
python3 spotcheck_pack.py <pack folder> 8 from the converter. It compares random records and trunk
tensors with his bytes.
Build mlx-serve
These are the "Server only" steps of mlx-serve's docs/building.md, plus the plugin kit:
git clone --recurse-submodules https://github.com/ddalcu/mlx-serve && cd mlx-serve
git checkout c1e256ad # or a later main; c1e256ad is the merge of ddalcu/mlx-serve#1239
git submodule update --init
brew bundle install --file=Brewfile
./scripts/fetch-zig.sh && export PATH="$PWD/.zig-toolchain:$PATH"
./scripts/fetch-llama.sh && ./scripts/build-mlx.sh
zig build -Doptimize=ReleaseFast -Dversion=26.11.3
zig build kit -Dversion=26.11.3
export PATH="$PWD/zig-out/bin:$PATH" # the server, zig-out/bin/mlx-serve
MLXS="$PWD" # this checkout, for the plugins' kit
cd ..
The kit is in zig-out/kit/mlx-serve-plugin-kit-26.11.3/. The plugins repo expects kit version 26.11.3 (its
KIT_VERSION file). -Dversion=26.11.3 gives the server the same version. Without it, the server takes 26.11.1 from
app/Info.plist. If xcrun -sdk macosx metal --version fails, run xcodebuild -downloadComponent MetalToolchain
first.
Build and install the plugins
Build both plugins against the kit of the same mlx-serve checkout, because the server refuses a plugin built against another MLX version. These commands build the two PR branches as they are when you fetch them.
git clone https://github.com/CommunityServe/plugins && cd plugins
git fetch origin pull/51/head:turboexl3_glm53f pull/54/head:glm53f
git checkout turboexl3_glm53f
MLX_SERVE_KIT="$MLXS/zig-out/kit/mlx-serve-plugin-kit-26.11.3" scripts/install.sh turboexl3_glm53f
git checkout glm53f
MLX_SERVE_KIT="$MLXS/zig-out/kit/mlx-serve-plugin-kit-26.11.3" scripts/install.sh glm53f
scripts/install.sh copies the kit into the checkout's .kit/ folder, runs the plugin's tests, builds it and
installs it into ~/.mlx-serve/plugins/. It reuses .kit/ on later runs, so delete .kit/ (rm -rf .kit) after
you build mlx-serve again.
mlx-serve loads no plugin unless ~/.mlx-serve/plugins.json holds "enabled": true. This command writes that file,
and replaces any file that is there:
printf '{"disabled": [], "enabled": true}\n' > ~/.mlx-serve/plugins.json
Start the server
These are the settings of our tests on a 128 GB Mac, except --ctx-size (see
What you can run today vs what we measured):
sudo sysctl iogpu.wired_limit_mb=114688
MLX_SERVE_GPU_CEILING_MB=112640 MLX_SERVE_CACHE_LIMIT=2147483648 \
mlx-serve --model ~/models/GLM-5.3-Flash-streaming-repack-exl3-sc_3.10bpw --serve --host 127.0.0.1 --port 8080 \
--ctx-size 32768 --prefill-chunk 16384 --wired-margin-gib 2 --mtp-min-depth 2 --mtp-max-depth 2
| Setting | What it does |
|---|---|
iogpu.wired_limit_mb=114688 |
The macOS limit on GPU-wired memory (112 GiB). Our test Mac had this value. The setting lasts until a restart. |
MLX_SERVE_GPU_CEILING_MB=112640 |
The memory ceiling, 110 GiB. Set it only on a 128 GB Mac: the server plans with this value in place of the Mac's limit. A bank fit applies only at this value. |
MLX_SERVE_CACHE_LIMIT=2147483648 |
Caps MLX's buffer cache at 2 GiB. |
--ctx-size 32768 |
The longest context. A larger value takes memory from the bank. |
--prefill-chunk 16384 |
The most prompt tokens in one forward pass. |
--wired-margin-gib 2 |
The memory that a plan leaves unused under iogpu.wired_limit_mb: 2 GiB here. The default is 8. |
--mtp-min-depth 2 --mtp-max-depth 2 |
Fixes the MTP draft at 2 tokens a round. |
On a Mac with less memory, leave MLX_SERVE_GPU_CEILING_MB and the sysctl unset. The server then plans with the
Mac's own limit and the bills. We did not test that.
What the load log shows
[diag] GPU ceiling forced to 112640 MB (MLX_SERVE_GPU_CEILING_MB): expected with that setting.[plugins] loaded turboexl3_glm53f (quant, sha256 ...)and[plugins] loaded glm53f (model, sha256 ...): the plugins loaded.glm5_next: bank fit key id=...: the server's bank-fit key. On a build of your own, expect nothe bank fits byline: the bank fits by the bills.glm5_next: streamed experts, ...: N prompt / M decode slot rows per layer: the slots that each layer got.
Troubleshooting
| Log line or error | Cause | What to do |
|---|---|---|
[plugins] refused <path>: plugin was built against another MLX: rebuild it with this release's kit, or ... plugin manifest version is not this host's: rebuild it with this release's kit |
The plugin was built against the kit of another mlx-serve build. | In the plugins checkout, run rm -rf .kit, then the install.sh commands above with MLX_SERVE_KIT set to the kit of the server's checkout. |
[plugins] refused <path>: has no install receipt: install it with `mlx-serve plugin install` , or changed since it was installed (hash differs from its receipt) |
The plugin file was copied in by hand, or rebuilt after install. | Run scripts/install.sh for that plugin again. |
[fetch-kit] cannot download ... from install.sh |
No mlx-serve release has kit 26.11.3 with these hooks, and MLX_SERVE_KIT is not set. |
Set MLX_SERVE_KIT as above. |
glm5_next: a streaming pack (expert_streaming): its experts.bin bank is not streamed by this plugin, or StreamedBankUnclaimed, and no [plugins] loaded turboexl3_glm53f line |
No plugin serves the pack: it is not installed, plugins are off, or the server ran with --no-plugins. |
Check that ~/.mlx-serve/plugins.json holds "enabled": true, and do not pass --no-plugins. |
turboexl3_glm53f: an M5 or newer GPU (NAX) is required |
The GPU is not M5-class. | Use an M5-class Mac. |
turboexl3_glm53f: the expert_streaming block is not the streaming repack's (LAYOUT.md) |
The folder is not one of these packs, for example our old 3.05 pack. | Download a branch of this repo. |
turboexl3_glm53f: the pack lacks its banks (experts.bin, mtp-experts.bin and their manifests), or glm5_next: a streaming pack without its experts.bin bank |
The download is not complete. | Download the branch again and run the checks above. |
turboexl3_glm53f: the codebook is not mul1 |
The pack's EXL3 codebook is not mul1. | Use a branch of this repo. All of them are mul1. |
StreamedRowsDoNotFit |
The bank does not get enough memory for its slots. | Close other apps, lower --ctx-size, or check MLX_SERVE_GPU_CEILING_MB. |
glm5_next: <file> not applied (<reason>): the bank fits by the bills |
See the reasons in Memory. | The server still runs, with fewer slots. |
[turboexl3_glm53f] <switch> is set, but this kit's expert stream (kit.streamed) has no seam for it: the bank is refused (set it to 0) |
TURBOEXL3_GLM53F_ADOPT_PIECES (or PROMPT_KEEP on an older kit) is set to 1, and the kit does not have its stream seam. mlx-serve main has no seam for ADOPT_PIECES. |
Unset the switch. |
Context sweep
The charts and tables below give every arm at all 9 prompt sizes.
Measured stack
We measured on an Apple M5 Max (40-core GPU, 128 GB) on 2026-11-19. The server was mlx-serve a5e30916 (the merge of
ddalcu/mlx-serve#1229) plus the hooks of ddalcu/mlx-serve#1239 as they were before its merge. We built the plugins from earlier commits of the
two PR branches, glm53f 013d661 and turboexl3_glm53f b4d4eed, plus our patches (binary sha256 9aacac0ede88
and dcc55fb9621b). Every switch was at its default, except PF_INT8 in the int8=4 arm. Each branch used its own bank fit, measured again on this
build.
We compared that patched source with the PR heads 82a4b15 and f5ad3b9, file by file. On this Mac, the served code
differs in two switches only, TURBOEXL3_GLM53F_READ_AHEAD3 and TURBOEXL3_GLM53F_ADOPT_PIECES. Both were on in the
sweep, both compute the same bytes, and mlx-serve main has neither. Their measured decode gain was within noise.
Every code difference between the sweep's plugins and the PR heads
ddalcu/mlx-serve#1239 dropped the hooks of both switches before its merge. So the PR heads no longer hold READ_AHEAD3's code,
and ADOPT_PIECES stays off on mlx-serve main; setting it to 1 there refuses the bank. In the last test before the
drop, on the standard 16K test, they moved decode from 41.30 to 41.43 tok/s and from 41.35 to 41.53 tok/s, each
change from one run.
We applied our patches to 013d661 and b4d4eed and compared each .zig and .metal file with 82a4b15 and
f5ad3b9. 013d661 has the same tree as 7442ff4 on the glm53f PR branch, and b4d4eed the same tree as bfb3273
on the turboexl3_glm53f PR branch. Their SHAs differ because the branch commits were signed again.
| Difference | Where | Effect on the 40-core M5 Max |
|---|---|---|
READ_AHEAD3: the switch, the bank's read-ahead pair (aheadScore, aheadIssue) and its call in the forward (read_ahead.zig) |
In the sweep, not in the heads | On in the sweep. Decode within noise (see above). |
A default-on KDA prompt route (GLM53F_KDA_PROMPT_CHUNK or GLM53F_KDA_PROMPT_PERCORE) that cannot serve the GPU is turned off at load, with an info line, unless the user set it. Before, the load failed. |
glm53f commit c9fe66b, in the heads |
None: both routes serve this GPU. |
| Comments, help text and the READMEs | Both plugins | None. |
ADOPT_PIECES has the same code in both. It defaults to 1 only on a kit with its seam, and mlx-serve main has
none. The other switches of the sweep are in the heads and on by default, among them TURBOEXL3_GLM53F_SHARED_HOIST
(f5ad3b9), GLM53F_TRUNK_F16_FREE (82a4b15) and TURBOEXL3_GLM53F_PROMPT_KEEP. mlx-serve c1e256ad has the seams
that SHARED_HOIST and PROMPT_KEEP need. On an older kit, both are 0 and TRUNK_F16_FREE does nothing.
Method
The prompts are our code-ctx family: one code task built to exactly 1,024 to 262,144 tokens, doubling. Each size had three arms:
- default greedy: the default settings (no int8), temperature 0;
- default sampled: the default settings with the model card's sampling, temperature 1.0 and top_p 0.95;
- int8=4 greedy: the default settings plus
TURBOEXL3_GLM53F_PF_INT8=4, temperature 0.
Each arm ran twice at each size, with seeds 1234 and 4321, and each value is the mean of the two runs. The seed has no effect at temperature 0: in both greedy arms, the two runs wrote the same text at every size. Each sampled run wrote its own text, and a different text decodes at a different speed. int8=4 changes only prefill (see Default or int8 prefill).
How each run was done, and what each column measures
Each run was one request on a fresh server. Before each server started, we evicted the pack's files from the page cache. Each request went out after the model had loaded and the chip had cooled below 40 C. The server ran with the settings of Start the server, except --ctx-size, which was the prompt size plus 1,024. max_tokens was 1,024, so each decode figure covers at most the first 1,024 output tokens. 12 of the 216 requests ended earlier, at the model's
own stop.
Prefill and decode tok/s come from the server's own timings. Prefill is prompt tokens over prompt time. Decode is output tokens per second after the first token, with the accepted MTP draft tokens included. TTFT runs from the request to the first token, and wall time from the request to the last token, both on the client's clock. The spec-decode hit rate is the share of MTP draft tokens that the model accepted. Server peak is the server process's peak memory footprint. 1 GB is 10^9 bytes, and 1 GiB is 2^30 bytes.
| Branch | Runs | Bank fit (sha256 starts) | Every run, method and raw data |
|---|---|---|---|
sc_4.10bpw |
54 | 64f64736c506 |
branch card, sweep/ |
sc_3.10bpw |
54 | 4791ae4634b2 |
branch card, sweep/ |
sc_2.60bpw |
54 | 8de2eca96713 |
branch card, sweep/ |
sc_2.10bpw |
54 | a59916583aec |
branch card, sweep/ |
Prefill (tok/s), every size
| Prompt | sc_4.10bpw default greedy |
sc_4.10bpw default sampled |
sc_4.10bpw int8=4 greedy |
sc_3.10bpw default greedy |
sc_3.10bpw default sampled |
sc_3.10bpw int8=4 greedy |
sc_2.60bpw default greedy |
sc_2.60bpw default sampled |
sc_2.60bpw int8=4 greedy |
sc_2.10bpw default greedy |
sc_2.10bpw default sampled |
sc_2.10bpw int8=4 greedy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1K | 160.6 | 161.7 | 152.2 | 300.6 | 303.0 | 258.7 | 413.7 | 418.2 | 426.8 | 828.5 | 819.9 | 692.7 |
| 2K | 319.1 | 319.0 | 318.6 | 549.1 | 547.6 | 574.2 | 719.9 | 716.8 | 770.2 | 1,038 | 1,041 | 1,328 |
| 4K | 597.0 | 597.2 | 610.4 | 911.8 | 911.3 | 976.3 | 1,109 | 1,111 | 1,265 | 1,143 | 1,143 | 1,458 |
| 8K | 1,030 | 1,031 | 1,114 | 1,201 | 1,204 | 1,470 | 1,204 | 1,203 | 1,578 | 1,213 | 1,212 | 1,598 |
| 16K | 1,174 | 1,174 | 1,536 | 1,191 | 1,190 | 1,588 | 1,194 | 1,195 | 1,588 | 1,215 | 1,205 | 1,615 |
| 32K | 1,135 | 1,143 | 1,514 | 1,161 | 1,161 | 1,557 | 1,158 | 1,161 | 1,562 | 1,172 | 1,170 | 1,582 |
| 64K | 1,102 | 1,102 | 1,449 | 1,122 | 1,122 | 1,501 | 1,122 | 1,123 | 1,504 | 1,135 | 1,135 | 1,526 |
| 128K | 1,049 | 1,049 | 1,357 | 1,080 | 1,080 | 1,430 | 1,085 | 1,085 | 1,440 | 1,094 | 1,093 | 1,456 |
| 256K | 970.5 | 970.7 | 1,219 | 1,017 | 1,018 | 1,321 | 1,022 | 1,022 | 1,334 | 1,038 | 1,038 | 1,361 |
Decode (tok/s), every size
| Prompt | sc_4.10bpw default greedy |
sc_4.10bpw default sampled |
sc_4.10bpw int8=4 greedy |
sc_3.10bpw default greedy |
sc_3.10bpw default sampled |
sc_3.10bpw int8=4 greedy |
sc_2.60bpw default greedy |
sc_2.60bpw default sampled |
sc_2.60bpw int8=4 greedy |
sc_2.10bpw default greedy |
sc_2.10bpw default sampled |
sc_2.10bpw int8=4 greedy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1K | 20.00 | 21.05 | 20.10 | 44.90 | 42.75 | 44.15 | 50.30 | 49.55 | 51.45 | 52.40 | 53.45 | 53.00 |
| 2K | 20.60 | 20.65 | 19.70 | 43.20 | 41.70 | 42.95 | 48.40 | 45.85 | 49.45 | 51.45 | 52.20 | 50.45 |
| 4K | 21.90 | 20.65 | 21.55 | 43.80 | 42.90 | 42.30 | 48.55 | 45.65 | 45.25 | 51.15 | 51.65 | 53.40 |
| 8K | 20.40 | 20.65 | 20.75 | 42.25 | 43.25 | 41.95 | 46.95 | 47.10 | 48.45 | 50.05 | 49.85 | 51.70 |
| 16K | 21.10 | 20.85 | 21.80 | 40.95 | 41.70 | 40.65 | 46.90 | 46.25 | 46.80 | 50.40 | 46.75 | 49.25 |
| 32K | 21.10 | 18.05 | 19.90 | 36.70 | 38.75 | 38.85 | 43.65 | 38.15 | 38.75 | 45.60 | 38.05 | 43.10 |
| 64K | 20.25 | 21.25 | 19.80 | 39.25 | 34.70 | 37.90 | 40.15 | 40.50 | 40.15 | 46.65 | 42.05 | 44.00 |
| 128K | 19.70 | 18.15 | 19.60 | 35.15 | 34.30 | 35.55 | 38.75 | 38.75 | 39.35 | 39.30 | 38.85 | 49.80 |
| 256K | 16.40 | 14.85 | 17.45 | 28.55 | 29.40 | 27.75 | 34.30 | 33.50 | 38.35 | 36.45 | 39.55 | 42.85 |
TTFT (s), every size
| Prompt | sc_4.10bpw default greedy |
sc_4.10bpw default sampled |
sc_4.10bpw int8=4 greedy |
sc_3.10bpw default greedy |
sc_3.10bpw default sampled |
sc_3.10bpw int8=4 greedy |
sc_2.60bpw default greedy |
sc_2.60bpw default sampled |
sc_2.60bpw int8=4 greedy |
sc_2.10bpw default greedy |
sc_2.10bpw default sampled |
sc_2.10bpw int8=4 greedy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1K | 6.38 | 6.34 | 6.76 | 3.42 | 3.38 | 4.11 | 2.48 | 2.46 | 2.41 | 1.24 | 1.26 | 1.66 |
| 2K | 6.43 | 6.43 | 6.43 | 3.74 | 3.75 | 3.58 | 2.85 | 2.86 | 2.67 | 1.98 | 1.98 | 1.55 |
| 4K | 6.87 | 6.87 | 6.72 | 4.50 | 4.50 | 4.20 | 3.70 | 3.69 | 3.25 | 3.59 | 3.60 | 2.82 |
| 8K | 7.96 | 7.95 | 7.37 | 6.83 | 6.82 | 5.58 | 6.82 | 6.82 | 5.20 | 6.76 | 6.76 | 5.13 |
| 16K | 13.96 | 13.96 | 10.68 | 13.77 | 13.79 | 10.32 | 13.73 | 13.72 | 10.33 | 13.49 | 13.61 | 10.15 |
| 32K | 28.88 | 28.68 | 21.66 | 28.25 | 28.25 | 21.05 | 28.32 | 28.25 | 21.00 | 27.96 | 28.02 | 20.73 |
| 64K | 59.51 | 59.50 | 45.25 | 58.41 | 58.44 | 43.70 | 58.42 | 58.39 | 43.59 | 57.78 | 57.77 | 42.98 |
| 128K | 125.02 | 124.99 | 96.62 | 121.36 | 121.38 | 91.67 | 120.84 | 120.80 | 91.08 | 119.88 | 119.90 | 90.08 |
| 256K | 270.17 | 270.10 | 215.03 | 257.77 | 257.64 | 198.56 | 256.56 | 256.52 | 196.54 | 252.56 | 252.72 | 192.67 |
Wall time (s), every size
| Prompt | sc_4.10bpw default greedy |
sc_4.10bpw default sampled |
sc_4.10bpw int8=4 greedy |
sc_3.10bpw default greedy |
sc_3.10bpw default sampled |
sc_3.10bpw int8=4 greedy |
sc_2.60bpw default greedy |
sc_2.60bpw default sampled |
sc_2.60bpw int8=4 greedy |
sc_2.10bpw default greedy |
sc_2.10bpw default sampled |
sc_2.10bpw int8=4 greedy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1K | 57.76 | 55.09 | 57.84 | 26.30 | 27.42 | 27.38 | 22.89 | 23.18 | 22.37 | 20.84 | 20.48 | 21.03 |
| 2K | 56.12 | 56.16 | 58.70 | 27.52 | 28.35 | 27.48 | 24.06 | 25.30 | 23.45 | 21.95 | 21.66 | 21.93 |
| 4K | 53.64 | 56.54 | 54.30 | 27.93 | 28.44 | 28.48 | 24.84 | 26.18 | 25.95 | 23.66 | 23.50 | 22.05 |
| 8K | 58.25 | 57.82 | 56.80 | 31.11 | 30.57 | 30.05 | 28.68 | 28.64 | 26.38 | 27.28 | 27.36 | 25.00 |
| 16K | 62.73 | 63.28 | 57.76 | 38.84 | 38.42 | 35.59 | 35.62 | 35.97 | 32.25 | 33.87 | 35.58 | 31.02 |
| 32K | 77.58 | 85.64 | 73.34 | 56.05 | 54.85 | 47.46 | 51.84 | 53.12 | 47.48 | 50.46 | 47.12 | 44.52 |
| 64K | 110.14 | 107.86 | 97.14 | 84.54 | 78.92 | 70.76 | 83.98 | 83.73 | 69.15 | 79.80 | 82.20 | 66.30 |
| 128K | 177.16 | 177.56 | 148.91 | 146.41 | 151.38 | 117.49 | 147.34 | 147.32 | 117.17 | 146.00 | 146.50 | 110.73 |
| 256K | 332.87 | 339.65 | 273.70 | 293.71 | 292.13 | 235.56 | 286.50 | 287.19 | 223.33 | 280.74 | 279.25 | 216.65 |
Spec-decode hit rate (%), every size
| Prompt | sc_4.10bpw default greedy |
sc_4.10bpw default sampled |
sc_4.10bpw int8=4 greedy |
sc_3.10bpw default greedy |
sc_3.10bpw default sampled |
sc_3.10bpw int8=4 greedy |
sc_2.60bpw default greedy |
sc_2.60bpw default sampled |
sc_2.60bpw int8=4 greedy |
sc_2.10bpw default greedy |
sc_2.10bpw default sampled |
sc_2.10bpw int8=4 greedy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1K | 81.8 | 76.5 | 81.2 | 80.3 | 75.6 | 79.7 | 77.9 | 76.8 | 78.8 | 70.6 | 73.5 | 71.6 |
| 2K | 73.1 | 75.4 | 82.8 | 77.7 | 74.2 | 77.2 | 76.1 | 70.6 | 78.7 | 71.4 | 74.4 | 69.0 |
| 4K | 78.1 | 81.3 | 76.3 | 81.5 | 80.2 | 73.6 | 77.5 | 71.3 | 68.3 | 72.4 | 73.3 | 76.3 |
| 8K | 79.9 | 71.8 | 76.4 | 73.8 | 79.0 | 75.2 | 74.6 | 76.2 | 79.0 | 71.8 | 73.1 | 73.2 |
| 16K | 80.6 | 79.2 | 77.5 | 69.1 | 77.3 | 69.1 | 76.5 | 76.2 | 75.4 | 77.0 | 68.2 | 71.2 |
| 32K | 71.1 | 64.1 | 73.2 | 68.4 | 70.8 | 71.9 | 75.6 | 60.8 | 61.5 | 73.2 | 55.1 | 66.3 |
| 64K | 68.2 | 69.8 | 64.7 | 72.3 | 62.1 | 68.7 | 65.1 | 66.3 | 65.3 | 73.2 | 60.9 | 68.0 |
| 128K | 65.6 | 67.7 | 72.4 | 67.3 | 67.5 | 67.9 | 67.7 | 69.3 | 70.2 | 60.6 | 58.4 | 85.3 |
| 256K | 71.8 | 64.2 | 69.9 | 70.5 | 67.8 | 67.2 | 68.3 | 62.6 | 87.0 | 64.5 | 76.5 | 81.8 |
Server peak, mean of 2 runs (GB), every size
| Prompt | sc_4.10bpw default greedy |
sc_4.10bpw default sampled |
sc_4.10bpw int8=4 greedy |
sc_3.10bpw default greedy |
sc_3.10bpw default sampled |
sc_3.10bpw int8=4 greedy |
sc_2.60bpw default greedy |
sc_2.60bpw default sampled |
sc_2.60bpw int8=4 greedy |
sc_2.10bpw default greedy |
sc_2.10bpw default sampled |
sc_2.10bpw int8=4 greedy |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1K | 104.3 | 104.0 | 104.0 | 106.2 | 105.8 | 105.8 | 105.7 | 105.3 | 105.3 | 94.5 | 94.5 | 94.4 |
| 2K | 104.1 | 104.1 | 104.1 | 105.8 | 105.8 | 105.6 | 105.2 | 105.1 | 105.0 | 95.5 | 95.3 | 94.9 |
| 4K | 104.2 | 104.2 | 104.1 | 105.9 | 105.9 | 105.7 | 104.9 | 105.1 | 105.1 | 96.4 | 96.4 | 95.9 |
| 8K | 104.0 | 104.2 | 104.3 | 106.0 | 105.8 | 105.9 | 105.1 | 105.2 | 105.1 | 98.7 | 98.8 | 98.2 |
| 16K | 104.4 | 104.4 | 104.4 | 106.2 | 106.2 | 106.0 | 105.3 | 105.3 | 105.3 | 103.2 | 103.2 | 102.9 |
| 32K | 103.8 | 103.8 | 103.8 | 107.2 | 107.2 | 106.7 | 104.9 | 104.9 | 104.9 | 105.0 | 105.0 | 104.5 |
| 64K | 103.2 | 103.2 | 102.8 | 105.6 | 105.7 | 105.2 | 104.1 | 104.1 | 104.1 | 103.2 | 103.7 | 102.5 |
| 128K | 102.1 | 102.3 | 102.1 | 103.8 | 104.0 | 103.9 | 103.5 | 103.5 | 103.2 | 99.5 | 99.1 | 98.3 |
| 256K | 99.5 | 99.5 | 99.5 | 101.3 | 101.5 | 101.4 | 100.8 | 100.2 | 100.1 | 100.2 | 100.1 | 100.3 |
Memory per prompt size
The bank takes the room under the ceiling, so a memory peak shows the server's plan, not the least memory that a pack needs. In all 216 runs, the whole Mac's non-file memory stayed under the ceiling of 110 GiB (118.1 GB). The bank holds fewer slots as the prompt grows.
Memory tables, every size (all arms)
The whole Mac's highest sample was 115.7 GB, and the smallest margin under the ceiling was
2.28 GiB, both for sc_3.10bpw at 32K. The bank fit aims for a margin of about 1 GiB at
its own request length. At 16K, near that length, the smallest margin was 3.18 to
4.79 GiB. We did not measure why the margin is larger than the aim. In sc_2.10bpw, the decode
bank held all 288 experts of each layer up to 128K.
"Whole machine" is the Mac's non-file memory (app and GPU memory, not the file cache), sampled once a second. For a longer context, mlx-serve takes the bills' growth past the fit's length out of the bank.
Whole machine, highest of the size's runs (GB)
| Prompt | sc_4.10bpw |
sc_3.10bpw |
sc_2.60bpw |
sc_2.10bpw |
|---|---|---|---|---|
| 1K | 114.6 | 114.7 | 114.3 | 104.2 |
| 2K | 114.2 | 114.4 | 114.2 | 105.0 |
| 4K | 114.4 | 114.4 | 114.2 | 106.0 |
| 8K | 114.3 | 114.5 | 114.3 | 108.4 |
| 16K | 114.5 | 114.7 | 114.3 | 113.0 |
| 32K | 113.9 | 115.7 | 113.9 | 114.7 |
| 64K | 113.3 | 114.2 | 113.3 | 114.0 |
| 128K | 111.8 | 112.1 | 111.8 | 109.0 |
| 256K | 109.7 | 110.1 | 110.2 | 109.9 |
Under the ceiling, lowest of the size's runs (GiB)
| Prompt | sc_4.10bpw |
sc_3.10bpw |
sc_2.60bpw |
sc_2.10bpw |
|---|---|---|---|---|
| 1K | 3.30 | 3.21 | 3.55 | 12.92 |
| 2K | 3.62 | 3.47 | 3.65 | 12.19 |
| 4K | 3.50 | 3.43 | 3.66 | 11.28 |
| 8K | 3.52 | 3.36 | 3.53 | 9.05 |
| 16K | 3.34 | 3.18 | 3.54 | 4.79 |
| 32K | 3.90 | 2.28 | 3.91 | 3.18 |
| 64K | 4.45 | 3.63 | 4.48 | 3.82 |
| 128K | 5.86 | 5.60 | 5.91 | 8.46 |
| 256K | 7.80 | 7.46 | 7.39 | 7.65 |
Server peak, highest of the size's runs (GB)
| Prompt | sc_4.10bpw |
sc_3.10bpw |
sc_2.60bpw |
sc_2.10bpw |
|---|---|---|---|---|
| 1K | 104.6 | 106.5 | 105.9 | 94.5 |
| 2K | 104.1 | 105.9 | 105.4 | 95.6 |
| 4K | 104.2 | 105.9 | 105.1 | 96.4 |
| 8K | 104.3 | 106.0 | 105.2 | 98.8 |
| 16K | 104.4 | 106.3 | 105.3 | 103.2 |
| 32K | 103.8 | 107.3 | 104.9 | 105.1 |
| 64K | 103.2 | 105.8 | 104.1 | 104.1 |
| 128K | 102.5 | 104.2 | 103.6 | 99.5 |
| 256K | 99.6 | 101.5 | 100.8 | 100.6 |
Slots a layer, prompt / decode (the fewest of the size's runs)
| Prompt | sc_4.10bpw |
sc_3.10bpw |
sc_2.60bpw |
sc_2.10bpw |
|---|---|---|---|---|
| 1K | 126 / 160 | 194 / 222 | 214 / 265 | 288 / 288 |
| 2K | 126 / 160 | 193 / 221 | 213 / 264 | 288 / 288 |
| 4K | 126 / 160 | 193 / 221 | 213 / 263 | 288 / 288 |
| 8K | 126 / 159 | 193 / 221 | 213 / 264 | 288 / 288 |
| 16K | 126 / 160 | 193 / 221 | 213 / 264 | 288 / 288 |
| 32K | 123 / 158 | 190 / 219 | 209 / 261 | 284 / 288 |
| 64K | 117 / 154 | 181 / 214 | 199 / 256 | 272 / 288 |
| 128K | 105 / 148 | 166 / 206 | 180 / 245 | 248 / 288 |
| 256K | 82 / 135 | 135 / 189 | 141 / 223 | 202 / 286 |
Gaps
| Gap | What closes it |
|---|---|
| No mlx-serve release has the plugin hooks, and both plugins are open pull requests. You must build from source. | The plugins merge, and an mlx-serve release ships a plugin kit with the hooks. |
| Our bank fits apply only on our build. On a build of your own, the bank fits by the bills, and we did not measure that speed. | A bank fit measured on a release build, plus published plugin binaries or byte-reproducible plugin builds, because the key holds each plugin binary's sha256. |
| You cannot make a bank fit for your own build. | We publish the tool that writes a fit. |
We did not run the sweep on the PR heads or on mlx-serve c1e256ad. The sweep's build had READ_AHEAD3 and ADOPT_PIECES on, and mlx-serve main has neither (see Measured stack). |
A sweep on mlx-serve main with the merged plugins. |
| We measured the four branches on one Apple M5 Max with 128 GB only. | Measurements on other M5-class Macs. |
| We did not measure KL divergence through mlx-serve. HumanEval ran only on an earlier build. | A KLD test through mlx-serve, and HumanEval on the shipped build. |
Smaller gaps
| Gap | What closes it |
|---|---|
| Every run was the first request on a fresh server with a cold page cache. We did not measure a warm server, a second request or a prefix-cache hit. | Runs on a warm server. |
| Each decode figure covers at most the first 1,024 output tokens. | Runs with a larger max_tokens. |
The card's server command (--ctx-size 32768) was not measured. |
A run with that command. |
| int8 levels 1 to 3 were not measured on the sweep's build. | A sweep at those levels. |
| The speed with stock bytes (the three rounding-class switches at 0) was not measured. | An A/B of that setting. |
| The NLL floor test of the three rounding-class switches ran on ddalcu/mlx-serve#1196, not on the plugin. | The NLL floor test on the plugin. |
| A streaming pack serves text only, and one request at a time. | Vision and batching on mlx-serve's streaming path. |
The quant plugin alone also lets mlx-serve's built-in glm5_next forward serve the packs. We did not run the sweep on that path. |
A sweep without glm53f. |
| We recorded each expert-usage profile on one request of coding.json. That prompt can start with more of its experts in memory than another prompt does. We did not measure how large this effect is. | A measurement on other prompts. |
| The context sweep ran one request at a time, on one code task built to each size. | Sweeps on other tasks. |
| 12 of the sweep's 216 requests ended before 1,024 output tokens, at the model's own stop. | Not a fault. Each branch card gives each run's output tokens. |
In the default arms, prefill tok/s is lower below 8K (below 16K for sc_4.10bpw). We did not measure the cause. |
A profile of short prompts. |
| At 16K, the memory margin under the ceiling was larger than the bank fit's aim of about 1 GiB. We did not measure why. | A study of the fit's margin. |
| On a Mac with less than 128 GB, we did not test the server. | Tests on such a Mac. |
| We did not test the packs on external drives. | Tests on an external drive. |
How the packs were made
Our converter copies turboderp's bytes into the pack layout and checks every record and tensor against his files.
How the converter works and checks
Our converter, convert_glm_sc.py in the converter repository, reads each of turboderp's
branches at a pinned commit from Hugging Face, with HTTP range requests. It writes the routed and MTP experts as
records, and every other tensor into new shards in the source order. It changes no weight byte.
For each source shard, the converter compares the sha256 of the streamed bytes with the shard's LFS hash at the
pinned commit. After it writes the pack, it reads every record and every trunk tensor back from disk. It compares each
with the sha256 of its source tensor. It writes a receipt for each shard, and it finishes a pack only when every
receipt passes. In all four packs, expert-manifest-v2.json and mtp-expert-manifest-v2.json record
parity.all_pass: true.
Credits
- turboderp made exllamav3, the EXL3 format and the self-calibrated quantizations that these packs contain: turboderp/GLM-5.3-Flash-exl3. EXL3 is based on QTIP by Cornell RelaxML.
- Z.ai made GLM-5.3-Flash: zai-org/GLM-5.3-Flash.
- ddalcu maintains mlx-serve, which has the
glm5_nextmodel. - turboexl3 has the Metal kernels that the plugin runs for the EXL3 weights.
License
MIT, as turboderp's repository. The LICENSE file is turboderp's, unchanged, and every branch carries it. The code
(mlx-serve, the plugin, turboexl3 and exllamav3) has its own licenses.
Terms
| Term | Meaning |
|---|---|
| bpw | Bits per weight. turboderp's count leaves out the embeddings and the output head. |
| SC | Self-calibrated: turboderp's name for these four branches. In them the bits per weight differ by layer and tensor type. |
| Routed expert | One of the 288 small networks in a MoE layer. The router picks 8 for each token. |
| Record | The bytes of one expert of one layer in experts.bin or mtp-experts.bin. |
| Trunk | Every tensor that is not a routed or MTP expert: attention, shared experts, dense layers, embeddings, LM head, vision tower. |
| Slot | A place in memory that holds one expert record. The log calls it a slot row. |
| Bank | The set of slots for the routed experts. |
| Bill | mlx-serve's upper bound on the memory that a phase of one request uses. |
| Bank fit | The slot plan from measured peaks (bank-fit/<id>.json) in place of the bills. |
| Bank-fit key | The build, machine, plugin binaries and pack that a bank fit was measured on. |
| Ceiling | The limit on the whole Mac's non-file memory that the server plans against (MLX_SERVE_GPU_CEILING_MB). |
| Quant plugin, model plugin | turboexl3_glm53f runs the EXL3 weights and streams the experts. glm53f runs the model's forward with our speed changes. |
| Kit | The plugin kit that zig build kit makes in an mlx-serve checkout. A plugin builds against it. |
| Switch | An environment variable that turns one speed change of a plugin on or off. |
| MTP | Multi-token prediction: the model's extra layer that proposes draft tokens, which the main model then checks. |
| Prefill | Processing the prompt, before the first output token. |
| TTFT | Time to first token, from the request to the first output token, on a loaded model. |
| Wall time | The time from the request to the last output token, on the client's clock. |
| KLD | Mean KL divergence between the original model's and the quantized model's next-token distributions. Lower is closer. |
| Context sweep | The same test at a series of prompt sizes, here 1K to 256K tokens. |
| Default (arm) | The plugins' default settings: no int8, and three rounding-class switches on. Not byte-identical to stock. |
| Sampled (arm) | The model card's sampling settings: temperature 1.0 and top_p 0.95. |
| Rounding-class | A switch that rounds differently from stock, uses no int8, and passed the NLL floor test. |
| int8=4 | TURBOEXL3_GLM53F_PF_INT8=4: inexact int8 prefill, opt-in. |
| NLL | Negative log-likelihood, in nats a token: how well the model predicts each real next token. |
| Whole machine | The Mac's non-file memory (app and GPU memory, not the file cache): what the ceiling bounds. It includes the server and every other process. |




