Schiltmans/Qwen3.8-Flash-Next-Coder-cut240

Qwen3.8-Flash-Next Coder cut240 for MLX on 48 GB Macs

ISTA-DASLab's IQ1_M Coder GGUF of Qwen3.8-Flash-Next (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF at 5348543e) with 768 of its 12,288 routed experts removed, served through MLX as an OpenAI-compatible endpoint for coding agents. Each layer keeps 185 to 255 of 256 routed experts, 240 on average; kept tensors are byte for byte as published. This repository is the model folder and its runner in one; the runtime is a TensorFold fork that reads the GGUF tensors in place through mlx-kquant. The fork builds on TensorFold 0.3.4 (MIT) and carries its Python engine, which upstream later replaced with a native one.

What the runner does

• Drafting is on by default: ggml-org's MTP head for Qwen3.8-Flash-Next (routed experts cut from 512 to 256 at load) proposes up to 3 tokens a round, a second proposer copies spans from the prompt, and the model checks each drafted token against its own sample. Decode throughput was 1.11 to 1.50 times that of drafting off, depending on the kind of request.
• When a client disconnects, the server stops generating for it.
• Agent session-title requests run in the background, uninterrupted once they have sent a token, so a title reply is never joined from two runs.
• SSD streaming is optional (STREAM=on): each layer keeps a fixed number of routed experts in memory and reads the others from the SSD as tokens need them. It gave the same tokens as the full-memory server at every slot count tested (16 to 200), with 12-17 GB of whole-machine wired memory instead of 32 GB. It decoded at 2.6-4.8 tok/s with other apps open; the full-memory server decoded the same requests at 23.7-25.3 tok/s on an idle Mac (SSD streaming, below).

DRAFTING=off ./serve.sh turns drafting off and keeps the disconnect and title behaviour. Some evidence/ pages were measured with drafting off and without the disconnect and title behaviour; each says which.

Use

Requires an Apple Silicon Mac with 48 GB (for 24 GB and 32 GB Macs, see below) and macOS 26 or later (measured on an M4 Pro), about 62 GB of free disk (57 GB model files, 2.2 GB head, about 1 GB environment), git and uv (the download below uses uvx; install.sh also works with python3.11). Close memory-heavy apps first: serve.sh will not start if macOS reports memory pressure or has less than an estimated 36 GB free or held by cached files. This is a startup check; it does not guarantee that memory will suffice later.

uvx --from huggingface_hub hf download Schiltmans/Qwen3.8-Flash-Next-Coder-cut240 --local-dir Flash-Next-Coder-cut240   # ~57 GB
cd Flash-Next-Coder-cut240
chmod +x install.sh setup.sh serve.sh   # the Hub does not keep the executable bit
./install.sh     # .venv with the pinned packages and TensorFold
./setup.sh       # downloads the MTP head at a pinned revision, checks every large file
./serve.sh       # http://127.0.0.1:8088/v1, model "local"

By default the server answers only requests from this Mac: it checks each request's Host and Origin headers. To serve other machines, set HOST, an API key (TENSORFOLD_API_KEY) and the host names clients use (TENSORFOLD_ALLOWED_HOSTS); see docs/api.md in the fork.

setup.sh stops on any checksum mismatch; opencode.example.jsonc holds settings tested with opencode 1.18.26 (used for the agentic results) and 2.0.32. Sampling defaults to temperature 1.0, top-p 0.95 and top-k 20, and request values take precedence; every request gets xhigh reasoning.

KV_PAGES=on CONTEXT=65536 ./serve.sh serves a 65536-token window, with the prompt cache's keys and values in pages that entries sharing a prefix share; close other apps first (Results). serve.sh then asks for an estimated 38 GB of free or cached memory at startup. It refuses a window above 32768 without KV_PAGES=on, and any window above 65536.

SSD streaming (STREAM=on)

STREAM=on ./serve.sh keeps 16 routed experts a layer in the server's own buffers and reads the others from experts-cut240.gguf on the SSD as tokens route to them. trunk.gguf (about 4.5 GB) and ngram.gguf (about 28.8 GB) stay memory-mapped, as with streaming off, and rely on the macOS file cache; the forward pass looks up n-gram embedding rows in ngram.gguf for every token in this mode too. The 5.59 GB in Results is the server process's footprint, not the memory the Mac needs. Streaming was tested only on a 48 GB Mac.

Streaming leaves the replies unchanged: every slot count tested (16 to 200) gave the same tokens as the full-memory server. Speed depends on the slot count and on what else the Mac is doing (Choosing settings, below); with other apps open, reading a 23K-31K-token prompt took 2.3 to 4.3 minutes.

With STREAM=on:

• drafting is off; DRAFTING=on exits with an error, as does KV_PAGES=on;
• CONTEXT defaults to 32768, and a larger value exits with an error;
• one request is served at a time; the others wait;
• the prompt cache is off; STREAM_CACHE_GIB sets 0 to 6 GiB (Results: with 0.5 GiB the replies differ from the cache-off reference and match a full-memory server with the same cache);
• STREAM_SLOTS (16 to 200) sets how many experts a layer keeps in memory (above; evidence/streaming.md);
• serve.sh refuses to start with less than an estimated 10 GB free or cached, plus about 0.1 GB per slot above 16 and the prompt cache's size. This check is a guard against starting under pressure, not a minimum-RAM figure; MIN_FREE_GB=0 skips it.

Upstream TensorFold's Python engine already streamed MLX 4-bit experts from the SSD; this mode reads this model's GGUF expert file. Details: evidence/streaming.md.

Choosing settings

Measured on one 48 GB M4 Pro, with other apps open unless noted. Other Macs were not tested, so treat these as starting points.

• STREAM=on: the least memory, 11.8-14.0 GB wired, at 2.6-3.3 tok/s.
• STREAM=on STREAM_SLOTS=64: 17.2 GB at 4.5-4.8 tok/s; 24 to 48 slots fall in between. Right after a full-memory run on an otherwise idle Mac, 64 slots reached 12.2-12.7 tok/s, and 128 to 200 slots added memory without a consistent gain.
• STREAM_CACHE_GIB=0.5 with streaming, for agents: a 27,110-token follow-up turn reused 27,091 cached tokens and was read in 2.7 s instead of 158.8 s (64 slots).
• Streaming off, when the whole model fits: 32.0 GB and 23.7-25.3 tok/s on the same requests (otherwise idle Mac, drafting and prompt cache off). With the defaults, median peaks of 37.1 GB at a full 32768-token window and 33.4-34.8 tok/s in the agentic passes.
• A longer window: KV_PAGES=on CONTEXT=65536 with streaming off, a median peak of 37.2 GB and 24.2-26.8 tok/s with ~61K-token prompts. The model's config allows 262144 tokens, but serve.sh refuses windows above 65536 (32768 with streaming), because larger ones were not tested.

Memory figures are whole-machine wired highs, macOS included; in every mode the memory-mapped trunk and n-gram files also use the file cache. The serve.sh start check guards against starting under memory pressure; it is not a measured minimum (MIN_FREE_GB=0 skips it).

24 GB and 32 GB Macs

With STREAM=on STREAM_CACHE_GIB=0.5, STREAM_SLOTS sets the server's memory footprint. The table gives the server process's phys_footprint with ~29K-31K-token prompts, measured on the 48 GB M4 Pro with browsers and Docker closed and an editor and small utilities open. No 24 GB or 32 GB Mac was tested. The file cache here was partly warm; a Mac with less memory reads more of the expert file from the SSD and will decode slower, so treat these settings as starting points. The serve.sh start check asks for an estimated 15 GB (48 slots) or 16 GB (64) free or cached; on a 32 GB Mac, close other apps first or skip the check with MIN_FREE_GB=0. Details: evidence/streaming.md.

Mac	STREAM_SLOTS	phys_footprint, highest	decode, tok/s
24 GB	16	6.3-7.0 GB	4.0 at 16K tokens
32 GB, other apps open	48	9.42 GB	5.75 at 28.8K tokens
32 GB, few apps open	64	10.5-11.3 GB	6.8 at 16K tokens

Results

M4 Pro, 48 GB, browsers closed unless a line says otherwise; methods and every run are in evidence/.

Same output with and without drafts

On one drafting server, 15 cases (3 short prompts; 12 of 2K-26K tokens, two seeds, a second turn) each gave the same token ids drafted and one token at a time ("draft": false). A DRAFTING=off server uses other matrix kernels for its one-token steps, which round differently: none of 24 seeded requests gave the same complete reply on both servers, though each server reproduced all 24 of its own in a second session. Both sample the same model (evidence/identity.md). With STREAM unset, this runner reproduced the replies recorded there: 15 of 15 drafting cases gave the same token ids, drafted and one token at a time, and 3 of 3 DRAFTING=off requests gave the same replies.

Decode throughput by kind of request

Drafting on and off, in tokens per second (evidence/drafting-speed.md). The two configurations generate different sequences, so the table compares whole configurations, not the same tokens. Accepted is the share of drafted tokens kept.

kind	on	off	ratio	accepted
chat	34.0	28.6	1.19	0.55
prose	33.6	28.6	1.18	0.57
code	33.5	28.6	1.17	0.54
Dutch / German	32.0	28.7	1.11	0.47
math	43.5	28.9	1.50	0.73
random-looking output	37.6	28.8	1.31	0.61

Agentic coding

24 Exercism Python exercises a pass, opencode, 600 s each (evidence/agentic-drafting.md); swap did not grow in any pass. Drafting on: 90 of 96 solved over 4 passes (21-24 a pass), decode 33.4-34.8 tok/s. Drafting off: 43 of 48 over 2 passes, decode 25.8-26.1 tok/s. Solved per hour was 30.2 on and 22.9 off, but the passes of one setting spread widely (25.9-36.9 an hour with drafting on) and were not designed to compare it. No agentic speed-up is claimed.

Memory

Highest whole-machine wired memory sample (every 0.5 s), macOS's own ~2.4 GB included, while a ~30.8K-token prompt fills the 32768-token window and four facts are recovered: median of 5 runs 37.05 GB with drafting on (36.50-37.37) and 35.62 GB off (35.05-36.29); all four facts and no swap-outs in every run (evidence/window.md). In the agentic passes the median was 32.5 GB on and 31.2 GB off. Large apps were closed; these figures do not measure how much memory other apps can use.

With KV_PAGES=on, a ~61K-token prompt in the 65536-token window recovered 4 of 4 facts in each of 6 runs; the median peak was 37.21 GB with one editor open and 37.49 GB with an editor and a chat app, against the runs' memory guard at 38.65 GB, with no swap-outs. These runs left little room for other apps. The first ~61K-token prompt, read without a cache, takes about 195 s; decode ran at 24.2-26.8 tok/s with one editor open and 22.9-26.4 with the chat app. In this mode MLX keeps 0.5 GiB of freed buffers for reuse (2 GiB otherwise) and the prompt cache holds one conversation. Greedy replies to 16K-token prompts were the same with KV pages on and off in 16 of 16 cases, and in 13 of 16 when a 64-page pool filled and evicted entries (evidence/window-64k.md; how it works: docs/kv-pages.md).

SSD streaming

STREAM=on with its defaults, three real agent requests of 23,334 to 30,770 prompt tokens with fixed seeds, sent one after another to one server start per setting, on a from-scratch install of this repository with other apps open; the idle-Mac row is a separate run (evidence/streaming.md).

	STREAM=on	STREAM=on STREAM_CACHE_GIB=0.5
server process phys_footprint, highest sample	5.59 GB	7.01 GB
whole-machine wired memory, highest sample (varies with the other apps)	13.98 GB	12.65 GB
decode, tok/s, other apps open	2.6-2.8	2.6-2.9
decode, tok/s, otherwise idle Mac	2.6-2.7
prompt read, no tokens reused from a cache, s	169-259	193-267
same tokens as a full-memory server, prompt cache off	3 of 3	0 of 3
same tokens as a full-memory server, 0.5 GiB prompt cache		3 of 3

phys_footprint does not include the memory-mapped trunk.gguf and ngram.gguf pages that macOS keeps in its file cache. The token match is an observation on three inputs, not a guarantee.

What the cut changes

cut240 against the unpruned GGUF on the same runtime, drafting off, without the disconnect and title behaviour (evidence/memory.md, evidence/agentic.md).

	cut240	unpruned
highest wired sample with the window full (one run each)	36.38 GB	38.03 GB
highest wired sample during agentic passes (4 each)	35.4 GB	36.8 GB
decode at 30K tokens	25.9 tok/s	25.8 tok/s
exercises solved over 4 passes	90 of 96	89 of 96

llama.cpp, serving the unpruned GGUF with its n-gram drafting, solved 94 of 96 of the same exercises (evidence/agentic.md).

Known issues

• In the 32K-window runs, wired memory spikes 3.4 to 4.7 GB above its median level within a few seconds of a request ending, around the time the next one starts. The cause is not known. These spikes set the highest samples above.
• A turn arriving while the server writes a session title waits for it (36 titles in the six agentic passes: median wait 8.8 s, once 50.5 s behind a 1,635-token title, 2.1% of pass time in all). A title request without max_tokens gets the server's limit of 16,384 tokens (evidence/background.md).
• About 5 of 24 opencode sessions a pass ended without a title; the cause is not known.
• A client resetting a kept-alive connection leaves a ConnectionResetError traceback in the server log; serving continues.
• Above the 32768-token window, only the 65536-token window with KV pages was measured, with little memory to spare.
• With a KV page pool smaller than one prompt (TF_KV_POOL_PAGES), copies that do not fit the pool are stored contiguous and save no page memory; the prompt cache keeps working. The default pool holds one window of pages plus 32.
• With SSD streaming, a request waits while another one, a session title included, is being served.
• With STREAM_CACHE_GIB=0.5, the three seeded replies differed from those of the full-memory server with the prompt cache off, although no request reused cached tokens. A full-memory server with the same cache gave the same replies as streaming (3 of 3), so the difference comes with the cache setting, not with streaming.
• With SSD streaming, the server tries to lock its expert buffers in memory, up to 0.4375 of the Mac's memory (21 GiB at most). Above that, or if macOS refuses the lock, it runs with the buffers unlocked and logs a warning. That path was tested only with simulated refusals.

Files

file	what
experts-cut240.gguf	the routed experts of ISTA-DASLab's shard 1 without 768 removed ones; kept bytes unchanged
trunk.gguf	every other tensor of ISTA-DASLab's shard 1, unchanged
ngram.gguf	ISTA-DASLab's shard 2, the per-layer n-gram embedding tables, unchanged
cut_experts.json and the other small files	the kept expert ids per layer, Qwen's tokenizer and chat template, and the model config that names the files above
mtp/keep256.json	the 256 MTP-head experts serve.sh keeps (how they were chosen: NOTICE)
install.sh, setup.sh, serve.sh	install the runtime, fetch the head and check checksums, start the server
SHA256SUMS	checksums of the large files and the head
evidence/	the measurements behind this card

Tensor elements: 56.6B in experts-cut240.gguf, 4.9B in trunk.gguf and 51.2B in ngram.gguf (n-gram embedding tables). The Hub's "Model size" counts only experts-cut240.gguf.

Licences

• Weights: Qwen Community License 1.0 (LICENSE), as a derivative of Qwen3.8-Flash-Next.
• ISTA-DASLab's source GGUF: Apache-2.0 (LICENSE-APACHE-2.0); NOTICE lists the changes made here.
• Scripts: MIT (LICENSE-MIT). setup.sh fetches the MTP head from ggml-org (NOTICE).