primitive-ai/Qwen3.8-Flash-Next-NVFP4

The 180B Flash-Next on one GPU.

Qwen3.8-Flash-Next is 360 GB in BF16 and needs two data-center GPUs in FP8.
This build serves it on one 96 GB Blackwell: 88.8 GiB of VRAM, the 51B n-gram table in host RAM, and no patched runtime.

---

Why this quant

• 🖥️ One 96 GB GPU. 88,828 MiB of VRAM at serve: NVFP4 experts plus the BF16 tail. The n-gram table lives in host RAM (~100 GB) with async prefetch. No other public build of this model can use the offload path at all.
• 🎯 92.2 knowledge on a 1,170-item, 9-benchmark suite (n=2 runs: 92.1, 92.2), 99.4% finished, zero request errors. GSM8K 98.0, MMLU-Pro 90.0.
• 🔧 84.6 call / 56.7 abstain on the 200-item tool-calling suite (n=3). It makes correct calls at the same rate as much smaller strong models and never declines to call — the abstention items are what pull a pooled number down, which is why both halves are printed.
• ⚡ 74.4 tok/s single-stream (12.2 ms/token), 483.8 tok/s at concurrency 32, measured prefix-cache-free with distinct seeds; the two seeds agreed within 0.04%.
• 🔀 MTP speculative decoding preserved. All 31 MTP tensors byte-identical to the source; {"method":"mtp","num_speculative_tokens":3} works as Qwen documents.
• 🧩 Stock image, no patches. vllm/vllm-openai:qwen38-flash-next exactly as published. The serve block below carries the two flags that make single-GPU work — without them the server hangs silently or times out.

---

Serve it

docker run --gpus all --ipc=host -p 8000:8000 \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800 \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-NVFP4 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Two flags are load-bearing on a single GPU. --distributed-executor-backend mp: the default single-GPU executor never starts the n-gram offload worker, and the first forward waits on it forever — the server looks healthy and hangs. VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800: the worker loads a 95 GB table before serving and the 600 s default can expire first. Host needs about 100 GB of free RAM. The reasoning parser and tool-call parser in this block are validated on this exact checkpoint: thinking lands in reasoning with no markup in content, and tool calls arrive as structured tool_calls with valid JSON arguments and finish_reason: tool_calls.

---

Not enough host RAM? Put the table on NVMe

The serve command above wants ~100 GB of free host RAM for the n-gram table. With fast local storage you can skip that: this repo ships a one-file overlay (worker_image_disk.py) that maps the table from a file instead. First boot writes 95.4 GB into the store directory; every later boot maps it instantly and skips the table's checkpoint reads.

hf download primitive-ai/Qwen3.8-Flash-Next-NVFP4 worker_image_disk.py --local-dir .
mkdir -p pledisk_store

docker run --gpus all --ipc=host -p 8000:8000 \
  -v $PWD/worker_image_disk.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py:ro \
  -v $PWD/pledisk_store:/pledisk_store \
  -e VLLM_PLE_DISK_OFFLOAD_DIR=/pledisk_store \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=3600 \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-NVFP4 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Measured on the mixed sibling — both repos ship byte-identical BF16 tables and the same worker path, so the disk behavior transfers; absolute tok/s columns are the sibling's. 8K in / 512 out, prefix-cache-free, two seeds per cell (shown a / b):

config	boot	tok/s @ 1	tok/s @ 32	median TTFT @ 1
table in RAM (command above)	302 s	84.5 / 84.4	516.8 / 523.6	569 / 573 ms
disk, container capped to 48 GB RAM — recommended	263 s	79.4 / 76.8	427.0 / 435.8	571 / 573 ms
disk, uncapped 176 GB host	303–344 s	50.4–62.1	196.7–396.7	1.8–2.9 s
disk, cold page cache	404 s	40.8 / 37.0	134.2 / 290.3	4.6 / 5.2 s
disk, first boot (writes the file)	504 s	—	—	—

Net cost of the disk path, run capped: −8% single-stream, −17% at concurrency 32, TTFT parity with the RAM baseline. The counterintuitive row is the uncapped one, and it reproduces across two boots and four seeds: without a container memory cap, the boot's own 172 GB checkpoint streaming flows through the global page cache and evicts the table it is about to need, so gathers fault back to NVMe mid-decode. A memory cap makes reclaim cgroup-local — the container's checkpoint reads can only evict the container's own cache, and the table stays resident. So on the disk path, always cap the serving container (48 GB is what we validated; --memory 48g --memory-swap 48g).

Accuracy is unaffected — the mapping serves the same bytes. Inside the 48 GB cap the 200-item tool-calling suite scored 78.5 with zero request errors and zero truncations (repeat spread on this suite: 78.0–80.5), and the generation-sanity gate passed on the first-boot and capped configurations. Cold cache is a floor, not a steady state: the two cold @ 32 runs went 134 → 290 tok/s back to back as the cache refilled. Boot times share one caveat: all were measured with the checkpoint at least partially page-cache-resident; a truly cold first read of the 172 GB weights adds its own disk time to any of them.

The overlay targets this exact image. The same change is a draft PR to vLLM — vllm-project/vllm#54070, branch feat/ple-disk-offload — stacked on the PLE CPU-offload PR (vllm-project/vllm#53899).

---

Quantized PLE tables: 49 GB or 32 GB instead of 95 GB

The table itself also quantizes well. We publish it in FP8 per-row (49 GB) and INT4 group-16 (32 GB), served memory-mapped by a two-file overlay — host RAM cost becomes page cache only, no container cap needed. Accuracy holds on both suites (knowledge 92.2 / 92.9 vs 92.2 for BF16; tool-calling inside the ±1.5 repeat spread), throughput lands within 5–6% of the in-RAM BF16 baseline, and MTP keeps most of its speed-up (129.6 tok/s single-stream with the INT4 table vs 142.6 in-RAM). Tables, overlay files, serve command, format spec, and the full measurement table: primitive-ai/Qwen3.8-Flash-Next-PLE-quant.

---

Speculative decoding (MTP)

The MTP tensors are preserved byte-identical, so vLLM's built-in draft path works — add:

  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Real-prompt A/B (measured on the mixed sibling), single stream, thinking on (60-item subset of our eval, decode rate = output tokens over wall time — not comparable to the bench-serve numbers above, and measured on real prompts because random-token benches overstate speculative gains):

speculative config	decode tok/s	strict score
none	91.2	91.7
num_speculative_tokens: 1	does not boot	—
num_speculative_tokens: 2	133.2	93.3
num_speculative_tokens: 3	142.6 (+56%)	93.3
num_speculative_tokens: 3, table on NVMe	77.5–82.3	93.3

Score differences are single-run noise on 60 items; the speed difference reproduces. num_speculative_tokens: 1 hangs this image at startup — the engine core blocks in shm_broadcast before the API server binds, reproduced on three boots (one waited 80 minutes). Use 3.

MTP and the disk-backed BF16 table do not combine well: speculation multiplies table-gather traffic, the working set outgrows the page cache, and the +56% collapses to roughly the no-speculation rate (capped 77.5, uncapped 82.3). With the BF16 table in RAM, use MTP; on a low-RAM host, pair MTP with the INT4 quantized table instead, which keeps 129.6 tok/s.

---

Measured

1,370 items across fourteen public benchmarks. A 1,170-item knowledge suite (MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500) and a 200-item tool-calling suite (BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, nvidia When2Call), under one fixed protocol: temperature 0.6 / top_p 0.95 / top_k 20, thinking forced on, a 16,384-token budget, no reasoning parser, the last ANSWER: scored. Concurrency 32 on the same single RTX PRO 6000 Blackwell the fit numbers come from. Auto-scored, no LLM judge.

build	size	overall	knowledge	call	abstain	runs k/a	finished	out/answer	tok/s @ 32	tok/s @ 1
this repo	186 GB	90.2	92.2	84.6	56.7	2/3	99.4%	664 tok	483.8	74.4

overall is one number over both suites: the 1,170 knowledge and 200 tool-calling items pooled as 1,370, weighted 85.4% and 14.6% by item count. call is accuracy on the 160 tool-calling items that require a call; abstain is the 40 whose correct action is to call nothing — they are never pooled into one number on our cards, because a model can be strong at one and weak at the other, and this one is exactly that: solid call accuracy, zero abstentions in the system-prompt tool format. Through the native tools= API it does abstain (validated above), so weight the two columns by how your application passes tools.

There is no comparison column because no other checkpoint of this model serves on this hardware — the next section is that story.

Knowledge is a mean of 2 runs (92.1, 92.2); tool-calling of 3 (78.5, 80.0, 78.5 pooled). Throughput is prefix-cache-free — --random-prefix-len 0, a distinct seed per run, warm-up seeded apart — and two seeds agreed within 0.04%. On our other models this suite's repeat spread runs to ±0.5 on knowledge and ±1.5 on tool calling; treat gaps inside that as ties.

---

Which checkpoints fit a single 96 GB GPU

checkpoint	on disk	serves on one 96 GB card
Qwen BF16	360 GB	no (240+ GB of weights)
Qwen FP8	185 GB	no — FP8 n-gram table crashes the offload worker; TP2 GB300 / TEP8 H200 per the vLLM recipe
RadixArk NVFP4	135 GB	no — same FP8-table crash; validated by its authors on SGLang, 2x GB300
Inferact NVFP4	183 GB	no — near-unquantized and FP8 table
lovedheart NVFP4-FP8	132.5 GB	no — FP8 n-gram table, and needs a patched SGLang fork (stock emits garbage silently)
this repo	186 GB	yes — 88.8 GiB VRAM + ~100 GB host RAM

GGUF and MLX conversions of this model exist for other runtimes (llama.cpp forks, Apple MLX); none serve on vLLM. Accuracy numbers published for other checkpoints came from different harnesses and hardware and are not comparable to the table above; they are not restated here.

The blocker is the n-gram table's storage format, not size arithmetic: VLLM_PLE_CPU_OFFLOAD builds the CPU-side embedding as a plain BF16 table, and every other checkpoint stores it as FP8 with a scale tensor, which that worker cannot load. Multi-GPU deployments (TP2 and up) can use any of them; a single card can use this one.

---

What's quantized to what

tensors	format
all 48 layers' routed experts (gate/up/down_proj, 120.8B params)	NVFP4 (group 16)
n-gram embedding table (51.2B, 128 shards)	BF16, pre-scaled — the vLLM offload worker loads no other format
attention, GDN linear-attention, shared experts, routers, MTP, vision, embeddings, norms	BF16, byte-identical to the source

Weights-only round-to-nearest, no calibration. The n-gram table carries the FP8 release's values, materialized in BF16 — the same numbers every runtime materializes at load.

model	shape	size	overall	knowledge	call	abstain
Laguna-XS-2.1	31 B MoE	19.3 GiB	81.7	83.8	68.4	73.5
Nemotron-3.5-Lightning-30B-A3B	30 B MoE+Mamba	19.2 GiB	87.1	87.9	85.4	70.5
Ornith-1.5-35B-A3B	35 B MoE	22.6 GiB	88.7	91.7	74.4	60.0
Muse-Glimmer-30B	30 B MoE	20.4 GiB	86.6	88.8	78.6	54.5
Qwen3.8-27B	27 B dense	20.7 GiB	88.8	90.4	85.5	54.5
Laguna-S-2.1	110 B MoE	64.0 GiB	84.3	87.1	64.6	81.0
Qwen3.8-Flash-Next	180 B MoE (6 B active)	185.8 GB	90.3	92.2	85.0	56.7

model	shape	size	overall	knowledge	call	abstain
Laguna-XS-2.1	31 B MoE	19.3 GiB	81.7	83.8	68.4	73.5
Nemotron-3.5-Lightning-30B-A3B	30 B MoE+Mamba	19.2 GiB	87.1	87.9	85.4	70.5
Ornith-1.5-35B-A3B	35 B MoE	22.6 GiB	88.7	91.7	74.4	60.0
Muse-Glimmer-30B	30 B MoE	20.4 GiB	86.6	88.8	78.6	54.5
Qwen3.8-27B	27 B dense	20.7 GiB	88.8	90.4	85.5	54.5
Laguna-S-2.1	110 B MoE	64.0 GiB	84.3	87.1	64.6	81.0
Qwen3.8-Flash-Next	180 B MoE (6 B active)	183.7 GB	90.3	92.2	84.8	56.7

---

_{primitive · more models · inference economics for production LLM systems}