Doesnt appear to work on WSL2. I had launched the correct recipe but my GPU was only hitting around 200w instead of 600w on a 3 lane full context setup for my agents. Ended up doing some debugging and the pinned memory is unavailable and throughput doesn't scale across lanes (130 tokens/s total instead of each). Going back to the ninfer build instead for my single RTX PRO 6000

i got it working on single pro 6000 with docker on wsl. Did not work out of the box though but now im running patched 0.32 nightly build (let the flash next to patch it, no idea what it did but now it works 😀 ). Got a good speed and also i think a bit of quality boost compared to earlier nvfp4 quants. Here is readme (basically what was done) if someone is interested:

Qwen3.8-Flash-Next — NVIDIA FP4-Local weights on mainline vLLM

Runs nvidia/Qwen3.8-Flash-Next-FP4-Local on the same kind of stack as
production (stock vLLM + WSL2 settings), not on the community fork.

Why nightly + one patch

• Nightly, pinned to commit 83c078d955cd (2026-10-23): the NVIDIA
  checkpoint's PLE table is NVFP4. Support for that was merged 10-14 and is
  not in v0.32.0, so no release can load it yet.
• CSF patch (apply_csf_patch.py): NVIDIA's expert scales are stored
  compressed ("csf"). The upstream fix, PR #60646, is still an unmerged draft.
  The patch backports it: a loader-only change that decodes the scales on the
  CPU at load and finds the shards under tensors/. It touches no GPU code.
  Tested against the pinned source: the result is identical to the PR, and
  re-running the patch is a no-op. The image build fails if the patch or its
  self-test fails.
• Not needed any more: the pinned-chunk patch (the NVFP4 table is small
  enough to pin as-is) and the fork image.

Files

Run everything from the same root WSL shell you downloaded from.

1. Unpack + build (production keeps running)

cd ~ && unzip vllm_qwen38_nvfp4_mainline.zip
cd ~/vllm_qwen38_nvfp4_mainline
docker compose build

2. Swap stacks + boot

docker stop vllm-qwen38-main
cd ~/vllm_qwen38_nvfp4_mainline
docker compose up -d
docker compose logs -f

docker compose logs | grep -iE "Decoded .* CSF|Resolved Engram|Model loading took|KV cache|Maximum concurrency|out of memory|Uvicorn running|Error|Traceback" | tail -60

The first load is slower than production's: the CPU decodes 73,728 scale
tensors, and the new kernels get compiled.

3. Smoke test

curl http://localhost:8079/v1/models
curl -N http://localhost:8079/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen/Qwen3.8-Flash-Next",
  "messages": [{"role": "user", "content": "Say hi in Finnish."}],
  "max_tokens": 200, "stream": true, "reasoning_effort": "low"
}'

4. Tuning steps (one per boot, after it runs well)

1. KV size: compare idle VRAM with production (~91.5 GB). Raise
   --kv-cache-memory-bytes by the headroom, keeping a ~2–3 GB margin.
2. FP8 KV: --kv-cache-dtype fp8 puts ~2× the tokens in the same bytes.
   Check output quality on a long prompt.
3. MTP: uncomment the two --speculative-config lines (3 tokens, as NVIDIA
   validated). Costs a few GiB (the drafter and its KV). Compare
   single-stream and 7-concurrent t/s against production.

Troubleshooting

When a nightly includes PR #60646, drop the patch: change FROM to that
nightly and delete the COPY + first RUN.

i got it working on single pro 6000 with docker on wsl. Did not work out of the box though but now im running patched 0.32 nightly build (let the flash next to patch it, no idea what it did but now it works 😀 ). Got a good speed and also i think a bit of quality boost compared to earlier nvfp4 quants. Here is readme (basically what was done) if someone is interested:

Qwen3.8-Flash-Next — NVIDIA FP4-Local weights on mainline vLLM

Runs nvidia/Qwen3.8-Flash-Next-FP4-Local on the same kind of stack as
production (stock vLLM + WSL2 settings), not on the community fork.

Why nightly + one patch

• Nightly, pinned to commit 83c078d955cd (2026-10-23): the NVIDIA
  checkpoint's PLE table is NVFP4. Support for that was merged 10-14 and is
  not in v0.32.0, so no release can load it yet.
• CSF patch (apply_csf_patch.py): NVIDIA's expert scales are stored
  compressed ("csf"). The upstream fix, PR #60646, is still an unmerged draft.
  The patch backports it: a loader-only change that decodes the scales on the
  CPU at load and finds the shards under tensors/. It touches no GPU code.
  Tested against the pinned source: the result is identical to the PR, and
  re-running the patch is a no-op. The image build fails if the patch or its
  self-test fails.
• Not needed any more: the pinned-chunk patch (the NVFP4 table is small
  enough to pin as-is) and the fork image.

Files

File	What
Dockerfile	FROM vllm/vllm-openai:nightly-83c078d…, applies + tests the CSF patch, checks NVFP4 PLE / modelopt_mixed / WSL pin flag exist
apply_csf_patch.py	Anchored, fail-loud patch of default_loader.py
csf_scales.py	Decoder module, verbatim from PR #60646
csf_selftest.py	Build-time CPU round-trip test of the installed decoder
docker-compose.yaml	Production settings with the NVIDIA weights

Run everything from the same root WSL shell you downloaded from.

1. Unpack + build (production keeps running)

cd ~ && unzip vllm_qwen38_nvfp4_mainline.zip
cd ~/vllm_qwen38_nvfp4_mainline
docker compose build

2. Swap stacks + boot

docker stop vllm-qwen38-main
cd ~/vllm_qwen38_nvfp4_mainline
docker compose up -d
docker compose logs -f

docker compose logs | grep -iE "Decoded .* CSF|Resolved Engram|Model loading took|KV cache|Maximum concurrency|out of memory|Uvicorn running|Error|Traceback" | tail -60

Line	Meaning
Decoded 73728 CSF expert scale tensors at load	Patch worked ✅
Cannot find any model weights / Weight files referenced in index but missing	Download incomplete or shards not found → send the log
CUDA error: out of memory during PLE allocation	WSL2 pin cap after all → send the log
Uvicorn running on http://0.0.0.0:8079	Ready

The first load is slower than production's: the CPU decodes 73,728 scale
tensors, and the new kernels get compiled.

3. Smoke test

curl http://localhost:8079/v1/models
curl -N http://localhost:8079/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen/Qwen3.8-Flash-Next",
  "messages": [{"role": "user", "content": "Say hi in Finnish."}],
  "max_tokens": 200, "stream": true, "reasoning_effort": "low"
}'

4. Tuning steps (one per boot, after it runs well)

1. KV size: compare idle VRAM with production (~91.5 GB). Raise
   --kv-cache-memory-bytes by the headroom, keeping a ~2–3 GB margin.
2. FP8 KV: --kv-cache-dtype fp8 puts ~2× the tokens in the same bytes.
   Check output quality on a long prompt.
3. MTP: uncomment the two --speculative-config lines (3 tokens, as NVIDIA
   validated). Costs a few GiB (the drafter and its KV). Compare
   single-stream and 7-concurrent t/s against production.

Troubleshooting

Symptom	Action
Build: FAIL: anchor N found 0 times	Base image isn't the pinned commit — don't edit the tag
Crash in V2 model runner	uncomment VLLM_USE_V2_MODEL_RUNNER: "0"
CUBLAS_STATUS_INTERNAL_ERROR / illegal memory access on multi-turn	--no-enable-prefix-caching (issue #54173)
CUDA out of memory mid-warmup	KV bytes −1 GiB (12.5 GiB = 13421772800)
CUDA driver error: unknown error	someone added expandable_segments — remove it
Unknown quant layer / MXFP8 / W4A16_NVFP4 error at load	send me the traceback (nightly gap; fork kit is plan B)

When a nightly includes PR #60646, drop the patch: change FROM to that
nightly and delete the COPY + first RUN.

Nice work, it's a lot of job!

I had it working on a single lane but couldn't get it to work in parallel for multiple users even though I had the context for 3 full lanes due to the WSL bug. Did you get round that or just launch a single lane?

Yeah in vllm currently using max sequences of 10. works great tbh (pushes upt to 530 tokens/seconds with that concurrency when kv cache is not totally full)