Inferact

/

Qwen3.8-Flash-Next-NVFP4

like 23

Follow

Inferact Inc. 117

I don't have a concrete answer here but my understanding is - yes. If you add up all the safetensor files you get ~180gb. I think the experts have been quantized to NVFP4 but the n-gram embedding layer isn't and it's like 95GB?

Yes The N-gram is still fp16 in offical recipe, the choices are bf16, fp16 and fp32 in sglang.
Waiting for the n-gram to be quantized to fp8/nvfp4

This thing flies on a single RTX Pro 6000 offloading all the n-gram stuff to system ram.

This thing flies on a single RTX Pro 6000 offloading all the n-gram stuff to system ram.

Can you share your recipe?

This thing flies on a single RTX Pro 6000 offloading all the n-gram stuff to system ram.

Can you share your recipe?
I run this in docker, but this is the recipe. If you want the full docker setup I can share that also.

# Requirements: native Linux (NOT WSL2), NVIDIA driver, and a vllm build that
# contains the qwen3_8_flash_next model (the NVIDIA fork our image is built from,
# v0.1.dev20073+g8e685d198 — stock pip vLLM may not have this model yet).
# Weights must be in the HF cache: export HF_HOME=/path/to/huggingface first.

VLLM_PLE_CPU_OFFLOAD=1 \
CUDA_DEVICE_ORDER=PCI_BUS_ID \
CUDA_VISIBLE_DEVICES=0 \
vllm serve Inferact/Qwen3.8-Flash-Next-NVFP4 \
  --served-model-name qwen38-flash-next \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 25 \
  --kv-cache-memory 11500000000 \
  --distributed-executor-backend mp \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Notes for the bare-metal folks:

• Only 3 changes vs our Docker run: VLLM_PLE_CPU_OFFLOAD=1 goes in the shell env, HF_HOME must point at the dir containing hub/models--Inferact--Qwen3.8-Flash-Next-NVFP4, and the CDI/device flags disappear (vLLM grabs the GPU directly via CUDA_VISIBLE_DEVICES). Everything else is identical to the verified config.
• RAM check first: free -g before serving — the PLE offload worker needs tens of GB of host RAM; 128 GB min, 192 GB comfortable.
• Optional bare-metal-only extra: PYTORCH_ALLOC_CONF=expandable_segments:True does work without Docker (no seccomp wall for the PLE worker's pidfd_getfd)

Measured performance (this exact config)

Metric Result
──────────────────────────────────────
Prefill 12,396 tok/s (5,457-tok prompt)
Decode, single stream 158 tok/s
Decode, 2 concurrent 287 tok/s system
Decode, 8 concurrent 715 tok/s system
25-concurrent JSON burst, thinking off 25/25 OK, 1.3s wall, p50 0.81s
Engine stability 0 crashes / 0 OOM / 0 × 500 across full battery

docker run --runtime nvidia --gpus all --ipc=host
--memory 120g
--shm-size 26g
-v /home/cheng/model/Qwen3.8-Flash-Next-NVFP4:/model
-p 8000:8000
--ipc=host
-e VLLM_PLE_CPU_OFFLOAD=1
-e PYTORCH_ALLOC_CONF=expandable_segments:True
-e CUDA_DEVICE_ORDER=PCI_BUS_ID
vllm/vllm-openai:qwen38-flash-next
/model
--served-model-name qwen38-flash-next
--host 0.0.0.0 --port 8000
--max-model-len 32768
--gpu-memory-utilization 0.85
--max-num-seqs 2
--tensor-parallel-size 4
--distributed-executor-backend mp
--enable-prefix-caching
--no-enable-flashinfer-autotune
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--enforce-eager

I tested several commands and they all overflowed the video memory. Mine is 4*3090 512G memory.