nvidia

/

Qwen3.6-27B-NVFP4

like 431

Follow

NVIDIA 66.3k

It will

It would work even in FP8 (with a pretty low context size)

I tried it on an RTX 5090, it does not work (out of memory). I used vllm nightly and the parameters from the model card: vllm serve nvidia/Qwen3.6-27B-NVFP4 --port 8000 --quantization modelopt --max-model-len 262144 --reasoning-parser qwen3. Even reducing max-model-len to 8192 I get an OOM. There is a line in the output which looks suspicious:

(EngineCore pid=40152) WARNING 07-01 13:04:02 [marlin.py:34] Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads.

AFAIK, RTX 5090 should support FP4. Any ideas why this fails?

yes, the OG model does fit

If anybody has a working configuration for RTX 5090, could you please post it? I cannot get it working, I always get OOM (even after reading https://www.reddit.com/r/LocalLLaMA/comments/1my3why/rtx_pro_6000_maxq_blackwell_for_llm/)

I'm in the same position Christian. I can't get any configuration to land.

Here's my working vllm 0.24.0. I still cannot go pass 256144 token window because we need room for JIT compilers.

CUDA_VISIBLE_DEVICES=0 NCCL_P2P_DISABLE=1 \
VLLM_SKIP_P2P_CHECK=1 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
VLLM_USE_FLASHINFER_SAMPLER=1 \
vllm serve ./nvidia/Qwen3.6-27B-NVFP4 \
  --host 0.0.0.0 \
  --port 8082 \
  --tensor-parallel-size 1 \
  --attention-backend flashinfer \
  --quantization modelopt \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 204800 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.92 \
  --safetensors-load-strategy prefetch \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

I'm running this on an RTX 5090 (32 GB) as well, but with the Unsloth Q6_K GGUF right now. I can fit 128k context entirely in VRAM, and I'll be switching to the official NVFP4 model this weekend.

The warning about "Your GPU does not have native support for FP4 computation" on a 5090 also looks wrong. Blackwell consumer GPUs should support FP4, so that makes me wonder if that nightly build wasn't correctly detecting the architecture.

I'll report back once I've tested NVFP4 on my own 5090. It should be interesting to compare VRAM usage, throughput, and how much context can realistically be pushed.

Try docker:

Working Docker Command:
docker run -d --name qwen-vllm --gpus all -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 -e TORCH_CUDA_ARCH_LIST="12.0" -v "D:/jarvis/models/nvidia/Qwen3.6-27B-NVFP4:/data/model" -p 8000:8000 --ipc=host --shm-size=16g vllm/vllm-openai:latest /data/model --served-model-name nvidia/Qwen3.6-27B-NVFP4 --quantization modelopt --dtype bfloat16 --kv-cache-dtype fp8 --max-model-len 16384 --max-num-seqs 96 --gpu-memory-utilization 0.85 --distributed-executor-backend mp --trust-remote-code

Try docker:

Working Docker Command:
docker run -d --name qwen-vllm --gpus all -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 -e TORCH_CUDA_ARCH_LIST="12.0" -v "D:/jarvis/models/nvidia/Qwen3.6-27B-NVFP4:/data/model" -p 8000:8000 --ipc=host --shm-size=16g vllm/vllm-openai:latest /data/model --served-model-name nvidia/Qwen3.6-27B-NVFP4 --quantization modelopt --dtype bfloat16 --kv-cache-dtype fp8 --max-model-len 16384 --max-num-seqs 96 --gpu-memory-utilization 0.85 --distributed-executor-backend mp --trust-remote-code

This is mostly a multi-user/high-throughput vLLM config rather than something optimized for a single 5090.

--max-num-seqs=96 is unnecessary for personal use and only increases scheduler/KV-cache bookkeeping overhead. On top of that, it implies a worst-case KV cache budget of roughly 96 * 16k tokens ~ 1.5 million tokens worth of KV cache capacity, which is outrageous. Set max-num-seqs to 1 and increase the context size.

Also, --distributed-executor-backend mp is irrelevant on one GPU, and TORCH_CUDA_ARCH_LIST doesn’t matter in a prebuilt container.

The core model settings are fine, but the rest is clearly tuned for throughput, not a single-user inference box.

If anyone wants to use Docker I would transform jpmrblood's command (you're free to select any port you want):

docker run --rm \
  --gpus '"device=0"' \
  --ipc=host \
  -p 8082:8082 \
  \
  -e CUDA_VISIBLE_DEVICES=0 \
  -e NCCL_P2P_DISABLE=1 \
  -e VLLM_SKIP_P2P_CHECK=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  \
  vllm/vllm-openai:latest \
  \
  --model nvidia/Qwen3.6-27B-NVFP4 \
  --host 0.0.0.0 \
  --port 8082 \
  --tensor-parallel-size 1 \
  \
  --attention-backend flashinfer \
  --quantization modelopt \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 204800 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.95 \
  \
  --safetensors-load-strategy prefetch \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --enable-auto-tool-choice

docker pull vllm/vllm-openai:v0.24.0

docker run -d --gpus all --shm-size 16G -p 11434:8000 -v E:\.cache\huggingface:/root/.cache/huggingface
-e HF_ENDPOINT=https://hf-mirror.com --name vllm-qwen3.6-nvfp4
vllm/vllm-openai:v0.24.0 nvidia/Qwen3.6-27B-NVFP4
--quantization modelopt --trust-remote-code
--served-model-name qwen3.6:27b --max-model-len 147456
--max-num-seqs 2 --max-num-batched-tokens 4096
--kv-offloading-size 8 --gpu-memory-utilization 0.91
--kv-cache-dtype fp8_e4m3 --enable-prefix-caching
--enable-chunked-prefill --reasoning-parser qwen3
--enable-auto-tool-choice --tool-call-parser qwen3_coder
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'