nvidia

/

Qwen3.8-Flash-Next-FP4-Local

Like 389

Follow

NVIDIA 74.7k

I had trouble running the card's DGX Spark recipe on one Spark (revision b9ce18a6, eugr/spark-vllm-b12x), so here's what worked in case it helps someone else. About 16 GB was already in use on the box before vLLM started, so a cleaner machine may have more room.

What happened

• At --gpu-memory-utilization 0.8, 0.72 and 0.65, memory fell sharply about 35–60 s after the weights loaded, and my memory guard stopped the server each time. Lowering utilization didn't help.
• Adding PYTORCH_ALLOC_CONF=per_process_memory_fraction:0.72 removed the spike. The card doesn't set this, but eugr's solo recipe sets an equivalent cap at 0.8. I didn't test the card's other settings with the cap on its own.
• The next failure was KV sizing: --max-model-len 262144 needed 4.93 GiB and only 3.93 GiB was available. Setting 131072 fixed that.

What worked: the card's recipe, plus the allocator cap, --gpu-memory-utilization 0.72, --max-model-len 131072, and --default-chat-template-kwargs '{"reasoning_effort":"medium"}'.

Measured

• Startup took about 11 minutes, and the lowest free memory was 11.7 GB.
• The KV pool was about 336K tokens, well short of the 1M the card mentions for a Spark. My context cap and memory limit are lower than the card's recipe, so this isn't a direct comparison.
• Speed was 35.3 tok/s on a single stream, 96.6 tok/s at 8 concurrent, and 5.15 s to process a 16K prompt.
• It scored 0.83 on my own 64-task suite, against 0.82 for Nemotron 3 Super and 0.84 for Claude Sonnet 5.5. Gaps under about 0.08 are noise on this suite.

If you see an OOM that looks different, also check the permissions on ~/.cache/b12x, which was the cause in discussion #8.