Tooooo BIG for a single node DGX Spark :(
theoretically this should be the sweetspot for this hw category, I do hope NVFP4 quant could be smaller. Much smaller to fit with KV cache too.
buy another one😂
Agreed
You need to wait for the ngram portion to get fp4 quantized whether it's in vllm or sglang code path, but right now it expands ngram to be bf16, 99GB
just wait for Qwen4.0 27b, I think it will be better
theoretically this should be the sweetspot for this hw category, I do hope NVFP4 quant could be smaller. Much smaller to fit with KV cache too.
working on it.
theoretically this should be the sweetspot for this hw category, I do hope NVFP4 quant could be smaller. Much smaller to fit with KV cache too.
Guys, here it is:
Working vLLM support for SM121/DGX Spark with PLE tables offloaded to NVMe - https://github.com/TamasS84/vllm-qwen38-flash-sm121-nvme. It runs on a single DGX SPARK with 256K context length
@realbazso
Single DGX Spark benchmark results
Test configuration:
- Prompt: 2,048 tokens
- Generation: 128 tokens
- Three runs per concurrency level
- Prefix caching disabled
- MTP speculative decoding enabled
- Maximum configured context: 262,144 tokens
| Concurrent requests | Aggregate generation | Per-request generation | Prompt processing | Peak aggregate | Mean TTFR |
|---|---|---|---|---|---|
| 1 | 26.81 ± 0.16 tok/s | 26.81 tok/s | 859.87 tok/s | 35.33 tok/s | 2.56 s |
| 2 | 31.08 ± 2.43 tok/s | 19.31 tok/s | 802.03 tok/s | 54.67 tok/s | 3.94 s |
| 5 | 36.79 ± 1.59 tok/s | 14.30 tok/s | 883.39 tok/s | 101.33 tok/s | 9.69 s |
| 10 | 41.14 ± 1.44 tok/s | 7.75 tok/s | 996.36 tok/s | 155.00 tok/s | 14.83 s |
The server supports a maximum context length of 262,144 tokens. This benchmark used a 2,048-token prompt and did not measure long-context performance.
Modified vLLM fork with SM121 support and NVMe-backed PLE tables
going for more.