Post
2259
14 days serving
unsloth/Qwen3.8-27B-NVFP4 to production agents on 2x RTX 5090 (vLLM 0.27, TP=2, 262K context, FP8 KV):
28,097 requests, 860.6M prompt tokens, 82.6% prefix-cache hit rate, TTFT p50 0.61 s, 0 engine errors.
The observation: prefix cache, not throughput, decides whether a 27B model keeps up with agents.
Mean request is 30,100 tokens in, 983 out, because every turn resends the whole session. Two flags mattered most: --max-num-seqs 12 (queue p95 went 9.4 s to 233.6 s past that) and --watermark 0.08 (preemptions 29 to 2). And thinking off for tool loops: 917 tokens in 11.7 s vs 11,170 in 144 s, same answer.
Full config and counters: scalably.io/blog/qwen3-8-27b-nvfp4-rtx-5090-production
Next we are preparing an 8x B300 node in an EU data center for open-weight serving. Which models or workloads are underserved for you?
unsloth/Qwen3.8-27B-NVFP4 to production agents on 2x RTX 5090 (vLLM 0.27, TP=2, 262K context, FP8 KV):
28,097 requests, 860.6M prompt tokens, 82.6% prefix-cache hit rate, TTFT p50 0.61 s, 0 engine errors.
The observation: prefix cache, not throughput, decides whether a 27B model keeps up with agents.
Mean request is 30,100 tokens in, 983 out, because every turn resends the whole session. Two flags mattered most: --max-num-seqs 12 (queue p95 went 9.4 s to 233.6 s past that) and --watermark 0.08 (preemptions 29 to 2). And thinking off for tool loops: 917 tokens in 11.7 s vs 11,170 in 144 s, same answer.
Full config and counters: scalably.io/blog/qwen3-8-27b-nvfp4-rtx-5090-production
Next we are preparing an 8x B300 node in an EU data center for open-weight serving. Which models or workloads are underserved for you?