vllm-sr

/

d3

Like 264

Follow

vLLM Semantic Router 527

Thanks for the request @BuiDoan !
We’re working on it: quantized model support in the vLLM Semantic Router model runtime first, then quantized d3 weights once they match the BF16 model closely enough.
We’ll update this thread when it’s ready

Thanks for waiting @BuiDoan ! Quantized d3 is ready:

• vllm-sr/d3-FP8: keeps 99.1% of d3's answers in 31.9 GB instead of 52.2 GB. Needs a GPU with FP8 units (NVIDIA compute capability 8.9+ or AMD MI300/MI350).
• vllm-sr/d3-W4A16-AWQ: int4, keeps about 98% in 16.5 GB, runs on any GPU with about 18 GB free.
• vllm-sr/d3-W4A16-GPTQ: int4, keeps about 98% in 16.4 GB, runs on any GPU with about 17 GB free.

They are served by the vLLM Semantic Router model runtime:

curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel dev
vllm-sr serve -e vllm-sr/d3-FP8

@subin please make the quantized versions be supported by stock vllm. The vllm-sr package is not as user friendly to use.

We do not want to download/install a CLI to run a container.

Hi @g-a-b-y ! d3 support landed on vLLM's main branch on 2026-10-14 (vllm-project/vllm#61565), so the quantized packages run in stock vLLM. Until a release includes it, install a nightly build as described in Run with vLLM on the d3 card.

vllm-sr/d3-FP8 works as it is:

vllm serve vllm-sr/d3-FP8

On AMD GPUs, the int4 packages (vllm-sr/d3-W4A16-AWQ and vllm-sr/d3-W4A16-GPTQ) currently fail at startup with the default context length, because of a bug in vLLM's ROCm int4 kernel (vllm-project/vllm#63049, fix in vllm-project/vllm#63054). Until the fix is merged, start them with a shorter context:

vllm serve vllm-sr/d3-W4A16-AWQ --max-model-len 32768

Thanks!

@subin found the issue. By default the model uses 261k max model len, which on a H100 throws an Out Of Memory even with the FP8 quantization.

Changing the max model length to 131k fixed the issue.

I'm using the nightly Docker image.

Thanks for tracking it down @g-a-b-y , and sorry, my “works as it is” was only checked on a 192 GB MI300X. d3 reads its prompt with bidirectional attention, so vLLM serves it without chunked prefill, and at startup it profiles one forward pass of the full max_model_len. At the default 262,144 tokens that pass alone needs about 45 GiB of activations next to 30 GiB of FP8 weights, which does not fit on an 80 GB GPU. --max-model-len 131072 (or lower) is the right fix; d3’s own requests are far shorter than that.

Should we use 131k or 64k or something else for max model len?

It would make sense to lower the default 262k in the config.json

It depends on the GPU, so I'd rather keep 262,144 in config.json: it is the model's position limit, and lowering it would cap larger GPUs too. For d3's own inputs, --max-model-len 32768 is plenty (a request with a full 32-frame video is about 6k tokens) and leaves the most memory free, so that's what I'd use on any GPU. I'll put it in the Run with vLLM section of the cards.

Perfect, thank you!