d3-NVFP4

This is vllm-sr/d3, the 26B model from the Decision 3.0 family, quantized to NVFP4 with NVIDIA TensorRT Model Optimizer. It's meant for Blackwell GPUs, which run FP4 natively.

We followed NVIDIA's recipe for this model family: the MLP and full-attention layers are in NVFP4, and the linear-attention layers, embeddings, vision encoder and decision readout are left as they were. That's why the files shrink less than you'd expect from 4-bit weights.

How it compares to the original

We scored both models on the public part of the Decision Index 0.3. It's a mix of general benchmarks (knowledge, language, retrieval, tool use and arts), not anything specific to one language or domain. To keep the runs short we used a fixed 1,000-request sample drawn with the kit's own stratified sampler, which covers every benchmark, and scored it with the kit's scorers. We built the suite from the kit's sources rather than downloading the official copy, so treat these as numbers for comparing the rows below, not as leaderboard scores.

	Decision Index (public)	Same answer as the original	Weights
vllm-sr/d3, original	65.48	-	52.2 GB
tensor-ai-inc/d3-NVFP4	63.12	98.8%	25.2 GB
Jev by TypeSafe, through Vercel AI Gateway, for reference	54.69	-	-

The NVFP4 weights are 52% smaller than the original. "Same answer" is how often the two models gave the same pick on the same question, across every question in the sample. Jev couldn't answer 7 of the 1,000 requests, and the kit counts those as wrong.

We ran both models on one H100 through the same code path. Hopper has no FP4 hardware, so the NVFP4 scores come from Model Optimizer's simulated FP4, which rounds weights and activations the same way the exported model does on Blackwell. We haven't measured speed on Blackwell yet.

Running it

Decision 3.0 support is on vLLM's main branch, so you need a nightly build:

pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
vllm serve tensor-ai-inc/d3-NVFP4

Then ask it questions about any input through the System One API:

curl localhost:8000/v1/systemone -H "content-type: application/json" -d '{
  "model": "tensor-ai-inc/d3-NVFP4",
  "state": "The order arrived damaged yesterday. The customer has a receipt and asks for a replacement today.",
  "questions": {
    "route": {"type": "choice", "instructions": "Which team should handle this request?",
               "criteria": {"returns": "Refunds, replacements and damaged deliveries", "billing": "Payments, invoices and charges"}},
    "receipt": {"type": "noul", "instructions": "Does the customer have a receipt?"}
  }
}'

The runtime code in this repo is the original package's, unchanged, so everything in the original model card about questions, images and video applies here too.

How it was made

• Model Optimizer 0.47.0, NVFP4 (FP4 values, FP8 scales per 16 values, one FP32 scale per tensor).
• Calibrated on 256 UltraChat prompts with generic choice, yes/no and score questions, nothing from any benchmark.
• Source revision: cf15549a78fc52598ddd1951f321c58372bbd64f.

License

Apache-2.0, like the original. This is a modified version of vllm-sr/d3 by the vLLM Semantic Router team; we changed only the weights' precision. Their LICENSE and NOTICE files are included.