license: apache-2.0
base_model: vllm-sr/d3-ultra
pipeline_tag: zero-shot-classification
tags:
- decision-model
- system-one
- fastflowlm
- amd-npu
- q4nx
D3-Ultra-NPU2
vllm-sr/d3-ultra (52ba9e7a; Decision 3.0, the 27B multimodal decision
model of vLLM Semantic Router, Qwen3.5-27B text tower, a newer fine-tune of the d3-turbo architecture) for AMD Ryzen AI NPUs,
served by FastFlowLM 1.0.11+.
It answers System One requests -- choice, yes / no and score questions about a state, optionally with images and videos,
with a probability for every option; box questions (where is it in an image or a video) and track questions (a box per
moment over a whole video) as well -- over the same POST /v1/systemone API as the upstream package. Up to 32,768 input
tokens per question. Like upstream d3-ultra it reads the state once per request and answers every question on it
("ask once, answer many"), so many questions about one input cost far less than one request per question.
flm pull d3:ultra # 16.6 GB download
flm serve --decision d3:ultra # optionally next to a chat model: flm serve qwen3.5:9b --decision d3:ultra
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
-d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
"criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'
Images go in an images list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most
1.6 MP), shared by every question of the request. d3:lite (0.8B), d3:nano (2B) and d3:mini (4B) are the smaller
siblings; they answer a single short question in well under a second, d3:ultra needs a few seconds (see Speed).
d3:turbo is the previous 27B model on the same engine.
What this package supports on the NPU
| Question / input | Status |
|---|---|
choice, noul (yes / no), score on text |
supported |
| the same with images in the state (several images per request) | supported |
| Maximum length | 32,768 tokens per question (a longer prompt is answered with max_length_exceeded; nothing is truncated). Upstream's validated_length is 131,072; this build keeps the 32,768-token limit of d3:turbo |
long_context=True (upstream: up to 131,072 tokens) |
not supported (the request is refused with HTTP 422) |
| Question / input | Status |
|---|---|
box (where is it in an image; also at a moment of a video with at) |
supported: one box per question with a calibrated confidence |
assets (images inside the state, a question's instructions or its options) |
supported |
| videos in the state (2 frames per second, 4 to 32 frames, at most 0.2 MP per frame) | supported; a video is at most 60 s and 16,384 vision tokens per request; decoding mp4 / webm / mkv / mov / gif needs the FFmpeg shared libraries on the machine (without them the request is refused with HTTP 422; frame lists {"frames": [...]} always work) |
track (a box per moment over a whole video, 1 or 2 updates per second) |
supported; at most 256 moments per request (D3_MOMENT_ROWS) |
GET /v1/systemone/models lists exactly what the running build accepts.
What is new against d3:turbo (both are inside this package's decision_config.json and are applied by the server as the
upstream runtime does):
prompt_layoutheader-v1: for a text request, the prompt of every question starts with the blockQuestions in this request:(all questions of the request, numbered, with their instructions and option texts), then the state and the question. The header is part of the shared state block, so it is computed once per request; it adds tokens to every request (13 to 3,875 tokens, median 219 on the test set). Requests with images or videos, andassets/box/trackrequests, use the plain layout, as upstream.temperature_by_questions: the softmax temperature is 1.046 for a request with exactly onechoice/noul/scorequestion and 1.0 for two or more;boxandtrackanswers keep the base temperature.
Requirements
- An AMD Ryzen AI processor with an XDNA 2 NPU (this package was tested on a Strix Halo machine), Linux with the XRT stack, FastFlowLM as above.
- Memory: the package is 16.6 GB on disk and about 15.9 GiB resident once loaded. Measured peak resident memory of the server process: 17.2 GiB after the load, 18.0 GiB for prompts up to 4k tokens and 27.0 GiB at 32,768 tokens (one request on a fresh server); the FLM catalog reserves a footprint of 28 GB (plus 8.6 GB of locked scratch memory). All numbers are from a 128 GB machine; smaller machines were not tested, plan for a 32 GB or larger system and do not run it next to other large models.
- Load time: about 13 s for the engine with the files in the page cache (
flm servealso checks the 16.6 GB of files against their hashes the first time it sees them or after an upgrade).
What was changed
- Weights: the Qwen3.5-27B text tower of d3-ultra quantized to FastFlowLM's Q4NX format: every projection Q4_K with
GPTQ-calibrated rounding (our own sequential GPTQ over all 64 layers, 400 tensors; calibration: 300 synthetic System
One questions with the header-v1 layout and the model's
prefix_bidirmask, 147,008 tokens, none from the Decision Index), token embedding Q8_0, norms and the Gated DeltaNetalpha/betaparameters kept in BF16. - Vision tower:
vision_weight.q4nxis made from d3-ultra's own vision encoder (27 layers, 1152 wide, 5120 out; it differs from stock Qwen3.5-27B's in every tensor and slightly from d3-turbo's), then quantized to Q4_K. It runs on FastFlowLM's existing vision kernels. - Attention numerics (
prefix_bidir): as upstream, the state block attends bidirectionally to itself, each question block attends to the state and to itself, never the other way, and the Gated DeltaNet layers stay causal. The engine runs the state once and the question blocks on its cached keys / values and recurrent state; a question's answer does not depend on the other questions of the request (the header, which lists them, is part of the state).usage.input_tokenscounts the state once plus every question block, as upstream. - Engine: none; d3:ultra runs on the same libraries and the same four kernels (
mm,attn,vision_mm,vision_attn) as d3:turbo. - Unchanged:
readout.safetensors,decision_config.json,tokenizer.json,tokenizer_config.json,chat_template.jinja,LICENSEare the upstream files byte for byte (MANIFEST.sha256);NOTICEadds the modification statement. Prompts, token ids and answer objects follow the upstream runtime, with one known difference: FastFlowLM tokenizes with the model's owntokenizer.json, while upstream'stransformerstokenizer drops the combining-mark class from its pre-tokenizer, so text with combining marks (for example an emoji variation selector) can have one token more upstream (3 of 212 text test requests, one token each).
Accuracy (against the upstream BF16 model)
All comparisons use the same requests: the upstream runtime in BF16 as the reference (run on an AMD iGPU, the vision tower of the image references on the CPU or in chunked BF16 on the GPU) and this package on the NPU.
| this package on the NPU | upstream BF16 | |
|---|---|---|
| Final-norm hidden state vs BF16, cosine (736 text questions, median / minimum) | 0.9985 / 0.919 | -- |
| Probability error vs BF16 (736 text questions, median / 99th percentile / maximum) | 0.006 / 0.22 / 0.95 | -- |
| Same top answer as BF16 (736 text questions) | 95.1% | -- |
| Same, 5 long prompts of 6k to 24k tokens (cosine median) | 5 / 5 (0.9988) | -- |
| Same, 2 prompts of 32,767 and 32,768 tokens (cosine) | 2 / 2 (0.9988) | -- |
| Same, 60 questions with images (cosine median) | 91.7% (0.9972) | -- |
The plain Q4_K quantization of the same model without GPTQ agreed on 91.5% of the text questions (cosine 0.9955), so GPTQ is what makes this package usable. Rounding noise of 4-bit weights accumulates over 64 layers, so this 27B model sits at the level of d3:turbo (95.7%) and d3:nano (95.7%) on this measure. The text set is deliberately hard (many near-ties); on the Decision Index rows below the agreement is 99%. The weakest items of the set are Japanese-language prompts (37 of 42 questions of the 20 multilingual requests agree, with the plain Q4_K package as well): non-English input is the part of the model that quantization hurts most. A request with one question is answered at temperature 1.046, as upstream.
| Decision Index public-suite subset (100 rows, same file and order as the d3:lite / nano / mini / turbo samples) | this package on the NPU | upstream BF16 (same requests) |
|---|---|---|
| top-1 agreement with BF16 on the same requests (716 choice answers) | 99.0% (709 / 716; 95% CI 98.0 to 99.5) | -- |
| accuracy against the expected labels (593 labelled choice answers) | 95.1% | 94.8% |
Accuracy, BF16 minus this package: -0.3 pt (95% CI -1.3 to +0.5 over the 100 rows). The mean absolute probability difference is 0.0060. The subset is the first 100 rows of the 4,002-row seeded sample (round-robin over 43 benchmarks; about 2 rows per benchmark, so per-benchmark numbers are not meaningful; one row of 32 questions and a 31k-token state does not fit the 80 GiB iGPU in BF16 and is left out). It is a small check, not a replacement for the full suite.
| Decision Index public vision suite subset (600 questions, 10 benchmarks x 60) | this package on the NPU | upstream BF16 |
|---|---|---|
| accuracy | 77.8% | 80.7% |
| top-1 agreement with BF16 on the same questions | 85.7% (514 / 600) | -- |
Paired accuracy, BF16 minus this package: +2.8 pt (95% CI +0.0 to +5.7; d3:turbo +2.0 pt, d3:mini +2.5 pt on the same rows). The largest gaps are in BLINK (75.0% vs 83.3%) and Mind2Web (83.3% vs 88.3%); Hateful Memes moderation is near chance for both models and flips between them (61.7% vs 66.7%, 42% agreement). The image rows are limited by the vision tower: on FastFlowLM's current vision kernels the features of the d3:turbo tower differ from the BF16 tower's by a relative L2 error of about 0.3 (measured there; d3-ultra's tower is a slight further fine-tune of it). These 600 rows are the sample used for d3:turbo, so the numbers are comparable.
Box and track questions against the BF16 answers of the same requests: box on images (29 requests, 99 boxes) median edge
difference 0.015 of the image size and median IoU 0.86 with BF16's box; assets (15 requests) top-1 100%, median IoU 0.95;
box at a video moment (11 requests, 22 boxes) edge 0.024 and IoU 0.77; track (13 requests, 120 moments) edge 0.017, IoU 0.83,
visible decision agrees on all; plain questions about videos (45 answers) top-1 97.8%. A few boxes of the image set miss the
reference object entirely (the 5th percentile of the IoU is about 0), a confident box on the wrong object.
Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM)
Measured over HTTP (flm serve --decision d3:ultra, this GPTQ package, FastFlowLM pull request build with the CI-built
libraries and xclbins, nothing else running on the NPU): median of 5 requests after a warm-up request (one request for 32k);
"tokens" is the usage.input_tokens of the request, the header included.
| request | tokens | latency |
|---|---|---|
| 1 question, short state | 241 | 3.10 s (2.96 to 3.10) |
| 1 question | 441 | 3.43 s (3.42 to 3.44) |
| 1 question | 1,131 | 9.50 s (9.43 to 9.51) |
| 1 question | 4,201 | 30.6 s (30.5 to 30.6) |
| 1 question | 32,750 | 314 s |
| 4 questions about one 330-token state | 717 | 6.33 s (6.30 to 6.36) |
| 16 questions about one 330-token state | 1,688 | 12.8 s (12.7 to 12.9) |
The floor is about 3 s per request (the 15.5 GB of weights are streamed from memory once per pass, so a short question is memory-bound; d3:mini needs 0.5 s for the same question), then roughly 5 to 10 ms per token. The header makes the prompts a little longer than d3:turbo's for the same request (a request lists its questions once), so the same text costs somewhat more than the d3:turbo numbers. State once gives bit-identical answers to the one-pass-per-question path (checked on 55 requests / 389 answers). Over HTTP the 736 text questions of the accuracy set (212 requests) took 3.3 s per request at the median.
Requests on the same machine, one pass over the accuracy request sets (GPTQ package, medians):
| request set | median latency (range) |
|---|---|
| plain question with 1 to 2 images (31 requests) | 4.1 s (max 15.7 s) |
box questions about one image (1 to 8 boxes, 29 requests) |
11.4 s (max 19.6 s) |
assets (images in options or instructions, 15 requests) |
27.5 s (max 113 s) |
| a plain question about a video in the state (11 requests) | 18.3 s (max 26.6 s) |
box at one moment of a video (11 requests) |
10.4 s (max 41 s) |
track (13 requests: clips of 4 to 11 s) |
11.1 s (max 103 s) |
Soak: 100 mixed requests (text, image and long-prompt requests, several questions per request, empty and malformed bodies) first one after the other, then the same schedule from 4 concurrent clients next to a streaming chat model: 0 status and 0 answer mismatches between the two runs (91 answered, 9 refused with 422 as expected), no crash; resident memory of the server 18.1 GB after the load, a peak of 26.1 GB, 19.6 GB afterwards.
Comparison
- Against d3:turbo (same engine): upstream reports (its own private tests) SR-Text 87.2 for d3-ultra against 83.1 for d3-turbo and SR-Box 58.2% against 60.0%. The NPU port costs the same kind of quantization noise as d3:turbo: 95.1% top-1 agreement on the hard text set (turbo 95.7%) and 99% on Decision Index rows; the vision gap to BF16 is +2.8 pt (turbo +2.0). Speed and memory are those of d3:turbo; the header adds tokens to a request.
- Against the smaller NPU siblings: d3:mini answers a 330-token question in 0.5 s and d3:nano in 0.25 s; d3:ultra takes about 3.4 s, but reads a state once for any number of questions (the NPU siblings in FastFlowLM read the prompt again for every question).
- Against the BF16 model on the same requests: see Accuracy. The quantization costs a few points of top-1 agreement on hard items, more than for d3:mini, and the image path costs somewhat more than text.
License
Apache-2.0, as the upstream model; see LICENSE and NOTICE. Built on Qwen/Qwen3.5-27B (Apache-2.0). Quantized and
repackaged for FastFlowLM by the FastFlowLM team; the readout, decision config, tokenizer and chat template are unmodified.
Citation
Please cite the upstream model (vllm-sr/d3-ultra):
@misc{d3ultra_2026,
title = {{d3-ultra}: A Multimodal Decision Model},
author = {{vLLM Semantic Router Team}},
year = {2026},
howpublished = {\url{https://huggingface.co/vllm-sr/d3-ultra}}
}