D3-Nano-NPU2

vllm-sr/d3-nano v3.1.3 (Decision 3.0, the decision model of vLLM Semantic Router; Qwen3.5-2B text tower) for AMD Ryzen AI NPUs, served by FastFlowLM 1.0.11+. It answers System One requests -- choice, yes / no and score questions about a state, optionally with images, with a probability for every option -- over the same POST /v1/systemone API as the upstream package. Up to 32,768 input tokens per question.

flm serve --decision d3:nano          # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:nano
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
  -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
       "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'

Images go in an images list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most 1.6 MP), shared by every question of the request. d3:lite is the smaller sibling (0.8B).

What was changed

Weights: the Qwen3.5-2B text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with GPTQ-calibrated rounding (calibration: 300 synthetic System One questions of up to 2k tokens, none from the Decision Index), token embedding Q8_0, norms and the Gated DeltaNet parameters unquantized. vision_weight.q4nx holds the vision tower (Qwen3.5-2B's blocks with d3-nano's own merger), used for requests with images.
Unchanged: readout.safetensors, decision_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja and LICENSE are the upstream files byte for byte (MANIFEST.sha256).
Same contract: prompts, token ids and answer objects follow the upstream runtime.

Accuracy (against the upstream BF16 model)

	this package on the NPU	upstream BF16
Decision Index public-suite subset, mean benchmark score (3,240 requests, 42 benchmarks)	55.5%	56.7%
Top-1 agreement with BF16 on the same requests (24,956 choice answers)	96.5%	--
Final-norm hidden state vs BF16, cosine (735 text answers, median)	0.998	--
Same top answer as BF16 (735 text answers / 60 image answers)	95.8% / 93.3%	--
Decision Index public vision suite subset, accuracy (600 questions, 10 benchmarks)	64.0%	64.3%
Top-1 agreement with BF16 on the same image questions	84.3%	--

Mean benchmark score, this package minus BF16: -1.25 pt (standard error 0.58 pt; about 65-80 requests per benchmark, so single benchmarks are noisy). The subset is a seeded round-robin sample over the suite's benchmarks, not the full suite. The vision subset is 60 seeded questions from each of 10 benchmarks (the suite without the gated Winoground); paired accuracy, BF16 minus this package: +0.3 pt (95% CI [-2.3, +2.8] pt).

Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.11)

request	median latency
1 question, 128 tokens	210 ms
1 question, 330 tokens	252 ms
1 question, 1,024 tokens	584 ms
1 question, 4,096 tokens	2,470 ms
1 question, 15,854 tokens	11.4 s
1 question, 31,491 tokens	27.1 s
1 question, 32,768 tokens	28.4 s
4 questions, 330 tokens each	796 ms
16 questions, 330 tokens each	2,889 ms
1 question + a 1024 x 768 image (876 tokens, HTTP)	1.3 s
1 question + two images (HTTP)	2.4 s
1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP)	3.0 s
8 questions + one image (one image encoding, HTTP)	5.0 s

From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is faster; at 31,491 tokens it is 3.3x faster than attention on the CPU). Prompts over 32,768 tokens are answered with max_length_exceeded (nothing is truncated). Text rows were measured at the engine (HTTP adds about a millisecond); the image rows over HTTP, one question, median of 3, on an otherwise idle NPU.