File size: 4,358 Bytes
c5f8dfa 6cb318a c5f8dfa 6cb318a c5f8dfa e144267 c5f8dfa e144267 c5f8dfa e144267 c5f8dfa e144267 c5f8dfa | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | ---
license: apache-2.0
base_model: vllm-sr/d3-nano
pipeline_tag: zero-shot-classification
tags:
- decision-model
- system-one
- fastflowlm
- amd-npu
- q4nx
---
# D3-Nano-NPU2
[vllm-sr/d3-nano](https://huggingface.co/vllm-sr/d3-nano) `v3.1.3` (Decision 3.0, the decision model of vLLM Semantic
Router; Qwen3.5-2B text tower) for AMD Ryzen AI NPUs, served by [FastFlowLM](https://github.com/ROCm/FastFlowLM) 1.0.11+.
It answers *System One* requests -- choice, yes / no and score questions about a state, optionally with images, with a
probability for every option -- over the same `POST /v1/systemone` API as the upstream package. Up to 32,768 input
tokens per question.
```shell
flm serve --decision d3:nano # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:nano
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
-d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
"criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'
```
Images go in an `images` list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most
1.6 MP), shared by every question of the request. `d3:lite` is the smaller sibling (0.8B).
## What was changed
- **Weights:** the Qwen3.5-2B text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with
GPTQ-calibrated rounding (calibration: 300 synthetic System One questions of up to 2k tokens, none from the Decision
Index), token embedding Q8_0, norms and the Gated DeltaNet parameters unquantized. `vision_weight.q4nx` holds the
vision tower (Qwen3.5-2B's blocks with d3-nano's own merger), used for requests with images.
- **Unchanged:** `readout.safetensors`, `decision_config.json`, `tokenizer.json`, `tokenizer_config.json`,
`chat_template.jinja` and `LICENSE` are the upstream files byte for byte (`MANIFEST.sha256`).
- **Same contract:** prompts, token ids and answer objects follow the upstream runtime.
## Accuracy (against the upstream BF16 model)
| | this package on the NPU | upstream BF16 |
|---|---|---|
| Decision Index public-suite subset, mean benchmark score (3,240 requests, 42 benchmarks) | 55.5% | 56.7% |
| Top-1 agreement with BF16 on the same requests (24,956 choice answers) | 96.5% | -- |
| Final-norm hidden state vs BF16, cosine (735 text answers, median) | 0.998 | -- |
| Same top answer as BF16 (735 text answers / 60 image answers) | 95.8% / 93.3% | -- |
| Decision Index public vision suite subset, accuracy (600 questions, 10 benchmarks) | 64.0% | 64.3% |
| Top-1 agreement with BF16 on the same image questions | 84.3% | -- |
Mean benchmark score, this package minus BF16: -1.25 pt (standard error 0.58 pt; about 65-80 requests per benchmark,
so single benchmarks are noisy). The subset is a seeded round-robin sample over the suite's benchmarks, not the full
suite. The vision subset is 60 seeded questions from each of 10 benchmarks (the suite without the gated Winoground);
paired accuracy, BF16 minus this package: +0.3 pt (95% CI [-2.3, +2.8] pt).
## Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.11)
| request | median latency |
|---|---|
| 1 question, 128 tokens | 210 ms |
| 1 question, 330 tokens | 252 ms |
| 1 question, 1,024 tokens | 584 ms |
| 1 question, 4,096 tokens | 2,470 ms |
| 1 question, 15,854 tokens | 11.4 s |
| 1 question, 31,491 tokens | 27.1 s |
| 1 question, 32,768 tokens | 28.4 s |
| 4 questions, 330 tokens each | 796 ms |
| 16 questions, 330 tokens each | 2,889 ms |
| 1 question + a 1024 x 768 image (876 tokens, HTTP) | 1.3 s |
| 1 question + two images (HTTP) | 2.4 s |
| 1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP) | 3.0 s |
| 8 questions + one image (one image encoding, HTTP) | 5.0 s |
From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is faster; at 31,491 tokens it is 3.3x faster than attention on the CPU). Prompts over 32,768
tokens are answered with `max_length_exceeded` (nothing is truncated). Text rows were measured at the engine (HTTP
adds about a millisecond); the image rows over HTTP, one question, median of 3, on an otherwise idle NPU.
## License
Apache-2.0, as the upstream model; see `LICENSE` and `NOTICE`. Built on Qwen/Qwen3.5-2B.
|