| --- |
| license: apache-2.0 |
| base_model: vllm-sr/d3-flash |
| pipeline_tag: zero-shot-classification |
| tags: |
| - decision-model |
| - system-one |
| - fastflowlm |
| - amd-npu |
| - q4nx |
| --- |
| |
| # D3-Flash-NPU2 |
|
|
| [vllm-sr/d3-flash](https://huggingface.co/vllm-sr/d3-flash) (Decision 3.0, the decision model of vLLM Semantic |
| Router; Qwen3.5-9B text tower, revision `13dc1cc8`) for AMD Ryzen AI NPUs, served by |
| [FastFlowLM](https://github.com/ROCm/FastFlowLM) 1.0.12 or later. It answers *System One* requests -- choice, yes / no |
| and score questions about a state, optionally with images, with a probability for every option -- over the same |
| `POST /v1/systemone` API as the upstream package. Up to 32,768 input tokens per question. Video is not supported (a |
| request with `videos` is refused with HTTP 422). |
|
|
| ```shell |
| flm serve --decision d3:flash # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:flash |
| curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \ |
| -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice", |
| "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}' |
| ``` |
|
|
| Images go in an `images` list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most |
| 1.6 MP), shared by every question of the request. `d3:lite` (0.8B), `d3:nano` (2B) and `d3:mini` (4B) are the smaller |
| siblings. |
|
|
| ## What was changed |
|
|
| - **Weights:** the Qwen3.5-9B text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with |
| GPTQ-calibrated rounding (calibration: 192 synthetic System One prompts of 100 to 2,048 tokens, none from the Decision |
| Index), token embedding Q8_0, norms and the Gated DeltaNet parameters unquantized. `vision_weight.q4nx` holds the |
| vision tower (identical to Qwen3.5-9B's, merger included), used for requests with images. |
| - **Unchanged:** `readout.safetensors`, `decision_config.json`, `tokenizer.json`, `tokenizer_config.json`, |
| `chat_template.jinja` and `LICENSE` are the upstream files byte for byte (`MANIFEST.sha256`). |
| - **Same contract:** prompts, token ids and answer objects follow the upstream runtime. |
|
|
| ## Accuracy (against the upstream BF16 model) |
|
|
| | | this package on the NPU | upstream BF16 | |
| |---|---|---| |
| | Decision Index public-suite sample, mean benchmark score (1,600 requests, 37 benchmarks) | 67.4% | 67.8% | |
| | Top-1 agreement with BF16 on the same requests (4,452 choice answers) | 97.0% | -- | |
| | Long prompts, 835 to 31,545 tokens (8 requests), same top answer as BF16 | 8 of 8 | -- | |
| | Decision Index public vision suite subset, accuracy (300 questions, 10 benchmarks) | 72.7% | 75.0% | |
| | Top-1 agreement with BF16 on the same image questions | 90.0% | -- | |
|
|
| Mean benchmark score, this package minus BF16: -0.36 pt (standard error 0.66 pt; about 41 requests per benchmark, so |
| single benchmarks are noisy). The sample is a seeded round-robin over the public suite's requests whose prompts are at |
| most 8,000 tokens (37 of its 40 benchmarks have such requests); RouterBench and SGD requests are left out because their |
| locally rebuilt payloads differ from the lab's. The BF16 scores are the upstream BF16 run's own answers on the same |
| requests. The vision subset is 30 seeded questions from each of 10 benchmarks (the suite without the gated Winoground); |
| paired accuracy, BF16 minus this package: +2.3 pt (95% CI -0.9 to +5.6 pt), so images may cost a little accuracy |
| while the answers agree with BF16 on 90.0% of the questions. |
|
|
| ## Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.12) |
|
|
| | request | median latency | |
| |---|---| |
| | 1 question, 128 tokens | 0.85 s | |
| | 1 question, 330 tokens | 1.0 s | |
| | 1 question, 1,024 tokens | 2.3 s | |
| | 1 question, 4,096 tokens | 9.0 s | |
| | 1 question, 16,383 tokens | 41 s | |
| | 1 question, 31,545 tokens | 92 s | |
| | 1 question, 32,768 tokens | 96 s | |
| | 4 questions, 330 tokens each | 3.1 s | |
| | 16 questions, 330 tokens each | 11.7 s | |
| | 1 question + a 1024 x 768 image (880 tokens, HTTP) | 3.3 s | |
| | 1 question + two images (HTTP) | 5.2 s | |
| | 1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP) | 7.0 s | |
| | 8 questions + one image (one image encoding, HTTP) | 18.3 s | |
|
|
| From 1,536 tokens attention runs on the NPU's attention kernel. Prompts over 32,768 tokens are answered with |
| `max_length_exceeded` (nothing is truncated). Resident memory of the server: 6.8 GB after loading, up to 14.1 GB at |
| 32,768 tokens with images in the mix. Median of 3 (16,383 tokens and longer: 1 to 2 runs) over HTTP on an otherwise idle |
| machine. |
|
|
| ## License |
|
|
| Apache-2.0, as the upstream model; see `LICENSE` and `NOTICE`. Built on Qwen/Qwen3.5-9B. |
|
|
