| --- |
| license: apache-2.0 |
| base_model: vllm-sr/d3-mini |
| pipeline_tag: zero-shot-classification |
| tags: |
| - decision-model |
| - system-one |
| - fastflowlm |
| - amd-npu |
| - q4nx |
| --- |
| |
| # D3-Mini-NPU2 |
|
|
| [vllm-sr/d3-mini](https://huggingface.co/vllm-sr/d3-mini) `v3.1.3` (Decision 3.0, the decision model of vLLM Semantic |
| Router; Qwen3.5-4B text tower) for AMD Ryzen AI NPUs, served by [FastFlowLM](https://github.com/ROCm/FastFlowLM) 1.0.11 or later. |
| It answers *System One* requests -- choice, yes / no and score questions about a state, optionally with images, with a |
| probability for every option -- over the same `POST /v1/systemone` API as the upstream package. Up to 32,768 input |
| tokens per question. Video is not supported (a request with `videos` is refused with HTTP 422). |
|
|
| ```shell |
| flm serve --decision d3:mini # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:mini |
| curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \ |
| -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice", |
| "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}' |
| ``` |
|
|
| Images go in an `images` list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most |
| 1.6 MP), shared by every question of the request. `d3:lite` (0.8B) and `d3:nano` (2B) are the smaller siblings. |
|
|
| ## What was changed |
|
|
| - **Weights:** the Qwen3.5-4B text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with |
| GPTQ-calibrated rounding (calibration: 300 synthetic System One questions of up to 2k tokens, none from the Decision |
| Index), token embedding Q8_0, norms and the Gated DeltaNet parameters unquantized. `vision_weight.q4nx` holds the |
| vision tower (identical to Qwen3.5-4B's, merger included), used for requests with images. |
| - **Unchanged:** `readout.safetensors`, `decision_config.json`, `tokenizer.json`, `tokenizer_config.json`, |
| `chat_template.jinja` and `LICENSE` are the upstream files byte for byte (`MANIFEST.sha256`). |
| - **Same contract:** prompts, token ids and answer objects follow the upstream runtime. |
|
|
| ## Accuracy (against the upstream BF16 model) |
|
|
| | | this package on the NPU | upstream BF16 | |
| |---|---|---| |
| | Decision Index public-suite subset, mean benchmark score (1,600 requests, 42 benchmarks) | 62.8% | 63.8% | |
| | Top-1 agreement with BF16 on the same requests (12,656 choice answers) | 97.0% | -- | |
| | Final-norm hidden state vs BF16, cosine (725 text answers, median) | 0.9994 | -- | |
| | Same top answer as BF16 (735 text answers / 60 image answers) | 96.3% / 96.7% | -- | |
| | Decision Index public vision suite subset, accuracy (600 questions, 10 benchmarks) | 69.2% | 71.7% | |
| | Top-1 agreement with BF16 on the same image questions | 89.8% | -- | |
|
|
| Mean benchmark score, this package minus BF16: -0.99 pt (standard error 0.57 pt; about 37 requests per benchmark, so |
| single benchmarks are noisy). The subset is a seeded round-robin sample over the suite's benchmarks, not the full |
| suite. The vision subset is 60 seeded questions from each of 10 benchmarks (the suite without the gated Winoground); |
| paired accuracy, BF16 minus this package: +2.5 pt (95% CI +0.5 to +4.5 pt), so images cost a little accuracy |
| (largest gaps on Mind2Web, MMMU-Pro, CV-Bench and CharXiv, each about 60 questions) while the answers agree with BF16 |
| on 89.8% of the questions. |
|
|
| ## Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.11) |
|
|
| | request | median latency | |
| |---|---| |
| | 1 question, 128 tokens | 473 ms | |
| | 1 question, 330 tokens | 536 ms | |
| | 1 question, 1,024 tokens | 1.3 s | |
| | 1 question, 4,096 tokens | 5.3 s | |
| | 1 question, 16,116 tokens | 25.9 s | |
| | 1 question, 31,545 tokens | 62.9 s | |
| | 1 question, 32,768 tokens | 66.5 s | |
| | 4 questions, 330 tokens each | 1.8 s | |
| | 16 questions, 330 tokens each | 6.6 s | |
| | 1 question + a 1024 x 768 image (876 tokens, HTTP) | 2.1 s | |
| | 1 question + two images (HTTP) | 3.8 s | |
| | 1 question + a 2000 x 1500 image (downscaled to 1.6 MP, HTTP) | 4.4 s | |
| | 8 questions + one image (one image encoding, HTTP) | 10.8 s | |
|
|
| From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is as fast or faster). Prompts over |
| 32,768 tokens are answered with `max_length_exceeded` (nothing is truncated). Peak resident memory of the server at |
| 32,768 tokens: 8.6 GB. Text rows were measured at the engine (HTTP adds about a millisecond); the image rows over HTTP, one question unless noted, median of 3, on an otherwise idle machine. |
|
|
| ## License |
|
|
| Apache-2.0, as the upstream model; see `LICENSE` and `NOTICE`. Built on Qwen/Qwen3.5-4B. |
|
|
