| --- |
| license: apache-2.0 |
| base_model: vllm-sr/d3-lite |
| pipeline_tag: zero-shot-classification |
| tags: |
| - decision-model |
| - system-one |
| - fastflowlm |
| - amd-npu |
| - q4nx |
| --- |
| |
| # D3-Lite-NPU2 |
|
|
| [vllm-sr/d3-lite](https://huggingface.co/vllm-sr/d3-lite) `v3.1.3` (Decision 3.0, the decision model of vLLM Semantic |
| Router) for AMD Ryzen AI NPUs, served by [FastFlowLM](https://github.com/ROCm/FastFlowLM) 1.0.11+. |
| It answers *System One* requests -- choice, yes / no and score questions about a state, optionally with images, with a |
| probability for every option -- over the same `POST /v1/systemone` API as the upstream package. Up to 32,768 input |
| tokens per question. |
|
|
| ```shell |
| flm serve --decision d3:lite # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:lite |
| curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \ |
| -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice", |
| "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}' |
| ``` |
|
|
| Images go in an `images` list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most |
| 1.6 MP), shared by every question of the request: |
|
|
| ```shell |
| curl -sLo panda.png https://raw.githubusercontent.com/ROCm/FastFlowLM/main/src/tb_files/panda.png |
| printf '{"questions": {"animal": {"type": "choice", "instructions": "Which animal is in the photo?", |
| "criteria": {"dog": "A dog", "panda": "A panda", "cat": "A cat"}}}, |
| "images": ["data:image/png;base64,%s"]}' "$(base64 -w0 panda.png)" > image_request.json |
| curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" -d @image_request.json |
| # ... "choice":"panda", "panda":0.999 ... |
| ``` |
|
|
| More in the [FastFlowLM d3 guide](https://fastflowlm.com/docs/models/d3/). |
|
|
| ## What was changed |
|
|
| - **Weights:** the Qwen3.5 text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with |
| GPTQ-calibrated rounding (calibration: 300 Decision Index prompts, none of them in the evaluation |
| sets below), token embedding Q8_0, norms and the Gated DeltaNet parameters unquantized. `vision_weight.q4nx` holds |
| the vision tower (BF16), used from FastFlowLM 1.0.11 for requests with images. |
| - **Unchanged:** `readout.safetensors`, `decision_config.json`, `tokenizer.json`, `tokenizer_config.json`, |
| `chat_template.jinja` and `LICENSE` are the upstream files byte for byte (`MANIFEST.sha256`). |
| - **Same contract:** prompts, token ids and answer objects are byte-identical to the upstream runtime (checked on every |
| FastFlowLM build against 1,365 upstream-rendered requests, 57 of them with images); image grids and input token |
| counts equal upstream's, and the image pixels are within 2 grey levels of the upstream processor's. |
|
|
| ## Accuracy (against the upstream BF16 model) |
|
|
| | | this package on the NPU | upstream BF16 | |
| |---|---|---| |
| | Decision Index public-suite subset, gold accuracy (6,824 questions, 35 benchmarks) | 60.5% | 61.2% | |
| | Top-1 agreement with BF16 on the same questions (7,226) | 90.6% | -- | |
| | Decision Index public vision suite, gold accuracy (600 questions, 10 benchmarks) | 61.7% | 60.2% | |
| | Top-1 agreement with BF16 on the same image questions | 84.5% | -- | |
|
|
| Paired gold accuracy, upstream BF16 minus this package: text +0.67 pt (95% CI [+0.08, +1.27] pt); images -1.50 pt |
| (95% CI [-3.92, +0.92] pt). The vision suite leaves out Winoground (gated); per-benchmark numbers are on the |
| [FastFlowLM d3 benchmark page](https://fastflowlm.com/docs/benchmarks/d3_results/). |
|
|
| ## Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.11) |
|
|
| | request | median latency | |
| |---|---| |
| | 1 question, 128 tokens | 109 ms | |
| | 1 question, 330 tokens | 151 ms | |
| | 1 question, 1,024 tokens | 430 ms | |
| | 1 question, 4,096 tokens | 1,587 ms | |
| | 1 question, 16,384 tokens | 8,425 ms | |
| | 1 question, 32,768 tokens | 22,696 ms | |
| | 4 questions, 330 tokens each | 536 ms | |
| | 16 questions, 330 tokens each | 2,067 ms | |
| | 1 question + a 640 x 480 image (398 tokens) | 393 ms | |
| | 1 question + a 1.6 MP image (1,698 tokens) | 1,529 ms | |
| | 8 questions + one 1.6 MP image (one image encoding) | 6,368 ms | |
|
|
| From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is faster). Prompts over 32,768 |
| tokens are answered with `max_length_exceeded` (nothing is truncated). Image rows were measured over HTTP; the others |
| at the engine (HTTP adds about a millisecond). |
|
|
| ## License |
|
|
| Apache-2.0, as the upstream model; see `LICENSE` and `NOTICE`. |
|
|
