README.md

---
license: apache-2.0
base_model: vllm-sr/d3-lite
pipeline_tag: zero-shot-classification
tags:
- decision-model
- system-one
- fastflowlm
- amd-npu
- q4nx
---

# D3-Lite-NPU2

[vllm-sr/d3-lite](https://huggingface.co/vllm-sr/d3-lite) `v3.1.3` (Decision 3.0, the decision model of vLLM Semantic
Router) for AMD Ryzen AI NPUs, served by [FastFlowLM](https://github.com/ROCm/FastFlowLM) 1.0.11+.
It answers *System One* requests -- choice, yes / no and score questions about a state, optionally with images, with a
probability for every option -- over the same `POST /v1/systemone` API as the upstream package. Up to 32,768 input
tokens per question.

```shell
flm serve --decision d3:lite          # optionally next to a chat model: flm serve qwen3.5:4b --decision d3:lite
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" \
  -d '{"state": "The order arrived damaged.", "questions": {"route": {"type": "choice",
       "criteria": {"returns": "Refunds and replacements", "billing": "Payments", "general": null}}}}'
```

Images go in an `images` list of base64 data URLs (PNG, JPEG or WebP; each at most 8 MB and 16 MP, resized to at most
1.6 MP), shared by every question of the request:

```shell
curl -sLo panda.png https://raw.githubusercontent.com/ROCm/FastFlowLM/main/src/tb_files/panda.png
printf '{"questions": {"animal": {"type": "choice", "instructions": "Which animal is in the photo?",
  "criteria": {"dog": "A dog", "panda": "A panda", "cat": "A cat"}}},
  "images": ["data:image/png;base64,%s"]}' "$(base64 -w0 panda.png)" > image_request.json
curl -s http://127.0.0.1:52625/v1/systemone -H "Content-Type: application/json" -d @image_request.json
# ... "choice":"panda", "panda":0.999 ...
```

More in the [FastFlowLM d3 guide](https://fastflowlm.com/docs/models/d3/).

## What was changed

- **Weights:** the Qwen3.5 text tower quantized to FastFlowLM's Q4NX format: every projection Q4_K with
  GPTQ-calibrated rounding (calibration: 300 Decision Index prompts, none of them in the evaluation
  sets below), token embedding Q8_0, norms and the Gated DeltaNet parameters unquantized. `vision_weight.q4nx` holds
  the vision tower (BF16), used from FastFlowLM 1.0.11 for requests with images.
- **Unchanged:** `readout.safetensors`, `decision_config.json`, `tokenizer.json`, `tokenizer_config.json`,
  `chat_template.jinja` and `LICENSE` are the upstream files byte for byte (`MANIFEST.sha256`).
- **Same contract:** prompts, token ids and answer objects are byte-identical to the upstream runtime (checked on every
  FastFlowLM build against 1,365 upstream-rendered requests, 57 of them with images); image grids and input token
  counts equal upstream's, and the image pixels are within 2 grey levels of the upstream processor's.

## Accuracy (against the upstream BF16 model)

| | this package on the NPU | upstream BF16 |
|---|---|---|
| Decision Index public-suite subset, gold accuracy (6,824 questions, 35 benchmarks) | 60.5% | 61.2% |
| Top-1 agreement with BF16 on the same questions (7,226) | 90.6% | -- |
| Decision Index public vision suite, gold accuracy (600 questions, 10 benchmarks) | 61.7% | 60.2% |
| Top-1 agreement with BF16 on the same image questions | 84.5% | -- |

Paired gold accuracy, upstream BF16 minus this package: text +0.67 pt (95% CI [+0.08, +1.27] pt); images -1.50 pt
(95% CI [-3.92, +0.92] pt). The vision suite leaves out Winoground (gated); per-benchmark numbers are on the
[FastFlowLM d3 benchmark page](https://fastflowlm.com/docs/benchmarks/d3_results/).

## Speed (AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FastFlowLM 1.0.11)

| request | median latency |
|---|---|
| 1 question, 128 tokens | 109 ms |
| 1 question, 330 tokens | 151 ms |
| 1 question, 1,024 tokens | 430 ms |
| 1 question, 4,096 tokens | 1,587 ms |
| 1 question, 16,384 tokens | 8,425 ms |
| 1 question, 32,768 tokens | 22,696 ms |
| 4 questions, 330 tokens each | 536 ms |
| 16 questions, 330 tokens each | 2,067 ms |
| 1 question + a 640 x 480 image (398 tokens) | 393 ms |
| 1 question + a 1.6 MP image (1,698 tokens) | 1,529 ms |
| 8 questions + one 1.6 MP image (one image encoding) | 6,368 ms |

From 1,536 tokens attention runs on the NPU's attention kernel (below that the CPU is faster). Prompts over 32,768
tokens are answered with `max_length_exceeded` (nothing is truncated). Image rows were measured over HTTP; the others
at the engine (HTTP adds about a millisecond).

## License

Apache-2.0, as the upstream model; see `LICENSE` and `NOTICE`.