NPU vision offload — the multimodal front-end belongs on the NPU
Thesis: for a multimodal model (image + text), put the vision front-end on the XDNA1 NPU and keep text generation on the CPU/iGPU. This is the NPU's designed purpose — Ryzen AI NPUs exist for edge vision/CNN inference — so unlike text decode (memory-bound, where the NPU loses), the vision path is compute-bound conv work the AIE array is built for.
Why vision, not text
| text decode | vision front-end | |
|---|---|---|
| bottleneck | memory bandwidth | compute (convs/matmuls) |
| NPU fit | poor (M0/M0.5: NPU loses) | strong (AIE = CNN engine) |
| parallelism | low (gemv) | high (patches/channels) |
| concurrency | — | runs while CPU/iGPU do the LLM |
| power | — | ~6.6 W |
The NPU stops being a reluctant matmul unit and becomes what it was made for.
Proven on silicon (this repo)
A full image-processing pipeline runs on the gen-1 NPU today — programming_examples/vision/ edge_detect (rgba2gray → 3×3 Laplacian filter2d → threshold), verified vs an OpenCV reference:
$ python3 edge_detect.py -W 512 -H 512
Number of differences: 1942, average L1 error: 0.96
PASS!
IRON ships the vision/CNN kernel set the front-end needs: conv2dk1, conv2dk3 (incl
depthwise + ReLU + pooling variants), conv2dk1_i8, filter2d, rgba2gray, gray2rgba,
rgba2hue, plus softmax/gelu/silu for the encoder blocks. These are exactly the ops a
ViT/CNN image tower (patch-embed conv, conv stem, channel projections) is made of.
The hook (proposed integration)
For a multimodal model with a vision encoder (PaliGemma / Gemma-vision / LLaVA-style towers):
- Image preprocessing + patch-embed conv → NPU (
conv2dk1/conv2dk3/filter2d). - Vision-encoder early conv/projection stages → NPU, streaming patch tiles.
- Hand off the image embeddings to the text path: decode → Radeon 780M, prefill → CPU.
- The NPU also runs the prune/collapse coprocessor on attention/FFN (see TIER3_DESIGN.md).
Net: a 4-way split on the integrated SoC — NPU = vision + prune, 780M = decode, CPU = prefill — all over the 96 GB unified pool, no discrete GPU.
Honest status
- ✅ Proven: conv / 2D-filter / colorspace vision kernels run on the NPU (edge_detect PASS). These are the building blocks of a vision tower.
- ✅ SigLIP/ViT patch-embed wired + measured (
examples/npu_patch_embed.py): the patch-projection matmul ([256×768]@[768×768], SigLIP-base/16) runs on the NPU, bit-exact. HONEST measurement: at base size the NPU (int16, 6.6 ms / 46 GFLOP/s, dispatch-bound) is ~3.2× SLOWER than fair CPU f32 BLAS (2.0 ms / 148 GFLOP/s). The win is offload + ~6.6 W + scaling (frees CPU/iGPU for the LLM, gap closes at larger image/batch), not raw latency at SigLIP-base. Don't quote int32-numpy as the CPU baseline — it's unoptimized (~1.5 GFLOP/s) and makes the NPU look 34× faster, which is false. - 🚧 Not yet done: full multi-stage encoder (conv stem + transformer blocks) + a batched/larger-image regime where the NPU's offload+power actually nets ahead; measure end-to-end encoder latency & SoC power.
- The "boost" is expected to be real because this is the NPU's native workload — but it is a projection from the proven primitives until a full vision tower is wired and measured.