| # NPU vision offload — the multimodal front-end belongs on the NPU |
|
|
| **Thesis:** for a multimodal model (image + text), put the **vision front-end on the XDNA1 |
| NPU** and keep text generation on the CPU/iGPU. This is the NPU's *designed* purpose — Ryzen AI |
| NPUs exist for edge vision/CNN inference — so unlike text decode (memory-bound, where the NPU |
| loses), the vision path is **compute-bound conv work the AIE array is built for.** |
|
|
| ## Why vision, not text |
|
|
| | | text decode | vision front-end | |
| |---|---|---| |
| | bottleneck | memory bandwidth | compute (convs/matmuls) | |
| | NPU fit | poor (M0/M0.5: NPU loses) | **strong** (AIE = CNN engine) | |
| | parallelism | low (gemv) | high (patches/channels) | |
| | concurrency | — | runs while CPU/iGPU do the LLM | |
| | power | — | ~6.6 W | |
|
|
| The NPU stops being a reluctant matmul unit and becomes what it was made for. |
|
|
| ## Proven on silicon (this repo) |
|
|
| A full image-processing pipeline runs on the gen-1 NPU today — `programming_examples/vision/ |
| edge_detect` (`rgba2gray → 3×3 Laplacian filter2d → threshold`), verified vs an OpenCV reference: |
|
|
| ``` |
| $ python3 edge_detect.py -W 512 -H 512 |
| Number of differences: 1942, average L1 error: 0.96 |
| PASS! |
| ``` |
|
|
| IRON ships the vision/CNN kernel set the front-end needs: `conv2dk1`, `conv2dk3` (incl |
| depthwise + ReLU + pooling variants), `conv2dk1_i8`, `filter2d`, `rgba2gray`, `gray2rgba`, |
| `rgba2hue`, plus `softmax`/`gelu`/`silu` for the encoder blocks. These are exactly the ops a |
| ViT/CNN image tower (patch-embed conv, conv stem, channel projections) is made of. |
|
|
| ## The hook (proposed integration) |
|
|
| For a multimodal model with a vision encoder (PaliGemma / Gemma-vision / LLaVA-style towers): |
| 1. **Image preprocessing + patch-embed conv → NPU** (`conv2dk1`/`conv2dk3`/`filter2d`). |
| 2. **Vision-encoder early conv/projection stages → NPU**, streaming patch tiles. |
| 3. Hand off the image embeddings to the text path: **decode → Radeon 780M, prefill → CPU.** |
| 4. The NPU also runs the **prune/collapse coprocessor** on attention/FFN (see TIER3_DESIGN.md). |
| |
| Net: a 4-way split on the integrated SoC — **NPU = vision + prune, 780M = decode, CPU = |
| prefill** — all over the 96 GB unified pool, no discrete GPU. |
| |
| ## Honest status |
| |
| - ✅ **Proven**: conv / 2D-filter / colorspace vision kernels run on the NPU (edge_detect PASS). |
| These are the building blocks of a vision tower. |
| - ✅ **SigLIP/ViT patch-embed wired + measured** (`examples/npu_patch_embed.py`): the patch-projection |
| matmul ([256×768]@[768×768], SigLIP-base/16) runs on the NPU, **bit-exact**. HONEST measurement: at |
| base size the NPU (int16, 6.6 ms / 46 GFLOP/s, dispatch-bound) is **~3.2× SLOWER than fair CPU f32 |
| BLAS** (2.0 ms / 148 GFLOP/s). The win is **offload + ~6.6 W + scaling** (frees CPU/iGPU for the LLM, |
| gap closes at larger image/batch), **not raw latency** at SigLIP-base. Don't quote int32-numpy as the |
| CPU baseline — it's unoptimized (~1.5 GFLOP/s) and makes the NPU look 34× faster, which is false. |
| - 🚧 **Not yet done**: full multi-stage encoder (conv stem + transformer blocks) + a batched/larger-image |
| regime where the NPU's offload+power actually nets ahead; measure end-to-end encoder latency & SoC power. |
| - The "boost" is expected to be real *because* this is the NPU's native workload — but it is a |
| projection from the proven primitives until a full vision tower is wired and measured. |
|
|
