File size: 4,615 Bytes
9e1c704 5264f19 9e1c704 5264f19 9e1c704 066fd62 9e1c704 5264f19 9e1c704 5264f19 9e1c704 5264f19 9e1c704 5264f19 9e1c704 066fd62 47d7a6d 066fd62 9e1c704 5264f19 066fd62 5264f19 9e1c704 26a2203 9e1c704 5264f19 9e1c704 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 |
---
license: apache-2.0
language:
- en
base_model:
- princeton-vl/RAFT-Stereo
pipeline_tag: depth-estimation
tags:
- Axera
- RAFT-Stereo
- NPU
- Stereo-Matching
- Depth-Estimation
- Computer-Vision
---
# RAFT-Stereo on Axera NPU
This repository contains the [RAFT-Stereo](https://arxiv.org/abs/2109.07547) model converted for high-performance inference on the Axera NPU.
This version is optimized with **w8a16** quantization and is compatible with **Pulsar2 version 4.2**.
## Convert Tools Links
For model conversion and deployment guidance:
* [AXera Platform GitHub Repo](https://github.com/AXERA-TECH/RAFT-Stereo.axera): Sample code and optimization guides for Axera NPU.
* [Pulsar2 Documentation](https://pulsar2-docs.readthedocs.io/en/latest/pulsar2/introduction.html): Guide for converting ONNX models to `.axmodel`.
## Support Platforms
- **AX650N/AX8850**
- [M4N-Dock(η±θ―ζ΄ΎPro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
- [M.2 Accelerator card](https://docs.m5stack.com/en/ai_hardware/LLM-8850_Card)
- **AX630C**
- [η±θ―ζ΄Ύ2](https://axera-pi-2-docs-cn.readthedocs.io/zh-cn/latest/index.html)
- [Module-LLM](https://docs.m5stack.com/zh_CN/module/Module-LLM)
- [LLM630 Compute Kit](https://docs.m5stack.com/zh_CN/core/LLM630%20Compute%20Kit)
- **AX637**
| Chips | Model Variant | Latency (ms) |
| --- | --- | --- |
| AX650 | raft_steoro256x640_r1.axmodel | 20.9 |
| AX650 | raft_steoro256x640_r4.axmodel | 111.8 |
| Chips | Model Variant | Latency (ms) |
| --- | --- | --- |
| AX630C| raft_steoro256x640_r1_npu2.axmodel | 317.765 |
| AX630C| raft_steoro256x640_r4_npu2.axmodel | 825.793 |
| Chips | Model Variant | Latency (ms) |
| --- | --- | --- |
| AX637| raft_steoro256x640_r1_npu1.axmodel | 61.033 |
| AX637| raft_steoro256x640_r4_npu1.axmodel | 99.094 |
## How to Use
Download the repository and ensure the directory structure is organized as follows:
```text
βββ ax630c
β βββ raft_steoro256x640_r1.onnx
β βββ raft_steoro256x640_r1_npu2.axmodel
β βββ raft_steoro256x640_r4.onnx
β βββ raft_steoro256x640_r4_npu2.axmodel
βββ ax650
β βββ raft_steoro256x640_r1.axmodel
β βββ raft_steoro256x640_r1.onnx
β βββ raft_steoro384x1280_r4.axmodel
β βββ raft_steoro384x1280_r4.onnx
βββ ax637
β βββ raft_steoro256x640_r1_npu1.axmodel
β βββ raft_steoro256x640_r4_npu1.axmodel
βββ examples
β βββ left
β β βββ 000051_11.png
β β βββ 000058_11.png
β β βββ 000059_10.png
β β βββ 000121_10.png
β β βββ 000164_10.png
β β βββ 000167_11.png
β β βββ 000172_11.png
β β βββ 000179_10.png
β β βββ 000193_10.png
β β βββ 000195_10.png
β βββ right
β βββ 000051_11.png
β βββ 000058_11.png
β βββ 000059_10.png
β βββ 000121_10.png
β βββ 000164_10.png
β βββ 000167_11.png
β βββ 000172_11.png
β βββ 000179_10.png
β βββ 000193_10.png
β βββ 000195_10.png
βββ infer.py
βββ infer_onnx.py
```
### Prerequisites
1. **Environment:** Ensure you have the required Python environment activated with the following core packages installed:
* **NPU Runtime:** [`axengine` (PyAXEngine)](https://github.com/AXERA-TECH/pyaxengine)
* **Core Libraries:** `numpy`, `opencv-python`, and `matplotlib`.
2. **Model/Data:** Ensure the compiled `.axmodel`, and input stereo pairs are available on the host.
### Inference Command
Run the inference script by providing the compiled model, and input data.
```bash
python3 infer.py --left examples/left/000051_11.png --right examples/right/000051_11.png --model raft_steoro256x640_r1_npu2.axmodel --width 640 --height 256
```
### Inference with AX630C Host
### Results
The model generates a dense disparity map representing the depth of the scene.
```
(base) root@ax630c:~/data# /python3 infer.py --left examples/left/000051_11.png --right examples/right/000051_11.png --model raft_steoro256x640_r4_npu2.axmodel --width 640 --height 256
[INFO] Available providers: ['AxEngineExecutionProvider']
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Chip type: ChipType.MC20E
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Engine version: 2.7.2a
[INFO] Model type: 1 (full core)
[INFO] Compiler version: 5.1-patch1-dirty e0fdc66b-dirty
Saved: output-ax.png
```
### Example Visualization
|