Hugging Face's logo

AXERA-TECH
/
MixFormerV2

ONNX
English
Transformer
Tracking
ONNX
Model card Files
xet
Community
1
MixFormerV2
File size: 4,052 Bytes 
43e6f9d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
f4f94e9
 
 
 
43e6f9d
2ab1513
 
 
 
43e6f9d
 
 
 
 
 
22616e1
43e6f9d
 
 
 
 
346a7f5
43e6f9d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
 
---
license: mit
language:
- en
tags:
- Transformer
- Tracking
- ONNX
---

# MixFormerV2

This version of MixFormerV2 has been converted to run on the Axera NPU using **w8a16** quantization.

This model has been optimized with the following LoRA: 

Compatible with Pulsar2 version: 3.4

## Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through 

- [The repo of original](https://github.com/MCG-NJU/MixFormerV2)

- [The repo of AXera Platform](https://github.com/Jordan-5i/ax650_mixformer2_demo), which you can get the detial of guide

- [Pulsar2 Link, How to Convert ONNX to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/pulsar2/introduction.html) 


## Support Platform

- AX650
  - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
  - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
- AX630C
  - [爱芯派2](https://axera-pi-2-docs-cn.readthedocs.io/zh-cn/latest/index.html)
  - [Module-LLM](https://docs.m5stack.com/zh_CN/module/Module-LLM)
  - [LLM630 Compute Kit](https://docs.m5stack.com/zh_CN/core/LLM630%20Compute%20Kit)
 
|Chips|npu1|
|--|--|
|AX650| 11 ms |
|AX630C| 33 ms |

## How to use

Download all files from this repository to the device

```
root@ax650:/mnt/qtang/MixFormerV2# tree -L 1
.
├── ax650
├── car.avi
├── config.json
├── onnx
├── README.md
├── run_mixformer2_axmodel.py
└── run_mixformer2_onnx.py
```

### python env requirement

#### pyaxengine

https://github.com/AXERA-TECH/pyaxengine

```
wget https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.1rc0/axengine-0.1.1-py3-none-any.whl
pip install axengine-0.1.1-py3-none-any.whl
```

#### others

```
pip install argparse numpy opencv-python glob2
```

#### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro)

```
root@ax650:/mnt/qtang/ax650_mixformer2_demo# python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10
[INFO] Available providers:  ['AxEngineExecutionProvider']
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Chip type: ChipType.MC50
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Engine version: 2.7.2a
[INFO] Model type: 0 (single core)
[INFO] Compiler version: 3.4-dirty 4ff37520-dirty
====================type================= [1079, 482] <class 'list'> <class 'list'>
第一帧初始化完毕！
Video: tracking     246.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Video: tracking     4.0fps
Reached the maximum number of frames (10). Exiting loop.
video: average finale average tracking fps 31.8 fps
root@ax650:/mnt/qtang/ax650_mixformer2_demo#
```

#### Inference with M.2 Accelerator card

[What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on Raspberry PI 5.

```
(axcl) axera@raspberrypi:~/samples/MixFormerV2 $ python3 run_mixformer2_axmodel.py --model-path ax650/mixformer_v2.axmodel --frame-path car.avi -r 10
[INFO] Available providers:  ['AXCLRTExecutionProvider']
[INFO] Using provider: AXCLRTExecutionProvider
[INFO] SOC Name: AX650N
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Compiler version: 3.4-dirty 4ff37520-dirty
====================type================= [1079, 482] <class 'list'> <class 'list'>
第一帧初始化完毕！
Video: tracking     925.0fps
Video: tracking     12.0fps
Video: tracking     12.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     11.0fps
Video: tracking     10.0fps
Video: tracking     10.0fps
Video: tracking     10.0fps
Reached the maximum number of frames (10). Exiting loop.
video: average finale average tracking fps 114.9 fps
(axcl) axera@raspberrypi:~/samples/MixFormerV2 $
```