README.md

---
license: bsd-3-clause
pipeline_tag: automatic-speech-recognition
---

# Whisper

OpenAI Whisper on Axera

- 目前支持 C++ 和 Python 两种语言
- 预编译模型下载
  - [Huggingface](https://huggingface.co/AXERA-TECH/Whisper)

- 如需自行转换请参考[模型转换](https://github.com/ml-inory/whisper.axera/blob/main/model_convert/README.md)

## 支持平台

- [x] AX650N
- [x] AX630C

## 模型转换

[模型转换](https://github.com/ml-inory/whisper.axera/blob/main/model_convert/README.md)

## 上板部署

- 基于 AX650N、AX630C 的设备已预装 Ubuntu22.04
- 链接互联网，确保设备能正常执行 `apt install`, `pip install` 等指令
- 已验证设备：
  - [爱芯派Pro(AX650N)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
  - [M.2 Accelerator card(AX650N)](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
  - [爱芯派2(AX630C)](https://axera-pi-2-docs-cn.readthedocs.io/zh-cn/latest/index.html)
  - [Module-LLM(AX630C)](https://docs.m5stack.com/zh_CN/module/Module-LLM)
  - [LLM630 Compute Kit(AX630C)](https://docs.m5stack.com/zh_CN/core/LLM630%20Compute%20Kit)
- 支持编程语言:
  - [Python](#Python)
  - [C++](#CPP)

<h3 id="Python">Python</h3>

#### Requirements

推荐在板上安装Miniconda管理虚拟环境，安装方法如下:
```
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh

source ~/miniconda3/bin/activate

conda init --all
```

安装Whisper依赖
```
cd python

conda create -n whisper python=3.12
conda activate whisper
pip3 install -r requirements.txt
```

####  安装pyaxenigne

参考 https://github.com/AXERA-TECH/pyaxengine 安装 NPU Python API

在0.1.3rc2上测试通过，可通过
```
pip install https://github.com/AXERA-TECH/pyaxengine/releases/download/0.1.3.rc2/axengine-0.1.3-py3-none-any.whl
```
安装，或把版本号更改为你想使用的版本


#### 运行

登陆开发板后

输入命令

```
cd python  
conda activate whisper
python3 main.py --model_type small --model_path ../models-ax650 --wav ../demo.wav --language zh
```

输出结果

```
root@ax650:/mnt/qtang/whisper.axera/python# python3 main.py --wav ../demo.wav --model_type small --model_path ../models/ --language zh
[INFO] Available providers:  ['AxEngineExecutionProvider']
wav: ../demo.wav
model_type: small
model_path: ../models/
language: zh
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Chip type: ChipType.MC50
[INFO] VNPU type: VNPUType.DISABLED
[INFO] Engine version: 2.10.1s
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 3.2-patch1 117f5fd4
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 3.2-patch1 117f5fd4
[INFO] Using provider: AxEngineExecutionProvider
[INFO] Model type: 2 (triple core)
[INFO] Compiler version: 3.2-patch1 117f5fd4
Load models take 2322.563409805298ms
Preprocess wav take 6971.68493270874ms
Run encoder take 211.52877807617188ms
Run decoder_main take 79.00094985961914ms
First token: 17556
Run decoder_loop take 101.91774368286133ms
Iter 0   Token: 20844
Run decoder_loop take 60.30416488647461ms
Iter 1   Token: 7781
Run decoder_loop take 60.22000312805176ms
Iter 2   Token: 20204
Run decoder_loop take 60.23716926574707ms
Iter 3   Token: 28455
Run decoder_loop take 60.214996337890625ms
Iter 4   Token: 31962
Run decoder_loop take 60.17565727233887ms
Iter 5   Token: 6336
Run decoder_loop take 60.94002723693848ms
Iter 6   Token: 254
Run decoder_loop take 60.71639060974121ms
Iter 7   Token: 2930
Run decoder_loop take 60.225725173950195ms
Iter 8   Token: 236
Run decoder_loop take 60.167789459228516ms
Iter 9   Token: 36135
Run decoder_loop take 60.29987335205078ms
Iter 10          Token: 15868
Run decoder_loop take 61.163902282714844ms
Iter 11          Token: 252
Run decoder_loop take 60.273170471191406ms
Iter 12          Token: 1546
Run decoder_loop take 60.23144721984863ms
Iter 13          Token: 46514
Run decoder_loop take 60.31966209411621ms
Iter 14          Token: 50257
Result: 甚至出现交易几乎停滞的情况
```

运行参数说明:  
| 参数名称 | 说明 | 默认值 |
| --- | --- | --- |
| --wav | 输入音频文件 | |
| --model_type/-t | 模型类型, tiny/base/small | |
| --model_path/-p | 模型所在目录 | ../models |
| --language/-l | 识别语言 | zh |


<h3 id="CPP">CPP</h3>

#### 运行

在 AX650N 设备上执行

```
cd cpp
./whisper -w ../demo.wav
```

或  

```
cd cpp
./whisper --model_type small --model_path ../models -w ../demo.wav
```

输出结果

```
root@ax650:/mnt/qtang/whisper.axera/cpp# ./install/whisper --wav ../demo.wav --model_type small --model_path ../models/ --language zh
wav_file: ../demo.wav
model_path: ../models/
model_type: small
language: zh
Encoder run take 188.30 ms
First token: 17556       take 81.88ms
Next Token: 20844        take 29.64ms
Next Token: 7781         take 29.70ms
Next Token: 20204        take 29.64ms
Next Token: 28455        take 29.65ms
Next Token: 31962        take 29.61ms
Next Token: 6336         take 29.67ms
Next Token: 254          take 29.63ms
Next Token: 2930         take 29.61ms
Next Token: 236          take 29.56ms
Next Token: 36135        take 29.64ms
Next Token: 15868        take 29.71ms
Next Token: 252          take 29.51ms
Next Token: 1546         take 29.63ms
Next Token: 46514        take 29.51ms
Next Token: 50257        take 29.69ms
All take 801.13 ms
Result: 甚至出现交易几乎停滞的情况
```

### 服务端

```
cd cpp
./whisper_srv --model_type tiny --model_path ../models-ax650 --language zh --port 8080
```

### 客户端

curl命令行测试(请自行替换IP和端口):  
```
ffmpeg -i demo.wav -f f32le -c:a pcm_f32le - 2>/dev/null | \
curl -X POST 10.126.33.192:8080/asr \
  -H "Content-Type: application/octet-stream" \
  --data-binary @-
```

## 模型性能

### Latency

RTF: Real-Time Factor

CPP:

| Models        | AX650N | AX630C |
| ------------- | ------ | ------ |
| Whisper-Tiny  | 0.08   |        |
| Whisper-Base  | 0.11   | 0.35   |
| Whisper-Small | 0.24   |        |
| Whisper-Turbo | 0.48   |        |

Python:  

| Models        | AX650N | AX630C |
| ------------- | ------ | ------ |
| Whisper-Tiny  | 0.12   |        |
| Whisper-Base  | 0.16   | 0.35   |
| Whisper-Small | 0.50   |        |
| Whisper-Turbo | 0.60   |        |

### Word Error Rate(Test on AIShell dataset)

| Models        | AX650N | AX630C |
| ------------- | ------ | ------ |
| Whisper-Tiny  |  0.24  |        |
| Whisper-Base  |  0.18  |        |
| Whisper-Small |  0.11  |        |
| Whisper-Turbo |  0.06  |        |

若要复现测试结果，请按照以下步骤:

解压数据集:
```
unzip datasets.zip
```

运行测试脚本:
```
cd python
conda activate whisper
python test_wer.py -d aishell --gt_path ../datasets/ground_truth.txt --model_type tiny

```

### MEM Usage

* CMM Stands for Physical memory used by Axera modules like VDEC(Video decoder), VENC(Video encoder), NPU, etc.

Python:  

| Models        | CMM(MB)| OS(MB) |
| ------------- | ------ | ------ |
| Whisper-Tiny  |  332   |  512   |
| Whisper-Base  |  533   |  644   |
| Whisper-Small |  1106  |  906   |
| Whisper-Turbo |  2065  |  2084  |

C++:  

| Models        | CMM(MB)| OS(MB) |
| ------------- | ------ | ------ |
| Whisper-Tiny  |  332   |  31    |
| Whisper-Base  |  533   |  54    |
| Whisper-Small |  1106  |  146   |
| Whisper-Turbo |  2065  |  86    |


## 技术讨论

- Github issues
- QQ 群: 139953715