| --- |
| license: apache-2.0 |
| language: |
| - zh |
| - en |
| library_name: onnx-asr |
| pipeline_tag: automatic-speech-recognition |
| tags: |
| - automatic-speech-recognition |
| - onnx |
| - onnx-asr |
| - paraformer |
| - funasr |
| base_model: |
| - csukuangfj/sherpa-onnx-paraformer-zh-2024-03-09 |
| --- |
| |
| # Paraformer-large ZH ONNX |
|
|
| Mandarin Paraformer-large for onnx-asr. The vocabulary is 8358 Chinese characters and English word pieces, so the model also reads code switched audio and several Mandarin dialects, though English alone is far better served by `OpenVoiceOS/paraformer-en-onnx`. |
|
|
| ## What this is |
|
|
| Paraformer is the Alibaba [FunASR](https://github.com/modelscope/FunASR) offline |
| non-autoregressive recognizer. A SAN-M encoder reads the audio, a CIF predictor decides |
| how many tokens the utterance has, and a single pass decoder emits all of them at once. |
| There is no decoding loop, so one forward pass gives the transcript. |
|
|
| The ONNX graphs here are copied byte for byte from the |
| [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) exports by |
| [csukuangfj](https://huggingface.co/csukuangfj/sherpa-onnx-paraformer-zh-2024-03-09). Only the side files changed: `tokens.txt` became |
| `vocab.txt`, and `config.json` carries the FunASR frontend statistics from `am.mvn`. |
|
|
| ## Usage |
|
|
| The `paraformer` model type is on the `feat/paraformer` branch of the TigreGotico |
| [onnx-asr](https://github.com/TigreGotico/onnx-asr) fork. |
|
|
| ```sh |
| pip install "onnx-asr[cpu,hub] @ git+https://github.com/TigreGotico/onnx-asr@feat/paraformer" |
| ``` |
|
|
| ```py |
| import onnx_asr |
| |
| model = onnx_asr.load_model("OpenVoiceOS/paraformer-zh-onnx") |
| print(model.recognize("audio.wav")) |
| ``` |
|
|
| ## Graph contract |
|
|
| | Item | Value | |
| |---|---| |
| | Input | `speech`, float32, `[batch, num_frames, 560]` | |
| | Input | `speech_lengths`, int32, `[batch]` | |
| | Output | `logits`, float32, `[batch, num_tokens, 8359]` | |
| | Output | `token_num`, int32, `[batch]`, the CIF token count | |
|
|
| The 560 dim input is the FunASR frontend: an 80 dim kaldi fbank of a waveform scaled to |
| the int16 range, then a low frame rate stack of 7 frames with a hop of 6, then the |
| `am.mvn` mean variance statistics. onnx-asr computes the fbank with its `wespeaker` |
| preprocessor and applies the LFR stack and the CMVN in the runtime. Decoding is one |
| argmax per logits row, stopping at `</s>` and never reading past `token_num`. |
|
|
| ## Streaming |
|
|
| A streaming Paraformer also exists upstream. It uses a different graph with encoder and |
| decoder states and needs a streaming runtime, which onnx-asr does not have yet |
| (upstream issue #21). Only the offline model is mirrored here. |
|
|
| ## Attribution and license |
|
|
| * Model: Alibaba DAMO Academy / FunASR, `iic/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8358-tensorflow1`, Apache-2.0. |
| * ONNX export: [sherpa-onnx](https://huggingface.co/csukuangfj/sherpa-onnx-paraformer-zh-2024-03-09) by Fangjun Kuang (csukuangfj), Xiaomi. |
| * This repository only repackages those files for onnx-asr. |
|
|
| ## Files |
|
|
| `model.onnx` (823 MB) and `model_int8.onnx` (227 MB). |
|
|
| ## Parity with native FunASR |
|
|
| 4 FLEURS clips, native FunASR on the same source checkpoint with `dither = 0`. |
|
|
| | Clip | fp32 | int8 | |
| | --- | --- | --- | |
| | zh_1 | identical | identical | |
| | zh_2 | identical | identical | |
| | en_1 | identical | identical | |
| | en_2 | identical | 4 words differ | |
|
|
|
|
