README.md

---
license: apache-2.0
base_model: Audio8/ARK-ASR-3B
base_model_relation: quantized
library_name: onnx-asr
pipeline_tag: automatic-speech-recognition
tags:
  - onnx
  - onnxruntime
  - automatic-speech-recognition
  - speech
  - audio
  - asr
  - int8
  - speech-llm
language:
  - zh
  - en
  - de
  - ja
  - fr
  - ko
  - es
  - pl
  - it
  - ro
  - hu
  - cs
  - nl
  - fi
  - hr
  - sk
  - sl
  - et
  - lt
---

# ARK-ASR-3B ONNX

ONNX export of [Audio8/ARK-ASR-3B](https://huggingface.co/Audio8/ARK-ASR-3B) for
[onnx-asr](https://github.com/istupakov/onnx-asr). All credit for the model goes to
Audio8 (AutoArk AI). This repository only contains the converted graphs; the
weights are the original ones.

The model is a speech-LLM: a Whisper-large-v3-style audio encoder with rotary
position embeddings, an MLP adapter that merges four encoder frames into one
embedding, and a Qwen2 3B causal language model that writes the transcription.

## Usage

```sh
pip install onnx-asr[cpu,hub]
```

```py
import onnx_asr

model = onnx_asr.load_model("speech-llm", "OpenVoiceOS/ARK-ASR-3B-onnx")
print(model.recognize("audio.wav"))
```

Pass `quantization="int8"` to use the quantized graphs instead of fp32:

```py
model = onnx_asr.load_model("speech-llm", "OpenVoiceOS/ARK-ASR-3B-onnx", quantization="int8")
```

## Files

| File | Contents |
| --- | --- |
| `encoder.onnx` / `encoder_int8.onnx` | audio encoder and MLP adapter, log-mel features in, LM embeddings out |
| `embed_tokens.onnx` / `embed_tokens_int8.onnx` | token embedding table |
| `decoder.onnx` / `decoder_int8.onnx` | Qwen2 decoder with KV cache, logits out |
| `config.json` | model type, prompt token ids, suppressed token ids |
| `vocab.json` | tokenizer vocabulary for detokenization |

int8 sizes: `encoder_int8.onnx` 668 MB, `embed_tokens_int8.onnx` 311 MB,
`decoder_int8.onnx` + `decoder_int8.onnx_data` 1.9 MB + 2.9 GiB (down from a 12 GiB
fp32 decoder). `encoder.onnx` and `embed_tokens.onnx` were quantized with
onnxruntime's `quantize_dynamic` (`QInt8`, `MatMulConstBOnly`). `decoder.onnx` is
too large for `quantize_dynamic` to hold in memory, so it was quantized with an
out-of-core streaming quantizer that reproduces the same dynamic-quantization
arithmetic (`DynamicQuantizeLinear` + `MatMulInteger`, per-tensor `amax/127` scale,
zero point 0) one weight tensor at a time, peaking at a few GB of RSS instead of
holding the whole model.

## Graph contract

| Graph | Inputs | Outputs |
| --- | --- | --- |
| `encoder.onnx` | `input_features (1, 128, frames)` | `audio_embeds (1, frames/8, 2048)` |
| `embed_tokens.onnx` | `input_ids (1, S)` | `inputs_embeds (1, S, 2048)` |
| `decoder.onnx` | `inputs_embeds (1, S, 2048)`, `attn_bias (1, 1, S, P+S)`, `position_ids (1, S)`, `past_key_values.{0..35}.{key,value} (1, 2, P, 128)` | `logits (1, S, 151936)`, `present.{0..35}.{key,value} (1, 2, P+S, 128)` |

## Accuracy

Four FLEURS clips (2 English, 2 Mandarin), greedy decoding, compared against the
PyTorch model in float32:

* fp32 ONNX: 4 of 4 transcriptions identical to PyTorch, character for character.
* int8 ONNX: 3 of 4 transcriptions identical to the fp32 ONNX output. The second
  Mandarin clip dropped one comma ("银和金等元素当然也是金属" instead of
  "银和金等元素，当然也是金属") but the transcription is otherwise complete and
  correct — no early stop or truncation. This is a smaller regression than the
  0.6B ARK model, whose int8 decoder had a Mandarin clip stop early.

Speed on a 12-core CPU under heavy load: RTF 1.1 to 3.2 (fp32), RTF 0.6 to 0.9
(int8). The ONNX graphs were two to four times faster than PyTorch on the same
clips, and int8 roughly doubled fp32 ONNX speed on top of that.

## Licence

Apache 2.0, the same licence as the source model. The model was published by
Audio8; see the [source repository](https://huggingface.co/Audio8/ARK-ASR-3B)
and the paper [arXiv:2605.28139](https://arxiv.org/abs/2605.28139).