ARK-ASR-3B ONNX

ONNX export of Audio8/ARK-ASR-3B for onnx-asr. All credit for the model goes to Audio8 (AutoArk AI). This repository only contains the converted graphs; the weights are the original ones.

The model is a speech-LLM: a Whisper-large-v3-style audio encoder with rotary position embeddings, an MLP adapter that merges four encoder frames into one embedding, and a Qwen2 3B causal language model that writes the transcription.

Usage

pip install onnx-asr[cpu,hub]

import onnx_asr

model = onnx_asr.load_model("speech-llm", "OpenVoiceOS/ARK-ASR-3B-onnx")
print(model.recognize("audio.wav"))

Pass quantization="int8" to use the quantized graphs instead of fp32:

model = onnx_asr.load_model("speech-llm", "OpenVoiceOS/ARK-ASR-3B-onnx", quantization="int8")

Files

File	Contents
`encoder.onnx` / `encoder_int8.onnx`	audio encoder and MLP adapter, log-mel features in, LM embeddings out
`embed_tokens.onnx` / `embed_tokens_int8.onnx`	token embedding table
`decoder.onnx` / `decoder_int8.onnx`	Qwen2 decoder with KV cache, logits out
`config.json`	model type, prompt token ids, suppressed token ids
`vocab.json`	tokenizer vocabulary for detokenization

int8 sizes: encoder_int8.onnx 668 MB, embed_tokens_int8.onnx 311 MB, decoder_int8.onnx + decoder_int8.onnx_data 1.9 MB + 2.9 GiB (down from a 12 GiB fp32 decoder). encoder.onnx and embed_tokens.onnx were quantized with onnxruntime's quantize_dynamic (QInt8, MatMulConstBOnly). decoder.onnx is too large for quantize_dynamic to hold in memory, so it was quantized with an out-of-core streaming quantizer that reproduces the same dynamic-quantization arithmetic (DynamicQuantizeLinear + MatMulInteger, per-tensor amax/127 scale, zero point 0) one weight tensor at a time, peaking at a few GB of RSS instead of holding the whole model.

Graph contract

Graph	Inputs	Outputs
`encoder.onnx`	`input_features (1, 128, frames)`	`audio_embeds (1, frames/8, 2048)`
`embed_tokens.onnx`	`input_ids (1, S)`	`inputs_embeds (1, S, 2048)`
`decoder.onnx`	`inputs_embeds (1, S, 2048)`, `attn_bias (1, 1, S, P+S)`, `position_ids (1, S)`, `past_key_values.{0..35}.{key,value} (1, 2, P, 128)`	`logits (1, S, 151936)`, `present.{0..35}.{key,value} (1, 2, P+S, 128)`

Accuracy

Four FLEURS clips (2 English, 2 Mandarin), greedy decoding, compared against the PyTorch model in float32:

fp32 ONNX: 4 of 4 transcriptions identical to PyTorch, character for character.
int8 ONNX: 3 of 4 transcriptions identical to the fp32 ONNX output. The second Mandarin clip dropped one comma ("银和金等元素当然也是金属" instead of "银和金等元素，当然也是金属") but the transcription is otherwise complete and correct — no early stop or truncation. This is a smaller regression than the 0.6B ARK model, whose int8 decoder had a Mandarin clip stop early.

Speed on a 12-core CPU under heavy load: RTF 1.1 to 3.2 (fp32), RTF 0.6 to 0.9 (int8). The ONNX graphs were two to four times faster than PyTorch on the same clips, and int8 roughly doubled fp32 ONNX speed on top of that.