samuli/mateli-onnx

Mateli (ONNX): streaming Finnish speech recognition

This is the ONNX export of Mateli, a small streaming speech recognition model for Finnish, for live recognition on phones and CPUs. The model, its training data, how it was evaluated and its limitations are described on the Mateli model card. This page covers only what is different here.

	samuli/mateli	this repository
runtime	PyTorch	ONNX Runtime (numpy, no PyTorch)
weights	fp32	int8 encoder (62 MiB) or fp32 encoder (225 MiB)
streaming	whole file, in the same 320 ms chunks a stream would use	live: feed audio blocks of any size as they arrive

With the fp32 encoder the output is the same as the PyTorch model's (identical text on all 2,390 evaluation clips). The int8 encoder gives the same words most of the time, and its error rates are within 0.2 points of fp32's.

Examples

The same eight clips as on the Mateli card are in examples/, with their sources, licences and references in examples/references.tsv. Their outputs are on the Mateli card. The fp32 encoder gives the same output as the PyTorch model on all eight, and the int8 encoder differs by one word, on voxpopuli_2 (säännöstyspaikka for säänestyspaikka, both wrong).

With --stream, the audio is fed in 100 ms blocks. Each line shows how much audio had arrived when the word came out, then the word's start and end time. A word comes out when the model emits the space after it, here 0.1 to 0.8 s after the word ends, or when the next word starts if a longer pause follows. The last word comes out when the stream is finished.

$ python transcribe.py examples/voxpopuli_1.wav --stream
[audio    2.60 s]    0.60    0.76  repo
[audio    2.90 s]    2.40    2.80  äänestin
[audio    3.30 s]    2.88    3.08  tämän
[audio    3.90 s]    3.28    3.68  mietinnön
[audio    4.90 s]    3.76    4.16  puolesta
[audio    5.20 s]    4.84    4.84  öö
[audio    5.50 s]    5.00    5.20  vaikka
[audio    6.10 s]    5.28    5.76  komission
[audio    6.80 s]    5.84    6.48  tiedonainnosta
[audio    6.80 s]    6.56    6.64  on
[audio    7.40 s]    6.68    7.04  kulunut
[audio    7.70 s]    7.16    7.20  jo
[audio    8.10 s]    7.56    7.68  yli
[audio    8.70 s]    7.96    8.32  kymmeneen
[audio    9.30 s]    8.36    8.60  vuotta
[audio    9.70 s]    9.08    9.28  eivät
[audio   10.00 s]    9.36    9.56  kaikki
[audio   10.30 s]    9.68   10.16  matkustajat
[audio   10.90 s]   10.24   10.76  edelleenkään
[audio   11.60 s]   10.92   11.12  tiedä
[end of audio  ]   11.64   12.44  oikeuksistaan

How to use

pip install numpy onnxruntime soundfile scipy
hf download samuli/mateli-onnx --local-dir mateli-onnx
python mateli-onnx/transcribe.py mateli-onnx/examples/commonvoice_2.wav

From Python (run next to the downloaded mateli-onnx directory):

import sys
sys.path.insert(0, "mateli-onnx")
from transcribe import Mateli, load_audio

asr = Mateli("mateli-onnx")                     # int8 encoder; Mateli("mateli-onnx", "fp32") for fp32
result = asr.transcribe(load_audio("mateli-onnx/examples/commonvoice_3.wav"))
print(result.text)
for word in result.words[:3]:
    print(word)

kaikki mikä tekis tästä tuskasta edes hitusen siedettävämpää oli kiellettyä
Word(text='kaikki', start=1.16, end=1.44)
Word(text='mikä', start=1.72, end=1.88)
Word(text='tekis', start=1.96, end=2.12)

For live use, create a stream with stream = asr.stream(). Each call to stream.accept(block) takes a block of float32 16 kHz mono samples (any block size) and returns the words completed so far, and stream.finish() returns the rest. Start a new stream for the next recording. Audio files at other sample rates are resampled by load_audio (scipy). Capture microphone audio at 16 kHz.

transcribe.py (numpy and onnxruntime only, about 270 lines) is meant to be read and adapted, for example ported to Kotlin or Swift. It contains the streaming log-mel front end, the chunk loop with the encoder state, the greedy decoder and the word timing.

Files

file	size	what
encoder.int8.onnx	62 MiB	encoder, dynamic int8 (per-channel weights for every MatMul/Gemm with a constant weight)
encoder.onnx	225 MiB	encoder, fp32
pred_step.onnx	2.3 MiB	prediction network, one step
joint.onnx	67 kB	joiner
mel_fb.f32	80 kB	mel filterbank, float32 little-endian, [80, 257]
config.json		feature settings, tensor names and shapes, vocabulary

Model interface

There are three graphs. encoder takes one 320 ms chunk of log-mel frames plus its state and returns the next state, pred_step advances the prediction network by one token, and joint scores the next token. The encoder state is a fixed set of six tensors (about 1.2 MB in total, whatever the length of the audio) that start as zeros and are fed back after every chunk. Their names and shapes are in config.json. transcribe.py shows the whole loop. When porting, match its log-mel front end exactly.

Tested with ONNX Runtime 1.30 and 1.31 on the CPU execution provider (x86-64 and Android arm64). Other execution providers (NNAPI, XNNPACK, GPU) have not been tested.

Evaluation

The same test sets and scoring as on the Mateli card, word and character error rates (%) for both encoders:

test set	WER int8	WER fp32	CER int8	CER fp32
Common Voice 17.0, fi test	15.02	15.03	3.09	3.08
FLEURS fi, SIB-Fleurs subset	18.47	18.47	7.48	7.49
VoxPopuli fi test	17.65	17.44	7.68	7.62

The output has no punctuation or capitals. samuli/mateli-punct-onnx adds them and can use these word times as pauses.

License

The model weights and the code in this repository are licensed under CC BY-NC 4.0. The example clips in examples/ are CC0 (Common Voice and VoxPopuli).