mateli / README.md

Initial release

3dd0f09 verified 5 days ago

10 kB

language:
  - fi
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: automatic-speech-recognition
tags:
  - speech-recognition
  - streaming
  - transducer
  - rnnt
  - mingru
  - finnish
model-index:
  - name: mateli
    results:
      - task:
          type: automatic-speech-recognition
        dataset:
          name: Common Voice 17.0 (fi, test)
          type: mozilla-foundation/common_voice_17_0
          config: fi
          split: test
        metrics:
          - type: wer
            value: 15.03
          - type: cer
            value: 3.08
      - task:
          type: automatic-speech-recognition
        dataset:
          name: FLEURS (fi, SIB-Fleurs subset)
          type: google/fleurs
          config: fi_fi
        metrics:
          - type: wer
            value: 18.47
          - type: cer
            value: 7.49
      - task:
          type: automatic-speech-recognition
        dataset:
          name: VoxPopuli (fi, test)
          type: facebook/voxpopuli
          config: fi
          split: test
        metrics:
          - type: wer
            value: 17.44
          - type: cer
            value: 7.62

Mateli: streaming Finnish speech recognition

Mateli is a small streaming speech recognition model for Finnish (59.6 M parameters). Audio is processed in 320 ms chunks, each chunk sees only itself and the past, and the state carried between chunks has a fixed size. That makes it cheap enough to run live on a phone. This repository has the PyTorch weights and a single-file reference implementation. For streaming on phones and CPUs, use the ONNX export, samuli/mateli-onnx.

Mateli is not the most accurate Finnish recogniser. It makes mistakes on every kind of speech, and more of them on spontaneous conversation than on read or prepared speech. Its strengths are streaming output, low latency and a low cost to run.

Its output is lowercase words with word times, without punctuation. samuli/mateli-punct adds punctuation and capitals.

Examples

Eight clips from public test sets are in examples/, and examples/references.tsv lists their sources and reference transcripts. The outputs below are from python mateli.py examples/<clip>.wav.

Sentences written in colloquial Finnish (Common Voice), with whisper-large-v3 for comparison:

clip	reference	whisper-large-v3	Mateli
`commonvoice_4`	mä en oo ehtiny vielä katsoa.	Mä en ole ehtinyt vielä katsoa.	mä en oo ehtiny vielä katsoa
`commonvoice_5`	miten me voidaa olla varmoja et sä puhut totta?	Miten me voidaan olla varmoja, että sä puhut totta?	miten me voidaan olla varmoja et sä puhu totta
`commonvoice_6`	sano vaa aika ja paikka ni miä tuun mukkaa kuulin itseni sanovan.	Sano vaan aika ja paikka, niin minä tuun mukaan, kuulin itseni sanoa.	sano vaa aika ja paikka niin minä tuun mukkaa kuulin itseni sanovan

Mateli writes the words in the form they were spoken (oo ehtiny, et, vaa, mukkaa), while Whisper turns them into standard written Finnish (ole ehtinyt, että, vaan, mukaan). Mateli also makes mistakes here. It writes voidaan for voidaa, puhu for puhut and niin minä for ni miä. Whisper was run with faster-whisper (int8 on CPU, beam 5, language set to Finnish).

Sentences in written Finnish (Common Voice and VoxPopuli):

clip	reference	output
`commonvoice_1`	äänestin päätöslauselmaa vastaan koska siinä ei käsitellä riittävästi näitä seikkoja.	äänestin päätöslauselmaa vastaan koska siinä ei käsitellä riittävästi näitä seikkoja
`commonvoice_2`	ne koostuivat eritavoin asetelluista ympyröistä.	ne koostuivat eri tavoin asetelluissa ympyröistä
`commonvoice_3`	kaikki mikä tekisi tästä tuskasta edes hitusen siedettävämpää oli kiellettyä.	kaikki mikä tekis tästä tuskasta edes hitusen siedettävämpää oli kiellettyä
`voxpopuli_1`	arvoisa puhemies äänestin tämän mietinnön puolesta. vaikka komission tiedonannosta on kulunut jo yli kymmenen vuotta eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan.	repo äänestin tämän mietinnön puolesta öö vaikka komission tiedonainnosta on kulunut jo yli kymmeneen vuotta eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan
`voxpopuli_2`	boikotoiminen vaaleissa ja mielenosoitus ovat eri asioita mutta erittäin huolestuttavia ovat tiedot siitä että useampi kuin joka kymmenes äänestyspaikka ei voinut avata oviaan opposition saarron takia.	boiko toiminen vaaleissa ja mielenosoitus ovat eri asioita mutta erittäin huolestuttavia ovat tiedot siitä että useampi kuin joka kymmenen säänestyspaikka ei voinut avata oviaan opposition saannon takia

Typical mistakes are wrong case endings (asetelluissa), split or misheard words (boiko toiminen, säänestyspaikka), an unclear start turned into a different word (repo for arvoisa puhemies), hesitation sounds written out (öö) and spoken forms kept as spoken (tekis).

Word times with --words (start and end in seconds):

$ python mateli.py examples/commonvoice_2.wav --words
   1.60    1.64  ne
   1.76    2.40  koostuivat
   2.60    2.72  eri
   2.84    3.16  tavoin
   3.28    4.04  asetelluissa
   4.20    4.72  ympyröistä

How to use

pip install torch safetensors soundfile scipy
hf download samuli/mateli --local-dir mateli
python mateli/mateli.py mateli/examples/commonvoice_2.wav

From Python (run next to the downloaded mateli directory):

import sys
sys.path.insert(0, "mateli")
from mateli import Mateli, load_audio

asr = Mateli.from_pretrained("mateli")          # GPU if available, else CPU
result = asr.transcribe(load_audio("mateli/examples/commonvoice_3.wav"))
print(result.text)
for word in result.words[:3]:
    print(word)

kaikki mikä tekis tästä tuskasta edes hitusen siedettävämpää oli kiellettyä
Word(text='kaikki', start=1.16, end=1.44)
Word(text='mikä', start=1.72, end=1.88)
Word(text='tekis', start=1.96, end=2.12)

transcribe() takes float32 mono 16 kHz samples and runs the same 320 ms chunk loop that live streaming would, so its output equals streaming recognition of the same audio, and memory use stays flat on long recordings. The output is the same as the ONNX export's fp32 encoder.

Files

file	what
`model.safetensors`	weights, fp32 (238 MB)
`config.json`	model settings and vocabulary
`mateli.py`	model definition, transcription, command line

Model


parameters	59.6 M
input	16 kHz mono, 80-bin log-mel (25 ms window, 10 ms hop)
encoder	two stride-2 Conv2d (40 ms frames), 16 blocks of feed-forward / MinGRU / causal depthwise convolution / feed-forward, windowed self-attention after blocks 4, 7, 10 and 13
attention window	the current 320 ms chunk + 64 earlier frames (2.56 s), no positional encoding
decoder	RNN-T: one-layer MinGRU prediction network, additive joiner, greedy search
output	44 characters (blank, space, a-z, å, ä, ö, 0-9, `. ? !`)
latency	320 ms chunks
compute	about 2 GMAC per second of audio

MinGRU is a minimal gated recurrent unit whose gates do not depend on the previous state, so a whole chunk is computed in parallel with a log-space scan while streaming still carries a single state vector per layer. The model was trained with the RNN-T loss and an auxiliary CTC loss (the CTC head is not included).

Training data

About 2,600 hours of Finnish speech from various publicly available sources and different registers. Transcripts are either human produced or from larger Finnish speech recognition models. Most of the machine transcripts were produced with GetmanY1/wav2vec2-xlarge-fi-150k-finetuned.

The transcripts of conversational speech are deliberately verbatim. They write down what was said, in the form it was said (mä, tekis, mahollisimman), including hesitations like öö, instead of turning it into written language. Spoken Finnish differs a lot from written Finnish, and a verbatim transcript keeps that information. It stays faithful to the speaker, every spoken word gets its own time, and written forms can still be produced afterwards. Words that a cleaned-up transcript dropped or rewrote cannot be recovered. Read and prepared speech was transcribed in standard written form. The model learned both styles, so its output follows the speech. The cost is a higher word error rate against written-language references (see Evaluation).

Evaluation

Word and character error rates (%). References and outputs are lowercased, with punctuation removed and foreign diacritics removed (é becomes e).

test set	clips	hours	words	WER	CER
Common Voice 17.0, fi test (read sentences)	1,763	2.50	11,135	15.03	3.08
FLEURS fi, SIB-Fleurs subset (read sentences)	175	0.64	2,766	18.47	7.49
VoxPopuli fi test (parliamentary speeches)	452	1.27	7,338	17.44	7.62

Part of every WER comes from spelling choices. Spoken forms (mä, tekis) and compound words written apart or together count as errors against a written-language reference.

Limitations

No punctuation or capitals, and numbers are written as words. Use samuli/mateli-punct for punctuation and casing.
Accuracy drops on spontaneous conversation, overlapping speakers, dialects, noise and rare names.
Greedy decoding only, no language model.
Finnish only.
Behaviour on music, other languages and long non-speech passages has not been evaluated.

License

The model weights and the code in this repository are licensed under CC BY-NC 4.0. The example clips in examples/ are CC0 (Common Voice and VoxPopuli).