mateli / README.md

Initial release

3dd0f09 verified 5 days ago

10 kB

---
language:
- fi
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: automatic-speech-recognition
tags:
- speech-recognition
- streaming
- transducer
- rnnt
- mingru
- finnish
model-index:
- name: mateli
  results:
  - task:
      type: automatic-speech-recognition
    dataset:
      name: Common Voice 17.0 (fi, test)
      type: mozilla-foundation/common_voice_17_0
      config: fi
      split: test
    metrics:
    - type: wer
      value: 15.03
    - type: cer
      value: 3.08
  - task:
      type: automatic-speech-recognition
    dataset:
      name: FLEURS (fi, SIB-Fleurs subset)
      type: google/fleurs
      config: fi_fi
    metrics:
    - type: wer
      value: 18.47
    - type: cer
      value: 7.49
  - task:
      type: automatic-speech-recognition
    dataset:
      name: VoxPopuli (fi, test)
      type: facebook/voxpopuli
      config: fi
      split: test
    metrics:
    - type: wer
      value: 17.44
    - type: cer
      value: 7.62
---
# Mateli: streaming Finnish speech recognition

Mateli is a small streaming speech recognition model for Finnish (59.6 M parameters). Audio is processed in 320 ms chunks, each chunk sees only itself and the past, and the state carried between chunks has a fixed size. That makes it cheap enough to run live on a phone. This repository has the PyTorch weights and a single-file reference implementation. For streaming on phones and CPUs, use the ONNX export, [samuli/mateli-onnx](https://huggingface.co/samuli/mateli-onnx).

Mateli is not the most accurate Finnish recogniser. It makes mistakes on every kind of speech, and more of them on spontaneous conversation than on read or prepared speech. Its strengths are streaming output, low latency and a low cost to run.

Its output is lowercase words with word times, without punctuation. [samuli/mateli-punct](https://huggingface.co/samuli/mateli-punct) adds punctuation and capitals.

## Examples

Eight clips from public test sets are in `examples/`, and `examples/references.tsv` lists their sources and reference transcripts. The outputs below are from `python mateli.py examples/<clip>.wav`.

Sentences written in colloquial Finnish (Common Voice), with [whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) for comparison:

| clip | reference | whisper-large-v3 | Mateli |
|---|---|---|---|
| `commonvoice_4` | mä en oo ehtiny vielä katsoa. | Mä en ole ehtinyt vielä katsoa. | mä en oo ehtiny vielä katsoa |
| `commonvoice_5` | miten me voidaa olla varmoja et sä puhut totta? | Miten me voidaan olla varmoja, että sä puhut totta? | miten me voidaan olla varmoja et sä puhu totta |
| `commonvoice_6` | sano vaa aika ja paikka ni miä tuun mukkaa kuulin itseni sanovan. | Sano vaan aika ja paikka, niin minä tuun mukaan, kuulin itseni sanoa. | sano vaa aika ja paikka niin minä tuun mukkaa kuulin itseni sanovan |

Mateli writes the words in the form they were spoken (*oo ehtiny*, *et*, *vaa*, *mukkaa*), while Whisper turns them into standard written Finnish (*ole ehtinyt*, *että*, *vaan*, *mukaan*). Mateli also makes mistakes here. It writes *voidaan* for *voidaa*, *puhu* for *puhut* and *niin minä* for *ni miä*. Whisper was run with faster-whisper (int8 on CPU, beam 5, language set to Finnish).

Sentences in written Finnish (Common Voice and VoxPopuli):

| clip | reference | output |
|---|---|---|
| `commonvoice_1` | äänestin päätöslauselmaa vastaan koska siinä ei käsitellä riittävästi näitä seikkoja. | äänestin päätöslauselmaa vastaan koska siinä ei käsitellä riittävästi näitä seikkoja |
| `commonvoice_2` | ne koostuivat eritavoin asetelluista ympyröistä. | ne koostuivat eri tavoin asetelluissa ympyröistä |
| `commonvoice_3` | kaikki mikä tekisi tästä tuskasta edes hitusen siedettävämpää oli kiellettyä. | kaikki mikä tekis tästä tuskasta edes hitusen siedettävämpää oli kiellettyä |
| `voxpopuli_1` | arvoisa puhemies äänestin tämän mietinnön puolesta. vaikka komission tiedonannosta on kulunut jo yli kymmenen vuotta eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan. | repo äänestin tämän mietinnön puolesta öö vaikka komission tiedonainnosta on kulunut jo yli kymmeneen vuotta eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan |
| `voxpopuli_2` | boikotoiminen vaaleissa ja mielenosoitus ovat eri asioita mutta erittäin huolestuttavia ovat tiedot siitä että useampi kuin joka kymmenes äänestyspaikka ei voinut avata oviaan opposition saarron takia. | boiko toiminen vaaleissa ja mielenosoitus ovat eri asioita mutta erittäin huolestuttavia ovat tiedot siitä että useampi kuin joka kymmenen säänestyspaikka ei voinut avata oviaan opposition saannon takia |

Typical mistakes are wrong case endings (*asetelluissa*), split or misheard words (*boiko toiminen*, *säänestyspaikka*), an unclear start turned into a different word (*repo* for *arvoisa puhemies*), hesitation sounds written out (*öö*) and spoken forms kept as spoken (*tekis*).

Word times with `--words` (start and end in seconds):

```
$ python mateli.py examples/commonvoice_2.wav --words
   1.60    1.64  ne
   1.76    2.40  koostuivat
   2.60    2.72  eri
   2.84    3.16  tavoin
   3.28    4.04  asetelluissa
   4.20    4.72  ympyröistä
```

## How to use

```
pip install torch safetensors soundfile scipy
hf download samuli/mateli --local-dir mateli
python mateli/mateli.py mateli/examples/commonvoice_2.wav
```

From Python (run next to the downloaded `mateli` directory):

```python
import sys
sys.path.insert(0, "mateli")
from mateli import Mateli, load_audio
asr = Mateli.from_pretrained("mateli")          # GPU if available, else CPU
result = asr.transcribe(load_audio("mateli/examples/commonvoice_3.wav"))
print(result.text)
for word in result.words[:3]:
    print(word)
```

```
kaikki mikä tekis tästä tuskasta edes hitusen siedettävämpää oli kiellettyä
Word(text='kaikki', start=1.16, end=1.44)
Word(text='mikä', start=1.72, end=1.88)
Word(text='tekis', start=1.96, end=2.12)
```

`transcribe()` takes float32 mono 16 kHz samples and runs the same 320 ms chunk loop that live streaming would, so its output equals streaming recognition of the same audio, and memory use stays flat on long recordings. The output is the same as the ONNX export's fp32 encoder.

### Files

| file | what |
|---|---|
| `model.safetensors` | weights, fp32 (238 MB) |
| `config.json` | model settings and vocabulary |
| `mateli.py` | model definition, transcription, command line |

## Model

![Mateli architecture and streaming](architecture.png)

| | |
|---|---|
| parameters | 59.6 M |
| input | 16 kHz mono, 80-bin log-mel (25 ms window, 10 ms hop) |
| encoder | two stride-2 Conv2d (40 ms frames), 16 blocks of feed-forward / MinGRU / causal depthwise convolution / feed-forward, windowed self-attention after blocks 4, 7, 10 and 13 |
| attention window | the current 320 ms chunk + 64 earlier frames (2.56 s), no positional encoding |
| decoder | RNN-T: one-layer MinGRU prediction network, additive joiner, greedy search |
| output | 44 characters (blank, space, a-z, å, ä, ö, 0-9, `. ? !`) |
| latency | 320 ms chunks |
| compute | about 2 GMAC per second of audio |

[MinGRU](https://arxiv.org/abs/2410.01201) is a minimal gated recurrent unit whose gates do not depend on the previous state, so a whole chunk is computed in parallel with a log-space scan while streaming still carries a single state vector per layer. The model was trained with the RNN-T loss and an auxiliary CTC loss (the CTC head is not included).

## Training data

About 2,600 hours of Finnish speech from various publicly available sources and different registers. Transcripts are either human produced or from larger Finnish speech recognition models. Most of the machine transcripts were produced with [GetmanY1/wav2vec2-xlarge-fi-150k-finetuned](https://huggingface.co/GetmanY1/wav2vec2-xlarge-fi-150k-finetuned).

The transcripts of conversational speech are deliberately verbatim. They write down what was said, in the form it was said (*mä*, *tekis*, *mahollisimman*), including hesitations like *öö*, instead of turning it into written language. Spoken Finnish differs a lot from written Finnish, and a verbatim transcript keeps that information. It stays faithful to the speaker, every spoken word gets its own time, and written forms can still be produced afterwards. Words that a cleaned-up transcript dropped or rewrote cannot be recovered. Read and prepared speech was transcribed in standard written form. The model learned both styles, so its output follows the speech. The cost is a higher word error rate against written-language references (see [Evaluation](#evaluation)).

## Evaluation

Word and character error rates (%). References and outputs are lowercased, with punctuation removed and foreign diacritics removed (é becomes e).

| test set | clips | hours | words | WER | CER |
|---|---:|---:|---:|---:|---:|
| Common Voice 17.0, fi test (read sentences) | 1,763 | 2.50 | 11,135 | 15.03 | 3.08 |
| FLEURS fi, SIB-Fleurs subset (read sentences) | 175 | 0.64 | 2,766 | 18.47 | 7.49 |
| VoxPopuli fi test (parliamentary speeches) | 452 | 1.27 | 7,338 | 17.44 | 7.62 |

Part of every WER comes from spelling choices. Spoken forms (*mä*, *tekis*) and compound words written apart or together count as errors against a written-language reference.

## Limitations

- No punctuation or capitals, and numbers are written as words. Use [samuli/mateli-punct](https://huggingface.co/samuli/mateli-punct) for punctuation and casing.
- Accuracy drops on spontaneous conversation, overlapping speakers, dialects, noise and rare names.
- Greedy decoding only, no language model.
- Finnish only.
- Behaviour on music, other languages and long non-speech passages has not been evaluated.

## License

The model weights and the code in this repository are licensed under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). The example clips in `examples/` are CC0 (Common Voice and VoxPopuli).

	---
	language:
	- fi
	license: cc-by-nc-4.0
	library_name: pytorch
	pipeline_tag: automatic-speech-recognition
	tags:
	- speech-recognition
	- streaming
	- transducer
	- rnnt
	- mingru
	- finnish
	model-index:
	- name: mateli
	results:
	- task:
	type: automatic-speech-recognition
	dataset:
	name: Common Voice 17.0 (fi, test)
	type: mozilla-foundation/common_voice_17_0
	config: fi
	split: test
	metrics:
	- type: wer
	value: 15.03
	- type: cer
	value: 3.08
	- task:
	type: automatic-speech-recognition
	dataset:
	name: FLEURS (fi, SIB-Fleurs subset)
	type: google/fleurs
	config: fi_fi
	metrics:
	- type: wer
	value: 18.47
	- type: cer
	value: 7.49
	- task:
	type: automatic-speech-recognition
	dataset:
	name: VoxPopuli (fi, test)
	type: facebook/voxpopuli
	config: fi
	split: test
	metrics:
	- type: wer
	value: 17.44
	- type: cer
	value: 7.62
	---

	# Mateli: streaming Finnish speech recognition

	Mateli is a small streaming speech recognition model for Finnish (59.6 M parameters). Audio is processed in 320 ms chunks, each chunk sees only itself and the past, and the state carried between chunks has a fixed size. That makes it cheap enough to run live on a phone. This repository has the PyTorch weights and a single-file reference implementation. For streaming on phones and CPUs, use the ONNX export, [samuli/mateli-onnx](https://huggingface.co/samuli/mateli-onnx).

	Mateli is not the most accurate Finnish recogniser. It makes mistakes on every kind of speech, and more of them on spontaneous conversation than on read or prepared speech. Its strengths are streaming output, low latency and a low cost to run.

	Its output is lowercase words with word times, without punctuation. [samuli/mateli-punct](https://huggingface.co/samuli/mateli-punct) adds punctuation and capitals.

	## Examples

	Eight clips from public test sets are in `examples/`, and `examples/references.tsv` lists their sources and reference transcripts. The outputs below are from `python mateli.py examples/<clip>.wav`.

	Sentences written in colloquial Finnish (Common Voice), with [whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) for comparison:

	\| clip \| reference \| whisper-large-v3 \| Mateli \|
	\|---\|---\|---\|---\|
	\| `commonvoice_4` \| mä en oo ehtiny vielä katsoa. \| Mä en ole ehtinyt vielä katsoa. \| mä en oo ehtiny vielä katsoa \|
	\| `commonvoice_5` \| miten me voidaa olla varmoja et sä puhut totta? \| Miten me voidaan olla varmoja, että sä puhut totta? \| miten me voidaan olla varmoja et sä puhu totta \|
	\| `commonvoice_6` \| sano vaa aika ja paikka ni miä tuun mukkaa kuulin itseni sanovan. \| Sano vaan aika ja paikka, niin minä tuun mukaan, kuulin itseni sanoa. \| sano vaa aika ja paikka niin minä tuun mukkaa kuulin itseni sanovan \|

	Mateli writes the words in the form they were spoken (oo ehtiny, et, vaa, mukkaa), while Whisper turns them into standard written Finnish (ole ehtinyt, että, vaan, mukaan). Mateli also makes mistakes here. It writes voidaan for voidaa, puhu for puhut and niin minä for ni miä. Whisper was run with faster-whisper (int8 on CPU, beam 5, language set to Finnish).

	Sentences in written Finnish (Common Voice and VoxPopuli):

	\| clip \| reference \| output \|
	\|---\|---\|---\|
	\| `commonvoice_1` \| äänestin päätöslauselmaa vastaan koska siinä ei käsitellä riittävästi näitä seikkoja. \| äänestin päätöslauselmaa vastaan koska siinä ei käsitellä riittävästi näitä seikkoja \|
	\| `commonvoice_2` \| ne koostuivat eritavoin asetelluista ympyröistä. \| ne koostuivat eri tavoin asetelluissa ympyröistä \|
	\| `commonvoice_3` \| kaikki mikä tekisi tästä tuskasta edes hitusen siedettävämpää oli kiellettyä. \| kaikki mikä tekis tästä tuskasta edes hitusen siedettävämpää oli kiellettyä \|
	\| `voxpopuli_1` \| arvoisa puhemies äänestin tämän mietinnön puolesta. vaikka komission tiedonannosta on kulunut jo yli kymmenen vuotta eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan. \| repo äänestin tämän mietinnön puolesta öö vaikka komission tiedonainnosta on kulunut jo yli kymmeneen vuotta eivät kaikki matkustajat edelleenkään tiedä oikeuksistaan \|
	\| `voxpopuli_2` \| boikotoiminen vaaleissa ja mielenosoitus ovat eri asioita mutta erittäin huolestuttavia ovat tiedot siitä että useampi kuin joka kymmenes äänestyspaikka ei voinut avata oviaan opposition saarron takia. \| boiko toiminen vaaleissa ja mielenosoitus ovat eri asioita mutta erittäin huolestuttavia ovat tiedot siitä että useampi kuin joka kymmenen säänestyspaikka ei voinut avata oviaan opposition saannon takia \|

	Typical mistakes are wrong case endings (asetelluissa), split or misheard words (boiko toiminen, säänestyspaikka), an unclear start turned into a different word (repo for arvoisa puhemies), hesitation sounds written out (öö) and spoken forms kept as spoken (tekis).

	Word times with `--words` (start and end in seconds):

	```
	$ python mateli.py examples/commonvoice_2.wav --words
	1.60 1.64 ne
	1.76 2.40 koostuivat
	2.60 2.72 eri
	2.84 3.16 tavoin
	3.28 4.04 asetelluissa
	4.20 4.72 ympyröistä
	```

	## How to use

	```
	pip install torch safetensors soundfile scipy
	hf download samuli/mateli --local-dir mateli
	python mateli/mateli.py mateli/examples/commonvoice_2.wav
	```

	From Python (run next to the downloaded `mateli` directory):

	```python
	import sys
	sys.path.insert(0, "mateli")
	from mateli import Mateli, load_audio

	asr = Mateli.from_pretrained("mateli") # GPU if available, else CPU
	result = asr.transcribe(load_audio("mateli/examples/commonvoice_3.wav"))
	print(result.text)
	for word in result.words[:3]:
	print(word)
	```

	```
	kaikki mikä tekis tästä tuskasta edes hitusen siedettävämpää oli kiellettyä
	Word(text='kaikki', start=1.16, end=1.44)
	Word(text='mikä', start=1.72, end=1.88)
	Word(text='tekis', start=1.96, end=2.12)
	```

	`transcribe()` takes float32 mono 16 kHz samples and runs the same 320 ms chunk loop that live streaming would, so its output equals streaming recognition of the same audio, and memory use stays flat on long recordings. The output is the same as the ONNX export's fp32 encoder.

	### Files

	\| file \| what \|
	\|---\|---\|
	\| `model.safetensors` \| weights, fp32 (238 MB) \|
	\| `config.json` \| model settings and vocabulary \|
	\| `mateli.py` \| model definition, transcription, command line \|

	## Model

	![Mateli architecture and streaming](architecture.png)

	\| \| \|
	\|---\|---\|
	\| parameters \| 59.6 M \|
	\| input \| 16 kHz mono, 80-bin log-mel (25 ms window, 10 ms hop) \|
	\| encoder \| two stride-2 Conv2d (40 ms frames), 16 blocks of feed-forward / MinGRU / causal depthwise convolution / feed-forward, windowed self-attention after blocks 4, 7, 10 and 13 \|
	\| attention window \| the current 320 ms chunk + 64 earlier frames (2.56 s), no positional encoding \|
	\| decoder \| RNN-T: one-layer MinGRU prediction network, additive joiner, greedy search \|
	\| output \| 44 characters (blank, space, a-z, å, ä, ö, 0-9, `. ? !`) \|
	\| latency \| 320 ms chunks \|
	\| compute \| about 2 GMAC per second of audio \|

	[MinGRU](https://arxiv.org/abs/2410.01201) is a minimal gated recurrent unit whose gates do not depend on the previous state, so a whole chunk is computed in parallel with a log-space scan while streaming still carries a single state vector per layer. The model was trained with the RNN-T loss and an auxiliary CTC loss (the CTC head is not included).

	## Training data

	About 2,600 hours of Finnish speech from various publicly available sources and different registers. Transcripts are either human produced or from larger Finnish speech recognition models. Most of the machine transcripts were produced with [GetmanY1/wav2vec2-xlarge-fi-150k-finetuned](https://huggingface.co/GetmanY1/wav2vec2-xlarge-fi-150k-finetuned).

	The transcripts of conversational speech are deliberately verbatim. They write down what was said, in the form it was said (mä, tekis, mahollisimman), including hesitations like öö, instead of turning it into written language. Spoken Finnish differs a lot from written Finnish, and a verbatim transcript keeps that information. It stays faithful to the speaker, every spoken word gets its own time, and written forms can still be produced afterwards. Words that a cleaned-up transcript dropped or rewrote cannot be recovered. Read and prepared speech was transcribed in standard written form. The model learned both styles, so its output follows the speech. The cost is a higher word error rate against written-language references (see [Evaluation](#evaluation)).

	## Evaluation

	Word and character error rates (%). References and outputs are lowercased, with punctuation removed and foreign diacritics removed (é becomes e).

	\| test set \| clips \| hours \| words \| WER \| CER \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| Common Voice 17.0, fi test (read sentences) \| 1,763 \| 2.50 \| 11,135 \| 15.03 \| 3.08 \|
	\| FLEURS fi, SIB-Fleurs subset (read sentences) \| 175 \| 0.64 \| 2,766 \| 18.47 \| 7.49 \|
	\| VoxPopuli fi test (parliamentary speeches) \| 452 \| 1.27 \| 7,338 \| 17.44 \| 7.62 \|

	Part of every WER comes from spelling choices. Spoken forms (mä, tekis) and compound words written apart or together count as errors against a written-language reference.

	## Limitations

	- No punctuation or capitals, and numbers are written as words. Use [samuli/mateli-punct](https://huggingface.co/samuli/mateli-punct) for punctuation and casing.
	- Accuracy drops on spontaneous conversation, overlapping speakers, dialects, noise and rare names.
	- Greedy decoding only, no language model.
	- Finnish only.
	- Behaviour on music, other languages and long non-speech passages has not been evaluated.

	## License

	The model weights and the code in this repository are licensed under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). The example clips in `examples/` are CC0 (Common Voice and VoxPopuli).