Malagasy 4-gram language model (KenLM) for CTC speech recognition
A word-level 4-gram KenLM model of Malagasy, built to decode the CTC output of Flo976/w2v-bert-malagasy. It brings that model from 14.33 % to 13.20 % word error rate on short utterances, and from 12.01 % to 10.41 % on full-length clips, when it receives log-probabilities and apostrophes are re-attached after decoding (see Usage).
A larger version trained on 134 M words, Flo976/malagasy-kenlm-4gram-v2, lowers these figures to 12.68 % and 10.13 %. It includes web text and has no single open license; this model remains the choice when you need a clean CC-BY-SA licence.
The chart shows the first published measurement, which passed raw logits to the decoder and did not re-attach apostrophes: 13.6 % and 10.6 %.
Files
| File | Content |
|---|---|
mg_4gram.bin |
KenLM binary (trie, 8-bit quantized), 89 MB |
unigrams.txt |
478,401 word forms seen at least twice, one per line, for pyctcdecode |
Training text
17.4 million words, normalized exactly as the ASR output: lowercase, punctuation removed, apostrophes and hyphens kept inside words, numbers as digits.
| Source | Words | License |
|---|---|---|
| Malagasy Wikipedia, via ffatty/plaintext-wikipedia-full-malagasy | 16.5 M | CC-BY-SA-3.0 |
| Transcripts of the train split of badrex/malagasy-speech-full | 0.9 M | not declared on the Hub |
No validation or test transcript was used. A later check found that 77 of the 3,101 test clips (75 of 3,099 in validation) share at least one 8-word sequence with a line of this corpus (Wikipedia or train transcripts); no corpus line equals a reference. The effect on the scores is likely small but was not measured. Version 2 removes every such line. Built with lmplz -o 4 --prune 0 0 1 1, then build_binary -q 8 -b 7 -a 256 trie.
Usage
pip install pyctcdecode https://github.com/kpu/kenlm/archive/master.zip
import re
import torch
from huggingface_hub import hf_hub_download
from pyctcdecode import build_ctcdecoder
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained("Flo976/w2v-bert-malagasy")
tok = processor.tokenizer
vocab = tok.get_vocab()
labels = ["" if t == tok.pad_token else " " if t == tok.word_delimiter_token else "⁇" if t == tok.unk_token else t
for t in sorted(vocab, key=vocab.get)]
repo = "Flo976/malagasy-kenlm-4gram"
unigrams = [w.strip() for w in open(hf_hub_download(repo, "unigrams.txt"), encoding="utf-8") if w.strip()]
decoder = build_ctcdecoder(labels, kenlm_model_path=hf_hub_download(repo, "mg_4gram.bin"),
unigrams=unigrams, alpha=0.3, beta=-1.0)
logp = torch.log_softmax(logits[0].float(), -1).numpy() # log-probabilities, not raw logits
text = decoder.decode(logp, beam_width=100)
text = re.sub(r"'\s+", "'", re.sub(r"\s+'", "'", text)) # "amin' ny" -> "amin'ny"
Tuned on the validation split: α = 0.3, β = −1.0 for utterances of 10 s or less, β = 0.0 for longer clips. A beam of 100 is used: in an earlier tuning on full-length clips, beams of 300 and 500 brought no gain over it. After decoding, re-attaching apostrophes (amin' ny → amin'ny) gains another 0.24 point on short utterances. Pass log-probabilities (log-softmax), not raw logits: with raw logits, as in the first version of this card, the scores are 0.2 point worse on short utterances.
pyctcdecode 0.5.0 declares numpy<2 but works with numpy 2; if that pin conflicts with your environment, install it with pip install --no-deps pyctcdecode pygtrie.
License
CC-BY-SA-3.0, inherited from Malagasy Wikipedia: share-alike applies to derivatives of this language model. The acoustic model Flo976/w2v-bert-malagasy is licensed separately under CC-BY-4.0.
Built by Flo976 for Milo Voice.