WERdict-pl

Estimates the word error rate of a Polish ASR transcript from the transcript alone. It reads the words an ASR system produced and returns the WER it thinks that transcript has.

Trained on two engine families and Polish only.

Use it to pick an ASR system for new recordings before anyone transcribes by hand, route transcripts that need a human, or watch quality on material that will never get a reference. It does not replace a WER measurement when you have a reference, and the per-word tags are an intermediate step, not an error report.

Quick start

from transformers import AutoModelForTokenClassification, AutoTokenizer

name = "Voicelab/WERdict-pl"
tokenizer = AutoTokenizer.from_pretrained(name, trust_remote_code=True)
model = AutoModelForTokenClassification.from_pretrained(name, trust_remote_code=True).eval()

model.estimate_wer("Dzień dobry, z tej strony Jan Kowalski. Dzwonię w sprawie mojej polisy.", tokenizer)
# 2.53

Both calls need trust_remote_code=True. estimate_wer lowercases and strips punctuation, so the same sentence written without them returns the same number. Call model.cuda() to run on a GPU. For your own normalisation, call model.tag(words, tokenizer) with a list of words and get the per-word tags and gap sizes instead.

Scoring transcripts you have on disk:

from pathlib import Path

for path in sorted(Path("transcripts").glob("*.txt")):
    wer = model.estimate_wer(path.read_text(encoding="utf-8"), tokenizer)
    print(f"{path.name}\t{wer:5.1f}")

Any plain-text file works (.txt, .asr, or a .vtt with the timestamps stripped) because only the sequence of words is read.

Tested with transformers 5.17.0.

Quality classes

One number is hard to act on, so we group it. Ranges are closed on the right.

class	WER	what it means
good	≤ 10	send downstream
acceptable	10–20	review
poor	> 20	correct by hand

Results

Held-out suite: 13 test sets, 37 corpora, 3,493 recordings, 6,982 ratings. Each recording is transcribed by two engines: one Moonlight (in-house ASR), one Whisper-family and each transcript is one rating. The second column is the best a single constant reaches on the same rows.

measure	model	best constant
class accuracy, three classes	69.4%	47.6%
mean absolute error	6.35 points	10.41
bias	+1.98 points	–
F1, poor class	73.7	0
recall, poor class	80.3%	0
precision, poor class	68.1%	–
ROC-AUC, poor against the rest	0.904	0.500
Spearman rank correlation	0.775	0
corpus WER, true vs estimated	15.51 vs 16.74	–

Given two transcripts of the same recording from two engines, the model puts the worse one lower 83.5% of the time.

By kind of material, over every held-out corpus:

material	ratings	true WER	estimated	class accuracy	best constant	MAE
spontaneous speech	2,830	15.74	17.07	72.5%	36.9%	5.99
YouTube	2,252	21.19	22.21	73.4%	45.0%	6.87
courtrooms	242	17.43	17.57	71.1%	39.3%	5.14
broadcast media	2,572	10.25	9.81	70.8%	62.0%	3.72
read speech	2,544	7.86	11.35	70.0%	72.4%	6.62
telephone, 8 kHz	1,162	16.90	17.50	61.3%	41.5%	7.05
parliament, local government	1,514	7.30	8.69	66.7%	78.1%	3.99

Six of the seven land within 1.4 WER points of the truth; read speech is the exception at 3.5 points high.

Example

753 clips from the Polish test split of Common Voice 20.0 (CC0), each transcribed by two ASR systems. The model saw only the transcripts.

system	true WER	estimated
A	6.21	6.37
B	11.02	9.91

True WER is the word-level alignment against the official sentences in test.tsv, both sides lowercased and stripped of punctuationand this is the same normalisation estimate_wer applies, so these numbers are reproducible from public data.

Per clip the model picks the worse of the two systems 86.4% of the time, over the 273 clips where they differ. A clip here is seven words long, so only the pooled figure means anything.

The train split of the same Common Voice release is in our training data; these 753 recordings are not.

Limitations

• Polish only.
• Trained on output from two ASR engine families.
• It compresses the range. On transcripts that are truly good it adds +3.8 points; on truly poor ones it takes off 1.7 and its mean absolute error there is 11.7.
• It under-predicts when words are missing rather than wrong. A transcript that reads cleanly but dropped material is the case i the ahrd case for the model to spot.
• Short transcripts are harder: MAE 7.2 under 20 reference words and 9.8 between 50 and 150, against 5.3 above 150.
• Returns one number and no confidence interval.

How it works

The encoder tags every word:

tag	meaning
OK	matches the reference
SUB	wrong word in the place of a reference word
INS	extra word, nothing behind it in the reference
DEL_AFTER	at least one reference word is missing after this one

A deleted word leaves no token to tag, so a second head predicts how many words are missing after each word, as a Poisson rate. The estimate is assembled from both:

missing   = sum of the predicted gaps
reference = OK + SUB + DEL_AFTER + missing
WER       = 100 * (SUB + INS + missing) / reference

Text longer than 256 words is cut into consecutive windows of 256 and the per-word outputs are concatenated. Windows do not overlap and carry no context across the boundary.

Normalisation

estimate_wer applies NFKC, lowercases and drops punctuation; a hyphen, apostrophe or dot inside a word survives. Training used a heavier in-house normaliser, but the quality class is unchanged for 97.2% of held-out transcripts.

normalize also drops hesitations (yy, mhm, eee and a dozen more) because the training references had none, so an engine that writes them down would have every one counted as an insertion. Pass drop_hesitations=False to keep them.

The WER this model estimates is WER measured after that step and after numbers are written consistently on both sides. If you score against a verbatim reference that keeps hesitations, expect your WER to be a few points higher than the estimate.

Normalise consistently when you compare runs, or call model.tag and do it yourself.

Training data

56,686 recordings, 113,363 transcript and reference pairs, 187,052 windows, 23.0 million words of ASR output, 1,507 hours of audio. Every recording contributes exactly two transcripts, one Moonlight-family and one Whisper-family. Output from other ASR systems was not in training.

material	% of words
spontaneous speech, podcasts, interviews	46.3
telephone conversations, 8 kHz	35.6
read and dictated speech	9.2
broadcast media	4.9
local council sessions	3.9

Two public corpora are in training by their official train splits: Common Voice and Multilingual LibriSpeech. No recording from either test split was used, but measure this model on those benchmarks knowing that.

Licence and contact

Released under Apache 2.0, the licence of the encoder it is built on, OPI-PIB/pl-ModernBERT-base.

Questions: kryspin.musiol@voicelab.ai