Abstract
A production bilingual Greek-English speech recognition system was engineered through iterative training, data filtering, ablation studies, and model ensembling to meet strict quality gates and achieve competitive benchmark results.
We report a multi-month engineering program to build Sophea, a production bilingual Greek-English automatic speech recognition system. We evaluate the system against nine production gates covering Greek and English word error rate, language identification, and hallucinations on non-speech audio. Across twenty-three training iterations and two model architectures, no training-data composition passed all nine gates simultaneously. Meeting the Greek noisy-environment target required about 1,500 steps of dense domain exposure, while preserving English language identification tolerated only about 250 steps, or about 1,250 with a rebalanced mix that reduced Greek accuracy. We describe a six-stage data pipeline in which calibrating an audio-quality filter against in-domain anchors reduced the discarded share of scored Greek audio from 98.7 percent to 10.6 percent. A pre-registered ablation isolated a hallucination defect to one training-data package. A three-model ROVER ensemble increased gate coverage from 4-7 of 9 for individual models to 9 of 9 and reduced overlapping-speech WER from 53.35 percent to 37.87 percent, a 29 percent relative improvement. A separate learned per-clip arbiter over two models is listed as sophea/asr-k1 (preview) on the public Open ASR Leaderboard, with 4.26 percent average WER across eight public English test sets, and reaches 25.88 percent WER on live Greek noisy-environment traffic. We also document five cases in which a measurement tool produced a plausible but incorrect result and seven substantial approaches that were evaluated but not shipped. No model weights or training data are released; we report methodology and quantitative results only.
Community
Hi folks, author here.
This is the story of getting Sophea ASR model, our Greek-English ASR, into production. We set nine quality gates (Greek WER, English WER, language ID, hallucination on silence, etc.) and wouldn't ship until all passed.
No single model ever did. Best of 23 training runs across two architectures was 7 of 9. Greek in noise wanted ~1,500 steps of in-domain data; English language ID broke after ~250. A three-model ROVER ensemble got us to 9 of 9 and cut overlapping-speech WER from 53% to 38%.
Two things worth your time: calibrating our audio filter against in-domain anchors dropped discarded Greek audio from 98.7% to 10.6%. And we document five times a measurement tool gave a plausible wrong answer, plus seven things we built and didn't ship.
Get this paper in your agent:
hf papers read 2609.13498 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper