BuzzASR

/

korean

Like 6

Follow

lemn-lab 20

Hi @Gonow1 :)

First of all, thank you for such a thorough evaluation! I'm a strong believer that evaluation is a research problem in itself, and your work on kocer is a great example of that. Genuinely appreciated!

Tbh, when I was first building BuzzASR/korean, Hangul gave me WER above 100% even on clean audio. After scratching our heads for quite a while, we traced it to the tokenizer. Our recipe replaces Whisper's GPT-2-style byte-level BPE with a new per-language tokenizer, and the pre-tokenizer splitting rule we wrote for it was passed to the Hugging Face tokenizers library as a plain string. The library treats a plain string as literal text, so the rule never matched and was effectively skipped.

Training still "worked", but the tokenizer could merge across boundaries we intended to preserve, producing tokens spanning entire phrases. The fix was one line, wrapping the rule as Regex(RULE), and after retraining things looked much healthier. We wrote it up here: https://lemn-lab.github.io/buzz-asr/tokenizer/#a-bug-worth-knowing-about

I mention it because both experiences highlight how much Korean text boundaries and normalization choices matter, whether during tokenization or evaluation. It's nice to see an evaluation that distinguishes spacing and number notation from actual recognition errors.

On your results: they broadly match our understanding. The model was fine-tuned only on FLEURS and Common Voice 25 training data, so a clear in-domain gain on CV, roughly a tie on FLEURS, and a small gap on Zeroth (which we never trained or evaluated on, and where number formatting appears to contribute to the difference) are consistent with what we'd expect. Thanks for the overlap check too :) that was useful context for me.

We'd also welcome a row with the model card's decoding settings (no_repeat_ngram_size=3, repetition_penalty=1.2) so the comparison lines up more closely with our reported setup.

One gap we list in the paper is that our evaluation focuses on read speech. If you ever try BuzzASR/korean on telephone or conversational audio, we'd love to hear how it does, and we'd be happy to explore working together on that. Feel free to reach me, I'd love a collab!

Thanks again!

Thank you for the kind reply and for sharing the tokenizer story! The plain-string vs Regex(RULE) pitfall is a great find, and it is a nice parallel: in both tokenization and evaluation, Korean boundaries quietly change the numbers.

As you suggested, I reran BuzzASR/korean with the model-card settings (no_repeat_ngram_size=3, repetition_penalty=1.2, otherwise unchanged), and since you mentioned conversational audio as an open gap, I also added KsponSpeech eval (spontaneous conversational speech, clean + other, 6,000 utterances).

• Model-card settings: slightly worse on all four sets in this setup, significantly so on the three public sets (sentence-level paired bootstrap). With these settings the spacing+punctuation-normalized CER on CV27 is 4.25, very close to the 4.24 in your card (different CV version and normalization, but consistent).
• Conversational speech (KsponSpeech): BuzzASR/korean is 4th of seven systems under both settings (spelling-side reference), behind TeamUNIVA/qwen3_asr_1.7b_ko (6.32), TeamUNIVA/qwen3_asr_1.7b_ko_beta (7.08) and Qwen/Qwen3-ASR-1.7B (8.63). As with Common Voice, each model seems strongest on data closest to what it was tuned for.
• Runaway outputs: 19 of 6,000 KsponSpeech utterances with greedy decoding, 17 with the card settings (criterion: the utterance's kocer error count exceeds its reference length; secondary check: output at least twice the reference length). Since repetition control barely changes this, it looks like a response to conversational audio rather than a decoding artifact. I can share the utterance IDs; the transcripts can't be redistributed under the AI-Hub terms.
• A correction to my original post: I wrote that BuzzASR/korean was 1st of six systems pooled over the three public sets. After adding TeamUNIVA/qwen3_asr_1.7b_ko, the two are statistically tied for first (3.90 vs 3.92).

On collaboration: I'd be glad to. Conversational audio is covered above, so telephone audio is the remaining gap. I'll run BuzzASR/korean and the same baselines on public Korean telephone speech from AI-Hub and share the results here. What's the best way to reach you?

Card-setting hypotheses for the public sets are in benchmarks/results/_buzzasr-korean-cardgen/ (https://github.com/Choi-kwan-u/kocer); pip install kocer==0.2.0 reproduces the kocer column.

As promised, here are the telephone results.

Data: AI-Hub "Low-quality telephone network speech recognition" (저음질 전화망 음성인식), validation split, HR domain (calls from job seekers and enquiries about corporate training): 1,936 utterances, 3.6 h, 85 calls. The audio is 8 kHz, upsampled to 16 kHz. Decoding is the same greedy setup as before; spelling-side reference.

• BuzzASR/korean is 2nd of seven, statistically tied with TeamUNIVA/qwen3_asr_1.7b_ko (0.15 pt, paired bootstrap 95% CI [−0.44, +0.12]) and clearly ahead of the rest. The one system ahead of it is whisper-large-v3 zero-shot, by 0.58 pt [0.43, 0.74], so on telephone audio the fine-tune does not beat its base model. That seems consistent with the read-speech training data.
• No runaway outputs: 0 of 1,936 for BuzzASR (19 of 6,000 on KsponSpeech). Since these calls are also spontaneous speech, the KsponSpeech runaways seem tied to something in that corpus rather than to conversational speech as such. I haven't looked further.
• Model-card settings: 6.50, worse than greedy by 0.90 pt [0.77, 1.02], still with no runaways. Across all five sets, greedy has been the better setting.
• Reference choice matters here too. The transcripts use the same dual notation as KsponSpeech. Switching from the spelling to the pronunciation side raises BuzzASR's spacing+punctuation CER by 4.65 pt (2.9–4.8 across models); with kocer the shift is 0.33 pt. On plain CER, TeamUNIVA ranks first because spacing makes up 38% of BuzzASR's raw errors versus 20% of TeamUNIVA's.
• Caveat: one domain (HR) of one dataset; the corpus has other domains I haven't run.

Per-model tables and scripts are in benchmarks/ (https://github.com/Choi-kwan-u/kocer); the audio and transcripts can't be redistributed under the AI-Hub terms, so only aggregate numbers are published. Happy to run other domains if that would help, and no rush on the contact question.

Hi @Gonow1 !

Wow, thank you for following through on both the conversational and telephone evaluations! This is genuinely useful, especially since neither domain was covered in our original evaluation.

A few things really stand out to me. First, greedy decoding consistently outperforming our model-card settings across all five datasets is something we should revisit. We'll check whether greedy also wins for other languages and update the model card if so. Second, the 19 runaway outputs on KsponSpeech versus zero on the telephone set is intriguing. It suggests something more specific is going on than conversational speech alone, and I'd be very interested in looking at those utterance IDs and the failure patterns.

The telephone results are encouraging too, even though Whisper-large-v3 retains an edge there. And your analysis of how reference notation changes both error rates and rankings is exactly why I think evaluation deserves more attention as a research problem. Thanks also for the correction on the pooled ranking, I really appreciate the care.

I'd love to explore this further with you. You can reach me directly at shs046@ucsd.edu. Perhaps we could set up a short call to discuss the findings, the decoding behavior, and whether there's an interesting direction for a joint evaluation study.

Thanks again for putting so much care into this work! :)