A fantastically accurate model!
I converted it to GGUF format and then compressed it using the q5_k_m quantization. This reduced the original size to one-third—just 1.1 GB. I’m using it on a Samsung S24 with the "Offline Voice Input" app. It’s a bit slow (running at about 0.5x the duration of the speech), but the recognition accuracy is unbeatable! I haven’t come across any other offline ASR model that handles the Hungarian language with such a low error rate. Thank you!
Épp most futtattam egy tesztet. A Saját sarpba/whisper-hu-large-v3-turbo-finetuned modellem, csak egy lehelletnyivel teljesít gyengébben cserébe kb 2,5* gyorsabb.
modell WER % CER % RTF (3090) megjegyzés
BuzzASR/hungarian (fp16) 9,96 1,84 34,1× legjobb pontosság
nvidia/parakeet-tdt-0.6b-v3 (bf16) 10,45 1,99 43,8× natív szó-időbélyeg
sarpba/whisper-hu-large-v3-turbo-finetuned (fp16) 10,85 1,94 79,9×
nvidia/canary-1b-v2 (bf16) 14,54 3,10 150,3× leggyorsabb; HF-portban nincs időbélyeg
openai/whisper-large-v3 (fp16) 17,80 4,14 22,9×
openai/whisper-large-v3-turbo (fp16) 19,96 4,68 81,6×
q8-ban semmít sem veszít szinte a pontosságából.
Szia 🇭🇺 @spy76573 ! Thank you so much, this made my day! :)
Running fully offline on a phone for everyday Hungarian voice input is exactly the kind of use we hoped for. And nice work getting the model down to just 1.1 GB with Q5_K_M!
Two quick questions:
- When you say 0.5× the duration of the speech, does a 10-second clip take roughly 5 seconds to transcribe, or closer to 20?
- Would you be open to sharing your GGUF conversion workflow and a few details about your Offline Voice Input setup?
We're exploring extending BuzzASR with quantized, edge-deployable checkpoints so people can run offline ASR directly on their phones and laptops, and your experience and feedback would be incredibly valuable for that.
Also, if you're attending EMNLP, feel free to reach out to Catherine Arnett (@catherinearnett ), our co-author, who'll be there in person. Unfortunately I couldn't make it this year, but I hope to visit Hungary someday!
Best,
Shivam
@sarpba Szia🇭🇺!! Thank you for running the comparison, really appreciated!
It's great to hear that Q8 loses almost nothing in accuracy, since we're exploring releasing official quantized checkpoints for on-device use. Thank you for sharing that!! However, part of your table seems to have come through garbled on our end. Could you share it again (Hungarian is perfectly fine), along with which test set, metrics and hardware you used?
Also, if you're attending EMNLP, feel free to reach out to Catherine Arnett (@catherinearnett ), our co-author, who'll be there in person. Unfortunately I couldn't make it this year, but I hope to visit Hungary someday!
Best,
Shivam
@sarpba Thank you so much for running these benchmarks and sharing the evaluation dataset! :)
It's really encouraging to see BuzzASR/hungarian performing so competitively, especially against newer ASR architectures. And congratulations on your own Hungarian Whisper Turbo fine-tune! The accuracy–speed tradeoff is quite impressive.
I also appreciate the effort you've put into making the evaluation reproducible, including the normalization details and test-set provenance.
I'm currently exploring the possibility of extending BuzzASR with quantized, edge-deployable checkpoints for offline inference on phones and laptops. I am really curious about your observation on Q8 version preserving accuracy .
Would you be interested in exploring FP16 vs. Q8/Q5_K_M evaluations for BuzzASR/hungarian using your benchmark? It could help us understand the accuracy–efficiency tradeoffs and potentially establish a reproducible evaluation setup for future quantized releases.
Thanks again for taking the time to benchmark our work. Really appreciate it!
Best,
Shivam