Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Gjusev 
posted an update 3 days ago
Post
94
A transcript can read fine and still get your phone number wrong. So I stopped scoring words and started scoring slots.

VoxEntity-EU is an open German/French benchmark: does a speech model capture a dictated name, phone number, email, date, address or reference code exactly? One wrong character fails the slot.

Whisper large-v3-turbo 34.2% · Voxtral Mini 3B 37.1% · correct-text ceiling 90.8%. 240 items per model, 95% CIs in the report. Emails: 0 to 20%.

My first extractor version produced a wrong headline: "ASR is the bottleneck." It was my parser. I retracted it in public commits and shipped what survived.

Read with care: synthetic speech (Qwen3-TTS), simulated phone line, provisional. A second TTS scores 10 points higher, so both results are published.

Dataset
Gjusev/voxentity-eu

Demo
Gjusev/voxentity-eu-results

Full record, retraction included https://github.com/Gjusev/voxentity-eu

10 of Whisper's 70 omissions are not mishearing. The decoder dropped an entity that is in the audio.

I went through the frozen transcripts in predictions/. On 10 of the 240 test files Whisper turbo keeps the framing words and loses the entity. 9 stop right after the lead-in: "Schicken Sie es an", "Ich buchstabiere die Adresse", "Je vous dicte.", "Langsam!". One keeps only the closing "Précisément." Clips run 5.8 to 13.9 s, so this is not a 30 s window cut.

Same master audio, and Voxtral writes the entity on all 10. "Je vous dicte : Fabrice soulignement de la croix verne @example .com". Whisper's own telephone copy of that clip has it too.

So the audio is fine, and tts_alert (both systems far) is right to keep them in. They split 5 clean, 5 telephone, so the channel delta is untouched. Dev adds one more (dev-fr-phone-005, clean), still 0 for Voxtral.

It does change what "omission" means. 10 files is 4.2 pp, bigger than the -2.9 pp Whisper vs Voxtral gap. Voxtral only gets 1 of those 10 right itself, so the gap barely moves. The label does: these are a decoding stop under greedy, not a model that heard nothing.

Would you split early-stop out as its own error class, or rerun Whisper with temperature fallback as a separate row?