Teaching Whisper to clean up speech as it transcribes
"So um basically I wanted to, well, you know, the the server is uh down and I mean we need to fix it."
What that person meant was "the server is down and we need to fix it." Whisper, OpenAI's speech-to-text model, transcribes the first version. It writes down what you said, including the fillers and the false starts.
FluentWhisper is a small LoRA adapter that teaches whisper-large-v3-turbo to skip the mess and write the clean version directly, in a single pass, with no second cleanup step and no cloud round trip. These are the field notes on how it got built. What worked, what broke, and where it still falls over.
The part that isn't obvious
Most people assume the hard job is deleting "um" and "uh." It isn't. Whisper already strips those on its own. It was trained to.
What Whisper keeps are the harder things:
- discourse markers like "you know," "I mean," and "well"
- accidental repetitions like "the the server"
- self-corrections like "I want to go to Bom, I mean Delhi"
The adapter handles all three. As of June 2026 it's the only open Apache-2.0 model I've found that removes filled pauses, discourse markers, repetitions, and self-repairs in one go. The commercial APIs (AssemblyAI, Deepgram, Google) only touch "um" and "uh" in their disfluency modes. So the demo deliberately shows what happens past the filler removal you already get for free.
Building a dataset that didn't exist
The first wall I hit: there was no training data for this. You need pairs of messy audio and clean text, and nobody had published that.
So I made it. The pipeline starts with clean sentences from LibriSpeech, then injects mess into the text using four categories: filler words, word repetitions, self-repairs, and a mixed bucket with all of them at once. Each corrupted sentence gets turned into audio with Kokoro, an Apache-2.0 text-to-speech engine, using 54 different voices so the model doesn't overfit to one speaker. Every sample has to clear two gates before it counts: the audio has to be valid, and vanilla Whisper has to transcribe the clean reference at under 8% word error rate, so I know the target is actually reachable.
That gave me 23,285 synthetic samples, plus 4,500 clips of real human disfluent speech from DisfluencySpeech, a research set of one speaker re-reading Switchboard conversations. The whole pipeline runs on Modal serverless GPUs with a single command, which is what made the fan-out generation and the dozens of eval runs affordable.
The experiments, and the bugs
The first real model, v1, trained on about 18,600 synthetic samples for 4,000 steps on an A10G. On the 250-clip test set it scored 8.87% WER against vanilla's 9.42%. Slightly better, barely significant.
Except my first reading said v1 scored 25.33%, which would have been a disaster. That number was wrong. v1 writes lowercase text with no punctuation, and I was scoring it against references that had both. Once I switched to whisper_norm, the standard normalization people use for this, v1 came out ahead. So before touching the model again, I wrote a proper scoring script. If the measurement is wrong, nothing downstream of it means anything.
v2 was supposed to be the jump. The idea was to blend in those 4,500 real human clips so the model heard real voices, not just TTS. Instead v2 looked terrible at 13.74% WER. The cause was a one-line bug: while normalizing the real-speech labels, I stripped the apostrophes, so "don't" became "dont" and "we're" became "were." The synthetic labels kept theirs. The model was being trained on two contradictory spelling conventions at once. When I mathematically restored the apostrophes in v2's output, the real score came out around 6.3%. So the approach had been fine all along, and the only thing broken was how I cleaned the labels.
Before committing to a long retrain, I ran three short 200-step probes to answer specific questions. Freezing the encoder and training only the decoder made things worse, which told me the audio side still matters. SpecAugment didn't help enough to keep. The one that surprised me was the control: same settings as v1 but stopped at 200 steps, and it scored 7.76%, beating v1's 4,000-step result. v1 had been overtrained. The model picks up the skill early and then slowly overfits to synthetic speech. That finding reshaped the whole final plan.
v3 used the fixed labels and two learning rates, with a checkpoint saved every 250 steps and scored on a held-out validation set. The gentler run (1e-4) at step 2000 came out best at 3.66% validation WER. I picked it over a marginally better earlier checkpoint because the checkpoints around step 2000 were all stable, whereas the earlier one looked like it could have been a lucky single reading. I'd rather trust a stable neighbourhood than one good number.
The one-shot test
I kept the 250-clip test set sealed the entire time. Every decision came from validation. Then I ran the winner on the test set exactly once.
| Model | WER (whisper_norm) |
|---|---|
| Vanilla Whisper | 9.42% |
| v1 | 8.87% |
| v3 winner | 3.42% |
That's roughly six points better than vanilla, with a 95% confidence interval of about [+5.0, +7.0], so I'm fairly sure the gain is real and not noise. Test at 3.42% against validation at 3.66% is a 0.24 point gap, which suggests it generalizes well inside this benchmark, though that's still one benchmark.
Where it breaks
The test set looks great, but it's one speaker reading scripted lines, so I built 100 synthetic probe sentences to stress the failure modes. The honest results:
Repeated digits get eaten. "Zero zero zero one one two" can collapse to something shorter, and only about 20% of repeated-digit cases survived. Do not point this at phone numbers or account IDs without a check afterward. Self-repairs are fixed correctly only about 40% of the time, and the model often keeps both the false start and the correction instead of just the correction. On the other hand, discourse-marker removal is its strongest skill, and filler-hedge removal improved over v1. These probes use TTS audio, not real speech, so read the numbers as directional.
I think that tradeoff is fine for dictation and transcript cleanup, which is what it's for, and clearly wrong for anything where a dropped digit matters. The model card says the same thing.
What I still don't know
I tested on one speaker. Whether it holds up across accents, second-language English, or genuinely spontaneous conversation isn't established. I also haven't checked whether Kokoro renders "um" and "uh" with realistic prosody, or whether placing disfluencies more deliberately would beat the random injection I used. Real disfluencies tend to cluster at the start of sentences and land on function words, and I never tried to match that. Mixing in more TTS engines like Piper or Parler is another loose end. Those are the next things I'd want to test.
The short version: the adapter cuts Whisper's error rate on disfluent speech from 9.4% to 3.4% on this benchmark. It's aggressive and it'll occasionally eat a number. It's also Apache-2.0, so anyone can pick it up and build on it.
Everything is public if you want to poke at it:
- Demo: FluentWhisper Space
- Model: pradachan/whisper-large-v3-turbo-disfluency-lora
- Dataset: pradachan/auto-whisper-disfluency
Trained end to end on Modal.