README.md

metadata

title: FluentWhisper
emoji: ☕
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: apache-2.0
suggested_hardware: zero-a10g
tags:
  - track:backyard
  - sponsor:modal
  - achievement:offgrid
  - achievement:welltuned
  - achievement:offbrand
  - achievement:fieldnotes

FluentWhisper

Speak messy. Read clean.

A small LoRA adapter that teaches whisper-large-v3-turbo to take the fillers, repeats, and false starts out of raw speech in one local pass, then show you exactly what it removed next to the vanilla model.

It runs on the laptop you already own. No API keys, no cloud round trip, no frontier model sitting in the loop. That makes it a workable alternative to cloud dictation for anyone who talks faster than they edit, especially non-native English speakers.

As of June 2026, to our knowledge it is the only Apache-2.0, commercially usable open-source model that removes filled pauses, discourse markers, repetitions, and self-repairs end to end. On the DisfluencySpeech test split it scores 3.4% WER (whisper-normalized, transcript C) against 9.4% for vanilla Whisper. That benchmark is a single-speaker acted set, N=250, with a 95% confidence interval of about ±1pp.

How it was trained

The training data is synthetic, and we built it ourselves. We started from clean text transcripts in LibriSpeech (the original audio was thrown away), injected disfluencies into that text with custom scripts plus the LARD tool, then voiced the messy versions with Kokoro TTS rotated across 54 voices. That gave us roughly 23k pairs of (disfluent audio, clean text) to fine-tune on. A later blend folded in about 4.5k rows from the DisfluencySpeech train split to tidy up the label formatting.

So the model is trained on LibriSpeech-derived synthetic speech, not on DisfluencySpeech. DisfluencySpeech is only the real-speech benchmark we report against.

• Base: openai/whisper-large-v3-turbo
• Adapter: pradachan/whisper-large-v3-turbo-disfluency-lora
• Training data: synthetic, generated from LibriSpeech transcripts and voiced with Kokoro 82M TTS
• Benchmark: DisfluencySpeech (arXiv:2406.08820)
• Trained on: Modal serverless GPUs, which ran both the LoRA fine-tuning and the eval harness
• Hardware: ZeroGPU (@spaces.GPU), so live transcription runs on an on-demand A10G

The gallery examples are real results from the DisfluencySpeech test set, plus one honest failure. Vanilla Whisper already deletes most "um/uh" fillers on its own, so the demo deliberately shows what it does past that point.

Limitation

The model deletes aggressively. Intentional repetitions like repeated digits, spelled IDs, and phone numbers, along with some self-repairs, can be lost. Do not use it for verbatim, legal, or numeric dictation without a downstream check.