README.md

---
title: FluentWhisper
emoji: ☕
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false
license: apache-2.0
suggested_hardware: zero-a10g
tags:
  - track:backyard
  - sponsor:modal
  - achievement:offgrid
  - achievement:welltuned
  - achievement:offbrand
  - achievement:fieldnotes
---

# FluentWhisper

**Speak messy. Read clean.**

A small LoRA adapter that teaches whisper-large-v3-turbo to take the fillers, repeats,
and false starts out of raw speech in one local pass, then show you exactly what it
removed next to the vanilla model.

It runs on the laptop you already own. No API keys, no cloud round trip, no frontier
model sitting in the loop. That makes it a workable alternative to cloud dictation for
anyone who talks faster than they edit, especially non-native English speakers.

As of June 2026, to our knowledge it is the only Apache-2.0, commercially usable
open-source model that removes filled pauses, discourse markers, repetitions, and
self-repairs end to end. On the DisfluencySpeech test split it scores 3.4% WER
(whisper-normalized, transcript C) against 9.4% for vanilla Whisper. That benchmark is a
single-speaker acted set, N=250, with a 95% confidence interval of about ±1pp.

## How it was trained

The training data is synthetic, and we built it ourselves. We started from clean text
transcripts in LibriSpeech (the original audio was thrown away), injected disfluencies
into that text with custom scripts plus the LARD tool, then voiced the messy versions
with Kokoro TTS rotated across 54 voices. That gave us roughly 23k pairs of
`(disfluent audio, clean text)` to fine-tune on. A later blend folded in about 4.5k rows
from the DisfluencySpeech train split to tidy up the label formatting.

So the model is trained on LibriSpeech-derived synthetic speech, not on DisfluencySpeech.
DisfluencySpeech is only the real-speech benchmark we report against.

- **Base:** `openai/whisper-large-v3-turbo`
- **Adapter:** [`pradachan/whisper-large-v3-turbo-disfluency-lora`](https://huggingface.co/pradachan/whisper-large-v3-turbo-disfluency-lora)
- **Training data:** synthetic, generated from [LibriSpeech](https://www.openslr.org/12/) transcripts and voiced with [Kokoro 82M TTS](https://huggingface.co/hexgrad/Kokoro-82M)
- **Benchmark:** [DisfluencySpeech](https://huggingface.co/datasets/amaai-lab/DisfluencySpeech) (arXiv:2406.08820)
- **Trained on:** [Modal](https://modal.com) serverless GPUs, which ran both the LoRA fine-tuning and the eval harness
- **Hardware:** ZeroGPU (`@spaces.GPU`), so live transcription runs on an on-demand A10G

The gallery examples are real results from the DisfluencySpeech test set, plus one honest
failure. Vanilla Whisper already deletes most "um/uh" fillers on its own, so the demo
deliberately shows what it does past that point.

## Limitation

The model deletes aggressively. Intentional repetitions like repeated digits, spelled IDs,
and phone numbers, along with some self-repairs, can be lost. Do not use it for verbatim,
legal, or numeric dictation without a downstream check.