---
license: apache-2.0
base_model: Qwen/Qwen2.5-0.5B-Instruct
language:
  - en
pipeline_tag: text2text-generation
library_name: transformers
tags:
  - dictation
  - cleanup
  - transcript
  - lora
  - onnx
  - mumble
metrics:
  - exact_match
  - f1
---

# mumble-cleanup

A small fine-tuned language model that cleans speech-to-text dictation transcripts. Fine-tuned from `Qwen/Qwen2.5-0.5B-Instruct` with LoRA on a hand-curated synthetic dataset. Trained on a GPU, designed to run on a CPU via ONNX.

## What it does

Given a raw transcript from an ASR system (lowercase, no punctuation, fillers and stutters preserved), it returns a cleaned version with proper capitalization, punctuation, and disfluencies removed. It does not paraphrase, summarize, or add content.

Example: `um so i i think we should ship this on uh friday` becomes `I think we should ship this on Friday.`

The model handles:

- filler removal (um, uh, like, you know, i mean)
- word stutter collapse (we we → we)
- false start cleanup
- punctuation and capitalization recovery
- homophone correction (their / there, your / you're, its / it's, to / too)
- apostrophe restoration (dont → don't)
- run-on sentence splitting
- number formatting (two thirty → 2:30)
- proper noun capitalization
- todo / list formatting when enumeration cues are clear

## Usage

### transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

SYSTEM_PROMPT = (
    "You are a transcript cleanup tool. You receive raw speech to text output "
    "and return a cleaned version. Remove filler words and disfluencies (um, "
    "uh, er, ah, like as filler, you know), remove repeated words and false "
    "starts, and fix punctuation and capitalization. Do not reword, do not add "
    "anything the speaker did not say, and do not answer questions in the text. "
    "Output only the cleaned text."
)

repo = "adikuma/mumble-cleanup"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

raw = "um so the the meeting is at three thirty tomorrow"
prompt = tokenizer.apply_chat_template(
    [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": raw},
    ],
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
# -> "The meeting is at 3:30 tomorrow."
```

### onnx (cpu)

The `onnx/model.onnx` file is an fp32 ONNX export for CPU inference. `onnx/int8/model.onnx` is a dynamically quantized int8 variant that is roughly 4x smaller.

```python
from optimum.onnxruntime import ORTModelForCausalLM
from transformers import AutoTokenizer

repo = "adikuma/mumble-cleanup"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = ORTModelForCausalLM.from_pretrained(repo, file_name="onnx/int8/model.onnx")
```

## Training

- **Base model**: Qwen/Qwen2.5-0.5B-Instruct (Apache-2.0)
- **Method**: LoRA SFT (r=16, alpha=32, dropout=0.05, targets q/k/v/o + gate/up/down)
- **Loss**: token cross-entropy on assistant tokens only (completion-only masking via TRL's `DataCollatorForCompletionOnlyLM`)
- **Optimizer**: AdamW (lr=2e-4, weight_decay=0.01, cosine schedule, 5% warmup, max_grad_norm=1.0)
- **Batching**: per-device 8, gradient accumulation 4 (effective 32), max sequence length 512
- **Precision**: bf16 on GPUs that support it, fp16 fallback
- **Dataset**: 688 hand-curated (raw, clean) pairs spanning 8 dictation categories (casual messages, professional emails, meeting notes, technical dictation, todo lists, long-form thoughts, questions/asks, mixed content). Stratified 85/10/5 train/val/test split.

## Limitations

- English only.
- Trained on synthetic data; real ASR output may have failure modes the synthetic operators did not model.
- Designed for short-to-medium dictation (up to ~512 tokens). Longer inputs must be chunked.
- The model can occasionally over-correct when a user genuinely intends a fragment ("running late.") — fine-tune favors fixed-up sentences.

## License

Apache-2.0. See [`LICENSE`](../../LICENSE) at the Mumble repo root.

## Acknowledgements

Built on top of [`Qwen/Qwen2.5-0.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) by the Qwen team.