# cleanup

> Status: implementation. Run `make smoke` first to validate wiring on CPU. Run `make all RUN_ID=r1` on a Vast.ai GPU for the real training pass. Run `make benchmark` locally on a laptop for the latency truth.

Fine-tune `Qwen/Qwen2.5-0.5B-Instruct` with LoRA to clean Mumble dictation transcripts. Remove fillers, repeats, and false starts. Fix punctuation and capitalization. Never paraphrase, summarize, or add content. Export to ONNX (fp32 + int8) for the Mumble Rust app's `ort` runtime.

See:
- [`docs/model_card.md`](docs/model_card.md) — published as the HuggingFace Hub README.
- [`docs/model_report.md`](docs/model_report.md) — human-readable evaluation report (filled in after a run).
- [`docs/HANDBOOK.md`](docs/HANDBOOK.md) — deep specs.
- [`../../handoffs/finetune-explainer.md`](../../handoffs/finetune-explainer.md) — conceptual walkthrough.
- [`../../handoffs/cleanup-execution-plan.md`](../../handoffs/cleanup-execution-plan.md) — operational plan with Vast.ai commands and budget.

## Pipeline

```
make sync                  install deps via uv
make data                  01: split the seed jsonl into train/val/test under data/pairs/
make train RUN_ID=r1       02: lora sft on qwen2.5-0.5b-instruct
make evaluate RUN_ID=r1    03: raw vs base vs fine-tune on the held-out test, plus adversarial check
make export RUN_ID=r1      04: merge lora and export fp32 + int8 onnx
make benchmark RUN_ID=r1   05: cpu latency, LOCAL only
make pack RUN_ID=r1        06: tar the run, print sha256 and the vastai copy line
make push RUN_ID=r1        publish to the hf hub
make all RUN_ID=r1         pipeline end to end on the gpu (no benchmark, no push)
make smoke                 tiny cpu run, no gpu needed
```

## Data

Source of truth: `data/seed/synthetic_pairs.jsonl` — **688 hand-curated (raw, clean) pairs** generated by a multi-agent workflow across 8 categories of dictation:

- `casual_messages` — Slack / texts
- `professional_emails` — work emails
- `meeting_notes` — voice memos during/after meetings
- `technical_dictation` — code comments, API notes, design sketches
- `todo_lists` — action items, shopping, plans
- `long_form_thoughts` — paragraph-length thinking out loud
- `questions_and_asks` — questions, requests, asks
- `mixed_content` — names, numbers, dates, times, addresses

Each pair is `{ raw: <Parakeet-shaped lowercase no-punct disfluent input>, clean: <proper English output> }`. The clean side is faithful by construction: every content word in `clean` exists in `raw` (modulo homophone fixes). The model is mechanically incapable of learning to invent content because the data never rewards it.

`scripts/01_download.py` splits the seed stratified-by-category into train/val/test (85/10/5) and writes `data/pairs/{train,val,test}.json` plus `data/pairs/meta.json`.

The 9 synthetic disfluency operators in `src/cleanup/data/inject.py` are kept for an optional v1.1 augmentation pass that expands the seed by 3-5x by stochastically corrupting clean sides. Off by default in v1.

## Layout

```
configs/
  data.yaml                seed path + split ratios
  inject.yaml              optional operator probabilities (v1.1)
  train.yaml               lora + adamw hyperparameters
src/cleanup/
  config.py                yaml -> typed dataclasses
  prompts.py               frozen system prompt
  infer.py                 model -> cleaned string (one-shot helper)
  data/
    download.py            load seed jsonl, split stratified, write data/pairs/
    inject.py              9 disfluency operators (v1.1 augmentation, off by default)
    tokenize.py            chatml format + the response_template for completion-only loss
  train/
    model.py               load qwen + apply lora + autodetect precision
    trainer.py             trl sftrainer wiring with completion-only collator
  eval/
    metrics.py             disfluency removal, punct f1, faithfulness, length, pass rate
    latency.py             cpu p50/p95/p99 sweep + realistic mix
    report.py              read jsons -> charts + markdown
  export/
    merge.py               peft merge_and_unload
    to_onnx.py             optimum onnx export
    quantize.py            ort int8 dynamic quantization
  pack/
    ship.py                tar + sha256 + vastai copy line
scripts/
  01_download.py           split seed jsonl into train/val/test
  02_train.py              lora sft on qwen2.5-0.5b-instruct
  03_evaluate.py           raw vs base vs fine-tune + adversarial questions
  04_export.py             merge + onnx + int8
  05_benchmark.py          cpu latency (LOCAL only)
  06_pack_and_ship.py      tar + sha256 + vastai copy line
  push_to_hub.py           hf hub publish (private by default)
docs/
  model_card.md            hf hub readme (with frontmatter)
  model_report.md          human-readable report (numbers filled after a run)
  HANDBOOK.md              deep specs
data/
  seed/synthetic_pairs.jsonl   the 612-pair source of truth
  seed/synthetic_pairs.csv     same data, human-inspectable
  seed/batches/                per-category breakdowns
  pairs/{train,val,test}.json  created by scripts/01_download.py
runs/<run-id>/
  model/                       peft adapter + tokenizer
  merged/                      transformers checkpoint after lora merge
  onnx/model.onnx              fp32 merged onnx
  onnx/int8/model.onnx         int8 quantized onnx
  eval.json                    quality metrics (raw, base, fine_tuned, adversarial)
  latency_benchmark.json       cpu latency (only after local benchmark)
  training_history.json        loss curves
dist/                          packed tarballs ready to scp
```