README.md

---
license: cc-by-nc-sa-4.0
task_categories:
  - automatic-speech-recognition
  - text2text-generation
language:
  - en
  - de
  - fr
  - ja
  - zh
tags:
  - asr
  - generative-error-correction
  - n-best
  - whisper
  - llm-memory
  - test-time-adaptation
  - agentic
pretty_name: Voice Memory
viewer: false
---

# Voice Memory

**A frozen LLM corrector + a small optimized text memory recovers ASR n-best oracle headroom — with zero weight updates and zero extra inference calls.**

Voice Memory stores the *skill* of ASR generative error correction (GER) outside the model, in a per-domain, human-readable [`memory.md`](memory.md) (typically under 1.5 KB). The memory is optimized purely in text space — rollout → reflect → bounded add/delete/replace edits, accepted only when a held-out selection metric strictly improves — and is simply prepended to the corrector's context at inference. The most valuable thing the memory learns is not knowledge but **calibrated restraint**: knowing when *not* to edit a hypothesis that was already right.

This repository releases the **memory artifacts**, the **inference-time code**, and the **result tables** from the paper.


## What's here

- **`memories/`** — 39 optimized memory files (plus 4 frozen backups) for 10 HyPoradise domains, 4 noise families, and 4 speech-translation directions. See [`memory.md`](memory.md) for the full index with per-memory results.
- **Inference & optimizer code** (top-level `.py`) — the complete pipeline: n-best loaders, frozen-corrector evaluation, the text-space memory optimizer, and metrics (WER suite, Recoverable Information Ratio ρ, Harmful Edit Rate).
- **Results** — `results.tsv`, `breadth_results.tsv`, `robust_results.tsv`, `robust_baseline.tsv`, plus analysis notes in [`docs/MEMORY_QUALITY.md`](docs/MEMORY_QUALITY.md) and [`docs/ROBUST_VM.md`](docs/ROBUST_VM.md).

The underlying corpora (HyPoradise v0, Robust HyPoradise) are **not redistributed** here — see [Data provenance](#data-provenance--attribution).

## Quickstart

```bash
pip install -e .
```

**Tier 1 — local baselines, no API key.** Download [HyPoradise v0](https://huggingface.co/datasets/PeacefulData/HyPoradise-v0) (Whisper 5-best JSON, `train/` + `test/`) and point `HYPO_ROOT` at it:

```bash
export HYPO_ROOT=/path/to/HyPoradise-v0
python baseline.py wsj_score        # 1-best vs n-best-oracle WER, no API calls
```

**Tier 2 — inference with a released memory.** Requires a MiniMax API key (`MINIMAX_API_KEY` in env or a local `.env`); the corrector is MiniMax-M3, frozen:

```bash
python run.py --domain wsj_score --memory memories/memory_wsj_score_sem.md --skip_d --workers 3
```

Arms: A = 1-best, B = n-best oracle, C = static GER (no memory), D = few-shot GER, E = Voice Memory. Results append to `results.tsv`.

**Tier 3 — re-optimize a memory from scratch** (spends API tokens; keep `--workers 3`):

```bash
python memory_opt.py chime4 --gate sem --workers 3
```

Gates: `wer` | `sem` | `joint` (`sem` — the paper's recommended gate — needs the `[sem]` extra: on-device MLX SBERT, Apple silicon only). Note `run.py --gate` accepts `wer|spr` (legacy surface), and `breadth.py` imports the semantic module at startup, so it also needs `[sem]`.

**Noise-robust runs** need the Robust-HyPoradise `.pt` corpora from the RobustGER release: set `RH_ROOT=/path/to/Robust-HyPoradise`, install `[robust]` (torch, first load only), and use e.g. `python robust_table.py --families chime4 --gate sem --workers 3`.

## Headline results

Full 10-domain breadth sweep (MiniMax-M3 corrector, semantic gate, full test splits — `breadth_results.tsv`); ρ = fraction of the 1-best→oracle headroom recovered, ρ>1 beats the 5-best oracle:

| Domain | 1-best WER | Oracle | Static GER | Voice Memory | ρ |
|---|---|---|---|---|---|
| atis | 7.89 | 4.74 | 6.71* | **4.86** | **0.96** |
| wsj_score | 4.90 | 3.60 | 5.76 | **4.28** | 0.48 |
| cv | 15.42 | 11.31 | 13.07 | **13.01** | 0.59 |
| chime4 | 9.85 | 7.48 | 9.68 | **9.36** | 0.21 |
| swbd | 15.61 | 12.34 | 15.25 | **15.04** | 0.17 |
| lrs2 | 11.66 | 6.70 | 11.14 | **10.86** | 0.16 |
| coraal | 25.57 | 23.68 | 25.11 | 25.28 | 0.15 |
| td3 | 4.08 | 2.92 | 4.36 | 4.17 | −0.09 |
| ls_other | 3.67 | 2.27 | 4.13 | 4.11 | −0.32 |
| ls_clean | 1.79 | 0.89 | 2.40 | 2.09 | −0.33 |

\* atis static-GER row from `results.tsv` (6.71%); its breadth-sweep row (6.31%) used the same commit's rerun.

The organizing result is a **scaling law**: ρ correlates with 1-best WER at r = +0.90 — correction pays where headroom exists and should abstain at the clean floor. With a WER-gated memory, wsj_score reaches **3.57% (ρ = 1.03)** and atis **4.12% (ρ = 1.20)** — *past the 5-best oracle* (`results.tsv`). Static GER over-corrects everywhere (up to 64% harmful edits on wsj); memory drops that to 35%.

Noise-robust results (`robust_results.tsv`, one memory per noise family): CHiME-4 real recorded noise wins 4-for-4 (e.g. dev_real 7.75→7.31 while static GER *raises* it to 8.13); NOIZEUS wins at low SNR (5 dB: 14.51→12.60); additive-noise-over-read-speech (VB-DEMAND, LS-FreeSound) is correctly near-null. See `docs/ROBUST_VM.md`.

**TSV schemas.** `results.tsv`: commit, domain, arm, wer, mer, wil, wip, cer, rho, her, n, desc. `breadth_results.tsv`: commit, domain, gate, arm, wer, cer, sem, rho, bytes, n. `robust_results.tsv`: domain, family, arm, wer, mer, cer, rho, her, n, mem_bytes, desc. `robust_baseline.tsv`: domain, family, arm, wer, mer, cer, rho, n.

## Data provenance & attribution

| This repo | Derived from / evaluated on | License / source |
|---|---|---|
| `memories/memory_<domain>_*.md`, `results.tsv`, `breadth_results.tsv` | **HyPoradise v0** (Whisper large-v2 5-best), Chen et al., NeurIPS 2023 | CC-BY-NC-4.0 — [PeacefulData/HyPoradise-v0](https://huggingface.co/datasets/PeacefulData/HyPoradise-v0) |
| `memories/memory_rh_*.md`, `robust_results.tsv`, `robust_baseline.tsv` | **Robust HyPoradise / RobustGER**, Hu et al., ICLR 2024, [arXiv:2401.10446](https://arxiv.org/abs/2401.10446) | see upstream release |
| `memories/memory_{de,fr,ja}_tr.md` | **CoVoST 2** X→En speech translation n-best (via GenTranslate/HypoTranslate) | CC-BY-NC-4.0 (see upstream) |
| `memories/memory_zh_tr.md` | **FLEURS** zh→En (via GenTranslate/HypoTranslate) | CC-BY-4.0 |

No corpus audio, references, or hypotheses are redistributed in this repository; result TSVs contain aggregate metrics only. The memory files are **LLM-generated content**: written by MiniMax-M3 (default), Claude Opus 4.7 agent loops (`*_claude*.md`), or MiniMax-M2.7 (`*_M27.md`, intentionally empty — see `memory.md`).

## Limitations

- Main-body results use a single frozen corrector (MiniMax-M3); the paper's appendix reproduces the effect with Qwen3-30B-A3B.
- The semantic gate runs on-device via MLX (Apple silicon). On other platforms use `--gate wer`, or swap in any sentence encoder (interface in `semantic.py`).
- Three memory files are intentionally empty (optimizer accepted no edits, or the writer model was below formation competence) — an empty memory is equivalent to static GER by construction.
- ρ depends on the n-best list (Whisper large-v2 5-best throughout); a different ASR front end changes the headroom.

## Upstream citations

```bibtex
@inproceedings{chen2023hyporadise,
  title     = {HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models},
  author    = {Chen, Chen and Hu, Yuchen and Yang, Chao-Han Huck and Siniscalchi, Sabato Marco and Chen, Pin-Yu and Chng, Eng Siong},
  booktitle = {NeurIPS},
  year      = {2023}
}
@inproceedings{hu2024robustger,
  title     = {Large Language Models are Efficient Learners of Noise-Robust Speech Recognition},
  author    = {Hu, Yuchen and Chen, Chen and Yang, Chao-Han Huck and Li, Ruizhe and Zhang, Chao and Chen, Pin-Yu and Chng, Eng Siong},
  booktitle = {ICLR},
  year      = {2024}
}
```

## License

CC-BY-NC-SA-4.0 for everything in this repository (memories, code, result tables), consistent with the non-commercial terms of the upstream corpora the memories were optimized on.