huckiyang/voice-memory

Voice Memory

A frozen LLM corrector + a small optimized text memory recovers ASR n-best oracle headroom — with zero weight updates and zero extra inference calls.

Voice Memory stores the skill of ASR generative error correction (GER) outside the model, in a per-domain, human-readable memory.md (typically under 1.5 KB). The memory is optimized purely in text space — rollout → reflect → bounded add/delete/replace edits, accepted only when a held-out selection metric strictly improves — and is simply prepended to the corrector's context at inference. The most valuable thing the memory learns is not knowledge but calibrated restraint: knowing when not to edit a hypothesis that was already right.

This repository releases the memory artifacts, the inference-time code, and the result tables from the paper.

What's here

• memories/ — 39 optimized memory files (plus 4 frozen backups) for 10 HyPoradise domains, 4 noise families, and 4 speech-translation directions. See memory.md for the full index with per-memory results.
• Inference & optimizer code (top-level .py) — the complete pipeline: n-best loaders, frozen-corrector evaluation, the text-space memory optimizer, and metrics (WER suite, Recoverable Information Ratio ρ, Harmful Edit Rate).
• Results — results.tsv, breadth_results.tsv, robust_results.tsv, robust_baseline.tsv, plus analysis notes in docs/MEMORY_QUALITY.md and docs/ROBUST_VM.md.

The underlying corpora (HyPoradise v0, Robust HyPoradise) are not redistributed here — see Data provenance.

Quickstart

pip install -e .

Tier 1 — local baselines, no API key. Download HyPoradise v0 (Whisper 5-best JSON, train/ + test/) and point HYPO_ROOT at it:

export HYPO_ROOT=/path/to/HyPoradise-v0
python baseline.py wsj_score        # 1-best vs n-best-oracle WER, no API calls

Tier 2 — inference with a released memory. Requires a MiniMax API key (MINIMAX_API_KEY in env or a local .env); the corrector is MiniMax-M3, frozen:

python run.py --domain wsj_score --memory memories/memory_wsj_score_sem.md --skip_d --workers 3

Arms: A = 1-best, B = n-best oracle, C = static GER (no memory), D = few-shot GER, E = Voice Memory. Results append to results.tsv.

Tier 3 — re-optimize a memory from scratch (spends API tokens; keep --workers 3):

python memory_opt.py chime4 --gate sem --workers 3

Gates: wer | sem | joint (sem — the paper's recommended gate — needs the [sem] extra: on-device MLX SBERT, Apple silicon only). Note run.py --gate accepts wer|spr (legacy surface), and breadth.py imports the semantic module at startup, so it also needs [sem].

Noise-robust runs need the Robust-HyPoradise .pt corpora from the RobustGER release: set RH_ROOT=/path/to/Robust-HyPoradise, install [robust] (torch, first load only), and use e.g. python robust_table.py --families chime4 --gate sem --workers 3.

Headline results

Full 10-domain breadth sweep (MiniMax-M3 corrector, semantic gate, full test splits — breadth_results.tsv); ρ = fraction of the 1-best→oracle headroom recovered, ρ>1 beats the 5-best oracle:

Domain	1-best WER	Oracle	Static GER	Voice Memory	ρ
atis	7.89	4.74	6.71*	4.86	0.96
wsj_score	4.90	3.60	5.76	4.28	0.48
cv	15.42	11.31	13.07	13.01	0.59
chime4	9.85	7.48	9.68	9.36	0.21
swbd	15.61	12.34	15.25	15.04	0.17
lrs2	11.66	6.70	11.14	10.86	0.16
coraal	25.57	23.68	25.11	25.28	0.15
td3	4.08	2.92	4.36	4.17	−0.09
ls_other	3.67	2.27	4.13	4.11	−0.32
ls_clean	1.79	0.89	2.40	2.09	−0.33

* atis static-GER row from results.tsv (6.71%); its breadth-sweep row (6.31%) used the same commit's rerun.

The organizing result is a scaling law: ρ correlates with 1-best WER at r = +0.90 — correction pays where headroom exists and should abstain at the clean floor. With a WER-gated memory, wsj_score reaches 3.57% (ρ = 1.03) and atis 4.12% (ρ = 1.20) — past the 5-best oracle (results.tsv). Static GER over-corrects everywhere (up to 64% harmful edits on wsj); memory drops that to 35%.

Noise-robust results (robust_results.tsv, one memory per noise family): CHiME-4 real recorded noise wins 4-for-4 (e.g. dev_real 7.75→7.31 while static GER raises it to 8.13); NOIZEUS wins at low SNR (5 dB: 14.51→12.60); additive-noise-over-read-speech (VB-DEMAND, LS-FreeSound) is correctly near-null. See docs/ROBUST_VM.md.

TSV schemas. results.tsv: commit, domain, arm, wer, mer, wil, wip, cer, rho, her, n, desc. breadth_results.tsv: commit, domain, gate, arm, wer, cer, sem, rho, bytes, n. robust_results.tsv: domain, family, arm, wer, mer, cer, rho, her, n, mem_bytes, desc. robust_baseline.tsv: domain, family, arm, wer, mer, cer, rho, n.

Data provenance & attribution

This repo	Derived from / evaluated on	License / source
memories/memory_<domain>_*.md, results.tsv, breadth_results.tsv	HyPoradise v0 (Whisper large-v2 5-best), Chen et al., NeurIPS 2023	CC-BY-NC-4.0 — PeacefulData/HyPoradise-v0
memories/memory_rh_*.md, robust_results.tsv, robust_baseline.tsv	Robust HyPoradise / RobustGER, Hu et al., ICLR 2024, arXiv:2401.10446	see upstream release
memories/memory_{de,fr,ja}_tr.md	CoVoST 2 X→En speech translation n-best (via GenTranslate/HypoTranslate)	CC-BY-NC-4.0 (see upstream)
memories/memory_zh_tr.md	FLEURS zh→En (via GenTranslate/HypoTranslate)	CC-BY-4.0

No corpus audio, references, or hypotheses are redistributed in this repository; result TSVs contain aggregate metrics only. The memory files are LLM-generated content: written by MiniMax-M3 (default), Claude Opus 4.7 agent loops (*_claude*.md), or MiniMax-M2.7 (*_M27.md, intentionally empty — see memory.md).

Limitations

• Main-body results use a single frozen corrector (MiniMax-M3); the paper's appendix reproduces the effect with Qwen3-30B-A3B.
• The semantic gate runs on-device via MLX (Apple silicon). On other platforms use --gate wer, or swap in any sentence encoder (interface in semantic.py).
• Three memory files are intentionally empty (optimizer accepted no edits, or the writer model was below formation competence) — an empty memory is equivalent to static GER by construction.
• ρ depends on the n-best list (Whisper large-v2 5-best throughout); a different ASR front end changes the headroom.

Upstream citations

@inproceedings{chen2023hyporadise,
  title     = {HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models},
  author    = {Chen, Chen and Hu, Yuchen and Yang, Chao-Han Huck and Siniscalchi, Sabato Marco and Chen, Pin-Yu and Chng, Eng Siong},
  booktitle = {NeurIPS},
  year      = {2023}
}
@inproceedings{hu2024robustger,
  title     = {Large Language Models are Efficient Learners of Noise-Robust Speech Recognition},
  author    = {Hu, Yuchen and Chen, Chen and Yang, Chao-Han Huck and Li, Ruizhe and Zhang, Chao and Chen, Pin-Yu and Chng, Eng Siong},
  booktitle = {ICLR},
  year      = {2024}
}

License

CC-BY-NC-SA-4.0 for everything in this repository (memories, code, result tables), consistent with the non-commercial terms of the upstream corpora the memories were optimized on.