---
license: apache-2.0
tags:
  - speech
  - speech-tokenizer
  - streaming
  - rvq
  - streamalign
library_name: pytorch
---

# StreamAlign R=32 RVQ Speech Tokenizer

Streaming speech tokenizer reproducing the StreamAlign paper recipe:
a char-level RNN-T aligner (frozen encoder) with a word-aligned acoustic
head quantized by a plain ResidualVQ (**R=32 layers, codebook 512,
dim 256**), decoding to CosyVoice3 speech tokens for streaming
reconstruction (chunk_size=4, left_context=32).

## Checkpoint

`final.pt` is the released tokenizer, taken from the end of the cosine
phase. Dict keys:
`{epoch, hubert_state_dict, optimizer_state_dict, train_loss}`.
Load `hubert_state_dict` into
`models/model_tokenizer.py::Data2VecSemanticAcousticModel`
(streamASR, branch `refactor/tokenizer-r32`) with env
`RVQ_R=32 RVQ_CODEBOOK_SIZE=512`.

## Training (3-phase, global batch 16)

| phase | trainer | LR | epochs |
|---|---|---|---|
| A continuous (RVQ_BYPASS=1) | train_tokenizer.py | 1e-4 | 15 |
| B RVQ on (subalign-init from A) | train_tokenizer.py | 1e-4 | 14 |
| C cosine finetune (from B) | train_tokenizer_cosine.py | 1e-5 -> 0 | 13 |

Data: LibriSpeech 960h + Emilia-EN 400h subset (precomputed CosyVoice3
features). Pipeline: `scripts/train_tokenizer_r32_pipeline.sh`.

## Results (LibriSpeech test-clean 2620, streaming reconstruction)

| metric | value | note |
|---|---|---|
| WER | 4.43% | whisper-large-v3 (paper: 4.41%) |
| CER | 1.92% | |
| UTMOS | 4.23 | versa pseudo_mos |
| SECS | 0.585 | versa speaker, RawNet3 |

## Dependencies at inference

Frozen char RNN-T encoder ckpt (stage 1), word/BPE streaming ASR +
tokenizer, boundary classifier, CosyVoice3-0.5B vocoder. See
`train_tokenizer_r32_pipeline.sh eval` for the exact wiring.

## Citation

Accepted to **Findings of EMNLP 2026**.

```bibtex
@inproceedings{kim2026streamalign,
  title     = {{StreamAlign: Streaming Text-Aligned Speech Tokenization}},
  author    = {Kim, Kang-wook and Park, Jinyoung and Kim, Jinsoo and
               Lee, Sehun and Woo, Tony and Kim, Gunhee},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026}
}
```