File size: 2,164 Bytes
0f88e30 c5586a6 0f88e30 dfa9770 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | ---
license: apache-2.0
tags:
- speech
- speech-tokenizer
- streaming
- rvq
- streamalign
library_name: pytorch
---
# StreamAlign R=32 RVQ Speech Tokenizer
Streaming speech tokenizer reproducing the StreamAlign paper recipe:
a char-level RNN-T aligner (frozen encoder) with a word-aligned acoustic
head quantized by a plain ResidualVQ (**R=32 layers, codebook 512,
dim 256**), decoding to CosyVoice3 speech tokens for streaming
reconstruction (chunk_size=4, left_context=32).
## Checkpoint
`final.pt` is the released tokenizer, taken from the end of the cosine
phase. Dict keys:
`{epoch, hubert_state_dict, optimizer_state_dict, train_loss}`.
Load `hubert_state_dict` into
`models/model_tokenizer.py::Data2VecSemanticAcousticModel`
(streamASR, branch `refactor/tokenizer-r32`) with env
`RVQ_R=32 RVQ_CODEBOOK_SIZE=512`.
## Training (3-phase, global batch 16)
| phase | trainer | LR | epochs |
|---|---|---|---|
| A continuous (RVQ_BYPASS=1) | train_tokenizer.py | 1e-4 | 15 |
| B RVQ on (subalign-init from A) | train_tokenizer.py | 1e-4 | 14 |
| C cosine finetune (from B) | train_tokenizer_cosine.py | 1e-5 -> 0 | 13 |
Data: LibriSpeech 960h + Emilia-EN 400h subset (precomputed CosyVoice3
features). Pipeline: `scripts/train_tokenizer_r32_pipeline.sh`.
## Results (LibriSpeech test-clean 2620, streaming reconstruction)
| metric | value | note |
|---|---|---|
| WER | 4.43% | whisper-large-v3 (paper: 4.41%) |
| CER | 1.92% | |
| UTMOS | 4.23 | versa pseudo_mos |
| SECS | 0.585 | versa speaker, RawNet3 |
## Dependencies at inference
Frozen char RNN-T encoder ckpt (stage 1), word/BPE streaming ASR +
tokenizer, boundary classifier, CosyVoice3-0.5B vocoder. See
`train_tokenizer_r32_pipeline.sh eval` for the exact wiring.
## Citation
Accepted to **Findings of EMNLP 2026**.
```bibtex
@inproceedings{kim2026streamalign,
title = {{StreamAlign: Streaming Text-Aligned Speech Tokenization}},
author = {Kim, Kang-wook and Park, Jinyoung and Kim, Jinsoo and
Lee, Sehun and Woo, Tony and Kim, Gunhee},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026}
}
```
|