| # Streamalign (R16) |
|
|
| Complete R16 speech tokenizer stack. Paired SLM: `Streamalign-SLM-R16` |
| (reported metrics: SALMon 69.1, StoryCloze 72.1). |
|
|
| ## Contents & roles |
|
|
| | Path | Role | |
| |------|------| |
| | `rvq_teacher/` | The acoustic tokenizer: the streaming encoder plus the R16 residual-VQ quantizer. Produces the R16 speech units. R16-specific. | |
| | `alignment_model/` | Char-level streaming Conformer-Transducer. Base scaffold the tokenizer is built on; supplies the RNN-T predictor/joiner for char-level alignment. R-independent. | |
| | `streaming_asr/` | Word-level streaming Conformer-Transducer. Generates the chunk TextGrids (word alignment) and drives the boundary classifier. R-independent. | |
| | `boundary_classifier/` | Word-boundary detector for streaming chunking; consumes `streaming_asr`. R-independent. | |
| | `alignment.yaml` | Extractor / `alignment_model` hparams. | |
|
|
| ## Pipeline |
| audio -> `streaming_asr` (word chunks + TextGrids) + `boundary_classifier` |
| -> `alignment_model` scaffold + `rvq_teacher` encoder -> R16 RVQ units. |
| |
| Only `rvq_teacher` is R16-specific; the ASR / alignment / boundary |
| components are shared across R8/R16/R32. |
|
|
| ## Tokenizer checkpoint |
|
|
| The checkpoint under `rvq_teacher/` is the final R16 tokenizer, trained on |
| LibriSpeech plus the full Emilia set. |
|
|
| Load it with `RVQ_R=16` and `RVQ_CODEBOOK_SIZE=512`, and leave |
| `RVQ_CODEBOOK_DIM` unset: `codebook_dim` defaults to `feat_dim=256`, which is |
| what these weights expect. |
|
|
| ## Citation |
|
|
| Accepted to **Findings of EMNLP 2026**. |
|
|
| ```bibtex |
| @inproceedings{kim2026streamalign, |
| title = {{StreamAlign: Streaming Text-Aligned Speech Tokenization}}, |
| author = {Kim, Kang-wook and Park, Jinyoung and Kim, Jinsoo and |
| Lee, Sehun and Woo, Tony and Kim, Gunhee}, |
| booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026}, |
| year = {2026} |
| } |
| ``` |
|
|
