| # Streamalign (R16) |
|
|
| Complete R16 speech tokenizer stack. Paired SLM: `Streamalign-SLM-R16` |
| (reported metrics: SALMon 69.1, StoryCloze 72.1). |
|
|
| ## Contents & roles |
|
|
| | Path | Role | |
| |------|------| |
| | `rvq_teacher/` | The acoustic tokenizer: the streaming encoder plus the R16 residual-VQ quantizer. Produces the R16 speech units. R16-specific. | |
| | `alignment_model/` | Char-level streaming Conformer-Transducer. Base scaffold the tokenizer is built on; supplies the RNN-T predictor/joiner for char-level alignment. R-independent. | |
| | `streaming_asr/` | Word-level streaming Conformer-Transducer. Generates the chunk TextGrids (word alignment) and drives the boundary classifier. R-independent. | |
| | `boundary_classifier/` | Word-boundary detector for streaming chunking; consumes `streaming_asr`. R-independent. | |
| | `alignment.yaml` | Extractor / `alignment_model` hparams. | |
|
|
| ## Pipeline |
| audio -> `streaming_asr` (word chunks + TextGrids) + `boundary_classifier` |
| -> `alignment_model` scaffold + `rvq_teacher` encoder -> R16 RVQ units. |
| |
| Only `rvq_teacher` is R16-specific; the ASR / alignment / boundary |
| components are shared across R8/R16/R32. |
|
|
| ## Tokenizer checkpoint |
|
|
| `rvq_teacher/epoch_22.pt` is the final R16 tokenizer, trained on LibriSpeech |
| plus the full Emilia set with a learnable codebook at a 10x codebook learning |
| rate (`streamalign_r16`). |
| It replaces the earlier LibriSpeech-only `epoch_16.pt`. |
|
|
| Load it with `RVQ_R=16` and `RVQ_CODEBOOK_SIZE=512`, and leave |
| `RVQ_CODEBOOK_DIM` unset: `codebook_dim` defaults to `feat_dim=256`, which is |
| what these weights expect. |
|
|
