File size: 1,580 Bytes
b345bef af7ca9e b345bef f326201 af7ca9e f326201 af7ca9e f326201 af7ca9e f326201 a05bc57 24f6fb4 a05bc57 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | # Streamalign (R16)
Complete R16 speech tokenizer stack. Paired SLM: `Streamalign-SLM-R16`
(reported metrics: SALMon 69.1, StoryCloze 72.1).
## Contents & roles
| Path | Role |
|------|------|
| `rvq_teacher/` | The acoustic tokenizer: the streaming encoder plus the R16 residual-VQ quantizer. Produces the R16 speech units. R16-specific. |
| `alignment_model/` | Char-level streaming Conformer-Transducer. Base scaffold the tokenizer is built on; supplies the RNN-T predictor/joiner for char-level alignment. R-independent. |
| `streaming_asr/` | Word-level streaming Conformer-Transducer. Generates the chunk TextGrids (word alignment) and drives the boundary classifier. R-independent. |
| `boundary_classifier/` | Word-boundary detector for streaming chunking; consumes `streaming_asr`. R-independent. |
| `alignment.yaml` | Extractor / `alignment_model` hparams. |
## Pipeline
audio -> `streaming_asr` (word chunks + TextGrids) + `boundary_classifier`
-> `alignment_model` scaffold + `rvq_teacher` encoder -> R16 RVQ units.
Only `rvq_teacher` is R16-specific; the ASR / alignment / boundary
components are shared across R8/R16/R32.
## Tokenizer checkpoint
`rvq_teacher/epoch_22.pt` is the final R16 tokenizer, trained on LibriSpeech
plus the full Emilia set with a learnable codebook at a 10x codebook learning
rate (`streamalign_r16`).
It replaces the earlier LibriSpeech-only `epoch_16.pt`.
Load it with `RVQ_R=16` and `RVQ_CODEBOOK_SIZE=512`, and leave
`RVQ_CODEBOOK_DIM` unset: `codebook_dim` defaults to `feat_dim=256`, which is
what these weights expect.
|