dd3434 commited on
Commit
af7ca9e
·
verified ·
1 Parent(s): 5a511a3

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +18 -6
README.md CHANGED
@@ -1,9 +1,21 @@
1
  # Streamalign (R16)
2
 
3
- StreamAlign RVQ teacher / speech tokenizer (R16, C512). Tokenizes speech into
4
- text-aligned RVQ units (16 quantizers, codebook size 512) consumed by
5
- Streamalign-SLM (R16).
6
 
7
- - Original checkpoint: `260507_RVQ_R16_C512_nodistill_codelr10x_from_nofsq_e13/epoch_16.pt`
8
- - Paired SLM: `Streamalign-SLM-R16`
9
- - Reported metrics (paired): SALMon 69.1, StoryCloze 72.1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  # Streamalign (R16)
2
 
3
+ Complete R16 speech tokenizer stack. Paired SLM: `Streamalign-SLM-R16`
4
+ (reported metrics: SALMon 69.1, StoryCloze 72.1).
 
5
 
6
+ ## Contents & how the tokenizer uses them
7
+
8
+ | Path | Role |
9
+ |------|------|
10
+ | `rvq_teacher/epoch_16.pt` | **The acoustic tokenizer.** Contains the fine-tuned streaming encoder **and** the R16 residual-VQ quantizer (`hubert_state_dict`: 408 encoder keys + `residual_vq`). This is the checkpoint that actually produces the R16 speech units. R16-specific. |
11
+ | `alignment_model/` | Char-level streaming Conformer-Transducer (`CKPT+2026-03-15`, `alignment.yaml`). Used as the base model the tokenizer is constructed from; supplies the RNN-T predictor/joiner for char-level alignment. Its encoder is overridden at load by `rvq_teacher`. R-independent. |
12
+ | `streaming_asr/` | Word-level streaming Conformer-Transducer (`3419_fastemit`, `chunk_streaming_word_fastemit.yaml`, `tokenizer.ckpt`). Generates the chunk TextGrids (word alignment) and drives the boundary classifier. R-independent. |
13
+ | `boundary_classifier/` | Word-boundary detector for streaming chunking; consumes `streaming_asr`. R-independent. |
14
+ | `alignment.yaml` | hparams for the extractor / `alignment_model`. |
15
+
16
+ ## Pipeline
17
+ audio -> `streaming_asr` (word chunks + TextGrids) + `boundary_classifier`
18
+ -> `alignment_model` scaffold + `rvq_teacher` encoder -> R16 RVQ units.
19
+
20
+ Only `rvq_teacher/epoch_16.pt` is R16-specific; the ASR / alignment /
21
+ boundary components are shared across R8/R16/R32.