| --- |
| language: |
| - vi |
| license: apache-2.0 |
| tags: |
| - automatic-speech-recognition |
| - vietnamese |
| - zipformer |
| - transducer |
| - rnnt |
| - icefall |
| - meeting |
| library_name: icefall |
| base_model: actableai/zipformer-rnnt-v2 |
| datasets: |
| - actableai/vi-meeting-soniox-wer16 |
| metrics: |
| - wer |
| pipeline_tag: automatic-speech-recognition |
| --- |
| |
| # zipformer-rnnt-v3 |
|
|
| Vietnamese **Zipformer2 transducer (RNNT)** fine-tuned for **Actable meeting audio**, starting from |
| [`actableai/zipformer-rnnt-v2`](https://huggingface.co/actableai/zipformer-rnnt-v2) and trained on |
| [`actableai/vi-meeting-soniox-wer16`](https://huggingface.co/datasets/actableai/vi-meeting-soniox-wer16) |
| with **Soniox transcripts as golden labels**. |
|
|
| | | | |
| |---|---| |
| | **Base** | `actableai/zipformer-rnnt-v2` (policy-domain Zipformer, BPE-3000) | |
| | **Fine-tune data** | Meeting segments with WER(Soniox, baseline Qwen) ≤ 16% | |
| | **Labels** | Soniox `text` only (mixed-case; not uppercased) | |
| | **Params** | ~70.7 M (transducer; no CTC head in this fine-tune) | |
| | **Sample rate** | 16 kHz, 80-dim log-Mel fbank (on-the-fly) | |
|
|
| --- |
|
|
| ## Performance |
|
|
| ### Meeting dev (Soniox references) |
|
|
| Eval set: `session_soniox_dev.jsonl` → 5,453 utterances, ~13.8 h |
| Decoding: **greedy search**, icefall `greedy_search_batch` |
| Text norm for WER: lowercase, strip punctuation / bracket tags, collapse whitespace |
| Checkpoint: `best-valid-loss.pt` using **`model_avg`** weights |
| |
| | Model | Meeting-dev WER | Errors / words | |
| |-------|----------------:|---------------:| |
| | **v2 baseline** (`actableai/zipformer-rnnt-v2`) | **31.07%** | 56,369 / 181,450 | |
| | **v3 (this model)** | **20.71%** | 37,580 / 181,450 | |
| | **Δ absolute** | **−10.36** | — | |
| | **Δ relative** | **−33.3%** | — | |
| |
| > Measured 2026-07-27 with `eval_zipformer_meeting_wer16.py` (full meeting dev). |
| |
| ### Validation RNNT loss (during fine-tune) |
| |
| Loss on meeting dev CutSet at the start of each epoch (icefall validation): |
| |
| | Epoch | Valid loss | Simple | Pruned | |
| |------:|-----------:|-------:|-------:| |
| | 1 | 0.4787 | 0.4529 | 0.2577 | |
| | 2 | 0.3846 | 0.3782 | 0.1805 | |
| | 3 | 0.3605 | 0.3684 | 0.1725 | |
| | 4 | 0.3478 | 0.3633 | 0.1662 | |
| | 5 | 0.3433 | 0.3604 | 0.1631 | |
| | 6 | 0.3416 | 0.3589 | 0.1622 | |
| | 7 | 0.3398 | 0.3578 | 0.1609 | |
| | 8 | 0.3385 | 0.3565 | 0.1603 | |
| | 9 | 0.3369 | 0.3558 | 0.1590 | |
| | **10 (best)** | **0.3362** | 0.3552 | 0.1586 | |
|
|
| - Best valid loss: **0.3362** @ epoch 10 |
| - Best train loss: **0.2452** @ epoch 10 |
| - Wall time: ~1 h 38 m (single NVIDIA A100 40GB) |
|
|
| --- |
|
|
| ## Training details |
|
|
| ### Data |
|
|
| | Split | Segments | Hours | Source | |
| |-------|--------:|------:|--------| |
| | train | 22,693 | ~68.1 | `actableai/vi-meeting-soniox-wer16` (= local `session_soniox_train_filtered.jsonl`) | |
| | dev | 5,453 | ~13.8 | `session_soniox_dev.jsonl` (Soniox labels) | |
|
|
| - **Filter:** keep train segments where WER(Soniox ref, baseline Qwen hyp) ≤ **16%** |
| - **Labels:** Soniox field `text` only — never Qwen hypotheses |
| - **Audio:** 16 kHz mono session WAVs (`full_session.wav` + start/end offsets) |
| - **Mix / replay:** none (`use_mux=0`) — pure meeting wer16 |
| - **Utt duration filter:** 0.3–30 s |
|
|
| ### Optimization |
|
|
| | Knob | Value | |
| |------|--------| |
| | Recipe | icefall Zipformer2 `finetune.py` | |
| | Init | load `encoder,encoder_embed,decoder,joiner,simple_am_proj,simple_lm_proj` from v2 | |
| | CTC | **disabled** (`use_ctc=0`) — joint CTC FT was unstable (CTC bias grad explosion under fp16) | |
| | Optimizer | ScaledAdam + Eden LR schedule | |
| | `base_lr` | 5e-4 | |
| | Epochs | 10 | |
| | Batching | `max_duration=300` s / batch, dynamic bucketing | |
| | Precision | fp16 | |
| | Features | on-the-fly 80-dim Kaldi fbank | |
| | SpecAugment | on (time-warp factor 80) | |
| | MUSAN | off | |
| | Speed perturb | off | |
| | Seed | 42 | |
| | Hardware | 1× A100-SXM4-40GB | |
| | Global steps | ~8,674 | |
|
|
| ### Architecture (unchanged from v2) |
|
|
| | Param | Value | |
| |-------|--------| |
| | `num_encoder_layers` | 2,2,3,4,3,2 | |
| | `encoder_dim` | 192,256,384,512,384,256 | |
| | `downsampling_factor` | 1,2,4,8,4,2 | |
| | `feedforward_dim` | 512,768,1024,1536,1024,768 | |
| | `num_heads` | 4,4,4,8,4,4 | |
| | `cnn_module_kernel` | 31,31,15,15,15,31 | |
| | `encoder_unmasked_dim` | 192,192,256,256,256,192 | |
| | `decoder_dim` / `joiner_dim` | 512 / 512 | |
| | Vocab | BPE-3000 (mixed-case Vietnamese; same as v2) | |
|
|
| --- |
|
|
| ## Files |
|
|
| | File | Description | |
| |------|-------------| |
| | `best-valid-loss.pt` | Best icefall checkpoint (`model` + `model_avg` + train metadata) | |
| | `bpe.model` | SentencePiece BPE-3000 (shared with v2) | |
| | `tokens.txt` | Token id map for sherpa / icefall | |
| | `config.json` | Architecture + training metadata | |
| | `train_zipformer_meeting_wer16.sh` | Launch script used for this run | |
| | `prep_lhotse_meeting_wer16.py` | Lhotse CutSet prep (Soniox labels) | |
| | `eval_zipformer_meeting_wer16.py` | Greedy WER eval helper | |
|
|
| --- |
|
|
| ## Usage (icefall) |
|
|
| ```python |
| import torch, sys |
| sys.path.insert(0, "/path/to/icefall") |
| sys.path.insert(0, "/path/to/zipformer_work") # finetune.py / model defs |
| |
| from finetune import get_model, get_params, add_model_arguments |
| from beam_search import greedy_search_batch |
| import argparse, sentencepiece as spm |
| |
| parser = argparse.ArgumentParser() |
| add_model_arguments(parser) |
| params = get_params() |
| params.update(vars(parser.parse_args([]))) |
| params.encoder_dim = "192,256,384,512,384,256" |
| params.num_encoder_layers = "2,2,3,4,3,2" |
| params.downsampling_factor = "1,2,4,8,4,2" |
| params.feedforward_dim = "512,768,1024,1536,1024,768" |
| params.num_heads = "4,4,4,8,4,4" |
| params.cnn_module_kernel = "31,31,15,15,15,31" |
| params.encoder_unmasked_dim = "192,192,256,256,256,192" |
| params.decoder_dim = 512 |
| params.joiner_dim = 512 |
| params.causal = False |
| params.vocab_size = 3000 |
| params.blank_id = 0 |
| params.context_size = 2 |
| params.use_transducer = True |
| params.use_ctc = False |
| |
| model = get_model(params) |
| ckpt = torch.load("best-valid-loss.pt", map_location="cpu") |
| state = ckpt.get("model_avg") or ckpt["model"] |
| model.load_state_dict(state, strict=False) |
| model.eval().cuda() |
| |
| sp = spm.SentencePieceProcessor() |
| sp.load("bpe.model") |
| |
| # feature: (1, T, 80) float32 fbank; feature_lens: (1,) int |
| with torch.no_grad(): |
| encoder_out, encoder_out_lens = model.forward_encoder(feature, feature_lens) |
| token_ids = greedy_search_batch(model, encoder_out, encoder_out_lens) |
| text = sp.decode(token_ids[0]) |
| print(text) |
| ``` |
|
|
| --- |
|
|
| ## Notes / limitations |
|
|
| - Optimized for **Vietnamese meeting / conversational** audio; not re-evaluated on open-domain policy or YouTube suites in this card. |
| - Greedy decoding only in the reported WER; beam search may improve further. |
| - Transcripts are **Soniox** pseudo-labels (filtered), not human gold. |
| - CTC head from v2 was **not** fine-tuned / not present in the shipped transducer-only graph. |
|
|
| ## Changelog |
|
|
| | Date | Event | |
| |------|--------| |
| | 2026-07-27 | Fine-tune v2 → meeting wer16, 10 epochs; valid loss 0.479 → 0.336 | |
| | 2026-07-27 | Meeting-dev WER 31.07% (v2) → **20.71%** (v3); push `actableai/zipformer-rnnt-v3` | |
|
|
