File size: 6,927 Bytes

2badf20

---
language:
- vi
license: apache-2.0
tags:
- automatic-speech-recognition
- vietnamese
- zipformer
- transducer
- rnnt
- icefall
- meeting
library_name: icefall
base_model: actableai/zipformer-rnnt-v2
datasets:
- actableai/vi-meeting-soniox-wer16
metrics:
- wer
pipeline_tag: automatic-speech-recognition
---

# zipformer-rnnt-v3

Vietnamese **Zipformer2 transducer (RNNT)** fine-tuned for **Actable meeting audio**, starting from
[`actableai/zipformer-rnnt-v2`](https://huggingface.co/actableai/zipformer-rnnt-v2) and trained on
[`actableai/vi-meeting-soniox-wer16`](https://huggingface.co/datasets/actableai/vi-meeting-soniox-wer16)
with **Soniox transcripts as golden labels**.

| | |
|---|---|
| **Base** | `actableai/zipformer-rnnt-v2` (policy-domain Zipformer, BPE-3000) |
| **Fine-tune data** | Meeting segments with WER(Soniox, baseline Qwen) ≤ 16% |
| **Labels** | Soniox `text` only (mixed-case; not uppercased) |
| **Params** | ~70.7 M (transducer; no CTC head in this fine-tune) |
| **Sample rate** | 16 kHz, 80-dim log-Mel fbank (on-the-fly) |

---

## Performance

### Meeting dev (Soniox references)

Eval set: `session_soniox_dev.jsonl` → 5,453 utterances, ~13.8 h  
Decoding: **greedy search**, icefall `greedy_search_batch`  
Text norm for WER: lowercase, strip punctuation / bracket tags, collapse whitespace  
Checkpoint: `best-valid-loss.pt` using **`model_avg`** weights

| Model | Meeting-dev WER | Errors / words |
|-------|----------------:|---------------:|
| **v2 baseline** (`actableai/zipformer-rnnt-v2`) | **31.07%** | 56,369 / 181,450 |
| **v3 (this model)** | **20.71%** | 37,580 / 181,450 |
| **Δ absolute** | **−10.36** | — |
| **Δ relative** | **−33.3%** | — |

> Measured 2026-07-27 with `eval_zipformer_meeting_wer16.py` (full meeting dev).

### Validation RNNT loss (during fine-tune)

Loss on meeting dev CutSet at the start of each epoch (icefall validation):

| Epoch | Valid loss | Simple | Pruned |
|------:|-----------:|-------:|-------:|
| 1 | 0.4787 | 0.4529 | 0.2577 |
| 2 | 0.3846 | 0.3782 | 0.1805 |
| 3 | 0.3605 | 0.3684 | 0.1725 |
| 4 | 0.3478 | 0.3633 | 0.1662 |
| 5 | 0.3433 | 0.3604 | 0.1631 |
| 6 | 0.3416 | 0.3589 | 0.1622 |
| 7 | 0.3398 | 0.3578 | 0.1609 |
| 8 | 0.3385 | 0.3565 | 0.1603 |
| 9 | 0.3369 | 0.3558 | 0.1590 |
| **10 (best)** | **0.3362** | 0.3552 | 0.1586 |

- Best valid loss: **0.3362** @ epoch 10  
- Best train loss: **0.2452** @ epoch 10  
- Wall time: ~1 h 38 m (single NVIDIA A100 40GB)

---

## Training details

### Data

| Split | Segments | Hours | Source |
|-------|--------:|------:|--------|
| train | 22,693 | ~68.1 | `actableai/vi-meeting-soniox-wer16` (= local `session_soniox_train_filtered.jsonl`) |
| dev | 5,453 | ~13.8 | `session_soniox_dev.jsonl` (Soniox labels) |

- **Filter:** keep train segments where WER(Soniox ref, baseline Qwen hyp) ≤ **16%**  
- **Labels:** Soniox field `text` only — never Qwen hypotheses  
- **Audio:** 16 kHz mono session WAVs (`full_session.wav` + start/end offsets)  
- **Mix / replay:** none (`use_mux=0`) — pure meeting wer16  
- **Utt duration filter:** 0.3–30 s  

### Optimization

| Knob | Value |
|------|--------|
| Recipe | icefall Zipformer2 `finetune.py` |
| Init | load `encoder,encoder_embed,decoder,joiner,simple_am_proj,simple_lm_proj` from v2 |
| CTC | **disabled** (`use_ctc=0`) — joint CTC FT was unstable (CTC bias grad explosion under fp16) |
| Optimizer | ScaledAdam + Eden LR schedule |
| `base_lr` | 5e-4 |
| Epochs | 10 |
| Batching | `max_duration=300` s / batch, dynamic bucketing |
| Precision | fp16 |
| Features | on-the-fly 80-dim Kaldi fbank |
| SpecAugment | on (time-warp factor 80) |
| MUSAN | off |
| Speed perturb | off |
| Seed | 42 |
| Hardware | 1× A100-SXM4-40GB |
| Global steps | ~8,674 |

### Architecture (unchanged from v2)

| Param | Value |
|-------|--------|
| `num_encoder_layers` | 2,2,3,4,3,2 |
| `encoder_dim` | 192,256,384,512,384,256 |
| `downsampling_factor` | 1,2,4,8,4,2 |
| `feedforward_dim` | 512,768,1024,1536,1024,768 |
| `num_heads` | 4,4,4,8,4,4 |
| `cnn_module_kernel` | 31,31,15,15,15,31 |
| `encoder_unmasked_dim` | 192,192,256,256,256,192 |
| `decoder_dim` / `joiner_dim` | 512 / 512 |
| Vocab | BPE-3000 (mixed-case Vietnamese; same as v2) |

---

## Files

| File | Description |
|------|-------------|
| `best-valid-loss.pt` | Best icefall checkpoint (`model` + `model_avg` + train metadata) |
| `bpe.model` | SentencePiece BPE-3000 (shared with v2) |
| `tokens.txt` | Token id map for sherpa / icefall |
| `config.json` | Architecture + training metadata |
| `train_zipformer_meeting_wer16.sh` | Launch script used for this run |
| `prep_lhotse_meeting_wer16.py` | Lhotse CutSet prep (Soniox labels) |
| `eval_zipformer_meeting_wer16.py` | Greedy WER eval helper |

---

## Usage (icefall)

```python
import torch, sys
sys.path.insert(0, "/path/to/icefall")
sys.path.insert(0, "/path/to/zipformer_work")  # finetune.py / model defs

from finetune import get_model, get_params, add_model_arguments
from beam_search import greedy_search_batch
import argparse, sentencepiece as spm

parser = argparse.ArgumentParser()
add_model_arguments(parser)
params = get_params()
params.update(vars(parser.parse_args([])))
params.encoder_dim = "192,256,384,512,384,256"
params.num_encoder_layers = "2,2,3,4,3,2"
params.downsampling_factor = "1,2,4,8,4,2"
params.feedforward_dim = "512,768,1024,1536,1024,768"
params.num_heads = "4,4,4,8,4,4"
params.cnn_module_kernel = "31,31,15,15,15,31"
params.encoder_unmasked_dim = "192,192,256,256,256,192"
params.decoder_dim = 512
params.joiner_dim = 512
params.causal = False
params.vocab_size = 3000
params.blank_id = 0
params.context_size = 2
params.use_transducer = True
params.use_ctc = False

model = get_model(params)
ckpt = torch.load("best-valid-loss.pt", map_location="cpu")
state = ckpt.get("model_avg") or ckpt["model"]
model.load_state_dict(state, strict=False)
model.eval().cuda()

sp = spm.SentencePieceProcessor()
sp.load("bpe.model")

# feature: (1, T, 80) float32 fbank; feature_lens: (1,) int
with torch.no_grad():
    encoder_out, encoder_out_lens = model.forward_encoder(feature, feature_lens)
    token_ids = greedy_search_batch(model, encoder_out, encoder_out_lens)
text = sp.decode(token_ids[0])
print(text)
```

---

## Notes / limitations

- Optimized for **Vietnamese meeting / conversational** audio; not re-evaluated on open-domain policy or YouTube suites in this card.
- Greedy decoding only in the reported WER; beam search may improve further.
- Transcripts are **Soniox** pseudo-labels (filtered), not human gold.
- CTC head from v2 was **not** fine-tuned / not present in the shipped transducer-only graph.

## Changelog

| Date | Event |
|------|--------|
| 2026-07-27 | Fine-tune v2 → meeting wer16, 10 epochs; valid loss 0.479 → 0.336 |
| 2026-07-27 | Meeting-dev WER 31.07% (v2) → **20.71%** (v3); push `actableai/zipformer-rnnt-v3` |