ka-fastconformer-v2: NVIDIA's Georgian FastConformer fine-tuned on 357 h

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+ka-fastconformer-v2.nemo filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,149 @@
+---
+language:
+- ka
+license: cc-by-4.0
+library_name: nemo
+pipeline_tag: automatic-speech-recognition
+base_model: nvidia/stt_ka_fastconformer_hybrid_large_pc
+datasets:
+- Peacockery/georgian-asr-corpus-v0
+- google/fleurs
+tags:
+- automatic-speech-recognition
+- speech
+- audio
+- georgian
+- FastConformer
+- Transducer
+- CTC
+- hybrid
+- NeMo
+---
+
+# ka-fastconformer-v2: Georgian speech recognition (FastConformer, 115M)
+
+A Georgian speech-to-text model. It is NVIDIA's
+[stt_ka_fastconformer_hybrid_large_pc](https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_large_pc)
+fine-tuned on 357 hours of Georgian speech, most of it unscripted speech from YouTube videos with creator-made subtitles.
+
+The base model was trained on read speech (Common Voice and FLEURS). On spontaneous speech from YouTube channels that neither
+model saw in training, this model gets 0.268 WER, against 0.456 for the base model. On read speech it is also a little
+better (FLEURS 0.115 vs 0.124).
+
+- **Size and decoders:** 115M parameters. A FastConformer encoder feeds two decoders: Transducer (the default, more
+  accurate) and CTC (faster).
+- **Output:** Georgian letters and spaces only. There is no punctuation, and numbers are written as words. The base model
+  writes punctuation; this one was trained on normalized labels.
+- **Input:** 16 kHz mono audio. It was trained on clips of 1–25 s; cut longer audio into windows of up to 30 s (see below).
+
+## Results
+
+Both models were scored on the same test clips. Every transcript was normalized the same way before scoring: lowercase,
+no punctuation, numbers as words. WER is the share of words wrong and CER the share of letters wrong; lower is better.
+
+| test set | clips | base model (zero-shot), WER / CER | this model, Transducer | this model, CTC decoder |
+|---|---|---|---|---|
+| YouTube, 7 channels never seen in training | 1,952 (7.8 h) | 0.456 / 0.225 | **0.268 / 0.119** | 0.307 / 0.121 |
+| FLEURS Georgian, test | 774 (2.3 h) | 0.124 / 0.030 | **0.115 / 0.027** | 0.141 / 0.033 |
+| Common Voice Georgian, test (see the note) | 3,000 (4.9 h) | 0.039 / 0.009 | 0.042 / 0.009 | 0.057 / 0.012 |
+
+- **YouTube:** the 95% bootstrap interval of the Transducer WER is 0.261–0.277. Per channel, this model is better than the
+  base model on all 7 channels, with WER between 0.217 and 0.346 (base: 0.280–0.579).
+- **FLEURS:** sentences that contain digits are left out, because the copy we used had their numbers spelled out wrongly.
+- **Common Voice:** don't read these numbers as a fair score for either model. The base model was trained on Common Voice 17,
+  and Common Voice reshuffles its splits between releases. Many of these test recordings were probably in the base model's
+  training data. Our fine-tuning used only Common Voice train splits.
+
+### Long recordings and speed
+
+- **Long recordings:** we took the first 10 minutes of one video from each test channel (7 videos, 67.5 min) and cut them
+  into fixed 30 s windows.
+  - Transducer: WER 0.254, CER 0.113.
+  - CTC decoder: WER 0.287, CER 0.117.
+  - Base model: WER 0.439, CER 0.218.
+- **Speed:** measured on one NVIDIA RTX 6000 Ada with batches of 16 windows.
+  - Transducer: 478× real time; a single 30 s window takes 264 ms.
+  - CTC decoder: 3,328× real time; a single window takes 40 ms.
+  - On an A100 80 GB the same network ran at 470× (Transducer) and 2,814× (CTC).
+
+## How to use
+
+```python
+# pip install "nemo_toolkit[asr]"
+import nemo.collections.asr as nemo_asr
+
+model = nemo_asr.models.ASRModel.from_pretrained("lukab/ka-fastconformer-v2")
+print(model.transcribe(["audio.wav"])[0].text)  # 16 kHz mono
+
+model.change_decoding_strategy(decoder_type="ctc")  # optional: the faster CTC decoder
+```
+
+To transcribe a long recording, cut it into windows of up to 30 s:
+
+```python
+import soundfile as sf
+
+audio, sr = sf.read("long.wav", dtype="float32")  # 16 kHz mono
+win = 30 * sr
+chunks = [audio[i:i + win] for i in range(0, len(audio), win)]
+text = " ".join(h.text for h in model.transcribe(chunks, batch_size=16))
+```
+
+Fixed cuts can split a word at a window edge. Cutting at pauses avoids that.
+
+## Training
+
+- **Starting point:** all weights of `nvidia/stt_ka_fastconformer_hybrid_large_pc`. The tokenizer (SentencePiece, 1,024
+  tokens) is unchanged.
+- **Data:** 124,861 clips, 356.8 hours.
+
+  | source | hours |
+  |---|---|
+  | YouTube: creator-subtitled videos with human speakers | 226.1 |
+  | YouTube: narration channels not verified as human | 28.8 |
+  | YouTube: AI-narrated channels (capped) | 5.0 |
+  | Common Voice Georgian (train splits) | 93.2 |
+  | FLEURS Georgian (train) | 3.7 |
+
+  - Common Voice and FLEURS come from the
+    [Peacockery/georgian-asr-corpus-v0](https://huggingface.co/datasets/Peacockery/georgian-asr-corpus-v0) collection,
+    with its own splits.
+  - The YouTube audio is not released.
+- **YouTube clips:** creator subtitles were merged into 3–20 s segments that end at a sentence end where possible.
+  - Segments were dropped if they had ambiguous numbers (times, decimals), foreign scripts or an implausible speaking rate.
+  - Videos are split by channel, so the 5 dev channels and 7 test channels never appear in training.
+- **Labels:** lowercase, no punctuation, numbers spelled out.
+- **Quality filter:** 3,262 training clips (2.5%) were dropped. In each, the transcript of Meta's untouched Omnilingual ASR
+  CTC 300M disagreed with the subtitle (CER above 0.5).
+- **Recipe (NeMo):**
+  - 3 epochs: 11,706 updates of 32 clips each.
+  - Optimizer: AdamW (betas 0.9 and 0.98, weight decay 1e-3).
+  - Learning rate: peak 2e-4, 10% warmup, cosine decay to 1e-6.
+  - Gradient clipping at 1.0 and bf16 mixed precision.
+  - Hybrid loss: Transducer + 0.3 × CTC.
+  - Augmentation: NVIDIA's SpecAugment plus speed perturbation (0.9 / 1.0 / 1.1).
+  - The full config is in `training_config.yaml`.
+- **Checkpoint:** chosen by WER on 150 YouTube dev clips from the 5 dev channels. The best was update 9,804, with dev WER
+  0.322 (0.510 before fine-tuning).
+- **Hardware:** one NVIDIA RTX 6000 Ada 48 GB, 1.8 hours.
+
+## Limitations
+
+- **Spontaneous speech is still hard.** On the YouTube test, 7.4% of the words are skipped (mostly in fast or overlapping
+  speech) and 16.7% are misheard.
+- **No punctuation or digits** in the output.
+- **Offline, not streaming:** the model needs the whole window before it writes text. Live captions need chunked decoding or
+  a streaming model.
+- **Narrow domain:** the YouTube data covers a limited set of channels and topics. Other audio, such as phone calls or
+  noisy far-field recordings, is untested.
+
+## License and credits
+
+- **License:** CC-BY-4.0, the license of the base model.
+- **Base model:** NVIDIA's
+  [stt_ka_fastconformer_hybrid_large_pc](https://huggingface.co/nvidia/stt_ka_fastconformer_hybrid_large_pc), trained on
+  about 163 h of Common Voice 17 and FLEURS.
+- **Toolkit:** [NVIDIA NeMo](https://github.com/NVIDIA/NeMo).
+- **FastConformer:** Rekesh et al., 2023, [arXiv:2305.05084](https://arxiv.org/abs/2305.05084).
+- **Training data:** Common Voice (CC0) and FLEURS (CC-BY-4.0) via the Peacockery collection (CC-BY-4.0), and Georgian
+  YouTube videos with creator-made subtitles, used for research. The audio is not redistributed.

ka-fastconformer-v2.nemo

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93c4b07f5a636f2547bea24d1418b9c83574f079e698ada6f14abe98b9991251
+size 459304960

training_config.yaml

@@ -0,0 +1,53 @@
+# Training config of ka-fastconformer-v2, as recorded by the training script (merged with the command-line overrides).
+# ctc_loss_weight: null keeps the base model's value, 0.3. Paths refer to the training machine.
+model_name_or_path: nvidia/stt_ka_fastconformer_hybrid_large_pc
+att_context_size: null
+spec_augment: true
+ctc_loss_weight: null
+data_dir: /data/prepared/v1
+data_version: prepared/v1
+train_sources:
+- youtube
+- youtube_unverified
+- youtube_ai
+- commonvoice
+- fleurs
+min_seconds: 1.0
+max_seconds: 25.0
+keep_ids: /data/scores_train.jsonl
+eval_rows_per_source: 150
+test_rows_per_source: 3000
+speed_perturb: true
+group: n2-fastconformer
+tags:
+- fastconformer
+- nvidia
+- prepared-v1
+- full-data
+- speed-perturb
+notes: NVIDIA's Georgian FastConformer (offline hybrid) fine-tuned on all of prepared/v1,
+  3 epochs, speed perturbation.
+log_model: true
+run_name: n2-fastconformer-ka-full
+output_dir: runs/n2-fastconformer-ka-full
+manifest_dir: /data/nemo
+num_train_epochs: 3
+per_device_train_batch_size: 16
+per_device_eval_batch_size: 32
+gradient_accumulation_steps: 2
+learning_rate: 0.0002
+min_learning_rate: 1.0e-06
+warmup_ratio: 0.1
+weight_decay: 0.001
+adam_beta1: 0.9
+adam_beta2: 0.98
+max_grad_norm: 1.0
+precision: bf16-mixed
+accelerator: gpu
+eval_steps: 4000
+eval_on_start: true
+logging_steps: 10
+dataloader_num_workers: 16
+seed: 42
+report_to:
+- wandb