File size: 6,534 Bytes
25aa57f 2dec448 77d85f9 2dec448 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 |
---
license: cc-by-nc-4.0
---
# 🎯 CTC Forced Aligner
We are open-sourcing the CTC forced aligner used in [Deskpai](https://www.deskpai.com).
With focus on production-ready model inference, it supports 18 different alignment models, including multilingual models(German, English, Spanish, French and Italian etc), and provides SRT and WebVTT alignment and generation out of box. It supports both ONNXRuntime and PyTorch for model serving.
[Home Page](https://github.com/deskpai/ctc_forced_aligner)
## 🚀 Installation
- CPU inference via ONNXRuntime
```bash
pip install ctc_forced_aligner
```
- GPU inference via ONNXRuntime
```bash
pip install ctc_forced_aligner[gpu]
```
- CPU/GPU inference via PyTorch
```bash
pip install ctc_forced_aligner[torch]
```
- Install all dependencies
```bash
pip install ctc_forced_aligner[all]
```
## 📝 Sample Inference Code
- CPU/GPU inference via ONNXRuntime
```python
from ctc_forced_aligner import AlignmentSingleton
alignment_service = AlignmentSingleton()
input_audio_path = "audio.mp3"
input_text_path = "input.txt"
output_srt_path = "output.srt"
ret = alignment_service.generate_srt(input_audio_path,
input_text_path,
output_srt_path)
if ret:
print(f"Aligned SRT is generated at {output_srt_path}")
output_vtt_path = "output.vtt"
ret = alignment_service.generate_webvtt(input_audio_path,
input_text_path,
output_vtt_path)
if ret:
print(f"aligned VTT is generated to {output_vtt_path}")
```
- CPU/GPU inference via PyTorch
```python
from ctc_forced_aligner import AlignmentTorch
at = AlignmentTorch()
ret = at.generate_srt(input_audio_path, input_text_path, output_srt_path)
if ret:
print(f"aligned srt is generated to {output_srt_path}")
ret = at.generate_webvtt(input_audio_path, input_text_path, output_vtt_path)
if ret:
print(f"aligned VTT is generated to {output_vtt_path}")
```
- Inference with multiple models
```python
from ctc_forced_aligner import AlignmentTorch
at = AlignmentTorch()
ret = at.generate_srt(input_audio_path, input_text_path, output_srt_path, model_type='WAV2VEC2_ASR_BASE_960H')
if ret:
print(f"aligned srt is generated to {output_srt_path}")
ret = at.generate_webvtt(input_audio_path, input_text_path, output_vtt_path, model_type='WAV2VEC2_ASR_BASE_960H')
if ret:
print(f"aligned VTT is generated to {output_vtt_path}")
```
## Models Supported
### ✅ Wav2Vec2 Models
These are fine-tuned models with a **CTC-based ASR head**:
- `WAV2VEC2_ASR_BASE_960H`
- `WAV2VEC2_ASR_BASE_100H`
- `WAV2VEC2_ASR_BASE_10M`
- `WAV2VEC2_ASR_LARGE_10M`
- `WAV2VEC2_ASR_LARGE_100H`
- `WAV2VEC2_ASR_LARGE_960H`
- `WAV2VEC2_ASR_LARGE_LV60K_10M`
- `WAV2VEC2_ASR_LARGE_LV60K_100H`
- `WAV2VEC2_ASR_LARGE_LV60K_960H`
### ✅ VoxPopuli Models (Multilingual)
These models are fine-tuned for **specific languages**:
- `VOXPOPULI_ASR_BASE_10K_DE` (German ASR)
- `VOXPOPULI_ASR_BASE_10K_EN` (English ASR)
- `VOXPOPULI_ASR_BASE_10K_ES` (Spanish ASR)
- `VOXPOPULI_ASR_BASE_10K_FR` (French ASR)
- `VOXPOPULI_ASR_BASE_10K_IT` (Italian ASR)
- Fine-tuned on **VoxPopuli** speech corpus.
### ✅ HuBERT Models
- `HUBERT_ASR_LARGE`
- `HUBERT_ASR_XLARGE`
## 💡 Which One and How to Use?
**For PyTorch serving**, use `AlignmentTorch` or `AlignmentTorchSingleton`.
- **For English ASR** → `WAV2VEC2_ASR_LARGE_960H` or `HUBERT_ASR_LARGE`
- **For multilingual ASR** → `VOXPOPULI_ASR_BASE_10K_*`
- **For low-resource ASR** → `WAV2VEC2_ASR_BASE_10M` (smallest model)
- **For best accuracy** → `WAV2VEC2_ASR_LARGE_LV60K_960H` or `HUBERT_ASR_XLARGE`
**For ONNXRuntime serving** with minimum dependencies, use `Alignment` or `AlignmentSingleton`.
Please contact [us](mailto:dev@deskpai.com) if you want to integrate your model into this package.
## 📄 License
### Code
- This project includes code from [pytorch/audio](https://github.com/pytorch/audio), licensed under the `BSD-2-Clause` license.
- This project includes code from [MahmoudAshraf97/ctc-forced-aligner](https://github.com/MahmoudAshraf97/ctc-forced-aligner), licensed under the `BSD` license.`This project is licensed under the BSD License, note that the default model has CC-BY-NC 4.0 License, so make sure to use a different model for commercial usage.`
- Modifications and additional code are contributed by [Deskpai.com](https://www.deskpai.com) and licensed under the [DOSL-1.0 license](https://github.com/deskpai/deskpai/blob/main/LICENSE).
### Model
- The following models are developed by Meta AI (formerly Facebook AI) under `MIT License` and redistributed with the same license.
- `WAV2VEC2_ASR_BASE_960H`
- `WAV2VEC2_ASR_BASE_100H`
- `WAV2VEC2_ASR_BASE_10M`
- `WAV2VEC2_ASR_LARGE_10M`
- `WAV2VEC2_ASR_LARGE_100H`
- `WAV2VEC2_ASR_LARGE_960H`
- `WAV2VEC2_ASR_LARGE_LV60K_10M`
- `WAV2VEC2_ASR_LARGE_LV60K_100H`
- `WAV2VEC2_ASR_LARGE_LV60K_960H`
- VoxPopuli and HuBERT models are also developed by Meta AI and are generally released under the MIT License. The specific licensing for these models can be found in their respective repositories or documentation. Please check it on your own.
- `VOXPOPULI_ASR_BASE_10K_DE`
- `VOXPOPULI_ASR_BASE_10K_EN`
- `VOXPOPULI_ASR_BASE_10K_ES`
- `VOXPOPULI_ASR_BASE_10K_FR`
- `VOXPOPULI_ASR_BASE_10K_IT`
- `HUBERT_ASR_LARGE`
- `HUBERT_ASR_XLARGE`
- The model `MMS_FA` is published by the authors of Scaling Speech Technology to 1,000+ Languages Pratap et al., 2023 under `CC-BY-NC 4.0 License`.
- The onnx model weights are created by [Deskpai.com](https://www.deskpai.com) based on the model of [MahmoudAshraf/mms-300m-1130-forced-aligner](https://huggingface.co/MahmoudAshraf/mms-300m-1130-forced-aligner) and under `CC-BY-NC 4.0 License`.
📝 Note: It's essential to verify the licensing terms from the official repositories or documentation before using these models.
## 🙏 Reference
- [LESS PEAKY AND MORE ACCURATE CTC FORCED ALIGNMENT BY LABEL PRIORS](https://arxiv.org/pdf/2406.02560)
- [Montreal Forced Aligner User Guide](https://montreal-forced-aligner.readthedocs.io/en/stable/user_guide/index.html)
- [Forced Alignment with Wav2Vec2](https://pytorch.org/audio/main/tutorials/forced_alignment_tutorial.html)
- [NeuFA: Neural Network Based End-to-End Forced Aligner](https://arxiv.org/abs/2203.16838)
- [Tradition or Innovation: A Comparison of Modern ASR Methods for Forced
Alignment](https://arxiv.org/pdf/2406.19363v1) |