|
Download README.md from cava-reader/cavaalign-v1: direct link, hf CLI and curl.
- Browser
- Download file 11.7 kB
-
https://huggingface.co/cava-reader/cavaalign-v1/resolve/main/README.md
- Command line
-
hf download hf://cava-reader/cavaalign-v1/README.md
-
curl -L -o README.md https://huggingface.co/cava-reader/cavaalign-v1/resolve/main/README.md
11.7 kB
| license: agpl-3.0 | |
| language: [af, be, bs, bg, ca, hr, cs, da, nl, en, et, fi, fr, gl, de, ha, hu, id, it, ky, lv, lt, pl, pt, ro, ru, sr, sk, sl, es, sv, tr, uk, vi, cy] | |
| library_name: onnxruntime | |
| pipeline_tag: automatic-speech-recognition | |
| tags: [forced-alignment, word-timestamps, audiobooks, read-along, ctc, onnx, on-device, multilingual] | |
| <div align="center"> | |
| # CavaAlign v1 | |
| **Audio + text → word timings, small enough for a phone. 35 languages. 16 MB.** | |
| Give a recording and its text; get back when every word is spoken. | |
| <a href="#"><img src="https://img.shields.io/badge/size-16%20MB-2a78d6" alt="16 MB"></a> | |
| <a href="#"><img src="https://img.shields.io/badge/languages-35-2a78d6" alt="35 languages"></a> | |
| <a href="#"><img src="https://img.shields.io/badge/runtime-ONNX%20Runtime%20CPU-2a78d6" alt="ONNX Runtime"></a> | |
| <a href="#"><img src="https://img.shields.io/badge/timing-20%20ms%20frames-2a78d6" alt="20 ms frames"></a> | |
| </div> | |
| <img class="block dark:hidden" src="https://huggingface.co/cava-reader/cavaalign-v1/resolve/main/assets/compare.png" alt="CavaAlign v1: 16 MB, 88.0% of word starts within 50 ms, 0.3% of words more than 250 ms off or untimed, compared with wav2vec2-base, PocketSphinx, Vosk small-en, MMS-300m, NeMo Conformer-CTC S and Whisper small.en"> | |
| <img class="hidden dark:block" src="https://huggingface.co/cava-reader/cavaalign-v1/resolve/main/assets/compare_dark.png" alt="CavaAlign v1: 16 MB, 88.0% of word starts within 50 ms, 0.3% of words more than 250 ms off or untimed, compared with wav2vec2-base, PocketSphinx, Vosk small-en, MMS-300m, NeMo Conformer-CTC S and Whisper small.en"> | |
| ## Highlights | |
| - **Precise.** On 70 LibriSpeech test speakers, 88% of word starts land within 50 ms of | |
| forced-alignment truth. That is close to the 89.5% where two independent references agree | |
| with each other. | |
| - **Small and fast.** 16 MB for all thirty-five languages, int8 weights, standard ONNX ops. Five | |
| minutes of audio align in 12 s on a 4-core laptop CPU, with 0.4 GB peak memory. | |
| - **Every word timed.** Each word in the text gets a time, and 0.3% are off by more than 250 ms. | |
| Speech recognisers with word timestamps leave 5–20% of words untimed or far off. | |
| - **Built for books.** Handles text that starts before the audio (front matter, chapter titles), | |
| unread footnotes and spoken announcements, without transcribing. | |
| ## Quick start | |
| ```bash | |
| pip install onnxruntime numpy soundfile regex num2words | |
| hf download cava-reader/cavaalign-v1 --local-dir cavaalign-v1 | |
| python cavaalign-v1/align.py speech.wav speech.txt --lang en | |
| ``` | |
| ```python | |
| import sys; sys.path.insert(0, "cavaalign-v1") | |
| import soundfile as sf | |
| from align import load, align | |
| sess, cfg = load() | |
| audio, sr = sf.read("speech.wav", dtype="float32") # 16 kHz mono | |
| with open("speech.txt", encoding="utf-8") as f: | |
| text = f.read() | |
| for w in align(sess, cfg, audio, text, lang="de")[:5]: | |
| print(f"{w['start']:7.2f} {w['end']:7.2f} {w['word']}") | |
| ``` | |
| `pip install numba` is optional and makes long recordings about 4× faster. Audio of any length | |
| works: `align.py` runs the model in short overlapping windows, so memory stays flat. | |
| `num2words` is used to read digits in languages other than English; `regex` is optional. | |
| ## Android (Kotlin) | |
| `kotlin/` contains the same romanizer and aligner as the Python code, plus an ONNX Runtime | |
| wrapper. Given the same model output, the Kotlin and Python aligners produce the same timings. | |
| 1. Add `com.microsoft.onnxruntime:onnxruntime-android` (1.17 or newer) and copy the three files | |
| in `kotlin/` into your project (package `dev.micr0.cavaalign`). | |
| 2. Ship `cavaalign.onnx` in your assets or download it on first use. | |
| 3. Align: | |
| ```kotlin | |
| val model = CavaAlignModel(context.assets.open("cavaalign.onnx").readBytes()) | |
| val emissions = model.logProbs(audio) // FloatArray, 16 kHz mono, about [-1, 1] | |
| val result = CavaAligner.fromText(text).align(emissions) | |
| for (w in result.words) { | |
| // w.wordIndex, w.startSec, w.endSec, w.measured | |
| } | |
| ``` | |
| `wordIndex` counts words in the text you passed in: runs of letters and digits, with | |
| apostrophes inside a word ("don't" is one word, "well-known" is two). `measured` is false for | |
| words the audio did not cover. Their times are inherited from the previous word. | |
| The aligner's memory grows with the length of the recording. Align long books one chapter or | |
| section at a time, passing the text for that stretch of audio. | |
| ## Benchmark | |
| English, 70 LibriSpeech test-clean / test-other speakers (about 60 s each, 12,195 words). | |
| Truth is Montreal Forced Aligner word starts. Every system gets the exact text and runs the way its | |
| users run it. Words a system leaves untimed count as misses. Systems with a constant timing delay | |
| are scored with that delay removed, so they are not penalised for a fixed offset. | |
| | Model | Size | Words within 50 ms | Off by > 250 ms or untimed | | |
| |---|---:|---:|---:| | |
| | **CavaAlign v1** | 16 MB | **88.0%** | **0.3%** | | |
| | wav2vec2-base (WhisperX) | 378 MB | 84.4% | 0.4% | | |
| | wav2vec2-large | 1.26 GB | 83.9% | 0.3% | | |
| | HuBERT-large | 1.26 GB | 83.9% | 0.3% | | |
| | data2vec-audio-base | 373 MB | 83.8% | 0.4% | | |
| | PocketSphinx | 10 MB | 83.8% | 8.6% | | |
| | Vosk en lgraph | 214 MB | 81.4% | 9.7% | | |
| | Vosk small-en | 71 MB | 80.4% | 12.7% | | |
| | MMS-300m aligner | 1.26 GB | 77.0% | 7.2% | | |
| | NeMo Conformer-CTC S | 46 MB | 67.3% | 5.7% | | |
| | NeMo Conformer-CTC M | 68 MB | 60.5% | 4.8% | | |
| | Zipformer-CTC | 70 MB | 55.2% | 3.9% | | |
| | Whisper small.en | 967 MB | 48.3% | 10.8% | | |
| | Whisper tiny.en | 151 MB | 45.5% | 13.2% | | |
| | NeMo Citrinet-512 | 38 MB | 44.0% | 7.0% | | |
| | Whisper base.en | 290 MB | 42.4% | 19.9% | | |
| Two independent MFA alignments of the same audio agree on **89.5%** of words within 50 ms, so this | |
| benchmark cannot separate systems above roughly 90%. On the words where the two references agree | |
| with each other, CavaAlign v1 is within 50 ms on **95.1%**. | |
| ## Languages | |
| One model covers all thirty-five. Text is romanized per word (Cyrillic, umlauts and accents → | |
| a–z), so there is no per-language download. Listed: every language tested with 70% or more of | |
| word starts within 50 ms. | |
| | Language | Words within 50 ms | > 250 ms off or untimed | | |
| |---|---:|---:| | |
| | Afrikaans † | 79.8% | 5.6% | | |
| | Belarusian | 74.2% | 11.1% | | |
| | Bosnian | 92.9% | 1.5% | | |
| | Bulgarian | 94.9% | 2.0% | | |
| | Catalan † | 85.8% | 3.4% | | |
| | Croatian | 82.2% | 3.6% | | |
| | Czech | 90.8% | 1.9% | | |
| | Danish † | 82.2% | 3.9% | | |
| | Dutch | 96.1% | 0.1% | | |
| | English | 89.7% | 0.1% | | |
| | Estonian † | 81.1% | 6.2% | | |
| | Finnish † | 90.9% | 2.6% | | |
| | French | 97.3% | 0.0% | | |
| | Galician † | 90.6% | 2.2% | | |
| | German | 96.0% | 0.0% | | |
| | Hausa | 79.2% | 3.6% | | |
| | Hungarian | 81.4% | 7.3% | | |
| | Indonesian † | 70.6% | 9.6% | | |
| | Italian | 80.9% | 6.4% | | |
| | Kyrgyz | 82.5% | 4.3% | | |
| | Latvian † | 89.4% | 1.9% | | |
| | Lithuanian † | 85.5% | 4.9% | | |
| | Polish | 98.9% | 0.0% | | |
| | Portuguese | 93.7% | 0.0% | | |
| | Romanian | 79.4% | 7.2% | | |
| | Russian | 92.6% | 1.1% | | |
| | Serbian | 85.3% | 1.5% | | |
| | Slovak † | 87.0% | 1.9% | | |
| | Slovenian † | 90.2% | 3.9% | | |
| | Spanish | 95.6% | 0.4% | | |
| | Swedish | 89.5% | 1.5% | | |
| | Turkish | 88.1% | 1.7% | | |
| | Ukrainian | 84.4% | 5.1% | | |
| | Vietnamese | 82.6% | 5.0% | | |
| | Welsh † | 75.0% | 7.6% | | |
| Average over all thirty-five (plain mean of the rows): **86.7%** of word starts within 50 ms, | |
| 3.4% more than 250 ms off or untimed. Read speech with MFA reference timing; words missing | |
| from MFA's pronunciation dictionaries get generated pronunciations (G2P), so every word is timed. | |
| † MFA has no model for these languages, so their reference is an MFA model trained on Common Voice | |
| audio with spelling-based pronunciations. On Czech, which has both, CavaAlign scores 88.4% against | |
| that kind of reference and 90.8% against MFA's own. | |
| Test audio: FLEURS for Afrikaans, Belarusian, Bosnian, Bulgarian, Catalan, Croatian, Czech, Danish, | |
| Estonian, Finnish, Galician, Hausa, Hungarian, Indonesian, Kyrgyz, Latvian, Lithuanian, Romanian, | |
| Serbian, Slovak, Slovenian, Swedish, Turkish, Ukrainian, Vietnamese and Welsh; LibriSpeech, | |
| Multilingual LibriSpeech and Russian LibriSpeech for Dutch, English, French, German, Italian, | |
| Polish, Portuguese, Russian and Spanish. | |
| ## Model | |
| | | | | |
| |---|---| | |
| | Architecture | raw-waveform front end + 8-layer streaming conformer (d 256), CTC head | | |
| | Parameters | 14.6 M | | |
| | Weights | int8 (dynamic quantisation), fp32 compute | | |
| | Input | `waveform` float32 `[batch, samples]` at 16 kHz mono, `lengths` int64 `[batch]` | | |
| | Output | `log_probs` `[batch, frames, 29]` at 50 frames/s, `out_lengths` | | |
| | Vocabulary | blank, a–z, apostrophe, unk (`vocab.json`) | | |
| | Aligner | monotonic dynamic program with text-skip, audio-skip and front-matter arcs (`cavaalign/`) | | |
| **Running it yourself:** | |
| - Feed the model 6.4 s windows with 3.2 s of preceding context, cut on 0.64 s chunk boundaries | |
| counted from the start of the audio (`align.py::log_probs`). Attention memory grows with the | |
| square of the input length, so whole chapters in one pass will not fit. | |
| - Create the ONNX Runtime session with the CPU memory arena and memory pattern disabled. | |
| - Romanize the text before tokenizing (`cavaalign/romanize.py`); a space maps to the blank id. | |
| - Frames are 20 ms (`frame_shift_sec` in `align_config.json`). | |
| ## Files | |
| | file | | | |
| |---|---| | |
| | `cavaalign.onnx` | the model | | |
| | `align.py`, `cavaalign/` | Python reference: romanizer and aligner (numpy) | | |
| | `align_config.json`, `vocab.json` | aligner settings, vocabulary, frame shift | | |
| | `kotlin/` | the same romanizer and aligner in Kotlin, plus an ONNX Runtime wrapper, for Android | | |
| | `reference/` | English, German and Russian clips with expected outputs, to check a port | | |
| | `SHA256SUMS` | checksums (`sha256sum -c SHA256SUMS`) | | |
| | `LICENSE` | GNU Affero General Public License v3 | | |
| To check a port, run each `reference/*.wav` with its `.txt` and compare with `.expected.json`: | |
| `romanized_text` must match exactly, `log_probs_frames` must be equal, and word times should match | |
| to the frame (an occasional one-frame difference from int8 rounding on ARM is normal). | |
| ## Limitations | |
| - It aligns given text; it does not transcribe. Speech that is not in the text is skipped. | |
| - Built for read speech (audiobooks). | |
| - Precision drops when the text window is much wider than the audio. Hand it the text for the | |
| stretch being played, with modest slack. | |
| - Digits are read with language-specific number words in Python; the Kotlin port uses English | |
| readings. | |
| - Latin and Cyrillic scripts only. Turkish ç and ü are romanized approximately (as "s" and "ue"), | |
| and Hausa's hooked letters (ɓ ɗ ƙ) are dropped; the words still align from their other letters. | |
| ## Reference clips | |
| The clips in `reference/` are 30 s excerpts of LibriVox recordings: | |
| - `en_*`: LibriSpeech test-clean, speaker 1089, chapter 134686 ([OpenSLR 12](https://openslr.org/12), CC BY 4.0). | |
| - `de_*`: Multilingual LibriSpeech German, speaker 3363, book 2124 ([OpenSLR 94](https://openslr.org/94), CC BY 4.0). | |
| - `ru_*`: Russian LibriSpeech ([OpenSLR 96](https://openslr.org/96), public domain in the USA). | |
| ## License | |
| Copyright (C) 2026 Cava Reader. | |
| CavaAlign (the model weights in `cavaalign.onnx` and all code in this repository) is free | |
| software: you can redistribute it and/or modify it under the terms of the GNU Affero General | |
| Public License as published by the Free Software Foundation, either version 3 of the License, | |
| or (at your option) any later version. See `LICENSE`. The reference clips keep their own | |
| licenses, listed above. | |
| It is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the | |
| implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. | |