--- license: agpl-3.0 language: [af, be, bs, bg, ca, hr, cs, da, nl, en, et, fi, fr, gl, de, ha, hu, id, it, ky, lv, lt, pl, pt, ro, ru, sr, sk, sl, es, sv, tr, uk, vi, cy] library_name: onnxruntime pipeline_tag: automatic-speech-recognition tags: [forced-alignment, word-timestamps, audiobooks, read-along, ctc, onnx, on-device, multilingual] ---
# CavaAlign v1 **Audio + text → word timings, small enough for a phone. 35 languages. 16 MB.** Give a recording and its text; get back when every word is spoken. 16 MB 35 languages ONNX Runtime 20 ms frames
CavaAlign v1: 16 MB, 88.0% of word starts within 50 ms, 0.3% of words more than 250 ms off or untimed, compared with wav2vec2-base, PocketSphinx, Vosk small-en, MMS-300m, NeMo Conformer-CTC S and Whisper small.en ## Highlights - **Precise.** On 70 LibriSpeech test speakers, 88% of word starts land within 50 ms of forced-alignment truth. That is close to the 89.5% where two independent references agree with each other. - **Small and fast.** 16 MB for all thirty-five languages, int8 weights, standard ONNX ops. Five minutes of audio align in 12 s on a 4-core laptop CPU, with 0.4 GB peak memory. - **Every word timed.** Each word in the text gets a time, and 0.3% are off by more than 250 ms. Speech recognisers with word timestamps leave 5–20% of words untimed or far off. - **Built for books.** Handles text that starts before the audio (front matter, chapter titles), unread footnotes and spoken announcements, without transcribing. ## Quick start ```bash pip install onnxruntime numpy soundfile regex num2words hf download cava-reader/cavaalign-v1 --local-dir cavaalign-v1 python cavaalign-v1/align.py speech.wav speech.txt --lang en ``` ```python import sys; sys.path.insert(0, "cavaalign-v1") import soundfile as sf from align import load, align sess, cfg = load() audio, sr = sf.read("speech.wav", dtype="float32") # 16 kHz mono with open("speech.txt", encoding="utf-8") as f: text = f.read() for w in align(sess, cfg, audio, text, lang="de")[:5]: print(f"{w['start']:7.2f} {w['end']:7.2f} {w['word']}") ``` `pip install numba` is optional and makes long recordings about 4× faster. Audio of any length works: `align.py` runs the model in short overlapping windows, so memory stays flat. `num2words` is used to read digits in languages other than English; `regex` is optional. ## Android (Kotlin) `kotlin/` contains the same romanizer and aligner as the Python code, plus an ONNX Runtime wrapper. Given the same model output, the Kotlin and Python aligners produce the same timings. 1. Add `com.microsoft.onnxruntime:onnxruntime-android` (1.17 or newer) and copy the three files in `kotlin/` into your project (package `dev.micr0.cavaalign`). 2. Ship `cavaalign.onnx` in your assets or download it on first use. 3. Align: ```kotlin val model = CavaAlignModel(context.assets.open("cavaalign.onnx").readBytes()) val emissions = model.logProbs(audio) // FloatArray, 16 kHz mono, about [-1, 1] val result = CavaAligner.fromText(text).align(emissions) for (w in result.words) { // w.wordIndex, w.startSec, w.endSec, w.measured } ``` `wordIndex` counts words in the text you passed in: runs of letters and digits, with apostrophes inside a word ("don't" is one word, "well-known" is two). `measured` is false for words the audio did not cover. Their times are inherited from the previous word. The aligner's memory grows with the length of the recording. Align long books one chapter or section at a time, passing the text for that stretch of audio. ## Benchmark English, 70 LibriSpeech test-clean / test-other speakers (about 60 s each, 12,195 words). Truth is Montreal Forced Aligner word starts. Every system gets the exact text and runs the way its users run it. Words a system leaves untimed count as misses. Systems with a constant timing delay are scored with that delay removed, so they are not penalised for a fixed offset. | Model | Size | Words within 50 ms | Off by > 250 ms or untimed | |---|---:|---:|---:| | **CavaAlign v1** | 16 MB | **88.0%** | **0.3%** | | wav2vec2-base (WhisperX) | 378 MB | 84.4% | 0.4% | | wav2vec2-large | 1.26 GB | 83.9% | 0.3% | | HuBERT-large | 1.26 GB | 83.9% | 0.3% | | data2vec-audio-base | 373 MB | 83.8% | 0.4% | | PocketSphinx | 10 MB | 83.8% | 8.6% | | Vosk en lgraph | 214 MB | 81.4% | 9.7% | | Vosk small-en | 71 MB | 80.4% | 12.7% | | MMS-300m aligner | 1.26 GB | 77.0% | 7.2% | | NeMo Conformer-CTC S | 46 MB | 67.3% | 5.7% | | NeMo Conformer-CTC M | 68 MB | 60.5% | 4.8% | | Zipformer-CTC | 70 MB | 55.2% | 3.9% | | Whisper small.en | 967 MB | 48.3% | 10.8% | | Whisper tiny.en | 151 MB | 45.5% | 13.2% | | NeMo Citrinet-512 | 38 MB | 44.0% | 7.0% | | Whisper base.en | 290 MB | 42.4% | 19.9% | Two independent MFA alignments of the same audio agree on **89.5%** of words within 50 ms, so this benchmark cannot separate systems above roughly 90%. On the words where the two references agree with each other, CavaAlign v1 is within 50 ms on **95.1%**. ## Languages One model covers all thirty-five. Text is romanized per word (Cyrillic, umlauts and accents → a–z), so there is no per-language download. Listed: every language tested with 70% or more of word starts within 50 ms. | Language | Words within 50 ms | > 250 ms off or untimed | |---|---:|---:| | Afrikaans † | 79.8% | 5.6% | | Belarusian | 74.2% | 11.1% | | Bosnian | 92.9% | 1.5% | | Bulgarian | 94.9% | 2.0% | | Catalan † | 85.8% | 3.4% | | Croatian | 82.2% | 3.6% | | Czech | 90.8% | 1.9% | | Danish † | 82.2% | 3.9% | | Dutch | 96.1% | 0.1% | | English | 89.7% | 0.1% | | Estonian † | 81.1% | 6.2% | | Finnish † | 90.9% | 2.6% | | French | 97.3% | 0.0% | | Galician † | 90.6% | 2.2% | | German | 96.0% | 0.0% | | Hausa | 79.2% | 3.6% | | Hungarian | 81.4% | 7.3% | | Indonesian † | 70.6% | 9.6% | | Italian | 80.9% | 6.4% | | Kyrgyz | 82.5% | 4.3% | | Latvian † | 89.4% | 1.9% | | Lithuanian † | 85.5% | 4.9% | | Polish | 98.9% | 0.0% | | Portuguese | 93.7% | 0.0% | | Romanian | 79.4% | 7.2% | | Russian | 92.6% | 1.1% | | Serbian | 85.3% | 1.5% | | Slovak † | 87.0% | 1.9% | | Slovenian † | 90.2% | 3.9% | | Spanish | 95.6% | 0.4% | | Swedish | 89.5% | 1.5% | | Turkish | 88.1% | 1.7% | | Ukrainian | 84.4% | 5.1% | | Vietnamese | 82.6% | 5.0% | | Welsh † | 75.0% | 7.6% | Average over all thirty-five (plain mean of the rows): **86.7%** of word starts within 50 ms, 3.4% more than 250 ms off or untimed. Read speech with MFA reference timing; words missing from MFA's pronunciation dictionaries get generated pronunciations (G2P), so every word is timed. † MFA has no model for these languages, so their reference is an MFA model trained on Common Voice audio with spelling-based pronunciations. On Czech, which has both, CavaAlign scores 88.4% against that kind of reference and 90.8% against MFA's own. Test audio: FLEURS for Afrikaans, Belarusian, Bosnian, Bulgarian, Catalan, Croatian, Czech, Danish, Estonian, Finnish, Galician, Hausa, Hungarian, Indonesian, Kyrgyz, Latvian, Lithuanian, Romanian, Serbian, Slovak, Slovenian, Swedish, Turkish, Ukrainian, Vietnamese and Welsh; LibriSpeech, Multilingual LibriSpeech and Russian LibriSpeech for Dutch, English, French, German, Italian, Polish, Portuguese, Russian and Spanish. ## Model | | | |---|---| | Architecture | raw-waveform front end + 8-layer streaming conformer (d 256), CTC head | | Parameters | 14.6 M | | Weights | int8 (dynamic quantisation), fp32 compute | | Input | `waveform` float32 `[batch, samples]` at 16 kHz mono, `lengths` int64 `[batch]` | | Output | `log_probs` `[batch, frames, 29]` at 50 frames/s, `out_lengths` | | Vocabulary | blank, a–z, apostrophe, unk (`vocab.json`) | | Aligner | monotonic dynamic program with text-skip, audio-skip and front-matter arcs (`cavaalign/`) | **Running it yourself:** - Feed the model 6.4 s windows with 3.2 s of preceding context, cut on 0.64 s chunk boundaries counted from the start of the audio (`align.py::log_probs`). Attention memory grows with the square of the input length, so whole chapters in one pass will not fit. - Create the ONNX Runtime session with the CPU memory arena and memory pattern disabled. - Romanize the text before tokenizing (`cavaalign/romanize.py`); a space maps to the blank id. - Frames are 20 ms (`frame_shift_sec` in `align_config.json`). ## Files | file | | |---|---| | `cavaalign.onnx` | the model | | `align.py`, `cavaalign/` | Python reference: romanizer and aligner (numpy) | | `align_config.json`, `vocab.json` | aligner settings, vocabulary, frame shift | | `kotlin/` | the same romanizer and aligner in Kotlin, plus an ONNX Runtime wrapper, for Android | | `reference/` | English, German and Russian clips with expected outputs, to check a port | | `SHA256SUMS` | checksums (`sha256sum -c SHA256SUMS`) | | `LICENSE` | GNU Affero General Public License v3 | To check a port, run each `reference/*.wav` with its `.txt` and compare with `.expected.json`: `romanized_text` must match exactly, `log_probs_frames` must be equal, and word times should match to the frame (an occasional one-frame difference from int8 rounding on ARM is normal). ## Limitations - It aligns given text; it does not transcribe. Speech that is not in the text is skipped. - Built for read speech (audiobooks). - Precision drops when the text window is much wider than the audio. Hand it the text for the stretch being played, with modest slack. - Digits are read with language-specific number words in Python; the Kotlin port uses English readings. - Latin and Cyrillic scripts only. Turkish ç and ü are romanized approximately (as "s" and "ue"), and Hausa's hooked letters (ɓ ɗ ƙ) are dropped; the words still align from their other letters. ## Reference clips The clips in `reference/` are 30 s excerpts of LibriVox recordings: - `en_*`: LibriSpeech test-clean, speaker 1089, chapter 134686 ([OpenSLR 12](https://openslr.org/12), CC BY 4.0). - `de_*`: Multilingual LibriSpeech German, speaker 3363, book 2124 ([OpenSLR 94](https://openslr.org/94), CC BY 4.0). - `ru_*`: Russian LibriSpeech ([OpenSLR 96](https://openslr.org/96), public domain in the USA). ## License Copyright (C) 2026 Cava Reader. CavaAlign (the model weights in `cavaalign.onnx` and all code in this repository) is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. See `LICENSE`. The reference clips keep their own licenses, listed above. It is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.