Usage agreement: free, non-commercial use only
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
This model is shared freely for the sake of Allah. By requesting access you agree: (1) you may use, fine-tune and redistribute it and its outputs ONLY in applications that are FREE to end users; (2) you may NOT sell it, place it behind a paid subscription or paywall, monetize it with ads, or earn any revenue from an app or service that uses this model or its outputs; (3) these terms pass on to anyone you share it with. وَما أَسأَلُكُم عَلَيهِ مِن أَجرٍ إِن أَجرِيَ إِلّا عَلىٰ رَبِّ العالَمِين
Log in or Sign Up to review the conditions and access this model content.
zipformer_p-arabic-v2
A streaming phoneme recognizer for Arabic. It maps audio to phoneme units (consonant plus haraka) representing what was actually pronounced, including dialect and tajweed.
Scope. This is a general Arabic model. It covers everyday speech (MSA, broadcast, and Egyptian, Gulf, Levantine and Maghrebi dialect) as well as Qur'anic recitation with tajweed. Qur'an-only models typically degrade sharply on ordinary Arabic; this model is trained for both domains in one.
Comparison with the models it replaces
| this model | zipformer_p-quran(Qur'an specialist) |
zipformer_p-arabic(v1, broad) |
|
|---|---|---|---|
| Qur'an, real phone audio (held-out PER) | 8.14% | 13.39% | 34.72% |
| Qur'an, clean recitation (PER) | 3.31% | 5.82% | not applicable |
| General Arabic (khutbah, CER) | 4.7% | 22.4% | 6.9% |
| General Arabic (khutbah, WER) | 9.4% | 36.9% | 11.4% |
The Qur'an specialist is accurate on clean recitation but loses ground on real phone audio, and its tajweed markers transfer poorly to everyday speech, which is what the general Arabic rows show. Training both domains jointly retained Qur'an accuracy and improved general Arabic.
Specification
| Architecture | Zipformer2 (icefall/k2), causal streaming, CTC |
| Parameters | 64.7M |
| Vocabulary | 250 phoneme units plus blank |
| Features | 80-bin kaldi fbank (povey), 16 kHz |
| Training | from scratch, 10 epochs, 2,039 h unique audio, last 3 epochs averaged |
| Streaming | decode_chunk_len 48, chunk 24, left context 256 |
| Formats | PyTorch, ONNX fp32, ONNX int8, CoreML fp16 and fp32 |
Benchmarks
Official Qur'an-Lab benchmark (600 clips)
Scored with the official score.py normalisation. Phonemes are converted to text by nearest-ayah
retrieval, the same method used for the published zipformer_p-quran entry.
| Metric | this model | zipformer_p-quran |
|---|---|---|
| Overall WER | 5.64 | 5.83 |
Phone WER (tlog_holdout) |
7.56 | 7.76 |
| EveryAyah WER | 7.72 | 7.64 |
| QUL WER | 2.19 | 2.66 |
| WER (alef-insensitive) | 4.40 | 4.59 |
Phoneme error rate
| Test set | this model | zipformer_p-quran |
|---|---|---|
| Held-out real-phone Qur'an (793 clips, deterministic gold) | 8.14% | 13.39% |
| Clean Qur'an recitation (400 clips) | 3.31% | 5.82% |
| Benchmark clips with gold phonemes (492) | 8.26% | 11.54% |
PER is the appropriate metric for this model, and the two tables measure different things. Nearest-ayah retrieval constrains output to one of about 9,112 ayat, so it corrects most acoustic errors before WER is computed: retrieval selects the correct ayah 99.8% of the time for this model against 99.6% for the specialist. With that little headroom, leaderboard WER compresses a 28% relative PER improvement into a 3% WER improvement. Use PER to compare acoustic quality and WER to compare end-to-end retrieval systems.
General Arabic (non-Qur'an)
Reverberant YouTube khutbah (MSA sermon), phonemes converted to text by an LLM, scored against ElevenLabs Scribe:
| WER | CER | |
|---|---|---|
| this model | 9.4% | 4.7% |
zipformer_p-arabic (v1) |
11.4% | 6.9% |
zipformer_p-quran |
36.9% | 22.4% |
Noise robustness
v1 was trained on clean, un-augmented audio. v2 applies three augmentations on GPU every batch, ramped in over the first 6,000 steps:
| Augmentation | Setting | Purpose |
|---|---|---|
| MUSAN additive noise | 1,983 clips (noise, music, babble), p=0.5, SNR 10 to 20 dB | rooms, traffic, crowds, PA systems, phone audio |
| Speed perturbation | 0.9x, 1.0x, 1.1x | fast and slow delivery |
| SpecAugment | 2 frequency masks (max 27 bins), 2 time masks (max 15% total) | reduces over-fitting to specific frames and bands |
The effect is largest where conditions are worst. On 793 held-out real phone recordings v2 scores 8.14% PER against 13.39% for the Qur'an specialist, a 39% relative reduction. On clean studio recitation the margin narrows to 3.31% against 5.82%.
MUSAN is narrowband: median 99% energy bandwidth is about 3.2 kHz, with under 0.6% of energy above 6 kHz. The model is therefore well hardened against low and mid frequency interference, which covers most real-world noise, but has seen comparatively little high frequency hiss.
Training data
2,039 h of unique audio.
Broad Arabic, 1,527 h (MSA, broadcast, and dialect coverage)
| Source | Hours | Share | Contribution |
|---|---|---|---|
| MASC | 859.8 h | 56.3% | dialectal YouTube speech (Egyptian, Gulf, Levantine) |
| SADA | 254.7 h | 16.7% | Saudi broadcast and drama, dialectal |
| MGB2 | 225.7 h | 14.8% | Al Jazeera broadcast MSA |
mostafa_mahmoud |
67.2 h | 4.4% | Egyptian colloquial broadcast speech |
| CommonVoice | 64.8 h | 4.2% | read MSA |
| ArVoice | 49.1 h | 3.2% | read and expressive |
| FLEURS | 5.8 h | 0.4% | read MSA |
Qur'an, 512 h
| Source | Hours | Share | Contribution |
|---|---|---|---|
| EveryAyah | 392.0 h | 76.6% | multi-reciter studio recitation |
tlog (tier S) |
85.2 h | 16.6% | real-phone recitation: phones, rooms, background noise |
| muaalem | 27.4 h | 5.4% | tajweed-annotated recitation |
tlog (tiers A and B) |
6.8 h | 1.3% | lower-confidence labels, excluded from this run |
EveryAyah and muaalem are repeated 4x per epoch. The effective training mix is 3,290 h per epoch:
| Component | Effective hours | Share of epoch |
|---|---|---|
| Clean studio Qur'an (EveryAyah) | 1,568 h | 47.7% |
| Broad Arabic | 1,527 h | 46.4% |
| Tajweed-annotated Qur'an (muaalem) | 110 h | 3.3% |
Real-phone Qur'an (tlog tier S) |
85 h | 2.6% |
Real-phone Qur'an is a small fraction of the mix, so the noise robustness reported above comes predominantly from the augmentation and from the acoustic variety of the broad Arabic corpora rather than from volume of in-domain noisy recitation.
For comparison, zipformer_p-quran was trained almost entirely on Qur'an, dominated by clean studio
recitation. Applications tuned against that model may need their thresholds recalibrated for v2, since
clean studio audio accounts for a smaller share of this model's training distribution.
Labels
The audio above comes from public and third-party corpora. The phoneme labelling is ours. No public Arabic phoneme corpus exists at this scale, so every phoneme label used in training was produced for this project and released as two datasets:
quran-phonemes, deterministic. Generated rule-based withquran-transcript(the generator behind the muaalem gold), so tajweed is exact rather than inferred: madd lengths, ghunna, idgham and qalqala follow Hafs rules.arabic-broad-phonemes, LLM-phonemized and source-aware. Dialectal sources are prompted for dialectal vowels and MSA sources for MSA, preserving dialect rather than normalising it. Being LLM-generated it carries some noise, mainly vowel-length artifacts.- Quality tiering. The
tlogreal-phone clips are tiered S, A and B by teacher-label confidence and weighted accordingly, so noisier labels count for less.
The tlog_holdout evaluation set is excluded from training, so the real-phone figures above are
measured on unseen clips of the same domain.
The 41-symbol Qur'an alphabet is a superset of the 37-symbol broad alphabet, so both are covered by one 250-unit tokenizer.
Files
| File | Size | Contents |
|---|---|---|
zipformer_p_arabic_v2.pt |
259 MB | PyTorch weights (64.7M params) plus blank_id |
zipformer_p_arabic_v2.onnx |
263 MB | cache-aware streaming CTC, sherpa-onnx compatible |
zipformer_p_arabic_v2.int8.onnx |
73 MB | same graph, dynamic INT8, for mobile and edge |
zipformer_p_arabic_v2.streaming.ane.mlpackage |
130 MB | CoreML fp16, streaming, Apple Neural Engine, iOS 17+ |
zipformer_p_arabic_v2.streaming.mlpackage |
258 MB | CoreML fp32, streaming, iOS 17+ (reference and CPU/GPU fallback) |
tokens.txt |
3 KB | symbol table for decoding |
phoneme_units.json |
3 KB | training-time tokenizer (greedy longest match) |
decode_with_confidence.py |
per-phoneme confidence, alternatives and timings | |
LICENSE |
free / non-commercial terms |
Symbol table
Decode with tokens.txt. It is authoritative and is the format sherpa-onnx expects.
phoneme_units.json is a training-time artifact whose indices do not describe the model's output
layer. It numbers <blank> as 0 and the phonemes 1 to 250, whereas the model emits phonemes at 0 to
249 with blank at 250. A symbol table built from the JSON is therefore shifted by one symbol. The model
loads, decodes and emits well-formed Arabic in that state, so the failure is silent and appears only as
degraded accuracy. Thanks to the QuranTracker team for identifying and reporting this.
Blank at 250 is intentional and matches zipformer_p-quran. It will not change without a corresponding
tokens.txt published alongside, so read the blank id from the file rather than hard-coding it.
ONNX
The ONNX graphs are chunk-by-chunk streaming with cache states, so they are not bit-identical to full-utterance PyTorch decoding. Accuracy on held-out real-phone Qur'an:
| Runtime | PER |
|---|---|
| PyTorch, offline full-utterance | 8.83% |
| ONNX fp32, streaming | 10.51% |
| ONNX int8, streaming | 10.74% |
The 1.7 point difference is the expected cost of limited lookahead relative to full-utterance context. INT8 costs a further 0.23 points while being 3.6x smaller, making it the default choice on device.
Streaming configuration is stored in the ONNX metadata (decode_chunk_len=48, T=61,
left_context_len=256,128,64,32,64,128), so sherpa-onnx reads it automatically.
CoreML
Both CoreML builds are cache-aware streaming, produced from the same icefall wrapper as the streaming ONNX, so CoreML and ONNX perform the identical per-chunk computation. Nothing is padded to a fixed utterance length.
Use .streaming.ane.mlpackage (fp16), since the Apple Neural Engine executes fp16 only. The fp32
package is a reference and CPU/GPU fallback.
Interface, per call:
Input x |
(1, 61, 80), one chunk of 80-bin kaldi fbank (povey, 16 kHz) |
Input state_0 .. state_97 |
98 cache tensors, zero-initialised on the first call |
Output log_probs |
(1, 12, 251) |
Output new_state_0 .. new_state_97 |
carry into the next call |
fp16 precision was verified rather than assumed:
| Check | Result |
|---|---|
| Peak activation across every leaf module, on real audio | 1,042.7, which is 0.016x the fp16 ceiling of 65,504. No module exceeds the limit. |
| fp16 against fp32 decoding, 15 real phone clips | 15/15 identical phoneme strings, 0 blank outputs |
A FastConformer model elsewhere in this project does blank on ANE in fp16, because its
RelPositionalEncoding multiplies activations by sqrt(512) and lifts peaks near 5,200 to about
118,000. Zipformer2 has no such multiply; it uses CompactRelPositionalEncoding, and the measured
headroom above is roughly 63x, so that failure mode does not apply here.
coremltools cannot execute a model outside macOS, and these packages were built on Windows and WSL. They are conversion-verified and precision-verified, but not runtime-verified on Apple hardware. The ONNX files were measured end to end against gold labels. When deploying on device, check the first outputs against the ONNX or PyTorch reference.
Per-phoneme confidence, alternatives and timings
Greedy CTC keeps the argmax and discards the rest of the 251-way posterior. For recitation grading that
discarded information is useful, because low model certainty and a genuine mispronunciation are
indistinguishable in a plain transcript. decode_with_confidence.py retains it, so every emitted
phoneme carries a confidence, its runner-up candidates and a time span.
$ python decode_with_confidence.py clip.wav --topk 2
ءِننننَرَببَهُ۾۾۾بِهِميَومَءِذِللَخَبِۦۦر
time ph conf margin alternatives
1.20- 1.24 ننننَ 1.000 1.000 <blank>:0.00 نَ:0.00
4.52- 4.60 ءِ 0.790 0.581 <blank>:0.21 ا:0.00
5.44- 5.48 خَ 1.000 0.999 حَ:0.00 <blank>:0.00
5.72- 5.80 بِ 0.894 0.789 <blank>:0.10 رِ:0.00
Each entry gives phoneme, confidence (0 to 1), margin (confidence minus runner-up), start and
end in seconds, and alternatives. Pass --json out.json for machine-readable output.
CTC posteriors are peaky. On the clip above the mean confidence is 0.983 and no value falls below 0.5,
so a fixed confidence < 0.5 gate will rarely fire and high confidence is not evidence of correctness.
Two signals carry more information:
marginrather thanconfidence. A phoneme at 0.79 with<blank>at 0.21 is a contested position; one at 1.000 with all alternatives at 0.00 is not.- The identity of the runner-up.
<blank>competing indicates uncertainty that anything was said at that position (onset, pause, clipped sound). A competing phoneme, for example خَ against حَ, indicates a sound was heard but its identity is ambiguous, which is the useful case for pronunciation feedback.
Thresholds are not calibrated probabilities and should be tuned on your own audio. For true calibration, temperature-scale the logits against a labelled set of your own recordings.
Timing resolution is one encoder frame, 40 ms. This separates phonemes reliably but is coarse for grading a precise 2-harakah hold, so treat spans as boundaries rather than fine duration measurements.
Usage
icefall Zipformer2, 80-bin kaldi fbank (povey window) at 16 kHz, CTC greedy decode to phoneme units.
Use torchaudio.compliance.kaldi.fbank, not torchaudio.transforms.MelSpectrogram. Slaney-mel
features roughly double the error rate (measured: 11.4% to 22.1% WER on identical audio) without
producing any error, and are the most common cause of poor results with this model.
import torch, torchaudio.compliance.kaldi as kaldi
feats = kaldi.fbank(wav, num_mel_bins=80, frame_length=25.0, frame_shift=10.0,
sample_frequency=16000, dither=0.0, snip_edges=False,
window_type="povey", use_energy=False, low_freq=20.0,
high_freq=-400.0, remove_dc_offset=True, preemphasis_coefficient=0.97)
For the 1000 ms look-ahead profile use chunk_size=(24,) and left_context_frames=(256,).
The ONNX files use the standard sherpa-onnx online zipformer2-CTC format (model_type=zipformer2), so
they load directly in sherpa-onnx, and in plain onnxruntime by feeding one chunk plus the cache tensors
and carrying the returned states forward.
When streaming ends, flush the buffer by passing one final chunk of silence (at least the right-context length, about 1.05 s). Audio still held in the right-context buffer is not emitted otherwise, which causes short final words to be dropped.
Limitations
- Phonemes, not text. Converting Arabic phonemes to orthography is a separate problem. Evaluate this model by PER rather than word-level text WER.
- Retrieval is a Qur'an-lexicon constraint. The leaderboard WER above uses nearest-ayah matching and does not transfer to general Arabic.
- Dialectal orthography is out of scope. The model transcribes dialect faithfully at the phonetic level, but reconstructing fluent dialectal text from it is a separate task.
- Broad-Arabic labels are LLM-generated and carry some vowel-length noise.
- Emphatic consonants (ط ض ظ ص) degrade on reciters absent from training. Measured on a held-out unseen reciter, ط reaches 24.4% error against 2.6% for ت, while on reciters seen in training ط is 9.3%. The cause is speaker generalisation rather than noise, and reciter diversity is the effective remedy.
- The CoreML builds have not been executed on Apple hardware, since coremltools cannot run a model outside macOS. They are conversion-verified and precision-verified only.
License
Shared freely for the sake of Allah under free, non-commercial terms, not Apache-2.0. Access is gated:
free apps only, no sales, no paid subscriptions or paywalls, no ad revenue, and no monetization of any
kind. These terms pass on to anyone you share it with. See LICENSE.
- Downloads last month
- 7