masumtechnonext/wav2vec2-arabic-letter-verifier

Arabic Isolated-Letter Pronunciation Verifier

Fine-tuned wav2vec2-large-xlsr-53-arabic for verifying whether a spoken clip matches one of the 28 Arabic alphabet letters, trained on masumtechnonext/test-data-set-Arabic-letter.

Why this exists

A plain 28-way classifier always has to pick some letter, even for audio that isn't any of the 28 letters at all (e.g. a name, noise, silence) or is a confusable near-miss (seen/sheen, dal/dhal, ta/taa/tha, ...). That gives false "correct" verdicts. This model adds:

1. A 29th trained "Unknown" class (silence, noise, time-reversed clips, cross-letter mixup) so out-of-vocabulary audio has somewhere to go instead of being forced into a real letter.
2. A calibrated confidence threshold (see calibration.json, confidence_threshold) — even if the model picks a real letter, the prediction is only reported as "correct" when max softmax probability clears the threshold. Below it, report "not recognized" rather than a guess.

Usage

from inference import LetterVerifier
import soundfile as sf

verifier = LetterVerifier("masumtechnonext/wav2vec2-arabic-letter-verifier")
audio, sr = sf.read("clip.wav", dtype="float32")  # 16kHz mono
result = verifier.predict(audio, sr, target_letter="Sheen")
print(result)  # {"predicted_label": ..., "confidence": ..., "accepted": ..., "is_correct": ...}

See inference.py in this repo for the full wrapper (handles the Unknown-class + threshold rejection logic).

Training

• Backbone: jonatasgrosman/wav2vec2-large-xlsr-53-arabic (encoder only; CTC head replaced with a classification head), feature encoder frozen.
• 29 classes: 28 letters + Unknown.
• SpecAugment (time/feature masking), label smoothing 0.1.
• Model selection by macro-F1 on a held-out validation split.
• See test_confusion_matrix.csv for per-letter confusions on the held-out test split, and calibration.json for the threshold calibration sweep.

Known limitation

The source recordings have no speaker ID, so train/validation/test are split by clip (stratified per letter, not per speaker). If the same speaker appears in both train and test, held-out accuracy may be optimistic relative to a genuinely new speaker/microphone. Validate on your own held-out speakers before deploying.