plan.md

Plan: Acoustic-Phonetic Pipeline Pivot

Overview

Replace the full-word DTW approach with a 3-phase pipeline:

1. Phoneme isolation via Wav2Vec2 CTC forced alignment
2. Pitch-agnostic feature extraction on isolated phoneme
3. Threshold-based evaluation (no reference audio needed)

Phase 1: phoneme_isolator.py (new module)

Model: imvladikon/wav2vec2-large-xlsr-53-hebrew (~1.26GB)

• Character-level Hebrew CTC output (vocab includes ש at ID 17, all Hebrew letters)
• Apache-2.0 license

Dependencies to install:

• torch (CPU-only, ~200MB)
• torchaudio (must match torch version)
• transformers

Implementation:

1. Load model via transformers.Wav2Vec2ForCTC + transformers.Wav2Vec2Processor
2. Run audio through model → get CTC logits → log_softmax → emissions
3. Map Hebrew target text to token IDs using model's vocabulary
4. Call torchaudio.functional.forced_align(emissions, targets, blank=pad_id)
5. Call torchaudio.functional.merge_tokens() → get TokenSpan list with start/end frames
6. Find the span matching the target phoneme (e.g., ש)
7. Convert frame indices to sample indices, extract audio snippet
8. Return: { audio: np.ndarray, sr: int, start_ms: float, end_ms: float }

Key function: isolate_phoneme(audio_path, word_text, target_char) → dict

Lazy loading: Model loaded once on first call, cached in module-level variable.

Phase 2: Feature extraction (refactor in score_engine.py)

New function: extract_phoneme_features(phoneme_audio, sr) → dict

Pitch-agnostic features computed on the isolated phoneme snippet:

• Spectral centroid (3-8kHz bandpassed)
• Spectral bandwidth/spread
• Spectral skewness (3rd moment)
• Energy ratio: high band (5-8kHz) vs mid "slushy" band (1-3kHz)
• S-band ratio (existing)

Phase 3: Threshold evaluation (refactor score_pronunciation)

New function: evaluate_phoneme(features) → dict

• No DTW, no reference audio comparison
• Pure threshold checks on extracted features
• Returns score, status, error_type, feedback
• Thresholds calibrated using existing correct/incorrect recordings

Files Changed

• NEW: phoneme_isolator.py — Phase 1 module
• EDIT: score_engine.py — Add Phase 2+3 functions, refactor score_pronunciation to use new pipeline with fallback to old pipeline when alignment fails
• EDIT: simulate.py — Pass word text to scoring, display alignment info
• EDIT: requirements.txt — Add torch, torchaudio, transformers
• KEEP: Old DTW pipeline as fallback (don't delete yet, keep behind a flag)