plan.md

# Plan: Acoustic-Phonetic Pipeline Pivot

## Overview
Replace the full-word DTW approach with a 3-phase pipeline:
1. **Phoneme isolation** via Wav2Vec2 CTC forced alignment
2. **Pitch-agnostic feature extraction** on isolated phoneme
3. **Threshold-based evaluation** (no reference audio needed)

## Phase 1: `phoneme_isolator.py` (new module)

**Model**: `imvladikon/wav2vec2-large-xlsr-53-hebrew` (~1.26GB)
- Character-level Hebrew CTC output (vocab includes ש at ID 17, all Hebrew letters)
- Apache-2.0 license

**Dependencies to install**:
- `torch` (CPU-only, ~200MB)
- `torchaudio` (must match torch version)
- `transformers`

**Implementation**:
1. Load model via `transformers.Wav2Vec2ForCTC` + `transformers.Wav2Vec2Processor`
2. Run audio through model → get CTC logits → log_softmax → emissions
3. Map Hebrew target text to token IDs using model's vocabulary
4. Call `torchaudio.functional.forced_align(emissions, targets, blank=pad_id)`
5. Call `torchaudio.functional.merge_tokens()` → get TokenSpan list with start/end frames
6. Find the span matching the target phoneme (e.g., ש)
7. Convert frame indices to sample indices, extract audio snippet
8. Return: `{ audio: np.ndarray, sr: int, start_ms: float, end_ms: float }`

**Key function**: `isolate_phoneme(audio_path, word_text, target_char) → dict`

**Lazy loading**: Model loaded once on first call, cached in module-level variable.

## Phase 2: Feature extraction (refactor in `score_engine.py`)

**New function**: `extract_phoneme_features(phoneme_audio, sr) → dict`

Pitch-agnostic features computed on the isolated phoneme snippet:
- Spectral centroid (3-8kHz bandpassed)
- Spectral bandwidth/spread
- Spectral skewness (3rd moment)
- Energy ratio: high band (5-8kHz) vs mid "slushy" band (1-3kHz)
- S-band ratio (existing)

## Phase 3: Threshold evaluation (refactor `score_pronunciation`)

**New function**: `evaluate_phoneme(features) → dict`

- No DTW, no reference audio comparison
- Pure threshold checks on extracted features
- Returns score, status, error_type, feedback
- Thresholds calibrated using existing correct/incorrect recordings

## Files Changed
- **NEW**: `phoneme_isolator.py` — Phase 1 module
- **EDIT**: `score_engine.py` — Add Phase 2+3 functions, refactor `score_pronunciation` to use new pipeline with fallback to old pipeline when alignment fails
- **EDIT**: `simulate.py` — Pass word text to scoring, display alignment info
- **EDIT**: `requirements.txt` — Add torch, torchaudio, transformers
- **KEEP**: Old DTW pipeline as fallback (don't delete yet, keep behind a flag)