README.md

metadata

title: Speechkid API
emoji: 🎙️
colorFrom: blue
colorTo: purple
sdk: docker
pinned: false

SpeechKid API - Hebrew 'Sh' Sound Pronunciation Trainer

AI-powered pronunciation trainer for children learning the Hebrew 'Sh' (ש) sound. Uses Wav2Vec2 forced alignment + acoustic physics to grade pronunciation as CORRECT or INCORRECT in real-time. No reference audio needed — pure AI inference.

Technology Stack

Layer	Technology
AI Model	Wav2Vec2 (imvladikon/wav2vec2-large-xlsr-53-hebrew) — CTC forced alignment
Audio Processing	librosa, torchaudio, scipy — spectral analysis, bandpass filtering
Backend	FastAPI + Uvicorn
Deployment	Hugging Face Spaces (Docker SDK), PyTorch CPU-only
Clinical Engine	OpenAI GPT-4o Structured Outputs — diagnostic report generation
Frontend	Expo React Native app (separate repo)

How the Algorithm Works

Processing Pipeline

Recording → Silence Trimming → Wav2Vec2 Forced Alignment → ש Segment Extraction
         → Bandpass Filter (3-8kHz) → Spectral Centroid → Binary Diagnosis

1. Silence Trimming — Removes leading silence (librosa.effects.trim, top_db=25) so the AI model sees speech immediately. Mobile recordings often have 1-2s of dead air.

2. Wav2Vec2 Forced Alignment — CTC forced alignment (torchaudio.functional.forced_align) maps each Hebrew character to a time span + confidence score. The model identifies exactly where the ש sound starts.

3. Segment Extraction — Extracts 120ms forward from the Wav2Vec2 onset of ש. Forward-only extraction avoids capturing pre-speech silence. Falls back to energy-peak scanning if alignment fails.

4. Bandpass Filter — 4th-order Butterworth bandpass (3-8kHz) isolates the fricative frequency range.

5. Spectral Centroid — Computes the frequency center of gravity on the bandpassed audio. This is device-invariant — doesn't change with recording volume or distance.

6. Binary Diagnosis — Two gates, both must pass:

Diagnosis Gates

Gate	Threshold	What it catches
AI Score	Wav2Vec2 confidence ≥ 0.70	Lisps, substitutions, wet CH, omission
Centroid	2700–5500 Hz	Too low = wet/lateral distortion; too high = S substitution

Result: CORRECT (score 100) or INCORRECT (score 0).

Key Techniques

• CTC Forced Alignment: Aligns Hebrew characters to audio frames using Connectionist Temporal Classification, providing per-character timestamps and confidence scores
• Spectral Centroid Analysis: Frequency center of gravity — robust across devices, volumes, and recording distances (unlike energy-based measures)
• Forward-Only Extraction: Always extract audio forward from the detected onset, never backward into silence
• Leading Silence Trimming: Prevents alignment failure caused by long pre-speech silence on mobile recordings
• Model Warmup: Pre-loads Wav2Vec2 at startup via FastAPI lifespan to eliminate cold-start latency (~10s warmup vs ~2min on first request)

API Endpoints

POST /evaluate

Score a pronunciation recording.

• file: Audio file (WAV, M4A, AAC, WebM, etc.)
• word: Target word key (shalom, shemesh, shir, shuk, geshem, shshshsh, shaon, shulchan)
• Returns: { score, status, diagnosis, feedback, details, evidence }

POST /diagnose_profile

Generate a clinical diagnostic report via GPT-4o.

• Body: Intake questionnaire (child age, error patterns, history)
• Returns: Clinical hypothesis, intervention recommendation, exercises, app roadmap

Supported Words

שלום (shalom), שמש (shemesh), שיר (shir), שוק (shuk), גשם (geshem), שששש (shshshsh), שעון (shaon), שולחן (shulchan)

Project Structure

speechkid-api/
├── score_engine.py       # Main scoring pipeline — trimming, extraction, diagnosis
├── phoneme_extractor.py  # Wav2Vec2 forced alignment & ש isolation
├── main.py               # FastAPI backend (HF Spaces deployment)
├── simulate_remote.py    # Terminal simulator — records mic, calls HF API
├── simulate.py           # Local terminal simulator — scores without API
├── calibrate.py          # Batch calibration tool
├── Dockerfile            # HF Spaces Docker deployment
├── requirements.txt      # Python dependencies
└── CLAUDE.md             # AI assistant context file

Quick Start

# Install
pip install -r requirements.txt

# Test via terminal (calls HF Spaces API)
python simulate_remote.py

# Score a single file locally
python score_engine.py "recording.wav" shalom --pretty

Deployment

Deployed on Hugging Face Spaces (Docker SDK) at: https://yoavzamir-speechkid-api.hf.space

License

MIT License