README.md

---
title: Speechkid API
emoji: 🎙️
colorFrom: blue
colorTo: purple
sdk: docker
pinned: false
---
# SpeechKid API - Hebrew 'Sh' Sound Pronunciation Trainer

AI-powered pronunciation trainer for children learning the Hebrew 'Sh' (ש) sound. Uses Wav2Vec2 forced alignment + acoustic physics to grade pronunciation as CORRECT or INCORRECT in real-time. No reference audio needed — pure AI inference.

## Technology Stack

| Layer | Technology |
|-------|-----------|
| **AI Model** | Wav2Vec2 (`imvladikon/wav2vec2-large-xlsr-53-hebrew`) — CTC forced alignment |
| **Audio Processing** | librosa, torchaudio, scipy — spectral analysis, bandpass filtering |
| **Backend** | FastAPI + Uvicorn |
| **Deployment** | Hugging Face Spaces (Docker SDK), PyTorch CPU-only |
| **Clinical Engine** | OpenAI GPT-4o Structured Outputs — diagnostic report generation |
| **Frontend** | Expo React Native app (separate repo) |

## How the Algorithm Works

### Processing Pipeline

```
Recording → Silence Trimming → Wav2Vec2 Forced Alignment → ש Segment Extraction
         → Bandpass Filter (3-8kHz) → Spectral Centroid → Binary Diagnosis
```

1. **Silence Trimming** — Removes leading silence (`librosa.effects.trim`, top_db=25) so the AI model sees speech immediately. Mobile recordings often have 1-2s of dead air.

2. **Wav2Vec2 Forced Alignment** — CTC forced alignment (`torchaudio.functional.forced_align`) maps each Hebrew character to a time span + confidence score. The model identifies exactly where the ש sound starts.

3. **Segment Extraction** — Extracts 120ms **forward** from the Wav2Vec2 onset of ש. Forward-only extraction avoids capturing pre-speech silence. Falls back to energy-peak scanning if alignment fails.

4. **Bandpass Filter** — 4th-order Butterworth bandpass (3-8kHz) isolates the fricative frequency range.

5. **Spectral Centroid** — Computes the frequency center of gravity on the bandpassed audio. This is device-invariant — doesn't change with recording volume or distance.

6. **Binary Diagnosis** — Two gates, both must pass:

### Diagnosis Gates

| Gate | Threshold | What it catches |
|------|-----------|-----------------|
| **AI Score** | Wav2Vec2 confidence ≥ 0.70 | Lisps, substitutions, wet CH, omission |
| **Centroid** | 2700–5500 Hz | Too low = wet/lateral distortion; too high = S substitution |

Result: **CORRECT** (score 100) or **INCORRECT** (score 0).

### Key Techniques

- **CTC Forced Alignment**: Aligns Hebrew characters to audio frames using Connectionist Temporal Classification, providing per-character timestamps and confidence scores
- **Spectral Centroid Analysis**: Frequency center of gravity — robust across devices, volumes, and recording distances (unlike energy-based measures)
- **Forward-Only Extraction**: Always extract audio forward from the detected onset, never backward into silence
- **Leading Silence Trimming**: Prevents alignment failure caused by long pre-speech silence on mobile recordings
- **Model Warmup**: Pre-loads Wav2Vec2 at startup via FastAPI lifespan to eliminate cold-start latency (~10s warmup vs ~2min on first request)

## API Endpoints

### `POST /evaluate`
Score a pronunciation recording.
- **file**: Audio file (WAV, M4A, AAC, WebM, etc.)
- **word**: Target word key (shalom, shemesh, shir, shuk, geshem, shshshsh, shaon, shulchan)
- **Returns**: `{ score, status, diagnosis, feedback, details, evidence }`

### `POST /diagnose_profile`
Generate a clinical diagnostic report via GPT-4o.
- **Body**: Intake questionnaire (child age, error patterns, history)
- **Returns**: Clinical hypothesis, intervention recommendation, exercises, app roadmap

## Supported Words

שלום (shalom), שמש (shemesh), שיר (shir), שוק (shuk), גשם (geshem), שששש (shshshsh), שעון (shaon), שולחן (shulchan)

## Project Structure

```
speechkid-api/
├── score_engine.py       # Main scoring pipeline — trimming, extraction, diagnosis
├── phoneme_extractor.py  # Wav2Vec2 forced alignment & ש isolation
├── main.py               # FastAPI backend (HF Spaces deployment)
├── simulate_remote.py    # Terminal simulator — records mic, calls HF API
├── simulate.py           # Local terminal simulator — scores without API
├── calibrate.py          # Batch calibration tool
├── Dockerfile            # HF Spaces Docker deployment
├── requirements.txt      # Python dependencies
└── CLAUDE.md             # AI assistant context file
```

## Quick Start

```bash
# Install
pip install -r requirements.txt

# Test via terminal (calls HF Spaces API)
python simulate_remote.py

# Score a single file locally
python score_engine.py "recording.wav" shalom --pretty
```

## Deployment

Deployed on **Hugging Face Spaces** (Docker SDK) at:
`https://yoavzamir-speechkid-api.hf.space`

## License

MIT License