docs/architecture_overview.md

Architecture Overview: Phoneme Pronunciation Correction System

1. System Philosophy

The system is designed to provide high-fidelity, phoneme-level pronunciation feedback using a modern embedding-based acoustic model. It prioritizes data efficiency (streaming), session resiliency (checkpoint syncing), and granular feedback (temporal alignment).

2. Model Architecture (Design A: Contrastive CTC)

The core model is a modified Wav2Vec2 architecture that uses a Cosine Similarity Embedding Head instead of a standard linear classification layer.

Acoustic Encoder

• Base: Wav2Vec2Model (e.g., facebook/wav2vec2-base).
• Input: 16kHz Raw Audio.
• Output: Temporal sequence of 768-dimensional hidden states (frames).

Embedding Head

• Audio Projection: Linear layer mapping hidden states to a shared embedding space (e.g., 256 or 768 dimensions).
• Phoneme Dictionary: A learnable nn.Embedding matrix containing a unique vector for every phoneme in the vocabulary + a CTC blank token.
• Matching Mechanism: L2-Normalization followed by Matrix Multiplication (Cosine Similarity).
• Logit Scaling: A learnable temperature parameter to adjust the sharpness of the probability distribution.

Optimization

• Loss Function: CTCLoss applied to the similarity logits.
• Rationale: Learns a robust acoustic-phoneme mapping that generalizes better across diverse accents than simple classification.

---

3. Data & Training Pipeline

Engineered to handle the NPTEL2020 dataset within strict compute/storage constraints.

Streaming Data Loader

• Source: Hugging Face Hub (skbose/indian-english-nptel-v0) in streaming=True mode.
• Workflow: Pulls shards from HF Hub, disables automatic torchcodec decoding to prevent driver incompatibilities, manually decodes raw bytes with soundfile, preprocesses audio, and discards raw data immediately.
• Advantages: Neatly handles the 130GB+ dataset with proper interleaving without multi-TB downloading or RAM exhaustion. No local disk usage required.

Session Resiliency (24-Hour Loop)

• Checkpointer: Integrated with Hugging Face Hub.
• Rotation: save_total_limit=2 to keep disk usage under 50GB.
• Auto-Sync: push_to_hub=True pushes weights to the cloud in the background.
• Resume Logic: Training logic checks for existing checkpoints on the Hub at startup.

---

4. Pronunciation Analysis Engine

Translates model predictions into actionable user feedback.

Temporal Alignment

• Acoustic Transcription: The model predicts a sequence of phonemes over time.
• Reference Mapping: The target word is converted to phonemes via dictionary lookup.
• Alignment (Levenshtein/DTW): Dynamically aligns the prediction against the reference to find:
  • Matches: Correct pronunciation.
  • Substitutions: Incorrect sound used.
  • Deletions: Sound skipped.
  • Insertions: Extra sound added.

Scoring Logic

• Accuracy: Percentage of correctly produced phonemes.
• Duration/Timing: Comparison of phoneme durations against typical distributions.
• Feedback: Visual mapping showing exactly which phonemes were missed or incorrect.

---

5. Technology Stack

• Languages: Python
• Frameworks: PyTorch, Hugging Face (Transformers, Datasets, Accelerate)
• Acoustic Processing: Librosa, Sounddevice, Torchaudio
• Hardware Optimization: CUDA, bfloat16 (H100 optimized)