README.md

metadata

title: Multilingual ASR
emoji: 🎙️
colorFrom: blue
colorTo: indigo
sdk: gradio
app_file: gradio_ui.py
pinned: false

🎙️ Multilingual Automatic Speech Recognition (ASR)

Live Demo: Hugging Face Space · Landing Page: Vercel

📌 Project Overview & Importance

An end-to-end Automatic Speech Recognition (ASR) pipeline that converts raw unstructured audio into transcribed text. ASR is notoriously difficult due to variability in audio data — background noise, hardware differences, varied accents, and speech speeds.

This project bridges Deep Learning model inference with full-stack software engineering, deploying a production-ready application that handles digital signal processing, neural network inference, and a user-facing REST API — all served from a single Python process.

⚙️ How It Works (The Pipeline)

1. Audio Ingestion & DSP: Raw audio (.mp3, .wav) is loaded via librosa and resampled to 16kHz — the exact rate the model was trained on.
2. Feature Extraction: The Wav2Vec2 Processor normalizes the waveform into padded PyTorch tensors.
3. Acoustic Model Inference: Wav2Vec2ForCTC (1.26GB, Large architecture) runs the forward pass using self-supervised learned speech representations.
4. CTC Decoding: Connectionist Temporal Classification decodes raw logits into the most probable character sequence.
5. Output & Storage: The transcript is persisted in local JSON history, downloadable as .txt, with full history exportable as .csv.

🛠️ Technology Stack

Layer	Technology
Acoustic Model	facebook/wav2vec2-large-960h-lv60-self
Deep Learning	PyTorch (CPU-forced inference)
DSP	Librosa
Backend API	FastAPI + Uvicorn
ML UI	Gradio (Tabbed Blocks)
Language Detection	LangDetect
Landing Page	React + Vite
ML Deployment	Hugging Face Spaces
Web Deployment	Vercel
CI/CD	GitHub Actions

🗂️ Project Structure

├── app/
│   ├── asr_model.py        # Wav2Vec2 model loading & inference (lazy-loaded)
│   ├── audio_processing.py # Librosa resampling to 16kHz
│   └── history.py          # JSON persistence, CSV/TXT export
├── landing page/           # React + Vite landing page (deployed to Vercel)
│   └── src/
│       ├── components/     # Nav, Hero, HowItWorks, TechStack, About, Footer
│       └── index.css
├── notebooks/
│   ├── 01_evaluation.ipynb # WER evaluation template
│   └── 02_finetuning.ipynb # Fine-tuning notebook (Colab-ready)
├── .github/workflows/
│   └── sync_to_hub.yml     # Auto-deploys to Hugging Face on every git push
├── gradio_ui.py            # Gradio Tabbed UI (Transcribe + History tabs)
├── main.py                 # FastAPI entry point, mounts Gradio at "/"
└── requirements.txt

🧠 Interview Talking Points (Key Technical Decisions)

1. Why Wav2Vec 2.0?

Traditional ASR models require massive amounts of perfectly transcribed audio. Wav2Vec 2.0 uses Self-Supervised Learning — it learns from raw, unlabeled audio by masking parts of speech and predicting the missing content (similar to BERT for text). This makes it highly accurate even when fine-tuning data is scarce.

2. Handling Apple Silicon Hardware Constraints

During development on M1 Mac, model inference hung indefinitely. I debugged this to a PyTorch limitation: the mps backend lacks support for CTC operations used by Wav2Vec. Solution: Hardware-fallback in asr_model.py forces CPU execution, prioritizing stability over theoretical GPU speed.

3. Lazy Loading Pattern

Loading a 1.26GB model on server boot blocks FastAPI's main thread and causes timeouts. Solution: The model is loaded only on the first transcription request. Server boot time stays under 1 second regardless of model size.

4. Unified Server Architecture

Rather than running two separate processes, the Gradio UI is mounted directly onto the FastAPI app (app.mount("/", gr.routes.App.create_app(demo))). One uvicorn process serves both the REST API and the interactive UI.

5. Dual CI/CD Pipelines

• ML Backend: sync_to_hub.yml (GitHub Actions) auto-deploys to Hugging Face Spaces on every push to main, using a scoped HF_TOKEN secret.
• Frontend: Vercel's GitHub integration auto-builds and deploys the React landing page on every push, with the landing page/ subfolder set as the root directory.