---
title: AudioShield AI
emoji: ๐ก๏ธ
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
---
# ๐ก๏ธ AudioShield AI
### Enterprise-Grade Deepfake Audio Detection & Forensic Analysis Platform
**Detect synthetic voices with 99.11% accuracy using self-supervised speech representations, temporal sequence modeling, and attention-based forensic reasoning.**
[](https://www.python.org/)
[](https://pytorch.org/)
[](https://huggingface.co/docs/transformers/model_doc/wavlm)
[](https://fastapi.tiangolo.com/)
[](https://react.dev/)
[](https://www.docker.com/)
[](https://hardik-25-audioshield.hf.space/)
[](https://hardik-25-audioshield.hf.space/)
[](https://hardik-25-audioshield.hf.space/)
[](LICENSE)
**[๐ Live Demo](https://hardik-25-audioshield.hf.space/)** ยท **[๐ Performance](#-diagnostic-performance-metrics)** ยท **[๐๏ธ Architecture](#๏ธ-system-architecture)**
---
## ๐ก Why AudioShield AI?
Voice cloning technology has crossed a threshold. Tools that once required studios and hours of training data can now clone a voice from seconds of audio โ and the output is good enough to fool the human ear. This isn't a hypothetical risk anymore; it's actively being used in financial fraud, impersonation scams, and disinformation.
Most academic deepfake detectors stop at a Jupyter notebook and an accuracy score. **AudioShield AI doesn't.** It's built as a complete, deployable forensic product:
- ๐ฏ A production-trained model evaluated with rigorous, leak-free methodology
- ๐ฅ๏ธ A full-stack web application with a real inference backend, not a mock UI
- ๐ An investigation-grade dashboard that explains *why* a clip was flagged, not just *what* it was flagged as
- ๐ Exportable forensic reports suitable for documentation and review
The goal was to build something closer to a real security tool than a portfolio script โ from raw waveform to actionable forensic verdict.
---
## โจ Key Features
| | |
|---|---|
| ๐ **Real-time Deepfake Detection** | Classifies uploaded audio as Real or Fake in under a second |
| ๐ง **WavLM Transformer Backbone** | Self-supervised speech foundation model captures prosody, speaker identity, and synthetic artifacts |
| ๐ **BiLSTM Temporal Modeling** | Captures sequential dependencies across the embedding sequence in both directions |
| ๐ฏ **Attention Pooling** | Learns to weight the most forensically relevant frames instead of naive averaging |
| ๐ **Confidence & Risk Scoring** | Authenticity score, risk assessment (Low/Medium/High/Critical), and calibrated probability |
| ๐บ๏ธ **Chronological Suspicion Heatmap** | Segment-by-segment breakdown showing exactly where a clip deviates from organic speech patterns |
| ๐ง **Interactive Waveform Diagnostics** | Playback with suspicious regions highlighted directly on the oscilloscope view |
| ๐น๏ธ **Forensic Auditory Playground** | Blind-listening challenge โ test your ear against the model on real/fake samples |
| ๐ **Exportable PDF Reports** | One-click forensic investigation reports for documentation and review |
| ๐๏ธ **Batch Verification Logs** | Audit trail of historical classifications with filterable risk levels |
| โก **FastAPI Inference Backend** | Model loads once at startup; serves predictions via a real API, not a notebook loop |
| ๐ณ **Dockerized Deployment** | Single-container deployment to Hugging Face Spaces, reproducible anywhere |
---
## ๐๏ธ System Architecture
### End-to-End Request Flow
```
โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโ
โ User โโโโโโถโ React Frontend โโโโโโถโ FastAPI Backend โ
โ (Browser) โ โ (Vite + Tailwind)โ โ (Uvicorn) โ
โโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโฌโโโโโโโโโ
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Audio Preprocessing โ
โ 16kHz ยท Mono ยท Normalizeโ
โโโโโโโโโโโโโโฌโโโโโโโโโโโโโโ
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ deployment_model.pt โ
โ (loaded once at startup)โ
โโโโโโโโโโโโโโฌโโโโโโโโโโโโโโ
โผ
Prediction + Confidence
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ JSON Response โ React โ
โ Forensic Dashboard โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโ
```
### Model Pipeline Architecture
```
Audio Input WavLM Base+ BiLSTM Attention Pooling Classifier
(WAV/MP3/FLAC) โโโถ SSL Representation โโโถ Temporal โโโถ Feature Weights โโโถ Softmax
(Contextual Embeddings) Context (Real / Fake)
```
| Stage | Role |
|---|---|
| **Preprocessing** | Resamples to 16kHz, converts to mono, normalizes amplitude, handles variable-length audio via padding/truncation |
| **WavLM Base+** | Microsoft's self-supervised speech foundation model โ extracts rich contextual embeddings capturing speaker traits, prosody, and synthetic artifacts |
| **BiLSTM** | Models temporal dependencies across the embedding sequence in both forward and backward directions |
| **Attention Pooling** | Learns to weight the most discriminative frames rather than pooling naively, improving sensitivity to localized synthetic artifacts |
| **Classifier** | Fully connected layer with dropout + GELU activation, outputting a calibrated Real/Fake probability |
> Pre-trained self-supervised speech representations provide strong zero-shot generalization to unseen deepfake generation methods โ a key reason this architecture outperforms handcrafted-feature baselines (MFCC, Chroma, Spectrogram).
---
## ๐งฐ Tech Stack
|
**Machine Learning**
- Python
- PyTorch
- Hugging Face Transformers
- WavLM Base+
- NumPy ยท Pandas
|
**Audio Processing**
- Librosa
- SoundFile
|
**Frontend**
- React
- Vite
- Tailwind CSS
|
**Backend & Deployment**
- FastAPI
- Uvicorn
- Docker
- Hugging Face Spaces
|
---
## ๐ Dataset
Trained and evaluated on a **combined dataset** merging **Fake-or-Real (FoR)** and **ASVspoof 2019 LA** โ **162,374** total samples.
| Dataset | Samples |
|---|---:|
| ASVspoof 2019 LA | 121,461 |
| Fake-or-Real (FoR) | 40,913 |
| **Combined Total** | **162,374** |
| Split | Samples |
|---:|---:|
| Training | 55,584 |
| Validation | 30,919 |
| Testing | 75,871 |
The test set (75,871 samples) combines 71,237 ASVspoof + 4,634 FoR samples, providing a rigorous held-out evaluation across diverse attack types and recording conditions.
---
## ๐ฌ Audio Preprocessing
Every recording is standardized before feature extraction:
1. **Loading** โ via Librosa / SoundFile
2. **Resampling** โ converted to 16kHz (WavLM's expected input rate)
3. **Mono conversion** โ removes channel-based variance
4. **Normalization** โ amplitude-normalized so volume doesn't bias predictions
5. **Padding / Truncation** โ handles variable-length clips during batching (max 96,000 samples / 6 sec)
---
## ๐ง Feature Extraction โ Why WavLM?
Traditional handcrafted features (MFCC, Chroma, Spectrograms) were deliberately avoided in favor of **WavLM Base+**, Microsoft's self-supervised speech foundation model.
```
Raw Audio โ WavLM Processor โ Transformer Encoder โ Contextual Embeddings โ BiLSTM โ Attention Pooling โ Fixed Vector
```
**Why this matters:** WavLM is pretrained on massive unlabeled speech corpora, learning representations of speaker characteristics, prosody, and speech dynamics that generalize far better than hand-engineered spectral features โ particularly against unseen deepfake generation methods, which is the failure mode that sinks most academic detectors in the real world.
---
## ๐๏ธ Training Pipeline
- **Loss:** Focal Loss (ฮฑ=0.25, ฮณ=2.0)
- **Optimizer:** AdamW (lr=1ร10โปโต, weight_decay=0.01)
- **Scheduler:** Cosine with linear warmup (10% warmup steps)
- **Batch Size:** 32
- **Max Audio Length:** 6 seconds (96,000 samples at 16kHz)
- **Epochs:** 7 (early stopping based on AUC improvement)
- **Mixed Precision:** FP16 via GradScaler
- **Freeze Strategy:** WavLM frozen for first 2 epochs, then unfrozen for fine-tuning
- **Output:** `deployment_model.pt` โ the exact artifact served in production
The combined dataset was assembled by merging FoR and ASVspoof 2019 LA, then splitting into train/validation/test sets. Training was performed on Kaggle with GPU acceleration.
---
## ๐ Diagnostic Performance Metrics
*Evaluated on the combined held-out test set (75,871 samples: 71,237 ASVspoof + 4,634 FoR).*
| Metric | Overall | Real (Bonafide) | Spoof (Fake) |
|---|---:|---:|---:|
| **Accuracy** | **99.11%** | โ | โ |
| **Precision** | โ | 94.14% | 99.87% |
| **Recall** | โ | 99.12% | 99.10% |
| **F1 Score** | โ | 96.57% | 99.49% |
| **ROC-AUC** | **0.9994** | โ | โ |
| **Equal Error Rate (EER)** | **0.89%** | โ | โ |
---
## โ
Why These Results Are Trustworthy
High numbers invite skepticism โ rightfully so. Here's why this isn't overfitting:
- **No data leakage** โ splits were created before training; test data was never seen during training or model selection.
- **Large, diverse test set** โ 75,871 held-out samples spanning two independent datasets with different attack types ensures robust evaluation.
- **Large training set** โ 55,584 labeled samples reduce memorization risk and support generalization.
- **Transfer learning, not memorization** โ performance is driven primarily by WavLM's pretrained representations, not example-level memorization.
- **Consistent validation/test alignment** โ closely matched performance across splits indicates genuine generalization rather than overfitting to validation.
---
## ๐ Combined Dataset Evaluation
The model was trained on a combined dataset merging **Fake-or-Real (FoR)** and **ASVspoof 2019 LA**, then evaluated on a held-out test set from the same combined distribution.
### Evaluation Methodology
| Component | Detail |
|---|---|
| **Training Datasets** | FoR + ASVspoof 2019 LA (combined) |
| **Total Samples** | 162,374 (55,584 train / 30,919 val / 75,871 test) |
| **Model Checkpoint** | `deployment_model.pt` โ best epoch by validation AUC |
| **Preprocessing** | 16kHz resample, mono, head truncation to 6s, WavLM feature extraction |
| **Evaluation Type** | In-distribution held-out test |
### Per-Class Metrics
| Class | Precision | Recall | F1-Score | Support |
|---|---:|---:|---:|---:|
| **Real (Bonafide)** | 94.14% | 99.12% | 96.57% | 9,619 |
| **Spoof (Fake)** | 99.87% | 99.10% | 99.49% | 66,252 |
| **Overall Accuracy** | **99.11%** | | | **75,871** |
The model achieves near-perfect spoof detection (99.87% precision, 99.10% recall) with a slight trade-off in real-class precision (94.14%), meaning it occasionally flags real audio as suspicious โ a conservative bias that favors security over false negatives.
### How to Reproduce
```bash
cd combined_dataset_evaluation
python run_inference_for.py # Run FoR inference
python compute_metrics_for.py # Compute metrics + visualizations
```
See the [`model evaluation`](model%20evaluation/) directory for full metrics, confusion matrix, ROC/PR curves, and score distributions.
---
## ๐ฅ๏ธ Web Application
### Forensic Sandbox
Drag-and-drop upload with reference benchmark cases pulled from the FoR and ASVspoof 2019 LA datasets, plus live duration/sample-rate/inference-time readouts per file.
### Investigation Dashboard โ Analysis Overview
Authenticity score, detection profile, model backbone, investigation findings, confidence distribution curve, and an acoustic anomaly radar chart (Spectral Inconsistency, Vocoder Footprint, Phase Coherence, Breath Mark Gaps, Jitter/Shimmer Ratio).
### Forensic Metrics โ Chronological Suspicion Heatmap
Segment-by-segment breakdown of suspicion scores across the clip timeline, paired with an interactive waveform oscilloscope that highlights suspicious regions directly on playback.
### Investigation Report
One-click downloadable forensic PDF report โ audio summary, key findings, and full timeline analysis, ready for documentation or review.
### Model Pipeline & Diagnostic Metrics
---
## ๐ Repository Structure
```
Audio-Shield-AI/
โ
โโโ combined_dataset_evaluation/ # In-distribution evaluation on combined dataset
โ โโโ download_asvspoof.py # ASVspoof 2019 LA download script
โ โโโ run_inference.py # Pure inference on ASVspoof
โ โโโ compute_metrics.py # Metrics + visualizations
โ โโโ error_analysis.py # FP/FN + attack-type analysis
โ โโโ generalization_report.py # Honest generalization report
โ โโโ run_all.py # Orchestrator
โ โโโ asvspoof_data/ # Dataset (downloaded)
โ โโโ results/ # Metrics + CSVs
โ โโโ figures/ # ROC, PR, confusion matrix, etc.
โ
โโโ notebooks/
โ โโโ 1_eda-data-validation.ipynb
โ โโโ 2_preprocessing.ipynb
โ โโโ 3_Model.ipynb
โ โโโ 4_Evaluation.ipynb
โ โโโ 5_inference.ipynb
โ
โโโ audioshield-app/ # React frontend source (Vite)
โ โโโ src/
โ โโโ App.tsx
โ โโโ index.css
โ
โโโ static/ # Built frontend (served by FastAPI)
โโโ assets/ # Screenshots and images
โโโ deployment_model.pt # Trained model checkpoint
โโโ app.py # FastAPI server
โโโ Dockerfile
โโโ requirements.txt
โโโ .gitignore
โโโ README.md
```
---
## ๐ Getting Started
### Reproduce Training (Kaggle)
The notebooks were developed entirely on **Kaggle** to take advantage of free GPU access and the dataset's native Kaggle hosting โ avoiding a ~70,000-file local download and giving anyone reproducing this work a zero-setup environment.
1. Open `notebooks/1_eda-data-validation.ipynb` in a new Kaggle Notebook
2. Attach the Fake-or-Real Audio Dataset
3. Enable GPU acceleration (Settings โ Accelerator โ GPU)
4. Run notebooks sequentially: `1 โ 2 โ 3 โ 4`
### Local Deployment
**Clone the repository**
```bash
git clone https://github.com/gautamhardik/Audio-Shield-AI.git
cd Audio-Shield-AI
```
**Backend**
```bash
pip install -r requirements.txt
uvicorn app:app --reload --port 7860
```
**Frontend (development)**
```bash
cd audioshield-app
npm install
npm run dev
```
**Docker (full stack)**
```bash
docker build -t audioshield-ai .
docker run -p 7860:7860 audioshield-ai
```
### Hugging Face Spaces Deployment
The production app runs as a **Docker Space** on Hugging Face, bundling the React frontend, FastAPI backend, and `deployment_model.pt` into a single reproducible container with a public URL:
**๐ [hardik-25-audioshield.hf.space](https://hardik-25-audioshield.hf.space/)**
---
## ๐ฎ Future Work
- ๐ Multilingual deepfake detection
- ๐๏ธ Streaming / real-time inference for live calls
- ๐ Explainable AI โ frame-level saliency visualization
- ๐ก๏ธ Adversarial robustness testing against evasion attacks
- ๐ฑ Mobile deployment (on-device inference)
- ๐ฆ Batch analysis API for bulk investigations
- ๐ฃ๏ธ Speaker attribution / voice fingerprinting
---
## ๐ฌ Model Evaluation
All evaluation code, results, metrics, and visualizations are available in the [`combined_dataset_evaluation/`](combined_dataset_evaluation/) and [`model evaluation`](model%20evaluation/) directories.
---
## ๐ Acknowledgements
- **Microsoft** โ for WavLM and the broader speech-SSL research that made this possible
- **Hugging Face** โ Transformers library and Spaces hosting
- **PyTorch** team
- **Kaggle** โ free GPU compute and dataset hosting
- Creators of the **Fake-or-Real** and **ASVspoof** datasets
---
## ๐ License
Distributed under the **MIT License**. See [`LICENSE`](LICENSE) for details.
---
### ๐ก๏ธ AudioShield AI
*Enterprise Deepfake Audio Detection Platform*
**[Live Demo](https://hardik-25-audioshield.hf.space/)**
If this project was useful or interesting, consider โญ starring the repo.