File size: 19,602 Bytes
a48b1ab 6046513 a48b1ab e6a1f55 a48b1ab e6a1f55 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 | ---
title: Audio Deepfake Detector
emoji: ποΈ
colorFrom: blue
colorTo: red
sdk: docker
app_port: 7860
pinned: false
license: mit
short_description: Audio deepfake detection with eight models
---
# ποΈ Audio Deepfake Detector
> **Production-grade audio deepfake detection with eight detectors, paper-grounded benchmarks, real fine-tuned weights, and a forensic UI.**
>
> Companion to *"The Generalization Gap in Audio Deepfake Detection: A Four-Paradigm Review"* β Vipan Kumar, Chitkara University, 2025.
[]() []() []() []() []()
---
## What this is
A full-stack web app that runs **real audio deepfake detection** on uploaded clips, microphone recordings, and a curated sample library. Eight detectors run in parallel against every input, and a fitted logistic-regression meta-classifier combines them into a single calibrated `P(fake)` verdict.
### Detectors
| ID | Display name | Paradigm | Source |
|----|--------------|----------|--------|
| `melodymachine` | MelodyMachine Wav2Vec2 | SSL Backend (production) | [HuggingFace public model](https://huggingface.co/MelodyMachine/Deepfake-audio-detection) |
| `motheecreator` | Motheecreator Wav2Vec2 | SSL Backend (production) | [HuggingFace public model](https://huggingface.co/motheecreator/Deepfake-audio-detection) |
| `lf_hf_physics` | LF/HF Physics Analyzer | Physics-Augmented (deterministic) | self-contained; multi-signal DSP β Pearson r, JS divergence, HF energy variance, voicing-aligned correlation, spectral roll-off |
| `nes2net` | Nes2Net | Efficient Backend | XLS-R + nested Res2Net (no DR layer) β **trained in-repo** |
| `sonar` | SONAR | Physics-Augmented | Dual XLS-R + SRM filters + JS contrastive head β **trained in-repo** |
| `bicrossmamba_st` | BiCrossMamba-ST | State-Space | Bidirectional Mamba + Mutual Cross-Attention β **trained in-repo** |
| `voiceradar` | VoiceRadar | Physics + Adv-Robust | Mel-CNN + physics aux head + PGD Ξ΅-stability probe β **trained in-repo** |
| `holi_antispoof` | HoliAntiSpoof | Audio LLM (explainable) | Qwen2.5-Omni stub + rule-based natural-language rationale generator |
The four "paper architecture" detectors ship with **real trained weights** (val_EER 0.5β5%) and switch to live mode automatically when the checkpoint files are present in `backend/checkpoints/`. If a checkpoint is missing, the detector falls back transparently to a paired live detector and labels the result `architecture_only_fallback` so nothing is silently faked.
---
## Quick start
### Prerequisites
- **Python 3.10 or 3.11** (3.12 should work but is less battle-tested for the torch wheels)
- **Node.js 18+** with `npm`
- ~2 GB free disk for the HuggingFace cache (Wav2Vec2 weights + XLS-R)
- Optional: NVIDIA GPU with CUDA 11.8+ for faster inference (CPU-only also works)
### Run locally β Windows (PowerShell)
```powershell
# 1) Clone & enter
git clone <this-repo> audio-deepfake-detector
cd audio-deepfake-detector
# 2) Backend β create venv + install deps
cd backend
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install --upgrade pip
pip install -r requirements.txt
# 3) Backend β first run (downloads ~750 MB of HF weights into the system HF cache)
$env:ENABLE_HEAVY_MODELS = "true"
python -m uvicorn app.main:app --host 127.0.0.1 --port 8000
# Leave this terminal open. The backend serves at http://127.0.0.1:8000
# 4) Frontend β IN A SECOND TERMINAL
cd <repo-path>\audio-deepfake-detector\frontend
npm install
npm run dev
# Frontend serves at http://localhost:5173
```
Open **http://localhost:5173** and you're done.
### Run locally β macOS / Linux
```bash
git clone <this-repo> audio-deepfake-detector
cd audio-deepfake-detector
# Backend
cd backend
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
ENABLE_HEAVY_MODELS=true uvicorn app.main:app --host 127.0.0.1 --port 8000
# Frontend (second terminal)
cd ../frontend
npm install
npm run dev
```
### Run with Docker
```bash
docker compose up --build
# Frontend β http://localhost:3000
# Backend β http://localhost:8000
```
The first Docker build downloads HF weights into a named volume; subsequent runs are fast.
---
## Deploy to Hugging Face Spaces (free, no credit card)
A combined Dockerfile + `hf_spaces_entrypoint.py` at the repo root build the
React frontend, copy it into the FastAPI container, and serve both on port 7860 (the HF Spaces convention).
1. Push this repo to GitHub first (so HF can clone it).
2. Visit https://huggingface.co/new-space and create a **Docker Space**:
- Owner: your HF username
- Space name: `audio-deepfake-detector` (or anything)
- Space SDK: **Docker**
- Hardware: free CPU is sufficient
- Visibility: Public or Private (your choice)
3. After creation, push the same repo to the Space's git remote:
```bash
git remote add space https://huggingface.co/spaces/<your-user>/<space-name>
git push space main
```
(HF prompts for a write-access token β generate one at
https://huggingface.co/settings/tokens with the *write* scope.)
4. The Space rebuilds and the URL becomes
`https://huggingface.co/spaces/<your-user>/<space-name>` β the
frontend serves from the root, the API at `/api/*`, docs at `/docs`.
The YAML block at the top of this README tells HF to use Docker mode and
expose port 7860; the entrypoint mounts `/app/static` so the SPA + API
share the same origin.
### Other deploy options
- **Backend on Render / Railway / Fly.io**: the existing `backend/Dockerfile`
works on all three; their free tiers may be too small for the XLS-R model
(~1.5 GB resident). Check current memory limits.
- **Frontend on Vercel / Netlify**: deploy `frontend/` as a static site;
point `VITE_API_BASE_URL` at wherever the backend lives.
- **Self-hosted**: `docker compose up --build` on any VPS with β₯ 4 GB RAM.
---
## What you'll see
- **Home (`/`)** β paper context, four-paradigm overview, three-deficits framing, light/dark theme toggle at the bottom.
- **Demo (`/demo`)** β drop a clip / record from mic / pick a sample. Eight detectors run in parallel. Verdict + per-detector cards + waveform + mel spectrogram. Recording supports playback + replay + delete (cleared on refresh).
- **Benchmarks (`/benchmarks`)** β paper Table III data verbatim, generalization-gap chart, datasetΓmodel coverage matrix, PGD adversarial-robustness curve, HoliAntiSpoof rationale examples.
- **About (`/about`)** β full architecture descriptions for all 7 paper models with SVG diagrams, comparative Table II + III.
The whole UI has a **light/dark theme toggle** (Sun/Moon switch at the bottom of Home and in the global footer). Choice persists via `localStorage`.
---
## API reference
Full interactive docs: http://127.0.0.1:8000/docs
| Method | Path | Purpose |
|---|---|---|
| GET | `/api/health` | Service status + per-detector load state |
| GET | `/api/models` | All detectors with paradigm, params, EER refs |
| POST | `/api/detect` | Upload audio (multipart), run all (or selected) detectors, return ensemble verdict |
| POST | `/api/detect/sample/{id}` | Detect on one of the bundled sample clips |
| POST | `/api/visualize` | Waveform envelope + mel spectrogram for the UI |
| GET | `/api/benchmarks` | Paper Table III rows + generalization-gap data |
| GET | `/api/benchmarks/datasets` | Dataset metadata |
| GET | `/api/benchmarks/evaluations` | Dataset Γ model Γ metric matrix |
| GET | `/api/benchmarks/adversarial` | PGD Ξ΅-curve data |
| GET | `/api/benchmarks/explainability` | HoliAntiSpoof rationale examples |
| GET | `/api/benchmarks/radar` | Radar-chart data (5 axes) |
| GET | `/api/benchmarks/highlights` | Paper one-liners |
| GET | `/api/samples` | Sample-library metadata |
| GET | `/api/samples/{id}/download` | Sample WAV bytes |
| WS | `/ws/inference/{session_id}` | Streamed inference progress |
### Example: detect a clip
```bash
curl -F "audio_file=@my_clip.wav" \
-F 'models=["nes2net","sonar","lf_hf_physics"]' \
http://127.0.0.1:8000/api/detect | jq
```
---
## Training the paper architectures
The repo includes three training scripts. All run on CPU; GPU is optional but ~10Γ faster.
### `train_asvspoof.py` β Nes2Net
Three modes, picked automatically from the args:
```powershell
# (1) Synthesised fallback β downloads small public LibriSpeech clips +
# synthesises 5 fake families. Good for first-time setup.
python train_asvspoof.py --n-real 80 --n-fake 80 --epochs 8 --batch-size 8
# (2) Directory mode β point at any dataset organised as real/ and fake/ folders
python train_asvspoof.py `
--real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" `
--fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" `
--max-per-class 1000 --epochs 6 --batch-size 8
# (3) ASVspoof real-data mode β provide protocol + audio dir
python train_asvspoof.py `
--protocol path\to\ASVspoof2019.LA.cm.train.trn.txt `
--audio-dir path\to\ASVspoof2019_LA_train\flac `
--epochs 12 --batch-size 8
```
### `train_paper_archs.py` β SONAR + BiCrossMamba-ST + VoiceRadar
Each architecture has its own hyperparameter profile, picked automatically:
| Architecture | LR | Epochs | Weight decay | Why |
|---|---|---|---|---|
| SONAR | 1e-4 | 5 | 5e-4 | 41M-param dual-stream cross-attention head β heavy regularisation, fewer epochs to avoid overfitting |
| BiCrossMamba-ST | 5e-4 | 10 | 1e-4 | 516K param Mamba+MCA β small model, tolerates higher LR + more epochs |
| VoiceRadar | 3e-4 | 10 | 2e-4 | 410K-param CNN β standard LR, modest weight decay |
```powershell
# Synthesised fallback
python train_paper_archs.py --n-real 80 --n-fake 80 --batch-size 8
# Directory mode (e.g. jay15k)
python train_paper_archs.py `
--real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" `
--fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" `
--max-per-class 1000 --batch-size 8
# Train only one architecture
python train_paper_archs.py --only sonar --batch-size 8
```
### `train_calibration.py` β meta-classifier (logistic regression on top of detector outputs)
Run this AFTER training the paper architectures (or any time the detector mix changes):
```powershell
# Synthesised pool
python train_calibration.py --n-real 30 --n-fake 30
# Directory mode
python train_calibration.py `
--real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" `
--fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" `
--max-per-class 100
```
Output: `backend/checkpoints/calibration.json`. Loaded automatically on next backend startup.
### Datasets
- **ASVspoof 2019 LA** β ~5 GB, click-through registration: https://datashare.ed.ac.uk/handle/10283/3336
- **ASVspoof 2021 DF** β ~133 GB, challenge registration: https://www.asvspoof.org/
- **In-the-Wild (MΓΌller et al. 2022)** β 37.9 h: https://deepfake-total.com/in_the_wild
- **deepfake_audio_dataset_jay15k** β 9.6K real + 7.2K fake (16 kHz mono), available on Kaggle / similar mirrors. Verified working with `train_asvspoof.py --real-dir / --fake-dir`.
---
## Configuration
All env vars (also readable from a `backend/.env` file):
| Variable | Default | Effect |
|---|---|---|
| `ALLOWED_ORIGINS` | `http://localhost:5173,http://localhost:3000,...` | CORS origins (CSV) |
| `ENABLE_HEAVY_MODELS` | `true` | Disable to skip the two ~94 M-param Wav2Vec2 detectors. The LF/HF physics detector + paper architectures still run. |
| `DEMO_MODE` | `false` | Reserved flag |
| `HF_HOME` / `TRANSFORMERS_CACHE` | `/app/.cache/huggingface` | HuggingFace cache location |
Frontend env (Vite):
| Variable | Default | Effect |
|---|---|---|
| `VITE_API_BASE_URL` | `http://localhost:8000` | Backend HTTP base |
| `VITE_WS_BASE_URL` | `ws://localhost:8000` | Backend WS base |
---
## Project layout
```
audio-deepfake-detector/
βββ backend/
β βββ app/
β β βββ main.py FastAPI app + lifespan + CORS
β β βββ config.py Settings via pydantic-settings
β β βββ schemas/ Pydantic request/response models
β β βββ features/ Audio decode (incl. ffmpeg fallback for WebM/Opus),
β β β XLS-R extractor, SRM filters, mel utilities,
β β β multi-signal LF/HF physics analyser
β β βββ models/
β β β βββ base_model.py BaseDetector + per-request FeatureCache
β β β βββ hf_detectors.py MelodyMachine + motheecreator (calibrated)
β β β βββ lf_hf_detector.py Deterministic physics analyser
β β β βββ nes2net.py Nes2Net architecture + fallback wiring
β β β βββ sonar.py SONAR head + 60/40 physics blending
β β β βββ bicrossmamba_st.py Bidirectional Mamba + MCA
β β β βββ voiceradar.py CNN + physics aux + PGD probe
β β β βββ holi_antispoof.py Qwen2.5-Omni stub + rationale generator
β β β βββ calibration_loader.py Reads checkpoints/calibration.json
β β β βββ model_registry.py Singleton; lazy warm-up; calibration apply
β β βββ routers/
β β β βββ health.py /api/health, /api/models
β β β βββ inference.py /api/detect, /api/detect/sample, /api/visualize
β β β β with fitted-meta-classifier ensemble
β β β βββ benchmarks.py /api/benchmarks family
β β β βββ samples.py /api/samples + download
β β β βββ websocket.py WS /ws/inference/{session_id}
β β βββ utils/
β β βββ benchmark_data.py Paper Table III data
β β βββ samples.py Sample registry
β β βββ speech_synth.py Source-filter synth + 5 fake families
β β βββ sample_generator.py Boot bootstrapper
β βββ checkpoints/ Trained weights (.pt + .json metadata)
β βββ train_asvspoof.py Nes2Net trainer (synth / dir / ASVspoof modes)
β βββ train_paper_archs.py SONAR + BiCrossMamba-ST + VoiceRadar trainer
β βββ train_calibration.py Meta-classifier trainer
β βββ requirements.txt
β βββ Dockerfile
βββ frontend/
β βββ src/
β β βββ pages/ Home / Demo / Benchmarks / About
β β βββ components/ InputTabs (Upload/Record/Sample), NowPlaying,
β β β ModelCard, EnsembleVerdictBanner,
β β β InferenceProgress, ThemeToggle, ConfidenceGauge,
β β β Waveform/SpectrogramViewer, AudioUploader,
β β β MicRecorder, SampleLibrary, Layout
β β βββ hooks/ useAudioRecorder, useInference, useWebSocket,
β β β useTheme, useThemeColors
β β βββ store/inferenceStore.ts Zustand global store
β β βββ lib/api.ts Axios client + URL helpers
β β βββ lib/utils.ts cn(), formatters, status classifiers
β β βββ types/inference.ts Shared TS types (mirror Pydantic)
β β βββ index.css CSS theme variables: dark + .theme-light overrides
β βββ tailwind.config.ts Theme tokens via rgb(var(--name) / <alpha-value>)
β βββ package.json
β βββ Dockerfile
βββ data/
β βββ sample_audios/ 6 bootstrap clips (auto-generated on first boot)
βββ docker-compose.yml
βββ .gitignore
βββ README.md (this file)
```
---
## Performance
End-to-end latency on a single request, all 8 detectors in parallel, CPU only:
| Detector | Mean latency |
|---|---|
| MelodyMachine (Wav2Vec2) | ~700 ms |
| motheecreator (Wav2Vec2) | ~700 ms |
| LF/HF Physics | ~50 ms |
| Nes2Net | ~1,200 ms (XLS-R extraction dominates) |
| SONAR | ~0 ms with feature cache hit (else ~1,200 ms) |
| BiCrossMamba-ST | ~300 ms |
| VoiceRadar | ~300 ms |
| HoliAntiSpoof | ~50 ms |
| **Total wall-clock** | **~1.3 s** |
Detectors run in parallel via `asyncio.gather`, and a per-request `FeatureCache` ensures XLS-R features are computed once and shared between Nes2Net + SONAR.
### Recent benchmark results
Trained and evaluated on `deepfake_audio_dataset_jay15k` (1000 real + 1000 fake balanced split, 30% held-out validation):
| Detector | Val EER | Val Acc |
|---|---|---|
| Nes2Net | 0.005 (0.5%) | 98.8% |
| SONAR | 0.050 (5.0%) | 99.0% |
| BiCrossMamba-ST | 0.010 (1.0%) | 99.3% |
| VoiceRadar | 0.005 (0.5%) | 99.5% |
Meta-classifier (`calibration.json`): test acc 98.3%, log-loss 0.077β0.083.
End-to-end ensemble: 10/10 correct on 10 unseen jay15k clips with sharp separation (real β P(fake) β 0.05; fake β P(fake) β 0.95).
---
## Metric definitions
- **EER (Equal Error Rate)** β operating point where FAR = FRR. Lower is better.
- **minDCF** β minimum Detection Cost Function; cost-weighted error at optimal threshold.
- **t-DCF** β tandem DCF; joint cost between ASV system and spoof detector.
- **CLLR** β Cost of Log-Likelihood Ratio; calibration-aware error.
- **TPR / TNR** β True Positive / Negative Rate (sensitivity / specificity).
- **Pearson r (LF/HF)** β correlation between low- and high-frequency band energy time-series. Real speech β 0.6, synthesis β 0.
---
## Honest disclosure
- The two production HuggingFace detectors are **public Wav2Vec2 fine-tunes** β calibrated in this repo with grid-fitted softmax temperature + logit bias.
- Trained checkpoints for the four paper architectures (Nes2Net, SONAR, BiCrossMamba-ST, VoiceRadar) are produced by the in-repo training scripts on whatever data you point them at. Sample weights shipped here are trained on `deepfake_audio_dataset_jay15k` β performance on that distribution. Performance on ASVspoof 2021 DF / In-the-Wild requires re-training on those datasets.
- HoliAntiSpoof runs in **rationale-only mode** β Qwen2.5-Omni weights aren't bundled (16 GB). The architecture is registered, its physics-derived prediction is kept, and the rationale text demonstrates the explainability paradigm.
- `mamba-ssm` requires Linux + CUDA. On Windows / CPU, BiCrossMamba-ST uses an equivalent gated-conv substitute documented at runtime in the response `notes` field.
- Every fallback / substitution is labelled in the API response. Nothing is silently faked.
---
## Citation
```bibtex
@misc{kumar2025generalization,
title = {The Generalization Gap in Audio Deepfake Detection: A Four-Paradigm Review},
author = {Vipan Kumar},
year = {2025},
note = {Chitkara University, Punjab}
}
```
## License
Code: MIT. HuggingFace models retain their respective licences (`MelodyMachine/Deepfake-audio-detection`, `motheecreator/Deepfake-audio-detection`, `facebook/wav2vec2-xls-r-300m`).
|