Spaces:

vipan-kumar
/

Audio_Deepfake_Detection

Sleeping

App Files Files Community

Audio_Deepfake_Detection / README.md

vipan-kumar

fix: shorten README short_description for HF Spaces

6046513 about 2 months ago

preview code

Raw

History Blame Contribute Delete

19.6 kB

metadata

title: Audio Deepfake Detector
emoji: 🎙️
colorFrom: blue
colorTo: red
sdk: docker
app_port: 7860
pinned: false
license: mit
short_description: Audio deepfake detection with eight models

🎙️ Audio Deepfake Detector

Production-grade audio deepfake detection with eight detectors, paper-grounded benchmarks, real fine-tuned weights, and a forensic UI.

Companion to "The Generalization Gap in Audio Deepfake Detection: A Four-Paradigm Review" — Vipan Kumar, Chitkara University, 2025.

What this is

A full-stack web app that runs real audio deepfake detection on uploaded clips, microphone recordings, and a curated sample library. Eight detectors run in parallel against every input, and a fitted logistic-regression meta-classifier combines them into a single calibrated P(fake) verdict.

Detectors

ID	Display name	Paradigm	Source
`melodymachine`	MelodyMachine Wav2Vec2	SSL Backend (production)	HuggingFace public model
`motheecreator`	Motheecreator Wav2Vec2	SSL Backend (production)	HuggingFace public model
`lf_hf_physics`	LF/HF Physics Analyzer	Physics-Augmented (deterministic)	self-contained; multi-signal DSP — Pearson r, JS divergence, HF energy variance, voicing-aligned correlation, spectral roll-off
`nes2net`	Nes2Net	Efficient Backend	XLS-R + nested Res2Net (no DR layer) — trained in-repo
`sonar`	SONAR	Physics-Augmented	Dual XLS-R + SRM filters + JS contrastive head — trained in-repo
`bicrossmamba_st`	BiCrossMamba-ST	State-Space	Bidirectional Mamba + Mutual Cross-Attention — trained in-repo
`voiceradar`	VoiceRadar	Physics + Adv-Robust	Mel-CNN + physics aux head + PGD ε-stability probe — trained in-repo
`holi_antispoof`	HoliAntiSpoof	Audio LLM (explainable)	Qwen2.5-Omni stub + rule-based natural-language rationale generator

The four "paper architecture" detectors ship with real trained weights (val_EER 0.5–5%) and switch to live mode automatically when the checkpoint files are present in backend/checkpoints/. If a checkpoint is missing, the detector falls back transparently to a paired live detector and labels the result architecture_only_fallback so nothing is silently faked.

Quick start

Prerequisites

Python 3.10 or 3.11 (3.12 should work but is less battle-tested for the torch wheels)
Node.js 18+ with npm
~2 GB free disk for the HuggingFace cache (Wav2Vec2 weights + XLS-R)
Optional: NVIDIA GPU with CUDA 11.8+ for faster inference (CPU-only also works)

Run locally — Windows (PowerShell)

# 1) Clone & enter
git clone <this-repo> audio-deepfake-detector
cd audio-deepfake-detector

# 2) Backend — create venv + install deps
cd backend
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install --upgrade pip
pip install -r requirements.txt

# 3) Backend — first run (downloads ~750 MB of HF weights into the system HF cache)
$env:ENABLE_HEAVY_MODELS = "true"
python -m uvicorn app.main:app --host 127.0.0.1 --port 8000

# Leave this terminal open. The backend serves at http://127.0.0.1:8000

# 4) Frontend — IN A SECOND TERMINAL
cd <repo-path>\audio-deepfake-detector\frontend
npm install
npm run dev

# Frontend serves at http://localhost:5173

Open http://localhost:5173 and you're done.

Run locally — macOS / Linux

git clone <this-repo> audio-deepfake-detector
cd audio-deepfake-detector

# Backend
cd backend
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
ENABLE_HEAVY_MODELS=true uvicorn app.main:app --host 127.0.0.1 --port 8000

# Frontend (second terminal)
cd ../frontend
npm install
npm run dev

Run with Docker

docker compose up --build
# Frontend → http://localhost:3000
# Backend  → http://localhost:8000

The first Docker build downloads HF weights into a named volume; subsequent runs are fast.

Deploy to Hugging Face Spaces (free, no credit card)

A combined Dockerfile + hf_spaces_entrypoint.py at the repo root build the React frontend, copy it into the FastAPI container, and serve both on port 7860 (the HF Spaces convention).

Push this repo to GitHub first (so HF can clone it).
Visit https://huggingface.co/new-space and create a Docker Space:
- Owner: your HF username
- Space name: audio-deepfake-detector (or anything)
- Space SDK: Docker
- Hardware: free CPU is sufficient
- Visibility: Public or Private (your choice)
After creation, push the same repo to the Space's git remote:
```
git remote add space https://huggingface.co/spaces/<your-user>/<space-name>
git push space main
```
(HF prompts for a write-access token — generate one at https://huggingface.co/settings/tokens with the write scope.)
The Space rebuilds and the URL becomes https://huggingface.co/spaces/<your-user>/<space-name> — the frontend serves from the root, the API at /api/*, docs at /docs.

The YAML block at the top of this README tells HF to use Docker mode and expose port 7860; the entrypoint mounts /app/static so the SPA + API share the same origin.

Other deploy options

Backend on Render / Railway / Fly.io: the existing backend/Dockerfile works on all three; their free tiers may be too small for the XLS-R model (~1.5 GB resident). Check current memory limits.
Frontend on Vercel / Netlify: deploy frontend/ as a static site; point VITE_API_BASE_URL at wherever the backend lives.
Self-hosted: docker compose up --build on any VPS with ≥ 4 GB RAM.

What you'll see

Home (/) — paper context, four-paradigm overview, three-deficits framing, light/dark theme toggle at the bottom.
Demo (/demo) — drop a clip / record from mic / pick a sample. Eight detectors run in parallel. Verdict + per-detector cards + waveform + mel spectrogram. Recording supports playback + replay + delete (cleared on refresh).
Benchmarks (/benchmarks) — paper Table III data verbatim, generalization-gap chart, dataset×model coverage matrix, PGD adversarial-robustness curve, HoliAntiSpoof rationale examples.
About (/about) — full architecture descriptions for all 7 paper models with SVG diagrams, comparative Table II + III.

The whole UI has a light/dark theme toggle (Sun/Moon switch at the bottom of Home and in the global footer). Choice persists via localStorage.

API reference

Full interactive docs: http://127.0.0.1:8000/docs

Method	Path	Purpose
GET	`/api/health`	Service status + per-detector load state
GET	`/api/models`	All detectors with paradigm, params, EER refs
POST	`/api/detect`	Upload audio (multipart), run all (or selected) detectors, return ensemble verdict
POST	`/api/detect/sample/{id}`	Detect on one of the bundled sample clips
POST	`/api/visualize`	Waveform envelope + mel spectrogram for the UI
GET	`/api/benchmarks`	Paper Table III rows + generalization-gap data
GET	`/api/benchmarks/datasets`	Dataset metadata
GET	`/api/benchmarks/evaluations`	Dataset × model × metric matrix
GET	`/api/benchmarks/adversarial`	PGD ε-curve data
GET	`/api/benchmarks/explainability`	HoliAntiSpoof rationale examples
GET	`/api/benchmarks/radar`	Radar-chart data (5 axes)
GET	`/api/benchmarks/highlights`	Paper one-liners
GET	`/api/samples`	Sample-library metadata
GET	`/api/samples/{id}/download`	Sample WAV bytes
WS	`/ws/inference/{session_id}`	Streamed inference progress

Example: detect a clip

curl -F "audio_file=@my_clip.wav" \
     -F 'models=["nes2net","sonar","lf_hf_physics"]' \
     http://127.0.0.1:8000/api/detect | jq

Training the paper architectures

The repo includes three training scripts. All run on CPU; GPU is optional but ~10× faster.

`train_asvspoof.py` — Nes2Net

Three modes, picked automatically from the args:

# (1) Synthesised fallback — downloads small public LibriSpeech clips +
#     synthesises 5 fake families. Good for first-time setup.
python train_asvspoof.py --n-real 80 --n-fake 80 --epochs 8 --batch-size 8

# (2) Directory mode — point at any dataset organised as real/ and fake/ folders
python train_asvspoof.py `
  --real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" `
  --fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" `
  --max-per-class 1000 --epochs 6 --batch-size 8

# (3) ASVspoof real-data mode — provide protocol + audio dir
python train_asvspoof.py `
  --protocol path\to\ASVspoof2019.LA.cm.train.trn.txt `
  --audio-dir path\to\ASVspoof2019_LA_train\flac `
  --epochs 12 --batch-size 8

`train_paper_archs.py` — SONAR + BiCrossMamba-ST + VoiceRadar

Each architecture has its own hyperparameter profile, picked automatically:

Architecture	LR	Epochs	Weight decay	Why
SONAR	1e-4	5	5e-4	41M-param dual-stream cross-attention head — heavy regularisation, fewer epochs to avoid overfitting
BiCrossMamba-ST	5e-4	10	1e-4	516K param Mamba+MCA — small model, tolerates higher LR + more epochs
VoiceRadar	3e-4	10	2e-4	410K-param CNN — standard LR, modest weight decay

# Synthesised fallback
python train_paper_archs.py --n-real 80 --n-fake 80 --batch-size 8

# Directory mode (e.g. jay15k)
python train_paper_archs.py `
  --real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" `
  --fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" `
  --max-per-class 1000 --batch-size 8

# Train only one architecture
python train_paper_archs.py --only sonar --batch-size 8

`train_calibration.py` — meta-classifier (logistic regression on top of detector outputs)

Run this AFTER training the paper architectures (or any time the detector mix changes):

# Synthesised pool
python train_calibration.py --n-real 30 --n-fake 30

# Directory mode
python train_calibration.py `
  --real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" `
  --fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" `
  --max-per-class 100

Output: backend/checkpoints/calibration.json. Loaded automatically on next backend startup.

Datasets

ASVspoof 2019 LA — ~5 GB, click-through registration: https://datashare.ed.ac.uk/handle/10283/3336
ASVspoof 2021 DF — ~133 GB, challenge registration: https://www.asvspoof.org/
In-the-Wild (Müller et al. 2022) — 37.9 h: https://deepfake-total.com/in_the_wild
deepfake_audio_dataset_jay15k — 9.6K real + 7.2K fake (16 kHz mono), available on Kaggle / similar mirrors. Verified working with train_asvspoof.py --real-dir / --fake-dir.

Configuration

All env vars (also readable from a backend/.env file):

Variable	Default	Effect
`ALLOWED_ORIGINS`	`http://localhost:5173,http://localhost:3000,...`	CORS origins (CSV)
`ENABLE_HEAVY_MODELS`	`true`	Disable to skip the two ~94 M-param Wav2Vec2 detectors. The LF/HF physics detector + paper architectures still run.
`DEMO_MODE`	`false`	Reserved flag
`HF_HOME` / `TRANSFORMERS_CACHE`	`/app/.cache/huggingface`	HuggingFace cache location

Frontend env (Vite):

Variable	Default	Effect
`VITE_API_BASE_URL`	`http://localhost:8000`	Backend HTTP base
`VITE_WS_BASE_URL`	`ws://localhost:8000`	Backend WS base

Project layout

audio-deepfake-detector/
├── backend/
│   ├── app/
│   │   ├── main.py                 FastAPI app + lifespan + CORS
│   │   ├── config.py               Settings via pydantic-settings
│   │   ├── schemas/                Pydantic request/response models
│   │   ├── features/               Audio decode (incl. ffmpeg fallback for WebM/Opus),
│   │   │                           XLS-R extractor, SRM filters, mel utilities,
│   │   │                           multi-signal LF/HF physics analyser
│   │   ├── models/
│   │   │   ├── base_model.py            BaseDetector + per-request FeatureCache
│   │   │   ├── hf_detectors.py          MelodyMachine + motheecreator (calibrated)
│   │   │   ├── lf_hf_detector.py        Deterministic physics analyser
│   │   │   ├── nes2net.py               Nes2Net architecture + fallback wiring
│   │   │   ├── sonar.py                 SONAR head + 60/40 physics blending
│   │   │   ├── bicrossmamba_st.py       Bidirectional Mamba + MCA
│   │   │   ├── voiceradar.py            CNN + physics aux + PGD probe
│   │   │   ├── holi_antispoof.py        Qwen2.5-Omni stub + rationale generator
│   │   │   ├── calibration_loader.py    Reads checkpoints/calibration.json
│   │   │   └── model_registry.py        Singleton; lazy warm-up; calibration apply
│   │   ├── routers/
│   │   │   ├── health.py                /api/health, /api/models
│   │   │   ├── inference.py             /api/detect, /api/detect/sample, /api/visualize
│   │   │   │                            with fitted-meta-classifier ensemble
│   │   │   ├── benchmarks.py            /api/benchmarks family
│   │   │   ├── samples.py               /api/samples + download
│   │   │   └── websocket.py             WS /ws/inference/{session_id}
│   │   └── utils/
│   │       ├── benchmark_data.py        Paper Table III data
│   │       ├── samples.py               Sample registry
│   │       ├── speech_synth.py          Source-filter synth + 5 fake families
│   │       └── sample_generator.py      Boot bootstrapper
│   ├── checkpoints/                Trained weights (.pt + .json metadata)
│   ├── train_asvspoof.py           Nes2Net trainer (synth / dir / ASVspoof modes)
│   ├── train_paper_archs.py        SONAR + BiCrossMamba-ST + VoiceRadar trainer
│   ├── train_calibration.py        Meta-classifier trainer
│   ├── requirements.txt
│   └── Dockerfile
├── frontend/
│   ├── src/
│   │   ├── pages/                  Home / Demo / Benchmarks / About
│   │   ├── components/             InputTabs (Upload/Record/Sample), NowPlaying,
│   │   │                           ModelCard, EnsembleVerdictBanner,
│   │   │                           InferenceProgress, ThemeToggle, ConfidenceGauge,
│   │   │                           Waveform/SpectrogramViewer, AudioUploader,
│   │   │                           MicRecorder, SampleLibrary, Layout
│   │   ├── hooks/                  useAudioRecorder, useInference, useWebSocket,
│   │   │                           useTheme, useThemeColors
│   │   ├── store/inferenceStore.ts Zustand global store
│   │   ├── lib/api.ts              Axios client + URL helpers
│   │   ├── lib/utils.ts            cn(), formatters, status classifiers
│   │   ├── types/inference.ts      Shared TS types (mirror Pydantic)
│   │   └── index.css               CSS theme variables: dark + .theme-light overrides
│   ├── tailwind.config.ts          Theme tokens via rgb(var(--name) / <alpha-value>)
│   ├── package.json
│   └── Dockerfile
├── data/
│   └── sample_audios/              6 bootstrap clips (auto-generated on first boot)
├── docker-compose.yml
├── .gitignore
└── README.md  (this file)

Performance

End-to-end latency on a single request, all 8 detectors in parallel, CPU only:

Detector	Mean latency
MelodyMachine (Wav2Vec2)	~700 ms
motheecreator (Wav2Vec2)	~700 ms
LF/HF Physics	~50 ms
Nes2Net	~1,200 ms (XLS-R extraction dominates)
SONAR	~0 ms with feature cache hit (else ~1,200 ms)
BiCrossMamba-ST	~300 ms
VoiceRadar	~300 ms
HoliAntiSpoof	~50 ms
Total wall-clock	~1.3 s

Detectors run in parallel via asyncio.gather, and a per-request FeatureCache ensures XLS-R features are computed once and shared between Nes2Net + SONAR.

Recent benchmark results

Trained and evaluated on deepfake_audio_dataset_jay15k (1000 real + 1000 fake balanced split, 30% held-out validation):

Detector	Val EER	Val Acc
Nes2Net	0.005 (0.5%)	98.8%
SONAR	0.050 (5.0%)	99.0%
BiCrossMamba-ST	0.010 (1.0%)	99.3%
VoiceRadar	0.005 (0.5%)	99.5%

Meta-classifier (calibration.json): test acc 98.3%, log-loss 0.077–0.083.

End-to-end ensemble: 10/10 correct on 10 unseen jay15k clips with sharp separation (real → P(fake) ≈ 0.05; fake → P(fake) ≈ 0.95).

Metric definitions

EER (Equal Error Rate) — operating point where FAR = FRR. Lower is better.
minDCF — minimum Detection Cost Function; cost-weighted error at optimal threshold.
t-DCF — tandem DCF; joint cost between ASV system and spoof detector.
CLLR — Cost of Log-Likelihood Ratio; calibration-aware error.
TPR / TNR — True Positive / Negative Rate (sensitivity / specificity).
Pearson r (LF/HF) — correlation between low- and high-frequency band energy time-series. Real speech ≈ 0.6, synthesis ≈ 0.

Honest disclosure

The two production HuggingFace detectors are public Wav2Vec2 fine-tunes — calibrated in this repo with grid-fitted softmax temperature + logit bias.
Trained checkpoints for the four paper architectures (Nes2Net, SONAR, BiCrossMamba-ST, VoiceRadar) are produced by the in-repo training scripts on whatever data you point them at. Sample weights shipped here are trained on deepfake_audio_dataset_jay15k — performance on that distribution. Performance on ASVspoof 2021 DF / In-the-Wild requires re-training on those datasets.
HoliAntiSpoof runs in rationale-only mode — Qwen2.5-Omni weights aren't bundled (16 GB). The architecture is registered, its physics-derived prediction is kept, and the rationale text demonstrates the explainability paradigm.
mamba-ssm requires Linux + CUDA. On Windows / CPU, BiCrossMamba-ST uses an equivalent gated-conv substitute documented at runtime in the response notes field.
Every fallback / substitution is labelled in the API response. Nothing is silently faked.

Citation

@misc{kumar2025generalization,
  title  = {The Generalization Gap in Audio Deepfake Detection: A Four-Paradigm Review},
  author = {Vipan Kumar},
  year   = {2025},
  note   = {Chitkara University, Punjab}
}

License

Code: MIT. HuggingFace models retain their respective licences (MelodyMachine/Deepfake-audio-detection, motheecreator/Deepfake-audio-detection, facebook/wav2vec2-xls-r-300m).