| --- |
| title: Audio Deepfake Detector |
| emoji: ποΈ |
| colorFrom: blue |
| colorTo: red |
| sdk: docker |
| app_port: 7860 |
| pinned: false |
| license: mit |
| short_description: Audio deepfake detection with eight models |
| --- |
| |
| # ποΈ Audio Deepfake Detector |
|
|
| > **Production-grade audio deepfake detection with eight detectors, paper-grounded benchmarks, real fine-tuned weights, and a forensic UI.** |
| > |
| > Companion to *"The Generalization Gap in Audio Deepfake Detection: A Four-Paradigm Review"* β Vipan Kumar, Chitkara University, 2025. |
|
|
| []() []() []() []() []() |
|
|
| --- |
|
|
| ## What this is |
|
|
| A full-stack web app that runs **real audio deepfake detection** on uploaded clips, microphone recordings, and a curated sample library. Eight detectors run in parallel against every input, and a fitted logistic-regression meta-classifier combines them into a single calibrated `P(fake)` verdict. |
|
|
| ### Detectors |
|
|
| | ID | Display name | Paradigm | Source | |
| |----|--------------|----------|--------| |
| | `melodymachine` | MelodyMachine Wav2Vec2 | SSL Backend (production) | [HuggingFace public model](https://huggingface.co/MelodyMachine/Deepfake-audio-detection) | |
| | `motheecreator` | Motheecreator Wav2Vec2 | SSL Backend (production) | [HuggingFace public model](https://huggingface.co/motheecreator/Deepfake-audio-detection) | |
| | `lf_hf_physics` | LF/HF Physics Analyzer | Physics-Augmented (deterministic) | self-contained; multi-signal DSP β Pearson r, JS divergence, HF energy variance, voicing-aligned correlation, spectral roll-off | |
| | `nes2net` | Nes2Net | Efficient Backend | XLS-R + nested Res2Net (no DR layer) β **trained in-repo** | |
| | `sonar` | SONAR | Physics-Augmented | Dual XLS-R + SRM filters + JS contrastive head β **trained in-repo** | |
| | `bicrossmamba_st` | BiCrossMamba-ST | State-Space | Bidirectional Mamba + Mutual Cross-Attention β **trained in-repo** | |
| | `voiceradar` | VoiceRadar | Physics + Adv-Robust | Mel-CNN + physics aux head + PGD Ξ΅-stability probe β **trained in-repo** | |
| | `holi_antispoof` | HoliAntiSpoof | Audio LLM (explainable) | Qwen2.5-Omni stub + rule-based natural-language rationale generator | |
|
|
| The four "paper architecture" detectors ship with **real trained weights** (val_EER 0.5β5%) and switch to live mode automatically when the checkpoint files are present in `backend/checkpoints/`. If a checkpoint is missing, the detector falls back transparently to a paired live detector and labels the result `architecture_only_fallback` so nothing is silently faked. |
| |
| --- |
| |
| ## Quick start |
| |
| ### Prerequisites |
| |
| - **Python 3.10 or 3.11** (3.12 should work but is less battle-tested for the torch wheels) |
| - **Node.js 18+** with `npm` |
| - ~2 GB free disk for the HuggingFace cache (Wav2Vec2 weights + XLS-R) |
| - Optional: NVIDIA GPU with CUDA 11.8+ for faster inference (CPU-only also works) |
| |
| ### Run locally β Windows (PowerShell) |
| |
| ```powershell |
| # 1) Clone & enter |
| git clone <this-repo> audio-deepfake-detector |
| cd audio-deepfake-detector |
| |
| # 2) Backend β create venv + install deps |
| cd backend |
| python -m venv .venv |
| .\.venv\Scripts\Activate.ps1 |
| pip install --upgrade pip |
| pip install -r requirements.txt |
| |
| # 3) Backend β first run (downloads ~750 MB of HF weights into the system HF cache) |
| $env:ENABLE_HEAVY_MODELS = "true" |
| python -m uvicorn app.main:app --host 127.0.0.1 --port 8000 |
| |
| # Leave this terminal open. The backend serves at http://127.0.0.1:8000 |
| |
| # 4) Frontend β IN A SECOND TERMINAL |
| cd <repo-path>\audio-deepfake-detector\frontend |
| npm install |
| npm run dev |
| |
| # Frontend serves at http://localhost:5173 |
| ``` |
| |
| Open **http://localhost:5173** and you're done. |
| |
| ### Run locally β macOS / Linux |
| |
| ```bash |
| git clone <this-repo> audio-deepfake-detector |
| cd audio-deepfake-detector |
| |
| # Backend |
| cd backend |
| python -m venv .venv |
| source .venv/bin/activate |
| pip install --upgrade pip |
| pip install -r requirements.txt |
| ENABLE_HEAVY_MODELS=true uvicorn app.main:app --host 127.0.0.1 --port 8000 |
| |
| # Frontend (second terminal) |
| cd ../frontend |
| npm install |
| npm run dev |
| ``` |
| |
| ### Run with Docker |
| |
| ```bash |
| docker compose up --build |
| # Frontend β http://localhost:3000 |
| # Backend β http://localhost:8000 |
| ``` |
| |
| The first Docker build downloads HF weights into a named volume; subsequent runs are fast. |
| |
| --- |
| |
| ## Deploy to Hugging Face Spaces (free, no credit card) |
| |
| A combined Dockerfile + `hf_spaces_entrypoint.py` at the repo root build the |
| React frontend, copy it into the FastAPI container, and serve both on port 7860 (the HF Spaces convention). |
| |
| 1. Push this repo to GitHub first (so HF can clone it). |
| 2. Visit https://huggingface.co/new-space and create a **Docker Space**: |
| - Owner: your HF username |
| - Space name: `audio-deepfake-detector` (or anything) |
| - Space SDK: **Docker** |
| - Hardware: free CPU is sufficient |
| - Visibility: Public or Private (your choice) |
| 3. After creation, push the same repo to the Space's git remote: |
| |
| ```bash |
| git remote add space https://huggingface.co/spaces/<your-user>/<space-name> |
| git push space main |
| ``` |
| |
| (HF prompts for a write-access token β generate one at |
| https://huggingface.co/settings/tokens with the *write* scope.) |
| 4. The Space rebuilds and the URL becomes |
| `https://huggingface.co/spaces/<your-user>/<space-name>` β the |
| frontend serves from the root, the API at `/api/*`, docs at `/docs`. |
| |
| The YAML block at the top of this README tells HF to use Docker mode and |
| expose port 7860; the entrypoint mounts `/app/static` so the SPA + API |
| share the same origin. |
| |
| ### Other deploy options |
| |
| - **Backend on Render / Railway / Fly.io**: the existing `backend/Dockerfile` |
| works on all three; their free tiers may be too small for the XLS-R model |
| (~1.5 GB resident). Check current memory limits. |
| - **Frontend on Vercel / Netlify**: deploy `frontend/` as a static site; |
| point `VITE_API_BASE_URL` at wherever the backend lives. |
| - **Self-hosted**: `docker compose up --build` on any VPS with β₯ 4 GB RAM. |
|
|
| --- |
|
|
| ## What you'll see |
|
|
| - **Home (`/`)** β paper context, four-paradigm overview, three-deficits framing, light/dark theme toggle at the bottom. |
| - **Demo (`/demo`)** β drop a clip / record from mic / pick a sample. Eight detectors run in parallel. Verdict + per-detector cards + waveform + mel spectrogram. Recording supports playback + replay + delete (cleared on refresh). |
| - **Benchmarks (`/benchmarks`)** β paper Table III data verbatim, generalization-gap chart, datasetΓmodel coverage matrix, PGD adversarial-robustness curve, HoliAntiSpoof rationale examples. |
| - **About (`/about`)** β full architecture descriptions for all 7 paper models with SVG diagrams, comparative Table II + III. |
|
|
| The whole UI has a **light/dark theme toggle** (Sun/Moon switch at the bottom of Home and in the global footer). Choice persists via `localStorage`. |
|
|
| --- |
|
|
| ## API reference |
|
|
| Full interactive docs: http://127.0.0.1:8000/docs |
|
|
| | Method | Path | Purpose | |
| |---|---|---| |
| | GET | `/api/health` | Service status + per-detector load state | |
| | GET | `/api/models` | All detectors with paradigm, params, EER refs | |
| | POST | `/api/detect` | Upload audio (multipart), run all (or selected) detectors, return ensemble verdict | |
| | POST | `/api/detect/sample/{id}` | Detect on one of the bundled sample clips | |
| | POST | `/api/visualize` | Waveform envelope + mel spectrogram for the UI | |
| | GET | `/api/benchmarks` | Paper Table III rows + generalization-gap data | |
| | GET | `/api/benchmarks/datasets` | Dataset metadata | |
| | GET | `/api/benchmarks/evaluations` | Dataset Γ model Γ metric matrix | |
| | GET | `/api/benchmarks/adversarial` | PGD Ξ΅-curve data | |
| | GET | `/api/benchmarks/explainability` | HoliAntiSpoof rationale examples | |
| | GET | `/api/benchmarks/radar` | Radar-chart data (5 axes) | |
| | GET | `/api/benchmarks/highlights` | Paper one-liners | |
| | GET | `/api/samples` | Sample-library metadata | |
| | GET | `/api/samples/{id}/download` | Sample WAV bytes | |
| | WS | `/ws/inference/{session_id}` | Streamed inference progress | |
|
|
| ### Example: detect a clip |
|
|
| ```bash |
| curl -F "audio_file=@my_clip.wav" \ |
| -F 'models=["nes2net","sonar","lf_hf_physics"]' \ |
| http://127.0.0.1:8000/api/detect | jq |
| ``` |
|
|
| --- |
|
|
| ## Training the paper architectures |
|
|
| The repo includes three training scripts. All run on CPU; GPU is optional but ~10Γ faster. |
|
|
| ### `train_asvspoof.py` β Nes2Net |
| |
| Three modes, picked automatically from the args: |
| |
| ```powershell |
| # (1) Synthesised fallback β downloads small public LibriSpeech clips + |
| # synthesises 5 fake families. Good for first-time setup. |
| python train_asvspoof.py --n-real 80 --n-fake 80 --epochs 8 --batch-size 8 |
|
|
| # (2) Directory mode β point at any dataset organised as real/ and fake/ folders |
| python train_asvspoof.py ` |
| --real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" ` |
| --fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" ` |
| --max-per-class 1000 --epochs 6 --batch-size 8 |
| |
| # (3) ASVspoof real-data mode β provide protocol + audio dir |
| python train_asvspoof.py ` |
| --protocol path\to\ASVspoof2019.LA.cm.train.trn.txt ` |
| --audio-dir path\to\ASVspoof2019_LA_train\flac ` |
| --epochs 12 --batch-size 8 |
| ``` |
| |
| ### `train_paper_archs.py` β SONAR + BiCrossMamba-ST + VoiceRadar |
| |
| Each architecture has its own hyperparameter profile, picked automatically: |
| |
| | Architecture | LR | Epochs | Weight decay | Why | |
| |---|---|---|---|---| |
| | SONAR | 1e-4 | 5 | 5e-4 | 41M-param dual-stream cross-attention head β heavy regularisation, fewer epochs to avoid overfitting | |
| | BiCrossMamba-ST | 5e-4 | 10 | 1e-4 | 516K param Mamba+MCA β small model, tolerates higher LR + more epochs | |
| | VoiceRadar | 3e-4 | 10 | 2e-4 | 410K-param CNN β standard LR, modest weight decay | |
| |
| ```powershell |
| # Synthesised fallback |
| python train_paper_archs.py --n-real 80 --n-fake 80 --batch-size 8 |
|
|
| # Directory mode (e.g. jay15k) |
| python train_paper_archs.py ` |
| --real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" ` |
| --fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" ` |
| --max-per-class 1000 --batch-size 8 |
|
|
| # Train only one architecture |
| python train_paper_archs.py --only sonar --batch-size 8 |
| ``` |
| |
| ### `train_calibration.py` β meta-classifier (logistic regression on top of detector outputs) |
| |
| Run this AFTER training the paper architectures (or any time the detector mix changes): |
| |
| ```powershell |
| # Synthesised pool |
| python train_calibration.py --n-real 30 --n-fake 30 |
| |
| # Directory mode |
| python train_calibration.py ` |
| --real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" ` |
| --fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" ` |
| --max-per-class 100 |
| ``` |
| |
| Output: `backend/checkpoints/calibration.json`. Loaded automatically on next backend startup. |
| |
| ### Datasets |
| |
| - **ASVspoof 2019 LA** β ~5 GB, click-through registration: https://datashare.ed.ac.uk/handle/10283/3336 |
| - **ASVspoof 2021 DF** β ~133 GB, challenge registration: https://www.asvspoof.org/ |
| - **In-the-Wild (MΓΌller et al. 2022)** β 37.9 h: https://deepfake-total.com/in_the_wild |
| - **deepfake_audio_dataset_jay15k** β 9.6K real + 7.2K fake (16 kHz mono), available on Kaggle / similar mirrors. Verified working with `train_asvspoof.py --real-dir / --fake-dir`. |
| |
| --- |
| |
| ## Configuration |
| |
| All env vars (also readable from a `backend/.env` file): |
| |
| | Variable | Default | Effect | |
| |---|---|---| |
| | `ALLOWED_ORIGINS` | `http://localhost:5173,http://localhost:3000,...` | CORS origins (CSV) | |
| | `ENABLE_HEAVY_MODELS` | `true` | Disable to skip the two ~94 M-param Wav2Vec2 detectors. The LF/HF physics detector + paper architectures still run. | |
| | `DEMO_MODE` | `false` | Reserved flag | |
| | `HF_HOME` / `TRANSFORMERS_CACHE` | `/app/.cache/huggingface` | HuggingFace cache location | |
| |
| Frontend env (Vite): |
| |
| | Variable | Default | Effect | |
| |---|---|---| |
| | `VITE_API_BASE_URL` | `http://localhost:8000` | Backend HTTP base | |
| | `VITE_WS_BASE_URL` | `ws://localhost:8000` | Backend WS base | |
| |
| --- |
| |
| ## Project layout |
| |
| ``` |
| audio-deepfake-detector/ |
| βββ backend/ |
| β βββ app/ |
| β β βββ main.py FastAPI app + lifespan + CORS |
| β β βββ config.py Settings via pydantic-settings |
| β β βββ schemas/ Pydantic request/response models |
| β β βββ features/ Audio decode (incl. ffmpeg fallback for WebM/Opus), |
| β β β XLS-R extractor, SRM filters, mel utilities, |
| β β β multi-signal LF/HF physics analyser |
| β β βββ models/ |
| β β β βββ base_model.py BaseDetector + per-request FeatureCache |
| β β β βββ hf_detectors.py MelodyMachine + motheecreator (calibrated) |
| β β β βββ lf_hf_detector.py Deterministic physics analyser |
| β β β βββ nes2net.py Nes2Net architecture + fallback wiring |
| β β β βββ sonar.py SONAR head + 60/40 physics blending |
| β β β βββ bicrossmamba_st.py Bidirectional Mamba + MCA |
| β β β βββ voiceradar.py CNN + physics aux + PGD probe |
| β β β βββ holi_antispoof.py Qwen2.5-Omni stub + rationale generator |
| β β β βββ calibration_loader.py Reads checkpoints/calibration.json |
| β β β βββ model_registry.py Singleton; lazy warm-up; calibration apply |
| β β βββ routers/ |
| β β β βββ health.py /api/health, /api/models |
| β β β βββ inference.py /api/detect, /api/detect/sample, /api/visualize |
| β β β β with fitted-meta-classifier ensemble |
| β β β βββ benchmarks.py /api/benchmarks family |
| β β β βββ samples.py /api/samples + download |
| β β β βββ websocket.py WS /ws/inference/{session_id} |
| β β βββ utils/ |
| β β βββ benchmark_data.py Paper Table III data |
| β β βββ samples.py Sample registry |
| β β βββ speech_synth.py Source-filter synth + 5 fake families |
| β β βββ sample_generator.py Boot bootstrapper |
| β βββ checkpoints/ Trained weights (.pt + .json metadata) |
| β βββ train_asvspoof.py Nes2Net trainer (synth / dir / ASVspoof modes) |
| β βββ train_paper_archs.py SONAR + BiCrossMamba-ST + VoiceRadar trainer |
| β βββ train_calibration.py Meta-classifier trainer |
| β βββ requirements.txt |
| β βββ Dockerfile |
| βββ frontend/ |
| β βββ src/ |
| β β βββ pages/ Home / Demo / Benchmarks / About |
| β β βββ components/ InputTabs (Upload/Record/Sample), NowPlaying, |
| β β β ModelCard, EnsembleVerdictBanner, |
| β β β InferenceProgress, ThemeToggle, ConfidenceGauge, |
| β β β Waveform/SpectrogramViewer, AudioUploader, |
| β β β MicRecorder, SampleLibrary, Layout |
| β β βββ hooks/ useAudioRecorder, useInference, useWebSocket, |
| β β β useTheme, useThemeColors |
| β β βββ store/inferenceStore.ts Zustand global store |
| β β βββ lib/api.ts Axios client + URL helpers |
| β β βββ lib/utils.ts cn(), formatters, status classifiers |
| β β βββ types/inference.ts Shared TS types (mirror Pydantic) |
| β β βββ index.css CSS theme variables: dark + .theme-light overrides |
| β βββ tailwind.config.ts Theme tokens via rgb(var(--name) / <alpha-value>) |
| β βββ package.json |
| β βββ Dockerfile |
| βββ data/ |
| β βββ sample_audios/ 6 bootstrap clips (auto-generated on first boot) |
| βββ docker-compose.yml |
| βββ .gitignore |
| βββ README.md (this file) |
| ``` |
| |
| --- |
| |
| ## Performance |
| |
| End-to-end latency on a single request, all 8 detectors in parallel, CPU only: |
| |
| | Detector | Mean latency | |
| |---|---| |
| | MelodyMachine (Wav2Vec2) | ~700 ms | |
| | motheecreator (Wav2Vec2) | ~700 ms | |
| | LF/HF Physics | ~50 ms | |
| | Nes2Net | ~1,200 ms (XLS-R extraction dominates) | |
| | SONAR | ~0 ms with feature cache hit (else ~1,200 ms) | |
| | BiCrossMamba-ST | ~300 ms | |
| | VoiceRadar | ~300 ms | |
| | HoliAntiSpoof | ~50 ms | |
| | **Total wall-clock** | **~1.3 s** | |
| |
| Detectors run in parallel via `asyncio.gather`, and a per-request `FeatureCache` ensures XLS-R features are computed once and shared between Nes2Net + SONAR. |
| |
| ### Recent benchmark results |
| |
| Trained and evaluated on `deepfake_audio_dataset_jay15k` (1000 real + 1000 fake balanced split, 30% held-out validation): |
|
|
| | Detector | Val EER | Val Acc | |
| |---|---|---| |
| | Nes2Net | 0.005 (0.5%) | 98.8% | |
| | SONAR | 0.050 (5.0%) | 99.0% | |
| | BiCrossMamba-ST | 0.010 (1.0%) | 99.3% | |
| | VoiceRadar | 0.005 (0.5%) | 99.5% | |
|
|
| Meta-classifier (`calibration.json`): test acc 98.3%, log-loss 0.077β0.083. |
|
|
| End-to-end ensemble: 10/10 correct on 10 unseen jay15k clips with sharp separation (real β P(fake) β 0.05; fake β P(fake) β 0.95). |
|
|
| --- |
|
|
| ## Metric definitions |
|
|
| - **EER (Equal Error Rate)** β operating point where FAR = FRR. Lower is better. |
| - **minDCF** β minimum Detection Cost Function; cost-weighted error at optimal threshold. |
| - **t-DCF** β tandem DCF; joint cost between ASV system and spoof detector. |
| - **CLLR** β Cost of Log-Likelihood Ratio; calibration-aware error. |
| - **TPR / TNR** β True Positive / Negative Rate (sensitivity / specificity). |
| - **Pearson r (LF/HF)** β correlation between low- and high-frequency band energy time-series. Real speech β 0.6, synthesis β 0. |
|
|
| --- |
|
|
| ## Honest disclosure |
|
|
| - The two production HuggingFace detectors are **public Wav2Vec2 fine-tunes** β calibrated in this repo with grid-fitted softmax temperature + logit bias. |
| - Trained checkpoints for the four paper architectures (Nes2Net, SONAR, BiCrossMamba-ST, VoiceRadar) are produced by the in-repo training scripts on whatever data you point them at. Sample weights shipped here are trained on `deepfake_audio_dataset_jay15k` β performance on that distribution. Performance on ASVspoof 2021 DF / In-the-Wild requires re-training on those datasets. |
| - HoliAntiSpoof runs in **rationale-only mode** β Qwen2.5-Omni weights aren't bundled (16 GB). The architecture is registered, its physics-derived prediction is kept, and the rationale text demonstrates the explainability paradigm. |
| - `mamba-ssm` requires Linux + CUDA. On Windows / CPU, BiCrossMamba-ST uses an equivalent gated-conv substitute documented at runtime in the response `notes` field. |
| - Every fallback / substitution is labelled in the API response. Nothing is silently faked. |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{kumar2025generalization, |
| title = {The Generalization Gap in Audio Deepfake Detection: A Four-Paradigm Review}, |
| author = {Vipan Kumar}, |
| year = {2025}, |
| note = {Chitkara University, Punjab} |
| } |
| ``` |
|
|
| ## License |
|
|
| Code: MIT. HuggingFace models retain their respective licences (`MelodyMachine/Deepfake-audio-detection`, `motheecreator/Deepfake-audio-detection`, `facebook/wav2vec2-xls-r-300m`). |
|
|
