--- title: Audio Deepfake Detector emoji: πŸŽ™οΈ colorFrom: blue colorTo: red sdk: docker app_port: 7860 pinned: false license: mit short_description: Audio deepfake detection with eight models --- # πŸŽ™οΈ Audio Deepfake Detector > **Production-grade audio deepfake detection with eight detectors, paper-grounded benchmarks, real fine-tuned weights, and a forensic UI.** > > Companion to *"The Generalization Gap in Audio Deepfake Detection: A Four-Paradigm Review"* β€” Vipan Kumar, Chitkara University, 2025. [![python](https://img.shields.io/badge/python-3.10%2B-blue)]() [![torch](https://img.shields.io/badge/torch-2.4-ee4c2c)]() [![fastapi](https://img.shields.io/badge/fastapi-0.115-009688)]() [![react](https://img.shields.io/badge/react-18-61dafb)]() [![tailwind](https://img.shields.io/badge/tailwind-3-38bdf8)]() --- ## What this is A full-stack web app that runs **real audio deepfake detection** on uploaded clips, microphone recordings, and a curated sample library. Eight detectors run in parallel against every input, and a fitted logistic-regression meta-classifier combines them into a single calibrated `P(fake)` verdict. ### Detectors | ID | Display name | Paradigm | Source | |----|--------------|----------|--------| | `melodymachine` | MelodyMachine Wav2Vec2 | SSL Backend (production) | [HuggingFace public model](https://huggingface.co/MelodyMachine/Deepfake-audio-detection) | | `motheecreator` | Motheecreator Wav2Vec2 | SSL Backend (production) | [HuggingFace public model](https://huggingface.co/motheecreator/Deepfake-audio-detection) | | `lf_hf_physics` | LF/HF Physics Analyzer | Physics-Augmented (deterministic) | self-contained; multi-signal DSP β€” Pearson r, JS divergence, HF energy variance, voicing-aligned correlation, spectral roll-off | | `nes2net` | Nes2Net | Efficient Backend | XLS-R + nested Res2Net (no DR layer) β€” **trained in-repo** | | `sonar` | SONAR | Physics-Augmented | Dual XLS-R + SRM filters + JS contrastive head β€” **trained in-repo** | | `bicrossmamba_st` | BiCrossMamba-ST | State-Space | Bidirectional Mamba + Mutual Cross-Attention β€” **trained in-repo** | | `voiceradar` | VoiceRadar | Physics + Adv-Robust | Mel-CNN + physics aux head + PGD Ξ΅-stability probe β€” **trained in-repo** | | `holi_antispoof` | HoliAntiSpoof | Audio LLM (explainable) | Qwen2.5-Omni stub + rule-based natural-language rationale generator | The four "paper architecture" detectors ship with **real trained weights** (val_EER 0.5–5%) and switch to live mode automatically when the checkpoint files are present in `backend/checkpoints/`. If a checkpoint is missing, the detector falls back transparently to a paired live detector and labels the result `architecture_only_fallback` so nothing is silently faked. --- ## Quick start ### Prerequisites - **Python 3.10 or 3.11** (3.12 should work but is less battle-tested for the torch wheels) - **Node.js 18+** with `npm` - ~2 GB free disk for the HuggingFace cache (Wav2Vec2 weights + XLS-R) - Optional: NVIDIA GPU with CUDA 11.8+ for faster inference (CPU-only also works) ### Run locally β€” Windows (PowerShell) ```powershell # 1) Clone & enter git clone audio-deepfake-detector cd audio-deepfake-detector # 2) Backend β€” create venv + install deps cd backend python -m venv .venv .\.venv\Scripts\Activate.ps1 pip install --upgrade pip pip install -r requirements.txt # 3) Backend β€” first run (downloads ~750 MB of HF weights into the system HF cache) $env:ENABLE_HEAVY_MODELS = "true" python -m uvicorn app.main:app --host 127.0.0.1 --port 8000 # Leave this terminal open. The backend serves at http://127.0.0.1:8000 # 4) Frontend β€” IN A SECOND TERMINAL cd \audio-deepfake-detector\frontend npm install npm run dev # Frontend serves at http://localhost:5173 ``` Open **http://localhost:5173** and you're done. ### Run locally β€” macOS / Linux ```bash git clone audio-deepfake-detector cd audio-deepfake-detector # Backend cd backend python -m venv .venv source .venv/bin/activate pip install --upgrade pip pip install -r requirements.txt ENABLE_HEAVY_MODELS=true uvicorn app.main:app --host 127.0.0.1 --port 8000 # Frontend (second terminal) cd ../frontend npm install npm run dev ``` ### Run with Docker ```bash docker compose up --build # Frontend β†’ http://localhost:3000 # Backend β†’ http://localhost:8000 ``` The first Docker build downloads HF weights into a named volume; subsequent runs are fast. --- ## Deploy to Hugging Face Spaces (free, no credit card) A combined Dockerfile + `hf_spaces_entrypoint.py` at the repo root build the React frontend, copy it into the FastAPI container, and serve both on port 7860 (the HF Spaces convention). 1. Push this repo to GitHub first (so HF can clone it). 2. Visit https://huggingface.co/new-space and create a **Docker Space**: - Owner: your HF username - Space name: `audio-deepfake-detector` (or anything) - Space SDK: **Docker** - Hardware: free CPU is sufficient - Visibility: Public or Private (your choice) 3. After creation, push the same repo to the Space's git remote: ```bash git remote add space https://huggingface.co/spaces// git push space main ``` (HF prompts for a write-access token β€” generate one at https://huggingface.co/settings/tokens with the *write* scope.) 4. The Space rebuilds and the URL becomes `https://huggingface.co/spaces//` β€” the frontend serves from the root, the API at `/api/*`, docs at `/docs`. The YAML block at the top of this README tells HF to use Docker mode and expose port 7860; the entrypoint mounts `/app/static` so the SPA + API share the same origin. ### Other deploy options - **Backend on Render / Railway / Fly.io**: the existing `backend/Dockerfile` works on all three; their free tiers may be too small for the XLS-R model (~1.5 GB resident). Check current memory limits. - **Frontend on Vercel / Netlify**: deploy `frontend/` as a static site; point `VITE_API_BASE_URL` at wherever the backend lives. - **Self-hosted**: `docker compose up --build` on any VPS with β‰₯ 4 GB RAM. --- ## What you'll see - **Home (`/`)** β€” paper context, four-paradigm overview, three-deficits framing, light/dark theme toggle at the bottom. - **Demo (`/demo`)** β€” drop a clip / record from mic / pick a sample. Eight detectors run in parallel. Verdict + per-detector cards + waveform + mel spectrogram. Recording supports playback + replay + delete (cleared on refresh). - **Benchmarks (`/benchmarks`)** β€” paper Table III data verbatim, generalization-gap chart, datasetΓ—model coverage matrix, PGD adversarial-robustness curve, HoliAntiSpoof rationale examples. - **About (`/about`)** β€” full architecture descriptions for all 7 paper models with SVG diagrams, comparative Table II + III. The whole UI has a **light/dark theme toggle** (Sun/Moon switch at the bottom of Home and in the global footer). Choice persists via `localStorage`. --- ## API reference Full interactive docs: http://127.0.0.1:8000/docs | Method | Path | Purpose | |---|---|---| | GET | `/api/health` | Service status + per-detector load state | | GET | `/api/models` | All detectors with paradigm, params, EER refs | | POST | `/api/detect` | Upload audio (multipart), run all (or selected) detectors, return ensemble verdict | | POST | `/api/detect/sample/{id}` | Detect on one of the bundled sample clips | | POST | `/api/visualize` | Waveform envelope + mel spectrogram for the UI | | GET | `/api/benchmarks` | Paper Table III rows + generalization-gap data | | GET | `/api/benchmarks/datasets` | Dataset metadata | | GET | `/api/benchmarks/evaluations` | Dataset Γ— model Γ— metric matrix | | GET | `/api/benchmarks/adversarial` | PGD Ξ΅-curve data | | GET | `/api/benchmarks/explainability` | HoliAntiSpoof rationale examples | | GET | `/api/benchmarks/radar` | Radar-chart data (5 axes) | | GET | `/api/benchmarks/highlights` | Paper one-liners | | GET | `/api/samples` | Sample-library metadata | | GET | `/api/samples/{id}/download` | Sample WAV bytes | | WS | `/ws/inference/{session_id}` | Streamed inference progress | ### Example: detect a clip ```bash curl -F "audio_file=@my_clip.wav" \ -F 'models=["nes2net","sonar","lf_hf_physics"]' \ http://127.0.0.1:8000/api/detect | jq ``` --- ## Training the paper architectures The repo includes three training scripts. All run on CPU; GPU is optional but ~10Γ— faster. ### `train_asvspoof.py` β€” Nes2Net Three modes, picked automatically from the args: ```powershell # (1) Synthesised fallback β€” downloads small public LibriSpeech clips + # synthesises 5 fake families. Good for first-time setup. python train_asvspoof.py --n-real 80 --n-fake 80 --epochs 8 --batch-size 8 # (2) Directory mode β€” point at any dataset organised as real/ and fake/ folders python train_asvspoof.py ` --real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" ` --fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" ` --max-per-class 1000 --epochs 6 --batch-size 8 # (3) ASVspoof real-data mode β€” provide protocol + audio dir python train_asvspoof.py ` --protocol path\to\ASVspoof2019.LA.cm.train.trn.txt ` --audio-dir path\to\ASVspoof2019_LA_train\flac ` --epochs 12 --batch-size 8 ``` ### `train_paper_archs.py` β€” SONAR + BiCrossMamba-ST + VoiceRadar Each architecture has its own hyperparameter profile, picked automatically: | Architecture | LR | Epochs | Weight decay | Why | |---|---|---|---|---| | SONAR | 1e-4 | 5 | 5e-4 | 41M-param dual-stream cross-attention head β€” heavy regularisation, fewer epochs to avoid overfitting | | BiCrossMamba-ST | 5e-4 | 10 | 1e-4 | 516K param Mamba+MCA β€” small model, tolerates higher LR + more epochs | | VoiceRadar | 3e-4 | 10 | 2e-4 | 410K-param CNN β€” standard LR, modest weight decay | ```powershell # Synthesised fallback python train_paper_archs.py --n-real 80 --n-fake 80 --batch-size 8 # Directory mode (e.g. jay15k) python train_paper_archs.py ` --real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" ` --fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" ` --max-per-class 1000 --batch-size 8 # Train only one architecture python train_paper_archs.py --only sonar --batch-size 8 ``` ### `train_calibration.py` β€” meta-classifier (logistic regression on top of detector outputs) Run this AFTER training the paper architectures (or any time the detector mix changes): ```powershell # Synthesised pool python train_calibration.py --n-real 30 --n-fake 30 # Directory mode python train_calibration.py ` --real-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\real" ` --fake-dir "..\data\jay15k\deepfake_audio_dataset_jay15k\fake" ` --max-per-class 100 ``` Output: `backend/checkpoints/calibration.json`. Loaded automatically on next backend startup. ### Datasets - **ASVspoof 2019 LA** β€” ~5 GB, click-through registration: https://datashare.ed.ac.uk/handle/10283/3336 - **ASVspoof 2021 DF** β€” ~133 GB, challenge registration: https://www.asvspoof.org/ - **In-the-Wild (MΓΌller et al. 2022)** β€” 37.9 h: https://deepfake-total.com/in_the_wild - **deepfake_audio_dataset_jay15k** β€” 9.6K real + 7.2K fake (16 kHz mono), available on Kaggle / similar mirrors. Verified working with `train_asvspoof.py --real-dir / --fake-dir`. --- ## Configuration All env vars (also readable from a `backend/.env` file): | Variable | Default | Effect | |---|---|---| | `ALLOWED_ORIGINS` | `http://localhost:5173,http://localhost:3000,...` | CORS origins (CSV) | | `ENABLE_HEAVY_MODELS` | `true` | Disable to skip the two ~94 M-param Wav2Vec2 detectors. The LF/HF physics detector + paper architectures still run. | | `DEMO_MODE` | `false` | Reserved flag | | `HF_HOME` / `TRANSFORMERS_CACHE` | `/app/.cache/huggingface` | HuggingFace cache location | Frontend env (Vite): | Variable | Default | Effect | |---|---|---| | `VITE_API_BASE_URL` | `http://localhost:8000` | Backend HTTP base | | `VITE_WS_BASE_URL` | `ws://localhost:8000` | Backend WS base | --- ## Project layout ``` audio-deepfake-detector/ β”œβ”€β”€ backend/ β”‚ β”œβ”€β”€ app/ β”‚ β”‚ β”œβ”€β”€ main.py FastAPI app + lifespan + CORS β”‚ β”‚ β”œβ”€β”€ config.py Settings via pydantic-settings β”‚ β”‚ β”œβ”€β”€ schemas/ Pydantic request/response models β”‚ β”‚ β”œβ”€β”€ features/ Audio decode (incl. ffmpeg fallback for WebM/Opus), β”‚ β”‚ β”‚ XLS-R extractor, SRM filters, mel utilities, β”‚ β”‚ β”‚ multi-signal LF/HF physics analyser β”‚ β”‚ β”œβ”€β”€ models/ β”‚ β”‚ β”‚ β”œβ”€β”€ base_model.py BaseDetector + per-request FeatureCache β”‚ β”‚ β”‚ β”œβ”€β”€ hf_detectors.py MelodyMachine + motheecreator (calibrated) β”‚ β”‚ β”‚ β”œβ”€β”€ lf_hf_detector.py Deterministic physics analyser β”‚ β”‚ β”‚ β”œβ”€β”€ nes2net.py Nes2Net architecture + fallback wiring β”‚ β”‚ β”‚ β”œβ”€β”€ sonar.py SONAR head + 60/40 physics blending β”‚ β”‚ β”‚ β”œβ”€β”€ bicrossmamba_st.py Bidirectional Mamba + MCA β”‚ β”‚ β”‚ β”œβ”€β”€ voiceradar.py CNN + physics aux + PGD probe β”‚ β”‚ β”‚ β”œβ”€β”€ holi_antispoof.py Qwen2.5-Omni stub + rationale generator β”‚ β”‚ β”‚ β”œβ”€β”€ calibration_loader.py Reads checkpoints/calibration.json β”‚ β”‚ β”‚ └── model_registry.py Singleton; lazy warm-up; calibration apply β”‚ β”‚ β”œβ”€β”€ routers/ β”‚ β”‚ β”‚ β”œβ”€β”€ health.py /api/health, /api/models β”‚ β”‚ β”‚ β”œβ”€β”€ inference.py /api/detect, /api/detect/sample, /api/visualize β”‚ β”‚ β”‚ β”‚ with fitted-meta-classifier ensemble β”‚ β”‚ β”‚ β”œβ”€β”€ benchmarks.py /api/benchmarks family β”‚ β”‚ β”‚ β”œβ”€β”€ samples.py /api/samples + download β”‚ β”‚ β”‚ └── websocket.py WS /ws/inference/{session_id} β”‚ β”‚ └── utils/ β”‚ β”‚ β”œβ”€β”€ benchmark_data.py Paper Table III data β”‚ β”‚ β”œβ”€β”€ samples.py Sample registry β”‚ β”‚ β”œβ”€β”€ speech_synth.py Source-filter synth + 5 fake families β”‚ β”‚ └── sample_generator.py Boot bootstrapper β”‚ β”œβ”€β”€ checkpoints/ Trained weights (.pt + .json metadata) β”‚ β”œβ”€β”€ train_asvspoof.py Nes2Net trainer (synth / dir / ASVspoof modes) β”‚ β”œβ”€β”€ train_paper_archs.py SONAR + BiCrossMamba-ST + VoiceRadar trainer β”‚ β”œβ”€β”€ train_calibration.py Meta-classifier trainer β”‚ β”œβ”€β”€ requirements.txt β”‚ └── Dockerfile β”œβ”€β”€ frontend/ β”‚ β”œβ”€β”€ src/ β”‚ β”‚ β”œβ”€β”€ pages/ Home / Demo / Benchmarks / About β”‚ β”‚ β”œβ”€β”€ components/ InputTabs (Upload/Record/Sample), NowPlaying, β”‚ β”‚ β”‚ ModelCard, EnsembleVerdictBanner, β”‚ β”‚ β”‚ InferenceProgress, ThemeToggle, ConfidenceGauge, β”‚ β”‚ β”‚ Waveform/SpectrogramViewer, AudioUploader, β”‚ β”‚ β”‚ MicRecorder, SampleLibrary, Layout β”‚ β”‚ β”œβ”€β”€ hooks/ useAudioRecorder, useInference, useWebSocket, β”‚ β”‚ β”‚ useTheme, useThemeColors β”‚ β”‚ β”œβ”€β”€ store/inferenceStore.ts Zustand global store β”‚ β”‚ β”œβ”€β”€ lib/api.ts Axios client + URL helpers β”‚ β”‚ β”œβ”€β”€ lib/utils.ts cn(), formatters, status classifiers β”‚ β”‚ β”œβ”€β”€ types/inference.ts Shared TS types (mirror Pydantic) β”‚ β”‚ └── index.css CSS theme variables: dark + .theme-light overrides β”‚ β”œβ”€β”€ tailwind.config.ts Theme tokens via rgb(var(--name) / ) β”‚ β”œβ”€β”€ package.json β”‚ └── Dockerfile β”œβ”€β”€ data/ β”‚ └── sample_audios/ 6 bootstrap clips (auto-generated on first boot) β”œβ”€β”€ docker-compose.yml β”œβ”€β”€ .gitignore └── README.md (this file) ``` --- ## Performance End-to-end latency on a single request, all 8 detectors in parallel, CPU only: | Detector | Mean latency | |---|---| | MelodyMachine (Wav2Vec2) | ~700 ms | | motheecreator (Wav2Vec2) | ~700 ms | | LF/HF Physics | ~50 ms | | Nes2Net | ~1,200 ms (XLS-R extraction dominates) | | SONAR | ~0 ms with feature cache hit (else ~1,200 ms) | | BiCrossMamba-ST | ~300 ms | | VoiceRadar | ~300 ms | | HoliAntiSpoof | ~50 ms | | **Total wall-clock** | **~1.3 s** | Detectors run in parallel via `asyncio.gather`, and a per-request `FeatureCache` ensures XLS-R features are computed once and shared between Nes2Net + SONAR. ### Recent benchmark results Trained and evaluated on `deepfake_audio_dataset_jay15k` (1000 real + 1000 fake balanced split, 30% held-out validation): | Detector | Val EER | Val Acc | |---|---|---| | Nes2Net | 0.005 (0.5%) | 98.8% | | SONAR | 0.050 (5.0%) | 99.0% | | BiCrossMamba-ST | 0.010 (1.0%) | 99.3% | | VoiceRadar | 0.005 (0.5%) | 99.5% | Meta-classifier (`calibration.json`): test acc 98.3%, log-loss 0.077–0.083. End-to-end ensemble: 10/10 correct on 10 unseen jay15k clips with sharp separation (real β†’ P(fake) β‰ˆ 0.05; fake β†’ P(fake) β‰ˆ 0.95). --- ## Metric definitions - **EER (Equal Error Rate)** β€” operating point where FAR = FRR. Lower is better. - **minDCF** β€” minimum Detection Cost Function; cost-weighted error at optimal threshold. - **t-DCF** β€” tandem DCF; joint cost between ASV system and spoof detector. - **CLLR** β€” Cost of Log-Likelihood Ratio; calibration-aware error. - **TPR / TNR** β€” True Positive / Negative Rate (sensitivity / specificity). - **Pearson r (LF/HF)** β€” correlation between low- and high-frequency band energy time-series. Real speech β‰ˆ 0.6, synthesis β‰ˆ 0. --- ## Honest disclosure - The two production HuggingFace detectors are **public Wav2Vec2 fine-tunes** β€” calibrated in this repo with grid-fitted softmax temperature + logit bias. - Trained checkpoints for the four paper architectures (Nes2Net, SONAR, BiCrossMamba-ST, VoiceRadar) are produced by the in-repo training scripts on whatever data you point them at. Sample weights shipped here are trained on `deepfake_audio_dataset_jay15k` β€” performance on that distribution. Performance on ASVspoof 2021 DF / In-the-Wild requires re-training on those datasets. - HoliAntiSpoof runs in **rationale-only mode** β€” Qwen2.5-Omni weights aren't bundled (16 GB). The architecture is registered, its physics-derived prediction is kept, and the rationale text demonstrates the explainability paradigm. - `mamba-ssm` requires Linux + CUDA. On Windows / CPU, BiCrossMamba-ST uses an equivalent gated-conv substitute documented at runtime in the response `notes` field. - Every fallback / substitution is labelled in the API response. Nothing is silently faked. --- ## Citation ```bibtex @misc{kumar2025generalization, title = {The Generalization Gap in Audio Deepfake Detection: A Four-Paradigm Review}, author = {Vipan Kumar}, year = {2025}, note = {Chitkara University, Punjab} } ``` ## License Code: MIT. HuggingFace models retain their respective licences (`MelodyMachine/Deepfake-audio-detection`, `motheecreator/Deepfake-audio-detection`, `facebook/wav2vec2-xls-r-300m`).