backend/README.md

metadata

title: Deepfake Forensics API
emoji: 🕵️
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false

Deepfake Forensics API (Backend)

This is the FastAPI backend for the Deepfake Forensics Platform. It provides a high-performance REST API to process video and audio files, extracting multi-modal anomaly scores across 15 distinct forensic dimensions.

Features

• Security & Integrity: Integrates python-magic for true binary MIME-type validation to prevent malicious payload uploads, overriding simple file-extension spoofing.
• Scene-Cut Extraction: Utilizes PySceneDetect for intelligent, context-aware frame extraction across the entire video duration, defeating deepfakes hidden in scene transitions.
• Batched Inference & Lazy Loading: Models are lazy-loaded into VRAM upon request, and frames are processed in 32-frame batches to prevent Out-Of-Memory (OOM) crashes.
• Real-Time SSE Streaming: Yields real-time telemetry and granular module logs back to the client via Server-Sent Events.
• Optimized Face Tracking: Integrates robust OpenCV tracking (KCF/CSRT) after an initial MediaPipe detection to radically speed up face extraction.
• True SHAP Explanations: Employs shap.KernelExplainer on the Meta-Classifier to compute mathematically rigorous feature importance.
• Concurrent Processing: Utilizes ThreadPoolExecutor to handle heavy OpenCV frame extractions and multi-model inferences in parallel.
• Rate Limiting & Stability: Secured with slowapi to restrict endpoints and uses robust per-module error trapping to prevent pipeline crashes.
• REST API: Fully documented interactive Swagger API accessible at /docs.
• Report Generation: Aggregates scores into comprehensive PDF forensics reports.

Advanced Mathematical Methodologies

The pipeline scripts in this backend utilize strict mathematical extraction techniques:

• rPPG Analysis (rppg_analysis.py): Uses MediaPipe landmarks to generate precise geometric polygon masks over the left/right cheeks and forehead to extract the mean RGB values, bypassing background noise to isolate cardiovascular blood flow.
• CFA Demosaicing (cfa_analysis.py): Applies a custom 3x3 high-frequency diagonal residual filter matrix to isolate the microscopic Bayer interpolation grid. It then computes an 8x8 block variance map to compare facial noise vs background noise.
• Corneal Specular Highlights (corneal_analysis.py): Converts eye-cropped regions into LAB color space and thresholds the top 10% brightness of the L (Lightness) channel to geometrically isolate lighting reflections for left/right eye structural similarity comparison.
• A/V Sync (audio_sync.py): Translates 16kHz audio into 100Hz 13-dimensional MFCC arrays using librosa, allowing the 3D-CNN SyncNet to map 0.2-second audio chunks directly to 5-frame video mouth crops.

Structure

• main.py: The FastAPI application entry point.
• pipeline/: Contains the forensic extraction logic (audio, video, geometry, XAI, etc.).
• kaggle_scripts/: Python scripts designed for training models in Kaggle GPU environments (kaggle_efficientnet_training.py, kaggle_meta_training.py, kaggle_voice_training.py).
• weights/: Pre-trained .pth and .onnx model weights (EfficientNet-B4, SyncNet, Voice Liveness 2D-CNN, Meta-Classifier MLP).

Installation & Setup

1. System Dependencies:

   • Ensure you have ffmpeg installed and available in your system's PATH, as it is required for video and audio processing.

2. Create a Virtual Environment:

   python -m venv venv
   # On Windows:
   .\venv\Scripts\Activate.ps1
   # On Linux/Mac:
   source venv/bin/activate
   

3. Install Dependencies:

   pip install -r requirements.txt
   

4. Run the Server:

   uvicorn main:app --reload
   

The API will be available at http://127.0.0.1:8000.

Environment Variables

The application uses the following optional environment variables for configuration:

• API_KEY: Secures the API endpoints. (Defaults to "deepforensics-dev-key"). You must pass this in the x-api-key header when making requests.
• ALLOWED_ORIGINS: A comma-separated list of origins for CORS. (Defaults to "*").

Required Model Weights

Ensure the following pre-trained weight files are placed in the weights/ directory for full functionality:

• improved_finetuned_model.pth or finetuned_model.pth: Custom EfficientNet-B4 weights. If missing, the system falls back to the standard ImageNet pre-trained timm model.
• ensemble_mlp.pth: Weights for the Meta-Classifier MLP.
• voice_spoofing.pth: Weights for the Acoustic Anti-Spoofing 2D-CNN.
• syncnet_v2.model: Pre-trained Wav2Lip SyncNet model (can be downloaded from the Wav2Lip official repository).

API Specifications

• Upload Limit: Maximum file size is strictly capped at 100 MB.
• Supported Formats: mp4, avi, mov, mkv, webm, png, jpg, jpeg.
• Processing Time limit: Video analyses are capped at the first 60 seconds of playback to prevent memory overflow.