fakeshield/docs/video_lab_documentation.md

📹 FakeShield: AI Video Forensic Lab Architecture & Technical Documentation

This document explains the technical implementation, model pipeline, and processing workflow of the AI Video Forensic Lab (Process 5.0) in the FakeShield system.

---

⚙️ Video Lab Processing Architecture Diagram

The flowchart below demonstrates the sequential and parallel data flow from the moment a user uploads a video file down to database logging.

graph TD
    %% Styling Definitions
    classDef step fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#fff;
    classDef model fill:#1e1b4b,stroke:#818cf8,stroke-width:2px,color:#fff;
    classDef logic fill:#18181b,stroke:#a1a1aa,stroke-width:2px,color:#fff;

    %% System Nodes
    VideoInput["User Video Input (MP4/WEBM Binary)"]:::step
    Loader["1.0 Video Loader & Splitter (Extract Audio & Frame Sampler)"]:::step
    ThreadPool["2.0 Parallel Thread Pool Execution (ThreadPoolExecutor)"]:::step

    %% Parallel Analyzers
    M_Spatial["2.1 Spatial Texture Ensemble (CLIP + SigLIP)"]:::model
    M_Temporal["2.2 Temporal Motion Engine (RAFT Optical Flow)"]:::model
    M_LipSync["2.3 Lip-Sync Alignment Module (Whisper + Mediapipe)"]:::model
    M_Reason["2.4 Physical Reasoning Engine (Moondream2 VLM)"]:::model
    M_Sensor["2.5 Sensor Noise & Frequency Auditor (PRNU + FFT)"]:::model

    %% Synthesis & Out
    Fusion["3.0 Adaptive Fusion Engine (Weighted Penalty Logic)"]:::logic
    MongoDB[("D1.5: video_forensics Collection (MongoDB)")]:::logic
    Output["UI Diagnostic Response (JSON Verdict & Heatmaps)"]:::logic

    %% Workflows
    VideoInput --> Loader
    Loader -->|Frames Stream| ThreadPool
    Loader -->|Extracted Audio| ThreadPool
    
    ThreadPool --> M_Spatial
    ThreadPool --> M_Temporal
    ThreadPool --> M_LipSync
    ThreadPool --> M_Reason
    ThreadPool --> M_Sensor

    M_Spatial --> Fusion
    M_Temporal --> Fusion
    M_LipSync --> Fusion
    M_Reason --> Fusion
    M_Sensor --> Fusion

    Fusion --> MongoDB
    Fusion --> Output

---

🔬 Core Stages of the Video Forensic Pipeline

1. Video Loading & Preprocessing (video_router.py)

• Frame Sampling: To guarantee sub-5s processing on standard CPU workloads, the video router limits analysis to 8-16 keyframes distributed evenly across the video length.
• Audio Demuxing: Uses ffmpeg or local sub-processes to isolate the audio track as a WAV file, which is mapped directly to the Lip-Sync module.

---

🛰️ Multi-Model Analyzer Specifications

🚀 Deep Learning Analyzers (Spatial, Temporal, & Reasoning)

A. Spatial Texture Ensemble (video_clip.py)

• Models: openai/clip-vit-base-patch32 (weight: 40%) & google/siglip-base-patch16-224 (weight: 60%).
• Method: Performs zero-shot classification at the frame level. Contrastive text prompt matrices are mapped to the images:
  • Real Prompts: "a frame from a real video recorded by a camera", "natural video footage".
  • AI Prompts: "a frame from an AI-generated synthetic video", "synthetic content from Sora/Runway".
• Aggregator: Rather than using a mean average (which dilutes localized frame edits like face swaps), it takes the 75th percentile score.

B. Temporal Flow Engine (video_tempo_raft.py)

• Model: torchvision.models.optical_flow.raft_small (pretrained).
• Method: Computes dense optical flow fields between consecutive frame pairs.
• Heuristics:
  1. PAVR (Peak-to-Average Velocity Ratio): Detects unnatural temporal morphing spikes.
  2. Flow Entropy: Measures spatial randomness of pixel movements.
  3. Temporal Coherence Residual: Computes the mean difference between consecutive flow vectors.
• Artifact Output: Converts raw optical flow tensors into BGR/RGB HSV optical flow heatmaps returned to the React frontend as base64 images.

C. Lip-Sync Alignment Module (video_audio.py)

• Speech Detection: Uses OpenAI's Whisper ("base" model) with greedy search parameters (beam_size=1, best_of=1) to extract precise word-level speaking timestamps from the audio track.
• Lip Motion Tracking: Uses Mediapipe Tasks FaceLandmarker (face_landmarker.task) to track lip openness over time by measuring the distance between upper lip indices [61, 185, 40, ...] and lower lip indices [146, 91, 181, ...] normalized by facial height (landmarks 10 to 152).
• Correlation Mismatch: Checks if lip movement aligns with the speech activity timeline. A high viseme-to-phoneme mismatch rate indicates dubbing or deepfake face synthesis.

D. Physical Reasoning Engine (video_reasoning.py)

• Model: Moondream2 (vikhyatk/moondream2 revision 2024-08-26).
• Method: Evaluates geometric and physical inconsistencies in target frames (shadow errors, floating pixels, merging features).
• Consolidated Forensic Prompt: "Analyze this frame for AI anomalies (warping, shadows, blurring). Keep response strictly under 15 words. End with exactly 'CONSISTENT' or 'INCONSISTENT'."
• Decision Rule: Scans the VLM output text for suspicious terms (warp, ghost, blur, inconsistent) to calculate the score.

---

🎛️ Digital Signal Processing (DSP) Analyzers

E. Sensor Noise & Frequency Auditor (video_forensics_v2.py)

• PRNU Consistency Check: Photo Response Non-Uniformity (PRNU) acts as a camera sensor fingerprint. The module extracts frame noise residuals using a Gaussian high-pass filter: $$\text{Noise} = \text{Frame}{\text{gray}} - \text{GaussianFilter}(\text{Frame}{\text{gray}}, \sigma=2.0)$$ It calculates the cosine correlation coefficient between consecutive frame residuals. Real video shows high cross-correlation due to the shared physical camera sensor. AI-generated video (where frames are generated independently) displays zero correlation.
• 2D FFT Spectral Decay: Applies a 2D Fast Fourier Transform (np.fft.fft2) on frames, mapping radial average power profiles. GAN/Diffusion generators leave periodic high-frequency checkerboard noise grid signatures, warping the expected $1/f^2$ spectral decay law.

---

🧠 The Adaptive Fusion Engine (video_fusion.py)

Unlike static classifiers, the Video Fusion Engine dynamically changes model weights based on the video's vertical resolution height ($h$):

1. Dynamic Weight Matrix

Resolution	Spatial (CLIP/SigLIP)	Temporal (RAFT)	Audio (Lip-Sync)	Forensic (PRNU/FFT)	Reasoning (VLM)
Low ($<480p$)	20%	20%	30%	5%	25%
Standard ($480p - 720p$)	25%	25%	20%	15%	15%
High ($\ge 1080p$)	25%	30%	15%	20%	10%

Rationale: In low-resolution files, fine sensor noise details (PRNU/FFT) are washed out by compression, so the engine relies heavily on audio lip-sync and VLM reasoning. In high-resolution files, high-frequency spatial and temporal vectors are highly reliable.

2. Consistency Penalty

If the spatial score (static texture) and temporal score (flow consistency) disagree significantly ($|\text{Score}{\text{spatial}} - \text{Score}{\text{temporal}}| > 0.40$), the engine applies a +0.10 penalty to the final probability, reflecting cross-modal morphing instabilities.

3. Verdict Mappings

• $\ge 75%$: DEEPFAKE (CRITICAL Threat)
• $55% - 74%$: LIKELY FAKE (HIGH Threat)
• $35% - 54%$: UNCERTAIN (MEDIUM Threat)
• $< 35%$: LIKELY REAL (LOW Threat)