<<<<<<< HEAD # UAIDE - AI-Generated Image & Video Detection A state-of-the-art deepfake detection system combining **EfficientNet-B2 + FFT Fusion** with comprehensive ethical assessment, Grad-CAM explainability, and a Gradio web interface. Supports both image and video analysis with multi-layered content safety checks. --- ## Model Performance Summary | Metric | Value | |--------|-------| | **Validation Accuracy** | 95.07% | | **Validation AUC** | 0.9908 | | **Validation Loss** | 0.0471 | | **Train Accuracy** | 83.91% | | **Train Loss** | 0.0770 | --- ## Quick Start ```bash # Create virtual environment python -m venv .venv # Activate (Windows PowerShell) .\.venv\Scripts\Activate.ps1 # Activate (Linux/Mac) source .venv/bin/activate # Install dependencies pip install -r requirements.txt pip install timm # Required for EfficientNet # Launch the web app python app.py ``` The app loads the trained model from `models_adv/best_model_weights.pt` and opens a Gradio UI at `http://localhost:7860`. --- ## Architecture ### EfficientNet-B2 + FFT Fusion Model The primary model fuses two complementary streams: ``` Input Image (224x224) │ ├──────────────────────┬────────────────────────┐ ▼ ▼ │ ┌───────────────┐ ┌───────────────┐ │ │ EfficientNet │ │ FFT │ │ │ B2 │ │ Extractor │ │ │ (1408 dim) │ │ (512 dim) │ │ └───────┬───────┘ └───────┬───────┘ │ │ │ │ └──────────┬───────────┘ │ ▼ │ ┌─────────────┐ │ │ Fusion │ │ │ Layer │ │ │ (1920→1024) │ │ └──────┬──────┘ │ ▼ │ ┌─────────────┐ │ │ Classifier │◄─────────────────────────────┘ │ (2 class) │ Grad-CAM └─────────────┘ ``` **Key Components:** 1. **Spatial Stream (EfficientNet-B2)**: Extracts high-level visual features using ImageNet-pretrained weights via `timm` library 2. **Frequency Stream (FFT)**: Analyzes frequency domain patterns that reveal GAN artifacts: - Magnitude statistics (mean, std, max, min) - Frequency band energies (low, mid-low, mid-high, high) - Threshold-based features for GAN fingerprint detection 3. **Fusion Layer**: Combines both streams with LayerNorm, GELU activation, and progressive dropout --- ## Training ### Advanced Training Script (`train_adv.py`) ```bash python train_adv.py \ --dataset "DeepfakeVsReal/Dataset" \ --backbone efficientnet_b2 \ --epochs 100 \ --batch_size 8 \ --lr 0.0001 ``` **Training Features:** - EfficientNet backbone (B0, B2, B4 supported) - FFT-based frequency domain analysis - Focal loss with adaptive class weighting - Exponential Moving Average (EMA) for better generalization - CutMix, Mixup, and RandAugment augmentation - Mixed precision training (AMP) - Cosine annealing with warm restarts - Test-Time Augmentation (TTA) - Early stopping with patience - Full classification report: AUC, Recall, Precision, F1, Confusion Matrix ### Quick Start Training ```bash python train_adv_quickstart.py --dataset "DeepfakeVsReal/Dataset" ``` --- ## Ethical Assessment UAIDE includes comprehensive ethical assessment for AI-generated content with **11 safety checks**: | Check | Description | Auto-Flag | |-------|-------------|-----------| | **Face Detection** | Detects human faces in AI content | `FACES_DETECTED` | | **NSFW Detection** | Skin exposure and explicit content analysis | `NSFW_CONTENT` | | **Age Estimation** | Protects against minor exploitation | `POTENTIAL_MINOR` | | **Celebrity Detection** | Image quality and symmetry analysis for impersonation | `POTENTIAL_CELEBRITY` | | **Emotion Analysis** | Facial expression manipulation scoring | `EMOTIONAL_MANIPULATION` | | **Metadata Analysis** | EXIF tampering and AI generation markers | `AI_METADATA_MARKERS` | | **Watermark Detection** | Detects watermark removal signs | `WATERMARK_REMOVAL` | | **Hate Symbol Detection** | Geometric patterns and color analysis | `POTENTIAL_HATE_SYMBOL` | | **Text Overlay Analysis** | Misleading text and clickbait detection | `MISLEADING_TEXT` | | **Document Forgery** | ID/document detection and forgery indicators | `DOCUMENT_DETECTED` | | **Jurisdiction Compliance** | Legal warnings for EU, US, UK, China, Korea, India | Region-specific | ### Ethical Classification ``` ETHICAL (Risk < 50%) ├── Safe for educational/artistic use ├── Clear synthetic artifacts visible └── No privacy/consent concerns UNETHICAL (Risk > 50%) ├── Faces detected → Consent required ├── High-quality deepfake → Misuse potential ├── NSFW content → Distribution restricted └── Document detected → Forgery risk ``` --- ## Video Detection Train ResNet-50 + LSTM on video datasets: ```bash python train_video.py \ --dataset "SDFVD/SDFVD" \ --out video_resnet_lstm.pt \ --frames_per_video 16 \ --epochs 10 ``` **Video Features:** - Temporal modeling with LSTM - Frame-level Grad-CAM visualization - Face tracking and cropping - Multi-frame analysis --- ## Project Structure ``` UAIDE/ ├── app.py # Gradio web interface ├── train_adv.py # Advanced EfficientNet training ├── train_adv_quickstart.py # Quick training script ├── train_video.py # Video model training ├── detector.py # Heuristic patch-based detector ├── ethical_assessment.py # Comprehensive ethical checks ├── video_model.py # ResNet-LSTM architecture ├── video_data.py # Video frame extraction ├── models_adv/ # Trained model weights │ ├── best_model_weights.pt # EfficientNet-B2 + FFT weights │ └── config.json # Model configuration ├── models_v2/ # Alternative model versions ├── DeepfakeVsReal/Dataset/ # Image dataset │ ├── Train/ │ ├── Validation/ │ └── Test/ └── requirements.txt ``` --- ## Technical Specifications | Component | Specification | |-----------|---------------| | **Framework** | PyTorch 2.0+ with CUDA support | | **Backbone** | EfficientNet-B2 (timm) | | **Backbone Features** | 1408 dimensions | | **FFT Features** | 512 dimensions (12 raw → 512 processed) | | **Fusion Dimension** | 1920 → 1024 → 512 → 2 | | **Input Size** | 224 x 224 (center crop, no resize) | | **Normalization** | ImageNet mean/std | | **Loss Function** | Focal Loss (gamma=2.0) | | **Optimizer** | AdamW (lr=1e-4, weight_decay=1e-4) | | **Scheduler** | Cosine Annealing with Warm Restarts | | **Regularization** | Dropout (0.4), LayerNorm, EMA | | **Augmentation** | CutMix, Mixup, RandAugment | | **Precision** | Mixed (FP16/FP32 AMP) | --- ## Dependencies ``` torch>=2.0.0 torchvision>=0.15.0 timm>=0.9.0 gradio>=4.0.0 opencv-python>=4.8.0 numpy>=1.24.0 scipy>=1.10.0 scikit-learn>=1.3.0 scikit-image>=0.21.0 Pillow>=10.0.0 pandas>=2.0.0 joblib>=1.3.0 ``` Install with: ```bash pip install -r requirements.txt pip install timm ``` --- ## Datasets | Dataset | Location | Contents | |---------|----------|----------| | DeepfakeVsReal | `DeepfakeVsReal/Dataset/` | Train/Validation/Test with Real and Fake folders | | AI vs Real img | `AI vs Real img/` | AI-generated art vs real art | | SDFVD | `SDFVD/SDFVD/` | Video dataset (`videos_real/`, `videos_fake/`) | --- ## Usage ### Web Interface ```bash python app.py ``` Opens Gradio UI with: - **Image Tab**: Upload images for deepfake detection - **Video Tab**: Analyze videos with frame-by-frame detection - **Ethical Assessment**: Automatic risk scoring and recommendations ### Python API ```python import torch import numpy as np from PIL import Image from torchvision import transforms # Load model device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model = torch.load('models_adv/best_model_weights.pt', map_location=device) model.eval() # Prepare image transform = transforms.Compose([ transforms.Resize((224, 224)), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) img = Image.open('test_image.jpg').convert('RGB') img_tensor = transform(img).unsqueeze(0).to(device) # Predict with torch.no_grad(): output = model(img_tensor) prob = torch.softmax(output, dim=1) fake_prob = prob[0, 1].item() print(f"AI-Generated Probability: {fake_prob:.2%}") ``` ### Ethical Assessment API ```python from ethical_assessment import EthicalAssessment, format_ethical_report import numpy as np # Load image as numpy array (H, W, C) normalized to [0, 1] img_arr = np.array(Image.open('image.jpg')).astype(np.float32) / 255.0 # Run comprehensive assessment assessment = EthicalAssessment.assess(img_arr) # Print report print(format_ethical_report(assessment)) # Access specific checks print(f"Is Ethical: {assessment['is_ethical']}") print(f"Risk Score: {assessment['risk_score']:.1%}") print(f"Flags: {assessment['flags']}") print(f"NSFW Score: {assessment['checks']['nsfw']['nsfw_score']:.2f}") ``` --- ## Troubleshooting | Problem | Solution | |---------|----------| | `ModuleNotFoundError: timm` | `pip install timm` | | CUDA out of memory | Reduce batch size or use CPU | | Model loading fails | Check `models_adv/best_model_weights.pt` exists | | Slow inference | Enable GPU with `torch.cuda.is_available()` | | Real images flagged as AI | Adjust ethical threshold slider in UI | --- ## Citation If you use UAIDE in your research, please cite: ```bibtex @software{uaide2024, title={UAIDE: AI-Generated Image and Video Detection}, author={Deshna}, year={2024}, url={https://github.com/Deshnaa2007/UAIDE} } ``` --- ## License MIT License - See LICENSE file for details. --- ## Repository https://github.com/Deshna24/UAIDE ======= # UAIDE — AI-Generated Image & Video Detection UAIDE is a deepfake detection toolkit that combines a ResNet-50 + FFT feature-fusion model with Grad-CAM explainability, ethical assessment, and a Gradio web interface. It supports both image and video analysis, with auto-calibrated thresholds to minimise false positives on real images. ## Quick Start ```powershell python -m venv .venv .\.venv\Scripts\Activate.ps1 pip install -r requirements.txt # Launch the web app python app.py ``` The app loads the trained fusion model (`model_fusion_best.joblib_info.pkl`), auto-calibrates a detection threshold against the validation set, and opens a Gradio UI where you can upload images for analysis. ## How It Works ### Architecture The primary model (`DeepfakeFeatureFusion`) fuses two streams: 1. **Spatial stream** — ResNet-50 backbone extracts high-level visual features. 2. **Frequency stream** — Block-wise FFT (16x16 blocks) produces per-block magnitude and phase statistics, processed by a small CNN. Phase information captures alignment errors that AI generators leave behind. A **cross-attention** layer lets the spatial stream guide where to look for frequency anomalies. The fused representation passes through a classification head with progressive dropout. ### Preprocessing Images are **padded and center-cropped** (224x224) instead of resized, preserving the original pixel-level detail that resize-based pipelines destroy. ### Training Augmentations - Random crop, flip, rotation, colour jitter, affine - JPEG compression (quality 50–95) and Gaussian blur — simulates real-world image degradation so the model works on compressed uploads ### Threshold Calibration At startup, `app.py` runs the real validation images through the model and sets a threshold at the 95th percentile of their fake-class probabilities, capping the false-positive rate on real images at ~5%. ## Trained Model The default model is stored across these files: | File | Contents | |------|----------| | `model_fusion_best.joblib_info.pkl` | Model metadata (type, state-dict path, optimal threshold) | | `model_fusion_best.joblib_best_improved` | PyTorch state dict | Stored metrics from training: - **Best F1**: 0.792 - **Best fake recall**: 0.681 - **Optimal threshold**: 0.371 Run `python evaluate_model.py` to compute full accuracy and ROC AUC on the Validation and Test splits. ## Training Default settings reproduce the shipped model: ```powershell python train.py --dataset "DeepfakeVsReal/Dataset" --max_per_class 1000 ``` This trains the `fusion` model type by default and writes `model_fusion_best.joblib_best_improved` + `model_fusion_best.joblib_info.pkl`. Other model types are available via `--model`: | `--model` | Architecture | |-----------|-------------| | `fusion` (default) | ResNet-50 + block-wise FFT + cross-attention | | `resnet` | ResNet-50 transfer learning | | `cnn` | Custom 4-layer CNN | | `cnn_kfold` | CNN with K-fold cross-validation | | `fusion_dual` | Dual-stream residual + ResNet | | `rf` | Random Forest (handcrafted features) | | `gb` | XGBoost with GPU support | ## Video Detection Train a ResNet-50 + LSTM on the SDFVD dataset: ```powershell python train_video.py --dataset "SDFVD/SDFVD" --out video_resnet_lstm.pt --frames_per_video 16 --epochs 10 ``` Run inference with Grad-CAM overlays: ```powershell python predict_video.py --video path\to\video.mp4 --checkpoint video_resnet_lstm.pt ``` ## Project Structure ``` UAIDE/ ├── app.py # Gradio web interface ├── train.py # Training (all model types) ├── train_video.py # Video model training ├── predict_video.py # Video inference + Grad-CAM ├── detector.py # Heuristic patch-based detector ├── ethical_assessment.py # Ethical risk scoring ├── evaluate_model.py # Validation / Test evaluation ├── compare_models.py # Side-by-side model comparison ├── diagnose_misclassification.py # Threshold sweep & FP analysis ├── print_report.py # Ethical classification report ├── video_model.py # ResNet-LSTM architecture ├── video_data.py # Video frame extraction ├── check_gpu.py # GPU availability check ├── requirements.txt ├── DeepfakeVsReal/Dataset/ # Train / Validation / Test splits ├── AI vs Real img/ # Additional AI art dataset ├── SDFVD/ # Video dataset └── model_fusion_best.* # Trained model artifacts ``` ## Datasets | Dataset | Location | Contents | |---------|----------|----------| | DeepfakeVsReal | `DeepfakeVsReal/Dataset/` | Train / Validation / Test splits with Real and Fake folders | | AI vs Real img | `AI vs Real img/` | AI-generated art vs real art | | SDFVD | `SDFVD/SDFVD/` | `videos_real/` and `videos_fake/` for video detection | ## Technical Details - **Framework**: PyTorch with CUDA support - **Backbone**: ResNet-50 (ImageNet pretrained) - **Frequency features**: Block-wise FFT magnitude + phase (16x16 blocks, 6-channel input) - **Attention**: Multi-head cross-attention (8 heads, 512-dim) - **Loss**: Focal loss (alpha=0.8, gamma=2.5, label smoothing=0.15) - **Optimiser**: AdamW with per-layer learning rates and cosine annealing - **Input**: 224x224 center crop (no resize) - **Regularisation**: Dropout (0.3–0.5), batch normalisation, weight decay, mixup (alpha=0.2) ## Troubleshooting | Problem | Fix | |---------|-----| | CUDA out of memory | Reduce `--max_per_class` or use smaller batch size | | Real images flagged as AI | The auto-threshold should handle this; if not, lower `TARGET_REAL_FPR` in `app.py` | | Grad-CAM errors | Ensure `opencv-python` is installed | | Slow startup | Threshold calibration runs on validation set at launch; reduce `MAX_CALIB_IMAGES` in `app.py` | ## Repository https://github.com/Deshnaa2007/UAIDE >>>>>>> 65ab9814191b6bb448da441c53a768594e7d1d59