MODEL_ARCHITECTURE.md

Model Architecture

AI vs Real Image Detector — 3-Branch Ensemble

Input: RGB image resized to 224×224, pixels in [0, 1]. Output: 2 logits → softmax. Label map: 0 = real, 1 = ai.

                        ┌─────────────────────────────┐
   224×224 RGB  ───────►│  Branch 1: CLIP  (frozen)    │──► 768
        │               └─────────────────────────────┘
        │               ┌─────────────────────────────┐
        ├──────────────►│  Branch 2: EfficientNet-B3   │──► 1536
        │               └─────────────────────────────┘
        │               ┌─────────────────────────────┐
        └──────────────►│  Branch 3: FFT-CNN           │──► 512
                        └─────────────────────────────┘
                                     │ concat (2816)
                                     ▼
                        ┌─────────────────────────────┐
                        │  Fusion MLP                  │
                        │  2816 → 512 → 128 → 2        │
                        └─────────────────────────────┘
                                     ▼
                              [P(real), P(ai)]

---

Branch 1 — CLIP (semantic view)

• Backbone: OpenAI CLIP ViT-L-14 visual encoder, fully frozen (requires_grad = False), run in eval() under torch.no_grad().
• Normalization: CLIP's own mean/std, applied as registered buffers.
• Adapter (trainable): Linear(768→768) → LayerNorm → GELU → Dropout(0.1) → Linear(768→768) → LayerNorm.
• Output: 768-d.
• Rationale: CLIP encodes high-level semantic structure; the adapter re-projects those features toward the real-vs-AI decision without disturbing the pretrained encoder (parameter-efficient, overfitting-resistant).

Branch 2 — EfficientNet-B3 (texture view)

• Backbone: torchvision EfficientNet-B3, ImageNet-pretrained, fine-tuned (features + avgpool).
• Normalization: ImageNet mean/std (registered buffers).
• Head: Dropout(0.4) after global average pooling.
• Output: 1536-d.
• Rationale: captures local texture and spatial artifacts — skin, edges, fine detail — where generators leave subtle inconsistencies.

Branch 3 — FFT-CNN (frequency view)

• Spectral transform: convert to grayscale (luma), 2D FFT, log1p(|·|) magnitude, fftshift, then per-image min-max normalize to [0, 1].
• CNN: 4 conv blocks 1→32→64→128→256 (3×3, BN, ReLU, MaxPool between the first three), AdaptiveAvgPool2d(1), flatten.
• Projection: Linear(256→512) → LayerNorm → GELU.
• Output: 512-d.
• Rationale: GAN/diffusion upsampling leaves periodic grid patterns in the frequency domain that are invisible in pixel space. This branch reads them directly. (Note: this is also why resizing an image — which injects interpolation artifacts — can fool the branch; see limitations.)

Fusion head

• Concatenate [768 + 1536 + 512] = 2816.
• Linear(2816→512) → BN → GELU → Dropout(0.4) → Linear(512→128) → BN → GELU → Dropout(0.2) → Linear(128→2).

Parameters

	Count
Total	~317,877,130
Trainable	~13,910,922

Trainable = CLIP adapter + EfficientNet + FFT branch + fusion head. The CLIP backbone (the bulk of the params) is frozen.

Training setup

• Optimizer: AdamW with per-branch learning rates — CLIP adapter 1e-5, EfficientNet 3e-5, FFT 1e-4, fusion head 1e-4; weight decay 1e-4.
• Scheduler: CosineAnnealingLR (eta_min = 1e-7).
• Loss: class-weighted CrossEntropy. Grad clipping: 1.0.
• Sampling: WeightedRandomSampler for class balance.
• Augmentation: horizontal flip, color jitter, random grayscale, Gaussian blur.

Checkpoint format

best_model.pt is a dict:

{"epoch": int, "model_state": state_dict, "val_f1": float, "cfg": {...}}

Load via model.py's load_detector(path, device), which rebuilds the network from the embedded cfg and loads model_state. AutoModel/pipeline() do not work — this is a custom nn.Module.