MUmairAB/deepfake-detection-ff-cn-transformer

Deepfake Detection on FaceForensics++ — CNN, Transformer, and Hybrid Models

This repository contains the trained checkpoints for a controlled study of video-level deepfake detection on FaceForensics++ (C23). It hosts five models: four established baselines and one proposed Hybrid CNN–Transformer.

The study is a scientific investigation with a negative result: adding a temporal Transformer on top of a strong per-frame CNN backbone does not improve in-domain detection or cross-manipulation generalization under the limited data and compute budget used here. These weights are released for reproducibility and educational purposes, not as a production-ready detector.

---

Models

The five models form a 2×2 design matrix over {pure-CNN, has-Transformer} × {spatial-only, spatiotemporal}:

Checkpoint	Architecture	Params	Temporal?
xception.pt	Xception (frame-averaged logits)	20.8M	No (spatial-only)
efficientnet_b3.pt	EfficientNet-B3 (frame-averaged logits)	10.7M	No (spatial-only)
cnn_lstm.pt	ResNet-18 + 2-layer BiLSTM	13.7M	Yes (recurrent)
vit_base.pt	ViT-Base/16 (frame-averaged logits)	85.8M	No (spatial-only)
hybrid_cnn_transformer.pt	EfficientNet-B3 + 4-layer Transformer encoder	74.9M	Yes (attention)

The proposed hybrid uses the same EfficientNet-B3 backbone as the EfficientNet baseline, so comparing the two isolates the effect of the temporal Transformer head (a controlled ablation).

---

Results

In-domain test (5 trained classes, shared test split):

Model	AUC	Accuracy	Precision	Recall	F1
Xception	0.9944	0.9744	0.9823	0.9858	0.9840
EfficientNet-B3	0.9976	0.9829	0.9825	0.9964	0.9894
CNN+BiLSTM	0.9805	0.9744	0.9823	0.9858	0.9840
ViT-Base/16	0.7066	0.6695	0.8910	0.6690	0.7642
Hybrid (ours)	0.9458	0.8462	0.9710	0.8327	0.8966

Cross-manipulation (in-domain AUC → held-out FaceShifter AUC):

Model	In-domain AUC	FaceShifter AUC	AUC drop
Xception	0.9944	0.7015	+0.2929
EfficientNet-B3	0.9976	0.7343	+0.2633
CNN+BiLSTM	0.9805	0.7591	+0.2214
ViT-Base/16	0.7066	0.6880	+0.0186
Hybrid (ours)	0.9458	0.6273	+0.3185

Key findings: EfficientNet-B3 is the strongest in-domain model; the hybrid scores below its own backbone; every model degrades sharply on the unseen manipulation; and the hybrid generalizes worst of all. ViT's small AUC drop is an artifact of its already-weak in-domain performance, not real robustness.

---

Intended use and scope

• Intended use: research and educational reproduction of the accompanying study; baseline checkpoints for further experiments on FaceForensics++.
• Out of scope: real-world or forensic deepfake detection, content moderation, or any decision affecting individuals. These models do not generalize to unseen manipulation types and must not be relied upon as a trustworthy detector.

---

How to use

Each checkpoint stores a model_state_dict plus the test metrics. The model class definitions live in the project notebooks (see the code repository). A typical load looks like:

import torch

ckpt = torch.load("hybrid_cnn_transformer.pt", map_location="cpu", weights_only=False)
model.load_state_dict(ckpt["model_state_dict"])   # `model` is the matching architecture
model.eval()
print(ckpt.get("metrics"))

Inputs are clips of shape [T=24, 3, 224, 224], ImageNet-normalized, produced by sampling 24 frames per video, detecting the face with MTCNN, cropping with a 30% margin, and resizing to 224×224. The frame-averaging models (Xception, EfficientNet, ViT) average per-frame logits; CNN+BiLSTM and the hybrid output a single clip-level logit directly. Apply a sigmoid to obtain the fake probability.

---

Training details

All five models were trained with an identical protocol for a fair architectural comparison:

• Loss: binary cross-entropy on the clip-level logit
• Optimizer: AdamW, lr 1e-4, weight decay 1e-4
• Schedule: cosine annealing over 25 epochs; early stopping (patience 5) on validation AUC; best checkpoint by validation AUC
• Mixed precision (fp16 autocast + gradient scaling)
• Class imbalance (~4:1 fake:real) handled with an inverse-frequency WeightedRandomSampler
• Augmentation (train only): horizontal flip (p=0.5), brightness jitter (±10%), ImageNet normalization
• Seed 42; trained on a single NVIDIA RTX 5080 (16 GB) with PyTorch 2.11 / CUDA 12.8

No per-model hyperparameter sweep was performed; this is a deliberate fair-comparison choice and a stated limitation. TensorBoard training logs are included in this repository.

---

Dataset

FaceForensics++ at the C23 (high-quality) compression level.

• Training classes: real original videos + Deepfakes, Face2Face, FaceSwap, NeuralTextures (700 videos per class)
• Held out for cross-manipulation testing: FaceShifter (400 videos), never seen during training

FaceForensics++ is distributed under its own research license. Obtain the dataset from its official source and comply with its terms; these weights are a derivative of that data and inherit its usage restrictions.

---

Limitations and ethical considerations

• Poor cross-manipulation generalization. Every model collapses on the held-out FaceShifter attack (AUC ~0.63–0.76). High in-domain accuracy does not imply a general notion of "manipulated."
• Not for deployment. These models can produce confident false negatives on unseen forgeries and false positives on genuine video. Do not use them to make consequential judgments about real people or content.
• Dataset bias. Performance reflects the actors, capture conditions, and manipulation methods present in FaceForensics++ and may not transfer elsewhere.
• Dual-use awareness. Detection research can inform evasion; please use responsibly.

---

Citation

If you use these checkpoints, please cite the accompanying report and the FaceForensics++ dataset:

@misc{deepfake_ff_cnn_transformer,
  title  = {Spatiotemporal Deepfake Detection with Cross-Manipulation
            Generalization: A Hybrid CNN--Transformer Approach on FaceForensics++},
  author = {Umair Akram and Ibrahim Dodo},
  year   = {2026},
  note   = {deepfake research study}
}

@inproceedings{rossler2019ff,
  title     = {FaceForensics++: Learning to Detect Manipulated Facial Images},
  author    = {R\"ossler, Andreas and Cozzolino, Davide and Verdoliva, Luisa and
               Riess, Christian and Thies, Justus and Nie{\ss}ner, Matthias},
  booktitle = {ICCV},
  year      = {2019}
}