ViT Deepfake Detector

AI-Powered Image Forensics | Vision Transformer | 96.56% Test Accuracy

An advanced computer vision model designed to distinguish authentic images from manipulated and AI-generated visual content. Built on Google's Vision Transformer (ViT-Base), this model leverages transfer learning and fine-tuning on the OpenForensics dataset to identify visual patterns associated with image manipulation.

Developed with a focus on accuracy, accessibility, and practical deployment, the model provides a lightweight foundation for image authenticity research, content verification pipelines, and computer vision applications.


Overview

The ViT Deepfake Detector is a binary image classification model that analyzes visual representations to predict whether an image is authentic or manipulated.

Built on google/vit-base-patch16-224-in21k, the model uses self-attention mechanisms to capture relationships across image patches, enabling it to learn complex visual features associated with manipulated imagery.

The model was fine-tuned on the OpenForensics dataset and evaluated on a held-out test set.

Key Highlights

  • 96.56% Test Accuracy on a held-out evaluation set.
  • Vision Transformer Architecture leveraging pretrained ViT representations.
  • 16,000 Training Images from the OpenForensics dataset.
  • 10-Epoch Fine-Tuning Pipeline using mixed-precision training.
  • Fast Inference through the Hugging Face Transformers ecosystem.
  • Ready for Integration into Python-based computer vision applications.

Model Specifications

Attribute Details
Architecture Vision Transformer (ViT-Base)
Base Model google/vit-base-patch16-224-in21k
Task Binary Image Classification
Classes Real / Fake
Training Dataset OpenForensics
Training Samples 16,000
Validation Samples 2,000
Test Samples 2,000
Input Resolution 224 ร— 224
Framework PyTorch / Hugging Face Transformers
Precision FP16 Mixed Precision
License MIT

Performance & Evaluation

The model was evaluated using a dedicated test set containing 2,000 images.

Benchmark Results

Evaluation Metric Score
Test Accuracy 96.56%
Validation Accuracy 96.22%
Validation F1 96.22%
Validation Precision 96.30%
Validation Recall 96.22%

Training Performance

The model demonstrated consistent convergence during fine-tuning, achieving its strongest validation accuracy at Epoch 5.

Epoch Training Loss Validation Loss Accuracy
1 0.2259 0.2567 92.89%
2 0.2002 0.2360 93.44%
3 0.1388 0.1925 96.11%
4 0.1322 0.2161 95.67%
5 0.1182 0.2208 96.22%

Note: Test accuracy is reported separately from validation performance.

Architecture

The model uses a pretrained Vision Transformer (ViT), which processes images as sequences of patches rather than relying exclusively on traditional convolutional operations.

The architecture consists of:

  1. Image Preprocessing: Images are resized and normalized using the pretrained ViT image processor.
  2. Patch Embedding: Input images are divided into fixed-size patches.
  3. Transformer Encoder: Self-attention layers learn spatial relationships and visual representations.
  4. Classification Head: The learned representations are mapped to two output classes.
  5. Prediction: Softmax probabilities provide the model's confidence scores for real and fake classifications.

Training Configuration

The model was fine-tuned using the following configuration:

Hyperparameter Value
Optimizer AdamW
Learning Rate 3e-5
Batch Size 24
Epochs 10
Weight Decay 0.02
Warmup Ratio 0.08
Scheduler Cosine
Label Smoothing 0.05
Mixed Precision FP16
Gradient Accumulation 1
Hardware NVIDIA Tesla T4
Training Duration Approximately 14 minutes

Preprocessing and normalization were performed using Hugging Face's ViTImageProcessor.

Quick Start

Get started with the model in just a few lines of Python.

1. Installation

pip install transformers torch pillow

2. Load the Model

import torch
from PIL import Image
from transformers import (
    ViTImageProcessor,
    ViTForImageClassification
)

MODEL_ID = "hamzenium/vit-deepfake-detector"

processor = ViTImageProcessor.from_pretrained(MODEL_ID)
model = ViTForImageClassification.from_pretrained(MODEL_ID)

model.eval()

3. Run Inference

image = Image.open("image.jpg").convert("RGB")

inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    probabilities = torch.softmax(outputs.logits, dim=-1)

predicted_class = probabilities.argmax(dim=-1).item()
confidence = probabilities[0][predicted_class].item()

labels = {0: "Real", 1: "Fake"}

print("Prediction:", labels[predicted_class])
print(f"Confidence: {confidence:.2%}")

4. Batch Inference

from transformers import pipeline

detector = pipeline(
    "image-classification",
    model=MODEL_ID
)

images = [
    "image1.jpg",
    "image2.jpg",
    "image3.jpg"
]

for image_path in images:
    result = detector(image_path)
    print(image_path, result)

Potential Applications

The model can serve as a building block for a variety of computer vision and digital forensics workflows.

  • Digital Image Forensics: Analyze images for potential manipulation.
  • Content Moderation: Support automated visual content screening.
  • Media Verification: Assist in evaluating the authenticity of digital imagery.
  • AI Research: Experiment with transformer-based approaches to synthetic image detection.
  • Computer Vision Pipelines: Integrate binary authenticity classification into larger ML systems.

Limitations & Responsible Use

Although the model demonstrates strong performance on its evaluation dataset, deepfake detection remains an evolving research challenge.

  • Dataset Generalization: Performance may vary on generation techniques and manipulation methods not represented in OpenForensics.
  • Domain Shift: Compression, image resizing, and different image sources may affect predictions.
  • Adversarial Robustness: The model has not been comprehensively evaluated against adversarial attacks.
  • Resolution Sensitivity: The model uses a 224 ร— 224 input resolution.
  • False Predictions: A high-confidence prediction does not establish the authenticity or provenance of an image.

This model should be treated as an analytical aid rather than a definitive authenticity verification system. Human review is recommended for consequential decisions.

Research & Development

This project explores the application of transformer-based architectures to digital image forensics, with an emphasis on accessible model distribution and reproducible inference.

The model is released to support further experimentation, benchmarking, and development within the computer vision and AI research communities.

Contributors

  • Dr. Lucy Liu
  • Muhammad Hamza Sohail
  • Ayaan Mohammed
  • Shadab Karim
  • Kirti Dhir

License

Released under the MIT License, allowing developers and researchers to use, modify, and integrate the model in accordance with the license terms.

Citation

If you use this model in research or development, please cite:

@misc{vit-deepfake-detector,
  title={ViT Deepfake Detector},
  author={Liu, Lucy and Sohail, Muhammad Hamza and
          Mohammed, Ayaan and Karim, Shadab and Dhir, Kirti},
  year={2024},
  publisher={Hugging Face},
  howpublished={https://huggingface.co/hamzenium/vit-deepfake-detector}
}

Built for experimentation. Released for the AI research community.

Downloads last month
211
Safetensors
Model size
85.8M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for hamzenium/ViT-Deepfake-Classifier

Finetuned
(2,589)
this model

Space using hamzenium/ViT-Deepfake-Classifier 1