GenD, re-implemented and trained under the MARS protocol
This is a GenD baseline from the MARS project: our re-implementation of GenD (Yermakov et al., WACV 2026), trained and evaluated under the same protocol as MARS. It is not the original authors' model or weights; training data, inputs, optimization and checkpoint selection follow the MARS protocol, so its numbers differ from those in the GenD paper.
GenD scores each frame with a Perception Encoder (PE-Core-L) in which only the normalization layers are trained. The pooled feature is L2-normalized and classified by one linear layer, and the clip score is the mean of the frame scores.
Training length. This checkpoint was trained for 10 epochs. The main comparison table in the MARS paper reports GenD under the shared 3-epoch recipe, where it reaches a 65.78 average AUROC; with 10 epochs it reaches 88.46 (below).
Files
| file | contents | size |
|---|---|---|
model.safetensors |
trained normalization layers and head; the rest of PE-Core-L is not included and is downloaded from timm on first use | 0.4 MB |
config.yaml |
model and data settings, read by the mars package |
1 KB |
model.onnx |
complete graph, encoder included; runs with ONNX Runtime alone | 1.28 GB |
Input
- Decode the video as RGB and keep every second frame (0, 2, 4, ...).
- Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is
padded by repeating its last frame;
n_validis the number of real frames. - Resize each frame to 336 x 336 (square, no crop, bicubic with antialiasing), scale to [0, 1], then to [-1, 1].
- The video score is the mean P(manipulated) over its clips.
Scores are not calibrated. If you need a decision, choose a threshold on validation data from your own setting.
Use with safetensors (PyTorch)
Requires the code repository:
git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from mars.config import load_yaml
from mars.data.sampling import clips_for_video
from mars.data.transforms import preprocess
from mars.data.video import count_frames, read_frames
from mars.models.factory import build_detector
folder = snapshot_download("vector-institute/mars-gend", allow_patterns=["config.yaml", "model.safetensors"])
raw = load_yaml(f"{folder}/config.yaml")
detector = build_detector(raw["model"]["name"], raw)
model = detector.build_network() # downloads PE-Core-L on first use
missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False)
assert not unexpected and all(key.startswith("encoder.") for key in missing) # only the frozen encoder weights are absent
model = model.cuda().eval()
data = detector.cfg.data
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
probs = []
for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride):
frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda()
valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
logits = model(frames, valid).logits
probs.append(logits.float().softmax(-1)[0, 1].item())
return sum(probs) / len(probs)
print(score("video.mp4"))
Use with ONNX (no PyTorch, no code repository)
pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime for CPU
| name | shape, type | meaning |
|---|---|---|
input frames |
(1, 64, 3, 336, 336), float32 | one clip in [-1, 1] |
input n_valid |
(1,), int64 | number of real frames in the clip, 1 to 64 |
output fake_prob |
(1,), float32 | P(manipulated) for the clip |
output logits |
(1, 2), float32 | authentic and manipulated logits |
Clip length, frame stride and image size are also stored in the file's metadata.
import av
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image
session = ort.InferenceSession(
hf_hub_download("vector-institute/mars-gend", "model.onnx"),
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
meta = session.get_modelmeta().custom_metadata_map
T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size"))
def to_input(rgb: np.ndarray) -> np.ndarray:
"""(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1]."""
x = rgb.astype(np.float32) / 255.0
if x.shape[:2] != (S, S):
channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)]
x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0)
return ((x - 0.5) / 0.5).transpose(2, 0, 1)
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
with av.open(path) as container:
kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0]
probs = []
for start in range(0, len(kept), T):
clip = kept[start : start + T]
n_valid = len(clip)
clip = clip + [clip[-1]] * (T - n_valid)
inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)}
probs.append(float(session.run(["fake_prob"], inputs)[0][0]))
return float(np.mean(probs))
print(score("video.mp4"))
The ONNX graph runs in float32, while the evaluation ran PyTorch in bfloat16, so scores differ
slightly. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with CUDA is
installed, import torch before creating the session also works.
Training
| training data | FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic) |
| validation | FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 10) |
| clips | 64 frames, every second frame, full frames at 336 x 336 |
| encoder | vit_pe_core_large_patch14_336.fb; every normalization layer is trained, the rest is frozen |
| loss | cross-entropy plus uniformity (weight 0.5) and alignment (weight 0.1) on the unit-norm features |
| optimization | AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 3,333 steps, 10 epochs, bfloat16, seed 0 |
| hardware | 4 x NVIDIA A40 |
Results
Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's score is the mean over its clips.
| test set | videos | AUROC | AP |
|---|---|---|---|
| FaceForensics++ (test split) | 700 | 96.68 | 99.11 |
| Celeb-DF v2 | 518 | 84.09 | 91.01 |
| DeepfakeTIMIT | 960 | 95.08 | 97.81 |
| UADFV | 98 | 97.00 | 97.53 |
| HiFiFace (paired with FF++ targets) | 280 | 76.42 | 78.91 |
| GenVideo (validation split, 10 generators) | 18,286 | 80.49 | 84.71 |
| DF26 (10 generators) | 2,691 | 89.44 | 98.27 |
| average | 88.46 |
License
The weights in this repository are released under Apache 2.0. The encoder is Meta's Perception Encoder (PE-Core-L), downloaded from timm under its own license (Apache 2.0).
Citation
Please cite the MARS paper for these weights and the protocol, and the GenD paper for the method.
@misc{radwan2026marsmultiscaleappearanceresidualstreams,
title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection},
author={Ahmed Y. Radwan and Shaina Raza},
year={2026},
eprint={2611.12508},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2611.12508},
}
@inproceedings{GenD,
title = {Deepfake detection that generalizes across benchmarks},
author = {Yermakov, Andrii and Cech, Jan and Matas, Jiri and Fritz, Mario},
year = 2026,
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
pages = {773--783}
}
- Downloads last month
- -
Model tree for vector-institute/mars-gend
Base model
timm/vit_pe_core_large_patch14_336.fb