v1: two-backbone ensemble (ResNet50+VGG16), calibrated, TTA-evaluated

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.keras filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,79 @@
+---
+license: mit
+tags: [deepfake-detection, image-classification, forensics, tensorflow, keras]
+library_name: keras
+pipeline_tag: image-classification
+---
+
+# OpenForensics Deepfake Detector (v1)
+
+A multi-backbone CNN ensemble that classifies face crops as **Real** or
+**Fake**. Backbones: resnet50, vgg16. Their pooled embeddings are
+concatenated and read by a shared classifier head.
+
+## Output
+
+A single sigmoid: **P(Real)**. Fake is `1 - p`.
+
+The published threshold is **0.500** — a neutral default, **not** a fitted operating point. A temperature of **1.099** was fitted on validation and is applied.
+
+Thresholds fitted on the validation split do not transfer to the test split for this dataset (see Limitations). Pick your own operating point from `threshold_sweep` in the evaluation report, on data resembling your deployment.
+
+## Test metrics
+
+| Metric | Value |
+|---|---|
+| Accuracy | 0.8860 |
+| ROC-AUC | 0.9421 |
+| PR-AUC | 0.9527 |
+| Real images called fake | 175 (17.5%) |
+
+Measured on a held-out test split with horizontal-flip test-time
+augmentation. The split is content-hash deduplicated against train and
+validation, so no image appears in more than one split.
+
+## Input
+
+Resize to 224x224, scale to `[0, 1]`, shape `(N, 224, 224, 3)` float32.
+Per-backbone normalisation happens **inside** the model — do not apply
+`preprocess_input` yourself.
+
+```python
+from huggingface_hub import snapshot_download
+import tensorflow as tf, numpy as np, json
+from PIL import Image
+
+path = snapshot_download("adarshcod30/openforensics-ensemble")
+model = tf.keras.models.load_model(f"{path}/model.keras", compile=False)
+card = json.load(open(f"{path}/serving.json"))
+
+img = Image.open("face.jpg").convert("RGB").resize((224, 224))
+x = np.asarray(img, dtype="float32")[None] / 255.0
+p = float(model.predict(x)[0, 0])
+print("Real" if p >= card["decision"]["threshold"] else "Fake", p)
+```
+
+Loading needs the `PreprocessLayer` custom layer from
+[the repo](https://github.com/adarshcod30/OpenForensics), or pass it via
+`custom_objects`.
+
+## Training data
+
+The face-cropped OpenForensics distribution (190,334 images at 256x256).
+Training used light augmentation only: horizontal flip, small brightness and contrast jitter. No corruption-matched augmentation.
+
+## Limitations
+
+- Trained on **face crops**. Behaviour on full scenes or non-face images is
+  undefined.
+- A score near the threshold is not evidence. Treat the margin as part of
+  the output.
+- Performance degrades on manipulation methods absent from OpenForensics.
+- Research and educational use. Not a forensic authority.
+- **Validation does not predict test performance on this dataset.** Recall on genuine images at threshold 0.5 is 0.984 on validation but 0.825 on test. The 10th percentile of scores on genuine images is 0.975 on validation and 0.138 on test — a subset of test images is confidently misread rather than the whole distribution shifting. The test split carries degradations (desaturation, noise, blocking, blur) that train and validation do not. Expect calibration to drift on degraded inputs.
+
+## Citation
+
+> Trung-Nghia Le, Huy H. Nguyen, Junichi Yamagishi, Isao Echizen,
+> "OpenForensics: Large-Scale Challenging Dataset For Multi-Face Forgery
+> Detection And Segmentation In-The-Wild", ICCV 2021.

config.json

@@ -0,0 +1,41 @@
+{
+  "name": "v1",
+  "out_dir": "/Users/adarsh/Desktop/Projects/OpenForensics/runs",
+  "data": {
+    "base_dir": "/Users/adarsh/Desktop/Projects/OpenForensics/Dataset",
+    "train_per_class": 10000,
+    "val_per_class": 3000,
+    "test_per_class": 1000,
+    "batch_size": 32,
+    "seed": 12345,
+    "img_size": [
+      224,
+      224
+    ],
+    "corruption_prob": 0.0,
+    "max_corruptions": 0
+  },
+  "model": {
+    "backbones": [
+      "resnet50",
+      "vgg16"
+    ],
+    "head_units": 256,
+    "dropout_branch": 0.4,
+    "dropout_merge": 0.4,
+    "dropout_final": 0.3
+  },
+  "train": {
+    "epochs": 20,
+    "lr": 0.0002,
+    "finetune_epochs": 10,
+    "finetune_lr": 1e-05,
+    "unfreeze_last": 50,
+    "freeze_batchnorm": true,
+    "monitor": "val_auc",
+    "monitor_mode": "max",
+    "early_stop_patience": 7,
+    "reduce_lr_patience": 3
+  },
+  "provenance": "Legacy run. Weights predate this package and were trained by the original two-stage pipeline; this config records the evaluation settings, not a reproducible training recipe."
+}

evaluation_report.json

@@ -0,0 +1,305 @@
+{
+  "model": "/Users/adarsh/Desktop/Projects/OpenForensics/runs/exp1_finetune/best_model_finetuned.keras",
+  "threshold_sweep": [
+    {
+      "threshold": 0.05,
+      "accuracy": 0.853,
+      "real_called_fake": 77,
+      "fake_called_real": 217
+    },
+    {
+      "threshold": 0.1,
+      "accuracy": 0.8795,
+      "real_called_fake": 87,
+      "fake_called_real": 154
+    },
+    {
+      "threshold": 0.15,
+      "accuracy": 0.8885,
+      "real_called_fake": 103,
+      "fake_called_real": 120
+    },
+    {
+      "threshold": 0.2,
+      "accuracy": 0.8915,
+      "real_called_fake": 113,
+      "fake_called_real": 104
+    },
+    {
+      "threshold": 0.25,
+      "accuracy": 0.8925,
+      "real_called_fake": 124,
+      "fake_called_real": 91
+    },
+    {
+      "threshold": 0.3,
+      "accuracy": 0.894,
+      "real_called_fake": 132,
+      "fake_called_real": 80
+    },
+    {
+      "threshold": 0.35,
+      "accuracy": 0.895,
+      "real_called_fake": 139,
+      "fake_called_real": 71
+    },
+    {
+      "threshold": 0.4,
+      "accuracy": 0.8945,
+      "real_called_fake": 151,
+      "fake_called_real": 60
+    },
+    {
+      "threshold": 0.45,
+      "accuracy": 0.8905,
+      "real_called_fake": 164,
+      "fake_called_real": 55
+    },
+    {
+      "threshold": 0.5,
+      "accuracy": 0.886,
+      "real_called_fake": 175,
+      "fake_called_real": 53
+    },
+    {
+      "threshold": 0.55,
+      "accuracy": 0.8825,
+      "real_called_fake": 188,
+      "fake_called_real": 47
+    },
+    {
+      "threshold": 0.6,
+      "accuracy": 0.88,
+      "real_called_fake": 198,
+      "fake_called_real": 42
+    },
+    {
+      "threshold": 0.65,
+      "accuracy": 0.8755,
+      "real_called_fake": 210,
+      "fake_called_real": 39
+    },
+    {
+      "threshold": 0.7,
+      "accuracy": 0.8735,
+      "real_called_fake": 219,
+      "fake_called_real": 34
+    },
+    {
+      "threshold": 0.75,
+      "accuracy": 0.8675,
+      "real_called_fake": 231,
+      "fake_called_real": 34
+    },
+    {
+      "threshold": 0.8,
+      "accuracy": 0.8635,
+      "real_called_fake": 243,
+      "fake_called_real": 30
+    },
+    {
+      "threshold": 0.85,
+      "accuracy": 0.8515,
+      "real_called_fake": 273,
+      "fake_called_real": 24
+    },
+    {
+      "threshold": 0.9,
+      "accuracy": 0.836,
+      "real_called_fake": 309,
+      "fake_called_real": 19
+    },
+    {
+      "threshold": 0.95,
+      "accuracy": 0.807,
+      "real_called_fake": 377,
+      "fake_called_real": 9
+    }
+  ],
+  "manifest_digest": "599c942765ed9553",
+  "tta": true,
+  "calibration": {
+    "temperature": 1.0986092864209605,
+    "val_ece_raw": 0.005462571293115611,
+    "val_ece_calibrated": 0.002969743043184234
+  },
+  "operating_point": {
+    "criterion": "fixed",
+    "threshold": 0.5,
+    "rationale": "Thresholds fitted on validation do not transfer to this test split. At 0.5 recall on Real is 0.984 on validation but 0.825 on test; at 0.90 the gap widens to 0.951 vs 0.691. The 10th percentile of scores on genuine images is 0.975 on validation and 0.138 on test, so a subset of test images is confidently misread rather than the whole distribution shifting. 0.5 is published as a neutral default and the full threshold sweep ships alongside so callers can pick their own operating point on data resembling their deployment."
+  },
+  "test_at_0.5": {
+    "threshold": 0.5,
+    "accuracy": 0.886,
+    "roc_auc": 0.94214,
+    "pr_auc": 0.9527261743419954,
+    "confusion_matrix": [
+      [
+        947,
+        53
+      ],
+      [
+        175,
+        825
+      ]
+    ],
+    "real_called_fake": 175,
+    "fake_called_real": 53,
+    "false_accusation_rate": 0.175,
+    "ece": 0.05536929062008858,
+    "n": 2000,
+    "fake_recall": 0.947,
+    "real_recall": 0.825
+  },
+  "test_at_threshold": {
+    "threshold": 0.5,
+    "accuracy": 0.886,
+    "roc_auc": 0.94214,
+    "pr_auc": 0.9527261743419954,
+    "confusion_matrix": [
+      [
+        947,
+        53
+      ],
+      [
+        175,
+        825
+      ]
+    ],
+    "real_called_fake": 175,
+    "fake_called_real": 53,
+    "false_accusation_rate": 0.175,
+    "ece": 0.05536929062008858,
+    "n": 2000,
+    "fake_recall": 0.947,
+    "real_recall": 0.825
+  },
+  "risk_coverage": [
+    {
+      "coverage": 1.0,
+      "n": 2000,
+      "accuracy": 0.886,
+      "min_confidence": 0.5004317760467529
+    },
+    {
+      "coverage": 0.953,
+      "n": 1905,
+      "accuracy": 0.90498687664042,
+      "min_confidence": 0.6455079317092896
+    },
+    {
+      "coverage": 0.905,
+      "n": 1810,
+      "accuracy": 0.9187845303867404,
+      "min_confidence": 0.7839853763580322
+    },
+    {
+      "coverage": 0.858,
+      "n": 1715,
+      "accuracy": 0.9276967930029154,
+      "min_confidence": 0.8622756600379944
+    },
+    {
+      "coverage": 0.811,
+      "n": 1621,
+      "accuracy": 0.9358420727945712,
+      "min_confidence": 0.9077473878860474
+    },
+    {
+      "coverage": 0.763,
+      "n": 1526,
+      "accuracy": 0.9403669724770642,
+      "min_confidence": 0.9426376223564148
+    },
+    {
+      "coverage": 0.716,
+      "n": 1431,
+      "accuracy": 0.944095038434661,
+      "min_confidence": 0.9612756371498108
+    },
+    {
+      "coverage": 0.668,
+      "n": 1336,
+      "accuracy": 0.9505988023952096,
+      "min_confidence": 0.9726122617721558
+    },
+    {
+      "coverage": 0.621,
+      "n": 1242,
+      "accuracy": 0.9565217391304348,
+      "min_confidence": 0.9810938835144043
+    },
+    {
+      "coverage": 0.574,
+      "n": 1147,
+      "accuracy": 0.95640802092415,
+      "min_confidence": 0.9872699975967407
+    },
+    {
+      "coverage": 0.526,
+      "n": 1052,
+      "accuracy": 0.9562737642585551,
+      "min_confidence": 0.9904727339744568
+    },
+    {
+      "coverage": 0.479,
+      "n": 957,
+      "accuracy": 0.955067920585162,
+      "min_confidence": 0.9927610158920288
+    },
+    {
+      "coverage": 0.432,
+      "n": 863,
+      "accuracy": 0.9559675550405562,
+      "min_confidence": 0.9945045709609985
+    },
+    {
+      "coverage": 0.384,
+      "n": 768,
+      "accuracy": 0.9557291666666666,
+      "min_confidence": 0.9956018924713135
+    },
+    {
+      "coverage": 0.337,
+      "n": 673,
+      "accuracy": 0.9554234769687965,
+      "min_confidence": 0.996526300907135
+    },
+    {
+      "coverage": 0.289,
+      "n": 578,
+      "accuracy": 0.9532871972318339,
+      "min_confidence": 0.9974053502082825
+    },
+    {
+      "coverage": 0.242,
+      "n": 484,
+      "accuracy": 0.9504132231404959,
+      "min_confidence": 0.9980881810188293
+    },
+    {
+      "coverage": 0.195,
+      "n": 389,
+      "accuracy": 0.9408740359897172,
+      "min_confidence": 0.9986152648925781
+    },
+    {
+      "coverage": 0.147,
+      "n": 294,
+      "accuracy": 0.9421768707482994,
+      "min_confidence": 0.9990310668945312
+    },
+    {
+      "coverage": 0.1,
+      "n": 200,
+      "accuracy": 0.945,
+      "min_confidence": 0.9993615746498108
+    }
+  ],
+  "distribution_shift": {
+    "recall_real_val_at_0.5": 0.9837,
+    "recall_real_test_at_0.5": 0.825,
+    "p10_real_scores_val": 0.9748,
+    "p10_real_scores_test": 0.1375
+  }
+}

model.keras

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bd92b1c334b738ae0e4dfb92dde7a5b77d08b86680ea02ab6d2b2c05272459e
+size 157119017

serving.json

@@ -0,0 +1,44 @@
+{
+  "weights": "model.keras",
+  "input": {
+    "size": [
+      224,
+      224
+    ],
+    "channels": 3,
+    "dtype": "float32",
+    "range": [
+      0,
+      1
+    ],
+    "note": "Resize to size, divide by 255. Per-backbone normalisation happens inside the model."
+  },
+  "output": {
+    "name": "probability_real",
+    "range": [
+      0,
+      1
+    ],
+    "note": "P(image is Real). Fake is 1 - p."
+  },
+  "params": 39092929,
+  "decision": {
+    "threshold": 0.5,
+    "temperature": 1.0986092864209605,
+    "criterion": "fixed",
+    "calibrated": true
+  },
+  "tta": {
+    "recommended": true,
+    "transform": "horizontal_flip",
+    "note": "Average P(Real) over the image and its mirror.",
+    "used_in_eval": true
+  },
+  "test_metrics": {
+    "accuracy": 0.886,
+    "roc_auc": 0.94214,
+    "pr_auc": 0.9527261743419954,
+    "real_called_fake": 175,
+    "false_accusation_rate": 0.175
+  }
+}