docs/CAPABILITIES.md

SynthScope capabilities (orientation for an agent building a new detector)

SynthScope is a reproduction + benchmarking library for AI-generated image and video detection. It hosts many published detectors behind one interface, reproduces each on its own benchmark, and serves a web UI + a Python/CLI API. Read this before adding a method, then follow docs/ADD_DETECTOR.md for the exact checklist.

Environment: everything runs in the SynthScope conda env — /home/ssalehi/miniconda3/bin/conda run -n SynthScope <cmd>. Config is Hydra (conf/).

---

1. What's already integrated (17 detectors)

synthscope.list_models() / from synthscope.api import list_specs. Each is an SynthScope<Name>Model with a matching conf/model/<modality>/<key>.yaml, a trainer.py._build_model branch, an interface/backend/registry.py ModelSpec, a per-model README.md, and a tests/test_<k>_smoke.py.

key	modality	predict_kind	weights	interp map	method
clip	image	ce	synthscope	–	UnivFD / CLIP linear probe
cnn	image	ce	synthscope	–	CNNDetection (ResNet-50)
dinov2,dinov3,adapted_dino	image	ce	synthscope	–	DINO probes / adapter-tuned
npr	image	bce_npr	official	npr_residual	Neighboring-Pixel Relationships
safe	image	ce	synthscope	safe_hf	High-freq DWT artifacts
aide	image	ce	official	aide_dct	ConvNeXt-XXL + DCT/SRM (native-res 5-view)
fat_former	image	ce	official	–	Forgery-aware adaptive transformer (CLIP)
spai	image	bce	synthscope*	spai_patch	Spectral, any-resolution (MFM ViT-B/16)
co_spy	image	bce	synthscope*	–	Semantic + artifact fusion (SigLIP + SD-VAE)
pgc	image	bce	official	pgc_peak	LoRA-DINOv2 + YCbCr residual, peak-guided
cure	image	ce	official	–	Frozen PE-Core L/14 + Q/V LoRA + source-response subspace (mixing-ratio regression)
vib_net	image	vib	synthscope	–	Variational information bottleneck (FLAVA)
d3	video	video	training-free	–	Disentangled detail dynamics (X-CLIP)
restrav	video	video	synthscope	–	Representation-trajectory geometry (DINOv2)
stall	video	video	training-free	–	Spatio-temporal whitened log-likelihoods (DINOv3)

* served via official weights remapped into the SynthScope class. predict_kind sets the score convention the engine applies to forward() logits: ce→softmax[:,1], bce→sigmoid, bce_npr (NPR residual transform then sigmoid), vib (tuple out), video (clip in).

---

2. The detector contract (what a new method must implement)

models/<modality>/<key>/<key>.py:

class SynthScope<Name>Model(nn.Module):
    def __init__(self, config: DictConfig): ...        # build arch from config.ModelConfig
    def forward(self, images) -> logits: ...           # RAW logits, not probs; model-specific
                                                       # preprocessing (residual/DWT/DCT) goes INSIDE forward
    def compute_loss(self, batch) -> {"loss", "outputs"}  # outputs = [N,2] real/fake logits, used by training

Repo rule: all imports + helper classes at module top level (no nested defs/imports). Vendor the official arch pieces verbatim where possible and document deviations in the per-model README.

To wire it in: conf/model/<modality>/<key>.yaml (model_type: <key>), a branch in trainer.py._build_model (dispatches on model_type), and a ModelSpec in the interface registry (see ADD_DETECTOR.md). The web engine (interface/backend/engine.py) lazy-imports by model_type.

---

3. Training infrastructure

• trainer.py (SynthScopeModelTrainer) + main.py entrypoint; Hydra config in conf/config.yaml with groups model/, dataset/, trainer/. Multi-GPU via 🤗 accelerate (accelerate_config.yaml).
• dataset.py — image/video dataset loaders; _decode_rgb feeds uint8 [3,H,W] tensors; sample_video_frames(..., edge_crop=) for video.
• transformations.py — composable named transforms / augmentations: resize_224, resize_short_{256,384,448}, center_crop_{224,256,384,448}, random_crop_*, pad_random_crop_224 (PGC), gaussian_blur, jpeg_compression, random_horizontal/vertical_flip, random_rotation_180, color_jitter_safe, to_tensor/to_dtype. Eval-transform fidelity is critical — artifact/ low-level detectors (SAFE/AIDE/SPAI/PGC/NPR) are wrecked by a wrong resize; match the official eval exactly (see repro/investigate/pgc_gap_investigation.md and the SAFE crop-not-resize lesson).
• metrics.py — torchmetrics binary Accuracy (real/fake), AveragePrecision (AP), AUROC. Threshold-free AP/AUROC are the headline metrics (esp. video, where per-gen real:fake ratios make mAP degenerate — prefer AUROC there).

---

4. Evaluation / benchmarking

• Image: benchmark/bench_eval.py --model <k> --benchmark {forensynths|ufd|pergen|pool} --data <root> [--ckpt <pth>] [--native-res] --tag <t>. Auto-discovers <gen>/{0_real,1_fake} (or {ai,nature}, shared-reals) layouts; prints per-generator acc/AP/AUROC + mean; writes repro/bench_results/<tag>.json.
• Video: benchmark/bench_eval_video.py --model {d3|restrav|stall} --dataset NAME (auto-download) or --benchmark {genvideo|fakeonly|pergen} --data DIR.
• --ckpt overrides the registry checkpoint (for official-vs-SynthScope side-by-side).
• benchmark/aggregate_results.py, leaderboard.py, evaluate.py, schema.json aggregate runs.
• Results live in repro/: RESULTS_SUMMARY.md (read first), AGGREGATED_BENCHMARK.md, BENCHMARK_COVERAGE_MATRIX.md, REPRODUCED_MODELS_BENCHMARKS.md, repro/bench_results/*.json, and per-method deep-dives in repro/investigate/.

---

5. Datasets (explorer registry: interface/backend/datasets.py, DATASET_DEFS)

Known image sets: cnndetection (ForenSynths/ProGAN), genimage, genimage_full (8-gen), universialfakedetection (Ojha UFD), aigibench, synthbuster, ditfake, diffusionforensics, diffusion1kstep, aigcdetect, commgen15, realchain, evalgen, synthwildx, wildrf, bfree_online, rrdataset, cospy/cospy_subset, spai_ldm, freqnetdetection. Video: genvideo_val, genvideo, videophy, vidprom, restrav_data, evalcrafter, genvidbench, videofeedback, comgenvid.

Each DatasetDef lists candidate roots; _resolve_root is node-portable — it remaps the configured /hddstore path onto whatever scratch mount exists (/hddstore↔/ssdstore↔/nvmestore), overridable with SYNTHSCOPE_DATA_PREFIXES. Labels are inferred from path tokens (0_real/nature→real, 1_fake/ai→fake). Not every dataset is physically present on every node — data is node-local scratch; check what's staged (e.g. CNNDetection, GenImage, GenImage_full are on cn029 /ssdstore).

---

6. The synthscope package (Python + CLI)

import synthscope
det = synthscope.load("pgc")                  # or load(key, checkpoint=PATH, device=...)
det.predict("img.png")                  # -> {fake_probability, verdict, confidence, interpret_map, ...}
synthscope.list_models(); synthscope.available_models()   # registered / weights-present

CLI: synthscope list, synthscope detect <file> --model <k>, synthscope pull <k> (fetch weights from the HF model zoo), synthscope push (maintainer). Weights resolve under $SYNTHSCOPE_CKPT_DIR (default /hddstore/ssalehi/SynthScope/checkpoints; override per node) or are auto-downloaded.

---

7. Web interface (interface/, FastAPI + uvicorn)

bash interface/run_backend.sh (SYNTHSCOPE_PORT, SYNTHSCOPE_HOST). Serves frontend/dist if built, else the no-build static/index.html. Endpoints (interface/backend/app.py): /api/models, /api/predict (image/video, lazy model load + LRU), /api/datasets[.../sample], /api/predict_dataset, /api/score_batch, /api/leaderboard, /api/generator_benchmarks, and a SOTA-vs-detector demo (/api/sota/{generators,real_image,compare}) that generates an image from a prompt (generators.py, API-keyed SOTA generators incl. local Ideogram) and scores real vs generated across detectors. On-cluster launch scripts: repro/webui_cn029_serve.sbatch (GPU on free cees node, checkpoints staged to /ssdstore), repro/webui_cn023_cpu.sbatch. Reach it via SSH tunnel through the ivi-h1 login host.

Interpretability maps

The engine returns a per-detector artifact map for the 5 native-signal detectors (spec.interpret): safe_hf (DWT high-freq band), npr_residual (pixel residual), pgc_peak (peak-guided residual score), aide_dct (native-res DCT patch energy), spai_patch (per-patch spectral fake-evidence). Each yields interpret_map (bare jet heatmap) + interpret_overlay (heatmap blended on the input, transform-aligned). A new method exposes one by setting interpret="<kind>" on its ModelSpec and adding a branch in engine._interpret_map that produces a 2-D saliency tensor.

---

8. New detectors & repo hygiene

New detectors are integrated on a detector/<key> branch and pushed for review — never auto-merged to main (see ADD_DETECTOR.md). Never commit weights (*.pth), .env, or caches.

---

9. Cluster cheat-sheet (IvI SLURM)

• cn023 (quva/quvausers, A6000) holds all detector checkpoints + most datasets on node-local /hddstore; GPUs are often booked for days. SYNTHSCOPE_CKPT_DIR=/hddstore/ssalehi/SynthScope/checkpoints.
• cees (ceesusers: cn029 etc., a5000, free) and cees6000 (cn030/31, rtx6000) use node-local /ssdstore / /nvmestore. To run there, stage checkpoints/data from cn023 (rsync -aL, dereference the symlinks) — note pam_slurm_adopt only allows ssh to a node where you hold an active job.
• /ivi/zfs/... is the shared home (24 GiB user quota — don't park big checkpoints there); HF cache is shared at /ivi/.../SynthScope/.cache_shared/hf (HF_HOME). DINOv3 weights gated, under dinov3_checkpoints/ (DINO_V3_WEIGHTS); SPAI MFM backbone via SYNTHSCOPE_MFM_CKPT.

---

10. Starting a new method — TL;DR

1. Confirm it passes both rules (public repo + strong results). Read ADD_DETECTOR.md.
2. git checkout -b detector/<key>; add the 5 files (model, conf, trainer dispatch, registry ModelSpec, README) + smoke test.
3. Match the official eval transform exactly in the ModelSpec (resize vs crop, norm) — this is the #1 silent-failure source.
4. Reproduce on the paper's benchmark with official weights via bench_eval.py --ckpt; fix remap_<k> for key mismatches; record per-gen AP/AUROC in the Paper/SynthScope_official/SynthScope_train schema.
5. (Optional) add an interpretability map. Push the branch; do not merge to main.