README.md

metadata

license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: image-classification
tags:
  - synthetic-image-detection
  - clip
  - lora

LEGO model card

• Paper: OpenReview
• Code: GitHub

Model summary

LEGO is a research detector for binary classification of real and synthetic images. It specializes multiple LoRA branches on different generator families and uses a shared router to compose their residual features.

The released checkpoint is intended for research evaluation, not automated content moderation, legal evidence, source attribution, or unsupervised deployment.

Architecture and CLIP parameters

• Base model: openai/clip-vit-large-patch14
• Vision architecture: ViT-L/14
• Input size: 224 × 224 RGB
• Patch size: 14 × 14
• Hidden size: 1,024
• Transformer layers: 24
• Attention heads: 16
• Frozen CLIP vision-backbone parameters: 303,179,776
• CLIP backbone: frozen during LEGO training
• Default LEGO branches: 3
• LoRA rank: 8
• LoRA scaling (alpha): 16
• LoRA targets: the linear projections in each CLIP self-attention block
• Router: shared MLP producing input-dependent branch weights
• Classification head: three serialized logits for released-checkpoint compatibility; training labels use only indices 0 (real) and 1 (fake)
• Output used for evaluation: fake probability at index 1; index 2 is unused

The checkpoint at Sean-xyt/LEGO contains the complete PyTorch model state: the CLIP vision backbone, LoRA parameters, shared router, attention head, and classification head. Its SHA-256 checksum is recorded in weights/SHA256SUMS.

For compatibility with the released checkpoint and reported scores, the implementation retains one research-code detail that differs from the minimal equations in the paper: router weights are normalized in the shared router and again inside each LoRA hub. This does not introduce additional training data or benchmark selection, but it should be reported when comparing a clean-room reimplementation. Legacy checkpoints also contain duplicate router entries and unused attention-pooling tensors; the loader accepts them, while newly saved checkpoints omit those redundant tensors. The routing loss and L1 penalty in this release follow the paper: soft cross-entropy is applied to router probabilities, and the shared router parameters are regularized exactly once (lambda_reg=0.001).

Training data sources

The three specialization branches correspond to held-out subsets derived from GenImage:

1. ADM
2. ProGAN
3. Stable Diffusion v1.4

Each prepared directory must contain real and fake samples under 0_real and 1_fake. The loader consumes every supported image present in those folders; there is no hidden sampling or hard-coded image-count cap. Users should report the exact number of files used in each reproduction.

To reproduce the paper, these directories must contain the same held-out GenImage subsets used by the experiment—fewer than 30,000 images in total. A configuration pointed at the complete GenImage training release is a different training protocol and must not be reported as the paper setting.

AIGIBench and Chameleon are evaluation-only datasets. They are not loaded by any training entry point. ProGAN is excluded from the AIGIBench aggregate because ProGAN is represented during training.

No dataset images are distributed with this repository. Dataset copyrights, consent conditions, access rules, and permitted uses remain governed by the original providers.

Preprocessing and optimization

Images are resized to 224 × 224 and normalized using CLIP statistics. Training augmentation may include horizontal flip, JPEG compression, Gaussian blur, brightness/contrast and hue/saturation perturbation, and Gaussian noise. Fake–fake interpolation is used at low probability during router alignment.

The default optimizer is AdamW with learning rate 2e-5, weight decay 5e-4, beta1=0.9, beta2=0.95, and eps=1e-8. Configuration files are the source of truth for an individual run.

Evaluation

Reported results use a fixed threshold of 0.5 for accuracy and macro-average the per-subset AIGIBench accuracy over 24 subsets, excluding ProGAN. Chameleon is evaluated independently after the checkpoint is fixed. See results/EVALUATION.md for the released checkpoint summary.

Limitations and risks

• Performance can degrade on unseen generators, image transformations, recompression, resizing, screenshots, or distributions unlike the benchmarks.
• A probability score is not proof that an image is synthetic or authentic.
• The model does not reliably identify the generator or provenance of an image.
• Thresholds are not calibrated for every domain or class prevalence.
• The released checkpoint retains an unused third classifier logit from the research implementation. It must not be interpreted as a third semantic class.
• The CLIP backbone may carry biases inherited from large-scale web data.
• Face-related benchmarks may not represent all demographic groups equally.
• False positives can harm creators and false negatives can miss manipulated content; human review and independent evidence are required for consequential decisions.
• The PyTorch .pth format uses serialization. Load checkpoints only from a trusted source and verify the published SHA-256 checksum.
• Commercial, surveillance, biometric, or high-stakes deployment has not been validated and is outside the intended scope of this release.

License

LEGO-specific code, documentation, and released parameters are made available for non-commercial research under CC BY-NC 4.0, subject to the rights of third-party components. The release does not grant rights to the CLIP base model or datasets beyond the terms supplied by their respective owners. See LICENSE and THIRD_PARTY_NOTICES.md before redistribution or deployment.