README.md

---
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: image-classification
tags:
  - synthetic-image-detection
  - clip
  - lora
---

# LEGO model card

- Paper: [OpenReview](https://openreview.net/pdf?id=WE4AHaFs6k)
- Code: [GitHub](https://github.com/seanstep/LEGO_LoRA_Enabled_Generator_Oriented_Framework)

## Model summary

LEGO is a research detector for binary classification of real and synthetic
images. It specializes multiple LoRA branches on different generator families
and uses a shared router to compose their residual features.

The released checkpoint is intended for research evaluation, not automated
content moderation, legal evidence, source attribution, or unsupervised
deployment.

## Architecture and CLIP parameters

- Base model: `openai/clip-vit-large-patch14`
- Vision architecture: ViT-L/14
- Input size: 224 × 224 RGB
- Patch size: 14 × 14
- Hidden size: 1,024
- Transformer layers: 24
- Attention heads: 16
- Frozen CLIP vision-backbone parameters: 303,179,776
- CLIP backbone: frozen during LEGO training
- Default LEGO branches: 3
- LoRA rank: 8
- LoRA scaling (`alpha`): 16
- LoRA targets: the linear projections in each CLIP self-attention block
- Router: shared MLP producing input-dependent branch weights
- Classification head: three serialized logits for released-checkpoint
  compatibility; training labels use only indices 0 (real) and 1 (fake)
- Output used for evaluation: fake probability at index 1; index 2 is unused

The checkpoint at `Sean-xyt/LEGO` contains the complete PyTorch model state:
the CLIP vision backbone, LoRA parameters, shared router, attention head, and
classification head. Its SHA-256 checksum is recorded in `weights/SHA256SUMS`.

For compatibility with the released checkpoint and reported scores, the
implementation retains one research-code detail that differs from the minimal
equations in the paper: router weights are normalized in the shared router and
again inside each LoRA hub. This does not introduce additional training data or
benchmark selection, but it should be reported when comparing a clean-room
reimplementation. Legacy checkpoints also contain duplicate router entries and
unused attention-pooling tensors; the loader accepts them, while newly saved
checkpoints omit those redundant tensors. The routing loss and L1 penalty in
this release follow the paper: soft cross-entropy is applied to router
probabilities, and the shared router parameters are regularized exactly once
(`lambda_reg=0.001`).

## Training data sources

The three specialization branches correspond to held-out subsets derived from
[GenImage](https://github.com/GenImage-Dataset/GenImage):

1. ADM
2. ProGAN
3. Stable Diffusion v1.4

Each prepared directory must contain real and fake samples under `0_real` and
`1_fake`. The loader consumes every supported image present in those folders;
there is no hidden sampling or hard-coded image-count cap. Users should report
the exact number of files used in each reproduction.

To reproduce the paper, these directories must contain the same held-out
GenImage subsets used by the experiment—fewer than 30,000 images in total. A
configuration pointed at the complete GenImage training release is a different
training protocol and must not be reported as the paper setting.

[AIGIBench](https://arxiv.org/abs/2505.12335) and
[Chameleon](https://arxiv.org/abs/2406.19435) are evaluation-only datasets.
They are not loaded by any training entry point. ProGAN is excluded from the
AIGIBench aggregate because ProGAN is represented during training.

No dataset images are distributed with this repository. Dataset copyrights,
consent conditions, access rules, and permitted uses remain governed by the
original providers.

## Preprocessing and optimization

Images are resized to 224 × 224 and normalized using CLIP statistics. Training
augmentation may include horizontal flip, JPEG compression, Gaussian blur,
brightness/contrast and hue/saturation perturbation, and Gaussian noise.
Fake–fake interpolation is used at low probability during router alignment.

The default optimizer is AdamW with learning rate `2e-5`, weight decay `5e-4`,
`beta1=0.9`, `beta2=0.95`, and `eps=1e-8`. Configuration files are the source of
truth for an individual run.

## Evaluation

Reported results use a fixed threshold of 0.5 for accuracy and macro-average
the per-subset AIGIBench accuracy over 24 subsets, excluding ProGAN. Chameleon
is evaluated independently after the checkpoint is fixed. See
`results/EVALUATION.md` for the released checkpoint summary.

## Limitations and risks

- Performance can degrade on unseen generators, image transformations,
  recompression, resizing, screenshots, or distributions unlike the benchmarks.
- A probability score is not proof that an image is synthetic or authentic.
- The model does not reliably identify the generator or provenance of an image.
- Thresholds are not calibrated for every domain or class prevalence.
- The released checkpoint retains an unused third classifier logit from the
  research implementation. It must not be interpreted as a third semantic
  class.
- The CLIP backbone may carry biases inherited from large-scale web data.
- Face-related benchmarks may not represent all demographic groups equally.
- False positives can harm creators and false negatives can miss manipulated
  content; human review and independent evidence are required for consequential
  decisions.
- The PyTorch `.pth` format uses serialization. Load checkpoints only from a
  trusted source and verify the published SHA-256 checksum.
- Commercial, surveillance, biometric, or high-stakes deployment has not been
  validated and is outside the intended scope of this release.

## License

LEGO-specific code, documentation, and released parameters are made available
for non-commercial research under CC BY-NC 4.0, subject to the rights of
third-party components. The release does not grant rights to the CLIP base
model or datasets beyond the terms supplied by their respective owners. See
`LICENSE` and `THIRD_PARTY_NOTICES.md` before redistribution or deployment.