Buckets:
| license: cc-by-nc-sa-4.0 | |
| pretty_name: "AIForge-Doc v2 — GPT-Image-2 Document Forgeries" | |
| language: | |
| - en | |
| size_categories: | |
| - 1K<n<10K | |
| task_categories: | |
| - image-classification | |
| - image-segmentation | |
| tags: | |
| - scam-ai | |
| - document-forgery | |
| - document-forensics | |
| - gpt-image-2 | |
| - ai-generated | |
| - tampering-detection | |
| - document-fraud | |
| gated: true | |
| extra_gated_heading: "Access Scam.AI Research Dataset" | |
| extra_gated_description: | | |
| This dataset is released by Scam.AI for academic research and non-commercial use. | |
| Please share a couple of details so we can understand how the community is using our work. | |
| Access is granted automatically after submission. | |
| extra_gated_button_content: "Agree and access dataset" | |
| extra_gated_fields: | |
| Full Name: text | |
| Email: text | |
| I agree to use this dataset for non-commercial research purposes only and to cite the corresponding paper if I publish results based on it: checkbox | |
| # AIForge-Doc v2: A Paired Benchmark of GPT-Image-2 Document Forgeries | |
| [](https://creativecommons.org/licenses/by/4.0/) | |
| []() | |
| []() | |
| []() | |
| **AIForge-Doc v2** is the first paired benchmark of document forgeries produced by | |
| **OpenAI's GPT-Image-2** (released April 2026). Every forged image is accompanied by | |
| its authentic source image and a pixel-precise tampered-region mask in | |
| DocTamper-compatible format. v2 reuses the forgery specifications of | |
| [AIForge-Doc v1](https://github.com/scamai/aiforge-doc) spec-for-spec and swaps only | |
| the generator, so any difference in detector behaviour between v1 and v2 is | |
| attributable to the generator change rather than to the documents being tampered. | |
| --- | |
| ## At a Glance | |
| | Attribute | Value | | |
| |---|---| | |
| | Total forged images | **3,066** | | |
| | Training split | 2,453 (80 %) | | |
| | Testing split | 613 (20 %) | | |
| | Authentic baseline images | 3,066 (one per forgery) | | |
| | AI inpainting tool | OpenAI GPT-Image-2 (April 2026) | | |
| | Source datasets | CORD v2, WildReceipt, SROIE, XFUND | | |
| | Document types | Receipts (86.4 %), Forms (13.6 %) | | |
| | Languages | 9 (EN, ID, DE, IT, ES, FR, PT, ZH, JA) | | |
| | Output format | DocTamper-compatible (binary grayscale masks) | | |
| | Mask convention | `0` = authentic · `255` = tampered pixel | | |
| | Paired with v1 | Yes — same spec list, same source images, different generator | | |
| --- | |
| ## Directory Layout | |
| ``` | |
| AIForge-Doc-v2.0/ | |
| ├── TrainingSet/ | |
| │ ├── images/ # 000000001.png … 000002453.png | |
| │ └── masks/ # same filenames; 0=authentic, 255=tampered | |
| ├── TestingSet/ | |
| │ ├── images/ # 000000001.png … 000000613.png | |
| │ └── masks/ | |
| ├── metadata.jsonl # full provenance for every image (see schema below) | |
| ├── dataset_info.json # machine-readable dataset metadata | |
| └── README.md # this file | |
| ``` | |
| File names are **9-digit zero-padded integers** (`000000001.png`), identical to the | |
| DocTamper dataset convention so that existing evaluation pipelines require no | |
| modification. | |
| --- | |
| ## Provenance — `metadata.jsonl` Schema | |
| Each line is a JSON object with the following fields: | |
| | Field | Type | Description | | |
| |---|---|---| | |
| | `spec_id` | str | Unique forgery spec identifier (shared with v1) | | |
| | `image_id` | str | Original image ID from source dataset | | |
| | `source_dataset` | str | `cord` / `wildreceipt` / `sroie` / `xfund` | | |
| | `doc_type` | str | `receipt` or `form` | | |
| | `language` | str | ISO 639-1 language code | | |
| | `field_name` | str | Annotation key of the tampered field | | |
| | `original_value` | str | Ground-truth text before tampering | | |
| | `forged_value` | str | Synthesised replacement text | | |
| | `bbox_xyxy` | list[int] | Tampered region `[x1, y1, x2, y2]` in full-image pixels | | |
| | `assigned_tool` | str | `openai-gpt-image-2` | | |
| | `v1_assigned_tool` | str | The v1 generator for the same spec, for cross-generator analysis | | |
| | `split` | str | `training` or `testing` | | |
| | `new_id` | str | 9-digit zero-padded filename stem | | |
| | `image` | str | Path of the forged image inside this package | | |
| | `mask` | str | Path of the ground-truth mask inside this package | | |
| | `generated_at` | str | ISO 8601 timestamp | | |
| --- | |
| ## Dataset Statistics | |
| ### By Inpainting Tool | |
| | Tool | API Provider | Images | Share | | |
| |---|---|---|---| | |
| | OpenAI GPT-Image-2 (`openai-gpt-image-2`) | OpenAI (April 2026) | 3,066 | 100 % | | |
| ### By Source Dataset | |
| | Source | Document Type | Images | Languages | | |
| |---|---|---|---| | |
| | WildReceipt | Receipt | 1,336 | EN | | |
| | CORD v2 | Receipt | 983 | ID | | |
| | SROIE | Receipt | 329 | EN | | |
| | XFUND | Form | 418 | DE, IT, ES, FR, PT, ZH, JA | | |
| ### By Language | |
| | Language | Code | Images | | |
| |---|---|---| | |
| | English | `en` | 1,665 | | |
| | Indonesian | `id` | 983 | | |
| | Italian | `it` | 80 | | |
| | German | `de` | 78 | | |
| | Spanish | `es` | 67 | | |
| | Portuguese | `pt` | 62 | | |
| | French | `fr` | 56 | | |
| | Chinese | `zh` | 38 | | |
| | Japanese | `ja` | 37 | | |
| ### Acceptance Rate | |
| The v1 spec catalogue contains 4,062 forgery specs; 3,066 of them passed deterministic | |
| upstream constraints on the GPT-Image-2 endpoint. The remaining 996 (24.5 %) failed | |
| \emph{geometric} rather than content-based constraints: ~94 % were rejected by the | |
| endpoint's `invalidAspectRatio` check (width-to-height bound `[1:3, 3:1]`, which rules | |
| out long-receipt context crops); ~4 % violated the `[128, 2048]`-pixel reference-height | |
| bound; and < 2 % were retry-exhausted timeouts. Selection is therefore on \emph{shape}, | |
| not on content. | |
| | Source | v1 specs | v2 produced | Acceptance | | |
| |---|---:|---:|---:| | |
| | CORD v2 | 1,000 | 983 | 98.3 % | | |
| | WildReceipt | 1,696 | 1,336 | 78.8 % | | |
| | SROIE | 946 | 329 | 34.8 % | | |
| | XFUND | 420 | 418 | 99.5 % | | |
| | **Total** | **4,062** | **3,066** | **75.5 %** | | |
| --- | |
| ## Forgery Generation Pipeline | |
| Each forgery is created in four steps to prevent global image drift: | |
| 1. **Field selection** — A numeric or key field (price, date, ID, phone) is chosen from | |
| the source annotation and a plausible replacement value is generated using v1's | |
| mutation rules (monetary fields scaled by `U(1.15, 3.0)` or `U(0.20, 0.85)`; dates | |
| perturbed within calendar bounds; document IDs digit-flipped). | |
| 2. **Context crop** — The bounding box is expanded 50 % on each side (minimum 100 px) | |
| to provide font and colour context for the inpainting model. | |
| 3. **Composite-marker mask + inpainting** — A 3-pixel **green outline** (not a fill) is | |
| drawn on the context crop to mark the editing region; v1's red fill caused chromatic | |
| bleed, the green outline does not. GPT-Image-2 is then driven with a fixed outer | |
| wrapper that reminds the model the rectangle is an overlay marker, plus an inner | |
| spec-specific layer that pins down character fidelity in five clauses (declared | |
| aspect ratio; exact target string with character-by-character spelling; no | |
| reformatting; typography matching; non-mask pixels visually identical). | |
| 4. **Patch-back** — Only the exact field bbox region is pasted back into the full image; | |
| the ground-truth mask marks those pixels as `255`. Every emitted PNG is rebuilt from | |
| raw pixel data so the file contains no metadata that could leak provenance to a | |
| detector. | |
| --- | |
| ## Baseline Results | |
| Evaluated on the **TestingSet** (613 forged + 613 authentic paired images). Numbers | |
| quoted on the full v2 paired test partition are from the accompanying paper (Wu et al., | |
| 2026). | |
| ### Image-Level Detection (AUC-ROC) | |
| | Method | AUC | 95 % CI | Note | | |
| |---|---|---|---| | |
| | Human ($N\!=\!120$, 2AFC, [CanUSpotAI.com](http://canuspotai.com/)) | **0.501** | [0.450, 0.553] | indistinguishable from chance | | |
| | TruFor (Guillaro et al., 2023) | 0.599 | [0.592, 0.606] | concentrated on WildReceipt (0.791) | | |
| | DocTamper (Qu et al., 2023) | 0.585 | [0.571, 0.600] | concentrated on SROIE (0.710) | | |
| | GPT-Image-2 self-judge | 0.532 | [0.525, 0.537] | minimal prompt, single trial | | |
| ### Calibration Sets (Same-Domain Traditional Tampering) | |
| | Method | Calibration AUC | v2 AUC | Drop | | |
| |---|---|---|---| | |
| | TruFor (cross-camera splicing) | **0.962** | 0.599 | −0.363 | | |
| | DocTamper (cross-doc OCR-token splice + 2-pass JPEG) | **0.852** | 0.585 | −0.267 | | |
| Both detectors retain near-published performance on our source domain when the | |
| tampering is traditional; switching the tampering to GPT-Image-2 inpainting drops | |
| detector AUC by 0.27–0.36, isolating an **AI-inpainting-specific detection gap** | |
| rather than a generic inability to operate on receipt and form scans. | |
| **Key finding:** No off-the-shelf defence — human inspection, generic forensic | |
| detection, document-specific forensic detection, or the generator itself queried | |
| in plain language — exceeds AUC 0.6 on AIForge-Doc v2 in any operationally | |
| meaningful sense. | |
| --- | |
| ## Pairing with AIForge-Doc v1 | |
| Of the 3,066 successful v2 specs, 3,062 also have a v1 forgery on disk | |
| (2,729 produced by Gemini 2.5 Flash Image; 333 by Ideogram v2 Edit). The | |
| `v1_assigned_tool` field in `metadata.jsonl` indicates which v1 generator produced | |
| the same-spec v1 forgery, supporting per-spec cross-generator detector comparisons. | |
| --- | |
| ## Licence | |
| The forged images are derived from the same source corpora as v1: | |
| - **CORD v2** — [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | |
| - **WildReceipt** — [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) | |
| - **SROIE** — ICDAR 2019 research use | |
| - **XFUND** — [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | |
| The AIForge-Doc v2 dataset itself (forged images + masks + metadata) is released under | |
| [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). You are free to share and | |
| adapt the material for any purpose provided you give appropriate credit. | |
| The accompanying generation pipeline, four-judge evaluation protocol, sanity-control | |
| calibration sets, and persistent-runner judge wrappers are released under the | |
| [MIT licence](https://opensource.org/licenses/MIT) at | |
| <https://github.com/scamai/gpt_image2_doc_forgery_paper>. | |
| --- | |
| ## Citation | |
| If you use AIForge-Doc v2 in your research, please cite: | |
| ```bibtex | |
| @dataset{aiforgedoc2v2026, | |
| title = {{AIForge-Doc} v2: A Paired Benchmark of GPT-Image-2 Document Forgeries}, | |
| author = {Wu, Jiaqi and Zhou, Yuchen and Tsang Ng, Dennis and Shen, Xingyu and | |
| Zewde, Kidus and Raj, Ankit and Duong, Tommy and Ren, Simiao}, | |
| year = {2026}, | |
| note = {Dataset paper under submission}, | |
| url = {https://github.com/scamai/gpt_image2_doc_forgery_paper} | |
| } | |
| ``` | |
| --- | |
| ## Contact | |
| For questions about the dataset or to report issues, please open a GitHub issue on | |
| <https://github.com/scamai/gpt_image2_doc_forgery_paper> or contact the corresponding | |
| author at `benren@scam.ai`. | |
| --- | |
| ## Related Research from Scam.AI | |
| This dataset is part of Scam.AI's broader research portfolio on deepfake detection, synthetic media forensics, and adversarial robustness. Other relevant work from our group: | |
| - **DOCFORGE-BENCH: A Comprehensive Benchmark for Document Forgery Detection and Analysis** — Zhao, Xia, Wei et al. (arXiv:2603.01433) | |
| - **When the Forger Is the Judge: GPT-Image-2 Cannot Recognize Its Own Faked Documents** — Wu, Zhou, Ng et al. (arXiv:2604.25213) | |
| - **AIForge-Doc: A Benchmark for Detecting AI-Forged Tampering in Financial and Form Documents** — Wu, Zhou, Xu et al. (arXiv:2602.20569) | |
| - **Can Multi-modal (reasoning) LLMs detect document manipulation?** — Liang, Zewde, Singh et al. (Google Scholar) | |
| Browse our full publications list and dataset catalog at **[scam.ai/research](https://www.scam.ai/en/research)**. | |
| ## About Scam.AI | |
| Scam.AI builds detection systems for AI-driven fraud — deepfakes, document forgery, AI-generated synthetic media, and adversarial attacks against identity verification. Learn more at **[scam.ai](https://www.scam.ai)**. | |
Xet Storage Details
- Size:
- 12 kB
- Xet hash:
- d50eef2a6059e2db8e0507d94583f08764253ae71da3e63be344c44930625775
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.