scripts/dataset/README.md

# Dataset curation

This folder contains the scripts that produced the **commercially-licensed
training dataset** for Stage 2 fine-tuning (DONE, 2026-05).

Stage 3A extends this folder with one new generation script per
approved open-source generator (initially AuraFlow, SDXL, and SD 3.5
Medium; PixArt-Σ is deferred pending license/intended-use clarification).
The rest of the pipeline
(`build_manifest.py`, `split.py`, `../precompute_embeddings.py`,
`../train_head.py`) is generator-agnostic and will be reused as-is —
the manifest format already supports multiple AI source labels.
See [`docs/plan.md`](../../docs/plan.md) Stage 3 for the full plan.

## What we're building

A dataset of roughly **100k images** total:

| Split      | Source                                | Class           | License            |
|------------|---------------------------------------|-----------------|--------------------|
| ~50k real  | Open Images V7 (sampled)              | `authentic`     | CC BY 2.0          |
| ~50k AI    | Self-generated with Flux.1-schnell    | `ai_generated`  | Apache 2.0 weights → outputs are unrestricted |

Stage 2 trains the CLIP classifier head on this 2-class data. The `deepfake`
and `edited` classes will be added later via separate scripts.

## Workflow

1. **`fetch_open_images.py`** — downloads a stratified sample of Open Images
   into `data/raw/real/`. Run once. Idempotent.

2. **`generate_flux_synthetic.py`** — generates AI images with Flux.1-schnell
   into `data/raw/ai_generated/`. Run on a rented cloud GPU
   (~$30 total at RunPod / Lambda Labs spot rates).

   **`generate_sdxl_synthetic.py`** — generates AI images with SDXL into
   `data/raw/ai_generated/sdxl/`. Use `--dry-run` locally before running on a
   GPU box.

   **`generate_sd35_synthetic.py`** — generates AI images with SD 3.5 Medium
   into `data/raw/ai_generated/sd35-medium/`. This source is conditionally
   approved under the Stability AI Community License; review `NOTICES.md`
   before running it.

   **`generate_auraflow_synthetic.py`** — generates AI images with AuraFlow
   v0.3 into `data/raw/ai_generated/auraflow-v0.3/`. Use `--dry-run` locally
   before running on a GPU box.

   Stage 3A generator scripts should reuse `generation_utils.py` for prompt
   loading, deterministic seeds, stable output keys, file hashes, image
   dimensions, approved generator metadata, and manifest-row construction.

3. **`build_manifest.py`** — produces `data/manifest.csv` with one row per
   image. Required columns are
   `path,class,source,license,license_url,sha256`. Stage 3A adds optional
   generator metadata columns:
   `generator,model_family,model_id,prompt,seed,width,height,generation_params_json`.
   Missing optional values are written as empty strings, and approved
   AI-generator metadata is backfilled from known `source` values when older
   fragments do not include the optional columns. You can pass fragments
   explicitly with `--inputs`, or point `--input-dir` at `data/raw` to discover
   all `*_manifest.csv` fragments recursively. This is the
   **legal record** that protects you when you commercialise and the
   source-of-truth for per-generator evaluation.

4. **`split.py`** — produces train/val/test splits stratified by class. Stage
   3A adds optional generator-aware splitting with
   `--stratify-by class-generator`, plus `--holdout-generator <name>` to write
   a `heldout.csv` evaluation split while excluding that generator from
   train/val/test.

After splitting, training itself lives outside this directory:

5. **`augment_images.py`** — creates deterministic Stage 3A robustness
   augmentations from any manifest or split CSV. It writes new JPEG images plus
   a manifest-compatible CSV with `original_path`, `augmentation`,
   `augmentation_seed`, and `augmentation_params_json`. Run it on `train.csv`
   for augmented training rows, or on `val.csv` / `test.csv` into separate
   manifests such as `test_augmented.csv` for robustness evaluation while
   keeping the clean splits unchanged.

6. **`../precompute_embeddings.py`** — encodes every image through frozen
   CLIP once and caches the 512-d feature vectors. Runs on CPU (1-4 hours
   on the dev laptop). One-shot per dataset version. By default it encodes
   clean `train.csv`, `val.csv`, and `test.csv`. Stage 3A adds
   `--train-augment-manifest train_augmented.csv` to append augmented rows to
   `train.npz`, and `--extra-split test_augmented=test_augmented.csv` for
   explicit augmented robustness-eval embeddings.

7. **`../train_head.py`** — trains the small classifier head on the cached
   features. Seconds per epoch on CPU; iterate hyperparameters freely. Stage
   3A reports overall metrics, per-source/per-generator/per-family metrics when
   embedding metadata is available, and augmentation robustness metrics for
   explicitly encoded augmented eval splits. Use `--report-out` to save the
   JSON report.

8. **`../evaluate_head.py`** — evaluates a candidate checkpoint, optionally
   against a Stage 2 baseline checkpoint, on any cached embedding split
   (`test`, `heldout`, `test_augmented`, etc.). It writes a reproducible JSON
   report with overall, per-generator, held-out-generator, uncertainty, and
   augmentation robustness metrics.

9. **`run_stage3a_smoke.py`** — runs a tiny local fixture pipeline before
   expensive GPU work. It builds fixture images/manifests, splits with an SDXL
   holdout, creates train/test augmentations, writes deterministic mocked
   512-d embeddings, trains a smoke head, and evaluates it against a constant
   baseline:

   ```bash
   python scripts/dataset/run_stage3a_smoke.py \
       --work-dir /tmp/deepfakescanner-stage3a-smoke
   ```

   This intentionally bypasses real CLIP precompute so the smoke test stays
   fast and offline; use `../precompute_embeddings.py` for real dataset runs.

10. **`../run_stage3a_pipeline.py`** — orchestrates the full Stage 3A sequence
    after the smoke test passes. It can plan the run, execute GPU generation,
    build the manifest, split, augment, precompute embeddings, train the
    candidate head, evaluate against the Stage 2 baseline, write a JSON
    ship/no-ship recommendation, and optionally upload an accepted checkpoint
    to the private HF Hub repo.

    ```bash
    python scripts/run_stage3a_pipeline.py --dry-run --stop-after evaluate
    python scripts/run_stage3a_pipeline.py --stop-after generate
    python scripts/run_stage3a_pipeline.py \
        --skip-generation \
        --baseline data/checkpoints/head_v1.pt
    ```

    Publishing is intentionally opt-in via `--publish-if-accepted` and requires
    `HF_TOKEN`.

For a collaborator-facing GPU handoff, use
[`docs/stage3a-gpu-collaborator-guide.md`](../../docs/stage3a-gpu-collaborator-guide.md).

## Why each piece is licensed for commercial use

- **Open Images V7** — Google's dataset, all images are CC BY 2.0 (commercial
  use allowed with attribution). Attribution lives in `manifest.csv`.
- **Flux.1-schnell** — released by Black Forest Labs under Apache 2.0. Outputs
  are not restricted; you own them. (Note: `flux.1-dev` is non-commercial —
  do NOT use it.)
- **Stage 3A approved additions** — SDXL, SD 3.5 Medium, and AuraFlow were
  reviewed on 2026-05-15. See [`NOTICES.md`](../../NOTICES.md) for the exact
  license status and constraints before generating any images.
- **NOT used here**: FaceForensics++, DFDC, Celeb-DF (research-only),
  outputs from closed-API generators (Gemini, Midjourney, DALL-E, Grok,
  Firefly — ToS restrictions or ambiguous terms), `flux.1-dev`
  (non-commercial), and PixArt-Σ until its model-card intended-use ambiguity
  is clarified. See [`NOTICES.md`](../../NOTICES.md) for the full list and
  [`docs/decisions.md`](../../docs/decisions.md) 2026-05-14 and 2026-05-15
  entries for the reasoning.

## Running it

The scripts are designed to be run independently. See each script's docstring
for prerequisites and command-line flags.