EVAL_REPRODUCTION.md

# Reproducing the eval.md numbers

Step-by-step guide to reproduce the FID / CLIP Score numbers in `eval.md` from
scratch. Unlike v0 (whose eval scripts lived in a throwaway env), everything
needed is **in this repo**: `fetch_coco_subset.py` builds the data,
`eval/run_eval.py` computes the metrics with torchmetrics.

CPU-only is fine (that's how the published numbers were produced). Budget
~7 GB disk: ~5 GB dataset shards + ~2 GB python deps and model weights
(InceptionV3, CLIP ViT-B/32).

## 1. Environment

```bash
python3.13 -m venv eval-venv
# Windows: eval-venv\Scripts\activate    macOS/Linux: source eval-venv/bin/activate

pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
pip install torchmetrics torch-fidelity transformers "datasets<3" safetensors pillow numpy
```

`torch-fidelity` is the backend torchmetrics uses for the FID InceptionV3
features; `transformers` backs the CLIPScore metric.

## 2. Build the data (train + eval split)

```bash
python fetch_coco_subset.py --out ../pm-work
```

This streams [`sayakpaul/coco-30-val-2014`](https://huggingface.co/datasets/sayakpaul/coco-30-val-2014)
(a public 30K-pair subset of MS-COCO val2014) and produces:

- `eval_real/` — stream rows 0–4999: real images, 256×256 center-crop JPEG (FID real set)
- `eval_captions.json` — the 5,000 captions for those rows (generation + CLIP Score)
- `coco_train.npz` + `coco_train.captions.json` — training pairs from stream
  rows 10,000+, at 64×64

**Leakage control:** the same COCO image appears in multiple rows with
different captions, so training rows are dropped if their image content hash
(md5 of a 64×64 bilinear thumbnail) appears in the eval set, and deduped
against each other. The counts printed at the end of the script report both.

Streaming row-by-row from the Hub is slow (~hours on a normal connection). The
much faster path is to download the 10 parquet shards first (~4.9 GB), then:

```bash
python fetch_coco_subset.py --out ../pm-work --parquet-dir ../pm-work/shards
```

Sorted shard filename order equals hub streaming order, so both paths produce
identical splits.

## 3. (Optional) retrain

```bash
python train.py --data ../pm-work/coco_train.npz --epochs 50
```

Deterministic given `--seed` (default 0): same data + same seed reproduces the
published `model.png` up to floating-point ordering. Skip this step to
evaluate the shipped `model.png` as-is.

## 4. Run the eval

```bash
python eval/run_eval.py --work ../pm-work --model model.png --n 5000
```

This:
1. Generates one 64×64 image per eval caption (native resolution — no
   upscaling before metrics).
2. **FID** via `torchmetrics.image.fid.FrechetInceptionDistance`
   (`feature=2048`, InceptionV3 pool3): real = 5,000 COCO val2014 images at
   256×256 center-crop, fake = the 5,000 generated 64×64 images. The metric's
   Inception backbone resizes both sets to 299×299 internally.
3. **CLIP Score** via `torchmetrics.multimodal.CLIPScore` with
   `openai/clip-vit-base-patch32` (the leaderboard default): each generated
   image against the caption that produced it, averaged.

Results are printed and written to `eval_results.json` in the `--work` dir.
First run downloads InceptionV3 (~95 MB) and CLIP ViT-B/32 (~600 MB).

## Notes

- n=5000 vs the standard n=30000: FID's 2048-dim covariance estimate benefits
  from more samples; n=5000 was chosen as a compromise that is far past the
  singular-covariance regime that made v0's n=40 number directional-only,
  while staying CPU-friendly. Raising `--n` (and `EVAL_N` in
  `fetch_coco_subset.py`) toward 30K tightens the estimate at proportional
  cost — but note this dataset has 30K *rows*, not 30K unique images, and
  rows past index 10,000 are used for training, so a larger eval set requires
  re-splitting.
- The generated set's resolution (64×64) is reported as the native generation
  resolution per the leaderboard rules; FID/CLIP preprocessing upsamples
  internally as part of the metric, not as part of generation.
- `INFERENCE.py` (safetensors) and `main.py` (model.png) are verified
  byte-identical, so either weights file reproduces the same metrics.