File size: 4,167 Bytes
c500926 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 | # Reproducing the eval.md numbers
Step-by-step guide to reproduce the FID / CLIP Score numbers in `eval.md` from
scratch. Unlike v0 (whose eval scripts lived in a throwaway env), everything
needed is **in this repo**: `fetch_coco_subset.py` builds the data,
`eval/run_eval.py` computes the metrics with torchmetrics.
CPU-only is fine (that's how the published numbers were produced). Budget
~7 GB disk: ~5 GB dataset shards + ~2 GB python deps and model weights
(InceptionV3, CLIP ViT-B/32).
## 1. Environment
```bash
python3.13 -m venv eval-venv
# Windows: eval-venv\Scripts\activate macOS/Linux: source eval-venv/bin/activate
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
pip install torchmetrics torch-fidelity transformers "datasets<3" safetensors pillow numpy
```
`torch-fidelity` is the backend torchmetrics uses for the FID InceptionV3
features; `transformers` backs the CLIPScore metric.
## 2. Build the data (train + eval split)
```bash
python fetch_coco_subset.py --out ../pm-work
```
This streams [`sayakpaul/coco-30-val-2014`](https://huggingface.co/datasets/sayakpaul/coco-30-val-2014)
(a public 30K-pair subset of MS-COCO val2014) and produces:
- `eval_real/` — stream rows 0–4999: real images, 256×256 center-crop JPEG (FID real set)
- `eval_captions.json` — the 5,000 captions for those rows (generation + CLIP Score)
- `coco_train.npz` + `coco_train.captions.json` — training pairs from stream
rows 10,000+, at 64×64
**Leakage control:** the same COCO image appears in multiple rows with
different captions, so training rows are dropped if their image content hash
(md5 of a 64×64 bilinear thumbnail) appears in the eval set, and deduped
against each other. The counts printed at the end of the script report both.
Streaming row-by-row from the Hub is slow (~hours on a normal connection). The
much faster path is to download the 10 parquet shards first (~4.9 GB), then:
```bash
python fetch_coco_subset.py --out ../pm-work --parquet-dir ../pm-work/shards
```
Sorted shard filename order equals hub streaming order, so both paths produce
identical splits.
## 3. (Optional) retrain
```bash
python train.py --data ../pm-work/coco_train.npz --epochs 50
```
Deterministic given `--seed` (default 0): same data + same seed reproduces the
published `model.png` up to floating-point ordering. Skip this step to
evaluate the shipped `model.png` as-is.
## 4. Run the eval
```bash
python eval/run_eval.py --work ../pm-work --model model.png --n 5000
```
This:
1. Generates one 64×64 image per eval caption (native resolution — no
upscaling before metrics).
2. **FID** via `torchmetrics.image.fid.FrechetInceptionDistance`
(`feature=2048`, InceptionV3 pool3): real = 5,000 COCO val2014 images at
256×256 center-crop, fake = the 5,000 generated 64×64 images. The metric's
Inception backbone resizes both sets to 299×299 internally.
3. **CLIP Score** via `torchmetrics.multimodal.CLIPScore` with
`openai/clip-vit-base-patch32` (the leaderboard default): each generated
image against the caption that produced it, averaged.
Results are printed and written to `eval_results.json` in the `--work` dir.
First run downloads InceptionV3 (~95 MB) and CLIP ViT-B/32 (~600 MB).
## Notes
- n=5000 vs the standard n=30000: FID's 2048-dim covariance estimate benefits
from more samples; n=5000 was chosen as a compromise that is far past the
singular-covariance regime that made v0's n=40 number directional-only,
while staying CPU-friendly. Raising `--n` (and `EVAL_N` in
`fetch_coco_subset.py`) toward 30K tightens the estimate at proportional
cost — but note this dataset has 30K *rows*, not 30K unique images, and
rows past index 10,000 are used for training, so a larger eval set requires
re-splitting.
- The generated set's resolution (64×64) is reported as the native generation
resolution per the leaderboard rules; FID/CLIP preprocessing upsamples
internally as part of the metric, not as part of generation.
- `INFERENCE.py` (safetensors) and `main.py` (model.png) are verified
byte-identical, so either weights file reproduces the same metrics.
|