scripts/METRICS_README.md

Metrics Evaluation Guide

This guide explains how to compute PSNR, SSIM, LPIPS, and FID metrics for your generated CXR images.

Overview

The evaluation pipeline computes four standard metrics:

• SSIM (Structural Similarity Index) ↑ Higher is better
• PSNR (Peak Signal-to-Noise Ratio) ↑ Higher is better
• LPIPS (Learned Perceptual Image Patch Similarity) ↓ Lower is better
• FID (Fréchet Inception Distance) ↓ Lower is better

Prerequisites

Install required Python packages:

pip install scikit-image lpips torch torchmetrics torchmetrics[image]

Or using conda:

conda install -c conda-forge scikit-image
pip install lpips torchmetrics

Directory Structure

The evaluation assumes a specific folder structure for both generated and ground truth images:

generated_dir/
├── LIDC-IDRI-0030/
│   ├── 0004.png
│   ├── 0005.png
│   └── ...
├── LIDC-IDRI-0089/
│   ├── 0004.png
│   └── ...
└── ...

ground_truth_dir/
├── LIDC-IDRI-0030/
│   ├── 0004.png
│   ├── 0005.png
│   └── ...
├── LIDC-IDRI-0089/
│   ├── 0004.png
│   └── ...
└── ...

Both directories should have the same patient ID and image filename structure.

Step 1: Prepare Ground Truth Images

First, you need to extract or prepare your ground truth images in the expected directory structure.

If you have images stored in a JSONL metadata file (like test_metadata.jsonl), you can use a helper script to extract them:

python scripts/prepare_gt_images.py \
    --metadata_jsonl dataset/cxr_radiomics/test_metadata.jsonl \
    --output_dir ./dataset/gt_images

Or if your images are already organized differently, manually create the directory structure as shown above.

Step 2: Run Evaluation

Option A: Using the shell script (recommended)

Edit scripts/eval_metrics.sh to set your paths:

# Edit these paths in the script:
GENERATED_DIR="./outputs/eval_eval_2026.04.29_03.59.44"
GROUND_TRUTH_DIR="./dataset/gt_images"  # Path to your ground truth
OUTPUT_DIR="./results/metrics_eval"

Then run:

bash scripts/eval_metrics.sh

Option B: Quick test with limited samples

bash scripts/eval_metrics.sh --max_samples 100

This will evaluate only the first 100 image pairs (useful for testing).

Option C: Direct Python call

python scripts/eval_metrics.py \
    --generated_dir ./outputs/eval_eval_2026.04.29_03.59.44 \
    --ground_truth_dir ./dataset/gt_images \
    --output_dir ./results/metrics_eval \
    --device cuda:0

Option D: Using different GPUs

export CUDA_VISIBLE_DEVICES=0,1,2,3
bash scripts/eval_metrics.sh --device cuda:0

Step 3: View Results

After evaluation completes, check the results:

# View summary metrics
cat results/metrics_eval/metrics_report.json

# View detailed logs
tail results/metrics_eval/metrics_eval.log

The metrics_report.json contains:

{
  "SSIM": 0.7234,
  "PSNR": 28.45,
  "LPIPS": 0.1234,
  "FID": 15.67,
  "num_evaluated": 512,
  "num_total": 512,
  "num_errors": 0,
  "per_patient": {
    "LIDC-IDRI-0030": {
      "SSIM": 0.7100,
      "PSNR": 28.12,
      "LPIPS": 0.1250,
      "count": 64
    },
    ...
  }
}

Troubleshooting

No valid pairs found

Problem: The script finds 0 image pairs to evaluate.

Solution: Check that:

1. Generated images are in subdirectories like patient_id/image_name.png
2. Ground truth images follow the exact same directory structure
3. File names match exactly (including extensions)

Debug: List both directories to compare structure:

ls -la ./outputs/eval_eval_2026.04.29_03.59.44/LIDC-IDRI-0030/ | head -5
ls -la ./dataset/gt_images/LIDC-IDRI-0030/ | head -5

Out of Memory

Problem: RuntimeError: CUDA out of memory

Solution: Reduce batch size or use CPU:

python scripts/eval_metrics.py \
    --generated_dir ./outputs/eval_eval_2026.04.29_03.59.44 \
    --ground_truth_dir ./dataset/gt_images \
    --output_dir ./results/metrics_eval \
    --device cpu

Or evaluate a subset:

bash scripts/eval_metrics.sh --max_samples 50

Missing dependencies

Problem: ModuleNotFoundError: No module named 'lpips'

Solution: Install missing package:

pip install lpips scikit-image torchmetrics

Understanding the Metrics

SSIM (Structural Similarity)

• Range: 0-1 (higher is better)
• Measures perceived structural similarity
• Commonly used in medical imaging

PSNR (Peak Signal-to-Noise Ratio)

• Range: 0-100 dB (higher is better)
• Measures peak signal vs. noise ratio
• Simple but not always perceptually accurate

LPIPS (Learned Perceptual Image Patch Similarity)

• Range: 0-1 (lower is better)
• Uses pretrained deep network (AlexNet)
• Better aligns with human perception than SSIM/PSNR

FID (Fréchet Inception Distance)

• Range: 0-∞ (lower is better)
• Measures distribution distance between real and generated images
• Uses InceptionV3 features
• Good for evaluating generative model quality

Example Usage with Your Data

For your specific setup:

# Navigate to project root
cd /data/wtchen/code/flow_grpo_cxr

# Quick test with first 50 samples
bash scripts/eval_metrics.sh --max_samples 50

# Full evaluation
bash scripts/eval_metrics.sh

# View results
cat results/metrics_eval/metrics_report.json | python -m json.tool

Notes

• Evaluation runs on GPU by default (much faster)
• LPIPS and FID computation require significant GPU memory
• For large datasets, consider running on GPU with sufficient VRAM (16GB+ recommended)
• Images are automatically converted to RGB for metric computation
• Per-patient metrics are provided for fine-grained analysis