# Metrics Evaluation Guide

This guide explains how to compute PSNR, SSIM, LPIPS, and FID metrics for your generated CXR images.

## Overview

The evaluation pipeline computes four standard metrics:
- **SSIM** (Structural Similarity Index) ↑ Higher is better
- **PSNR** (Peak Signal-to-Noise Ratio) ↑ Higher is better  
- **LPIPS** (Learned Perceptual Image Patch Similarity) ↓ Lower is better
- **FID** (Fréchet Inception Distance) ↓ Lower is better

## Prerequisites

Install required Python packages:

```bash
pip install scikit-image lpips torch torchmetrics torchmetrics[image]
```

Or using conda:

```bash
conda install -c conda-forge scikit-image
pip install lpips torchmetrics
```

## Directory Structure

The evaluation assumes a specific folder structure for both generated and ground truth images:

```
generated_dir/
├── LIDC-IDRI-0030/
│   ├── 0004.png
│   ├── 0005.png
│   └── ...
├── LIDC-IDRI-0089/
│   ├── 0004.png
│   └── ...
└── ...

ground_truth_dir/
├── LIDC-IDRI-0030/
│   ├── 0004.png
│   ├── 0005.png
│   └── ...
├── LIDC-IDRI-0089/
│   ├── 0004.png
│   └── ...
└── ...
```

Both directories should have the same patient ID and image filename structure.

## Step 1: Prepare Ground Truth Images

First, you need to extract or prepare your ground truth images in the expected directory structure.

If you have images stored in a JSONL metadata file (like `test_metadata.jsonl`), you can use a helper script to extract them:

```bash
python scripts/prepare_gt_images.py \
    --metadata_jsonl dataset/cxr_radiomics/test_metadata.jsonl \
    --output_dir ./dataset/gt_images
```

Or if your images are already organized differently, manually create the directory structure as shown above.

## Step 2: Run Evaluation

### Option A: Using the shell script (recommended)

Edit `scripts/eval_metrics.sh` to set your paths:

```bash
# Edit these paths in the script:
GENERATED_DIR="./outputs/eval_eval_2026.04.29_03.59.44"
GROUND_TRUTH_DIR="./dataset/gt_images"  # Path to your ground truth
OUTPUT_DIR="./results/metrics_eval"
```

Then run:

```bash
bash scripts/eval_metrics.sh
```

### Option B: Quick test with limited samples

```bash
bash scripts/eval_metrics.sh --max_samples 100
```

This will evaluate only the first 100 image pairs (useful for testing).

### Option C: Direct Python call

```bash
python scripts/eval_metrics.py \
    --generated_dir ./outputs/eval_eval_2026.04.29_03.59.44 \
    --ground_truth_dir ./dataset/gt_images \
    --output_dir ./results/metrics_eval \
    --device cuda:0
```

### Option D: Using different GPUs

```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
bash scripts/eval_metrics.sh --device cuda:0
```

## Step 3: View Results

After evaluation completes, check the results:

```bash
# View summary metrics
cat results/metrics_eval/metrics_report.json

# View detailed logs
tail results/metrics_eval/metrics_eval.log
```

The `metrics_report.json` contains:

```json
{
  "SSIM": 0.7234,
  "PSNR": 28.45,
  "LPIPS": 0.1234,
  "FID": 15.67,
  "num_evaluated": 512,
  "num_total": 512,
  "num_errors": 0,
  "per_patient": {
    "LIDC-IDRI-0030": {
      "SSIM": 0.7100,
      "PSNR": 28.12,
      "LPIPS": 0.1250,
      "count": 64
    },
    ...
  }
}
```

## Troubleshooting

### No valid pairs found

**Problem**: The script finds 0 image pairs to evaluate.

**Solution**: Check that:
1. Generated images are in subdirectories like `patient_id/image_name.png`
2. Ground truth images follow the exact same directory structure
3. File names match exactly (including extensions)

**Debug**: List both directories to compare structure:
```bash
ls -la ./outputs/eval_eval_2026.04.29_03.59.44/LIDC-IDRI-0030/ | head -5
ls -la ./dataset/gt_images/LIDC-IDRI-0030/ | head -5
```

### Out of Memory

**Problem**: `RuntimeError: CUDA out of memory`

**Solution**: Reduce batch size or use CPU:
```bash
python scripts/eval_metrics.py \
    --generated_dir ./outputs/eval_eval_2026.04.29_03.59.44 \
    --ground_truth_dir ./dataset/gt_images \
    --output_dir ./results/metrics_eval \
    --device cpu
```

Or evaluate a subset:
```bash
bash scripts/eval_metrics.sh --max_samples 50
```

### Missing dependencies

**Problem**: `ModuleNotFoundError: No module named 'lpips'`

**Solution**: Install missing package:
```bash
pip install lpips scikit-image torchmetrics
```

## Understanding the Metrics

### SSIM (Structural Similarity)
- Range: 0-1 (higher is better)
- Measures perceived structural similarity
- Commonly used in medical imaging

### PSNR (Peak Signal-to-Noise Ratio)  
- Range: 0-100 dB (higher is better)
- Measures peak signal vs. noise ratio
- Simple but not always perceptually accurate

### LPIPS (Learned Perceptual Image Patch Similarity)
- Range: 0-1 (lower is better)
- Uses pretrained deep network (AlexNet)
- Better aligns with human perception than SSIM/PSNR

### FID (Fréchet Inception Distance)
- Range: 0-∞ (lower is better)
- Measures distribution distance between real and generated images
- Uses InceptionV3 features
- Good for evaluating generative model quality

## Example Usage with Your Data

For your specific setup:

```bash
# Navigate to project root
cd /data/wtchen/code/flow_grpo_cxr

# Quick test with first 50 samples
bash scripts/eval_metrics.sh --max_samples 50

# Full evaluation
bash scripts/eval_metrics.sh

# View results
cat results/metrics_eval/metrics_report.json | python -m json.tool
```

## Notes

- Evaluation runs on GPU by default (much faster)
- LPIPS and FID computation require significant GPU memory
- For large datasets, consider running on GPU with sufficient VRAM (16GB+ recommended)
- Images are automatically converted to RGB for metric computation
- Per-patient metrics are provided for fine-grained analysis