# Eval Flow

## Command Reconstruction

`scripts/single_node/eval_4gpu.sh` launches:

```bash
python3 -m accelerate.commands.launch \
  --config_file scripts/accelerate_configs/multi_gpu.yaml \
  --num_processes=${NUM_PROCESSES:-4} \
  --main_process_port ${MAIN_PROCESS_PORT:-29501} \
  scripts/eval_omnigen.py \
  --config ${CONFIG_ENTRY:-config/grpo.py:general_radiomics_omnigen_4gpu_kl_eval} \
  --eval_lora_path ${EVAL_LORA_PATH:-logs/radiomics/omnigen-lora-r32-a64-4gpu-bs32-evalbs24-kl-beta0p005-from-cxr-lora30000/checkpoints/checkpoint-60/lora} \
  "$@"
```

Evidence: `scripts/single_node/eval_4gpu.sh:21-33`.

## Inputs

| Input | Default/source | Evidence |
|---|---|---|
| Config | `config/grpo.py:general_radiomics_omnigen_4gpu_kl_eval` | `eval_4gpu.sh:23`, `config/grpo.py:757-760` |
| Model | `Shitao/OmniGen-v1` | `config/grpo.py:630-632` |
| Merged base CXR LoRA | Current server `/home/wenting/zr/gen_code/results/cxr_finetune_lora/checkpoints/0030000` | `config/grpo.py:20-25`, `:627-628`, `:699-705` |
| Evaluation LoRA adapter | `logs/radiomics/.../checkpoint-60/lora` unless overridden | `eval_4gpu.sh:24`, `scripts/eval_omnigen.py:98-107` |
| Dataset | `dataset/cxr_radiomics_current_server/test_metadata.jsonl` on current server | `config/grpo.py:68-70`, `scripts/eval_omnigen.py:126` |

## Output Directory And Files

If `--resume_dir` is not passed, `scripts/eval_omnigen.py` constructs `output_dir = FLAGS.output_dir/{run_name}_eval_{timestamp}` where `FLAGS.output_dir` defaults to `/data/wtchen/code/flow_grpo_cxr/outputs` and `run_name` defaults to `config.run_name` or `eval` (`scripts/eval_omnigen.py:32-37`, `:57-70`).

| File | Meaning | Evidence |
|---|---|---|
| `eval_lora_path.json` | Adapter path used for reproducibility. | `scripts/eval_omnigen.py:118-122` |
| `{patient_id}/{image_name}` | Generated PNG for each metadata record. | `scripts/eval_omnigen.py:41-44`, `:192-203`, `:273-279` |
| `failed_rank_{rank}.jsonl` | Per-rank generation/save failures. | `scripts/eval_omnigen.py:253-268`, `:280-298` |
| `status_rank_{rank}.json` | Assigned/saved/skipped/failed counts for the rank. | `scripts/eval_omnigen.py:299-311` |
| `missing_images.jsonl` | Global missing/empty generated image list on main process. | `scripts/eval_omnigen.py:315-329` |
| `eval_summary.json` | Dataset size, complete images, missing images, output dir, resume dir. | `scripts/eval_omnigen.py:330-342` |

## Generation Logic

| Step | Code |
|---|---|
| Load model/VAE/processor | `load_omnigen_components(config, accelerator.device, weight_dtype)` at `scripts/eval_omnigen.py:81-83` |
| Merge pretrained base CXR LoRA if configured | `scripts/eval_omnigen.py:87-96` |
| Load evaluation adapter | `PeftModel.from_pretrained(..., is_trainable=False)` at `scripts/eval_omnigen.py:98-110` |
| Build dataset | `RadiomicsEditDataset(config.dataset, "test")` at `scripts/eval_omnigen.py:126` |
| Partition across ranks | `all_eval_indices[accelerator.process_index::accelerator.num_processes]` at `scripts/eval_omnigen.py:145-146` |
| Generate | `pipeline_with_logprob(..., eval_num_steps, eval_guidance_scale, eval_img_guidance_scale, noise_level=0.0)` at `scripts/eval_omnigen.py:166-190` |
| Save | `_to_rgb_pil(image_array).save(tmp_path, format="PNG")`, atomic replace at `scripts/eval_omnigen.py:192-203` |

## Metrics

`scripts/eval_omnigen.py` does not compute image-quality metrics. It only checks output completeness and writes status/summary files. The image metric implementation for generated-vs-GT pairs exists in `scripts/test_omnigen_cxr.py:evaluate`:

| Metric | Formula/code |
|---|---|
| SSIM | `skimage.metrics.structural_similarity(gt_gray, gen_gray, data_range=255)` at `scripts/test_omnigen_cxr.py:337-341` |
| PSNR | `skimage.metrics.peak_signal_noise_ratio(gt_gray, gen_gray, data_range=255)` at `scripts/test_omnigen_cxr.py:337-342` |
| LPIPS | `lpips_fn(gen_batch, gt_batch)` after RGB tensors scaled to `[-1, 1]` at `scripts/test_omnigen_cxr.py:322-330`, `:354-355` |
| FID | `FrechetInceptionDistance(feature=2048)` updated with uint8 CHW real/fake tensors, then `compute()` at `scripts/test_omnigen_cxr.py:301-302`, `:315-320`, `:356-357`, `:385` |

## Relationship To Test Output

`eval_4gpu.sh` and `test_omnigen_4gpu.sh` both save generated images as `output_dir/patient_id/image_name`. `test_omnigen_cxr.py` additionally computes metrics against the same metadata GT paths and writes `metrics_report.json`; `eval_omnigen.py` writes completeness summaries and missing-image records but no metric report.

## Pipeline Diagram

```mermaid
flowchart TD
  sh[eval_4gpu.sh] --> acc[accelerate x4]
  acc --> py[scripts/eval_omnigen.py]
  py --> cfg[general_radiomics_omnigen_4gpu_kl_eval]
  cfg --> ds[test_metadata.jsonl]
  cfg --> model[OmniGen + merged CXR LoRA]
  py --> eval_lora[checkpoint-60/lora adapter]
  ds --> shard[rank strided subset]
  shard --> gen[pipeline_with_logprob noise_level=0]
  model --> gen
  eval_lora --> gen
  gen --> png[patient_id/image_name PNG]
  png --> summary[status/missing/eval_summary JSON]
```