docs/rl_framework_analysis/05_eval_flow.md

Eval Flow

Command Reconstruction

scripts/single_node/eval_4gpu.sh launches:

python3 -m accelerate.commands.launch \
  --config_file scripts/accelerate_configs/multi_gpu.yaml \
  --num_processes=${NUM_PROCESSES:-4} \
  --main_process_port ${MAIN_PROCESS_PORT:-29501} \
  scripts/eval_omnigen.py \
  --config ${CONFIG_ENTRY:-config/grpo.py:general_radiomics_omnigen_4gpu_kl_eval} \
  --eval_lora_path ${EVAL_LORA_PATH:-logs/radiomics/omnigen-lora-r32-a64-4gpu-bs32-evalbs24-kl-beta0p005-from-cxr-lora30000/checkpoints/checkpoint-60/lora} \
  "$@"

Evidence: scripts/single_node/eval_4gpu.sh:21-33.

Inputs

Input	Default/source	Evidence
Config	config/grpo.py:general_radiomics_omnigen_4gpu_kl_eval	eval_4gpu.sh:23, config/grpo.py:757-760
Model	Shitao/OmniGen-v1	config/grpo.py:630-632
Merged base CXR LoRA	Current server /home/wenting/zr/gen_code/results/cxr_finetune_lora/checkpoints/0030000	config/grpo.py:20-25, :627-628, :699-705
Evaluation LoRA adapter	logs/radiomics/.../checkpoint-60/lora unless overridden	eval_4gpu.sh:24, scripts/eval_omnigen.py:98-107
Dataset	dataset/cxr_radiomics_current_server/test_metadata.jsonl on current server	config/grpo.py:68-70, scripts/eval_omnigen.py:126

Output Directory And Files

If --resume_dir is not passed, scripts/eval_omnigen.py constructs output_dir = FLAGS.output_dir/{run_name}_eval_{timestamp} where FLAGS.output_dir defaults to /data/wtchen/code/flow_grpo_cxr/outputs and run_name defaults to config.run_name or eval (scripts/eval_omnigen.py:32-37, :57-70).

File	Meaning	Evidence
eval_lora_path.json	Adapter path used for reproducibility.	scripts/eval_omnigen.py:118-122
{patient_id}/{image_name}	Generated PNG for each metadata record.	scripts/eval_omnigen.py:41-44, :192-203, :273-279
failed_rank_{rank}.jsonl	Per-rank generation/save failures.	scripts/eval_omnigen.py:253-268, :280-298
status_rank_{rank}.json	Assigned/saved/skipped/failed counts for the rank.	scripts/eval_omnigen.py:299-311
missing_images.jsonl	Global missing/empty generated image list on main process.	scripts/eval_omnigen.py:315-329
eval_summary.json	Dataset size, complete images, missing images, output dir, resume dir.	scripts/eval_omnigen.py:330-342

Generation Logic

Step	Code
Load model/VAE/processor	load_omnigen_components(config, accelerator.device, weight_dtype) at scripts/eval_omnigen.py:81-83
Merge pretrained base CXR LoRA if configured	scripts/eval_omnigen.py:87-96
Load evaluation adapter	PeftModel.from_pretrained(..., is_trainable=False) at scripts/eval_omnigen.py:98-110
Build dataset	RadiomicsEditDataset(config.dataset, "test") at scripts/eval_omnigen.py:126
Partition across ranks	all_eval_indices[accelerator.process_index::accelerator.num_processes] at scripts/eval_omnigen.py:145-146
Generate	pipeline_with_logprob(..., eval_num_steps, eval_guidance_scale, eval_img_guidance_scale, noise_level=0.0) at scripts/eval_omnigen.py:166-190
Save	_to_rgb_pil(image_array).save(tmp_path, format="PNG"), atomic replace at scripts/eval_omnigen.py:192-203

Metrics

scripts/eval_omnigen.py does not compute image-quality metrics. It only checks output completeness and writes status/summary files. The image metric implementation for generated-vs-GT pairs exists in scripts/test_omnigen_cxr.py:evaluate:

Metric	Formula/code
SSIM	skimage.metrics.structural_similarity(gt_gray, gen_gray, data_range=255) at scripts/test_omnigen_cxr.py:337-341
PSNR	skimage.metrics.peak_signal_noise_ratio(gt_gray, gen_gray, data_range=255) at scripts/test_omnigen_cxr.py:337-342
LPIPS	lpips_fn(gen_batch, gt_batch) after RGB tensors scaled to [-1, 1] at scripts/test_omnigen_cxr.py:322-330, :354-355
FID	FrechetInceptionDistance(feature=2048) updated with uint8 CHW real/fake tensors, then compute() at scripts/test_omnigen_cxr.py:301-302, :315-320, :356-357, :385

Relationship To Test Output

eval_4gpu.sh and test_omnigen_4gpu.sh both save generated images as output_dir/patient_id/image_name. test_omnigen_cxr.py additionally computes metrics against the same metadata GT paths and writes metrics_report.json; eval_omnigen.py writes completeness summaries and missing-image records but no metric report.

Pipeline Diagram

flowchart TD
  sh[eval_4gpu.sh] --> acc[accelerate x4]
  acc --> py[scripts/eval_omnigen.py]
  py --> cfg[general_radiomics_omnigen_4gpu_kl_eval]
  cfg --> ds[test_metadata.jsonl]
  cfg --> model[OmniGen + merged CXR LoRA]
  py --> eval_lora[checkpoint-60/lora adapter]
  ds --> shard[rank strided subset]
  shard --> gen[pipeline_with_logprob noise_level=0]
  model --> gen
  eval_lora --> gen
  gen --> png[patient_id/image_name PNG]
  png --> summary[status/missing/eval_summary JSON]