File size: 5,078 Bytes
535fb25 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 | # Eval Flow
## Command Reconstruction
`scripts/single_node/eval_4gpu.sh` launches:
```bash
python3 -m accelerate.commands.launch \
--config_file scripts/accelerate_configs/multi_gpu.yaml \
--num_processes=${NUM_PROCESSES:-4} \
--main_process_port ${MAIN_PROCESS_PORT:-29501} \
scripts/eval_omnigen.py \
--config ${CONFIG_ENTRY:-config/grpo.py:general_radiomics_omnigen_4gpu_kl_eval} \
--eval_lora_path ${EVAL_LORA_PATH:-logs/radiomics/omnigen-lora-r32-a64-4gpu-bs32-evalbs24-kl-beta0p005-from-cxr-lora30000/checkpoints/checkpoint-60/lora} \
"$@"
```
Evidence: `scripts/single_node/eval_4gpu.sh:21-33`.
## Inputs
| Input | Default/source | Evidence |
|---|---|---|
| Config | `config/grpo.py:general_radiomics_omnigen_4gpu_kl_eval` | `eval_4gpu.sh:23`, `config/grpo.py:757-760` |
| Model | `Shitao/OmniGen-v1` | `config/grpo.py:630-632` |
| Merged base CXR LoRA | Current server `/home/wenting/zr/gen_code/results/cxr_finetune_lora/checkpoints/0030000` | `config/grpo.py:20-25`, `:627-628`, `:699-705` |
| Evaluation LoRA adapter | `logs/radiomics/.../checkpoint-60/lora` unless overridden | `eval_4gpu.sh:24`, `scripts/eval_omnigen.py:98-107` |
| Dataset | `dataset/cxr_radiomics_current_server/test_metadata.jsonl` on current server | `config/grpo.py:68-70`, `scripts/eval_omnigen.py:126` |
## Output Directory And Files
If `--resume_dir` is not passed, `scripts/eval_omnigen.py` constructs `output_dir = FLAGS.output_dir/{run_name}_eval_{timestamp}` where `FLAGS.output_dir` defaults to `/data/wtchen/code/flow_grpo_cxr/outputs` and `run_name` defaults to `config.run_name` or `eval` (`scripts/eval_omnigen.py:32-37`, `:57-70`).
| File | Meaning | Evidence |
|---|---|---|
| `eval_lora_path.json` | Adapter path used for reproducibility. | `scripts/eval_omnigen.py:118-122` |
| `{patient_id}/{image_name}` | Generated PNG for each metadata record. | `scripts/eval_omnigen.py:41-44`, `:192-203`, `:273-279` |
| `failed_rank_{rank}.jsonl` | Per-rank generation/save failures. | `scripts/eval_omnigen.py:253-268`, `:280-298` |
| `status_rank_{rank}.json` | Assigned/saved/skipped/failed counts for the rank. | `scripts/eval_omnigen.py:299-311` |
| `missing_images.jsonl` | Global missing/empty generated image list on main process. | `scripts/eval_omnigen.py:315-329` |
| `eval_summary.json` | Dataset size, complete images, missing images, output dir, resume dir. | `scripts/eval_omnigen.py:330-342` |
## Generation Logic
| Step | Code |
|---|---|
| Load model/VAE/processor | `load_omnigen_components(config, accelerator.device, weight_dtype)` at `scripts/eval_omnigen.py:81-83` |
| Merge pretrained base CXR LoRA if configured | `scripts/eval_omnigen.py:87-96` |
| Load evaluation adapter | `PeftModel.from_pretrained(..., is_trainable=False)` at `scripts/eval_omnigen.py:98-110` |
| Build dataset | `RadiomicsEditDataset(config.dataset, "test")` at `scripts/eval_omnigen.py:126` |
| Partition across ranks | `all_eval_indices[accelerator.process_index::accelerator.num_processes]` at `scripts/eval_omnigen.py:145-146` |
| Generate | `pipeline_with_logprob(..., eval_num_steps, eval_guidance_scale, eval_img_guidance_scale, noise_level=0.0)` at `scripts/eval_omnigen.py:166-190` |
| Save | `_to_rgb_pil(image_array).save(tmp_path, format="PNG")`, atomic replace at `scripts/eval_omnigen.py:192-203` |
## Metrics
`scripts/eval_omnigen.py` does not compute image-quality metrics. It only checks output completeness and writes status/summary files. The image metric implementation for generated-vs-GT pairs exists in `scripts/test_omnigen_cxr.py:evaluate`:
| Metric | Formula/code |
|---|---|
| SSIM | `skimage.metrics.structural_similarity(gt_gray, gen_gray, data_range=255)` at `scripts/test_omnigen_cxr.py:337-341` |
| PSNR | `skimage.metrics.peak_signal_noise_ratio(gt_gray, gen_gray, data_range=255)` at `scripts/test_omnigen_cxr.py:337-342` |
| LPIPS | `lpips_fn(gen_batch, gt_batch)` after RGB tensors scaled to `[-1, 1]` at `scripts/test_omnigen_cxr.py:322-330`, `:354-355` |
| FID | `FrechetInceptionDistance(feature=2048)` updated with uint8 CHW real/fake tensors, then `compute()` at `scripts/test_omnigen_cxr.py:301-302`, `:315-320`, `:356-357`, `:385` |
## Relationship To Test Output
`eval_4gpu.sh` and `test_omnigen_4gpu.sh` both save generated images as `output_dir/patient_id/image_name`. `test_omnigen_cxr.py` additionally computes metrics against the same metadata GT paths and writes `metrics_report.json`; `eval_omnigen.py` writes completeness summaries and missing-image records but no metric report.
## Pipeline Diagram
```mermaid
flowchart TD
sh[eval_4gpu.sh] --> acc[accelerate x4]
acc --> py[scripts/eval_omnigen.py]
py --> cfg[general_radiomics_omnigen_4gpu_kl_eval]
cfg --> ds[test_metadata.jsonl]
cfg --> model[OmniGen + merged CXR LoRA]
py --> eval_lora[checkpoint-60/lora adapter]
ds --> shard[rank strided subset]
shard --> gen[pipeline_with_logprob noise_level=0]
model --> gen
eval_lora --> gen
gen --> png[patient_id/image_name PNG]
png --> summary[status/missing/eval_summary JSON]
```
|