README.md

# Rubrics-RL: Rubric-Guided Reinforcement Learning for Medical Report Generation

This repository contains the code, training configurations, evaluation outputs, and processed data for the Rubrics-RL project (ICLR 2027 submission). The project investigates rubric-guided reinforcement learning for medical report generation, focusing on the "scissors gap" between LLM-judge rewards and verifiable faithfulness metrics.

## Repository Structure

```
├── code/                          # All training and evaluation scripts
│   ├── audit/                     # M1 judge vulnerability audit
│   ├── m2/                        # M2-M4 closed-form VQA experiments
│   ├── m4/                        # M4 counterfactual faithfulness
│   └── m5/                        # M5-M19 open-ended report generation
│       ├── report_reward.py       # M5 reward (5 conditions)
│       ├── report_reward_m6.py    # M6 routed gate reward
│       ├── report_reward_m9.py    # M9 RadGraph dual-gate reward
│       ├── chexbert_server.py     # CheXbert lie-detector service
│       ├── radgraph_server.py     # RadGraph lie-detector service
│       ├── rubric_gen.py          # RubricHub rubric generation
│       ├── atom_router.py         # Atom routing for gated reward
│       ├── train_m5.sh            # M5 training script
│       ├── train_m6.sh            # M6 training script
│       ├── train_m9.sh            # M9 training script
│       ├── plot_m5.py             # M5 scissors gap plot
│       ├── plot_m6.py             # M6 routed gate plot
│       ├── plot_m9.py             # M9 RadGraph gate plot
│       ├── m11_calibration.py     # M11 clinical calibration
│       ├── m11_adversarial_probe.py # M11 adversarial CheXbert probe
│       ├── m13a_overcredit.py     # M13 over-credit analysis
│       └── ...
├── data/                          # Project-specific processed data
│   ├── m2_vqarad/                 # Processed VQA-RAD for M2 training
│   ├── m5_iuxray/                 # Processed IU-Xray for M5 training
│   └── new_datasets/
│       ├── CRIMSON/               # RadJudge/RadPref calibration data
│       └── TAIX-VQA/              # ICU CXR faithfulness eval
├── outputs/                       # Evaluation results, plots, caches
│   ├── *.json                     # Evaluation result JSONs
│   ├── *.png                      # Visualization plots
│   ├── *.csv                      # Per-sample results
│   ├── *_cache/                   # Judge inference caches
│   └── ...
├── logs/                          # Training logs
├── env_backup/                    # Conda environment specifications
└── README.md                      # This file
```

## Publicly Available Datasets (not included)

The following datasets used in this project are publicly available. Download them from their respective sources:

| Dataset | Access | Source | Size | Usage |
|---------|--------|--------|------|-------|
| **VQA-RAD** | OPEN | HuggingFace: `flaviagiammarino/vqa-rad` | 34 MB | M1-M4 VQA warm-up |
| **SLAKE** | OPEN | HuggingFace: `mdwiratathya/SLAKE-vqa-english` (or `BoKelvin/SLAKE`) | <1 GB | VQA warm-up + spatial |
| **IU-Xray** | OPEN | Open-i / HuggingFace: `dz-osamu/IU-Xray` | ~14 GB | M5-M19 main dataset |
| **TAIX-VQA** | OPEN | HuggingFace: `jomoll/TAIX-VQA` (images: `TLAIM/TAIX-Ray`) | small | ICU CXR faithfulness |
| **CRIMSON** | OPEN | GitHub: `rajpurkarlab/CRIMSON` | small | RadJudge/RadPref calibration |
| **ReXGradient-160K** | HF-GATED | HuggingFace: `rajpurkarlab/ReXGradient-160K` (accept license first) | ~160 GB | Scale-up training |
| **MIMIC-CXR** | PHYSIONET | PhysioNet: `mimic-cxr` / `mimic-cxr-jpg` | ~570 GB | (Future scale-up) |

**Download example:**
```bash
# Open datasets
pip install huggingface_hub
huggingface-cli download flaviagiammarino/vqa-rad --local-dir data/vqa-rad
huggingface-cli download mdwiratathya/SLAKE-vqa-english --local-dir data/SLAKE

# IU-Xray (or use the HF mirror)
huggingface-cli download dz-osamu/IU-Xray --local-dir data/IU-Xray-RRG

# Gated dataset (accept license on HF page first)
huggingface-cli download rajpurkarlab/ReXGradient-160K --local-dir data/ReXGradient-160K --token YOUR_HF_TOKEN
```

## Base Models (not included)

All base models are publicly available on HuggingFace:

| Model | HuggingFace ID | Size |
|-------|---------------|------|
| **Qwen2.5-VL-3B-Instruct** | `Qwen/Qwen2.5-VL-3B-Instruct` | 7.1 GB |
| **Qwen2.5-VL-7B-Instruct** | `Qwen/Qwen2.5-VL-7B-Instruct` | 16 GB |
| **Qwen3-VL-4B-Instruct** | `Qwen/Qwen3-VL-4B-Instruct` | 8.3 GB |
| **HuatuoGPT-Vision-7B** | `FreedomIntelligence/HuatuoGPT-Vision-7B` | 17 GB |

## Training Framework

This project uses [EasyR1](https://github.com/hiyouga/EasyR1) (commit `dd71bbd`) with one compatibility patch:

```diff
# verl/workers/fsdp_workers.py — transformers>=5 compatibility
-from transformers.modeling_utils import no_init_weights
+try:
+    from transformers.modeling_utils import no_init_weights
+except ImportError:
+    from transformers.initialization import no_init_weights
```

## External Tools

- **GREEN** (Grounded Radiology Report Evaluation): https://github.com/Stanford-AIMI/GREEN
- **CheXbert**: Used as a verifiable lie-detector (14-label chest finding classifier)
- **RadGraph**: Entity-relation extraction for fine-grained faithfulness verification

## Experiment Overview

| Milestone | Description | Key Finding |
|-----------|-------------|-------------|
| M1 | Judge vulnerability audit | LLM judges exploitable via master-key attacks |
| M2-M4 | Closed-form VQA RL | Strong judge resists short-horizon hacking; gate prototype works |
| M5 | Open-ended report RL (5 conditions) | **Scissors gap confirmed** (judge ~0.5 vs CheXbert-F1 ~0.12) |
| M6 | Routed per-atom gate | Gate preserves faithfulness; high verif_rate makes gate invisible |
| M7 | Master-key attack (report-level) | **Does not reproduce** (honest negative result) |
| M8 | 2nd-seed replication | M6 conclusions seed-stable |
| M9 | RadGraph dual-gate | **Gate becomes visible** — finer signal exposes cross-stream gate effect |
| M11 | Clinical calibration | CheXbert-F1 vs radiologist ρ=0.63; dualgate fixes single-stream blindspots |
| M12-M19 | Extended experiments | Confound controls, long training, emergent downstream harm |

## Environment Setup

Three conda environments were used:
- `easyr1`: Training (GRPO + EasyR1)
- `r1`: Evaluation, lie-detector services (requires nvJitLink fix)
- `chx`: CheXbert CPU inference

Environment specs are in `env_backup/`.

## Gated Access

This repository requires manual approval for access. Please provide your affiliation and intended use when requesting access.

## License

Research use only. Please contact the authors for commercial licensing.