README.md

---
title: DEFER-RL Reader Study
emoji: 🩻
colorFrom: indigo
colorTo: gray
sdk: gradio
sdk_version: 6.17.3
app_file: app.py
pinned: false
---

# DEFER-RL Radiologist Reader Study

A blinded, multi-reader appropriateness study for the DEFER-RL project. Each case shows one
reference imaging panel beside several anonymized, order-randomized **deferral-system decision
panels** (DEFER-RL plus baselines). Readers rate each panel's decision; the backend derives any
best/worst rankings. No model names are shown.

## What the reader rates (per case)

* **Per panel (one row each):** decision appropriateness (1-5), evidence-gathering adequacy (1-5),
  reading soundness (1-5, only when the panel chose *Trust*), and a yes/no *misleading* judgement
  with an inline definition and worked example.
* **Per case:** "should this case **not** be auto-read?" (yes/partial/no) and "is the reference
  imaging adequate to judge?" (yes/partial/no).

Every scale legend is printed inline, every rating value has a hover tooltip, and each image has its
own display-only zoom / brightness / contrast strip directly beneath it.

## Deploy as a Hugging Face Space

1. Create a **Gradio** Space (SDK version `6.17.3`, set by this README) and upload `app.py`,
   `requirements.txt`, `README.md`. (Gradio is installed from `sdk_version`; do not pin it in
   `requirements.txt`.)
2. Add a **private dataset** for responses (e.g. `your-org/deferrl-reader-responses`).
3. In the Space **Settings -> Variables and secrets**, set:
   * `ANNOTATORS` (secret) - JSON of per-user credentials, e.g.
     `{"dr_smith":"s3cret-a","dr_lee":"s3cret-b"}`. The username each reader types is their
     annotator name and keys their own response file.
   * `DATASET_REPO` (variable) - the private dataset id above.
   * `HF_TOKEN` (secret) - a token with **write** access to that dataset.
   * optional: `COMMIT_EVERY_MIN` (default `1`), `MAX_ITEMS` (default `5`), `DATA_DIR`, `RESP_DIR`.
4. (Recommended) Enable **persistent storage** on the Space so `responses_local/` survives restarts
   between dataset syncs.

If `DATASET_REPO`/`HF_TOKEN` are unset the app still runs and writes responses locally (good for a
dry run). With them set, every **Save & Next** is streamed to the private dataset by a
`CommitScheduler`.

## Loading real cases

Replace the auto-generated sample data by committing your own `data/cases.json` and `data/images/`.
Schema for each case:

```json
{
  "case_id": "C001",
  "cohort": "LIDC-IDRI chest CT",
  "reference_image": "images/C001_ref.png",
  "show_trail": true,
  "ground_truth": {"image": "images/C001_gt.png", "text": "Reference standard: ..."},
  "items": [
    {"item_id": "defer_rl", "action": "Defer", "reading": "(routed to radiologist)",
     "image": "images/C001_defer_rl.png", "trail": ["images/C001_defer_rl_t0.png", "..."]},
    {"item_id": "atcxr", "action": "Trust", "reading": "No suspicious finding. BI-RADS 1.",
     "image": "images/C001_atcxr.png", "trail": ["..."]}
  ]
}
```

* `item_id` is the true system name (never shown to readers); the UI shows blinded "Panel A/B/...".
* `action` is `"Trust"` or `"Defer"`; soundness is only asked for `Trust` panels.
* `show_trail` is the per-case **evidence-trail ablation** condition (saved with every response).
* Put a balanced mix of difficulty / cohort / routing in the manifest for stratified analysis.

If `data/cases.json` is absent, the app generates six synthetic sample cases so the Space runs
immediately; delete them once real data is in place.

## Response schema (robust to UI changes)

Responses are append-only JSONL, one line per **(annotator, case_id, item_id, dimension) -> value**:

```json
{"schema_version":"deferrl-reader-1","ts":"...Z","annotator":"dr_smith","case_id":"C001",
 "item_id":"defer_rl","dimension":"appropriateness","value":"4","presented_pos":2,
 "item_action":"Defer","case_condition_show_trail":true}
```

Case-level answers use `item_id":"__case__"`. Because each value is an atomic, self-describing row,
later changes to layout, controls, or wording can never overwrite or invalidate prior annotations,
and best/worst rankings are derived offline from the per-panel scores.

## Analysis pointers (offline)

* `P_app` = fraction of *Defer* decisions (per system) with median reader rating >= 4.
* Inter-rater agreement: weighted Cohen's kappa and Gwet's AC1 over the ordinal ratings.
* Evidence-trail effect: compare ratings on `show_trail=true` vs `false` cases.
* Rankings: order systems within each case by appropriateness; readers are never asked to rank.

## Notes / limits

* "Zero scrolling" is best on a wide display; each panel is self-contained (its controls, legend, and
  tooltips sit with its image), so you never scroll to learn what a control means. With `MAX_ITEMS`
  large on a small screen, rows may extend below the fold - lower `MAX_ITEMS` or use a wide monitor.
* Zoom/brightness/contrast are pure CSS on the displayed image and never alter stored data.
* Closing the tab keeps you logged in (session cookie); use **Log out** to end the session.