README.md

metadata

title: DEFER-RL Reader Study
emoji: 🩻
colorFrom: indigo
colorTo: gray
sdk: gradio
sdk_version: 6.17.3
app_file: app.py
pinned: false

DEFER-RL Radiologist Reader Study

A blinded, multi-reader appropriateness study for the DEFER-RL project. Each case shows one reference imaging panel beside several anonymized, order-randomized deferral-system decision panels (DEFER-RL plus baselines). Readers rate each panel's decision; the backend derives any best/worst rankings. No model names are shown.

What the reader rates (per case)

• Per panel (one row each): decision appropriateness (1-5), evidence-gathering adequacy (1-5), reading soundness (1-5, only when the panel chose Trust), and a yes/no misleading judgement with an inline definition and worked example.
• Per case: "should this case not be auto-read?" (yes/partial/no) and "is the reference imaging adequate to judge?" (yes/partial/no).

Every scale legend is printed inline, every rating value has a hover tooltip, and each image has its own display-only zoom / brightness / contrast strip directly beneath it.

Deploy as a Hugging Face Space

1. Create a Gradio Space (SDK version 6.17.3, set by this README) and upload app.py, requirements.txt, README.md. (Gradio is installed from sdk_version; do not pin it in requirements.txt.)
2. Add a private dataset for responses (e.g. your-org/deferrl-reader-responses).
3. In the Space Settings -> Variables and secrets, set:
   • ANNOTATORS (secret) - JSON of per-user credentials, e.g. {"dr_smith":"s3cret-a","dr_lee":"s3cret-b"}. The username each reader types is their annotator name and keys their own response file.
   • DATASET_REPO (variable) - the private dataset id above.
   • HF_TOKEN (secret) - a token with write access to that dataset.
   • optional: COMMIT_EVERY_MIN (default 1), MAX_ITEMS (default 5), DATA_DIR, RESP_DIR.
4. (Recommended) Enable persistent storage on the Space so responses_local/ survives restarts between dataset syncs.

If DATASET_REPO/HF_TOKEN are unset the app still runs and writes responses locally (good for a dry run). With them set, every Save & Next is streamed to the private dataset by a CommitScheduler.

Loading real cases

Replace the auto-generated sample data by committing your own data/cases.json and data/images/. Schema for each case:

{
  "case_id": "C001",
  "cohort": "LIDC-IDRI chest CT",
  "reference_image": "images/C001_ref.png",
  "show_trail": true,
  "ground_truth": {"image": "images/C001_gt.png", "text": "Reference standard: ..."},
  "items": [
    {"item_id": "defer_rl", "action": "Defer", "reading": "(routed to radiologist)",
     "image": "images/C001_defer_rl.png", "trail": ["images/C001_defer_rl_t0.png", "..."]},
    {"item_id": "atcxr", "action": "Trust", "reading": "No suspicious finding. BI-RADS 1.",
     "image": "images/C001_atcxr.png", "trail": ["..."]}
  ]
}

• item_id is the true system name (never shown to readers); the UI shows blinded "Panel A/B/...".
• action is "Trust" or "Defer"; soundness is only asked for Trust panels.
• show_trail is the per-case evidence-trail ablation condition (saved with every response).
• Put a balanced mix of difficulty / cohort / routing in the manifest for stratified analysis.

If data/cases.json is absent, the app generates six synthetic sample cases so the Space runs immediately; delete them once real data is in place.

Response schema (robust to UI changes)

Responses are append-only JSONL, one line per (annotator, case_id, item_id, dimension) -> value:

{"schema_version":"deferrl-reader-1","ts":"...Z","annotator":"dr_smith","case_id":"C001",
 "item_id":"defer_rl","dimension":"appropriateness","value":"4","presented_pos":2,
 "item_action":"Defer","case_condition_show_trail":true}

Case-level answers use item_id":"__case__". Because each value is an atomic, self-describing row, later changes to layout, controls, or wording can never overwrite or invalidate prior annotations, and best/worst rankings are derived offline from the per-panel scores.

Analysis pointers (offline)

• P_app = fraction of Defer decisions (per system) with median reader rating >= 4.
• Inter-rater agreement: weighted Cohen's kappa and Gwet's AC1 over the ordinal ratings.
• Evidence-trail effect: compare ratings on show_trail=true vs false cases.
• Rankings: order systems within each case by appropriateness; readers are never asked to rank.

Notes / limits

• "Zero scrolling" is best on a wide display; each panel is self-contained (its controls, legend, and tooltips sit with its image), so you never scroll to learn what a control means. With MAX_ITEMS large on a small screen, rows may extend below the fold - lower MAX_ITEMS or use a wide monitor.
• Zoom/brightness/contrast are pure CSS on the displayed image and never alter stored data.
• Closing the tab keeps you logged in (session cookie); use Log out to end the session.