File size: 3,653 Bytes
4ec95a7 74cc258 4ec95a7 74cc258 464011c 4ec95a7 74cc258 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | ---
title: GRACE Reader Study
emoji: 🩺
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 4.44.1
python_version: "3.11"
app_file: app.py
pinned: false
---
# GRACE Reader Study annotation app
Blinded, resume-safe radiologist reader study for the GRACE project. One case per screen,
no scrolling: reference CXR on the left, each anonymized system's output on the right with its
rating controls directly beneath the image it refers to. Scale legends and subjective-term
definitions are printed inline; per-image zoom / brightness / contrast are display-only.
## What a reader does per case
For each anonymized item: **answer correctness** (Correct / Incorrect / Indeterminate),
**decision appropriateness** (1-5), **grounding relevance** (Relevant / Partial / Not relevant),
optional note. Plus one case-level question: is the reference (ground-truth) annotation acceptable.
No best/worst ranking is asked; rankings are derived in the backend from per-item scores.
## Run locally
```bash
pip install -r requirements.txt
python build_cases_example.py # generates data/cases.json + placeholder images (demo)
python app.py # http://127.0.0.1:7860
```
Demo credentials (when READER_CREDENTIALS is unset): `reader1 / changeme`, `reader2 / changeme`.
## Deploy as a Hugging Face Space
1. Create a **Gradio** Space.
2. Upload `app.py`, `requirements.txt`, `README.md`, and (for a demo) the `data/` folder.
For the real study do NOT commit patient images: put `cases.json` + images in a **private
dataset** and set `CASES_DATASET` instead (see below).
3. In **Settings -> Variables and secrets**, add these **secrets** (never commit them):
- `HF_TOKEN` = your HF token with **write** permission (the value in `Keys.txt`; do not paste it into any file).
- `READER_CREDENTIALS` = JSON, e.g. `{"Dr Kotak":"<pw1>","Dr B":"<pw2>","Resident C":"<pw3>"}`.
- `RESPONSE_DATASET` = e.g. `DrSyedFaizan/grace-reader-responses` (created automatically, private).
- `CASES_DATASET` (optional) = e.g. `DrSyedFaizan/grace-reader-cases` (private; pulled at boot).
- `APP_SECRET` (optional) = any random string that signs resume tokens.
## Data flow and storage
- Responses stream to the private dataset `RESPONSE_DATASET` on every **Save & Next**, as
`responses/<reader>.jsonl`, and to a local backup under `local_responses/` (so an HF write
hiccup never loses data).
- **Robust schema:** one append-only record per `(annotator, case_id, item_id)` with a
`dims` dict of value-per-dimension, plus a `__case__` record for case-level answers. Later
changes to layout or wording can never overwrite or invalidate saved annotations; reads take
the latest record per key.
## Auth / session / resume
- First visit always lands on the login page; the app is gated by per-reader credentials.
- On login the reader resumes at their first unfinished case and sees a live `X / N` counter
(their own completed count, not a reset to 1). Item order is shuffled deterministically per
`(reader, case)` so a resumed case looks identical.
- Closing the tab does NOT log out (a signed token is kept in `localStorage`); use the explicit
**Logout** button to end the session.
## Preparing the real cases.json
Run `build_cases_example.py` and read its docstring for the exact schema. Each item's `item_id`
is the TRUE system id (used only in the backend for the blinded GRACE-vs-baseline comparison and
derived rankings); the reader never sees it. Keep `items` per case small (2-3) so the no-scroll
goal holds. Enrich the case set per `../reader_protocol.md` (near-boundary, deferrals, errors,
rare pathology, easy anchors).
|