| # Oncology RAG Benchmark |
|
|
| An oncology-specific, CC-BY-4.0 retrieval-augmented-generation (RAG) benchmark |
| built from the Cancer Knowledge Base. Answer keys are **provable from the KB's own |
| structured data** — every question carries golden docs, an evidence level, and a |
| citation. |
|
|
| ## Files |
|
|
| | File | Contents | |
| |------|----------| |
| | `retrieval_pool.parquet` | ~1,100 chunked retrieval units (sections with heading weighting, markdown tables, and domain tables: codes / drugs / staging / prognosis / subtypes / biomarkers) | |
| | `mcq_benchmark.parquet` | 112 multiple-choice questions with `options`, `answer`, `answer_index`, `difficulty`, `evidence_level`, `citation`, `golden_docs`, `sources` | |
| | `mcq_robustness.parquet` | 336 MedRGB-style variants — `sufficiency` (golden docs), `noise` (golden + 5 irrelevant), `robustness` (golden + a counterfactually edited doc) | |
| | `leaderboard.json` | recorded results from `eval_mcq.py --write-leaderboard` | |
| | `mirage_format.jsonl` | 112 questions in MIRAGE-family interchange format (context = golden docs) | |
| | `medrgb_format.jsonl` | 336 robustness variants in MedRGB interchange format (incl. `counterfactual_doc`) | |
|
|
| See [`MIRAGE_INTEGRATION.md`](MIRAGE_INTEGRATION.md) for how this maps onto the |
| MIRAGE/MedRGB evaluation ecosystem and how to submit a system there. |
|
|
| ## Scenarios |
|
|
| * **sufficiency** — answer from the golden docs only (upper-bound signal). |
| * **noise** — answer from golden + irrelevant docs (measures retrieval robustness). |
| * **robustness** — golden + one adversarially-edited doc that contradicts the answer (measures hallucination resistance). |
|
|
| ## Evaluate your system |
|
|
| Install the repo scripts (needs `pyarrow`): |
|
|
| ```sh |
| # score your own predictions (JSONL: {"question_id": 1, "answer_index": 2}) |
| python3 scripts/eval_mcq.py --benchmark hf_dataset/benchmark --predictions preds.jsonl |
| |
| # baseline with real BM25 retrieval (no deps) |
| python3 scripts/eval_mcq.py --benchmark hf_dataset/benchmark --retriever bm25 --topk 3 |
| |
| # record your result on the leaderboard |
| python3 scripts/eval_mcq.py --retriever bm25 --topk 3 --write-leaderboard \ |
| --system-name "my-system" |
| |
| # compare leaderboard entries |
| python3 scripts/eval_mcq.py --compare my-system retriever:bm25@top3 |
| ``` |
|
|
| Prediction JSONL format: |
|
|
| ```json |
| {"question_id": 1, "answer_index": 2} |
| {"question_id": 2, "answer_index": 0} |
| ``` |
|
|
| ## Submission |
|
|
| 1. Run `eval_mcq.py` with `--write-leaderboard --system-name "<your system>"`. |
| 2. Open a PR / issue against this dataset repo describing the system (LLM, |
| retriever, top-k) and attach your `preds.jsonl` for reproducibility. |
|
|
| ## Baseline |
|
|
| Current `retriever:bm25@top3` baseline (dependency-free BM25): |
|
|
| * sufficiency accuracy: **33.0%** |
| * retrieval@3 (golden doc in top-3): **49.1%** |
|
|
| Random guessing is 25% (4-option questions), so the baseline beats chance and |
| shows real headroom for better retrievers (embeddings) and generators. |
|
|
