# Oncology RAG Benchmark

An oncology-specific, CC-BY-4.0 retrieval-augmented-generation (RAG) benchmark
built from the Cancer Knowledge Base. Answer keys are **provable from the KB's own
structured data** — every question carries golden docs, an evidence level, and a
citation.

## Files

| File | Contents |
|------|----------|
| `retrieval_pool.parquet` | ~1,100 chunked retrieval units (sections with heading weighting, markdown tables, and domain tables: codes / drugs / staging / prognosis / subtypes / biomarkers) |
| `mcq_benchmark.parquet` | 112 multiple-choice questions with `options`, `answer`, `answer_index`, `difficulty`, `evidence_level`, `citation`, `golden_docs`, `sources` |
| `mcq_robustness.parquet` | 336 MedRGB-style variants — `sufficiency` (golden docs), `noise` (golden + 5 irrelevant), `robustness` (golden + a counterfactually edited doc) |
| `leaderboard.json` | recorded results from `eval_mcq.py --write-leaderboard` |
| `mirage_format.jsonl` | 112 questions in MIRAGE-family interchange format (context = golden docs) |
| `medrgb_format.jsonl` | 336 robustness variants in MedRGB interchange format (incl. `counterfactual_doc`) |

See [`MIRAGE_INTEGRATION.md`](MIRAGE_INTEGRATION.md) for how this maps onto the
MIRAGE/MedRGB evaluation ecosystem and how to submit a system there.

## Scenarios

* **sufficiency** — answer from the golden docs only (upper-bound signal).
* **noise** — answer from golden + irrelevant docs (measures retrieval robustness).
* **robustness** — golden + one adversarially-edited doc that contradicts the answer (measures hallucination resistance).

## Evaluate your system

Install the repo scripts (needs `pyarrow`):

```sh
# score your own predictions (JSONL: {"question_id": 1, "answer_index": 2})
python3 scripts/eval_mcq.py --benchmark hf_dataset/benchmark --predictions preds.jsonl

# baseline with real BM25 retrieval (no deps)
python3 scripts/eval_mcq.py --benchmark hf_dataset/benchmark --retriever bm25 --topk 3

# record your result on the leaderboard
python3 scripts/eval_mcq.py --retriever bm25 --topk 3 --write-leaderboard \
    --system-name "my-system"

# compare leaderboard entries
python3 scripts/eval_mcq.py --compare my-system retriever:bm25@top3
```

Prediction JSONL format:

```json
{"question_id": 1, "answer_index": 2}
{"question_id": 2, "answer_index": 0}
```

## Submission

1. Run `eval_mcq.py` with `--write-leaderboard --system-name "<your system>"`.
2. Open a PR / issue against this dataset repo describing the system (LLM,
   retriever, top-k) and attach your `preds.jsonl` for reproducibility.

## Baseline

Current `retriever:bm25@top3` baseline (dependency-free BM25):

* sufficiency accuracy: **33.0%**
* retrieval@3 (golden doc in top-3): **49.1%**

Random guessing is 25% (4-option questions), so the baseline beats chance and
shows real headroom for better retrievers (embeddings) and generators.