# Eval set v2

A harder retrieval benchmark for Lumen RAG. It exists because the original set
(`data/eval.jsonl`, 7 documents, 19 questions) is saturated: with `k=5` the
retriever returns 5 of the 7 documents, so recall is about 1.0 for any
configuration and cannot show an improvement.

## Contents

- `docs/` — 120 short documents (60 to 110 words each), one markdown file per
  document. The file name (without `.md`) is the document id.
- `eval.jsonl` — 216 labelled questions.

## Provenance and licence

All documents and questions were written for this repository about a
**fictional** company, "Halden Robotics" (warehouse picking robots). No real
company, person or customer data is used and nothing is copied from another
source, so there are no third-party licence terms. The data is released under
the repository's MIT licence.

## Question categories

| Category | Count | What it tests |
|---|---|---|
| `direct` | 42 | Wording overlaps the document; the easy baseline. |
| `paraphrase` | 51 | Same fact, different words ("holiday days" vs "paid leave"). |
| `disambiguation` | 31 | Several near-identical documents (per region, plan, model or team); only one is right. |
| `multihop` | 29 | The answer needs two documents; both must be retrieved. |
| `unanswerable` | 63 | The corpus does not contain the answer. No relevant docs; not scored for retrieval, used to calibrate abstention. The last 40 are *near misses*: on-topic for a document that exists, but the fact asked for is absent ("Which brand of hardware security key does the company issue?"). |

The corpus is built from 24 topic families of about five documents each (time
off, equipment, expenses, on-call, runbooks, release process, pricing, support
terms, security, data retention, hiring, reviews, robot hardware, robot safety,
fleet API, deployment, procurement, finance, legal, marketing, facilities,
internal tools, engineering practice, customer success). Near-duplicate
documents within a family are deliberate distractors.

## Row format

```json
{"question": "...", "relevant_doc_ids": ["vacation-eu"], "expected_answer": "30 working days.",
 "category": "direct", "answerable": true,
 "evidence": [{"doc_id": "vacation-eu", "span": "receive 30 working days of annual paid leave"}]}
```

`evidence[].span` is a verbatim excerpt of the cited document, so every answerable
label can be checked mechanically. Unanswerable rows have `relevant_doc_ids: []`,
`answerable: false`, no `evidence` and no `expected_answer`.

## Using it

```bash
python scripts/validate_eval_set.py            # consistency checks (ids, spans, duplicates, categories)
python scripts/benchmark.py --set v2           # configurations x categories table
lumen ingest data/eval_v2/docs
lumen eval data/eval_v2/eval.jsonl --k 5       # one-off scorecard with per-category breakdown
lumen eval-answers data/eval_v2/eval.jsonl --claims   # why answers go wrong: retrieval vs generation
```

## Baseline (offline hashing embedder, no reranker, k=5)

| Configuration | recall@5 | MRR | nDCG@5 |
|---|---|---|---|
| vector-only | 0.74 | 0.69 | 0.69 |
| BM25 only | 0.90 | 0.88 | 0.87 |
| hybrid (BM25 + RRF) | 0.82 | 0.73 | 0.74 |

The spread between best and worst configuration is 0.19 MRR (the M0 target was at
least 0.10). `tests/test_eval_set_v2.py` guards this.

## Known limits

- The documents are short, so chunking has no effect on this set.
- Paraphrases still share some vocabulary with the source documents, which keeps
  BM25 strong. A set written against a real semantic embedder would be harder.
- The cross-encoder reranker was not measured (it needs a model download).
- Written by one author with LLM assistance; questions and answers were checked
  mechanically (spans, ids) but not by independent annotators.

## Extending it

Add a document under `docs/`, add rows to `eval.jsonl` with a verbatim `evidence`
span, then run `python scripts/validate_eval_set.py`. Keep one fact per question
and add distractor documents when you add a new topic.