# Evaluation

All numbers below were produced by actually running `python scripts/evaluate.py`
in this project's development environment and are saved verbatim in
`data/eval_results.json`. Nothing here is invented. Reproduce them yourself
with the same command — results will differ (likely improve) on a real
Hugging Face Space where the primary transformer models can load; see the
"Backend in use" line below and `docs/architecture.md`.

**Backend in use for this run:** `tfidf-fallback (scikit-learn)` embeddings,
`mmr-fallback` reranking. Corpus: 17 documents / 45 chunks. Eval set: 8
hand-labeled questions with gold-relevant document ids
(`data/eval_questions.json`).

## 12.1 System comparison (mean over the 8-question eval set)

| System | Recall@K | Context Relevance | Evidence Coverage | Citation Correctness | Hallucination Rate (proxy) | Iterations | Latency (s) |
|---|---|---|---|---|---|---|---|
| Basic Vector RAG | 0.669 | 0.231 | 1.000 | 0.594 | 0.406 | 1.00 | 0.0015 |
| Hybrid RAG | 0.794 | 0.031 | 1.000 | 0.648 | 0.352 | 1.00 | 0.0019 |
| Agentic RAG | 0.631 | 0.177 | 0.896 | 0.345 | 0.655 | 1.25 | 0.0330 |

### Honest reading of these numbers

This is not a case of the fancier system automatically winning — and we're
reporting that plainly rather than reshaping the eval set or metrics until
it does:

- **Hybrid RAG has the best recall and citation correctness** on this run.
  Reciprocal Rank Fusion genuinely helps here: it catches keyword-critical
  matches (acronyms like "NMT", "BLEU") that pure semantic search under the
  TF-IDF fallback sometimes misses, without losing the semantic signal.
- **Agentic RAG has the worst citation correctness (0.345) and the highest
  hallucination-rate proxy (0.655) on this run.** The Planner decomposes a
  single gold-labeled question into several sub-questions (e.g. "what
  datasets are used", "what limitations are reported"), and each
  sub-question pulls in its own passages — several of which are relevant to
  the *sub-question* but not to the original question's specific gold
  document set used for scoring. In other words: decomposition trades
  precision against the original question's narrow gold set for broader
  topical coverage. This is a real, known tension in agentic RAG, not a
  bug being hidden — see Future Work below for how we'd address it.
- **Agentic RAG's context relevance (0.177) sits between Basic (0.231) and
  Hybrid's oddly low 0.031.** Hybrid RAG's very low context-relevance number
  is a measurement artifact worth flagging honestly: `context_relevance`
  is computed by re-embedding each passage against the *original* question
  using the Evidence Agent's scorer, but Hybrid RAG's own passages were
  selected by RRF-fused rank, not by that scorer — so passages that ranked
  well under hybrid fusion don't necessarily score well under the
  independent relevance check. This mismatch is a limitation of comparing
  systems with one shared metric function that only one of them was
  designed against; a fairer comparison would give every system's own
  native relevance signal equal standing, which is left for future work.
- **Agentic RAG is ~20x slower** (0.033s vs ~0.002s) — expected, since it
  runs multiple retrieval cycles and several extra scoring passes per
  question. On the TF-IDF fallback this cost is trivial in absolute terms;
  with the real transformer models (embedding + cross-encoder + NLI + LLM
  generation, all CPU-bound on a free Space), expect several seconds per
  question, not milliseconds — still fast enough for an interactive demo.

**Takeaway:** on this small corpus, with the honest fallback backend, the
simpler Hybrid RAG baseline is actually the stronger system by the
precision-oriented metrics used here. The agentic system's real advantage —
catching contradictions and honestly reporting insufficient evidence rather
than guessing — isn't captured by Recall@K/citation-correctness at all; see
§12.3.

## 12.2 Ablation study (mean over the same 8 questions)

| Configuration | Recall@K | Context Relevance | Evidence Coverage | Citation Correctness | Hallucination Rate (proxy) | Iterations | Latency (s) |
|---|---|---|---|---|---|---|---|
| Full Agentic RAG | 0.631 | 0.177 | 0.896 | 0.345 | 0.655 | 1.25 | 0.0103 |
| Without Planner | 0.675 | 0.242 | 1.000 | 0.615 | 0.385 | 1.00 | 0.0031 |
| Without Evidence Verification | 0.631 | **1.000** | 0.969 | 0.281 | 0.719 | 1.00 | 0.0046 |
| Without Adaptive Retrieval | 0.631 | 0.177 | 0.896 | 0.345 | 0.655 | 1.00 | 0.0095 |
| Without Contradiction Detection | 0.631 | 0.177 | 0.896 | 0.345 | 0.655 | 1.25 | 0.0105 |
| Without Final Verification | 0.631 | 0.177 | 0.896 | 0.345 | 0.655 | 1.00 | 0.0093 |

### Honest reading of the ablation

- **"Without Planner" scores better on every precision metric**, confirming
  the same effect described above: on this eval set (built around single,
  fairly specific questions), decomposition spreads retrieval across
  sub-questions the gold-label set didn't anticipate. This is a genuine,
  useful finding about *this eval set's design*, not just the system: an
  eval set built from multi-part research questions (the kind ResearchPilot
  is actually meant for, e.g. "compare X and Y and identify the gaps")
  would need gold labels *per sub-question*, not one flat gold set per
  question, to fairly credit decomposition. That's flagged in Future Work.
- **"Without Evidence Verification" shows context_relevance jumping to a
  perfect 1.000** — but this is an artifact of that ablation's own
  mechanism, not genuine quality: disabling evidence verification forces
  every retrieved passage's `relevance_label` to `SUPPORTED` and
  `relevance_score` to `1.0` by construction (see `_ablation_run` in
  `src/evaluation/benchmark.py`), so of course the average of a
  constant `1.0` is `1.0`. It's included for completeness and to make
  exactly this kind of artifact visible, not to claim the un-verified
  system is actually better.
- **Removing contradiction detection, adaptive retrieval, or final
  verification alone changes almost nothing in this run** because none of
  these 8 questions happened to trigger a second retrieval cycle after the
  domain-floor/threshold fixes, except where noted (`iterations` stays 1.0
  for those ablations vs 1.25 for the full system and the
  contradiction-detection ablation, which still loops since contradiction
  detection isn't what gates the sufficiency check). This is a real
  limitation of an 8-question eval set: it's too small to reliably exercise
  every branch. Expanding `data/eval_questions.json` (see Future Work)
  would give these ablations more room to differentiate.

## 12.3 What these metrics don't capture

The system-comparison table above cannot see two of ResearchPilot's actual
differentiators, because neither Basic nor Hybrid RAG has any notion of
either:

1. **Explicit contradiction surfacing.** Only the Agentic RAG system detects
   and reports the real, verifiable conflict between two papers in the
   corpus reporting different BLEU-point improvements from
   back-translation/data-augmentation methods (`Generalized Data
   Augmentation for Low-Resource Translation` reports 1.5–8 BLEU;
   `Data Augmentation for Low-Resource Neural Machine Translation` reports
   2.9 BLEU). Basic and Hybrid RAG would silently cite both without
   flagging the discrepancy.
2. **Honest insufficiency.** Only the Agentic RAG system will say
   "Insufficient evidence to provide a reliable answer" for an
   off-corpus question (verified directly:
   `tests/test_pipeline.py::test_off_domain_question_reports_insufficiency`)
   instead of returning *some* passages regardless of fit, which is what
   Basic/Hybrid RAG do by construction (they always return their top-K).

Neither of these behaviors is rewarded — or even measured — by Recall@K,
context relevance, or citation correctness. A faithfulness/abstention-aware
metric (e.g., did the system correctly decline to answer an unanswerable
question; did it correctly flag a genuine contradiction) would be needed to
see the agentic approach's actual value in a table, and is a natural
addition to `src/evaluation/metrics.py` (see Future Work).

## 12.4 Reproducing this evaluation

```bash
python scripts/evaluate.py
```

This regenerates `data/eval_results.json` from a live run. On a Hugging
Face Space with the primary transformer models loaded, re-run this and
compare — the README and this document will not be silently "corrected" to
match; if you get materially different results, that's expected and is the
point of shipping the script rather than a static table only.