| # FEVER Benchmark — Controlled Retrieval Evaluation |
|
|
| A frozen, reproducible benchmark for testing retrieval methods on the BEIR FEVER dataset. |
|
|
| ## Quick Start |
|
|
| ```python |
| from fever_benchmark import FEVERBenchmark |
| |
| bench = FEVERBenchmark() |
| |
| # Load the BM25 top-100 pool |
| pool = bench.load_pool("beir_pool.json") |
| |
| # Re-rank with your method |
| my_rankings = {} |
| for qid, docs in pool.items(): |
| # docs is [(docid, bm25_score), ...] |
| # Replace with your scores: |
| my_rankings[qid] = [(docid, your_score(docid, qid)) |
| for docid, _ in docs] |
| # Sort descending by score |
| my_rankings[qid].sort(key=lambda x: -x[1]) |
| |
| # Evaluate |
| results = bench.evaluate(my_rankings) |
| print(f"NDCG@10: {results['ndcg@10']:.4f}") |
| |
| # Compare against baselines |
| bench.verify_reproduction({"Your Method": results["ndcg@10"]}) |
| ``` |
|
|
| ## What's Included |
|
|
| | File | Description | |
| |---|---| |
| | `beir_pool.json` | BM25 top-100 pool (k1=1.2, b=0.75) — 6,666 queries | |
| | `fever_benchmark.py` | Benchmark evaluation class | |
| | `setup_fever_benchmark.py` | Regenerate pool from scratch | |
| | `query_deltas.csv` | Per-query NDCG@10 for all Hadith variants | |
| | `benchmark_manifest.md` | Frozen configuration and checksums | |
|
|
| ## Baseline Scores |
|
|
| | System | NDCG@10 | |
| |---|---| |
| | BM25 (k1=1.2, b=0.75) | 0.5214 | |
| | MiniLM Dense | 0.6497 | |
| | Dense + Muttafaq (best Hadith) | 0.6461 | |
|
|
| To verify reproduction: run your rankings through `bench.evaluate()` and check against these numbers. |
|
|
| ## Setup from Scratch |
|
|
| ```bash |
| python setup_fever_benchmark.py |
| ``` |
|
|
| This downloads BEIR FEVER (~3.3 GB), builds the Pyserini index (~6 min), and runs BM25 retrieval (~7 min). |
|
|
| ## Requirements |
|
|
| - Python 3.8+ |
| - `pyserini>=0.14.0` |
| - Java 11+ (for Pyserini/Lucene) |
|
|
| ## License |
|
|
| Same as BEIR FEVER — research use. |
|
|
