Upload README.md with huggingface_hub

README.md

@@ -11,7 +11,7 @@ tags:
 
 <br/><br/>
 
-[![AstaBench PFB](https://img.shields.io/badge/AstaBench_PaperFindingBench-0.386_(validation)-4c72b0?style=flat-square)](#results)
+[![AstaBench PFB](https://img.shields.io/badge/AstaBench_PaperFindingBench-0.382_(validation)-4c72b0?style=flat-square)](#results)
 [![Cost](https://img.shields.io/badge/cost-%240.13%2Fquery-2ca02c?style=flat-square)](#results)
 [![Python](https://img.shields.io/badge/python-3.11%2B-blue?style=flat-square)](#quickstart)
 [![License](https://img.shields.io/badge/license-MIT-green?style=flat-square)](LICENSE)
@@ -28,7 +28,7 @@ IRIS takes a natural language literature query like
 
 and returns a ranked list of papers where every stated constraint has been checked against the paper's actual text, with verbatim evidence attached to each result.
 
-It is scored under the official AstaBench harness (`astabench 0.5.4`, live GPT-4o grading) and was submitted to Ai2's AstaBench leaderboard as "Rasyn IRIS" on 2026-08-19 (currently in Ai2's review queue).
+It is scored under the official AstaBench harness (`astabench 0.5.4`, live GPT-4o grading). The run submitted to Ai2's AstaBench leaderboard as "Rasyn IRIS" on 2026-08-19 scored 0.386 and is in Ai2's review queue; the code in this repository, re-run afterwards on all 66 validation queries, scores 0.382.
 
 ## Results
 
@@ -36,7 +36,7 @@ AstaBench PaperFindingBench, validation split, official harness:
 
 | System | Adjusted F1 | Cost / query | Source |
 |---|---|---|---|
-| **IRIS (this repo)** | **0.386** | ~$0.13 | our runs, official harness |
+| **IRIS (this repo)** | **0.382** | ~$0.13 | this exact code, official harness, all 66 queries |
 | Asta Paper Finder (Ai2) | 0.380 | ~$0.07 | Ai2's leaderboard results dataset |
 
 On the test split, which IRIS has not run yet, Ai2's Asta Paper Finder scores 0.433 at ~$0.35/query and RoboPhD currently leads at 0.440. Validation and test are different query sets; do not compare numbers across them.
@@ -45,9 +45,10 @@ On the test split, which IRIS has not run yet, Ai2's Asta Paper Finder scores 0.
 
 We would rather you read this here than discover it in the comments.
 
-- **0.386 is the best of six official runs, and here are all six.** 0.384 (early configuration, before the citation-graph fix), 0.366 and 0.357 (fully live retrieval), 0.376 (cache-served, earlier configuration), 0.386 (the shipped configuration, cache-served), 0.384 (a 3-pass tournament variant). Judge nondeterminism moves repeat runs by a few thousandths. Cost ($0.1298/query mean) comes from the certified run's per-sample metering and includes the gpt-4o topic-verification calls on the specific-paper channel (`PFBMAX_TOPIC_MODEL`).
-- **Read 0.386 vs 0.380 as parity under cached corpus access, not victory.** The gap is about the size of our own run-to-run noise, and Ai2's agent costs roughly half as much per query. With fully live retrieval we currently score below them (0.357 and 0.366 vs 0.380); the remaining gap is retrieval infrastructure, not modeling. The claim we stand behind: an open-source system matching Ai2's production paper finder on this split when corpus access is not the bottleneck.
-- **Validation is the public development split.** Like the published baselines, we report validation, and it is also the split we developed against. Treat 0.386 as a dev-split result until our test-split run lands.
+- **We headline the number this code produces, not our best run.** Seven official-harness runs, all of them: 0.384 (early configuration, before the citation-graph fix), 0.366 and 0.357 (fully live retrieval), 0.376 (cache-served, earlier configuration), 0.386 (the run submitted to Ai2), 0.384 (a 3-pass tournament variant), and 0.382 (this repository's code, re-run after the pre-release bug fixes below). The whole spread sits in the semantic slice, which measures 0.228 to 0.249 across repeats because the LLM judge is nondeterministic; the metadata slice (0.682) and specific slice (0.880) are stable. Cost is $0.131/query mean from the re-run's own per-sample metering, and includes the gpt-4o topic-verification calls on the specific-paper channel (`PFBMAX_TOPIC_MODEL`).
+- **Read 0.382 vs 0.380 as a tie, not a win.** The difference is far smaller than our own run-to-run noise, and Ai2's agent costs roughly half as much per query. With fully live retrieval we score below them (0.357 and 0.366 vs 0.380); that gap is retrieval infrastructure, not modeling. The claim we stand behind: an open-source system that matches Ai2's production paper finder on this split when corpus access is not the bottleneck.
+- **Validation is the public development split.** Like the published baselines, we report validation, and it is also the split we developed against. Treat this as a dev-split result until our test-split run lands.
+- **The repository code differs slightly from the submitted run.** Pre-release review found four real bugs (a citation-graph shortcut that dropped query conditions, a rate limiter running at double the configured rate, a truncation that discarded query expansions at raised settings, and silent zero-score runs on missing credentials). They are fixed here and the 0.382 figure is measured with them in place.
 - **Certified runs served corpus retrieval from a local response cache** (built from the same corpus API) because Semantic Scholar's public endpoint aggressively rate-limits; our fully live-retrieval runs scored 0.357 and 0.366. The harness entry point always wraps the corpus client in a read-through cache that fills `pfbmax/cache/` as it runs; `PFBMAX_CACHE_ONLY=1` makes a run serve exclusively from it. If you hit 429s, set `PFBMAX_USE_MCP=1` to route through Asta's MCP gateway (MCP keys are issued at allenai.org/asta/resources, separate from the Semantic Scholar key).
 - **The 923M-edge citation graph is not shipped.** It is a ~360GB build from Semantic Scholar's bulk datasets; this repo does not include the build tooling yet. Without it, citation-constrained metadata queries fall back to live APIs.
 - **The public history is a single clean commit.** The development repository contains benchmark reference data that we are not licensed to redistribute, so the public tree is a clean export rather than the full history.