Card tables relabeled

README.md

@@ -37,14 +37,14 @@ It is `answerdotai/ModernBERT-large` fine-tuned end to end with the public openj
 
 Two numbers are reported throughout. **Answer survival:** among questions whose top-10 holds an answer-bearing passage, the share that still hold one after the filter. **Tokens kept:** kept passage tokens as a share of top-10 passage tokens, over every question. All at the default point rho_99.
 
-| set | questions | answer survival | tokens kept |
-|---|---|---|---|
-| In-distribution test, all 11 sources | 64,948 with an answer passage / 84,874 | 99.1% | 36.8% |
-| Rebuilt retrieval lists: FiQA, SciFact, NQ, FEVER (bge-large top-10) | 7,310 / 7,714 | 98.6% | 22.7% |
-| TREC DL 2019, never trained on | 22 / 43 | 100% | 59.1% |
-| TREC DL 2020 | 38 / 54 | 100% | 61.9% |
-| TREC DL 2023 | 39 / 82 | 94.9% | 56.7% |
-| TREC-COVID | 47 / 50 | 100% | 81.1% |
+| set | questions with an answer-bearing passage | all questions | answer survival | tokens kept |
+|---|---:|---:|---:|---:|
+| In-distribution test, all 11 sources | 64,948 | 84,874 | 99.1% | 36.8% |
+| Rebuilt retrieval lists: FiQA, SciFact, NQ, FEVER (bge-large top-10) | 7,310 | 7,714 | 98.6% | 22.7% |
+| TREC DL 2019, never trained on | 22 | 43 | 100% | 59.1% |
+| TREC DL 2020, never trained on | 38 | 54 | 100% | 61.9% |
+| TREC DL 2023, never trained on | 39 | 82 | 94.9% | 56.7% |
+| TREC-COVID, never trained on | 47 | 50 | 100% | 81.1% |
 
 With the generator reading only the kept passages, answers stay as good: on RGB (1,500 lists, gemini-3.8-flash, RAGAS answer_correctness) 0.736 against 0.732 with every document, at 80% of the context tokens; on CRAG (500 lists) 0.397 against 0.404 at 60% of the tokens. Details under "Answer quality".
 
@@ -156,61 +156,63 @@ Run end to end on a stand-in for your logs (TREC Deep Learning 2019, 2020 and 20
 
 ## Evaluation
 
-Operating-point values come from the calibration split only. Head-to-head tokens are tiktoken `cl100k_base`; in-distribution and external tokens are ModernBERT tokenizer counts. Each cell is survival, then tokens kept.
+Operating-point values come from the calibration split only. Tokens on the rebuilt retrieval lists are tiktoken `cl100k_base`; in-distribution and external tokens are ModernBERT tokenizer counts. In every table, answer survival is the share of the questions with an answer-bearing passage that keep at least one after filtering, and tokens kept is the share of candidate passage tokens kept, over every question of the set.
 
-### External sets (never trained on), at the calibration points
+### External sets, never trained on
 
-TREC DL 2019 / 2020 / 2023: the 10 judged passages BM25 ranks highest within the NIST-judged pool. TREC-COVID: BM25 top-10, 94% judged. Survival is at least one top-grade passage kept (grade 3 TREC DL, grade 2 TREC-COVID); n is questions with a top-grade passage / all.
+TREC DL 2019 / 2020 / 2023: the 10 judged passages BM25 ranks highest within the NIST-judged pool. TREC-COVID: BM25 top-10, 94% judged. An answer-bearing passage here is one judged top grade (grade 3 on TREC DL, grade 2 on TREC-COVID). Four stored operating points; rho_99 is the shipped default.
 
-| set | n | tau_98 | tau_99 | rho_98 | **rho_99 (shipped)** |
-|---|---|---|---|---|---|
-| TREC DL 2019 | 22 / 43 | 100%, 60.4% | 100%, 69.6% | 95.5%, 47.2% | **100%, 59.1%** |
-| TREC DL 2020 | 38 / 54 | 100%, 59.2% | 100%, 69.3% | 100%, 50.0% | **100%, 61.9%** |
-| TREC DL 2023 | 39 / 82 | 94.9%, 51.7% | 94.9%, 63.5% | 94.9%, 46.5% | **94.9%, 56.7%** |
-| TREC-COVID | 47 / 50 | 97.9%, 76.6% | 97.9%, 85.8% | 100%, 66.0% | **100%, 81.1%** |
+| set | questions with a top-grade passage | all questions | tau_98 survival | tau_98 tokens kept | tau_99 survival | tau_99 tokens kept | rho_98 survival | rho_98 tokens kept | rho_99 survival | rho_99 tokens kept |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| TREC DL 2019 | 22 | 43 | 100% | 60.4% | 100% | 69.6% | 95.5% | 47.2% | **100%** | **59.1%** |
+| TREC DL 2020 | 38 | 54 | 100% | 59.2% | 100% | 69.3% | 100% | 50.0% | **100%** | **61.9%** |
+| TREC DL 2023 | 39 | 82 | 94.9% | 51.7% | 94.9% | 63.5% | 94.9% | 46.5% | **94.9%** | **56.7%** |
+| TREC-COVID | 47 | 50 | 97.9% | 76.6% | 97.9% | 85.8% | 100% | 66.0% | **100%** | **81.1%** |
 
 ### In-distribution test, per source
 
-| set | n with a relevant passage / all | tau_98 | tau_99 | rho_98 | **rho_99 (shipped)** |
-|---|---|---|---|---|---|
-| all sources | 64,948 / 84,874 | 98.1%, 37.2% | 99.1%, 44.5% | 98.2%, 31.1% | **99.1%, 36.8%** |
-| 2WikiMultihopQA | 9,913 / 9,913 | 100%, 44.8% | 100%, 46.1% | 100%, 41.6% | 100%, 42.4% |
-| FEVER (sentences) | 6,779 / 6,779 | 99.7%, 49.2% | 99.8%, 56.8% | 99.3%, 32.6% | 99.6%, 37.8% |
-| FEVER (pages) | 8,471 / 8,582 | 99.5%, 17.1% | 99.7%, 21.2% | 99.4%, 12.4% | 99.6%, 13.5% |
-| HotpotQA | 9,853 / 9,853 | 100%, 46.4% | 100%, 53.7% | 100%, 28.7% | 100%, 33.9% |
-| MS MARCO | 15,435 / 25,106 | 94.4%, 46.3% | 97.4%, 57.4% | 94.7%, 45.9% | 97.6%, 57.0% |
-| MuSiQue | 2,300 / 2,300 | 100%, 55.2% | 100%, 61.0% | 99.9%, 40.0% | 100%, 44.6% |
-| NQ | 7,354 / 14,940 | 96.6%, 50.6% | 98.2%, 61.1% | 96.7%, 48.5% | 98.2%, 59.3% |
-| QASPER | 372 / 420 | 98.9%, 83.1% | 99.7%, 91.2% | 98.4%, 59.3% | 99.2%, 72.7% |
-| SciFact | 63 / 83 | 98.4%, 30.2% | 98.4%, 37.0% | 100%, 32.7% | 100%, 44.0% |
-| SciTail | 147 / 147 | 100%, 73.7% | 100%, 83.9% | 99.3%, 48.2% | 100%, 54.8% |
-| SQuAD2 | 4,139 / 6,465 | 98.9%, 34.8% | 99.6%, 45.5% | 99.6%, 23.2% | 99.9%, 30.0% |
-| WikiQA | 122 / 286 | 95.1%, 32.7% | 96.7%, 41.5% | 96.7%, 31.4% | 98.4%, 39.0% |
+The held-out test split, 84,874 questions, by source. Four stored operating points; rho_99 is the shipped default.
+
+| source | questions with a relevant passage | all questions | tau_98 survival | tau_98 tokens kept | tau_99 survival | tau_99 tokens kept | rho_98 survival | rho_98 tokens kept | rho_99 survival | rho_99 tokens kept |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| all sources | 64,948 | 84,874 | 98.1% | 37.2% | 99.1% | 44.5% | 98.2% | 31.1% | **99.1%** | **36.8%** |
+| 2WikiMultihopQA | 9,913 | 9,913 | 100% | 44.8% | 100% | 46.1% | 100% | 41.6% | 100% | 42.4% |
+| FEVER (sentences) | 6,779 | 6,779 | 99.7% | 49.2% | 99.8% | 56.8% | 99.3% | 32.6% | 99.6% | 37.8% |
+| FEVER (pages) | 8,471 | 8,582 | 99.5% | 17.1% | 99.7% | 21.2% | 99.4% | 12.4% | 99.6% | 13.5% |
+| HotpotQA | 9,853 | 9,853 | 100% | 46.4% | 100% | 53.7% | 100% | 28.7% | 100% | 33.9% |
+| MS MARCO | 15,435 | 25,106 | 94.4% | 46.3% | 97.4% | 57.4% | 94.7% | 45.9% | 97.6% | 57.0% |
+| MuSiQue | 2,300 | 2,300 | 100% | 55.2% | 100% | 61.0% | 99.9% | 40.0% | 100% | 44.6% |
+| NQ | 7,354 | 14,940 | 96.6% | 50.6% | 98.2% | 61.1% | 96.7% | 48.5% | 98.2% | 59.3% |
+| QASPER | 372 | 420 | 98.9% | 83.1% | 99.7% | 91.2% | 98.4% | 59.3% | 99.2% | 72.7% |
+| SciFact | 63 | 83 | 98.4% | 30.2% | 98.4% | 37.0% | 100% | 32.7% | 100% | 44.0% |
+| SciTail | 147 | 147 | 100% | 73.7% | 100% | 83.9% | 99.3% | 48.2% | 100% | 54.8% |
+| SQuAD2 | 4,139 | 6,465 | 98.9% | 34.8% | 99.6% | 45.5% | 99.6% | 23.2% | 99.9% | 30.0% |
+| WikiQA | 122 | 286 | 95.1% | 32.7% | 96.7% | 41.5% | 96.7% | 31.4% | 98.4% | 39.0% |
 
 ### Rebuilt retrieval lists, per corpus
 
-bge-large-en-v1.5 dense + BM25 fused top-10, 7,714 queries (7,310 with gold in the top-10), never trained on.
+bge-large-en-v1.5 dense + BM25 fused top-10, 7,714 queries (7,310 with gold in the top-10), never trained on. At the shipped point rho_99.
 
-| corpus | queries with gold in top-10 / all | survival | tokens kept | passages kept |
-|---|---|---:|---:|---:|
-| FEVER | 4,911 / 5,000 | 99.3% | 14.1% | 1.37 |
-| NQ | 1,862 / 1,993 | 97.9% | 53.9% | 5.18 |
-| SciFact | 67 / 73 | 97.0% | 35.1% | 3.45 |
-| FiQA | 470 / 648 | 93.8% | 71.7% | 6.57 |
-| pooled | 7,310 / 7,714 | 98.6% | 22.7% | 2.81 |
+| corpus | queries with gold in top-10 | all queries | answer survival | tokens kept | passages kept per query |
+|---|---:|---:|---:|---:|---:|
+| FEVER | 4,911 | 5,000 | 99.3% | 14.1% | 1.37 |
+| NQ | 1,862 | 1,993 | 97.9% | 53.9% | 5.18 |
+| SciFact | 67 | 73 | 97.0% | 35.1% | 3.45 |
+| FiQA | 470 | 648 | 93.8% | 71.7% | 6.57 |
+| pooled | 7,310 | 7,714 | 98.6% | 22.7% | 2.81 |
 
 ### Choosing an operating point
 
-Survival, then tokens kept, for every stored point. Any point can be named in the call (`point='rho_98'`).
+Every stored point on five sets: the calibration split (81,796 questions, where the values were chosen), the in-distribution test split (84,874), the rebuilt lists pooled (7,714 queries), FiQA alone (648) and TREC-COVID (50). Any point can be named in the call (`point='rho_98'`).
 
-| point | value | calibration | in-distribution test | rebuilt lists pooled | FiQA | TREC-COVID |
-|---|---:|---|---|---|---|---|
-| tau_95 | 0.0874 | 95.0%, 29.3% | 95.2%, 28.4% | 93.4%, 18.6% | 64.7%, 30.7% | 93.6%, 63.5% |
-| tau_98 | 0.0376 | 98.0%, 38.2% | 98.1%, 37.2% | 97.1%, 25.2% | 83.0%, 54.2% | 97.9%, 76.6% |
-| tau_99 | 0.0195 | 99.0%, 45.4% | 99.1%, 44.5% | 98.3%, 31.4% | 88.9%, 71.2% | 97.9%, 85.8% |
-| rho_95 | 0.4897 | 95.0%, 24.2% | 95.3%, 23.4% | 94.2%, 14.5% | 75.5%, 32.6% | 97.9%, 44.7% |
-| rho_98 | 0.2484 | 98.0%, 32.4% | 98.2%, 31.1% | 97.1%, 18.8% | 86.2%, 55.5% | 100%, 66.0% |
-| **rho_99** | **0.1410** | 99.0%, 38.4% | 99.1%, 36.8% | 98.6%, 22.7% | 93.8%, 71.7% | 100%, 81.1% |
+| point | threshold value | calibration survival | calibration tokens kept | test survival | test tokens kept | rebuilt lists survival | rebuilt lists tokens kept | FiQA survival | FiQA tokens kept | TREC-COVID survival | TREC-COVID tokens kept |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| tau_95 | 0.0874 | 95.0% | 29.3% | 95.2% | 28.4% | 93.4% | 18.6% | 64.7% | 30.7% | 93.6% | 63.5% |
+| tau_98 | 0.0376 | 98.0% | 38.2% | 98.1% | 37.2% | 97.1% | 25.2% | 83.0% | 54.2% | 97.9% | 76.6% |
+| tau_99 | 0.0195 | 99.0% | 45.4% | 99.1% | 44.5% | 98.3% | 31.4% | 88.9% | 71.2% | 97.9% | 85.8% |
+| rho_95 | 0.4897 | 95.0% | 24.2% | 95.3% | 23.4% | 94.2% | 14.5% | 75.5% | 32.6% | 97.9% | 44.7% |
+| rho_98 | 0.2484 | 98.0% | 32.4% | 98.2% | 31.1% | 97.1% | 18.8% | 86.2% | 55.5% | 100% | 66.0% |
+| **rho_99 (shipped)** | **0.1410** | **99.0%** | **38.4%** | **99.1%** | **36.8%** | **98.6%** | **22.7%** | **93.8%** | **71.7%** | **100%** | **81.1%** |
 
 At matched survival rho uses fewer tokens than tau on calibration, on the rebuilt lists and on every external set.
 
@@ -218,17 +220,19 @@ At matched survival rho uses fewer tokens than tau on calibration, on the rebuil
 
 Two public RAG benchmarks, the same generator (gemini-3.8-flash, temperature 0) and the benchmark's own prompt three times per question: A gets every document, B gets only the documents Ragzilla keeps at rho_99, C gets none. Scored with RAGAS 0.4.3 (gemini-3.8-flash as judge, gemini-embedding-001 for the embedding terms). RGB: 300 English questions x noise ratio 0 to 0.8 = 1,500 lists of 5 documents, every document tagged answer-bearing or not. CRAG: task 1 dev, 500 questions, the baseline's top-10 sentence chunks.
 
-| | RGB, 1,500 lists | CRAG, 500 lists |
-|---|---|---|
-| documents kept | 3.98 of 5 (95.4% of the answer-bearing ones, 57.8% of the others) | 4.40 of 10 |
-| context tokens B / A | 1,046 / 1,304 = 80.2% | 335 / 557 = 60.2% |
-| answer_correctness A / B / C | 0.732 / 0.736 / 0.131 | 0.404 / 0.397 / 0.286 |
-| faithfulness A / B | 0.974 / 0.968 | 0.330 / 0.314 |
-| answer_relevancy A / B | 0.934 / 0.932 | 0.207 / 0.192 |
-| context_precision A / B | 0.688 / 0.780 | 0.170 / 0.283 |
-| benchmark's own scorer A / B | string accuracy 0.958 / 0.957 | three-way score 0.134 / 0.126 |
-| B vs A per list: wins / ties / losses | 125 / 1,272 / 103 | 24 / 447 / 29 |
-| mean change B minus A, 95% interval | +0.004 [-0.001, +0.008] | -0.007 [-0.026, +0.012] |
+| benchmark | lists | documents per list | documents kept per list | context tokens, all documents | context tokens, Ragzilla-kept | answer_correctness, all documents | answer_correctness, Ragzilla-kept | answer_correctness, no documents | faithfulness, all documents | faithfulness, Ragzilla-kept | answer_relevancy, all documents | answer_relevancy, Ragzilla-kept |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| RGB | 1,500 | 5 | 3.98 | 1,304 | 1,046 | 0.732 | 0.736 | 0.131 | 0.974 | 0.968 | 0.934 | 0.932 |
+| CRAG | 500 | 10 | 4.40 | 557 | 335 | 0.404 | 0.397 | 0.286 | 0.330 | 0.314 | 0.207 | 0.192 |
+
+Per list, Ragzilla-kept against all documents, on answer_correctness (a win or loss is a difference of at least 0.05):
+
+| benchmark | lists | wins | ties | losses | mean change | 95% bootstrap interval | benchmark's own scorer, all documents | benchmark's own scorer, Ragzilla-kept |
+|---|---:|---:|---:|---:|---:|---|---:|---:|
+| RGB | 1,500 | 125 | 1,272 | 103 | +0.004 | -0.001 to +0.008 | string accuracy 0.958 | string accuracy 0.957 |
+| CRAG | 500 | 24 | 447 | 29 | -0.007 | -0.026 to +0.012 | three-way score 0.134 | three-way score 0.126 |
+
+On RGB the filter keeps 95.4% of the answer-bearing documents and 57.8% of the others.
 
 On RGB the filter leaves answer quality unchanged at every noise ratio while cutting the context to 80% of its tokens (66% at the highest noise); 2 of 1,500 lists went from right to wrong on RGB's own scorer because every answer-bearing document was dropped. On CRAG the context drops to 60% at the cost of 10 correct answers of 119 (7 become "I don't know", 3 become wrong); the losses sit in the lists cut to 1 to 3 chunks.
 
@@ -236,10 +240,12 @@ On RGB the filter leaves answer quality unchanged at every noise ratio while cut
 
 Measured on 480 held-out questions, each a question plus its top-10 passages: 240 from the in-distribution test split (mean 1,150 ModernBERT tokens per question, 9.15 passages) and 240 from the rebuilt retrieval lists, 60 per corpus (2,673 tokens, 10 passages). The policy is loaded once (4.7 s on the GPU, 3.5 s on the CPU), 32 warm-up questions are excluded, the point is rho_99. Hardware: one A100-40GB of an a2-highgpu-8g (the box has 48 physical cores; the CPU rows use 12 threads). **Batch 1** is one `decide` call per question. **Packed** is `decide_many` over the set. Both modes kept the same passages on 479 of 480 questions.
 
-| device | trunk dtype | batch 1, median s/question (test / rebuilt lists) | batch 1, p90 s (test / rebuilt lists) | batch 1, questions/s (test / rebuilt lists) | packed, questions/s (test / rebuilt lists) | peak memory |
-|---|---|---:|---:|---:|---:|---|
-| A100-40GB | bf16 | 0.030 / 0.054 | 0.041 / 0.095 | 28.2 / 17.0 | 51.0 / 19.1 | 2.2 GB GPU, 3.4 GB RSS |
-| CPU, 12 threads | fp32 | 0.749 / 3.13 | 2.13 / 7.30 | 0.74 / 0.27 | 0.57 / 0.21 | 5.3 GB RSS |
+| device | trunk dtype | question set | questions | batch 1, median seconds per question | batch 1, p90 seconds per question | batch 1, questions per second | packed, questions per second | peak memory |
+|---|---|---|---:|---:|---:|---:|---:|---|
+| A100-40GB | bf16 | test split | 240 | 0.030 | 0.041 | 28.2 | 51.0 | 2.2 GB GPU, 3.4 GB RSS |
+| A100-40GB | bf16 | rebuilt retrieval lists | 240 | 0.054 | 0.095 | 17.0 | 19.1 | 2.2 GB GPU, 3.4 GB RSS |
+| CPU, 12 threads | fp32 | test split | 240 | 0.749 | 2.13 | 0.74 | 0.57 | 5.3 GB RSS |
+| CPU, 12 threads | fp32 | rebuilt retrieval lists | 240 | 3.13 | 7.30 | 0.27 | 0.21 | 5.3 GB RSS |
 
 On the CPU the bf16 weights are upcast to fp32. Time tracks tokens: the rebuilt lists are 2.3x longer than the test questions.