| === Generating auto-qrel files for trec-dl-2019 === | |
| Generating: pool-40-systems-top10-rerank-judge | |
| [direct] queries=43 judged_docs=2187 relevant=1874 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.direct.txt | |
| [thresholding@0.5] queries=43 judged_docs=2187 relevant=1874 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.thresholding@0.5.txt | |
| [rank@10] queries=43 judged_docs=2187 relevant=430 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.rank@10.txt | |
| [largest_gap@1] queries=43 judged_docs=2187 relevant=1097 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.largest_gap@1.txt | |
| [quantile_binary@0.75] queries=43 judged_docs=2187 relevant=1322 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.quantile_binary@0.75.txt | |
| [quantile_bucket] queries=41 judged_docs=2069 relevant=1422 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.quantile_bucket.txt | |
| [optimal_per_topic] queries=43 judged_docs=2187 relevant=1823 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.optimal_per_topic.txt | |
| [optimal_global] queries=43 judged_docs=2187 relevant=1874 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.optimal_global.txt | |
| [optimal_precision] queries=43 judged_docs=2187 relevant=1080 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.optimal_precision.txt | |
| [optimal_recall] queries=43 judged_docs=2187 relevant=1867 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge/trec-dl-2019/autollmqrel.optimal_recall.txt | |
| Generating: pool-40-systems-top10-rerank-judge_expr | |
| [direct] queries=43 judged_docs=2187 relevant=1615 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.direct.txt | |
| [thresholding@0.5] queries=43 judged_docs=2187 relevant=1615 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.thresholding@0.5.txt | |
| [rank@10] queries=43 judged_docs=2187 relevant=430 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.rank@10.txt | |
| [largest_gap@1] queries=43 judged_docs=2187 relevant=1510 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.largest_gap@1.txt | |
| [quantile_binary@0.75] queries=43 judged_docs=2187 relevant=562 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.quantile_binary@0.75.txt | |
| [quantile_bucket] queries=43 judged_docs=2187 relevant=1949 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.quantile_bucket.txt | |
| [optimal_per_topic] queries=43 judged_docs=2187 relevant=1753 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.optimal_per_topic.txt | |
| [optimal_global] queries=43 judged_docs=2187 relevant=1782 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.optimal_global.txt | |
| [optimal_precision] queries=43 judged_docs=2187 relevant=411 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.optimal_precision.txt | |
| [optimal_recall] queries=43 judged_docs=2187 relevant=1819 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-judge_expr/trec-dl-2019/autollmqrel.optimal_recall.txt | |
| Generating: pool-40-systems-top10-rerank-point | |
| [direct] queries=43 judged_docs=2187 relevant=1611 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.direct.txt | |
| [thresholding@0.5] queries=43 judged_docs=2187 relevant=1617 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.thresholding@0.5.txt | |
| [rank@10] queries=43 judged_docs=2187 relevant=430 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.rank@10.txt | |
| [largest_gap@1] queries=43 judged_docs=2187 relevant=1611 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.largest_gap@1.txt | |
| [quantile_binary@0.75] queries=43 judged_docs=2187 relevant=580 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.quantile_binary@0.75.txt | |
| [quantile_bucket] queries=43 judged_docs=2187 relevant=1946 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.quantile_bucket.txt | |
| [optimal_per_topic] queries=43 judged_docs=2187 relevant=1697 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.optimal_per_topic.txt | |
| [optimal_global] queries=43 judged_docs=2187 relevant=1767 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.optimal_global.txt | |
| [optimal_precision] queries=43 judged_docs=2187 relevant=427 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.optimal_precision.txt | |
| [optimal_recall] queries=43 judged_docs=2187 relevant=1806 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-point/trec-dl-2019/autollmqrel.optimal_recall.txt | |
| Generating: pool-40-systems-top10-rerank-rankgpt | |
| [direct] queries=43 judged_docs=2187 relevant=43 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.direct.txt | |
| [thresholding@0.5] queries=43 judged_docs=2187 relevant=86 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.thresholding@0.5.txt | |
| [rank@10] queries=43 judged_docs=2187 relevant=430 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.rank@10.txt | |
| [largest_gap@1] queries=43 judged_docs=2187 relevant=43 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.largest_gap@1.txt | |
| [quantile_binary@0.75] queries=43 judged_docs=2187 relevant=562 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.quantile_binary@0.75.txt | |
| [quantile_bucket] queries=43 judged_docs=2187 relevant=1949 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.quantile_bucket.txt | |
| [optimal_per_topic] queries=43 judged_docs=2187 relevant=1638 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.optimal_per_topic.txt | |
| [optimal_global] queries=43 judged_docs=2187 relevant=2096 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.optimal_global.txt | |
| [optimal_precision] queries=43 judged_docs=2187 relevant=419 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.optimal_precision.txt | |
| [optimal_recall] queries=43 judged_docs=2187 relevant=1771 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-rankgpt/trec-dl-2019/autollmqrel.optimal_recall.txt | |
| Generating: pool-40-systems-top10-rerank-setmaxheaptopk | |
| [direct] queries=43 judged_docs=2187 relevant=43 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.direct.txt | |
| [thresholding@0.5] queries=43 judged_docs=2187 relevant=86 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.thresholding@0.5.txt | |
| [rank@10] queries=43 judged_docs=2187 relevant=430 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.rank@10.txt | |
| [largest_gap@1] queries=43 judged_docs=2187 relevant=43 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.largest_gap@1.txt | |
| [quantile_binary@0.75] queries=43 judged_docs=2187 relevant=562 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.quantile_binary@0.75.txt | |
| [quantile_bucket] queries=43 judged_docs=2187 relevant=1949 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.quantile_bucket.txt | |
| [optimal_per_topic] queries=43 judged_docs=2187 relevant=1799 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.optimal_per_topic.txt | |
| [optimal_global] queries=43 judged_docs=2187 relevant=2168 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.optimal_global.txt | |
| [optimal_precision] queries=43 judged_docs=2187 relevant=408 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.optimal_precision.txt | |
| [optimal_recall] queries=43 judged_docs=2187 relevant=1921 -> /users/judylan1/APRIL/qrel-analysis/autoqrels/pool-40-systems-top10-rerank-setmaxheaptopk/trec-dl-2019/autollmqrel.optimal_recall.txt | |
| === nDCG@10 — human qrel baseline === | |
| run nDCG@10 | |
| bm25 0.5058 | |
| splade-v3 0.7270 | |
| nomicai-modernbert-embed 0.6960 | |
| qwen3-embed-600m 0.6693 | |
| colbert-small 0.7213 | |
| bm25-rerank-judge 0.6488 | |
| bm25-rerank-judge_expr 0.7018 | |
| bm25-rerank-point 0.6298 | |
| bm25-rerank-rankgpt 0.7442 | |
| bm25-rerank-setmaxheaptopk 0.7074 | |
| bm25-rerank-rankfirst 0.7284 | |
| bm25-rerank-rankzephyr 0.7378 | |
| splade-v3-rerank-judge 0.6625 | |
| splade-v3-rerank-judge_expr 0.7180 | |
| splade-v3-rerank-point 0.6417 | |
| splade-v3-rerank-rankgpt 0.7676 | |
| splade-v3-rerank-setmaxheaptopk 0.7495 | |
| splade-v3-rerank-rankfirst 0.7655 | |
| splade-v3-rerank-rankzephyr 0.7748 | |
| nomicai-modernbert-embed-rerank-judge 0.6567 | |
| nomicai-modernbert-embed-rerank-judge_expr 0.6999 | |
| nomicai-modernbert-embed-rerank-point 0.6479 | |
| nomicai-modernbert-embed-rerank-rankgpt 0.7809 | |
| nomicai-modernbert-embed-rerank-setmaxheaptopk 0.7330 | |
| nomicai-modernbert-embed-rerank-rankfirst 0.7551 | |
| nomicai-modernbert-embed-rerank-rankzephyr 0.7738 | |
| qwen3-embed-600m-rerank-judge 0.6533 | |
| qwen3-embed-600m-rerank-judge_expr 0.7074 | |
| qwen3-embed-600m-rerank-point 0.6457 | |
| qwen3-embed-600m-rerank-rankgpt 0.7498 | |
| qwen3-embed-600m-rerank-setmaxheaptopk 0.7142 | |
| qwen3-embed-600m-rerank-rankfirst 0.7383 | |
| qwen3-embed-600m-rerank-rankzephyr 0.7567 | |
| colbert-small-rerank-judge 0.6643 | |
| colbert-small-rerank-judge_expr 0.7153 | |
| colbert-small-rerank-point 0.6452 | |
| colbert-small-rerank-rankgpt 0.7656 | |
| colbert-small-rerank-setmaxheaptopk 0.7422 | |
| colbert-small-rerank-rankfirst 0.7717 | |
| colbert-small-rerank-rankzephyr 0.7683 | |
| === nDCG@10 — auto-qrels === | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.direct.txt | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.largest_gap@1.txt | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.optimal_global.txt | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.optimal_per_topic.txt | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.optimal_precision.txt | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.optimal_recall.txt | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.quantile_binary@0.75.txt | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.quantile_bucket.txt | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.rank@10.txt | |
| Writing: pool-40-systems-top10-rerank-judge.autollmqrel.thresholding@0.5.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.direct.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.largest_gap@1.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.optimal_global.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.optimal_per_topic.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.optimal_precision.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.optimal_recall.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.quantile_binary@0.75.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.quantile_bucket.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.rank@10.txt | |
| Writing: pool-40-systems-top10-rerank-judge_expr.autollmqrel.thresholding@0.5.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.direct.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.largest_gap@1.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.optimal_global.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.optimal_per_topic.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.optimal_precision.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.optimal_recall.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.quantile_binary@0.75.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.quantile_bucket.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.rank@10.txt | |
| Writing: pool-40-systems-top10-rerank-point.autollmqrel.thresholding@0.5.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.direct.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.largest_gap@1.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.optimal_global.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.optimal_per_topic.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.optimal_precision.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.optimal_recall.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.quantile_binary@0.75.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.quantile_bucket.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.rank@10.txt | |
| Writing: pool-40-systems-top10-rerank-rankgpt.autollmqrel.thresholding@0.5.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.direct.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.largest_gap@1.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.optimal_global.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.optimal_per_topic.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.optimal_precision.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.optimal_recall.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.quantile_binary@0.75.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.quantile_bucket.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.rank@10.txt | |
| Writing: pool-40-systems-top10-rerank-setmaxheaptopk.autollmqrel.thresholding@0.5.txt | |
Xet Storage Details
- Size:
- 16.2 kB
- Xet hash:
- 5b3765e33b0860c3a4902e9b9041ba81cc6c17176f43bd2a1afadb042e72ba7b
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.