| NOTICE |
| ====== |
|
|
| This repository is the public research artifact for the EMNLP 2026 paper |
| "LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration |
| Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks" |
| (arXiv:2608.14927). |
|
|
| Code in this repository is released under the MIT License (see LICENSE). |
|
|
| -------------------------------------------------------------------------- |
| Upstream benchmarks |
| -------------------------------------------------------------------------- |
|
|
| This artifact reports measured outcomes of language-model protocol executions |
| on problems drawn from the benchmarks below. We do NOT redistribute upstream |
| problem text or gold answers. We release stable problem identifiers, derived |
| per-protocol outcomes, and instructions for reconstructing the inputs from the |
| pinned upstream releases. Each benchmark remains subject to its own license. |
|
|
| * Omni-MATH - https://huggingface.co/datasets/KbsdJames/Omni-MATH |
| License: Apache-2.0. |
| Used as a filtered subset of 4,181 competition-mathematics problems. |
|
|
| * JEEBench - https://huggingface.co/datasets/daman1209arora/jeebench |
| License: MIT. 515 problems. |
|
|
| * SciBench - https://huggingface.co/datasets/xw27/scibench |
| License: MIT (see the upstream repository's LICENSE file; the Hugging Face |
| dataset card does not currently declare a license tag). 565 problems. |
|
|
| * LAB-Bench - https://huggingface.co/datasets/futurehouse/lab-bench |
| License: CC-BY-SA-4.0. FutureHouse. |
| Pinned upstream revision: 5c77cec648430f30611808808861eb86f81d5eaa. |
| Two text-only slices are used: a strict slice (741 problems) and a broader |
| text-no-tool slice (1,542 problems). Image-dependent subsets (FigQA, TableQA) |
| are excluded because the protocols studied here are text-only. |
|
|
| Please cite the original benchmark authors when using these identifiers. |
|
|
| -------------------------------------------------------------------------- |
| Foundation models |
| -------------------------------------------------------------------------- |
|
|
| The solver families evaluated here (openai/gpt-oss-120b and |
| google/gemma-4-31B-it) are third-party models. This artifact does not |
| redistribute, retrain, or claim ownership of those model weights. |
|
|
| -------------------------------------------------------------------------- |
| Acknowledgments |
| -------------------------------------------------------------------------- |
|
|
| This research used resources of the Argonne Leadership Computing Facility, a |
| U.S. Department of Energy (DOE) Office of Science user facility at Argonne |
| National Laboratory (ANL) operated under Contract No. DE-AC02-06CH11357. |
|
|
