R2-Bench

R2-Bench is a benchmark dataset for evaluating LLM routing with joint model and token budget optimization. It contains 30,968 queries evaluated across 10 LLMs at 16 token budget levels, with LLM-judge quality scores.

Associated with R2-Router (code), under review at ICML 2026.

Dataset Structure

data/
├── meta-llama/
│   ├── Llama-3.1-70B-Instruct/
│   │   ├── 10_judge.csv
│   │   ├── 20_judge.csv
│   │   ├── ...
│   │   └── 8000_judge.csv
│   └── Llama-3.2-3B-Instruct/
├── Qwen/
│   ├── Qwen3-235B-A22B-Instruct-2507/
│   ├── Qwen3-Next-80B-A3B-Instruct/
│   ├── Qwen3-30B-A3B-Instruct-2507/
│   ├── Qwen2.5-Math-7B-Instruct/
│   ├── Qwen2.5-Math-1.5B-Instruct/
│   └── Qwen3-0.6B/
└── zai-org/
    ├── GLM-4.5-Air/
    └── GLM-4.6/

Models (10)

Token Budgets (16)

10, 20, 30, 40, 50, 80, 100, 150, 200, 300, 500, 800, 1200, 2000, 4000, 8000

Each model is evaluated at every budget level. The LLM is instructed to respond within the given token budget via a system prompt.

File Format

Each {budget}_judge.csv file contains:

Statistics

• Queries: 30,968
• Models: 10
• Budgets: 16
• Total evaluations: ~4.95M (30,968 × 10 × 16)
• Dataset size: ~25 GB

Usage

import pandas as pd

# Load a specific model + budget
df = pd.read_csv("data/Qwen/Qwen3-235B-A22B-Instruct-2507/100_judge.csv")
print(f"Queries: {len(df)}")
print(f"Mean score: {df['correctness_score'].mean():.3f}")
print(f"Mean tokens: {df['actual_token_count'].mean():.0f}")

Usage with R2-Router

These are the training labels for R2-Router's Ridge regression predictors:

from r2_router import R2Router

# The checkpoints in the r2-router repo were trained on this data
router = R2Router.from_pretrained("./r2_router")

Citation

@inproceedings{r2router2026,
  title={R2-Router: A New Paradigm for LLM Routing with Reasoning},
  author={Anonymous},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2026}
}

License

MIT License