|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- multilingual |
|
|
base_model: |
|
|
- Qwen/Qwen3-4B-Instruct-2507 |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- Model Routing |
|
|
- LLM reasoning |
|
|
--- |
|
|
|
|
|
# SCOPE: Scalable and Controllable Outcome Performance Estimator |
|
|
[📄 Paper (arXiv:2601.22323)](https://www.arxiv.org/abs/2601.22323) |
|
|
|
|
|
This repository accompanies the paper “**Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning**”, which introduces SCOPE (Scalable and Controllable Outcome Performance Estimator) — a new framework for large language model (LLM) routing. |
|
|
SCOPE reframes model routing as a pre-hoc estimation problem: instead of directly selecting a model from a fixed candidate set, it predicts each model’s expected performance (correctness) and inference cost (token length) before execution, based on the model’s historical behaviors on similar queries. This enables training-free generalization to unseen models and allows users to flexibly control the trade-off between accuracy and cost through a budget-aware utility function. |
|
|
Overall, SCOPE provides a scalable, explainable, and controllable solution for allocating test-time compute across heterogeneous model portfolios. |
|
|
|
|
|
<p align="center"> |
|
|
<img src="assets/1.png" width="500"> |
|
|
</p> |
|
|
The figure above illustrates the core difference between traditional routers and SCOPE. |
|
|
Conventional LLM routers treat routing as a closed-set classification problem, simply memorizing model names and selecting one model per query. In contrast, SCOPE reasons over models’ past behaviors, explicitly predicting outcome correctness and token cost, and then makes a budget-aware decision based on these estimates. This design allows SCOPE to generalize to unseen models and supports dynamic cost–accuracy control at inference time. |
|
|
|
|
|
|
|
|
## Model Description |
|
|
|
|
|
- **Task**: Performance prediction for LLMs |
|
|
- **Base Model**: Qwen/Qwen3-4B-Instruct-2507 |
|
|
- **Training**: Supervised Fine-Tuning (SFT) + Reinforcement Learning (GRPO) |
|
|
- **Input**: Target question + k anchor questions with performance data |
|
|
- **Output**: Predicted length (tokens) and correctness (yes/no) |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
SCOPE is designed to: |
|
|
- Predict whether an LLM will answer a question correctly before running expensive inference |
|
|
- Estimate the output token length for resource planning |
|
|
- Enable efficient LLM routing and selection |
|
|
|
|
|
## Quick Start |
|
|
|
|
|
### Installation |
|
|
|
|
|
```bash |
|
|
pip install transformers>=4.51.0 torch datasets |
|
|
# For vLLM inference (optional but recommended) |
|
|
pip install vllm |
|
|
``` |
|
|
|
|
|
### Input Format |
|
|
|
|
|
SCOPE uses the following prompt format: |
|
|
|
|
|
``` |
|
|
### Task |
|
|
You are a performance prediction expert. Given a target question, 5 anchor questions with their performance results, and a target AI model, predict how the model will perform on the target question, specifically the output length and correctness after related reasoning analysis. |
|
|
|
|
|
### Target Model |
|
|
{model_name} |
|
|
|
|
|
Example 1: |
|
|
Question: {anchor_question_1} |
|
|
Performance: {len: {length}, correct: {yes/no}} |
|
|
|
|
|
Example 2: |
|
|
Question: {anchor_question_2} |
|
|
Performance: {len: {length}, correct: {yes/no}} |
|
|
|
|
|
... |
|
|
|
|
|
### Target Question |
|
|
{your_target_question} |
|
|
|
|
|
### Output Format (STRICT) |
|
|
Analysis: [Your comprehensive analysis covering anchor patterns, target question characteristics, and reasoning.] |
|
|
Predicted Performance: {len: [integer], correct: [yes/no]} |
|
|
|
|
|
### Output: |
|
|
``` |
|
|
|
|
|
### Output Format |
|
|
|
|
|
The model outputs: |
|
|
``` |
|
|
Analysis: [Reasoning about the question difficulty based on anchor patterns...] |
|
|
Predicted Performance: {len: 256, correct: yes} |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Inference Methods |
|
|
|
|
|
### Method 1: Using Transformers (Recommended for Single Inference) |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
# Load model |
|
|
model_name = "Cooolder/SCOPE" |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
model = AutoModelForCausalLM.from_pretrained( |
|
|
model_name, |
|
|
torch_dtype="auto", |
|
|
device_map="auto" |
|
|
) |
|
|
|
|
|
# Prepare the prompt (see "Prompt Examples" section below) |
|
|
prompt = """### Task |
|
|
You are a performance prediction expert. Given a target question, 5 anchor questions with their performance results, and a target AI model, predict how the model will perform on the target question, specifically the output length and correctness after related reasoning analysis. |
|
|
|
|
|
### Target Model |
|
|
Qwen/Qwen3-8B-Instruct |
|
|
|
|
|
Example 1: |
|
|
Question: What is the capital of France? |
|
|
Performance: {len: 45, correct: yes} |
|
|
|
|
|
Example 2: |
|
|
Question: Solve: 2 + 2 = ? |
|
|
Performance: {len: 32, correct: yes} |
|
|
|
|
|
Example 3: |
|
|
Question: Explain quantum entanglement in simple terms. |
|
|
Performance: {len: 512, correct: yes} |
|
|
|
|
|
Example 4: |
|
|
Question: What is the 50th prime number? |
|
|
Performance: {len: 128, correct: no} |
|
|
|
|
|
Example 5: |
|
|
Question: Write a haiku about programming. |
|
|
Performance: {len: 78, correct: yes} |
|
|
|
|
|
### Target Question |
|
|
What is the derivative of x^3 + 2x^2 - 5x + 7? |
|
|
|
|
|
### Output Format (STRICT) |
|
|
Analysis: [Your comprehensive analysis covering anchor patterns, target question characteristics, and reasoning.] |
|
|
Predicted Performance: {len: [integer], correct: [yes/no]} |
|
|
|
|
|
### Output:""" |
|
|
|
|
|
# Format as chat message |
|
|
messages = [{"role": "user", "content": prompt}] |
|
|
text = tokenizer.apply_chat_template( |
|
|
messages, |
|
|
tokenize=False, |
|
|
add_generation_prompt=True, |
|
|
) |
|
|
|
|
|
# Generate |
|
|
model_inputs = tokenizer([text], return_tensors="pt").to(model.device) |
|
|
generated_ids = model.generate( |
|
|
**model_inputs, |
|
|
max_new_tokens=1536, |
|
|
temperature=0.7, |
|
|
top_p=0.8, |
|
|
top_k=20, |
|
|
) |
|
|
|
|
|
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() |
|
|
response = tokenizer.decode(output_ids, skip_special_tokens=True) |
|
|
print(response) |
|
|
``` |
|
|
|
|
|
### Method 2: Using vLLM (Recommended for Batch Inference) |
|
|
|
|
|
```python |
|
|
import os |
|
|
import re |
|
|
from vllm import LLM, SamplingParams |
|
|
|
|
|
# Load model with vLLM |
|
|
model_name = "Cooolder/SCOPE" |
|
|
llm = LLM( |
|
|
model=model_name, |
|
|
dtype="bfloat16", |
|
|
gpu_memory_utilization=0.90, |
|
|
max_model_len=8192, |
|
|
trust_remote_code=True, |
|
|
) |
|
|
|
|
|
# Prepare prompts (batch processing) |
|
|
prompts = [] |
|
|
raw_prompt = """### Task |
|
|
You are a performance prediction expert. Given a target question, 5 anchor questions with their performance results, and a target AI model, predict how the model will perform on the target question, specifically the output length and correctness after related reasoning analysis. |
|
|
|
|
|
### Target Model |
|
|
Qwen/Qwen3-8B-Instruct |
|
|
|
|
|
Example 1: |
|
|
Question: What is the capital of France? |
|
|
Performance: {len: 45, correct: yes} |
|
|
|
|
|
Example 2: |
|
|
Question: Solve: 2 + 2 = ? |
|
|
Performance: {len: 32, correct: yes} |
|
|
|
|
|
Example 3: |
|
|
Question: Explain quantum entanglement in simple terms. |
|
|
Performance: {len: 512, correct: yes} |
|
|
|
|
|
Example 4: |
|
|
Question: What is the 50th prime number? |
|
|
Performance: {len: 128, correct: no} |
|
|
|
|
|
Example 5: |
|
|
Question: Write a haiku about programming. |
|
|
Performance: {len: 78, correct: yes} |
|
|
|
|
|
### Target Question |
|
|
What is the derivative of x^3 + 2x^2 - 5x + 7? |
|
|
|
|
|
### Output Format (STRICT) |
|
|
Analysis: [Your comprehensive analysis covering anchor patterns, target question characteristics, and reasoning.] |
|
|
Predicted Performance: {len: [integer], correct: [yes/no]} |
|
|
|
|
|
### Output:""" |
|
|
|
|
|
# Wrap in Qwen3 chat template |
|
|
chat_prompt = f"<|im_start|>user\n{raw_prompt}<|im_end|>\n<|im_start|>assistant\n" |
|
|
prompts.append(chat_prompt) |
|
|
|
|
|
# Sampling parameters |
|
|
sampling_params = SamplingParams( |
|
|
temperature=0.6, |
|
|
max_tokens=1536, |
|
|
top_p=0.95, |
|
|
top_k=20, |
|
|
n=8, # Generate multiple samples for better confidence estimation |
|
|
stop=["<|im_end|>", "<|endoftext|>"], |
|
|
stop_token_ids=[151645, 151643] |
|
|
) |
|
|
|
|
|
# Run inference |
|
|
outputs = llm.generate(prompts, sampling_params) |
|
|
|
|
|
# Parse results |
|
|
for output in outputs: |
|
|
for single_output in output.outputs: |
|
|
response = single_output.text.strip() |
|
|
print(response) |
|
|
print("-" * 50) |
|
|
``` |
|
|
|
|
|
### Parsing the Output |
|
|
|
|
|
```python |
|
|
import re |
|
|
|
|
|
def parse_prediction(response: str): |
|
|
"""Parse SCOPE model output to extract predictions.""" |
|
|
# Clean up formatting variations |
|
|
response = response.replace('**Analysis**', 'Analysis:') |
|
|
response = response.replace('**Predicted Performance:**', 'Predicted Performance:') |
|
|
|
|
|
# Extract analysis |
|
|
analysis = "" |
|
|
if 'Analysis:' in response: |
|
|
analysis_start = response.find('Analysis:') + len('Analysis:') |
|
|
perf_start = response.find('Predicted Performance:') |
|
|
if perf_start > analysis_start: |
|
|
analysis = response[analysis_start:perf_start].strip() |
|
|
|
|
|
# Parse len and correct |
|
|
len_match = re.search(r'len:\s*(\d+)', response) |
|
|
correct_match = re.search(r'correct:\s*(yes|no)', response, re.IGNORECASE) |
|
|
|
|
|
if not len_match or not correct_match: |
|
|
return None |
|
|
|
|
|
return { |
|
|
'analysis': analysis, |
|
|
'predicted_length': int(len_match.group(1)), |
|
|
'predicted_correct': correct_match.group(1).lower() |
|
|
} |
|
|
|
|
|
# Example usage |
|
|
result = parse_prediction(response) |
|
|
print(f"Predicted Length: {result['predicted_length']}") |
|
|
print(f"Predicted Correct: {result['predicted_correct']}") |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Anchor and Prompt Examples |
|
|
|
|
|
### Example 1: Math Question Prediction |
|
|
|
|
|
```python |
|
|
anchor_text = """Example 1: |
|
|
Question: What is 15 + 27? |
|
|
Performance: {len: 28, correct: yes} |
|
|
|
|
|
Example 2: |
|
|
Question: Calculate the area of a circle with radius 5. |
|
|
Performance: {len: 156, correct: yes} |
|
|
|
|
|
Example 3: |
|
|
Question: Solve the quadratic equation x^2 - 5x + 6 = 0. |
|
|
Performance: {len: 245, correct: yes} |
|
|
|
|
|
Example 4: |
|
|
Question: What is the integral of sin(x)? |
|
|
Performance: {len: 89, correct: yes} |
|
|
|
|
|
Example 5: |
|
|
Question: Prove that the square root of 2 is irrational. |
|
|
Performance: {len: 478, correct: no} |
|
|
""" |
|
|
|
|
|
target_question = "Find the limit of (x^2 - 1)/(x - 1) as x approaches 1." |
|
|
model_name = "Qwen/Qwen3-8B-Instruct" |
|
|
``` |
|
|
|
|
|
### Example 2: Coding Question Prediction |
|
|
|
|
|
```python |
|
|
anchor_text = """Example 1: |
|
|
Question: Write a Python function to check if a number is even. |
|
|
Performance: {len: 67, correct: yes} |
|
|
|
|
|
Example 2: |
|
|
Question: Implement binary search in Python. |
|
|
Performance: {len: 234, correct: yes} |
|
|
|
|
|
Example 3: |
|
|
Question: Write a function to reverse a linked list. |
|
|
Performance: {len: 312, correct: yes} |
|
|
|
|
|
Example 4: |
|
|
Question: Implement a LRU cache in Python. |
|
|
Performance: {len: 456, correct: no} |
|
|
|
|
|
Example 5: |
|
|
Question: Write a recursive function to compute Fibonacci numbers. |
|
|
Performance: {len: 178, correct: yes} |
|
|
""" |
|
|
|
|
|
target_question = "Write a Python function to find the longest palindromic substring." |
|
|
model_name = "deepseek-ai/DeepSeek-V2-Chat" |
|
|
``` |
|
|
|
|
|
### Example 3: General Knowledge Prediction |
|
|
|
|
|
```python |
|
|
anchor_text = """Example 1: |
|
|
Question: Who wrote "Romeo and Juliet"? |
|
|
Performance: {len: 34, correct: yes} |
|
|
|
|
|
Example 2: |
|
|
Question: What is the chemical formula for water? |
|
|
Performance: {len: 42, correct: yes} |
|
|
|
|
|
Example 3: |
|
|
Question: Explain the theory of relativity. |
|
|
Performance: {len: 687, correct: yes} |
|
|
|
|
|
Example 4: |
|
|
Question: What year did World War II end? |
|
|
Performance: {len: 51, correct: yes} |
|
|
|
|
|
Example 5: |
|
|
Question: Who was the 23rd President of the United States? |
|
|
Performance: {len: 89, correct: no} |
|
|
""" |
|
|
|
|
|
target_question = "What is the speed of light in a vacuum?" |
|
|
model_name = "meta-llama/Llama-3-70B-Instruct" |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Using with Cooolder/kshot_inference Dataset |
|
|
|
|
|
The model is designed to work with the [Cooolder/kshot_inference](https://huggingface.co/datasets/Cooolder/kshot_inference) dataset: |
|
|
|
|
|
```python |
|
|
from datasets import load_dataset |
|
|
|
|
|
# Load the dataset |
|
|
dataset = load_dataset("Cooolder/kshot_inference", split="train") |
|
|
|
|
|
# Each sample contains: |
|
|
# - id: unique identifier |
|
|
# - prompt: pre-formatted prompt with anchors and target question |
|
|
# - gt_is_correct: ground truth correctness |
|
|
# - gt_token_count: ground truth token count |
|
|
# - source_model: the target model being predicted |
|
|
# - retrieved_anchors: the anchor questions used |
|
|
|
|
|
# Example: Run inference on the dataset |
|
|
for sample in dataset: |
|
|
prompt = sample['prompt'] |
|
|
# Wrap in chat template and run inference... |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Performance Tips |
|
|
|
|
|
1. **Multiple Sampling**: Generate 8+ samples and aggregate predictions for better accuracy |
|
|
2. **Temperature**: Use 0.6-0.7 for balanced diversity |
|
|
3. **Batch Processing**: Use vLLM for high-throughput batch inference |
|
|
4. **Anchor Selection**: Choose anchors similar to your target question domain |
|
|
|
|
|
## Citation |
|
|
|
|
|
```bibtex |
|
|
@misc{cao2026modelsscopescalablecontrollable, |
|
|
title={Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning}, |
|
|
author={Qi Cao and Shuhao Zhang and Ruizhe Zhou and Ruiyi Zhang and Peijia Qin and Pengtao Xie}, |
|
|
year={2026}, |
|
|
eprint={2601.22323}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.LG}, |
|
|
url={https://arxiv.org/abs/2601.22323}, |
|
|
} |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
Apache 2.0 |
|
|
|
