Upload 2 files

.gitattributes

@@ -37,3 +37,4 @@ tokenizer.json filter=lfs diff=lfs merge=lfs -text
 checkpoint-6000/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 checkpoint-6500/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 checkpoint-6695/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/1.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,75 +1,413 @@
 ---
-library_name: transformers
-license: other
-base_model: Qwen/Qwen3-4B-Instruct-2507
+license: apache-2.0
+language:
+- multilingual
+base_model:
+- Qwen/Qwen3-4B-Instruct-2507
+pipeline_tag: text-generation
 tags:
-- llama-factory
-- full
-- generated_from_trainer
-model-index:
-- name: sft_direct
-  results: []
+- Model Routing
+- LLM reasoning
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# SCOPE-Direct: Scalable and Controllable Outcome Performance Estimator (Direct Prediction)
+[📄 Paper (arXiv:2601.22323)](https://www.arxiv.org/abs/2601.22323)
 
-# sft_direct
+This repository accompanies the paper "**Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning**", which introduces SCOPE (Scalable and Controllable Outcome Performance Estimator) — a new framework for large language model (LLM) routing.
+SCOPE reframes model routing as a pre-hoc estimation problem: instead of directly selecting a model from a fixed candidate set, it predicts each model's expected performance (correctness) and inference cost (token length) before execution, based on the model's historical behaviors on similar queries. This enables training-free generalization to unseen models and allows users to flexibly control the trade-off between accuracy and cost through a budget-aware utility function.
+Overall, SCOPE provides a scalable, explainable, and controllable solution for allocating test-time compute across heterogeneous model portfolios.
 
-This model is a fine-tuned version of [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) on the scope_sft_direct dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.3041
+<p align="center">
+  <img src="assets/1.png" width="500">
+</p>
+The figure above illustrates the core difference between traditional routers and SCOPE.
+Conventional LLM routers treat routing as a closed-set classification problem, simply memorizing model names and selecting one model per query. In contrast, SCOPE reasons over models' past behaviors, explicitly predicting outcome correctness and token cost, and then makes a budget-aware decision based on these estimates. This design allows SCOPE to generalize to unseen models and supports dynamic cost–accuracy control at inference time.
 
-## Model description
+**SCOPE-Direct** is the direct prediction variant that outputs predictions without explicit chain-of-thought analysis, making it faster and more efficient for production use cases.
 
-More information needed
+## Model Description
 
-## Intended uses & limitations
+- **Task**: Performance prediction for LLMs
+- **Base Model**: Qwen/Qwen3-4B-Instruct-2507
+- **Training**: Supervised Fine-Tuning (SFT)
+- **Input**: Target question + k anchor questions with performance data
+- **Output**: Predicted length (tokens) and correctness (yes/no)
 
-More information needed
+## Intended Use
 
-## Training and evaluation data
+SCOPE-Direct is designed to:
+- Predict whether an LLM will answer a question correctly before running expensive inference
+- Estimate the output token length for resource planning
+- Enable efficient LLM routing and selection
 
-More information needed
+## Quick Start
 
-## Training procedure
+### Installation
 
-### Training hyperparameters
+```bash
+pip install transformers>=4.51.0 torch datasets
+# For vLLM inference (optional but recommended)
+pip install vllm
+```
 
-The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 1
-- eval_batch_size: 2
-- seed: 42
-- gradient_accumulation_steps: 8
-- total_train_batch_size: 8
-- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_ratio: 0.1
-- num_epochs: 1.0
+### Input Format
 
-### Training results
+SCOPE-Direct uses the following prompt format (without Analysis):
 
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 0.3289        | 0.0747 | 500  | 0.3394          |
-| 0.3445        | 0.1494 | 1000 | 0.3315          |
-| 0.3383        | 0.2241 | 1500 | 0.3268          |
-| 0.3352        | 0.2987 | 2000 | 0.3224          |
-| 0.3027        | 0.3734 | 2500 | 0.3197          |
-| 0.3272        | 0.4481 | 3000 | 0.3166          |
-| 0.3216        | 0.5228 | 3500 | 0.3138          |
-| 0.3223        | 0.5975 | 4000 | 0.3108          |
-| 0.3346        | 0.6722 | 4500 | 0.3076          |
-| 0.3098        | 0.7468 | 5000 | 0.3065          |
-| 0.2938        | 0.8215 | 5500 | 0.3059          |
-| 0.315         | 0.8962 | 6000 | 0.3046          |
-| 0.301         | 0.9709 | 6500 | 0.3040          |
+```
+### Task
+You are a performance prediction expert. Given a target question, 5 anchor questions with their performance results, and a target AI model, predict how the model will perform on the target question, specifically the output length and correctness.
 
+### Target Model
+{model_name}
 
-### Framework versions
+Example 1:
+Question: {anchor_question_1}
+Performance: {len: {length}, correct: {yes/no}}
 
-- Transformers 4.57.1
-- Pytorch 2.7.1+cu126
-- Datasets 3.6.0
-- Tokenizers 0.22.1
+Example 2:
+Question: {anchor_question_2}
+Performance: {len: {length}, correct: {yes/no}}
+
+...
+
+### Target Question
+{your_target_question}
+
+### Output Format (STRICT)
+Predicted Performance: {len: [integer], correct: [yes/no]}
+
+### Output:
+```
+
+### Output Format
+
+The model directly outputs:
+```
+Predicted Performance: {len: 256, correct: yes}
+```
+
+---
+
+## Inference Methods
+
+### Method 1: Using Transformers (Recommended for Single Inference)
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+# Load model
+model_name = "Cooolder/SCOPE-Direct"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map="auto"
+)
+
+# Prepare the prompt (see "Prompt Examples" section below)
+prompt = """### Task
+You are a performance prediction expert. Given a target question, 5 anchor questions with their performance results, and a target AI model, predict how the model will perform on the target question, specifically the output length and correctness.
+
+### Target Model
+Qwen/Qwen3-8B-Instruct
+
+Example 1:
+Question: What is the capital of France?
+Performance: {len: 45, correct: yes}
+
+Example 2:
+Question: Solve: 2 + 2 = ?
+Performance: {len: 32, correct: yes}
+
+Example 3:
+Question: Explain quantum entanglement in simple terms.
+Performance: {len: 512, correct: yes}
+
+Example 4:
+Question: What is the 50th prime number?
+Performance: {len: 128, correct: no}
+
+Example 5:
+Question: Write a haiku about programming.
+Performance: {len: 78, correct: yes}
+
+### Target Question
+What is the derivative of x^3 + 2x^2 - 5x + 7?
+
+### Output Format (STRICT)
+Predicted Performance: {len: [integer], correct: [yes/no]}
+
+### Output:"""
+
+# Format as chat message
+messages = [{"role": "user", "content": prompt}]
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+
+# Generate
+model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+generated_ids = model.generate(
+    **model_inputs,
+    max_new_tokens=64,  # Direct prediction needs fewer tokens
+    temperature=0.7,
+    top_p=0.8,
+    top_k=20,
+)
+
+output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
+response = tokenizer.decode(output_ids, skip_special_tokens=True)
+print(response)
+```
+
+### Method 2: Using vLLM (Recommended for Batch Inference)
+
+```python
+import os
+import re
+from vllm import LLM, SamplingParams
+
+# Load model with vLLM
+model_name = "Cooolder/SCOPE-Direct"
+llm = LLM(
+    model=model_name,
+    dtype="bfloat16",
+    gpu_memory_utilization=0.90,
+    max_model_len=8192,
+    trust_remote_code=True,
+)
+
+# Prepare prompts (batch processing)
+prompts = []
+raw_prompt = """### Task
+You are a performance prediction expert. Given a target question, 5 anchor questions with their performance results, and a target AI model, predict how the model will perform on the target question, specifically the output length and correctness.
+
+### Target Model
+Qwen/Qwen3-8B-Instruct
+
+Example 1:
+Question: What is the capital of France?
+Performance: {len: 45, correct: yes}
+
+Example 2:
+Question: Solve: 2 + 2 = ?
+Performance: {len: 32, correct: yes}
+
+Example 3:
+Question: Explain quantum entanglement in simple terms.
+Performance: {len: 512, correct: yes}
+
+Example 4:
+Question: What is the 50th prime number?
+Performance: {len: 128, correct: no}
+
+Example 5:
+Question: Write a haiku about programming.
+Performance: {len: 78, correct: yes}
+
+### Target Question
+What is the derivative of x^3 + 2x^2 - 5x + 7?
+
+### Output Format (STRICT)
+Predicted Performance: {len: [integer], correct: [yes/no]}
+
+### Output:"""
+
+# Wrap in Qwen3 chat template
+chat_prompt = f"<|im_start|>user\n{raw_prompt}<|im_end|>\n<|im_start|>assistant\n"
+prompts.append(chat_prompt)
+
+# Sampling parameters
+sampling_params = SamplingParams(
+    temperature=0.6,
+    max_tokens=64,  # Direct prediction needs fewer tokens
+    top_p=0.95,
+    top_k=20,
+    n=8,  # Generate multiple samples for better confidence estimation
+    stop=["<|im_end|>", "<|endoftext|>"],
+    stop_token_ids=[151645, 151643]
+)
+
+# Run inference
+outputs = llm.generate(prompts, sampling_params)
+
+# Parse results
+for output in outputs:
+    for single_output in output.outputs:
+        response = single_output.text.strip()
+        print(response)
+        print("-" * 50)
+```
+
+### Parsing the Output
+
+```python
+import re
+
+def parse_prediction(response: str):
+    """Parse SCOPE-Direct model output to extract predictions."""
+    # Clean up formatting variations
+    response = response.replace('**Predicted Performance:**', 'Predicted Performance:')
+    response = response.replace('**Predicted Performance**:', 'Predicted Performance:')
+
+    # Parse len and correct
+    len_match = re.search(r'len:\s*(\d+)', response)
+    correct_match = re.search(r'correct:\s*(yes|no)', response, re.IGNORECASE)
+
+    if not len_match or not correct_match:
+        return None
+
+    return {
+        'predicted_length': int(len_match.group(1)),
+        'predicted_correct': correct_match.group(1).lower()
+    }
+
+# Example usage
+result = parse_prediction(response)
+print(f"Predicted Length: {result['predicted_length']}")
+print(f"Predicted Correct: {result['predicted_correct']}")
+```
+
+---
+
+## Anchor and Prompt Examples
+
+### Example 1: Math Question Prediction
+
+```python
+anchor_text = """Example 1:
+Question: What is 15 + 27?
+Performance: {len: 28, correct: yes}
+
+Example 2:
+Question: Calculate the area of a circle with radius 5.
+Performance: {len: 156, correct: yes}
+
+Example 3:
+Question: Solve the quadratic equation x^2 - 5x + 6 = 0.
+Performance: {len: 245, correct: yes}
+
+Example 4:
+Question: What is the integral of sin(x)?
+Performance: {len: 89, correct: yes}
+
+Example 5:
+Question: Prove that the square root of 2 is irrational.
+Performance: {len: 478, correct: no}
+"""
+
+target_question = "Find the limit of (x^2 - 1)/(x - 1) as x approaches 1."
+model_name = "Qwen/Qwen3-8B-Instruct"
+```
+
+### Example 2: Coding Question Prediction
+
+```python
+anchor_text = """Example 1:
+Question: Write a Python function to check if a number is even.
+Performance: {len: 67, correct: yes}
+
+Example 2:
+Question: Implement binary search in Python.
+Performance: {len: 234, correct: yes}
+
+Example 3:
+Question: Write a function to reverse a linked list.
+Performance: {len: 312, correct: yes}
+
+Example 4:
+Question: Implement a LRU cache in Python.
+Performance: {len: 456, correct: no}
+
+Example 5:
+Question: Write a recursive function to compute Fibonacci numbers.
+Performance: {len: 178, correct: yes}
+"""
+
+target_question = "Write a Python function to find the longest palindromic substring."
+model_name = "deepseek-ai/DeepSeek-V2-Chat"
+```
+
+### Example 3: General Knowledge Prediction
+
+```python
+anchor_text = """Example 1:
+Question: Who wrote "Romeo and Juliet"?
+Performance: {len: 34, correct: yes}
+
+Example 2:
+Question: What is the chemical formula for water?
+Performance: {len: 42, correct: yes}
+
+Example 3:
+Question: Explain the theory of relativity.
+Performance: {len: 687, correct: yes}
+
+Example 4:
+Question: What year did World War II end?
+Performance: {len: 51, correct: yes}
+
+Example 5:
+Question: Who was the 23rd President of the United States?
+Performance: {len: 89, correct: no}
+"""
+
+target_question = "What is the speed of light in a vacuum?"
+model_name = "meta-llama/Llama-3-70B-Instruct"
+```
+
+---
+
+## Using with Cooolder/kshot_inference_direct Dataset
+
+The model is designed to work with the [Cooolder/kshot_inference_direct](https://huggingface.co/datasets/Cooolder/kshot_inference_direct) dataset:
+
+```python
+from datasets import load_dataset
+
+# Load the dataset
+dataset = load_dataset("Cooolder/kshot_inference_direct", split="train")
+
+# Each sample contains:
+# - id: unique identifier
+# - prompt: pre-formatted prompt with anchors and target question
+# - gt_is_correct: ground truth correctness
+# - gt_token_count: ground truth token count
+# - source_model: the target model being predicted
+# - retrieved_anchors: the anchor questions used
+
+# Example: Run inference on the dataset
+for sample in dataset:
+    prompt = sample['prompt']
+    # Wrap in chat template and run inference...
+```
+
+---
+
+## Performance Tips
+
+1. **Multiple Sampling**: Generate 8+ samples and aggregate predictions for better accuracy
+2. **Temperature**: Use 0.6-0.7 for balanced diversity
+3. **Batch Processing**: Use vLLM for high-throughput batch inference
+4. **Anchor Selection**: Choose anchors similar to your target question domain
+
+## Citation
+
+```bibtex
+@misc{cao2026modelsscopescalablecontrollable,
+      title={Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning},
+      author={Qi Cao and Shuhao Zhang and Ruizhe Zhou and Ruiyi Zhang and Peijia Qin and Pengtao Xie},
+      year={2026},
+      eprint={2601.22323},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2601.22323},
+}
+```
+
+## License
+
+Apache 2.0