---
library_name: transformers
pipeline_tag: text-classification
tags:
  - regression
  - prompt
  - complexity-estimation
  - semantic-routing
  - llm-routing
base_model: microsoft/deberta-v3-base
license: apache-2.0
---

# PromptComplexityEstimator

A lightweight regressor that estimates the complexity of an LLM prompt on a scale between 0 and 1.

- **Input:** a string prompt  
- **Output:** a scalar score in [0, 1] (higher = more complex)

The model is designed primarily to be used as a core building block for semantic routing systems, especially LLM vs. SLM (Small Language Model) routers.  
Any router that aims to intelligently decide *which model should handle a request* needs a reliable signal for *how complex the request is*. This is the gap this model aims to close.

---

## Intended use

### Primary use case: LLM vs. SLM routing

This model is intended to be used as part of a semantic router, where:
- *Simple* prompts are handled by a **small / fast / cheap model**
- *Complex* prompts are routed to a **large / capable / expensive model**

The complexity score provides a learned signal for this decision.

### Additional use cases
- Prompt analytics and monitoring
- Dataset stratification by difficulty
- Adaptive compute allocation
- Cost-aware or latency-aware inference pipelines

### Not intended for
- Safety classification, toxicity detection, or policy enforcement
- Guaranteed difficulty estimation for a specific target model
- Multimodal inputs or tool-augmented workflows (RAG/tools)

---

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModel

repo_id = "ilya-kolchinsky/PromptComplexityEstimator"

tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()

prompt = "Design a distributed consensus protocol with Byzantine fault tolerance..."
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, padding=True)

with torch.no_grad():
    score = model(**inputs).logits.squeeze(-1).item()

print(float(score))
```


### Example: Simple LLM vs. SLM routing

```python
THRESHOLD = 0.45  # chosen empirically

def route_prompt(prompt: str) -> str:
    inputs = tokenizer(prompt, return_tensors="pt", truncation=True, padding=True)
    with torch.no_grad():
        complexity = model(**inputs).logits.squeeze(-1).item()

    return "LLM" if complexity > THRESHOLD else "SLM"
```

---


## Model and Training Details

### Datasets
- Cross-Difficulty (https://huggingface.co/datasets/BatsResearch/Cross-Difficulty)
- [Easy2Hard-Bench](https://huggingface.co/datasets/furonghuang-lab/Easy2Hard-Bench)
- [MATH](https://huggingface.co/datasets/EleutherAI/hendrycks_math)
- [ARC](https://huggingface.co/datasets/allenai/ai2_arc)
- [RACE](https://huggingface.co/datasets/ehovy/race)
- [ANLI (R1/R2/R3)](https://huggingface.co/datasets/facebook/anli)

### Training Configuration
- **Epochs:** 3
- **Batch Size:** 16
- **Loss:** huber
- **Regressor Learning Rate:** 7.5e-5
- **Encoder Learning Rate:** 1.0e-5
- **Encoder Weight Decay:** 0.01
- **Optimizer**: AdamW
- **Schedule**: Cosine (warmup_ratio=0.06)
- **Dropout**: 0.1

### Model
- **Backbone encoder:** microsoft/deberta-v3-base
- Mask-aware **mean pooling** over token embeddings + **LayerNorm**
- **Regression head:** Linear → ReLU → Linear → Sigmoid
- **Max input length:** 512 tokens
- The model outputs a bounded score in [0, 1]. In the examples below, the score is read from `outputs.logits` (shape `[batch, 1]`).


Full training code and configuration are available at https://github.com/ilya-kolchinsky/ComplexityEstimator.

---

## Performance

On the held-out evaluation set used during development, the released checkpoint achieved:

- **MAE:** **0.0855**
- **Spearman correlation:** **0.735**

---

## Citation

```bibtex
@misc{kolchinsky_promptcomplexityestimator_2026,
  title        = {PromptComplexityEstimator},
  author       = {Ilya Kolchinsky},
  year         = {2026},
  howpublished = {Hugging Face Hub model: ilya-kolchinsky/PromptComplexityEstimator}
}
```