Text Classification
Transformers
Safetensors
English
modernbert
router
llm-routing
difficulty-estimation
text-embeddings-inference
Instructions to use bthe0/switchyard-decider with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use bthe0/switchyard-decider with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="bthe0/switchyard-decider")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("bthe0/switchyard-decider") model = AutoModelForSequenceClassification.from_pretrained("bthe0/switchyard-decider", device_map="auto") - Notebooks
- Google Colab
- Kaggle
File size: 10,479 Bytes
1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 030e4a3 1b4d100 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 | ---
license: apache-2.0
base_model: answerdotai/ModernBERT-large
pipeline_tag: text-classification
library_name: transformers
language:
- en
tags:
- router
- llm-routing
- modernbert
- difficulty-estimation
---
# switchyard-decider
## Summary
switchyard-decider is `answerdotai/ModernBERT-large` fine-tuned to score how hard a coding-assistant request is. A
local LLM router uses the score to send a request either to a cheaper mid-tier model or to a frontier model. v1.2
reaches AUC 0.9794 on a frozen 200-row holdout built to catch length shortcuts (v1.1: 0.8396) and 0.8966 on the
validation set val2 (v1.1: 0.8952), at 62.8 ms p50 end to end on Apple M4 Max MPS.
v1.2 needs the head+tail input cut from the v1.2 code (`switchyard_decider/serve.py`, package version 1.2.0). Do not
serve these weights with the v1.1 server, which keeps only the first 768 tokens.
- Weights: https://huggingface.co/bthe0/switchyard-decider (primary) and the GitHub release
https://github.com/bthe0/switchyard-decider/releases/tag/v1.2
- Code: https://github.com/bthe0/switchyard-decider
Files: `model.safetensors` (about 792 MB, fp16), `config.json`, `tokenizer.json`, `tokenizer_config.json`,
`calib.npz`.
```sh
hf download bthe0/switchyard-decider --local-dir switchyard-decider
# or
gh release download v1.2 -R bthe0/switchyard-decider -D switchyard-decider
```
## Intended use
Routing single requests to coding assistants by estimated difficulty: lower scores go to a frontier model, higher
scores to a mid-tier model. It is not a general-purpose classifier and has not been evaluated outside English
coding-assistant and chat prompts.
## How to use
With transformers and numpy:
```python
import numpy as np
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
d = "switchyard-decider"
tok = AutoTokenizer.from_pretrained(d)
model = AutoModelForSequenceClassification.from_pretrained(d, dtype=torch.float32).eval()
calib = np.load(f"{d}/calib.npz")
text = "Rename the variable foo to bar in utils.py"
ids = tok(text[:20000])["input_ids"]
if len(ids) > 768: # head+tail: keep the first 384 and last 384 tokens
ids = ids[:384] + ids[-384:]
with torch.no_grad():
out = model(input_ids=torch.tensor([ids]), output_hidden_states=True)
lg, hid = out.logits[0].float(), out.hidden_states[-1][0].float()
pr = (lg[:3].softmax(-1) + lg[3:6].softmax(-1)) / 2
h = (np.searchsorted(calib["aj"], pr[2].item()) + np.searchsorted(calib["g"], -lg[6].item())) / 2 / len(calib["aj"])
emb = torch.cat([hid[0], hid.mean(0)]).numpy() # CLS + mean of the final hidden states
r = np.searchsorted(calib["rs"], emb @ calib["ra"] + calib["rb"]) / len(calib["rs"])
hardness = (h + calib["rw"] * r) / (1 + calib["rw"])
q = np.searchsorted(calib["h"], hardness) / len(calib["h"])
p_solve = float(np.quantile(calib["out"], 1 - q)) # probability the mid-tier model can handle it
```
Or run the HTTP server from the code repository:
```sh
pip install -e .
switchyard-decider-serve switchyard-decider --port 4126
curl -s -X POST http://127.0.0.1:4126/v1/systemone \
-d '{"state": {"opening_task": "Rename the variable foo to bar in utils.py", "latest_follow_up": ""}}'
# {"answers": {"solve": {"noul": 0.84}}}
```
## Outputs
Seven logits:
| logits | meaning |
|---|---|
| 0–2 | arena-derived easy / medium / hard |
| 3–5 | LLM-judge easy / medium / hard |
| 6 | gold score from decision-model teachers; higher means easier |
The hardness score is the mean of two validation-set percentiles: P(hard) averaged over the arena and judge heads, and
the negated gold logit. That is rank-blended with a ridge probe on the CLS+mean embedding (`ra`, `rb`, `rs`, weight
`rw`). `calib.npz` holds these reference arrays (`aj`, `g`, `h`, `out`, `ra`, `rb`, `rs`, `rw`) and maps hardness
onto a target output distribution so that a router's existing cutoffs keep routing the same share of traffic.
## Training data
About 153k prompts from WildChat-1M (ODC-BY-1.0), LMArena arena-human-preference-140k (CC-BY-4.0), OpenAssistant
oasst2 (Apache-2.0), Magicoder-Evol-Instruct-110K (Apache-2.0), CodeFeedback-Filtered-Instruction (Apache-2.0),
SWE-bench (MIT), SWE-Gym (MIT) and SWE-rebench (CC-BY-4.0). Labels came from arena battle outcomes, LLM judges
(Anthropic Claude Haiku, OpenAI GPT-6-luna decisions) and OpenRouter decision models (`nace-ai/drex-max`,
`perplexity/pplx-decider`).
Training (v1.0): lr 3e-5, max length 768, bf16, single NVIDIA RTX 5070 Ti, best checkpoint at step 3,600, followed by
a greedy weight soup. v1.1 added the ridge probe.
v1.2 retune, from the v1.1 weights: 1 epoch at lr 1e-5 on 20k replay rows of the original training set plus 4,653
label-preserving augmentation rows. The augmentation covers easy asks wrapped in long logs, JSON or code pastes; terse
hard questions; and matched pairs that put an easy ask and a hard ask after the same paste. Inputs over 768 tokens
keep the first 384 and last 384 tokens. The ridge probe was refit with no augmentation weight. Training took about
45 minutes on one RTX 5070 Ti (about $0.25 rented).
The selection rule was fixed before any holdout metric was computed: take the highest dev-aug AUC among runs with
val2 AUC ≥ 0.8902. Candidates scored (val2 / dev-aug): r5 0.8936 / 0.9463, r10 0.8968 / 0.9483, r20 0.8987 / 0.9505,
r10ht 0.8966 / 0.9576 (chosen). Probe refits that weighted the augmentation rows failed the val2 gate.
## Evaluation
### v1.2 against v1.1
The holdout is 200 frozen rows: 35 each of short_easy, short_hard, long_easy and long_hard, 20 matched pairs (same
paste, one easy ask and one hard ask), and 20 borderline rows. Its sha256 is
`967dc16e4d61e90ed14a1f4c048a14be285264f8193a1b14b211a87ed6c61357`. It was never used for training or tuning. It has
0 exact matches and 0 near-duplicates (8-gram overlap ≥ 50%) against the training, augmentation, dev-aug and fresh100
sets. Length alone scores AUC 0.5065 on it, so length does not separate its classes.
| metric | v1.1 | v1.2 |
|---|---|---|
| holdout hard-vs-easy AUC | 0.8396 | **0.9794** |
| matched pairs ordered correctly | 13/20 | 19/20 |
| long_easy on the wrong side of the cutoff | 30/35 | 13/35 |
| pair_easy on the wrong side | 16/20 | 8/20 |
| short_hard on the wrong side | 2/35 | 0/35 |
| short_easy, long_hard on the wrong side (controls) | 0 | 0 |
| val2 AUC | 0.8952 | 0.8966 |
| fresh100 AUC | 0.9908 | 0.9956 |
| dev-aug AUC | 0.7312 | 0.9576 |
Holdout AUC difference +0.1398. Paired bootstrap with 2,000 resamples gives a 95% lower bound of +0.0906.
### Validation history (v1.0, v1.1)
Validation set: 1,994 held-out prompts (easy vs hard pairs; medium excluded from AUC).
| model | AUC (labels) | AUC (arena) | AUC (coding) | hard → frontier @40% | easy → mid-tier @40% |
|---|---|---|---|---|---|
| **+ ridge probe on embedding (v1.1, served)** | **0.895** | | | | |
| this model (step 3600, head only, v1.0) | 0.886 | 0.769 | 0.929 | 0.693 | 0.862 |
| same recipe without gold head | 0.859 | | | | |
| previous LLM decider (0.8B, MLX 4-bit) | 0.687 | | | 0.538 | 0.712 |
On a fixed 300-prompt subset used to benchmark about 40 OpenRouter decision models, this model scores AUC 0.904. The
best teachers scored 0.879 (`nace-ai/drex-max-finance`), 0.873 (`nace-ai/drex-max`) and 0.871
(`perplexity/pplx-decider-v1.3`).
The checkpoint was chosen on this validation set, so these numbers are slightly optimistic.
## Speed
v1.2 measurement: served path end to end (20,000-character cut, tokenization, head+tail, forward pass, ridge probe),
batch 1, fp16, Apple M4 Max MPS, on the 200 holdout texts. Most of these texts are long. Same harness for both models:
| model | p50 | p90 |
|---|---|---|
| v1.1 | 64.0 ms | |
| v1.2 | 62.8 ms | 64.9 ms |
The earlier figures below come from a forward-pass harness on mostly short validation prompts (median 80 tokens).
They are not comparable with the table above, which includes tokenization of long inputs. Neither model meets a
40 ms goal on long inputs; batch-1 MPS inference is bound by kernel launch overhead.
Earlier measurement (v1.1): Apple M4 Max (48 GB), PyTorch, single request, batch 1. Forward pass only at fixed lengths; "real prompts" includes
tokenization on the 1,994 validation prompts (median 80 tokens). The ridge probe adds one 2048-wide dot product.
| tokens | MPS fp16 p50 | MPS fp16 p95 | CPU fp32 p50 | CPU fp32 p95 |
|---|---|---|---|---|
| 32 | 11.4 ms | 11.7 ms | 49.4 ms | 55.3 ms |
| 128 | 17.7 ms | 18.1 ms | 125.1 ms | 141.7 ms |
| 256 | 27.6 ms | 29.3 ms | 150.3 ms | 161.1 ms |
| 512 | 107.8 ms | 320.3 ms | 294.1 ms | 317.5 ms |
| 768 | 145.4 ms | 161.6 ms | 473.3 ms | 522.6 ms |
| real prompts | 30.2 ms (p99 151 ms) | 107.5 ms | 78.7 ms (p99 320 ms, 200 prompts) | 219.0 ms |
Through the HTTP server on MPS (500 prompts): p50 23 ms, p95 74 ms, p99 97 ms. Model load 0.7 s, resident memory
about 560 MB. The previous LLM decider took about 720 ms per decision.
## Limitations
- English only; tuned on coding-assistant and chat prompts.
- The model scores the first 20,000 characters of the opening task plus the latest follow-up. Inputs over 768 tokens
keep only the first 384 and last 384 tokens, so text in the middle of a long paste is not seen.
- Long pastes still bias v1.2 toward "hard": 13/35 long_easy and 8/20 pair_easy holdout rows land on the wrong side of
the cutoff.
- The holdout is synthetic. Its asks were written by an LLM and assembled by deterministic generator scripts. A TF-IDF
classifier separates its classes at CV AUC 0.597, a borderline amount of lexical leakage.
- dev-aug is built the same way as the training augmentation, so the dev-aug gain is optimistic.
- The v1.2 weights need the head+tail serve path (v1.2 code). The v1.1 `serve.py` truncates to the first 768 tokens.
- Labels are noisy teacher outputs; the model ranks difficulty, it does not verify that a model can solve a task.
- Evaluation numbers are slightly optimistic because the checkpoint was selected on the validation set.
## License and usage risk
Code and weights are released under Apache-2.0, matching the ModernBERT base model. The training labels are outputs of
commercial models. Some providers' terms restrict using outputs to build competing models. This model is a
binary-ish routing classifier, not a general model, but check the relevant terms before any commercial use. The
WildChat (ODC-BY) and CC-BY sources require attribution, which the list above provides.
|