bthe0 commited on
Commit
1b4d100
·
verified ·
1 Parent(s): 88545ea

v1.1: gold3 weights, calib with ridge probe, model card

Browse files
Files changed (6) hide show
  1. README.md +146 -0
  2. calib.npz +3 -0
  3. config.json +102 -0
  4. model.safetensors +3 -0
  5. tokenizer.json +0 -0
  6. tokenizer_config.json +17 -0
README.md ADDED
@@ -0,0 +1,146 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: answerdotai/ModernBERT-large
4
+ pipeline_tag: text-classification
5
+ library_name: transformers
6
+ language:
7
+ - en
8
+ tags:
9
+ - router
10
+ - llm-routing
11
+ - modernbert
12
+ - difficulty-estimation
13
+ ---
14
+
15
+ # switchyard-decider
16
+
17
+ ## Summary
18
+
19
+ switchyard-decider is `answerdotai/ModernBERT-large` fine-tuned to score how hard a coding-assistant request is. A
20
+ local LLM router uses the score to send a request either to a cheaper mid-tier model or to a frontier model. It
21
+ answers in about 25 ms (p50, Apple-silicon MPS) and reaches AUC 0.895 on held-out easy vs hard prompts.
22
+
23
+ - Weights: https://huggingface.co/bthe0/switchyard-decider (primary) and the GitHub release
24
+ https://github.com/bthe0/switchyard-decider/releases/tag/v1.1
25
+ - Code: https://github.com/bthe0/switchyard-decider
26
+
27
+ Files: `model.safetensors` (about 792 MB, fp16), `config.json`, `tokenizer.json`, `tokenizer_config.json`,
28
+ `calib.npz`.
29
+
30
+ ```sh
31
+ hf download bthe0/switchyard-decider --local-dir switchyard-decider
32
+ # or
33
+ gh release download v1.1 -R bthe0/switchyard-decider -D switchyard-decider
34
+ ```
35
+
36
+ ## Intended use
37
+
38
+ Routing single requests to coding assistants by estimated difficulty: lower scores go to a frontier model, higher
39
+ scores to a mid-tier model. It is not a general-purpose classifier and has not been evaluated outside English
40
+ coding-assistant and chat prompts.
41
+
42
+ ## How to use
43
+
44
+ With transformers and numpy:
45
+
46
+ ```python
47
+ import numpy as np
48
+ import torch
49
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
50
+
51
+ d = "switchyard-decider"
52
+ tok = AutoTokenizer.from_pretrained(d)
53
+ model = AutoModelForSequenceClassification.from_pretrained(d, dtype=torch.float32).eval()
54
+ calib = np.load(f"{d}/calib.npz")
55
+
56
+ text = "Rename the variable foo to bar in utils.py"
57
+ with torch.no_grad():
58
+ lg = model(**tok(text, truncation=True, max_length=768, return_tensors="pt")).logits[0].float()
59
+ pr = (lg[:3].softmax(-1) + lg[3:6].softmax(-1)) / 2
60
+ hardness = (np.searchsorted(calib["aj"], pr[2].item()) + np.searchsorted(calib["g"], -lg[6].item())) / 2 / len(calib["aj"])
61
+ q = np.searchsorted(calib["h"], hardness) / len(calib["h"])
62
+ p_solve = float(np.quantile(calib["out"], 1 - q)) # probability the mid-tier model can handle it
63
+ ```
64
+
65
+ Or run the HTTP server from the code repository:
66
+
67
+ ```sh
68
+ pip install -e .
69
+ switchyard-decider-serve switchyard-decider --port 4126
70
+ curl -s -X POST http://127.0.0.1:4126/v1/systemone \
71
+ -d '{"state": {"opening_task": "Rename the variable foo to bar in utils.py", "latest_follow_up": ""}}'
72
+ # {"answers": {"solve": {"noul": 0.84}}}
73
+ ```
74
+
75
+ ## Outputs
76
+
77
+ Seven logits:
78
+
79
+ | logits | meaning |
80
+ |---|---|
81
+ | 0–2 | arena-derived easy / medium / hard |
82
+ | 3–5 | LLM-judge easy / medium / hard |
83
+ | 6 | gold score from decision-model teachers; higher means easier |
84
+
85
+ The hardness score is the mean of two validation-set percentiles: P(hard) averaged over the arena and judge heads, and
86
+ the negated gold logit. `calib.npz` (keys `aj`, `g`, `h`, `out`) holds the reference arrays and maps hardness onto a
87
+ target output distribution so that a router's existing cutoffs keep routing the same share of traffic.
88
+
89
+ ## Training data
90
+
91
+ About 153k prompts from WildChat-1M (ODC-BY-1.0), LMArena arena-human-preference-140k (CC-BY-4.0), OpenAssistant
92
+ oasst2 (Apache-2.0), Magicoder-Evol-Instruct-110K (Apache-2.0), CodeFeedback-Filtered-Instruction (Apache-2.0),
93
+ SWE-bench (MIT), SWE-Gym (MIT) and SWE-rebench (CC-BY-4.0). Labels came from arena battle outcomes, LLM judges
94
+ (Anthropic Claude Haiku, OpenAI GPT-6-luna decisions) and OpenRouter decision models (`nace-ai/drex-max`,
95
+ `perplexity/pplx-decider`).
96
+
97
+ Training: lr 3e-5, max length 768, bf16, single NVIDIA RTX 5070 Ti, best checkpoint at step 3,600, followed by a
98
+ greedy weight soup.
99
+
100
+ ## Evaluation
101
+
102
+ Validation set: 1,994 held-out prompts (easy vs hard pairs; medium excluded from AUC).
103
+
104
+ | model | AUC (labels) | AUC (arena) | AUC (coding) | hard → frontier @40% | easy → mid-tier @40% |
105
+ |---|---|---|---|---|---|
106
+ | **+ ridge probe on embedding (v1.1, served)** | **0.895** | | | | |
107
+ | this model (step 3600, head only, v1.0) | 0.886 | 0.769 | 0.929 | 0.693 | 0.862 |
108
+ | same recipe without gold head | 0.859 | | | | |
109
+ | previous LLM decider (0.8B, MLX 4-bit) | 0.687 | | | 0.538 | 0.712 |
110
+
111
+ On a fixed 300-prompt subset used to benchmark about 40 OpenRouter decision models, this model scores AUC 0.904. The
112
+ best teachers scored 0.879 (`nace-ai/drex-max-finance`), 0.873 (`nace-ai/drex-max`) and 0.871
113
+ (`perplexity/pplx-decider-v1.3`).
114
+
115
+ The checkpoint was chosen on this validation set, so these numbers are slightly optimistic.
116
+
117
+ ## Speed
118
+
119
+ Apple M4 Max (48 GB), PyTorch, single request, batch 1. Forward pass only at fixed lengths; "real prompts" includes
120
+ tokenization on the 1,994 validation prompts (median 80 tokens). The ridge probe adds one 2048-wide dot product.
121
+
122
+ | tokens | MPS fp16 p50 | MPS fp16 p95 | CPU fp32 p50 | CPU fp32 p95 |
123
+ |---|---|---|---|---|
124
+ | 32 | 11.4 ms | 11.7 ms | 49.4 ms | 55.3 ms |
125
+ | 128 | 17.7 ms | 18.1 ms | 125.1 ms | 141.7 ms |
126
+ | 256 | 27.6 ms | 29.3 ms | 150.3 ms | 161.1 ms |
127
+ | 512 | 107.8 ms | 320.3 ms | 294.1 ms | 317.5 ms |
128
+ | 768 | 145.4 ms | 161.6 ms | 473.3 ms | 522.6 ms |
129
+ | real prompts | 30.2 ms (p99 151 ms) | 107.5 ms | 78.7 ms (p99 320 ms, 200 prompts) | 219.0 ms |
130
+
131
+ Through the HTTP server on MPS (500 prompts): p50 23 ms, p95 74 ms, p99 97 ms. Model load 0.7 s, resident memory
132
+ about 560 MB. The previous LLM decider took about 720 ms per decision.
133
+
134
+ ## Limitations
135
+
136
+ - English only; tuned on coding-assistant and chat prompts.
137
+ - Only the first 3,000 characters of the opening task plus the latest follow-up are scored, truncated to 768 tokens.
138
+ - Labels are noisy teacher outputs; the model ranks difficulty, it does not verify that a model can solve a task.
139
+ - Evaluation numbers are slightly optimistic because the checkpoint was selected on the validation set.
140
+
141
+ ## License and usage risk
142
+
143
+ Code and weights are released under Apache-2.0, matching the ModernBERT base model. The training labels are outputs of
144
+ commercial models. Some providers' terms restrict using outputs to build competing models. This model is a
145
+ binary-ish routing classifier, not a general model, but check the relevant terms before any commercial use. The
146
+ WildChat (ODC-BY) and CC-BY sources require attribution, which the list above provides.
calib.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:63027ab6a877df0b29a63bc6d84005c658bb361884e476b53d335008b0d11303
3
+ size 65940
config.json ADDED
@@ -0,0 +1,102 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "ModernBertForSequenceClassification"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 50281,
8
+ "classifier_activation": "gelu",
9
+ "classifier_bias": false,
10
+ "classifier_dropout": 0.0,
11
+ "classifier_pooling": "mean",
12
+ "cls_token_id": 50281,
13
+ "decoder_bias": true,
14
+ "deterministic_flash_attn": false,
15
+ "dtype": "float32",
16
+ "embedding_dropout": 0.0,
17
+ "eos_token_id": 50282,
18
+ "global_attn_every_n_layers": 3,
19
+ "gradient_checkpointing": false,
20
+ "hidden_activation": "gelu",
21
+ "hidden_size": 1024,
22
+ "id2label": {
23
+ "0": "LABEL_0",
24
+ "1": "LABEL_1",
25
+ "2": "LABEL_2",
26
+ "3": "LABEL_3",
27
+ "4": "LABEL_4",
28
+ "5": "LABEL_5",
29
+ "6": "LABEL_6"
30
+ },
31
+ "initializer_cutoff_factor": 2.0,
32
+ "initializer_range": 0.02,
33
+ "intermediate_size": 2624,
34
+ "label2id": {
35
+ "LABEL_0": 0,
36
+ "LABEL_1": 1,
37
+ "LABEL_2": 2,
38
+ "LABEL_3": 3,
39
+ "LABEL_4": 4,
40
+ "LABEL_5": 5,
41
+ "LABEL_6": 6
42
+ },
43
+ "layer_norm_eps": 1e-05,
44
+ "layer_types": [
45
+ "full_attention",
46
+ "sliding_attention",
47
+ "sliding_attention",
48
+ "full_attention",
49
+ "sliding_attention",
50
+ "sliding_attention",
51
+ "full_attention",
52
+ "sliding_attention",
53
+ "sliding_attention",
54
+ "full_attention",
55
+ "sliding_attention",
56
+ "sliding_attention",
57
+ "full_attention",
58
+ "sliding_attention",
59
+ "sliding_attention",
60
+ "full_attention",
61
+ "sliding_attention",
62
+ "sliding_attention",
63
+ "full_attention",
64
+ "sliding_attention",
65
+ "sliding_attention",
66
+ "full_attention",
67
+ "sliding_attention",
68
+ "sliding_attention",
69
+ "full_attention",
70
+ "sliding_attention",
71
+ "sliding_attention",
72
+ "full_attention"
73
+ ],
74
+ "local_attention": 128,
75
+ "max_position_embeddings": 8192,
76
+ "mlp_bias": false,
77
+ "mlp_dropout": 0.0,
78
+ "model_type": "modernbert",
79
+ "norm_bias": false,
80
+ "norm_eps": 1e-05,
81
+ "num_attention_heads": 16,
82
+ "num_hidden_layers": 28,
83
+ "pad_token_id": 50283,
84
+ "position_embedding_type": "absolute",
85
+ "rope_parameters": {
86
+ "full_attention": {
87
+ "rope_theta": 160000.0,
88
+ "rope_type": "default"
89
+ },
90
+ "sliding_attention": {
91
+ "rope_theta": 10000.0,
92
+ "rope_type": "default"
93
+ }
94
+ },
95
+ "sep_token_id": 50282,
96
+ "sparse_pred_ignore_index": -100,
97
+ "sparse_prediction": false,
98
+ "tie_word_embeddings": true,
99
+ "transformers_version": "5.23.0",
100
+ "use_cache": false,
101
+ "vocab_size": 50368
102
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e6716ac8aa2d58d65389ec946b2e0efe5a4ade72ee2e40a0c039e570fccb5e08
3
+ size 791695054
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "clean_up_tokenization_spaces": true,
4
+ "cls_token": "[CLS]",
5
+ "is_local": false,
6
+ "local_files_only": false,
7
+ "mask_token": "[MASK]",
8
+ "model_input_names": [
9
+ "input_ids",
10
+ "attention_mask"
11
+ ],
12
+ "model_max_length": 8192,
13
+ "pad_token": "[PAD]",
14
+ "sep_token": "[SEP]",
15
+ "tokenizer_class": "TokenizersBackend",
16
+ "unk_token": "[UNK]"
17
+ }