MyeongHoJeong commited on
Commit
5557317
·
verified ·
1 Parent(s): 4488cd6

docs: download snapshot before installing its requirements

Browse files
Files changed (1) hide show
  1. README.md +98 -1
README.md CHANGED
@@ -16,21 +16,31 @@ tags:
16
  ---
17
 
18
 
 
 
19
  # Standard One 3B SH
20
 
21
 
 
 
22
  > **Updated weights (v3, 2026-10-13).** v3 is trained further on decision data, mostly our own synthetic items. Decision Index 0.3 public index 51.53 (self-scored with the official kit; v2 42.90).
23
  > Earlier releases stay available under the tags `v2` and `v1`.
24
 
25
 
 
 
26
  **Version:** v3
27
 
28
 
 
 
29
  **Standard One 3B SH** is Standard One 3B with a **joint schema head** (SH). It reads the scenario once and
30
  scores every option of every question in a single forward pass, then returns probabilities through the same
31
  `POST /v1/systemone` contract as Standard One. It does not generate text.
32
 
33
 
 
 
34
  | | |
35
  |---|---|
36
  | Backbone | Standard One 3B v2.2 (`mistral3`, Ministral 3 3B text + Pixtral vision tower), fine-tuned for the head and merged; weights in this repository |
@@ -43,23 +53,33 @@ scores every option of every question in a single forward pass, then returns pro
43
  | License | Apache-2.0 |
44
 
45
 
 
 
46
  ## Why a schema head
47
 
48
 
 
 
49
  Standard One reads one answer letter per option, so a question is limited by the label alphabet and every
50
  question in a request needs its own pass over the scenario. The schema head instead attends over the hidden states
51
  of the whole request (the final layer and one intermediate layer), places every question and option in one joint
52
  graph, and returns a calibrated distribution for each question at once. In practice:
53
 
54
 
 
 
55
  - **Many options.** Intent and routing questions with 100–255 options keep nearly the accuracy of small option sets.
56
  - **Many questions per scenario.** All questions of a request share one pass over the scenario.
57
  - **Long scenarios.** Decisions buried in long documents are found far more reliably than with Standard One v2.2.
58
 
59
 
 
 
60
  ## Results
61
 
62
 
 
 
63
  | Benchmark | SH v2 | **SH v3 (this release)** |
64
  |---|---:|---:|
65
  | Decision Index 0.3, public index | 42.90 | **51.53** |
@@ -70,6 +90,8 @@ graph, and returns a calibrated distribution for each question at once. In pract
70
  | JevBench public: easy / original / hard (48 / 72 / 111 items) | 100.00 / 98.61 / 63.96 | 100.00 / 98.61 / 63.96 |
71
 
72
 
 
 
73
  Decision Index rows (DI) are chance-corrected skill scores (0 = random guessing, 100 = perfect) from our own runs of the public
74
  0.3 suite with the official kit, all 140,178 scoreable requests; every benchmark of both runs is in [standardone-3b-sh-v2](https://huggingface.co/datasets/StandardThinking/StandardOne-SH-decision-index/tree/main/runs/standardone-3b-sh-v2)
75
  and [standardone-3b-sh-v3](https://huggingface.co/datasets/StandardThinking/StandardOne-SH-decision-index/tree/main/runs/standardone-3b-sh-v3). For some DI benchmarks the public train split or the benchmark's public
@@ -77,48 +99,76 @@ generator was part of training (HoVer among them); evaluated items never were. T
77
  on identical items: the v2 column with the reference server in `code/`, the v3 column with the Standard One engine.
78
 
79
 
 
 
80
  ## Benchmark charts
81
 
82
 
 
 
83
  Decision Index 0.3 public index, board data of 2026-10-12; Standard One self-scored with the official kit (see the footnote of each chart).
84
  The same charts with notes: [Standard One SH on the Decision Index](https://standardthinking.ai/resources/guides/standard-one-sh-decision-index/).
85
 
86
 
 
 
87
  ![Public index against model size](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/01-index-vs-size.png)
88
 
89
 
 
 
90
  ![Retrieval & Classification](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/02-retrieval-classification.png)
91
 
92
 
 
 
93
  ![Tools & Automation](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/03-tools-automation.png)
94
 
95
 
 
 
96
  ![Product search relevance (Amazon ESCI)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/04-product-search.png)
97
 
98
 
 
 
99
  ![Multi-hop claim verification (HoVer)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/05-claim-verification.png)
100
 
101
 
 
 
102
  ![Smart-home request handling (Home appliance simulator)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/06-home-appliance.png)
103
 
104
 
 
 
105
  ![Contract understanding (ContractNLI)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/07-contract-nli.png)
106
 
107
 
 
 
108
  ![Reading chords from a score (POP909-CL)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/08-chords.png)
109
 
110
 
 
 
111
  ## Changes in v3
112
 
113
 
 
 
114
  - **Further training on synthetic decision data.** v3 continues from v2 on decision items built for this round, mostly our own
115
  synthetic items (code-checked, LLM-written in varied styles), with replay of earlier data. See "Results" for the
116
  Decision Index 0.3 and the long-context checks.
117
 
118
 
 
 
119
  ## Changes in v2
120
 
121
 
 
 
122
  - **Hard decision tasks.** v2 is fine-tuned on about 2,200 new decision items written in the families where v1 was
123
  weakest: long policies with exceptions and overrides, multi-criteria trade-offs, judging a response against a
124
  rubric, and date and number reasoning. The items were authored by large language models, every gold answer was
@@ -132,20 +182,31 @@ The same charts with notes: [Standard One SH on the Decision Index](https://stan
132
  label sets are within one point of v1. Keep `v1` if policy-judgement repair matters most to you.
133
 
134
 
 
 
135
  ## Prompt format
136
 
137
 
 
 
138
  The included `chat_template.jinja` has an empty default system message, and the reference server renders requests
139
  without one (`SH_SYSTEM_PROMPT=none`, the default in `code/render.py`). Do not add the Ministral default system
140
  prompt: the head was trained without it, and adding it changes the hidden states it reads.
141
 
142
 
 
 
143
  ## Quick start
144
 
145
 
 
 
146
  ```bash
147
- pip install -r code/requirements.txt # torch, transformers, safetensors
148
  hf download StandardThinking/StandardOne-3B-SH --local-dir StandardOne-3B-SH
 
 
 
149
 
150
 
151
  python StandardOne-3B-SH/code/serve_head.py \
@@ -154,6 +215,8 @@ python StandardOne-3B-SH/code/serve_head.py \
154
  ```
155
 
156
 
 
 
157
  ```bash
158
  curl -s localhost:30171/v1/systemone -H 'content-type: application/json' -d '{
159
  "state": {"message": "Please close my card, I lost it yesterday."},
@@ -167,6 +230,8 @@ curl -s localhost:30171/v1/systemone -H 'content-type: application/json' -d '{
167
  ```
168
 
169
 
 
 
170
  The server batches concurrent requests (`--max-batch-requests`, `--max-batch-tokens`) and binds `127.0.0.1` by
171
  default. It runs on one GPU with the backbone in bfloat16 and the head in float32. On Blackwell GPUs (B200/B300)
172
  install a CUDA 13 build of PyTorch first (`pip install torch --index-url https://download.pytorch.org/whl/cu130`).
@@ -174,18 +239,27 @@ For CPU or Apple-silicon use, see [StandardOne-3B-SH-GGUF](https://huggingface.c
174
 
175
 
176
 
 
 
 
177
  ## Run locally with llama.cpp
178
 
 
179
  For CPU or Apple-silicon use, follow the download, build and server instructions in [StandardOne-3B-SH-GGUF](https://huggingface.co/StandardThinking/StandardOne-3B-SH-GGUF#quick-start-with-llamacpp). The SH head reads hidden states from the backbone, so it requires the supplied state extractor and Python runner; stock `llama-server -hf` alone does not execute the schema head. Once the runner is listening on port 30171, use the requests below unchanged.
180
 
 
181
  ## Practical examples
182
 
 
183
  Use the running server at `http://127.0.0.1:30171/v1/systemone`. SH scores all questions jointly without generating text. These examples are illustrative workloads, not measured accuracy results; inspect the actual response on your chosen model and quantization.
184
 
 
185
  ### Customer support: route, handoff and priority
186
 
 
187
  Run this in a second terminal after starting the server:
188
 
 
189
  ```bash
190
  curl --fail-with-body http://127.0.0.1:30171/v1/systemone \
191
  -H "Content-Type: application/json" --data-binary @- <<'JSON'
@@ -212,16 +286,21 @@ curl --fail-with-body http://127.0.0.1:30171/v1/systemone \
212
  JSON
213
  ```
214
 
 
215
  The intended routing label here is `billing`, with a human handoff. This describes the task, not a promised model output. `choice` selects from supplied categories; `noul` scores a yes/no question; `score` uses an ordered scale and returns an expected zero-based level that can be fractional.
216
 
 
217
  ### Ticket routing, incident severity, agent completion and document routing
218
 
 
219
  This Python client uses only the standard library and prints the complete server response. The requests work with the BF16 reference server and the SH GGUF runner on port 30171.
220
 
 
221
  ```python
222
  import json
223
  from urllib.request import Request, urlopen
224
 
 
225
  def decide(state, questions):
226
  request = Request(
227
  "http://127.0.0.1:30171/v1/systemone",
@@ -232,6 +311,7 @@ def decide(state, questions):
232
  with urlopen(request, timeout=300) as response:
233
  return json.load(response)
234
 
 
235
  # Route a ticket to one queue from your own taxonomy.
236
  print(json.dumps(decide(
237
  "I cannot sign in after resetting my password. The reset link has expired.",
@@ -242,6 +322,7 @@ print(json.dumps(decide(
242
  "other": "None of these queues"}}}
243
  ), indent=2))
244
 
 
245
  # Score an ordered incident severity scale.
246
  print(json.dumps(decide(
247
  "Production checkout is unavailable for all customers. The incident is ongoing.",
@@ -252,12 +333,14 @@ print(json.dumps(decide(
252
  "Critical outage affecting all customers"]}}
253
  ), indent=2))
254
 
 
255
  # Check an observed result against the original goal.
256
  print(json.dumps(decide(
257
  "Goal: send the weekly report to the team. Observed state: draft saved; the message has not been sent.",
258
  {"finished": {"type": "noul", "instructions": "Has the agent completed the goal?"}}
259
  ), indent=2))
260
 
 
261
  # Classify a document before selecting an extraction workflow.
262
  print(json.dumps(decide(
263
  "INVOICE 2048. Amount due: USD 240. Payment due within 30 days.",
@@ -269,11 +352,15 @@ print(json.dumps(decide(
269
  ), indent=2))
270
  ```
271
 
 
272
  Each `choice` question selects one category. For independent tags, submit one `noul` question per tag in the same request. Choose probability thresholds on representative validation data before automating routing or handoff. These examples use text input, the validated path for this release.
273
 
 
274
  ## Files
275
 
276
 
 
 
277
  | Path | Contents |
278
  |---|---|
279
  | `model-0000{1,2}-of-00002.safetensors`, `config.json`, tokenizer files | merged backbone, bfloat16 (`Mistral3ForConditionalGeneration`) |
@@ -283,9 +370,13 @@ Each `choice` question selects one category. For independent tags, submit one `n
283
  | `SHA256SUMS`, `release-manifest.json` | checksums; source and merge details |
284
 
285
 
 
 
286
  ## Limitations
287
 
288
 
 
 
289
  - Requests longer than 32,768 tokens are refused, not truncated.
290
  - On our hard held-out decision suite (600 items; long policies, multi-step lookups, date and number reasoning)
291
  it scores 41.67 against 44.67 for Standard One 3B v2.2. Prefer Standard One 3B for that kind of workload.
@@ -294,9 +385,15 @@ Each `choice` question selects one category. For independent tags, submit one `n
294
  - Probabilities are calibrated on our held-out data; refit thresholds on your own distribution before relying on them.
295
 
296
 
 
 
297
  ## License
298
 
299
 
 
 
300
  Apache-2.0. Built on [Standard One 3B](https://huggingface.co/StandardThinking/StandardOne-3B) (Apache-2.0), which is
301
  built on [Ministral 3 3B](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512) (Apache-2.0).
302
 
 
 
 
16
  ---
17
 
18
 
19
+
20
+
21
  # Standard One 3B SH
22
 
23
 
24
+
25
+
26
  > **Updated weights (v3, 2026-10-13).** v3 is trained further on decision data, mostly our own synthetic items. Decision Index 0.3 public index 51.53 (self-scored with the official kit; v2 42.90).
27
  > Earlier releases stay available under the tags `v2` and `v1`.
28
 
29
 
30
+
31
+
32
  **Version:** v3
33
 
34
 
35
+
36
+
37
  **Standard One 3B SH** is Standard One 3B with a **joint schema head** (SH). It reads the scenario once and
38
  scores every option of every question in a single forward pass, then returns probabilities through the same
39
  `POST /v1/systemone` contract as Standard One. It does not generate text.
40
 
41
 
42
+
43
+
44
  | | |
45
  |---|---|
46
  | Backbone | Standard One 3B v2.2 (`mistral3`, Ministral 3 3B text + Pixtral vision tower), fine-tuned for the head and merged; weights in this repository |
 
53
  | License | Apache-2.0 |
54
 
55
 
56
+
57
+
58
  ## Why a schema head
59
 
60
 
61
+
62
+
63
  Standard One reads one answer letter per option, so a question is limited by the label alphabet and every
64
  question in a request needs its own pass over the scenario. The schema head instead attends over the hidden states
65
  of the whole request (the final layer and one intermediate layer), places every question and option in one joint
66
  graph, and returns a calibrated distribution for each question at once. In practice:
67
 
68
 
69
+
70
+
71
  - **Many options.** Intent and routing questions with 100–255 options keep nearly the accuracy of small option sets.
72
  - **Many questions per scenario.** All questions of a request share one pass over the scenario.
73
  - **Long scenarios.** Decisions buried in long documents are found far more reliably than with Standard One v2.2.
74
 
75
 
76
+
77
+
78
  ## Results
79
 
80
 
81
+
82
+
83
  | Benchmark | SH v2 | **SH v3 (this release)** |
84
  |---|---:|---:|
85
  | Decision Index 0.3, public index | 42.90 | **51.53** |
 
90
  | JevBench public: easy / original / hard (48 / 72 / 111 items) | 100.00 / 98.61 / 63.96 | 100.00 / 98.61 / 63.96 |
91
 
92
 
93
+
94
+
95
  Decision Index rows (DI) are chance-corrected skill scores (0 = random guessing, 100 = perfect) from our own runs of the public
96
  0.3 suite with the official kit, all 140,178 scoreable requests; every benchmark of both runs is in [standardone-3b-sh-v2](https://huggingface.co/datasets/StandardThinking/StandardOne-SH-decision-index/tree/main/runs/standardone-3b-sh-v2)
97
  and [standardone-3b-sh-v3](https://huggingface.co/datasets/StandardThinking/StandardOne-SH-decision-index/tree/main/runs/standardone-3b-sh-v3). For some DI benchmarks the public train split or the benchmark's public
 
99
  on identical items: the v2 column with the reference server in `code/`, the v3 column with the Standard One engine.
100
 
101
 
102
+
103
+
104
  ## Benchmark charts
105
 
106
 
107
+
108
+
109
  Decision Index 0.3 public index, board data of 2026-10-12; Standard One self-scored with the official kit (see the footnote of each chart).
110
  The same charts with notes: [Standard One SH on the Decision Index](https://standardthinking.ai/resources/guides/standard-one-sh-decision-index/).
111
 
112
 
113
+
114
+
115
  ![Public index against model size](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/01-index-vs-size.png)
116
 
117
 
118
+
119
+
120
  ![Retrieval & Classification](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/02-retrieval-classification.png)
121
 
122
 
123
+
124
+
125
  ![Tools & Automation](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/03-tools-automation.png)
126
 
127
 
128
+
129
+
130
  ![Product search relevance (Amazon ESCI)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/04-product-search.png)
131
 
132
 
133
+
134
+
135
  ![Multi-hop claim verification (HoVer)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/05-claim-verification.png)
136
 
137
 
138
+
139
+
140
  ![Smart-home request handling (Home appliance simulator)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/06-home-appliance.png)
141
 
142
 
143
+
144
+
145
  ![Contract understanding (ContractNLI)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/07-contract-nli.png)
146
 
147
 
148
+
149
+
150
  ![Reading chords from a score (POP909-CL)](https://huggingface.co/StandardThinking/StandardOne-3B-SH/resolve/main/assets/charts/08-chords.png)
151
 
152
 
153
+
154
+
155
  ## Changes in v3
156
 
157
 
158
+
159
+
160
  - **Further training on synthetic decision data.** v3 continues from v2 on decision items built for this round, mostly our own
161
  synthetic items (code-checked, LLM-written in varied styles), with replay of earlier data. See "Results" for the
162
  Decision Index 0.3 and the long-context checks.
163
 
164
 
165
+
166
+
167
  ## Changes in v2
168
 
169
 
170
+
171
+
172
  - **Hard decision tasks.** v2 is fine-tuned on about 2,200 new decision items written in the families where v1 was
173
  weakest: long policies with exceptions and overrides, multi-criteria trade-offs, judging a response against a
174
  rubric, and date and number reasoning. The items were authored by large language models, every gold answer was
 
182
  label sets are within one point of v1. Keep `v1` if policy-judgement repair matters most to you.
183
 
184
 
185
+
186
+
187
  ## Prompt format
188
 
189
 
190
+
191
+
192
  The included `chat_template.jinja` has an empty default system message, and the reference server renders requests
193
  without one (`SH_SYSTEM_PROMPT=none`, the default in `code/render.py`). Do not add the Ministral default system
194
  prompt: the head was trained without it, and adding it changes the hidden states it reads.
195
 
196
 
197
+
198
+
199
  ## Quick start
200
 
201
 
202
+
203
+
204
  ```bash
205
+ python -m pip install --upgrade huggingface_hub
206
  hf download StandardThinking/StandardOne-3B-SH --local-dir StandardOne-3B-SH
207
+ python -m pip install -r StandardOne-3B-SH/code/requirements.txt
208
+
209
+
210
 
211
 
212
  python StandardOne-3B-SH/code/serve_head.py \
 
215
  ```
216
 
217
 
218
+
219
+
220
  ```bash
221
  curl -s localhost:30171/v1/systemone -H 'content-type: application/json' -d '{
222
  "state": {"message": "Please close my card, I lost it yesterday."},
 
230
  ```
231
 
232
 
233
+
234
+
235
  The server batches concurrent requests (`--max-batch-requests`, `--max-batch-tokens`) and binds `127.0.0.1` by
236
  default. It runs on one GPU with the backbone in bfloat16 and the head in float32. On Blackwell GPUs (B200/B300)
237
  install a CUDA 13 build of PyTorch first (`pip install torch --index-url https://download.pytorch.org/whl/cu130`).
 
239
 
240
 
241
 
242
+
243
+
244
+
245
  ## Run locally with llama.cpp
246
 
247
+
248
  For CPU or Apple-silicon use, follow the download, build and server instructions in [StandardOne-3B-SH-GGUF](https://huggingface.co/StandardThinking/StandardOne-3B-SH-GGUF#quick-start-with-llamacpp). The SH head reads hidden states from the backbone, so it requires the supplied state extractor and Python runner; stock `llama-server -hf` alone does not execute the schema head. Once the runner is listening on port 30171, use the requests below unchanged.
249
 
250
+
251
  ## Practical examples
252
 
253
+
254
  Use the running server at `http://127.0.0.1:30171/v1/systemone`. SH scores all questions jointly without generating text. These examples are illustrative workloads, not measured accuracy results; inspect the actual response on your chosen model and quantization.
255
 
256
+
257
  ### Customer support: route, handoff and priority
258
 
259
+
260
  Run this in a second terminal after starting the server:
261
 
262
+
263
  ```bash
264
  curl --fail-with-body http://127.0.0.1:30171/v1/systemone \
265
  -H "Content-Type: application/json" --data-binary @- <<'JSON'
 
286
  JSON
287
  ```
288
 
289
+
290
  The intended routing label here is `billing`, with a human handoff. This describes the task, not a promised model output. `choice` selects from supplied categories; `noul` scores a yes/no question; `score` uses an ordered scale and returns an expected zero-based level that can be fractional.
291
 
292
+
293
  ### Ticket routing, incident severity, agent completion and document routing
294
 
295
+
296
  This Python client uses only the standard library and prints the complete server response. The requests work with the BF16 reference server and the SH GGUF runner on port 30171.
297
 
298
+
299
  ```python
300
  import json
301
  from urllib.request import Request, urlopen
302
 
303
+
304
  def decide(state, questions):
305
  request = Request(
306
  "http://127.0.0.1:30171/v1/systemone",
 
311
  with urlopen(request, timeout=300) as response:
312
  return json.load(response)
313
 
314
+
315
  # Route a ticket to one queue from your own taxonomy.
316
  print(json.dumps(decide(
317
  "I cannot sign in after resetting my password. The reset link has expired.",
 
322
  "other": "None of these queues"}}}
323
  ), indent=2))
324
 
325
+
326
  # Score an ordered incident severity scale.
327
  print(json.dumps(decide(
328
  "Production checkout is unavailable for all customers. The incident is ongoing.",
 
333
  "Critical outage affecting all customers"]}}
334
  ), indent=2))
335
 
336
+
337
  # Check an observed result against the original goal.
338
  print(json.dumps(decide(
339
  "Goal: send the weekly report to the team. Observed state: draft saved; the message has not been sent.",
340
  {"finished": {"type": "noul", "instructions": "Has the agent completed the goal?"}}
341
  ), indent=2))
342
 
343
+
344
  # Classify a document before selecting an extraction workflow.
345
  print(json.dumps(decide(
346
  "INVOICE 2048. Amount due: USD 240. Payment due within 30 days.",
 
352
  ), indent=2))
353
  ```
354
 
355
+
356
  Each `choice` question selects one category. For independent tags, submit one `noul` question per tag in the same request. Choose probability thresholds on representative validation data before automating routing or handoff. These examples use text input, the validated path for this release.
357
 
358
+
359
  ## Files
360
 
361
 
362
+
363
+
364
  | Path | Contents |
365
  |---|---|
366
  | `model-0000{1,2}-of-00002.safetensors`, `config.json`, tokenizer files | merged backbone, bfloat16 (`Mistral3ForConditionalGeneration`) |
 
370
  | `SHA256SUMS`, `release-manifest.json` | checksums; source and merge details |
371
 
372
 
373
+
374
+
375
  ## Limitations
376
 
377
 
378
+
379
+
380
  - Requests longer than 32,768 tokens are refused, not truncated.
381
  - On our hard held-out decision suite (600 items; long policies, multi-step lookups, date and number reasoning)
382
  it scores 41.67 against 44.67 for Standard One 3B v2.2. Prefer Standard One 3B for that kind of workload.
 
385
  - Probabilities are calibrated on our held-out data; refit thresholds on your own distribution before relying on them.
386
 
387
 
388
+
389
+
390
  ## License
391
 
392
 
393
+
394
+
395
  Apache-2.0. Built on [Standard One 3B](https://huggingface.co/StandardThinking/StandardOne-3B) (Apache-2.0), which is
396
  built on [Ministral 3 3B](https://huggingface.co/mistralai/Ministral-3-3B-Instruct-2512) (Apache-2.0).
397
 
398
+
399
+