Hugging Face's logo

rdaum
/
decisive-dory

Text Generation
Safetensors
GGUF
English
dory
decision-making
lora
eider
bf16
nvfp4
llama-cpp
conversational
decision-model
Model card Files
xet
 Community
rdaum commited on
Commit
ca51159
·
verified ·
1 Parent(s): 3eefc3a

Add matched-question comparisons with hosted decision models

Browse files
Files changed (4) hide show
  1. README.md +26 -2
  2. evaluation/README.md +104 -0
  3. evaluation/openrouter-comparison.json +393 -0
  4. evaluation/questions-512.json +0 -0
README.md CHANGED
@@ -96,7 +96,7 @@ The exact prompt and label mapping matter; the GGUF includes the decision templa
96
 
97
  The following sources provide the decision questions and targets used for fine-tuning.
98
  [training-data.json](training-data.json) records source versions, licences, hashes, sampling, and split rules.
99
- Source datasets are not redistributed here.
100
 
101
  | Area | Source | Training task |
102
  |---|---|---|
@@ -131,7 +131,10 @@ NVFP4 uses packed E2M1 weights with E4M3 block scales and an F32 tensor scale in
131
 
132
  ## Evaluation
133
 
134
- Both comparisons use the same 512 questions, two recurrent loops, the same typed-decision prompts, and no probability calibration.
 
 
 
135
  Within each comparison, base Dory and the fine-tune use the same quantization and runtime.
136
 
137
  The model was selected using a separate 1,024-question validation set before the test questions were scored.
@@ -170,6 +173,27 @@ These are adapted decision tasks, not scores under the source datasets' original
170
 
171
  The BF16 weights have not received the same accuracy evaluation.
172
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
173
  ## Limitations
174
 
175
  Probabilities are normalized over the supplied answers and have no fitted calibration profile.
 
96
 
97
  The following sources provide the decision questions and targets used for fine-tuning.
98
  [training-data.json](training-data.json) records source versions, licences, hashes, sampling, and split rules.
99
+ The full source datasets are not redistributed here. The [evaluation subset](evaluation/README.md) includes source attribution and licence terms.
100
 
101
  | Area | Source | Training task |
102
  |---|---|---|
 
131
 
132
  ## Evaluation
133
 
134
+ The [512-question test set](evaluation/questions-512.json) includes the exact inputs, answer options, and target distributions used in all tables below.
135
+ [Scoring rules and source attribution](evaluation/README.md) explain how to reproduce the results.
136
+
137
+ Both base-versus-fine-tune comparisons use the same 512 questions, two recurrent loops, the same typed-decision prompts, and no probability calibration.
138
  Within each comparison, base Dory and the fine-tune use the same quantization and runtime.
139
 
140
  The model was selected using a separate 1,024-question validation set before the test questions were scored.
 
173
 
174
  The BF16 weights have not received the same accuracy evaluation.
175
 
176
+ ### Comparison with hosted decision models
177
+
178
+ The same 512 questions were submitted to four models through OpenRouter's native decision API on 21 October 2026.
179
+ Each request used the same context, question, answer descriptions, and option order as the local evaluations. Reference answers were not sent.
180
+ Hosted models used their providers' prompt templates and probability settings; no prompts or calibration were fitted for this comparison.
181
+
182
+ | Model | Accuracy | Brier ↓ |
183
+ |---|---:|---:|
184
+ | Dory Typed Decisions — Q8_0 / llama.cpp | 77.73% | 0.2599 |
185
+ | Dory Typed Decisions — NVFP4 / Eider | 75.59% | 0.2706 |
186
+ | [Perplexity Decider 1.2](https://openrouter.ai/perplexity/pplx-decider-v1.2) | 74.61% | 0.3178 |
187
+ | [Liquid d1](https://openrouter.ai/liquid/d1) | 66.21% | 0.4156 |
188
+ | [TypeSafe Jev 1.14](https://openrouter.ai/typesafe/jev-1.14) | 66.02% | 0.3843 |
189
+ | [Kev 4B](https://openrouter.ai/jaredpalmer/kev-4b) | 42.58% | 0.5134 |
190
+
191
+ This test measures new examples from task families included in this fine-tune's training. It does not establish a general ranking of decision models.
192
+ The hosted models' training overlap with these sources is unknown.
193
+ Small rounding errors in returned distributions were normalized before Brier scoring; this preserves their answer ranking.
194
+ [Comparison details](evaluation/openrouter-comparison.json) include exact served model versions, per-task scores, and normalization counts.
195
+ Latency is not ranked here: remote requests include network and provider overhead, while the local models ran on DGX Spark.
196
+
197
  ## Limitations
198
 
199
  Probabilities are normalized over the supplied answers and have no fitted calibration profile.
evaluation/README.md ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Decision evaluation: 512 questions
2
+
3
+ [Download the questions and targets](questions-512.json).
4
+ This is the exact file used for the base Dory, fine-tuned Dory, and hosted-model comparisons in the model card.
5
+
6
+ | Task | Questions | Source and target |
7
+ |---|---:|---|
8
+ | State tracking | 192 | SCONE Alchemy and Tangrams: questions about recorded world states after instructions. |
9
+ | Planning | 192 | Generated block-stacking and package-delivery problems: simulated legality and search-derived optimal actions. |
10
+ | Human judgements | 128 | HelpSteer2: questions with distributions derived from released human ratings and preference votes. |
11
+
12
+ These examples were withheld from this model's training. Their task families are represented in training.
13
+ The results measure performance on this task mix, not general decision-making ability or the source datasets' original benchmarks.
14
+ Related examples share a `group_id`; 512 questions should not be interpreted as 512 independent source situations.
15
+ Hosted providers' training overlap with these sources is unknown.
16
+
17
+ ## File integrity
18
+
19
+ SHA-256 of `questions-512.json`:
20
+
21
+ ```text
22
+ e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625
23
+ ```
24
+
25
+ The file is a JSON array. Original identifiers and metadata are preserved so its hash matches the evaluation reports.
26
+
27
+ ## Fields
28
+
29
+ | Field | Meaning |
30
+ |---|---|
31
+ | `id` | Unique question identifier. |
32
+ | `state` | Context presented to the model. |
33
+ | `instructions` | Question or judgement requested. |
34
+ | `kind` | `noul` for binary, `choice` for named options, or `score` for ordered levels. |
35
+ | `options` | Ordered list of `[answer_name, answer_description]` pairs. Preserve this order. |
36
+ | `target` | Target probability for each option, in the same order. Never send these values to the model. |
37
+ | `target_basis` | How the reference distribution was constructed. |
38
+ | `source`, `dataset`, `domain` | Source and task identifiers. |
39
+ | `group_id` | Related source episode, prompt, or canonical planning problem. |
40
+ | `tokens`, `labels` | Dory prompt token IDs and corresponding answer-label token IDs used in local evaluation. Other models use their own tokenizers. |
41
+ | `role`, `component` | Dataset assembly metadata; not model input. |
42
+
43
+ ## Requests
44
+
45
+ Send one question at a time with its unchanged `state`, `instructions`, and answer descriptions.
46
+ For choice and binary questions, map answer names to descriptions. For scores, send the descriptions as an ordered list.
47
+ For example, construct an OpenRouter native decision request from a row as follows:
48
+
49
+ ```python
50
+ criteria = (
51
+ [description for name, description in row["options"]]
52
+ if row["kind"] == "score"
53
+ else dict(row["options"])
54
+ )
55
+ request = {
56
+ "model": model_id,
57
+ "state": row["state"],
58
+ "questions": {
59
+ row["id"]: {
60
+ "type": row["kind"],
61
+ "instructions": row["instructions"],
62
+ "criteria": criteria,
63
+ }
64
+ },
65
+ }
66
+ ```
67
+
68
+ Hosted comparisons use `/api/alpha/decisions` at OpenRouter. The llama.cpp comparison uses `/v1/systemone`.
69
+ Providers control their internal prompts and probability settings. Local Dory evaluations use two recurrent loops and no fitted calibration.
70
+ No generated reasoning or additional examples are requested.
71
+
72
+ ## Scoring
73
+
74
+ Read probabilities in `options` order. For a binary response `p`, use `p` for `true` and `1-p` for `false`.
75
+ All reported comparisons require a valid response for every question; no failed questions are omitted.
76
+
77
+ - **Accuracy:** choose the option with the largest predicted probability. Resolve prediction ties by the original option order. Count it as correct if its target probability is within `1e-6` of the largest target probability.
78
+ - **Brier error:** sum `(prediction[i] - target[i])**2` over options, then average over questions. Lower is better. This is a sum over options, not an average over options.
79
+ - **Combined results:** average over all 512 questions, rather than averaging the three task scores equally.
80
+
81
+ Hosted responses are checked for finite probabilities between zero and one and the exact expected options.
82
+ If their sum differs from one by at most 0.02, divide by the sum before scoring. Larger discrepancies are rejected.
83
+ This handles provider rounding without changing the ordering of answers.
84
+ The [hosted comparison report](openrouter-comparison.json) records normalization counts, observed discrepancies, and exact served model versions.
85
+
86
+ Targets can represent a recorded fact, several equally optimal actions, or disagreement among human raters.
87
+ Matching their most probable answer does not by itself establish calibrated confidence.
88
+
89
+ ## Sources, changes, and licences
90
+
91
+ The evaluation data have separate terms from the model weights' OpenMDW licence.
92
+
93
+ - **SCONE:** Long, Pasupat, and Liang (2016), [Sequential CONtext-dependent Execution](https://nlp.stanford.edu/projects/scone/), under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Episodes and observed states were converted into typed questions and reference distributions. These adapted rows remain under CC BY-SA 4.0.
94
+ - **HelpSteer2:** NVIDIA and Scale AI, [HelpSteer2](https://huggingface.co/datasets/nvidia/HelpSteer2), under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Prompts, responses, ratings, and preference votes were converted into typed judgement questions and empirical distributions. Source revision: `990b2711a36180dd19d9c94b8627844866f8982a`.
95
+ - **Planning:** independently generated block-stacking and delivery problems, inspired by the task families in [PlanBench](https://github.com/karthikv792/LLMs-Planning/tree/main/plan-bench). Published PlanBench instances and generator code were not used. The generated questions and targets are provided by Ryan Daum under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
96
+
97
+ Collection and question-format adaptations are provided under CC BY-SA 4.0; the source-specific terms above remain applicable.
98
+ [Source provenance](../training-data.json) includes source archive hashes and the split policy.
99
+
100
+ ## Results
101
+
102
+ - [Base versus fine-tuned Dory, NVFP4 / Eider](base-comparison.json)
103
+ - [Base versus fine-tuned Dory, Q8_0 / llama.cpp](llama-comparison.json)
104
+ - [Hosted decision models on the same questions](openrouter-comparison.json)
evaluation/openrouter-comparison.json ADDED
@@ -0,0 +1,393 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "protocol": {
3
+ "endpoint": "https://openrouter.ai/api/alpha/decisions",
4
+ "models": [
5
+ "typesafe/jev-1.14",
6
+ "liquid/d1",
7
+ "perplexity/pplx-decider-v1.2",
8
+ "jaredpalmer/kev-4b"
9
+ ],
10
+ "test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
11
+ "questions": 512,
12
+ "protocol": "One request per question. Identical state, question instructions, answer meanings, and option order; no reference targets sent. Native provider prompts and calibration remain provider-controlled. No prompt optimization or fitting.",
13
+ "concurrency": "Four models concurrently, one request at a time per model. HTTP latency includes network/provider overhead and is not local GPU compute latency.",
14
+ "probability_rounding": "Returned distributions within 0.02 of unit mass are renormalized before Brier scoring; raw API responses are retained. Argmax option order breaks probability ties.",
15
+ "cost_limit_usd": 1.0,
16
+ "date": "2026-10-21"
17
+ },
18
+ "models": {
19
+ "liquid/d1": {
20
+ "questions": 512,
21
+ "modal_accuracy": 0.662109375,
22
+ "brier": 0.4156228551936487,
23
+ "median_http_ms": 153.96152925677598,
24
+ "cost_usd": 0.0032266,
25
+ "renormalized_responses": 0,
26
+ "max_probability_sum_error": 4.440892098500626e-16,
27
+ "requested_model": "liquid/d1",
28
+ "served_models": [
29
+ "liquid/d1-20260930"
30
+ ],
31
+ "by_source": {
32
+ "helpsteer2": {
33
+ "questions": 128,
34
+ "modal_accuracy": 0.5859375,
35
+ "brier": 0.31772152271267684,
36
+ "median_http_ms": 156.79931757040322,
37
+ "cost_usd": 0.00130398
38
+ },
39
+ "planning": {
40
+ "questions": 192,
41
+ "modal_accuracy": 0.7916666666666666,
42
+ "brier": 0.36154586401436944,
43
+ "median_http_ms": 160.43087700381875,
44
+ "cost_usd": 0.00111626
45
+ },
46
+ "scone": {
47
+ "questions": 192,
48
+ "modal_accuracy": 0.5833333333333334,
49
+ "brier": 0.5349674013602426,
50
+ "median_http_ms": 146.55435550957918,
51
+ "cost_usd": 0.00080636
52
+ }
53
+ },
54
+ "by_kind": {
55
+ "choice": {
56
+ "questions": 241,
57
+ "modal_accuracy": 0.6970954356846473,
58
+ "brier": 0.45294190031856313,
59
+ "median_http_ms": 153.44042284414172,
60
+ "cost_usd": 0.0013192
61
+ },
62
+ "noul": {
63
+ "questions": 94,
64
+ "modal_accuracy": 0.7127659574468085,
65
+ "brier": 0.36126946919251096,
66
+ "median_http_ms": 152.50918315723538,
67
+ "cost_usd": 0.00040766
68
+ },
69
+ "score": {
70
+ "questions": 177,
71
+ "modal_accuracy": 0.5875706214689266,
72
+ "brier": 0.39367555806936955,
73
+ "median_http_ms": 156.5550728701055,
74
+ "cost_usd": 0.00149974
75
+ }
76
+ }
77
+ },
78
+ "perplexity/pplx-decider-v1.2": {
79
+ "questions": 512,
80
+ "modal_accuracy": 0.74609375,
81
+ "brier": 0.3177875424868089,
82
+ "median_http_ms": 190.27893454767764,
83
+ "cost_usd": 0.00346588,
84
+ "renormalized_responses": 0,
85
+ "max_probability_sum_error": 4.440892098500626e-16,
86
+ "requested_model": "perplexity/pplx-decider-v1.2",
87
+ "served_models": [
88
+ "perplexity/pplx-decider-v1.2-20261016"
89
+ ],
90
+ "by_source": {
91
+ "helpsteer2": {
92
+ "questions": 128,
93
+ "modal_accuracy": 0.5703125,
94
+ "brier": 0.2787386993739065,
95
+ "median_http_ms": 197.84710323438048,
96
+ "cost_usd": 0.00139706
97
+ },
98
+ "planning": {
99
+ "questions": 192,
100
+ "modal_accuracy": 0.8802083333333334,
101
+ "brier": 0.26224774345602686,
102
+ "median_http_ms": 189.46263939142227,
103
+ "cost_usd": 0.00119936
104
+ },
105
+ "scone": {
106
+ "questions": 192,
107
+ "modal_accuracy": 0.7291666666666666,
108
+ "brier": 0.3993599035928593,
109
+ "median_http_ms": 185.11412316001952,
110
+ "cost_usd": 0.00086946
111
+ }
112
+ },
113
+ "by_kind": {
114
+ "choice": {
115
+ "questions": 241,
116
+ "modal_accuracy": 0.8091286307053942,
117
+ "brier": 0.3446775651495988,
118
+ "median_http_ms": 189.45799581706524,
119
+ "cost_usd": 0.0014693
120
+ },
121
+ "noul": {
122
+ "questions": 94,
123
+ "modal_accuracy": 0.8191489361702128,
124
+ "brier": 0.24647724911756672,
125
+ "median_http_ms": 183.42466722242534,
126
+ "cost_usd": 0.00042356
127
+ },
128
+ "score": {
129
+ "questions": 177,
130
+ "modal_accuracy": 0.6214689265536724,
131
+ "brier": 0.3190455770346982,
132
+ "median_http_ms": 192.91105587035418,
133
+ "cost_usd": 0.00157302
134
+ }
135
+ }
136
+ },
137
+ "typesafe/jev-1.14": {
138
+ "questions": 512,
139
+ "modal_accuracy": 0.66015625,
140
+ "brier": 0.38428551299771707,
141
+ "median_http_ms": 160.70113889873028,
142
+ "cost_usd": 0.012567408,
143
+ "renormalized_responses": 6,
144
+ "max_probability_sum_error": 0.009999999999999898,
145
+ "requested_model": "typesafe/jev-1.14",
146
+ "served_models": [
147
+ "typesafe/jev-1.14-20261014"
148
+ ],
149
+ "by_source": {
150
+ "helpsteer2": {
151
+ "questions": 128,
152
+ "modal_accuracy": 0.5546875,
153
+ "brier": 0.3501162689075094,
154
+ "median_http_ms": 161.725880112499,
155
+ "cost_usd": 0.00407988
156
+ },
157
+ "planning": {
158
+ "questions": 192,
159
+ "modal_accuracy": 0.7447916666666666,
160
+ "brier": 0.3210494803889059,
161
+ "median_http_ms": 162.16431255452335,
162
+ "cost_usd": 0.004622646
163
+ },
164
+ "scone": {
165
+ "questions": 192,
166
+ "modal_accuracy": 0.6458333333333334,
167
+ "brier": 0.4703010416666667,
168
+ "median_http_ms": 158.17508311010897,
169
+ "cost_usd": 0.003864882
170
+ }
171
+ },
172
+ "by_kind": {
173
+ "choice": {
174
+ "questions": 241,
175
+ "modal_accuracy": 0.6804979253112033,
176
+ "brier": 0.4145994016191929,
177
+ "median_http_ms": 159.9918780848384,
178
+ "cost_usd": 0.005741064
179
+ },
180
+ "noul": {
181
+ "questions": 94,
182
+ "modal_accuracy": 0.7446808510638298,
183
+ "brier": 0.2824255319148936,
184
+ "median_http_ms": 159.92187475785613,
185
+ "cost_usd": 0.001838382
186
+ },
187
+ "score": {
188
+ "questions": 177,
189
+ "modal_accuracy": 0.5875706214689266,
190
+ "brier": 0.3971058014949472,
191
+ "median_http_ms": 162.21308708190918,
192
+ "cost_usd": 0.004987962
193
+ }
194
+ }
195
+ },
196
+ "jaredpalmer/kev-4b": {
197
+ "questions": 512,
198
+ "modal_accuracy": 0.42578125,
199
+ "brier": 0.5133862273141486,
200
+ "median_http_ms": 374.62099245749414,
201
+ "cost_usd": 0.00583884,
202
+ "renormalized_responses": 166,
203
+ "max_probability_sum_error": 0.00019999999999997797,
204
+ "requested_model": "jaredpalmer/kev-4b",
205
+ "served_models": [
206
+ "jaredpalmer/kev-4b-20260924"
207
+ ],
208
+ "by_source": {
209
+ "helpsteer2": {
210
+ "questions": 128,
211
+ "modal_accuracy": 0.578125,
212
+ "brier": 0.2507105103891895,
213
+ "median_http_ms": 391.88018697313964,
214
+ "cost_usd": 0.002482536
215
+ },
216
+ "planning": {
217
+ "questions": 192,
218
+ "modal_accuracy": 0.3645833333333333,
219
+ "brier": 0.5232131482077674,
220
+ "median_http_ms": 364.3412641249597,
221
+ "cost_usd": 0.00194796
222
+ },
223
+ "scone": {
224
+ "questions": 192,
225
+ "modal_accuracy": 0.3854166666666667,
226
+ "brier": 0.6786764510371691,
227
+ "median_http_ms": 369.4418070372194,
228
+ "cost_usd": 0.001408344
229
+ }
230
+ },
231
+ "by_kind": {
232
+ "choice": {
233
+ "questions": 241,
234
+ "modal_accuracy": 0.26556016597510373,
235
+ "brier": 0.648331459991458,
236
+ "median_http_ms": 369.08131977543235,
237
+ "cost_usd": 0.002369598
238
+ },
239
+ "noul": {
240
+ "questions": 94,
241
+ "modal_accuracy": 0.6170212765957447,
242
+ "brier": 0.4348702625531915,
243
+ "median_http_ms": 367.6399344112724,
244
+ "cost_usd": 0.000701316
245
+ },
246
+ "score": {
247
+ "questions": 177,
248
+ "modal_accuracy": 0.5423728813559322,
249
+ "brier": 0.37134498218589096,
250
+ "median_http_ms": 380.40076615288854,
251
+ "cost_usd": 0.002767926
252
+ }
253
+ }
254
+ }
255
+ },
256
+ "errors": {},
257
+ "total_cost_usd": 0.02509872800000002,
258
+ "local_results": {
259
+ "Dory Typed Decisions \u2014 Q8_0 / llama.cpp": {
260
+ "questions": 512,
261
+ "modal_accuracy": 0.77734375,
262
+ "brier": 0.2599162990600157,
263
+ "median_request_ms": 109.77665497921407,
264
+ "mean_request_ms": 135.0687600670426,
265
+ "by_source": {
266
+ "helpsteer2": {
267
+ "questions": 128,
268
+ "modal_accuracy": 0.515625,
269
+ "brier": 0.23377195573417558,
270
+ "median_request_ms": 211.45723038353026,
271
+ "mean_request_ms": 208.28389131929725
272
+ },
273
+ "planning": {
274
+ "questions": 192,
275
+ "modal_accuracy": 0.96875,
276
+ "brier": 0.14051244957828152,
277
+ "median_request_ms": 112.26724414154887,
278
+ "mean_request_ms": 119.26124551003643
279
+ },
280
+ "scone": {
281
+ "questions": 192,
282
+ "modal_accuracy": 0.7604166666666666,
283
+ "brier": 0.3967497107589765,
284
+ "median_request_ms": 103.08765154331923,
285
+ "mean_request_ms": 102.06618712254567
286
+ }
287
+ },
288
+ "by_kind": {
289
+ "choice": {
290
+ "questions": 241,
291
+ "modal_accuracy": 0.8257261410788381,
292
+ "brier": 0.3344495604077365,
293
+ "median_request_ms": 110.2855778299272,
294
+ "mean_request_ms": 120.80057657659981
295
+ },
296
+ "noul": {
297
+ "questions": 94,
298
+ "modal_accuracy": 0.9680851063829787,
299
+ "brier": 0.07220308279158415,
300
+ "median_request_ms": 101.73197486437857,
301
+ "mean_request_ms": 97.29551964815944
302
+ },
303
+ "score": {
304
+ "questions": 177,
305
+ "modal_accuracy": 0.6101694915254238,
306
+ "brier": 0.2581226625878792,
307
+ "median_request_ms": 134.17465519160032,
308
+ "mean_request_ms": 174.55642571999024
309
+ }
310
+ },
311
+ "model_sha256": "bb36b8ed263ce633649ccfad9c6e1dc065e9c3d333624fc27028f496089a38b8",
312
+ "loops": 2,
313
+ "prompt_parity_questions": 512,
314
+ "command": [
315
+ "llama-server",
316
+ "--model",
317
+ "dory-decisions-Q8_0.gguf",
318
+ "--alias",
319
+ "dory-gguf-eval",
320
+ "--host",
321
+ "127.0.0.1",
322
+ "--port",
323
+ "18091",
324
+ "--ctx-size",
325
+ "4096",
326
+ "--parallel",
327
+ "1",
328
+ "--threads",
329
+ "4",
330
+ "--threads-batch",
331
+ "4",
332
+ "--batch-size",
333
+ "512",
334
+ "--ubatch-size",
335
+ "512",
336
+ "--fit",
337
+ "off",
338
+ "--gpu-layers",
339
+ "all",
340
+ "--no-warmup"
341
+ ]
342
+ },
343
+ "Dory Typed Decisions \u2014 NVFP4 / Eider": {
344
+ "questions": 512,
345
+ "modal_accuracy": 0.755859375,
346
+ "brier": 0.2705710538872297,
347
+ "cross_entropy": 1.0354930422480668,
348
+ "by_kind": {
349
+ "choice": {
350
+ "questions": 241,
351
+ "modal_accuracy": 0.8215767634854771,
352
+ "brier": 0.30593746240810665,
353
+ "cross_entropy": 1.0288481446263855
354
+ },
355
+ "noul": {
356
+ "questions": 94,
357
+ "modal_accuracy": 0.8936170212765957,
358
+ "brier": 0.1853453372075174,
359
+ "cross_entropy": 0.4718470608079094
360
+ },
361
+ "score": {
362
+ "questions": 177,
363
+ "modal_accuracy": 0.5932203389830508,
364
+ "brier": 0.26767790651074164,
365
+ "cross_entropy": 1.34387802858818
366
+ }
367
+ },
368
+ "by_source": {
369
+ "helpsteer2": {
370
+ "questions": 128,
371
+ "modal_accuracy": 0.5078125,
372
+ "brier": 0.22898458428820745,
373
+ "cross_entropy": 1.5346420909767997
374
+ },
375
+ "planning": {
376
+ "questions": 192,
377
+ "modal_accuracy": 0.921875,
378
+ "brier": 0.1909474730913934,
379
+ "cross_entropy": 0.8915435249464682
380
+ },
381
+ "scone": {
382
+ "questions": 192,
383
+ "modal_accuracy": 0.7552083333333334,
384
+ "brier": 0.3779189477490808,
385
+ "cross_entropy": 0.8466765270638437
386
+ }
387
+ }
388
+ }
389
+ },
390
+ "scope": "512 test questions withheld from this fine-tune. Task families are represented in its training. Provider training overlap is unknown. This is not an official source benchmark or a general model ranking.",
391
+ "scoring": "Accuracy accepts any answer tied for highest target probability. Brier is the sum of squared differences between predicted and target probabilities, averaged over questions.",
392
+ "cost_scope": "Cost covers scored requests and calls made during the final resumed run. Earlier failed-response and diagnostic probe costs are excluded."
393
+ }
evaluation/questions-512.json ADDED
The diff for this file is too large to render. See raw diff