RandomMountainMan commited on
Commit
ce407d3
·
verified ·
1 Parent(s): 61e2c88

Restore verified peer comparison, efficiency context, and Ollama support

Browse files
Modelfile ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ FROM ./coneml-810m-alpha-Q8_0.gguf
2
+
3
+ TEMPLATE """{{- range .Messages }}
4
+ {{- if eq .Role "system" }}System:
5
+ {{ .Content }}
6
+ {{- else if eq .Role "user" }}User:
7
+ {{ .Content }}
8
+ {{- else if eq .Role "assistant" }}Assistant:
9
+ {{ .Content }}<|endoftext|>
10
+ {{- end }}
11
+ {{- end }}Assistant:
12
+ """
13
+
14
+ PARAMETER num_ctx 8192
15
+ PARAMETER repeat_penalty 1.15
16
+ PARAMETER stop "<|endoftext|>"
README.md CHANGED
@@ -65,7 +65,47 @@ Behavioral screens:
65
  - Everyday-reasoning screen: 11/20 by manual adjudication; the
66
  phrase-sensitive automatic scorer recorded 6/20.
67
 
68
- ## 2. Standard benchmark limits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69
 
70
  Zero-shot, greedy decoding with a 256-token generation cap. GSM8K uses
71
  strict final-number matching; HumanEval reports pass@1 from executed
@@ -80,7 +120,7 @@ These results are weak and define an important boundary: the narrower
80
  task-family results above do not transfer to broad GSM8K or HumanEval
81
  performance.
82
 
83
- ## 3. Supported prompt formats
84
 
85
  Evaluated decoding is greedy with repetition penalty 1.15, as pinned in
86
  `generation_config.json`.
@@ -100,7 +140,7 @@ Dialogue generation ends at `<|endoftext|>`. Prompting outside these
100
  formats, including paraphrases, was not systematically evaluated except
101
  for the published raw-completion spot check.
102
 
103
- ## 4. Quantized variants
104
 
105
  Q8_0 GGUF was evaluated separately under llama.cpp. That runtime and its
106
  stop handling differ from the bf16 harness, so the scores are not directly
@@ -116,7 +156,23 @@ comparable and differences cannot be attributed solely to quantization.
116
 
117
  Use bf16 when the bf16 profile is required.
118
 
119
- ## 5. Loading
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
120
 
121
  ```python
122
  import torch
@@ -148,7 +204,7 @@ print(answer)
148
  This is a text-only causal language model. Load it with
149
  `AutoModelForCausalLM` or `LlamaForCausalLM`, not a multimodal model class.
150
 
151
- ## 6. Intended use and safety
152
 
153
  This release is intended for research, constrained local inference,
154
  evaluation-harness development, and experiments whose prompts match the
@@ -159,7 +215,7 @@ financial decisions, autonomous action, or unsandboxed execution of
159
  generated code. Generated code may be incomplete or unsafe; inspect and
160
  test it in an isolated environment.
161
 
162
- ## 7. Known limitations
163
 
164
  - Not a general code model: complete programs scored 0/10, executable SQL
165
  1/10, and bash 1/5. The 69% result is for single functions on ConeML's
@@ -173,7 +229,7 @@ test it in an isolated environment.
173
  - Multi-step GSM-class reasoning remained weak.
174
  - Frameless prompting was spot-checked, not systematically certified.
175
 
176
- ## 8. Evidence and reproducibility
177
 
178
  The `eval/` directory publishes:
179
 
@@ -184,6 +240,9 @@ The `eval/` directory publishes:
184
  - separate Q8_0 aggregate results; and
185
  - `PRIVATE_EVIDENCE_SHA256SUMS.txt`: commitments to the complete
186
  row-level evidence retained by ConeML.
 
 
 
187
 
188
  Complete proprietary probe rows and generations are retained privately to
189
  avoid releasing the full measurement instruments. The hashes permit later
@@ -191,7 +250,7 @@ disclosure to be checked against the evidence frozen at release time.
191
  Public GSM8K and HumanEval rows are not redistributed; only aggregate
192
  results are published. `SHA256SUMS.txt` covers every shipped file.
193
 
194
- ## 9. License and commercial enquiries
195
 
196
  Released for non-commercial use under CC BY-NC 4.0. Commercial use is not
197
  granted by this release. See `LICENSE.md`.
 
65
  - Everyday-reasoning screen: 11/20 by manual adjudication; the
66
  phrase-sensitive automatic scorer recorded 6/20.
67
 
68
+ ## 2. Peer comparison and efficiency
69
+
70
+ The pair was tested against five post-trained peers on the same reduced
71
+ diagnostic screen, through each model's native instruction interface,
72
+ with greedy decoding, repetition penalty 1.15, and matched short-answer
73
+ budgets.
74
+
75
+ | model | parameters | mixed arithmetic (n=585) | executed functions (n=100) | designated refusals (n=17) |
76
+ |---|---:|---:|---:|---:|
77
+ | ConeML Alpha | 0.81B | 421 (72.0%) | 83 | **13** |
78
+ | ConeML Arithmetic | 0.81B | **442 (75.6%)** | 35 | 11 |
79
+ | Qwen3.5 | 0.8B | 154 (26.3%) | 93 | 0 |
80
+ | Qwen3 | 0.6B | 223 (38.1%) | **98** | 1 |
81
+ | Llama 3.2 Instruct | 1.24B | 343 (58.6%) | 79 | 1 |
82
+ | TinyLlama Chat | 1.1B | 113 (19.3%) | 51 | 0 |
83
+ | SmolLM2 Instruct | 1.7B | 380 (65.0%) | 96 | 1 |
84
+
85
+ This is not a general leaderboard: the task families match ConeML's
86
+ trained surfaces. On this screen, the ConeML pair led mixed arithmetic;
87
+ Alpha exceeded Llama 3.2 and TinyLlama on function writing but trailed
88
+ Qwen3.5, Qwen3, and SmolLM2. All models recorded zero over-refusals on
89
+ five in-scope contrasts.
90
+
91
+ Qwen3.5 with thinking enabled reached 451/585 (77.1%), compared with
92
+ ConeML Arithmetic's 442/585 (75.6%), while using at least 17.6 times the
93
+ generated-token budget per item and approximately 24 times the recorded
94
+ wall time. Runtime ratios are implementation-specific.
95
+
96
+ ConeML's selected base consumed 12.32B token positions—15.2 tokens per
97
+ parameter and approximately 5.99e19 training FLOPs under the `6ND`
98
+ convention. Pretraining took approximately 11 days on one RTX 5090.
99
+ Assuming, rather than claiming to have metered, a 0.70 kW average wall
100
+ draw gives 184.8 kWh; at CHF 0.14–0.30/kWh, that is approximately
101
+ CHF 26–55 of marginal pretraining electricity. This excludes hardware,
102
+ labor, SFT, evaluation, conversion, and emissions accounting.
103
+
104
+ The complete category table, peer wins, adverse base result, evaluated
105
+ revisions, assumptions, and frozen evidence commitments are in
106
+ [`eval/PEER_COMPARISON.md`](eval/PEER_COMPARISON.md).
107
+
108
+ ## 3. Standard benchmark limits
109
 
110
  Zero-shot, greedy decoding with a 256-token generation cap. GSM8K uses
111
  strict final-number matching; HumanEval reports pass@1 from executed
 
120
  task-family results above do not transfer to broad GSM8K or HumanEval
121
  performance.
122
 
123
+ ## 4. Supported prompt formats
124
 
125
  Evaluated decoding is greedy with repetition penalty 1.15, as pinned in
126
  `generation_config.json`.
 
140
  formats, including paraphrases, was not systematically evaluated except
141
  for the published raw-completion spot check.
142
 
143
+ ## 5. Quantized variants
144
 
145
  Q8_0 GGUF was evaluated separately under llama.cpp. That runtime and its
146
  stop handling differ from the bf16 harness, so the scores are not directly
 
156
 
157
  Use bf16 when the bf16 profile is required.
158
 
159
+ For local GGUF inference:
160
+
161
+ ```bash
162
+ llama-cli \
163
+ -m coneml-810m-alpha-Q8_0.gguf \
164
+ -p $'Question: What is 7 + 8?\nAnswer:' \
165
+ -n 32 --repeat-penalty 1.15 --no-conversation --single-turn
166
+ ```
167
+
168
+ For Ollama, download `Modelfile` beside the Q8_0 file, then run:
169
+
170
+ ```bash
171
+ ollama create coneml-810m-alpha -f Modelfile
172
+ ollama run coneml-810m-alpha "What is 7 + 8?"
173
+ ```
174
+
175
+ ## 6. Loading with Transformers
176
 
177
  ```python
178
  import torch
 
204
  This is a text-only causal language model. Load it with
205
  `AutoModelForCausalLM` or `LlamaForCausalLM`, not a multimodal model class.
206
 
207
+ ## 7. Intended use and safety
208
 
209
  This release is intended for research, constrained local inference,
210
  evaluation-harness development, and experiments whose prompts match the
 
215
  generated code. Generated code may be incomplete or unsafe; inspect and
216
  test it in an isolated environment.
217
 
218
+ ## 8. Known limitations
219
 
220
  - Not a general code model: complete programs scored 0/10, executable SQL
221
  1/10, and bash 1/5. The 69% result is for single functions on ConeML's
 
229
  - Multi-step GSM-class reasoning remained weak.
230
  - Frameless prompting was spot-checked, not systematically certified.
231
 
232
+ ## 9. Evidence and reproducibility
233
 
234
  The `eval/` directory publishes:
235
 
 
240
  - separate Q8_0 aggregate results; and
241
  - `PRIVATE_EVIDENCE_SHA256SUMS.txt`: commitments to the complete
242
  row-level evidence retained by ConeML.
243
+ - `peer-comparison-summary.json`, `PEER_COMPARISON.md`, and
244
+ `PEER_EVIDENCE_SHA256SUMS.txt`: the peer results, interpretation, and
245
+ commitments to the retained peer-generation evidence.
246
 
247
  Complete proprietary probe rows and generations are retained privately to
248
  avoid releasing the full measurement instruments. The hashes permit later
 
250
  Public GSM8K and HumanEval rows are not redistributed; only aggregate
251
  results are published. `SHA256SUMS.txt` covers every shipped file.
252
 
253
+ ## 10. License and commercial enquiries
254
 
255
  Released for non-commercial use under CC BY-NC 4.0. Commercial use is not
256
  granted by this release. See `LICENSE.md`.
SHA256SUMS.txt CHANGED
@@ -1,14 +1,18 @@
1
  846a8e2297580b3c8d57f69a07d88833d370f976010b31d9fc9b93e9a3f04464 LICENSE.md
2
- c16d4402a3123ce06c2684cebc34294c25e1b9463799de8020cd985e2d937c5a README.md
 
3
  fdf72939738a346a2134cd5ede1e9921d52d4d85ac0449249118322de48447e7 chat_template.jinja
4
  a09065a6d6c5ee5c518ed913085e878e7d3807f7d6cb8bcd1d6ce59653bd41b5 coneml-810m-alpha-Q8_0.gguf
5
  16835ec01c1fa344c5bd3788c7697ab67e68676ebae3fc374b62f7b30749a6bb coneml-810m-alpha-f16.gguf
6
  6cd7853f16b53fdb6bcc44438be39c33726379dc9848798fdcc81ba7f7555a58 config.json
7
  8a56ef50eebf7604d0408b52249591e258a31ff1c9ebf3785f748c0316637c56 conversion.json
8
  83b8f89b84387d8c2b22bc2b5d8c20a564fea9f79c7f6e6ab4fa9acee38040c5 eval/EVALUATION_METHODOLOGY.md
 
 
9
  b2414d6dddbac7128724d27a7ce54e2fa3586943ece7b94e6c3c5188a1f04484 eval/PRIVATE_EVIDENCE_SHA256SUMS.txt
10
  0108c09e9b835a0da81c5093b40f0b7937b1553de3abdcb5c207c75888714db2 eval/certification-Q8_0.json
11
  b4cc4f98ccd8ba91206e4446c38f6b1e5f6e7a07f3c9ec963ee16d6a0619dc71 eval/code-cert-Q8_0.json
 
12
  bff2af0c69ec8cce87cd93a69c22045b4062588e60f1dafb4ba0b85ac68f2c8d eval/representative-samples.json
13
  f6ce9e02abad733c0d5ae9fdaa58f290441a24c4f24c7673c4f50ca5150e1d70 eval/summary.json
14
  2e2ddba5318c74a4fcc31cfdfdc6e067564176b6404da484932f95a7fdc9712a generation_config.json
 
1
  846a8e2297580b3c8d57f69a07d88833d370f976010b31d9fc9b93e9a3f04464 LICENSE.md
2
+ dae2b5a811b8d4fd08801460bc80968d8792391bec75a235b6e79d0ba2fd952c Modelfile
3
+ 86ad0bc96b23050d854c8c1ef6353aa1ecc36581f21038b8e418a54de3c37483 README.md
4
  fdf72939738a346a2134cd5ede1e9921d52d4d85ac0449249118322de48447e7 chat_template.jinja
5
  a09065a6d6c5ee5c518ed913085e878e7d3807f7d6cb8bcd1d6ce59653bd41b5 coneml-810m-alpha-Q8_0.gguf
6
  16835ec01c1fa344c5bd3788c7697ab67e68676ebae3fc374b62f7b30749a6bb coneml-810m-alpha-f16.gguf
7
  6cd7853f16b53fdb6bcc44438be39c33726379dc9848798fdcc81ba7f7555a58 config.json
8
  8a56ef50eebf7604d0408b52249591e258a31ff1c9ebf3785f748c0316637c56 conversion.json
9
  83b8f89b84387d8c2b22bc2b5d8c20a564fea9f79c7f6e6ab4fa9acee38040c5 eval/EVALUATION_METHODOLOGY.md
10
+ 3817126000edea9bc2fcb75a783fa73de234590cdfe89e1aea9b569bae809caa eval/PEER_COMPARISON.md
11
+ 73d8100a4b78d7693aabb7a22f3b9948774f7a418d9fa162d19d6ff630baffd4 eval/PEER_EVIDENCE_SHA256SUMS.txt
12
  b2414d6dddbac7128724d27a7ce54e2fa3586943ece7b94e6c3c5188a1f04484 eval/PRIVATE_EVIDENCE_SHA256SUMS.txt
13
  0108c09e9b835a0da81c5093b40f0b7937b1553de3abdcb5c207c75888714db2 eval/certification-Q8_0.json
14
  b4cc4f98ccd8ba91206e4446c38f6b1e5f6e7a07f3c9ec963ee16d6a0619dc71 eval/code-cert-Q8_0.json
15
+ baff252791bdf43653763a53a57024d0075bb9a1f6b35cf1fcc94a9191cc73df eval/peer-comparison-summary.json
16
  bff2af0c69ec8cce87cd93a69c22045b4062588e60f1dafb4ba0b85ac68f2c8d eval/representative-samples.json
17
  f6ce9e02abad733c0d5ae9fdaa58f290441a24c4f24c7673c4f50ca5150e1d70 eval/summary.json
18
  2e2ddba5318c74a4fcc31cfdfdc6e067564176b6404da484932f95a7fdc9712a generation_config.json
eval/PEER_COMPARISON.md ADDED
@@ -0,0 +1,128 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ConeML 810M peer diagnostic and resource context
2
+
3
+ Date: 2026-07-30
4
+
5
+ This is an interface-specific diagnostic screen, not a neutral ranking of
6
+ general model capability. Each instruction-tuned model was evaluated
7
+ through its own instruction interface. All models used greedy decoding
8
+ with repetition penalty 1.15. Arithmetic and function-writing used
9
+ matched short-answer generation budgets.
10
+
11
+ The task families match ConeML's supervised training surfaces. That gives
12
+ the comparison practical value for locating the release's envelope, but
13
+ it also means these results must not be generalized to unrelated tasks.
14
+ Standard GSM8K and HumanEval results remain separately disclosed in the
15
+ model cards.
16
+
17
+ ## Instruct-model results
18
+
19
+ | measured surface | ConeML Alpha | ConeML Arithmetic | Qwen3.5 0.8B | Qwen3 0.6B | Llama 3.2 1B Instruct | TinyLlama Chat | SmolLM2 1.7B Instruct |
20
+ |---|---:|---:|---:|---:|---:|---:|---:|
21
+ | mixed arithmetic screen (n=585) | 421 (72.0%) | **442 (75.6%)** | 154 (26.3%) | 223 (38.1%) | 343 (58.6%) | 113 (19.3%) | 380 (65.0%) |
22
+ | four core arithmetic lanes (n=225) | 177 (78.7%) | **221 (98.2%)** | 150 (66.7%) | 208 (92.4%) | **221 (98.2%)** | 92 (40.9%) | **221 (98.2%)** |
23
+ | executed single functions (n=100) | 83 | 35 | 93 | **98** | 79 | 51 | 96 |
24
+ | designated-refusal prompts (n=17) | **13** | 11 | 0 | 1 | 1 | 0 | 1 |
25
+ | over-refusals on contrasts (n=5) | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
26
+
27
+ The honest result is mixed:
28
+
29
+ - At matched short-answer budgets, the ConeML pair led this post-trained
30
+ peer group on the broader arithmetic screen.
31
+ - On the four core arithmetic lanes, the Arithmetic variant tied Llama
32
+ 3.2 Instruct and SmolLM2 at this reduced sample size. Its separate
33
+ full-size internal result is 1,093/1,116 (97.9%).
34
+ - Alpha exceeded Llama 3.2 Instruct and TinyLlama Chat on the executed
35
+ function screen, but Qwen3.5, Qwen3, and SmolLM2 scored higher.
36
+ - The refusal row measures a trained response policy on designated
37
+ prompts, not factual correctness or general epistemic calibration.
38
+
39
+ The peer-harness refusal results above differ from the dedicated ConeML
40
+ probe's 17/17 for each release because the generation loops differ. Both
41
+ measurements are reported rather than merged.
42
+
43
+ ## Qwen3.5 thinking-mode sensitivity
44
+
45
+ Qwen3.5 0.8B was also evaluated with thinking enabled on the same 585
46
+ arithmetic items:
47
+
48
+ | metric | Qwen3.5 thinking | Qwen3.5 short-answer | ConeML Arithmetic short-answer |
49
+ |---|---:|---:|---:|
50
+ | accuracy | 451/585 (77.1%) | 154/585 (26.3%) | 442/585 (75.6%) |
51
+ | generated tokens per item | mean 844; median 699 | at most 48 | at most 48 |
52
+ | p95 / generation cap | 1,536 / 1,536 | 48 / 48 | 48 / 48 |
53
+ | truncation rate | 138/585 (23.6%) | approximately 0% | approximately 0% |
54
+ | mean wall time per item | 2.43 s | approximately 0.1 s | approximately 0.1 s |
55
+ | correct answers per 1,000 generated tokens | 0.91 | 5.5 | approximately 15.7 or higher |
56
+
57
+ Thinking mode recovered Qwen3.5 to statistical parity with ConeML
58
+ Arithmetic on this screen, while using at least 17.6 times the
59
+ per-answer generation budget and approximately 24 times the measured wall
60
+ time. Wall-time ratios are specific to the recorded hardware, batching,
61
+ and implementation. This is an inference-cost comparison, not a claim
62
+ that thinking mode is intrinsically inferior.
63
+
64
+ ## Against-interest base result
65
+
66
+ On the same 585 arithmetic items, through a task frame native to neither
67
+ base model, Qwen3.5 0.8B Base scored 492/585 (84.1%), while ConeML base
68
+ checkpoint 188 scored 156/585 (26.7%). This result is included because it
69
+ prevents an absolute-superiority reading: high-exposure base models may
70
+ already contain strong task-formatted behavior, and post-training can
71
+ move capability between output surfaces.
72
+
73
+ ## Training-resource context
74
+
75
+ ConeML's selected base consumed approximately 12.32B token positions:
76
+ 15.2 tokens per parameter and approximately 5.99e19 training FLOPs under
77
+ the `6 × parameters × tokens` convention.
78
+
79
+ | model family | disclosed pretraining tokens | approximate tokens/parameter | approximate training FLOPs vs ConeML |
80
+ |---|---:|---:|---:|
81
+ | ConeML 810M | 12.32B | 15.2 | 1× |
82
+ | TinyLlama 1.1B | 3T | 2,727 | 330× |
83
+ | Llama 3.2 1B | up to 9T | 7,258 | 1,118×, plus distillation |
84
+ | SmolLM2 1.7B | 11T | 6,471 | 1,873× |
85
+ | Qwen3 0.6B | 36T | 60,000 | 2,164× |
86
+ | Qwen3.5 0.8B | not disclosed | not stated | not stated |
87
+
88
+ These are estimated pretraining FLOP ratios, not historical electricity
89
+ or monetary costs for the peer models.
90
+
91
+ ConeML pretraining took approximately 11 days on one local RTX 5090. At
92
+ an explicitly assumed average wall draw of 0.70 kW, that corresponds to
93
+ 184.8 kWh. Applying an assumed Swiss residential tariff range of
94
+ CHF 0.14–0.30/kWh gives approximately CHF 26–55 of marginal pretraining
95
+ electricity.
96
+
97
+ | ConeML pretraining unit | estimate |
98
+ |---|---:|
99
+ | average throughput | 12,963 token positions/s |
100
+ | wall time per billion token positions | 21.4 h |
101
+ | energy per billion token positions | 15.0 kWh |
102
+ | energy per million token positions | 15 Wh |
103
+ | estimated wall energy per token position | 0.054 J |
104
+ | marginal electricity per billion token positions | CHF 2.10–4.50 |
105
+
106
+ The energy and cost figures are estimates derived from the stated power
107
+ and tariff assumptions, not meter readings. They exclude hardware,
108
+ depreciation, labor, supervised fine-tuning, evaluation, conversion,
109
+ datacenter PUE, and carbon intensity. No emissions claim is made.
110
+
111
+ ## Evaluated revisions
112
+
113
+ - `Qwen/Qwen3.5-0.8B@2fc06364715b967f1860aea9cf38778875588b17`
114
+ - `Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca`
115
+ - `unsloth/Llama-3.2-1B-Instruct@5a8abab4a5d6f164389b1079fb721cfab8d7126c`
116
+ - `TinyLlama/TinyLlama-1.1B-Chat-v1.0@fe8a4ea1ffedaf415f4da2f062534de366a451e6`
117
+ - `HuggingFaceTB/SmolLM2-1.7B-Instruct@31b70e2e869a7173562077fd711b654946d38674`
118
+
119
+ The exact aggregates are in `peer-comparison-summary.json`. Complete
120
+ generation rows are retained privately; their frozen SHA-256 commitments
121
+ are published in `PEER_EVIDENCE_SHA256SUMS.txt`.
122
+
123
+ ## Primary resource disclosures
124
+
125
+ - Qwen3 pretraining: [Qwen3 release post](https://qwenlm.github.io/blog/qwen3/)
126
+ - Llama 3.2 token count and distillation: [Meta Llama 3.2 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md)
127
+ - TinyLlama token count: [TinyLlama model card](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)
128
+ - SmolLM2 token count: [Hugging Face SmolLM2 model card](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)
eval/PEER_EVIDENCE_SHA256SUMS.txt ADDED
@@ -0,0 +1,31 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 96a805aed179f77d9f1959216f9a39e8bf8d50dea5465eeea1eb24e96502a6af SOURCE_PEER_COMPARISON_V2.md
2
+ a5d6cd4d7a2643b316cefb1dcb6f43296bb69da4064dbc3b3ab4a123bf36b2b1 base-reference/coneml-base188.json
3
+ 83643ce110655e708d50f4eb3ee29ab08ab8e003768eb5b5da9a3e545bc9b8ef base-reference/coneml-base188.rows.jsonl
4
+ ca387426c55ac72d50041e2f082bd4cdc747eedb51deca7929cf7a4acea2c5b6 base-reference/coneml-native-transitive.json
5
+ 43ef584d37c6a276b190aeb741597b54bfebbf7bbd9c37df5d82a2e15a8ee2d1 base-reference/qwen35-0.8b-base.json
6
+ 25de33180d4073777d29871882b57f1556733e9a8c1e10eba14a8cac062b5c35 base-reference/qwen35-0.8b-base.rows.jsonl
7
+ 2003c73a54602f3a30980d6c7f85feef39b63fd11cef9fc20a90044d3569ba77 instruct/coneml-810m-alpha.json
8
+ 8da107545bd6717c7795ff4e827698e5e269d451f2390f24dccc516fc390ab44 instruct/coneml-810m-alpha.rows.jsonl
9
+ 77d514c14e14557c4ac30e8a8c3f4c5887f0db0dff39b8ea14baea164327d027 instruct/coneml-810m-arithmetic.json
10
+ a60ed3e533c0c4a77b34a21f80b8df8de85e3df074f3cacdc3bbd4b5e0741cce instruct/coneml-810m-arithmetic.rows.jsonl
11
+ aa35b93f46778ffb22351ac411f2fc04e645775e416b5a9423f925bb6d6b1609 instruct/llama32-1b-instruct-transchat.json
12
+ ee514c35d86f27172a004325b26da915f869164adae59edf8d95b01d958e683f instruct/llama32-1b-instruct-transchat.rows.jsonl
13
+ f4553ef34844e55364a87f631fac03a3545acbb14cd12dcf182f16151b5e5f8a instruct/llama32-1b-instruct.json
14
+ d5c0fd8b00b50754ac7dede96081afeaf3143795e005134cf5543483859340bf instruct/llama32-1b-instruct.rows.jsonl
15
+ a61b56681810410dfc3a8d8e758b8ccd11d01ebcce2d963d6bdbca4644c2cee9 instruct/qwen3-0.6b-transchat.json
16
+ f7c763441edc0d5e37a34902a3cc4efcacc6a35b47056adb9356879a4407cb31 instruct/qwen3-0.6b-transchat.rows.jsonl
17
+ 3f34342d79095a5754512ebf57192fdf98a16b364a250712993ac1da0b65e252 instruct/qwen3-0.6b.json
18
+ 05847f4187651780008331a85c9cd5fa84d3b636320c9a809fb3c307f1a230d6 instruct/qwen3-0.6b.rows.jsonl
19
+ 21984e00a4a6da3d2887138ede25c03e9007fee0be41de462ef58dc58db89554 instruct/qwen35-0.8b-transchat.json
20
+ 48a53b6bf3829c207953b1075160df9f6702b1fc2162426577aa69530856e555 instruct/qwen35-0.8b-transchat.rows.jsonl
21
+ fa11fbf67c047ac541ecd38571867f9fe68bfb8aec12b84f2addda8a6d561642 instruct/qwen35-0.8b.json
22
+ 74a6debf6ae55db9895f556e654b6905fde871358a94c9c4a2730a418b0e2608 instruct/qwen35-0.8b.rows.jsonl
23
+ 72598668bfce72497ac67a4d83d1cad7538c63ec933e8a1d1198876d85dc8a67 instruct/qwen35-thinking-sensitivity.json
24
+ 8917677a4c9d29124c5ade229f7080bbd0ccb83471c3d7d39a6f958224596a8f instruct/smollm2-1.7b-instruct-transchat.json
25
+ deb9eb805475e4ad2d158283b469830138fdf42ee0c9a903bf3611b43a7869ff instruct/smollm2-1.7b-instruct-transchat.rows.jsonl
26
+ 94a3440cdaae14eb27beaf36a1a0103dab4a6d292bc382981f0514a82a93ed0f instruct/smollm2-1.7b-instruct.json
27
+ 1b06370ad7b940ed05b59ffe4acd19035341cfe078ed7b01d59d168abc8d0c71 instruct/smollm2-1.7b-instruct.rows.jsonl
28
+ bdd7cf2867643439e90a3e04cb209e6130839af9422e492bcd2eb1c2cba4556e instruct/tinyllama-chat-transchat.json
29
+ 472aad0c169c4f943fbc9a8367d65fa2d5c41acbbee66d099e16a286f8676d1c instruct/tinyllama-chat-transchat.rows.jsonl
30
+ 3a7c2e7ebcaa78b4b7d9b6d84a5b90a495f260f97287bad3282d54fc84e0c009 instruct/tinyllama-chat.json
31
+ 5926a31e926279f8d9459d19cbbe9730a06071bb8b4029c2f9174804ab447943 instruct/tinyllama-chat.rows.jsonl
eval/peer-comparison-summary.json ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "date": "2026-07-30",
4
+ "scope": "Interface-specific diagnostic screen, not a neutral ranking of general model capability. Each instruct model used its native instruction interface. Arithmetic and function-writing used matched short-answer generation budgets.",
5
+ "decoding": {
6
+ "strategy": "greedy",
7
+ "repetition_penalty": 1.15,
8
+ "arithmetic_n": 585,
9
+ "core_arithmetic_n": 225,
10
+ "function_writing_n": 100,
11
+ "designated_refusal_n": 17,
12
+ "refusal_contrast_n": 5
13
+ },
14
+ "evaluated_revisions": {
15
+ "ConeML/coneml-810m-alpha": "local release artifact e5df8f2edd773ad079ba00d383ace81b96cbb9579d64c570db505f9efc032b76",
16
+ "ConeML/coneml-810m-alpha-arithmetic": "local release artifact e1de27f5f91ac43130ce2a57a4bf9a036b84b2e5b589789b5981ecf79489e0d1",
17
+ "Qwen/Qwen3.5-0.8B": "2fc06364715b967f1860aea9cf38778875588b17",
18
+ "Qwen/Qwen3-0.6B": "c1899de289a04d12100db370d81485cdf75e47ca",
19
+ "unsloth/Llama-3.2-1B-Instruct": "5a8abab4a5d6f164389b1079fb721cfab8d7126c",
20
+ "TinyLlama/TinyLlama-1.1B-Chat-v1.0": "fe8a4ea1ffedaf415f4da2f062534de366a451e6",
21
+ "HuggingFaceTB/SmolLM2-1.7B-Instruct": "31b70e2e869a7173562077fd711b654946d38674"
22
+ },
23
+ "instruct_results": [
24
+ {
25
+ "model": "ConeML/coneml-810m-alpha",
26
+ "parameters_billions": 0.8101,
27
+ "arithmetic_mixed": {"correct": 421, "n": 585, "accuracy": 0.7197},
28
+ "arithmetic_core_four_lanes": {"correct": 177, "n": 225, "accuracy": 0.7867},
29
+ "executed_functions": {"correct": 83, "n": 100, "accuracy": 0.83},
30
+ "designated_refusals": {"correct": 13, "n": 17, "accuracy": 0.7647},
31
+ "over_refusals": {"count": 0, "n": 5}
32
+ },
33
+ {
34
+ "model": "ConeML/coneml-810m-alpha-arithmetic",
35
+ "parameters_billions": 0.8101,
36
+ "arithmetic_mixed": {"correct": 442, "n": 585, "accuracy": 0.7556},
37
+ "arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
38
+ "executed_functions": {"correct": 35, "n": 100, "accuracy": 0.35},
39
+ "designated_refusals": {"correct": 11, "n": 17, "accuracy": 0.6471},
40
+ "over_refusals": {"count": 0, "n": 5}
41
+ },
42
+ {
43
+ "model": "Qwen/Qwen3.5-0.8B",
44
+ "parameters_billions": 0.8,
45
+ "mode": "thinking disabled for matched short-answer budget",
46
+ "arithmetic_mixed": {"correct": 154, "n": 585, "accuracy": 0.2632},
47
+ "arithmetic_core_four_lanes": {"correct": 150, "n": 225, "accuracy": 0.6667},
48
+ "executed_functions": {"correct": 93, "n": 100, "accuracy": 0.93},
49
+ "designated_refusals": {"correct": 0, "n": 17, "accuracy": 0.0},
50
+ "over_refusals": {"count": 0, "n": 5}
51
+ },
52
+ {
53
+ "model": "Qwen/Qwen3-0.6B",
54
+ "parameters_billions": 0.6,
55
+ "arithmetic_mixed": {"correct": 223, "n": 585, "accuracy": 0.3812},
56
+ "arithmetic_core_four_lanes": {"correct": 208, "n": 225, "accuracy": 0.9244},
57
+ "executed_functions": {"correct": 98, "n": 100, "accuracy": 0.98},
58
+ "designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
59
+ "over_refusals": {"count": 0, "n": 5}
60
+ },
61
+ {
62
+ "model": "unsloth/Llama-3.2-1B-Instruct",
63
+ "parameters_billions": 1.24,
64
+ "arithmetic_mixed": {"correct": 343, "n": 585, "accuracy": 0.5863},
65
+ "arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
66
+ "executed_functions": {"correct": 79, "n": 100, "accuracy": 0.79},
67
+ "designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
68
+ "over_refusals": {"count": 0, "n": 5}
69
+ },
70
+ {
71
+ "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
72
+ "parameters_billions": 1.1,
73
+ "arithmetic_mixed": {"correct": 113, "n": 585, "accuracy": 0.1932},
74
+ "arithmetic_core_four_lanes": {"correct": 92, "n": 225, "accuracy": 0.4089},
75
+ "executed_functions": {"correct": 51, "n": 100, "accuracy": 0.51},
76
+ "designated_refusals": {"correct": 0, "n": 17, "accuracy": 0.0},
77
+ "over_refusals": {"count": 0, "n": 5}
78
+ },
79
+ {
80
+ "model": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
81
+ "parameters_billions": 1.7,
82
+ "arithmetic_mixed": {"correct": 380, "n": 585, "accuracy": 0.6496},
83
+ "arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
84
+ "executed_functions": {"correct": 96, "n": 100, "accuracy": 0.96},
85
+ "designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
86
+ "over_refusals": {"count": 0, "n": 5}
87
+ }
88
+ ],
89
+ "qwen35_thinking_sensitivity": {
90
+ "model": "Qwen/Qwen3.5-0.8B",
91
+ "mode": "thinking enabled",
92
+ "arithmetic_mixed": {"correct": 451, "n": 585, "accuracy": 0.771},
93
+ "generated_tokens": {"mean": 844, "median": 699, "p95": 1536},
94
+ "generation_cap": 1536,
95
+ "truncation": {"count": 138, "n": 585, "rate": 0.2359},
96
+ "wall_seconds_per_item_mean": 2.43,
97
+ "correct_per_1000_generated_tokens": 0.913,
98
+ "comparison": "Qwen thinking reached parity with ConeML Arithmetic's 442/585 while using at least 17.6 times the generated-token budget per item. The recorded wall-time ratio was approximately 24 times. Runtime measurements are hardware- and implementation-specific."
99
+ },
100
+ "against_interest_base_result": {
101
+ "scope": "Same arithmetic items through a task frame not native to either base model.",
102
+ "ConeML_base_188": {"correct": 156, "n": 585, "accuracy": 0.2667},
103
+ "Qwen3.5_0.8B_Base": {"correct": 492, "n": 585, "accuracy": 0.841},
104
+ "interpretation": "The peer base result prevents an absolute-superiority interpretation and illustrates sensitivity to training exposure and interface."
105
+ },
106
+ "training_resource_context": {
107
+ "ConeML": {
108
+ "pretraining_token_positions": 12320000000,
109
+ "tokens_per_parameter": 15.2,
110
+ "estimated_training_flops_6ND": 5.99e19,
111
+ "pretraining_wall_time_days": 11,
112
+ "hardware": "one NVIDIA RTX 5090",
113
+ "power_assumption_kw_average_wall": 0.70,
114
+ "estimated_pretraining_energy_kwh": 184.8,
115
+ "assumed_swiss_residential_tariff_chf_per_kwh": [0.14, 0.30],
116
+ "estimated_marginal_pretraining_electricity_chf": [25.87, 55.44],
117
+ "exclusions": ["hardware", "depreciation", "labor", "SFT", "evaluation", "conversion", "grid-carbon estimate"]
118
+ },
119
+ "peer_compute_context": [
120
+ {"model": "TinyLlama 1.1B", "published_pretraining_tokens": 3000000000000, "approx_tokens_per_parameter": 2727, "approx_flops_vs_coneml": 330},
121
+ {"model": "Llama 3.2 1B", "published_pretraining_tokens": 9000000000000, "approx_tokens_per_parameter": 7258, "approx_flops_vs_coneml": 1118, "note": "plus distillation"},
122
+ {"model": "SmolLM2 1.7B", "published_pretraining_tokens": 11000000000000, "approx_tokens_per_parameter": 6471, "approx_flops_vs_coneml": 1873},
123
+ {"model": "Qwen3 0.6B", "published_pretraining_tokens": 36000000000000, "approx_tokens_per_parameter": 60000, "approx_flops_vs_coneml": 2164},
124
+ {"model": "Qwen3.5 0.8B", "published_pretraining_tokens": null, "approx_tokens_per_parameter": null, "approx_flops_vs_coneml": null}
125
+ ]
126
+ },
127
+ "evidence": {
128
+ "public_interpretation": "PEER_COMPARISON.md",
129
+ "private_rows": "retained by ConeML",
130
+ "hash_commitment": "PEER_EVIDENCE_SHA256SUMS.txt"
131
+ }
132
+ }