Text Generation
Transformers
Safetensors
GGUF
English
llama
conversational
text-generation-inference
Instructions to use ConeML/coneml-810m-alpha with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ConeML/coneml-810m-alpha with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="ConeML/coneml-810m-alpha") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("ConeML/coneml-810m-alpha") model = AutoModelForCausalLM.from_pretrained("ConeML/coneml-810m-alpha", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ConeML/coneml-810m-alpha with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ConeML/coneml-810m-alpha:Q8_0 # Run inference directly in the terminal: llama cli -hf ConeML/coneml-810m-alpha:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ConeML/coneml-810m-alpha:Q8_0 # Run inference directly in the terminal: llama cli -hf ConeML/coneml-810m-alpha:Q8_0
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf ConeML/coneml-810m-alpha:Q8_0 # Run inference directly in the terminal: ./llama-cli -hf ConeML/coneml-810m-alpha:Q8_0
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf ConeML/coneml-810m-alpha:Q8_0 # Run inference directly in the terminal: ./build/bin/llama-cli -hf ConeML/coneml-810m-alpha:Q8_0
Use Docker
docker model run hf.co/ConeML/coneml-810m-alpha:Q8_0
- LM Studio
- Jan
- vLLM
How to use ConeML/coneml-810m-alpha with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ConeML/coneml-810m-alpha" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ConeML/coneml-810m-alpha", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ConeML/coneml-810m-alpha:Q8_0
- SGLang
How to use ConeML/coneml-810m-alpha with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ConeML/coneml-810m-alpha" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ConeML/coneml-810m-alpha", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ConeML/coneml-810m-alpha" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ConeML/coneml-810m-alpha", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use ConeML/coneml-810m-alpha with Ollama:
ollama run hf.co/ConeML/coneml-810m-alpha:Q8_0
- Unsloth Studio
How to use ConeML/coneml-810m-alpha with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for ConeML/coneml-810m-alpha to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for ConeML/coneml-810m-alpha to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for ConeML/coneml-810m-alpha to start chatting
- Docker Model Runner
How to use ConeML/coneml-810m-alpha with Docker Model Runner:
docker model run hf.co/ConeML/coneml-810m-alpha:Q8_0
- Lemonade
How to use ConeML/coneml-810m-alpha with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ConeML/coneml-810m-alpha:Q8_0
Run and chat with the model
lemonade run user.coneml-810m-alpha-Q8_0
List all available models
lemonade list
- Atomic Chat
Restore verified peer comparison, efficiency context, and Ollama support
Browse files- Modelfile +16 -0
- README.md +67 -8
- SHA256SUMS.txt +5 -1
- eval/PEER_COMPARISON.md +128 -0
- eval/PEER_EVIDENCE_SHA256SUMS.txt +31 -0
- eval/peer-comparison-summary.json +132 -0
Modelfile
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
FROM ./coneml-810m-alpha-Q8_0.gguf
|
| 2 |
+
|
| 3 |
+
TEMPLATE """{{- range .Messages }}
|
| 4 |
+
{{- if eq .Role "system" }}System:
|
| 5 |
+
{{ .Content }}
|
| 6 |
+
{{- else if eq .Role "user" }}User:
|
| 7 |
+
{{ .Content }}
|
| 8 |
+
{{- else if eq .Role "assistant" }}Assistant:
|
| 9 |
+
{{ .Content }}<|endoftext|>
|
| 10 |
+
{{- end }}
|
| 11 |
+
{{- end }}Assistant:
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
PARAMETER num_ctx 8192
|
| 15 |
+
PARAMETER repeat_penalty 1.15
|
| 16 |
+
PARAMETER stop "<|endoftext|>"
|
README.md
CHANGED
|
@@ -65,7 +65,47 @@ Behavioral screens:
|
|
| 65 |
- Everyday-reasoning screen: 11/20 by manual adjudication; the
|
| 66 |
phrase-sensitive automatic scorer recorded 6/20.
|
| 67 |
|
| 68 |
-
## 2.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
|
| 70 |
Zero-shot, greedy decoding with a 256-token generation cap. GSM8K uses
|
| 71 |
strict final-number matching; HumanEval reports pass@1 from executed
|
|
@@ -80,7 +120,7 @@ These results are weak and define an important boundary: the narrower
|
|
| 80 |
task-family results above do not transfer to broad GSM8K or HumanEval
|
| 81 |
performance.
|
| 82 |
|
| 83 |
-
##
|
| 84 |
|
| 85 |
Evaluated decoding is greedy with repetition penalty 1.15, as pinned in
|
| 86 |
`generation_config.json`.
|
|
@@ -100,7 +140,7 @@ Dialogue generation ends at `<|endoftext|>`. Prompting outside these
|
|
| 100 |
formats, including paraphrases, was not systematically evaluated except
|
| 101 |
for the published raw-completion spot check.
|
| 102 |
|
| 103 |
-
##
|
| 104 |
|
| 105 |
Q8_0 GGUF was evaluated separately under llama.cpp. That runtime and its
|
| 106 |
stop handling differ from the bf16 harness, so the scores are not directly
|
|
@@ -116,7 +156,23 @@ comparable and differences cannot be attributed solely to quantization.
|
|
| 116 |
|
| 117 |
Use bf16 when the bf16 profile is required.
|
| 118 |
|
| 119 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
|
| 121 |
```python
|
| 122 |
import torch
|
|
@@ -148,7 +204,7 @@ print(answer)
|
|
| 148 |
This is a text-only causal language model. Load it with
|
| 149 |
`AutoModelForCausalLM` or `LlamaForCausalLM`, not a multimodal model class.
|
| 150 |
|
| 151 |
-
##
|
| 152 |
|
| 153 |
This release is intended for research, constrained local inference,
|
| 154 |
evaluation-harness development, and experiments whose prompts match the
|
|
@@ -159,7 +215,7 @@ financial decisions, autonomous action, or unsandboxed execution of
|
|
| 159 |
generated code. Generated code may be incomplete or unsafe; inspect and
|
| 160 |
test it in an isolated environment.
|
| 161 |
|
| 162 |
-
##
|
| 163 |
|
| 164 |
- Not a general code model: complete programs scored 0/10, executable SQL
|
| 165 |
1/10, and bash 1/5. The 69% result is for single functions on ConeML's
|
|
@@ -173,7 +229,7 @@ test it in an isolated environment.
|
|
| 173 |
- Multi-step GSM-class reasoning remained weak.
|
| 174 |
- Frameless prompting was spot-checked, not systematically certified.
|
| 175 |
|
| 176 |
-
##
|
| 177 |
|
| 178 |
The `eval/` directory publishes:
|
| 179 |
|
|
@@ -184,6 +240,9 @@ The `eval/` directory publishes:
|
|
| 184 |
- separate Q8_0 aggregate results; and
|
| 185 |
- `PRIVATE_EVIDENCE_SHA256SUMS.txt`: commitments to the complete
|
| 186 |
row-level evidence retained by ConeML.
|
|
|
|
|
|
|
|
|
|
| 187 |
|
| 188 |
Complete proprietary probe rows and generations are retained privately to
|
| 189 |
avoid releasing the full measurement instruments. The hashes permit later
|
|
@@ -191,7 +250,7 @@ disclosure to be checked against the evidence frozen at release time.
|
|
| 191 |
Public GSM8K and HumanEval rows are not redistributed; only aggregate
|
| 192 |
results are published. `SHA256SUMS.txt` covers every shipped file.
|
| 193 |
|
| 194 |
-
##
|
| 195 |
|
| 196 |
Released for non-commercial use under CC BY-NC 4.0. Commercial use is not
|
| 197 |
granted by this release. See `LICENSE.md`.
|
|
|
|
| 65 |
- Everyday-reasoning screen: 11/20 by manual adjudication; the
|
| 66 |
phrase-sensitive automatic scorer recorded 6/20.
|
| 67 |
|
| 68 |
+
## 2. Peer comparison and efficiency
|
| 69 |
+
|
| 70 |
+
The pair was tested against five post-trained peers on the same reduced
|
| 71 |
+
diagnostic screen, through each model's native instruction interface,
|
| 72 |
+
with greedy decoding, repetition penalty 1.15, and matched short-answer
|
| 73 |
+
budgets.
|
| 74 |
+
|
| 75 |
+
| model | parameters | mixed arithmetic (n=585) | executed functions (n=100) | designated refusals (n=17) |
|
| 76 |
+
|---|---:|---:|---:|---:|
|
| 77 |
+
| ConeML Alpha | 0.81B | 421 (72.0%) | 83 | **13** |
|
| 78 |
+
| ConeML Arithmetic | 0.81B | **442 (75.6%)** | 35 | 11 |
|
| 79 |
+
| Qwen3.5 | 0.8B | 154 (26.3%) | 93 | 0 |
|
| 80 |
+
| Qwen3 | 0.6B | 223 (38.1%) | **98** | 1 |
|
| 81 |
+
| Llama 3.2 Instruct | 1.24B | 343 (58.6%) | 79 | 1 |
|
| 82 |
+
| TinyLlama Chat | 1.1B | 113 (19.3%) | 51 | 0 |
|
| 83 |
+
| SmolLM2 Instruct | 1.7B | 380 (65.0%) | 96 | 1 |
|
| 84 |
+
|
| 85 |
+
This is not a general leaderboard: the task families match ConeML's
|
| 86 |
+
trained surfaces. On this screen, the ConeML pair led mixed arithmetic;
|
| 87 |
+
Alpha exceeded Llama 3.2 and TinyLlama on function writing but trailed
|
| 88 |
+
Qwen3.5, Qwen3, and SmolLM2. All models recorded zero over-refusals on
|
| 89 |
+
five in-scope contrasts.
|
| 90 |
+
|
| 91 |
+
Qwen3.5 with thinking enabled reached 451/585 (77.1%), compared with
|
| 92 |
+
ConeML Arithmetic's 442/585 (75.6%), while using at least 17.6 times the
|
| 93 |
+
generated-token budget per item and approximately 24 times the recorded
|
| 94 |
+
wall time. Runtime ratios are implementation-specific.
|
| 95 |
+
|
| 96 |
+
ConeML's selected base consumed 12.32B token positions—15.2 tokens per
|
| 97 |
+
parameter and approximately 5.99e19 training FLOPs under the `6ND`
|
| 98 |
+
convention. Pretraining took approximately 11 days on one RTX 5090.
|
| 99 |
+
Assuming, rather than claiming to have metered, a 0.70 kW average wall
|
| 100 |
+
draw gives 184.8 kWh; at CHF 0.14–0.30/kWh, that is approximately
|
| 101 |
+
CHF 26–55 of marginal pretraining electricity. This excludes hardware,
|
| 102 |
+
labor, SFT, evaluation, conversion, and emissions accounting.
|
| 103 |
+
|
| 104 |
+
The complete category table, peer wins, adverse base result, evaluated
|
| 105 |
+
revisions, assumptions, and frozen evidence commitments are in
|
| 106 |
+
[`eval/PEER_COMPARISON.md`](eval/PEER_COMPARISON.md).
|
| 107 |
+
|
| 108 |
+
## 3. Standard benchmark limits
|
| 109 |
|
| 110 |
Zero-shot, greedy decoding with a 256-token generation cap. GSM8K uses
|
| 111 |
strict final-number matching; HumanEval reports pass@1 from executed
|
|
|
|
| 120 |
task-family results above do not transfer to broad GSM8K or HumanEval
|
| 121 |
performance.
|
| 122 |
|
| 123 |
+
## 4. Supported prompt formats
|
| 124 |
|
| 125 |
Evaluated decoding is greedy with repetition penalty 1.15, as pinned in
|
| 126 |
`generation_config.json`.
|
|
|
|
| 140 |
formats, including paraphrases, was not systematically evaluated except
|
| 141 |
for the published raw-completion spot check.
|
| 142 |
|
| 143 |
+
## 5. Quantized variants
|
| 144 |
|
| 145 |
Q8_0 GGUF was evaluated separately under llama.cpp. That runtime and its
|
| 146 |
stop handling differ from the bf16 harness, so the scores are not directly
|
|
|
|
| 156 |
|
| 157 |
Use bf16 when the bf16 profile is required.
|
| 158 |
|
| 159 |
+
For local GGUF inference:
|
| 160 |
+
|
| 161 |
+
```bash
|
| 162 |
+
llama-cli \
|
| 163 |
+
-m coneml-810m-alpha-Q8_0.gguf \
|
| 164 |
+
-p $'Question: What is 7 + 8?\nAnswer:' \
|
| 165 |
+
-n 32 --repeat-penalty 1.15 --no-conversation --single-turn
|
| 166 |
+
```
|
| 167 |
+
|
| 168 |
+
For Ollama, download `Modelfile` beside the Q8_0 file, then run:
|
| 169 |
+
|
| 170 |
+
```bash
|
| 171 |
+
ollama create coneml-810m-alpha -f Modelfile
|
| 172 |
+
ollama run coneml-810m-alpha "What is 7 + 8?"
|
| 173 |
+
```
|
| 174 |
+
|
| 175 |
+
## 6. Loading with Transformers
|
| 176 |
|
| 177 |
```python
|
| 178 |
import torch
|
|
|
|
| 204 |
This is a text-only causal language model. Load it with
|
| 205 |
`AutoModelForCausalLM` or `LlamaForCausalLM`, not a multimodal model class.
|
| 206 |
|
| 207 |
+
## 7. Intended use and safety
|
| 208 |
|
| 209 |
This release is intended for research, constrained local inference,
|
| 210 |
evaluation-harness development, and experiments whose prompts match the
|
|
|
|
| 215 |
generated code. Generated code may be incomplete or unsafe; inspect and
|
| 216 |
test it in an isolated environment.
|
| 217 |
|
| 218 |
+
## 8. Known limitations
|
| 219 |
|
| 220 |
- Not a general code model: complete programs scored 0/10, executable SQL
|
| 221 |
1/10, and bash 1/5. The 69% result is for single functions on ConeML's
|
|
|
|
| 229 |
- Multi-step GSM-class reasoning remained weak.
|
| 230 |
- Frameless prompting was spot-checked, not systematically certified.
|
| 231 |
|
| 232 |
+
## 9. Evidence and reproducibility
|
| 233 |
|
| 234 |
The `eval/` directory publishes:
|
| 235 |
|
|
|
|
| 240 |
- separate Q8_0 aggregate results; and
|
| 241 |
- `PRIVATE_EVIDENCE_SHA256SUMS.txt`: commitments to the complete
|
| 242 |
row-level evidence retained by ConeML.
|
| 243 |
+
- `peer-comparison-summary.json`, `PEER_COMPARISON.md`, and
|
| 244 |
+
`PEER_EVIDENCE_SHA256SUMS.txt`: the peer results, interpretation, and
|
| 245 |
+
commitments to the retained peer-generation evidence.
|
| 246 |
|
| 247 |
Complete proprietary probe rows and generations are retained privately to
|
| 248 |
avoid releasing the full measurement instruments. The hashes permit later
|
|
|
|
| 250 |
Public GSM8K and HumanEval rows are not redistributed; only aggregate
|
| 251 |
results are published. `SHA256SUMS.txt` covers every shipped file.
|
| 252 |
|
| 253 |
+
## 10. License and commercial enquiries
|
| 254 |
|
| 255 |
Released for non-commercial use under CC BY-NC 4.0. Commercial use is not
|
| 256 |
granted by this release. See `LICENSE.md`.
|
SHA256SUMS.txt
CHANGED
|
@@ -1,14 +1,18 @@
|
|
| 1 |
846a8e2297580b3c8d57f69a07d88833d370f976010b31d9fc9b93e9a3f04464 LICENSE.md
|
| 2 |
-
|
|
|
|
| 3 |
fdf72939738a346a2134cd5ede1e9921d52d4d85ac0449249118322de48447e7 chat_template.jinja
|
| 4 |
a09065a6d6c5ee5c518ed913085e878e7d3807f7d6cb8bcd1d6ce59653bd41b5 coneml-810m-alpha-Q8_0.gguf
|
| 5 |
16835ec01c1fa344c5bd3788c7697ab67e68676ebae3fc374b62f7b30749a6bb coneml-810m-alpha-f16.gguf
|
| 6 |
6cd7853f16b53fdb6bcc44438be39c33726379dc9848798fdcc81ba7f7555a58 config.json
|
| 7 |
8a56ef50eebf7604d0408b52249591e258a31ff1c9ebf3785f748c0316637c56 conversion.json
|
| 8 |
83b8f89b84387d8c2b22bc2b5d8c20a564fea9f79c7f6e6ab4fa9acee38040c5 eval/EVALUATION_METHODOLOGY.md
|
|
|
|
|
|
|
| 9 |
b2414d6dddbac7128724d27a7ce54e2fa3586943ece7b94e6c3c5188a1f04484 eval/PRIVATE_EVIDENCE_SHA256SUMS.txt
|
| 10 |
0108c09e9b835a0da81c5093b40f0b7937b1553de3abdcb5c207c75888714db2 eval/certification-Q8_0.json
|
| 11 |
b4cc4f98ccd8ba91206e4446c38f6b1e5f6e7a07f3c9ec963ee16d6a0619dc71 eval/code-cert-Q8_0.json
|
|
|
|
| 12 |
bff2af0c69ec8cce87cd93a69c22045b4062588e60f1dafb4ba0b85ac68f2c8d eval/representative-samples.json
|
| 13 |
f6ce9e02abad733c0d5ae9fdaa58f290441a24c4f24c7673c4f50ca5150e1d70 eval/summary.json
|
| 14 |
2e2ddba5318c74a4fcc31cfdfdc6e067564176b6404da484932f95a7fdc9712a generation_config.json
|
|
|
|
| 1 |
846a8e2297580b3c8d57f69a07d88833d370f976010b31d9fc9b93e9a3f04464 LICENSE.md
|
| 2 |
+
dae2b5a811b8d4fd08801460bc80968d8792391bec75a235b6e79d0ba2fd952c Modelfile
|
| 3 |
+
86ad0bc96b23050d854c8c1ef6353aa1ecc36581f21038b8e418a54de3c37483 README.md
|
| 4 |
fdf72939738a346a2134cd5ede1e9921d52d4d85ac0449249118322de48447e7 chat_template.jinja
|
| 5 |
a09065a6d6c5ee5c518ed913085e878e7d3807f7d6cb8bcd1d6ce59653bd41b5 coneml-810m-alpha-Q8_0.gguf
|
| 6 |
16835ec01c1fa344c5bd3788c7697ab67e68676ebae3fc374b62f7b30749a6bb coneml-810m-alpha-f16.gguf
|
| 7 |
6cd7853f16b53fdb6bcc44438be39c33726379dc9848798fdcc81ba7f7555a58 config.json
|
| 8 |
8a56ef50eebf7604d0408b52249591e258a31ff1c9ebf3785f748c0316637c56 conversion.json
|
| 9 |
83b8f89b84387d8c2b22bc2b5d8c20a564fea9f79c7f6e6ab4fa9acee38040c5 eval/EVALUATION_METHODOLOGY.md
|
| 10 |
+
3817126000edea9bc2fcb75a783fa73de234590cdfe89e1aea9b569bae809caa eval/PEER_COMPARISON.md
|
| 11 |
+
73d8100a4b78d7693aabb7a22f3b9948774f7a418d9fa162d19d6ff630baffd4 eval/PEER_EVIDENCE_SHA256SUMS.txt
|
| 12 |
b2414d6dddbac7128724d27a7ce54e2fa3586943ece7b94e6c3c5188a1f04484 eval/PRIVATE_EVIDENCE_SHA256SUMS.txt
|
| 13 |
0108c09e9b835a0da81c5093b40f0b7937b1553de3abdcb5c207c75888714db2 eval/certification-Q8_0.json
|
| 14 |
b4cc4f98ccd8ba91206e4446c38f6b1e5f6e7a07f3c9ec963ee16d6a0619dc71 eval/code-cert-Q8_0.json
|
| 15 |
+
baff252791bdf43653763a53a57024d0075bb9a1f6b35cf1fcc94a9191cc73df eval/peer-comparison-summary.json
|
| 16 |
bff2af0c69ec8cce87cd93a69c22045b4062588e60f1dafb4ba0b85ac68f2c8d eval/representative-samples.json
|
| 17 |
f6ce9e02abad733c0d5ae9fdaa58f290441a24c4f24c7673c4f50ca5150e1d70 eval/summary.json
|
| 18 |
2e2ddba5318c74a4fcc31cfdfdc6e067564176b6404da484932f95a7fdc9712a generation_config.json
|
eval/PEER_COMPARISON.md
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ConeML 810M peer diagnostic and resource context
|
| 2 |
+
|
| 3 |
+
Date: 2026-07-30
|
| 4 |
+
|
| 5 |
+
This is an interface-specific diagnostic screen, not a neutral ranking of
|
| 6 |
+
general model capability. Each instruction-tuned model was evaluated
|
| 7 |
+
through its own instruction interface. All models used greedy decoding
|
| 8 |
+
with repetition penalty 1.15. Arithmetic and function-writing used
|
| 9 |
+
matched short-answer generation budgets.
|
| 10 |
+
|
| 11 |
+
The task families match ConeML's supervised training surfaces. That gives
|
| 12 |
+
the comparison practical value for locating the release's envelope, but
|
| 13 |
+
it also means these results must not be generalized to unrelated tasks.
|
| 14 |
+
Standard GSM8K and HumanEval results remain separately disclosed in the
|
| 15 |
+
model cards.
|
| 16 |
+
|
| 17 |
+
## Instruct-model results
|
| 18 |
+
|
| 19 |
+
| measured surface | ConeML Alpha | ConeML Arithmetic | Qwen3.5 0.8B | Qwen3 0.6B | Llama 3.2 1B Instruct | TinyLlama Chat | SmolLM2 1.7B Instruct |
|
| 20 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 21 |
+
| mixed arithmetic screen (n=585) | 421 (72.0%) | **442 (75.6%)** | 154 (26.3%) | 223 (38.1%) | 343 (58.6%) | 113 (19.3%) | 380 (65.0%) |
|
| 22 |
+
| four core arithmetic lanes (n=225) | 177 (78.7%) | **221 (98.2%)** | 150 (66.7%) | 208 (92.4%) | **221 (98.2%)** | 92 (40.9%) | **221 (98.2%)** |
|
| 23 |
+
| executed single functions (n=100) | 83 | 35 | 93 | **98** | 79 | 51 | 96 |
|
| 24 |
+
| designated-refusal prompts (n=17) | **13** | 11 | 0 | 1 | 1 | 0 | 1 |
|
| 25 |
+
| over-refusals on contrasts (n=5) | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
|
| 26 |
+
|
| 27 |
+
The honest result is mixed:
|
| 28 |
+
|
| 29 |
+
- At matched short-answer budgets, the ConeML pair led this post-trained
|
| 30 |
+
peer group on the broader arithmetic screen.
|
| 31 |
+
- On the four core arithmetic lanes, the Arithmetic variant tied Llama
|
| 32 |
+
3.2 Instruct and SmolLM2 at this reduced sample size. Its separate
|
| 33 |
+
full-size internal result is 1,093/1,116 (97.9%).
|
| 34 |
+
- Alpha exceeded Llama 3.2 Instruct and TinyLlama Chat on the executed
|
| 35 |
+
function screen, but Qwen3.5, Qwen3, and SmolLM2 scored higher.
|
| 36 |
+
- The refusal row measures a trained response policy on designated
|
| 37 |
+
prompts, not factual correctness or general epistemic calibration.
|
| 38 |
+
|
| 39 |
+
The peer-harness refusal results above differ from the dedicated ConeML
|
| 40 |
+
probe's 17/17 for each release because the generation loops differ. Both
|
| 41 |
+
measurements are reported rather than merged.
|
| 42 |
+
|
| 43 |
+
## Qwen3.5 thinking-mode sensitivity
|
| 44 |
+
|
| 45 |
+
Qwen3.5 0.8B was also evaluated with thinking enabled on the same 585
|
| 46 |
+
arithmetic items:
|
| 47 |
+
|
| 48 |
+
| metric | Qwen3.5 thinking | Qwen3.5 short-answer | ConeML Arithmetic short-answer |
|
| 49 |
+
|---|---:|---:|---:|
|
| 50 |
+
| accuracy | 451/585 (77.1%) | 154/585 (26.3%) | 442/585 (75.6%) |
|
| 51 |
+
| generated tokens per item | mean 844; median 699 | at most 48 | at most 48 |
|
| 52 |
+
| p95 / generation cap | 1,536 / 1,536 | 48 / 48 | 48 / 48 |
|
| 53 |
+
| truncation rate | 138/585 (23.6%) | approximately 0% | approximately 0% |
|
| 54 |
+
| mean wall time per item | 2.43 s | approximately 0.1 s | approximately 0.1 s |
|
| 55 |
+
| correct answers per 1,000 generated tokens | 0.91 | 5.5 | approximately 15.7 or higher |
|
| 56 |
+
|
| 57 |
+
Thinking mode recovered Qwen3.5 to statistical parity with ConeML
|
| 58 |
+
Arithmetic on this screen, while using at least 17.6 times the
|
| 59 |
+
per-answer generation budget and approximately 24 times the measured wall
|
| 60 |
+
time. Wall-time ratios are specific to the recorded hardware, batching,
|
| 61 |
+
and implementation. This is an inference-cost comparison, not a claim
|
| 62 |
+
that thinking mode is intrinsically inferior.
|
| 63 |
+
|
| 64 |
+
## Against-interest base result
|
| 65 |
+
|
| 66 |
+
On the same 585 arithmetic items, through a task frame native to neither
|
| 67 |
+
base model, Qwen3.5 0.8B Base scored 492/585 (84.1%), while ConeML base
|
| 68 |
+
checkpoint 188 scored 156/585 (26.7%). This result is included because it
|
| 69 |
+
prevents an absolute-superiority reading: high-exposure base models may
|
| 70 |
+
already contain strong task-formatted behavior, and post-training can
|
| 71 |
+
move capability between output surfaces.
|
| 72 |
+
|
| 73 |
+
## Training-resource context
|
| 74 |
+
|
| 75 |
+
ConeML's selected base consumed approximately 12.32B token positions:
|
| 76 |
+
15.2 tokens per parameter and approximately 5.99e19 training FLOPs under
|
| 77 |
+
the `6 × parameters × tokens` convention.
|
| 78 |
+
|
| 79 |
+
| model family | disclosed pretraining tokens | approximate tokens/parameter | approximate training FLOPs vs ConeML |
|
| 80 |
+
|---|---:|---:|---:|
|
| 81 |
+
| ConeML 810M | 12.32B | 15.2 | 1× |
|
| 82 |
+
| TinyLlama 1.1B | 3T | 2,727 | 330× |
|
| 83 |
+
| Llama 3.2 1B | up to 9T | 7,258 | 1,118×, plus distillation |
|
| 84 |
+
| SmolLM2 1.7B | 11T | 6,471 | 1,873× |
|
| 85 |
+
| Qwen3 0.6B | 36T | 60,000 | 2,164× |
|
| 86 |
+
| Qwen3.5 0.8B | not disclosed | not stated | not stated |
|
| 87 |
+
|
| 88 |
+
These are estimated pretraining FLOP ratios, not historical electricity
|
| 89 |
+
or monetary costs for the peer models.
|
| 90 |
+
|
| 91 |
+
ConeML pretraining took approximately 11 days on one local RTX 5090. At
|
| 92 |
+
an explicitly assumed average wall draw of 0.70 kW, that corresponds to
|
| 93 |
+
184.8 kWh. Applying an assumed Swiss residential tariff range of
|
| 94 |
+
CHF 0.14–0.30/kWh gives approximately CHF 26–55 of marginal pretraining
|
| 95 |
+
electricity.
|
| 96 |
+
|
| 97 |
+
| ConeML pretraining unit | estimate |
|
| 98 |
+
|---|---:|
|
| 99 |
+
| average throughput | 12,963 token positions/s |
|
| 100 |
+
| wall time per billion token positions | 21.4 h |
|
| 101 |
+
| energy per billion token positions | 15.0 kWh |
|
| 102 |
+
| energy per million token positions | 15 Wh |
|
| 103 |
+
| estimated wall energy per token position | 0.054 J |
|
| 104 |
+
| marginal electricity per billion token positions | CHF 2.10–4.50 |
|
| 105 |
+
|
| 106 |
+
The energy and cost figures are estimates derived from the stated power
|
| 107 |
+
and tariff assumptions, not meter readings. They exclude hardware,
|
| 108 |
+
depreciation, labor, supervised fine-tuning, evaluation, conversion,
|
| 109 |
+
datacenter PUE, and carbon intensity. No emissions claim is made.
|
| 110 |
+
|
| 111 |
+
## Evaluated revisions
|
| 112 |
+
|
| 113 |
+
- `Qwen/Qwen3.5-0.8B@2fc06364715b967f1860aea9cf38778875588b17`
|
| 114 |
+
- `Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca`
|
| 115 |
+
- `unsloth/Llama-3.2-1B-Instruct@5a8abab4a5d6f164389b1079fb721cfab8d7126c`
|
| 116 |
+
- `TinyLlama/TinyLlama-1.1B-Chat-v1.0@fe8a4ea1ffedaf415f4da2f062534de366a451e6`
|
| 117 |
+
- `HuggingFaceTB/SmolLM2-1.7B-Instruct@31b70e2e869a7173562077fd711b654946d38674`
|
| 118 |
+
|
| 119 |
+
The exact aggregates are in `peer-comparison-summary.json`. Complete
|
| 120 |
+
generation rows are retained privately; their frozen SHA-256 commitments
|
| 121 |
+
are published in `PEER_EVIDENCE_SHA256SUMS.txt`.
|
| 122 |
+
|
| 123 |
+
## Primary resource disclosures
|
| 124 |
+
|
| 125 |
+
- Qwen3 pretraining: [Qwen3 release post](https://qwenlm.github.io/blog/qwen3/)
|
| 126 |
+
- Llama 3.2 token count and distillation: [Meta Llama 3.2 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md)
|
| 127 |
+
- TinyLlama token count: [TinyLlama model card](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)
|
| 128 |
+
- SmolLM2 token count: [Hugging Face SmolLM2 model card](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)
|
eval/PEER_EVIDENCE_SHA256SUMS.txt
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
96a805aed179f77d9f1959216f9a39e8bf8d50dea5465eeea1eb24e96502a6af SOURCE_PEER_COMPARISON_V2.md
|
| 2 |
+
a5d6cd4d7a2643b316cefb1dcb6f43296bb69da4064dbc3b3ab4a123bf36b2b1 base-reference/coneml-base188.json
|
| 3 |
+
83643ce110655e708d50f4eb3ee29ab08ab8e003768eb5b5da9a3e545bc9b8ef base-reference/coneml-base188.rows.jsonl
|
| 4 |
+
ca387426c55ac72d50041e2f082bd4cdc747eedb51deca7929cf7a4acea2c5b6 base-reference/coneml-native-transitive.json
|
| 5 |
+
43ef584d37c6a276b190aeb741597b54bfebbf7bbd9c37df5d82a2e15a8ee2d1 base-reference/qwen35-0.8b-base.json
|
| 6 |
+
25de33180d4073777d29871882b57f1556733e9a8c1e10eba14a8cac062b5c35 base-reference/qwen35-0.8b-base.rows.jsonl
|
| 7 |
+
2003c73a54602f3a30980d6c7f85feef39b63fd11cef9fc20a90044d3569ba77 instruct/coneml-810m-alpha.json
|
| 8 |
+
8da107545bd6717c7795ff4e827698e5e269d451f2390f24dccc516fc390ab44 instruct/coneml-810m-alpha.rows.jsonl
|
| 9 |
+
77d514c14e14557c4ac30e8a8c3f4c5887f0db0dff39b8ea14baea164327d027 instruct/coneml-810m-arithmetic.json
|
| 10 |
+
a60ed3e533c0c4a77b34a21f80b8df8de85e3df074f3cacdc3bbd4b5e0741cce instruct/coneml-810m-arithmetic.rows.jsonl
|
| 11 |
+
aa35b93f46778ffb22351ac411f2fc04e645775e416b5a9423f925bb6d6b1609 instruct/llama32-1b-instruct-transchat.json
|
| 12 |
+
ee514c35d86f27172a004325b26da915f869164adae59edf8d95b01d958e683f instruct/llama32-1b-instruct-transchat.rows.jsonl
|
| 13 |
+
f4553ef34844e55364a87f631fac03a3545acbb14cd12dcf182f16151b5e5f8a instruct/llama32-1b-instruct.json
|
| 14 |
+
d5c0fd8b00b50754ac7dede96081afeaf3143795e005134cf5543483859340bf instruct/llama32-1b-instruct.rows.jsonl
|
| 15 |
+
a61b56681810410dfc3a8d8e758b8ccd11d01ebcce2d963d6bdbca4644c2cee9 instruct/qwen3-0.6b-transchat.json
|
| 16 |
+
f7c763441edc0d5e37a34902a3cc4efcacc6a35b47056adb9356879a4407cb31 instruct/qwen3-0.6b-transchat.rows.jsonl
|
| 17 |
+
3f34342d79095a5754512ebf57192fdf98a16b364a250712993ac1da0b65e252 instruct/qwen3-0.6b.json
|
| 18 |
+
05847f4187651780008331a85c9cd5fa84d3b636320c9a809fb3c307f1a230d6 instruct/qwen3-0.6b.rows.jsonl
|
| 19 |
+
21984e00a4a6da3d2887138ede25c03e9007fee0be41de462ef58dc58db89554 instruct/qwen35-0.8b-transchat.json
|
| 20 |
+
48a53b6bf3829c207953b1075160df9f6702b1fc2162426577aa69530856e555 instruct/qwen35-0.8b-transchat.rows.jsonl
|
| 21 |
+
fa11fbf67c047ac541ecd38571867f9fe68bfb8aec12b84f2addda8a6d561642 instruct/qwen35-0.8b.json
|
| 22 |
+
74a6debf6ae55db9895f556e654b6905fde871358a94c9c4a2730a418b0e2608 instruct/qwen35-0.8b.rows.jsonl
|
| 23 |
+
72598668bfce72497ac67a4d83d1cad7538c63ec933e8a1d1198876d85dc8a67 instruct/qwen35-thinking-sensitivity.json
|
| 24 |
+
8917677a4c9d29124c5ade229f7080bbd0ccb83471c3d7d39a6f958224596a8f instruct/smollm2-1.7b-instruct-transchat.json
|
| 25 |
+
deb9eb805475e4ad2d158283b469830138fdf42ee0c9a903bf3611b43a7869ff instruct/smollm2-1.7b-instruct-transchat.rows.jsonl
|
| 26 |
+
94a3440cdaae14eb27beaf36a1a0103dab4a6d292bc382981f0514a82a93ed0f instruct/smollm2-1.7b-instruct.json
|
| 27 |
+
1b06370ad7b940ed05b59ffe4acd19035341cfe078ed7b01d59d168abc8d0c71 instruct/smollm2-1.7b-instruct.rows.jsonl
|
| 28 |
+
bdd7cf2867643439e90a3e04cb209e6130839af9422e492bcd2eb1c2cba4556e instruct/tinyllama-chat-transchat.json
|
| 29 |
+
472aad0c169c4f943fbc9a8367d65fa2d5c41acbbee66d099e16a286f8676d1c instruct/tinyllama-chat-transchat.rows.jsonl
|
| 30 |
+
3a7c2e7ebcaa78b4b7d9b6d84a5b90a495f260f97287bad3282d54fc84e0c009 instruct/tinyllama-chat.json
|
| 31 |
+
5926a31e926279f8d9459d19cbbe9730a06071bb8b4029c2f9174804ab447943 instruct/tinyllama-chat.rows.jsonl
|
eval/peer-comparison-summary.json
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema_version": 1,
|
| 3 |
+
"date": "2026-07-30",
|
| 4 |
+
"scope": "Interface-specific diagnostic screen, not a neutral ranking of general model capability. Each instruct model used its native instruction interface. Arithmetic and function-writing used matched short-answer generation budgets.",
|
| 5 |
+
"decoding": {
|
| 6 |
+
"strategy": "greedy",
|
| 7 |
+
"repetition_penalty": 1.15,
|
| 8 |
+
"arithmetic_n": 585,
|
| 9 |
+
"core_arithmetic_n": 225,
|
| 10 |
+
"function_writing_n": 100,
|
| 11 |
+
"designated_refusal_n": 17,
|
| 12 |
+
"refusal_contrast_n": 5
|
| 13 |
+
},
|
| 14 |
+
"evaluated_revisions": {
|
| 15 |
+
"ConeML/coneml-810m-alpha": "local release artifact e5df8f2edd773ad079ba00d383ace81b96cbb9579d64c570db505f9efc032b76",
|
| 16 |
+
"ConeML/coneml-810m-alpha-arithmetic": "local release artifact e1de27f5f91ac43130ce2a57a4bf9a036b84b2e5b589789b5981ecf79489e0d1",
|
| 17 |
+
"Qwen/Qwen3.5-0.8B": "2fc06364715b967f1860aea9cf38778875588b17",
|
| 18 |
+
"Qwen/Qwen3-0.6B": "c1899de289a04d12100db370d81485cdf75e47ca",
|
| 19 |
+
"unsloth/Llama-3.2-1B-Instruct": "5a8abab4a5d6f164389b1079fb721cfab8d7126c",
|
| 20 |
+
"TinyLlama/TinyLlama-1.1B-Chat-v1.0": "fe8a4ea1ffedaf415f4da2f062534de366a451e6",
|
| 21 |
+
"HuggingFaceTB/SmolLM2-1.7B-Instruct": "31b70e2e869a7173562077fd711b654946d38674"
|
| 22 |
+
},
|
| 23 |
+
"instruct_results": [
|
| 24 |
+
{
|
| 25 |
+
"model": "ConeML/coneml-810m-alpha",
|
| 26 |
+
"parameters_billions": 0.8101,
|
| 27 |
+
"arithmetic_mixed": {"correct": 421, "n": 585, "accuracy": 0.7197},
|
| 28 |
+
"arithmetic_core_four_lanes": {"correct": 177, "n": 225, "accuracy": 0.7867},
|
| 29 |
+
"executed_functions": {"correct": 83, "n": 100, "accuracy": 0.83},
|
| 30 |
+
"designated_refusals": {"correct": 13, "n": 17, "accuracy": 0.7647},
|
| 31 |
+
"over_refusals": {"count": 0, "n": 5}
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"model": "ConeML/coneml-810m-alpha-arithmetic",
|
| 35 |
+
"parameters_billions": 0.8101,
|
| 36 |
+
"arithmetic_mixed": {"correct": 442, "n": 585, "accuracy": 0.7556},
|
| 37 |
+
"arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
|
| 38 |
+
"executed_functions": {"correct": 35, "n": 100, "accuracy": 0.35},
|
| 39 |
+
"designated_refusals": {"correct": 11, "n": 17, "accuracy": 0.6471},
|
| 40 |
+
"over_refusals": {"count": 0, "n": 5}
|
| 41 |
+
},
|
| 42 |
+
{
|
| 43 |
+
"model": "Qwen/Qwen3.5-0.8B",
|
| 44 |
+
"parameters_billions": 0.8,
|
| 45 |
+
"mode": "thinking disabled for matched short-answer budget",
|
| 46 |
+
"arithmetic_mixed": {"correct": 154, "n": 585, "accuracy": 0.2632},
|
| 47 |
+
"arithmetic_core_four_lanes": {"correct": 150, "n": 225, "accuracy": 0.6667},
|
| 48 |
+
"executed_functions": {"correct": 93, "n": 100, "accuracy": 0.93},
|
| 49 |
+
"designated_refusals": {"correct": 0, "n": 17, "accuracy": 0.0},
|
| 50 |
+
"over_refusals": {"count": 0, "n": 5}
|
| 51 |
+
},
|
| 52 |
+
{
|
| 53 |
+
"model": "Qwen/Qwen3-0.6B",
|
| 54 |
+
"parameters_billions": 0.6,
|
| 55 |
+
"arithmetic_mixed": {"correct": 223, "n": 585, "accuracy": 0.3812},
|
| 56 |
+
"arithmetic_core_four_lanes": {"correct": 208, "n": 225, "accuracy": 0.9244},
|
| 57 |
+
"executed_functions": {"correct": 98, "n": 100, "accuracy": 0.98},
|
| 58 |
+
"designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
|
| 59 |
+
"over_refusals": {"count": 0, "n": 5}
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"model": "unsloth/Llama-3.2-1B-Instruct",
|
| 63 |
+
"parameters_billions": 1.24,
|
| 64 |
+
"arithmetic_mixed": {"correct": 343, "n": 585, "accuracy": 0.5863},
|
| 65 |
+
"arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
|
| 66 |
+
"executed_functions": {"correct": 79, "n": 100, "accuracy": 0.79},
|
| 67 |
+
"designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
|
| 68 |
+
"over_refusals": {"count": 0, "n": 5}
|
| 69 |
+
},
|
| 70 |
+
{
|
| 71 |
+
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
|
| 72 |
+
"parameters_billions": 1.1,
|
| 73 |
+
"arithmetic_mixed": {"correct": 113, "n": 585, "accuracy": 0.1932},
|
| 74 |
+
"arithmetic_core_four_lanes": {"correct": 92, "n": 225, "accuracy": 0.4089},
|
| 75 |
+
"executed_functions": {"correct": 51, "n": 100, "accuracy": 0.51},
|
| 76 |
+
"designated_refusals": {"correct": 0, "n": 17, "accuracy": 0.0},
|
| 77 |
+
"over_refusals": {"count": 0, "n": 5}
|
| 78 |
+
},
|
| 79 |
+
{
|
| 80 |
+
"model": "HuggingFaceTB/SmolLM2-1.7B-Instruct",
|
| 81 |
+
"parameters_billions": 1.7,
|
| 82 |
+
"arithmetic_mixed": {"correct": 380, "n": 585, "accuracy": 0.6496},
|
| 83 |
+
"arithmetic_core_four_lanes": {"correct": 221, "n": 225, "accuracy": 0.9822},
|
| 84 |
+
"executed_functions": {"correct": 96, "n": 100, "accuracy": 0.96},
|
| 85 |
+
"designated_refusals": {"correct": 1, "n": 17, "accuracy": 0.0588},
|
| 86 |
+
"over_refusals": {"count": 0, "n": 5}
|
| 87 |
+
}
|
| 88 |
+
],
|
| 89 |
+
"qwen35_thinking_sensitivity": {
|
| 90 |
+
"model": "Qwen/Qwen3.5-0.8B",
|
| 91 |
+
"mode": "thinking enabled",
|
| 92 |
+
"arithmetic_mixed": {"correct": 451, "n": 585, "accuracy": 0.771},
|
| 93 |
+
"generated_tokens": {"mean": 844, "median": 699, "p95": 1536},
|
| 94 |
+
"generation_cap": 1536,
|
| 95 |
+
"truncation": {"count": 138, "n": 585, "rate": 0.2359},
|
| 96 |
+
"wall_seconds_per_item_mean": 2.43,
|
| 97 |
+
"correct_per_1000_generated_tokens": 0.913,
|
| 98 |
+
"comparison": "Qwen thinking reached parity with ConeML Arithmetic's 442/585 while using at least 17.6 times the generated-token budget per item. The recorded wall-time ratio was approximately 24 times. Runtime measurements are hardware- and implementation-specific."
|
| 99 |
+
},
|
| 100 |
+
"against_interest_base_result": {
|
| 101 |
+
"scope": "Same arithmetic items through a task frame not native to either base model.",
|
| 102 |
+
"ConeML_base_188": {"correct": 156, "n": 585, "accuracy": 0.2667},
|
| 103 |
+
"Qwen3.5_0.8B_Base": {"correct": 492, "n": 585, "accuracy": 0.841},
|
| 104 |
+
"interpretation": "The peer base result prevents an absolute-superiority interpretation and illustrates sensitivity to training exposure and interface."
|
| 105 |
+
},
|
| 106 |
+
"training_resource_context": {
|
| 107 |
+
"ConeML": {
|
| 108 |
+
"pretraining_token_positions": 12320000000,
|
| 109 |
+
"tokens_per_parameter": 15.2,
|
| 110 |
+
"estimated_training_flops_6ND": 5.99e19,
|
| 111 |
+
"pretraining_wall_time_days": 11,
|
| 112 |
+
"hardware": "one NVIDIA RTX 5090",
|
| 113 |
+
"power_assumption_kw_average_wall": 0.70,
|
| 114 |
+
"estimated_pretraining_energy_kwh": 184.8,
|
| 115 |
+
"assumed_swiss_residential_tariff_chf_per_kwh": [0.14, 0.30],
|
| 116 |
+
"estimated_marginal_pretraining_electricity_chf": [25.87, 55.44],
|
| 117 |
+
"exclusions": ["hardware", "depreciation", "labor", "SFT", "evaluation", "conversion", "grid-carbon estimate"]
|
| 118 |
+
},
|
| 119 |
+
"peer_compute_context": [
|
| 120 |
+
{"model": "TinyLlama 1.1B", "published_pretraining_tokens": 3000000000000, "approx_tokens_per_parameter": 2727, "approx_flops_vs_coneml": 330},
|
| 121 |
+
{"model": "Llama 3.2 1B", "published_pretraining_tokens": 9000000000000, "approx_tokens_per_parameter": 7258, "approx_flops_vs_coneml": 1118, "note": "plus distillation"},
|
| 122 |
+
{"model": "SmolLM2 1.7B", "published_pretraining_tokens": 11000000000000, "approx_tokens_per_parameter": 6471, "approx_flops_vs_coneml": 1873},
|
| 123 |
+
{"model": "Qwen3 0.6B", "published_pretraining_tokens": 36000000000000, "approx_tokens_per_parameter": 60000, "approx_flops_vs_coneml": 2164},
|
| 124 |
+
{"model": "Qwen3.5 0.8B", "published_pretraining_tokens": null, "approx_tokens_per_parameter": null, "approx_flops_vs_coneml": null}
|
| 125 |
+
]
|
| 126 |
+
},
|
| 127 |
+
"evidence": {
|
| 128 |
+
"public_interpretation": "PEER_COMPARISON.md",
|
| 129 |
+
"private_rows": "retained by ConeML",
|
| 130 |
+
"hash_commitment": "PEER_EVIDENCE_SHA256SUMS.txt"
|
| 131 |
+
}
|
| 132 |
+
}
|