eval/PEER_COMPARISON.md

# ConeML 810M peer diagnostic and resource context

Date: 2026-07-30

This is an interface-specific diagnostic screen, not a neutral ranking of
general model capability. Each instruction-tuned model was evaluated
through its own instruction interface. All models used greedy decoding
with repetition penalty 1.15. Arithmetic and function-writing used
matched short-answer generation budgets.

The task families match ConeML's supervised training surfaces. That gives
the comparison practical value for locating the release's envelope, but
it also means these results must not be generalized to unrelated tasks.
Standard GSM8K and HumanEval results remain separately disclosed in the
model cards.

## Instruct-model results

| measured surface | ConeML 810M Alpha | ConeML 810M Alpha-Arithmetic | Qwen3.5 0.8B | Qwen3 0.6B | Llama 3.2 1B Instruct | TinyLlama Chat | SmolLM2 1.7B Instruct |
|---|---:|---:|---:|---:|---:|---:|---:|
| mixed arithmetic screen (n=585) | 421 (72.0%) | **442 (75.6%)** | 154 (26.3%) | 223 (38.1%) | 343 (58.6%) | 113 (19.3%) | 380 (65.0%) |
| four core arithmetic lanes (n=225) | 177 (78.7%) | **221 (98.2%)** | 150 (66.7%) | 208 (92.4%) | **221 (98.2%)** | 92 (40.9%) | **221 (98.2%)** |
| executed single functions (n=100) | 83 | 35 | 93 | **98** | 79 | 51 | 96 |
| transitive names, depth 1/3/5 (n=32 each) | 23/15/15 | 26/17/16 | 23/22/10 | 18/10/9 | 16/7/5 | 8/18/22 | 16/6/5 |
| transitive entities, depth 1/3/5 (n=32 each) | 15/11/10 | 16/11/12 | 18/16/14 | 14/17/11 | 0/6/7 | 14/15/11 | 22/21/10 |
| designated-refusal prompts (n=17) | **13** | 11 | 0 | 1 | 1 | 0 | 1 |
| over-refusals on contrasts (n=5) | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

The honest result is mixed:

- At matched short-answer budgets, the ConeML pair led this post-trained
  peer group on the broader arithmetic screen.
- On the four core arithmetic lanes, the Arithmetic variant tied Llama
  3.2 Instruct and SmolLM2 at this reduced sample size. Its separate
  full-size internal result is 1,093/1,116 (97.9%).
- ConeML 810M Alpha exceeded Llama 3.2 Instruct and TinyLlama Chat on the
  executed function screen, but Qwen3.5, Qwen3, and SmolLM2 scored higher.
- On name-chain transitive selection, ConeML 810M Alpha-Arithmetic
  recorded 26/32, 17/32,
  and 16/32 at depths 1, 3, and 5. It exceeded Qwen3 and Llama 3.2 at all
  three shown depths and exceeded Qwen3.5 at depths 1 and 5; Qwen3.5 led
  it at depth 3, and TinyLlama led the group at depth 5.
- Entity-chain transfer was mixed: ConeML 810M Alpha recorded 15/11/10
  and ConeML 810M Alpha-Arithmetic 16/11/12 at depths 1/3/5, while different peers led each
  depth. Chance is 1/(depth+1). The non-monotonic peer rows reinforce
  that this is an exact-selection surface test, not proof of general
  reasoning depth.
- The refusal row measures a trained response policy on designated
  prompts, not factual correctness or general epistemic calibration.

The peer-harness refusal results above differ from the dedicated ConeML
probe's 17/17 for each release because the generation loops differ. Both
measurements are reported rather than merged.

## Qwen3.5 thinking-mode sensitivity

Qwen3.5 0.8B was also evaluated with thinking enabled on the same 585
arithmetic items:

| metric | Qwen3.5 thinking | Qwen3.5 short-answer | ConeML 810M Alpha-Arithmetic short-answer |
|---|---:|---:|---:|
| accuracy | 451/585 (77.1%) | 154/585 (26.3%) | 442/585 (75.6%) |
| generated tokens per item | mean 844; median 699 | at most 48 | at most 48 |
| p95 / generation cap | 1,536 / 1,536 | 48 / 48 | 48 / 48 |
| truncation rate | 138/585 (23.6%) | approximately 0% | approximately 0% |
| mean wall time per item | 2.43 s | approximately 0.1 s | approximately 0.1 s |
| correct answers per 1,000 generated tokens | 0.91 | 5.5 | approximately 15.7 or higher |

Thinking mode recovered Qwen3.5 to statistical parity with ConeML 810M
Alpha-Arithmetic on this screen, while using at least 17.6 times the
per-answer generation budget and approximately 24 times the measured wall
time. Wall-time ratios are specific to the recorded hardware, batching,
and implementation. This is an inference-cost comparison, not a claim
that thinking mode is intrinsically inferior.

## Against-interest base result

On the same 585 arithmetic items, through a task frame native to neither
base model, Qwen3.5 0.8B Base scored 492/585 (84.1%), while ConeML base
checkpoint 188 scored 156/585 (26.7%). This result is included because it
prevents an absolute-superiority reading: high-exposure base models may
already contain strong task-formatted behavior, and post-training can
move capability between output surfaces.

## Training-resource context

ConeML's selected base consumed approximately 12.32B token positions:
15.2 tokens per parameter and approximately 5.99e19 training FLOPs under
the `6 × parameters × tokens` convention.

| model family | disclosed pretraining tokens | approximate tokens/parameter | approximate training FLOPs vs ConeML |
|---|---:|---:|---:|
| ConeML 810M | 12.32B | 15.2 | 1× |
| TinyLlama 1.1B | 3T | 2,727 | 330× |
| Llama 3.2 1B | up to 9T | 7,258 | 1,118×, plus distillation |
| SmolLM2 1.7B | 11T | 6,471 | 1,873× |
| Qwen3 0.6B | 36T | 60,000 | 2,164× |
| Qwen3.5 0.8B | not disclosed | not stated | not stated |

These are estimated pretraining FLOP ratios, not historical electricity
or monetary costs for the peer models.

ConeML pretraining took approximately 11 days on one local RTX 5090. At
an explicitly assumed average wall draw of 0.70 kW, that corresponds to
184.8 kWh. Applying an assumed Swiss residential tariff range of
CHF 0.14–0.30/kWh gives approximately CHF 26–55 of marginal pretraining
electricity.

| ConeML pretraining unit | estimate |
|---|---:|
| average throughput | 12,963 token positions/s |
| wall time per billion token positions | 21.4 h |
| energy per billion token positions | 15.0 kWh |
| energy per million token positions | 15 Wh |
| estimated wall energy per token position | 0.054 J |
| marginal electricity per billion token positions | CHF 2.10–4.50 |

The energy and cost figures are estimates derived from the stated power
and tariff assumptions, not meter readings. They exclude hardware,
depreciation, labor, supervised fine-tuning, evaluation, conversion,
datacenter PUE, and carbon intensity. No emissions claim is made.

## Evaluated revisions

- `Qwen/Qwen3.5-0.8B@2fc06364715b967f1860aea9cf38778875588b17`
- `Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca`
- `unsloth/Llama-3.2-1B-Instruct@5a8abab4a5d6f164389b1079fb721cfab8d7126c`
- `TinyLlama/TinyLlama-1.1B-Chat-v1.0@fe8a4ea1ffedaf415f4da2f062534de366a451e6`
- `HuggingFaceTB/SmolLM2-1.7B-Instruct@31b70e2e869a7173562077fd711b654946d38674`

The exact aggregates are in `peer-comparison-summary.json`. Complete
generation rows are retained privately; their frozen SHA-256 commitments
are published in `PEER_EVIDENCE_SHA256SUMS.txt`.

## Primary resource disclosures

- Qwen3 pretraining: [Qwen3 release post](https://qwenlm.github.io/blog/qwen3/)
- Llama 3.2 token count and distillation: [Meta Llama 3.2 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md)
- TinyLlama token count: [TinyLlama model card](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)
- SmolLM2 token count: [Hugging Face SmolLM2 model card](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B)