| # ConeML 810M — evaluation methodology and variant lineage (public record) |
|
|
| Two released variants share one from-scratch 810M base checkpoint and |
| differ only in supervised fine-tuning; both were evaluated with the same |
| bf16 instruments. |
|
|
| - `coneml-810m-alpha` — function-writing / mixed-arithmetic profile |
| - `coneml-810m-alpha-arithmetic` — carry/borrow arithmetic variant |
|
|
| ## Methodology |
|
|
| - **ConeML internal certification**: large-n (300–500 per task family) |
| batteries of unseen instances in the trained prompt formats, greedy |
| decoding, repetition |
| penalty 1.15. Code tasks are scored by EXECUTING the generated function |
| against held-out tests; SQL by executing against a live sqlite database; |
| scripts by captured stdout. No pattern-match scoring for anything |
| execution can decide. |
| - **Behavior probes**: held-out designated-refusal prompts plus in-scope |
| contrast pairs in matching wording; one separately rerun, scripted |
| eight-turn conversation; and an everyday-reasoning screen with manual |
| adjudication. Aggregate results and deterministic representative |
| ConeML-owned rows are public; complete rows are retained privately. |
| - **Public benchmarks**: GSM8K (first 200, zero-shot, strict final-number |
| match) and HumanEval (pass@1, tests executed), identical harness across |
| variants and the pretrained base. Aggregate results are public; |
| benchmark rows are not redistributed. |
| - **Quantized variants are evaluated separately**: each shipped GGUF carries |
| its own result table. The GGUF and bf16 harnesses differ in runtime and |
| stop handling, so score gaps are not attributed solely to quantization. |
|
|
| ## Why two variants |
|
|
| Under fixed architecture and base checkpoint, modest differences in |
| fine-tuning composition produced large, cross-surface capability |
| tradeoffs (e.g., one build scores 96.8% on held-out carry addition while |
| the other scores 79.0% on held-out two-step word problems and 69.0% on |
| held-out function writing). Rather than presenting one checkpoint as |
| dominant, each variant ships with its measured envelope and a |
| recommendation for which profile |
| fits which pipeline. A checkpoint's profile is measured, never inferred. |
|
|
| ## Shared limitations |
|
|
| Function-writing is evaluated for single Python functions only — neither |
| variant is a general code model (complete programs, executable SQL, and |
| bash scored near zero for both and are stated out of scope). Both variants |
| refused every designated refusal prompt, including real factual questions; |
| this targeted behavior is not evidence of general epistemic calibration. |
| Multi-step GSM-class reasoning is weak in every variant tested this cycle. |
|
|
| ## Evidence disclosure |
|
|
| The public folders contain aggregate result tables, deterministic |
| representative examples from ConeML-owned instruments, and SHA-256 |
| commitments to the complete row-level evidence. Full proprietary probe |
| rows are retained by ConeML rather than distributed. This preserves the |
| measurement instruments while allowing later disclosure to be checked |
| against the evidence frozen for this release. |
|
|
