File size: 3,055 Bytes
031dd3b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | # ConeML 810M — evaluation methodology and variant lineage (public record)
Two released variants share one from-scratch 810M base checkpoint and
differ only in supervised fine-tuning; both were evaluated with the same
bf16 instruments.
- `coneml-810m-alpha` — function-writing / mixed-arithmetic profile
- `coneml-810m-alpha-arithmetic` — carry/borrow arithmetic variant
## Methodology
- **ConeML internal certification**: large-n (300–500 per task family)
batteries of unseen instances in the trained prompt formats, greedy
decoding, repetition
penalty 1.15. Code tasks are scored by EXECUTING the generated function
against held-out tests; SQL by executing against a live sqlite database;
scripts by captured stdout. No pattern-match scoring for anything
execution can decide.
- **Behavior probes**: held-out designated-refusal prompts plus in-scope
contrast pairs in matching wording; one separately rerun, scripted
eight-turn conversation; and an everyday-reasoning screen with manual
adjudication. Aggregate results and deterministic representative
ConeML-owned rows are public; complete rows are retained privately.
- **Public benchmarks**: GSM8K (first 200, zero-shot, strict final-number
match) and HumanEval (pass@1, tests executed), identical harness across
variants and the pretrained base. Aggregate results are public;
benchmark rows are not redistributed.
- **Quantized variants are evaluated separately**: each shipped GGUF carries
its own result table. The GGUF and bf16 harnesses differ in runtime and
stop handling, so score gaps are not attributed solely to quantization.
## Why two variants
Under fixed architecture and base checkpoint, modest differences in
fine-tuning composition produced large, cross-surface capability
tradeoffs (e.g., one build scores 96.8% on held-out carry addition while
the other scores 79.0% on held-out two-step word problems and 69.0% on
held-out function writing). Rather than presenting one checkpoint as
dominant, each variant ships with its measured envelope and a
recommendation for which profile
fits which pipeline. A checkpoint's profile is measured, never inferred.
## Shared limitations
Function-writing is evaluated for single Python functions only — neither
variant is a general code model (complete programs, executable SQL, and
bash scored near zero for both and are stated out of scope). Both variants
refused every designated refusal prompt, including real factual questions;
this targeted behavior is not evidence of general epistemic calibration.
Multi-step GSM-class reasoning is weak in every variant tested this cycle.
## Evidence disclosure
The public folders contain aggregate result tables, deterministic
representative examples from ConeML-owned instruments, and SHA-256
commitments to the complete row-level evidence. Full proprietary probe
rows are retained by ConeML rather than distributed. This preserves the
measurement instruments while allowing later disclosure to be checked
against the evidence frozen for this release.
|