eval/README.md

# ConeML 348M Gamma Evaluation Package

This folder contains the local artifacts behind the Gamma model-card numbers. It is meant to make the release claims
auditable without publishing training data or private infrastructure details.

## Gamma Artifacts

| File | What it validates |
|---|---|
| `artifacts/gamma/humaneval164_fullctx.json` | HumanEval full-context run: 22/164, pass@1 13.4%, truncation 4.9%. |
| `artifacts/gamma/gsm8k1319_fullctx.json` | GSM8K full test run: 35/1319, exact-match 2.65%, truncation 1.7%. |
| `artifacts/gamma/internal_coverage_418.json` | Internal 418-item capability battery: 219/418 with per-bucket breakdown and raw generations. |
| `artifacts/gamma/epistemic_32.json` | Epistemic 32-item battery: 23/32 (abstention 10/10, contradiction 4/6, false premise 5/10, sycophancy 4/6) with raw generations. |
| `artifacts/gamma/heldout_transitive_probe.json` | Held-out transitive-binding probe with generated rows, examples, N=128 per depth for Gamma. Includes a non-name entity control suite. |

Checkpoint identifiers inside the artifacts are scrubbed to the release name (`coneml-348m-gamma`); all scores,
rows, and generations are exactly as run against the released checkpoint.

## Qwen Local Reference Artifacts

These are local reference runs, not a fully reproducible public Qwen baseline. They are included so the reader can see
the exact saved outputs behind the card's local comparison table.

| File | What it contains |
|---|---|
| `artifacts/qwen/qwen25_05b_ollama_full418.json` | Qwen `qwen2.5:0.5b` on the full 418-item internal battery. |
| `artifacts/qwen/qwen25_05b_ollama_objective.json` | Qwen objective-only subset. |
| `artifacts/qwen/qwen25_05b_ollama_epistemic.json` | Qwen epistemic subset. |
| `artifacts/qwen/qwen25_05b_ollama_transitive.json` | Qwen transitive-binding summary by depth. |

Recoverable Qwen run details:

- Model tag in saved artifact: `qwen2.5:0.5b`
- Runtime path: Ollama `/api/chat`
- Prompt shape for FULL418: one user message containing the internal prompt
- Explicit system prompt in harness: none
- FULL418 decoding options: `temperature=0`, `repeat_penalty=1.15`, `num_predict=2048`
- Transitive decoding options: `temperature=0`, `num_predict=16`
- Scoring: same functional scorers as the Gamma internal battery
- Raw generations: saved in the JSON files above

Missing Qwen baseline details:

- Ollama version
- Model digest
- GGUF source and quantization level
- Server-side chat template
- Upstream checkpoint revision
- Any server default context settings not captured by `/api/chat` request body

Because these details were not captured, the Qwen numbers should be read as a local reference run, not as a public
leaderboard-quality baseline. A cleaner public comparison would rerun `Qwen/Qwen2.5-0.5B-Instruct` through Transformers
with a pinned revision and `tokenizer.apply_chat_template(...)`.

## Battery And Scripts

| File | Purpose |
|---|---|
| `batteries/coverage_all_418.jsonl` | Internal 418-item battery definition used by the coverage runs. |
| `scripts/eval_coverage.py` | Gamma internal coverage evaluator. |
| `scripts/eval_api_full.py` | Ollama/API evaluator for a chat model on the same 418-item battery. |
| `scripts/heldout_transitive_probe.py` | Gamma held-out transitive-binding evaluator. |
| `scripts/qwen_transitive_probe.py` | Ollama/API transitive-binding evaluator. |

The copied API scripts use `http://127.0.0.1:11434` as a scrubbed default host. The original private endpoint is not
published.

## Headline Results

Public benchmarks for Gamma:

- HumanEval: 22/164 = 13.4%
- GSM8K: 35/1319 = 2.65%

Internal FULL418 comparison, all 15 categories, sorted by Gamma-minus-Qwen delta:

| Category (scorer) | Gamma | Qwen local reference | Δ |
|---|---:|---:|---:|
| Algorithms/data-structures (code_exec) | **11/20** | 1/20 | +10 |
| Code full-program (code_exec) | **9/20** | 0/20 | +9 |
| Code-write (code_exec) | **13/40** | 10/40 | +3 |
| Math-computation (math_exact) | **17/56** | 14/56 | +3 |
| Factual (factual_terms) | 24/30 | 24/30 | 0 |
| Relational-binding (factual_terms) | 27/30 | **28/30** | −1 |
| Instruction (bleed) | 23/30 | **25/30** | −2 |
| Explanation (rubric) | 17/20 | **20/20** | −3 |
| Narrative (rubric) | 26/30 | **29/30** | −3 |
| Math word-problem (math_exact) | 1/42 | **5/42** | −4 |
| Reasoning-decomposition (factual_terms) | 23/30 | **28/30** | −5 |
| Data/SQL (sql_shape) | 6/15 | **12/15** | −6 |
| Math-explanation (rubric) | 8/20 | **14/20** | −6 |
| Systems/devops (rubric) | 4/15 | **12/15** | −8 |
| Reasoning-everyday (factual_terms) | 10/20 | **19/20** | −9 |
| **Overall** | 219/418 | **241/418** | −22 |

The split follows the scorer type. **Machine-verified aggregate** — the four categories scored by code execution or
exact numeric match: **Gamma 50/136 (37%) vs Qwen 25/136 (18%), a 2.0× margin.** Qwen's overall lead comes from the
rubric/keyword-scored breadth categories (everyday reasoning, systems/devops, SQL shape, explanation, narrative,
instruction) plus word problems. Both facts are real; neither cancels the other.

The local code-slice numbers are prompt-surface-sensitive. The Qwen artifact's generations on the full-program and
algorithms slices frequently reproduce the signature and docstring without an implementation body (15/20 full-program
failures are `IndentationError: expected an indented block after function definition`), with wrong-algorithm and
name/signature-mismatch failures making up most of the rest. This does not contradict Qwen's stronger published
public HumanEval performance; it is evidence about this battery under this harness, scored identically for both
models.

Epistemic internal probe:

| Metric | Gamma | Qwen local reference |
|---|---:|---:|
| Epistemic total | **23/32** | 17/32 |
| Abstention | **10/10** | 2/10 |
| Contradiction | **4/6** | 1/6 |
| False premise | 5/10 | **8/10** |
| Sycophancy | 4/6 | **6/6** |

## Binding Denominators

Gamma chat-surface held-out name probes use N=128 at each depth:

| Suite | Depth 1 | Depth 2 | Depth 3 | Depth 4 | Depth 5 |
|---|---:|---:|---:|---:|---:|
| Held-out names, trained query surface | 125/128 | 127/128 | 118/128 | 110/128 | 94/128 |
| Held-out names, unseen query phrasing | 127/128 | 120/128 | 111/128 | 90/128 | 69/128 |
| Older-relation held-out names | 118/128 | 116/128 | 108/128 | 88/128 | 76/128 |
| Non-name entity control (colored cards) | 77/128 | 73/128 | 69/128 | 57/128 | 50/128 |

Chance is 1/(depth+1): the answer is one of the chain's depth+1 items. The non-name control suite is included in the
Gamma artifact (the Qwen artifact covers only the three name suites) and is reported on the card as a limitation:
binding strength is name-based, and the entity control runs 39–60% — above chance at every depth, well below the
name suites. The Qwen artifact stores rounded percentages without per-row counts; Gamma's cells are exact counts.

## Privacy And IP Boundary

Included:

- Eval task definitions for the 418 battery
- Eval scripts and scorer logic
- Public benchmark raw outputs
- Internal battery raw outputs
- Local Qwen reference outputs

Not included:

- SFT training rows
- Pretraining corpus files
- Private service hostnames or IPs
- API tokens or credentials
- Full corpus mixture internals beyond the release-card training-budget disclosure