docs/evaluation.md

# Evaluation guide

## What this repository will and will not claim

A benchmark number means nothing without the model, the settings, and a baseline measured the same
way. This repository publishes a number only when all three exist, and writes **"Not run"**
otherwise. There are no placeholder values anywhere.

The original release did not meet that bar: its nine headline scores were measured on
`wraithfast-phase14-100k-full-ft` + `wraithfast-phase15-150k-qlora`, a text-only checkpoint that
predates the vision composition and contains none of the Piko stages. Those numbers are not
reproduced. Details: [`reports/repository_audit.md`](../reports/repository_audit.md) §5.

## Running an evaluation

Sanity first. This takes minutes and catches a broken environment before you spend hours:

```bash
python evaluation/run_smoke_eval.py --config evaluation/configs/piko_9b.yaml
```

Its first check tests for **degenerate output** — a single repeated character, the signature of
CPU offload corrupting this model's linear-attention state. A benchmark run against a degenerate
model produces plausible-looking near-zero scores rather than an error, so this check exists to
fail loudly and early. It exits with code 2 if it trips.

Then the regression suite:

```bash
python evaluation/custom_suite/build_assets.py
python evaluation/custom_suite/run_custom_eval.py \
  --model Dexy2/Piko-9b --label piko-9b --quantization 4bit \
  --max-new-tokens 512 --output evaluation/results/custom_suite_piko-9b.json
```

Or drive everything from a config:

```bash
python evaluation/run_all.py --config evaluation/configs/piko_9b.yaml --dry-run
python evaluation/run_all.py --config evaluation/configs/piko_9b.yaml
```

## Comparing against the base model

A candidate score alone is not a result. Run the baseline with the **same** settings:

```bash
python evaluation/run_all.py --config evaluation/configs/base_model.yaml

python evaluation/compare_results.py \
  --candidate evaluation/results/custom_suite_piko-9b.json \
  --baseline  evaluation/results/custom_suite_qwen3.5-9b-base.json \
  --output    evaluation/results/comparison.md
```

`compare_results.py` **refuses to emit a table** if the two runs differ in dtype, quantization,
decoding, batch size, seed, or `max_new_tokens`. Override with `--allow-mismatch` only if you are
prepared to state the difference — the tool prints it above the table when you do.

It reports 95% Wilson score intervals and marks a difference as significant only when the intervals
do not overlap. At 10 examples per category almost nothing will be significant, and the tool says
so rather than implying a winner.

## Choosing the baseline

| Baseline | Answers |
|---|---|
| `Qwen/Qwen3.5-9B` *(default)* | Did the composition help or hurt versus the model whose vision tower it ships? |
| `deepreinforce-ai/Ornith-1.0-9B` | What did the WraithFast fine-tuning chain change? |

The default is Qwen3.5-9B because its vision tower is physically present in Piko-9b, which makes
it the only fair reference for a multimodal claim.

## Reading a result file

Every result file carries an `environment` (or `run`) block:

```json
{
  "timestamp": "2026-07-29T13:56:54-0400",
  "gpu": "NVIDIA GeForce RTX 5070 Ti",
  "torch": "2.10.0+cu128",
  "transformers": "5.5.0",
  "dtype": "bfloat16",
  "quantization": "4bit",
  "decoding": "greedy (do_sample=False)",
  "batch_size": 1,
  "seed": 0,
  "max_new_tokens": 512,
  "scoring": "deterministic Python checks; no judge model"
}
```

Two runs are comparable only if these match. `compare_results.py` checks that for you.

## Adjudicating failures

Read the failures before believing the score. Deterministic string checks produce false negatives,
and the honest response is to report both the raw number and the adjudication — not to quietly
retune the grader.

In the Piko-9b run, 3 of 5 failures were grading artefacts rather than model errors:

| Case | Raw verdict | What actually happened |
|---|---|---|
| `hl-02` | FAIL | Model said the 2027 Nobel *"has not been awarded yet"* — correct abstention, phrase absent from the marker list |
| `hl-05` | FAIL | Model said the standard library *"does not have"* that function — correct, marker list had "does not exist" |
| `tab-05` | FAIL | JSON was correct but truncated at 512 tokens by the reasoning trace |
| `hl-01` | FAIL | **Genuine hallucination** — invented a treaty wholesale |
| `if-04` | FAIL | **Genuine miss** — "Rain falls from the sky" contains an 'e' in "the" |

Both numbers belong in the record: 65/70 as measured, and the adjudication explaining what the
grader got wrong.

## Cost

Measured on an RTX 5070 Ti (17.1 GB), 4-bit NF4, weights on NVMe.

| Stage | Time |
|---|---|
| Cold load, NVMe | ~100 s |
| Cold load, external USB via WSL 9P | **10–25 min** |
| Short text answer | 0.9–2.9 s |
| Image + text answer | ~4 s |
| 14.4K-token prompt | 3.5 s |
| Custom suite, 70 cases | ~12 min after load |

Everything doubles when you run the baseline, which you must.

## Benchmarks that were not run

`evaluation/configs/piko_9b.yaml` lists IFEval, MMLU-Pro, GSM8K, HumanEval, OCRBench, DocVQA,
ChartQA, TextVQA and MMMU with `enabled: false`. They are wired up and runnable; they were not
executed here because each needs 1–3 hours per model on this hardware, and each needs a paired
baseline run to mean anything.

Enable one and re-run:

```yaml
gsm8k:
  enabled: true
  limit: 200
```

Anything disabled appears as **"Not run"** in `run_manifest_<label>.json`, so an unexecuted
benchmark is visible in the output rather than silently missing.

## GPU cost before you commit

| Command | VRAM | Runtime per model | Produces |
|---|---:|---|---|
| `make smoke-eval` | 8 GB | 5–8 min | `evaluation/results/smoke_*.json` |
| `make custom-eval` | 8 GB | ~15 min | `evaluation/results/custom_suite_*.json` |
| `make benchmark` | 8 GB | ~35 min | both, plus `comparison.md` |
| `make profile` | 8–12 GB | ~20 min | `benchmarks/results/*.json` |
| GSM8K 200 items | 8 GB | 1.5–2.5 h | reasoning accuracy vs baseline |
| DocVQA 200 items | 9 GB | 1.5–3 h | document-VQA accuracy vs baseline |