docs/evaluation.md

Evaluation guide

What this repository will and will not claim

A benchmark number means nothing without the model, the settings, and a baseline measured the same way. This repository publishes a number only when all three exist, and writes "Not run" otherwise. There are no placeholder values anywhere.

The original release did not meet that bar: its nine headline scores were measured on wraithfast-phase14-100k-full-ft + wraithfast-phase15-150k-qlora, a text-only checkpoint that predates the vision composition and contains none of the Piko stages. Those numbers are not reproduced. Details: reports/repository_audit.md §5.

Running an evaluation

Sanity first. This takes minutes and catches a broken environment before you spend hours:

python evaluation/run_smoke_eval.py --config evaluation/configs/piko_9b.yaml

Its first check tests for degenerate output — a single repeated character, the signature of CPU offload corrupting this model's linear-attention state. A benchmark run against a degenerate model produces plausible-looking near-zero scores rather than an error, so this check exists to fail loudly and early. It exits with code 2 if it trips.

Then the regression suite:

python evaluation/custom_suite/build_assets.py
python evaluation/custom_suite/run_custom_eval.py \
  --model Dexy2/Piko-9b --label piko-9b --quantization 4bit \
  --max-new-tokens 512 --output evaluation/results/custom_suite_piko-9b.json

Or drive everything from a config:

python evaluation/run_all.py --config evaluation/configs/piko_9b.yaml --dry-run
python evaluation/run_all.py --config evaluation/configs/piko_9b.yaml

Comparing against the base model

A candidate score alone is not a result. Run the baseline with the same settings:

python evaluation/run_all.py --config evaluation/configs/base_model.yaml

python evaluation/compare_results.py \
  --candidate evaluation/results/custom_suite_piko-9b.json \
  --baseline  evaluation/results/custom_suite_qwen3.5-9b-base.json \
  --output    evaluation/results/comparison.md

compare_results.py refuses to emit a table if the two runs differ in dtype, quantization, decoding, batch size, seed, or max_new_tokens. Override with --allow-mismatch only if you are prepared to state the difference — the tool prints it above the table when you do.

It reports 95% Wilson score intervals and marks a difference as significant only when the intervals do not overlap. At 10 examples per category almost nothing will be significant, and the tool says so rather than implying a winner.

Choosing the baseline

Baseline	Answers
Qwen/Qwen3.5-9B (default)	Did the composition help or hurt versus the model whose vision tower it ships?
deepreinforce-ai/Ornith-1.0-9B	What did the WraithFast fine-tuning chain change?

The default is Qwen3.5-9B because its vision tower is physically present in Piko-9b, which makes it the only fair reference for a multimodal claim.

Reading a result file

Every result file carries an environment (or run) block:

{
  "timestamp": "2026-07-29T13:56:54-0400",
  "gpu": "NVIDIA GeForce RTX 5070 Ti",
  "torch": "2.10.0+cu128",
  "transformers": "5.5.0",
  "dtype": "bfloat16",
  "quantization": "4bit",
  "decoding": "greedy (do_sample=False)",
  "batch_size": 1,
  "seed": 0,
  "max_new_tokens": 512,
  "scoring": "deterministic Python checks; no judge model"
}

Two runs are comparable only if these match. compare_results.py checks that for you.

Adjudicating failures

Read the failures before believing the score. Deterministic string checks produce false negatives, and the honest response is to report both the raw number and the adjudication — not to quietly retune the grader.

In the Piko-9b run, 3 of 5 failures were grading artefacts rather than model errors:

Case	Raw verdict	What actually happened
hl-02	FAIL	Model said the 2027 Nobel "has not been awarded yet" — correct abstention, phrase absent from the marker list
hl-05	FAIL	Model said the standard library "does not have" that function — correct, marker list had "does not exist"
tab-05	FAIL	JSON was correct but truncated at 512 tokens by the reasoning trace
hl-01	FAIL	Genuine hallucination — invented a treaty wholesale
if-04	FAIL	Genuine miss — "Rain falls from the sky" contains an 'e' in "the"

Both numbers belong in the record: 65/70 as measured, and the adjudication explaining what the grader got wrong.

Cost

Measured on an RTX 5070 Ti (17.1 GB), 4-bit NF4, weights on NVMe.

Stage	Time
Cold load, NVMe	~100 s
Cold load, external USB via WSL 9P	10–25 min
Short text answer	0.9–2.9 s
Image + text answer	~4 s
14.4K-token prompt	3.5 s
Custom suite, 70 cases	~12 min after load

Everything doubles when you run the baseline, which you must.

Benchmarks that were not run

evaluation/configs/piko_9b.yaml lists IFEval, MMLU-Pro, GSM8K, HumanEval, OCRBench, DocVQA, ChartQA, TextVQA and MMMU with enabled: false. They are wired up and runnable; they were not executed here because each needs 1–3 hours per model on this hardware, and each needs a paired baseline run to mean anything.

Enable one and re-run:

gsm8k:
  enabled: true
  limit: 200

Anything disabled appears as "Not run" in run_manifest_<label>.json, so an unexecuted benchmark is visible in the output rather than silently missing.

GPU cost before you commit

Command	VRAM	Runtime per model	Produces
make smoke-eval	8 GB	5–8 min	evaluation/results/smoke_*.json
make custom-eval	8 GB	~15 min	evaluation/results/custom_suite_*.json
make benchmark	8 GB	~35 min	both, plus comparison.md
make profile	8–12 GB	~20 min	benchmarks/results/*.json
GSM8K 200 items	8 GB	1.5–2.5 h	reasoning accuracy vs baseline
DocVQA 200 items	9 GB	1.5–3 h	document-VQA accuracy vs baseline