# Evaluation guide ## What this repository will and will not claim A benchmark number means nothing without the model, the settings, and a baseline measured the same way. This repository publishes a number only when all three exist, and writes **"Not run"** otherwise. There are no placeholder values anywhere. The original release did not meet that bar: its nine headline scores were measured on `wraithfast-phase14-100k-full-ft` + `wraithfast-phase15-150k-qlora`, a text-only checkpoint that predates the vision composition and contains none of the Piko stages. Those numbers are not reproduced. Details: [`reports/repository_audit.md`](../reports/repository_audit.md) §5. ## Running an evaluation Sanity first. This takes minutes and catches a broken environment before you spend hours: ```bash python evaluation/run_smoke_eval.py --config evaluation/configs/piko_9b.yaml ``` Its first check tests for **degenerate output** — a single repeated character, the signature of CPU offload corrupting this model's linear-attention state. A benchmark run against a degenerate model produces plausible-looking near-zero scores rather than an error, so this check exists to fail loudly and early. It exits with code 2 if it trips. Then the regression suite: ```bash python evaluation/custom_suite/build_assets.py python evaluation/custom_suite/run_custom_eval.py \ --model Dexy2/Piko-9b --label piko-9b --quantization 4bit \ --max-new-tokens 512 --output evaluation/results/custom_suite_piko-9b.json ``` Or drive everything from a config: ```bash python evaluation/run_all.py --config evaluation/configs/piko_9b.yaml --dry-run python evaluation/run_all.py --config evaluation/configs/piko_9b.yaml ``` ## Comparing against the base model A candidate score alone is not a result. Run the baseline with the **same** settings: ```bash python evaluation/run_all.py --config evaluation/configs/base_model.yaml python evaluation/compare_results.py \ --candidate evaluation/results/custom_suite_piko-9b.json \ --baseline evaluation/results/custom_suite_qwen3.5-9b-base.json \ --output evaluation/results/comparison.md ``` `compare_results.py` **refuses to emit a table** if the two runs differ in dtype, quantization, decoding, batch size, seed, or `max_new_tokens`. Override with `--allow-mismatch` only if you are prepared to state the difference — the tool prints it above the table when you do. It reports 95% Wilson score intervals and marks a difference as significant only when the intervals do not overlap. At 10 examples per category almost nothing will be significant, and the tool says so rather than implying a winner. ## Choosing the baseline | Baseline | Answers | |---|---| | `Qwen/Qwen3.5-9B` *(default)* | Did the composition help or hurt versus the model whose vision tower it ships? | | `deepreinforce-ai/Ornith-1.0-9B` | What did the WraithFast fine-tuning chain change? | The default is Qwen3.5-9B because its vision tower is physically present in Piko-9b, which makes it the only fair reference for a multimodal claim. ## Reading a result file Every result file carries an `environment` (or `run`) block: ```json { "timestamp": "2026-07-29T13:56:54-0400", "gpu": "NVIDIA GeForce RTX 5070 Ti", "torch": "2.10.0+cu128", "transformers": "5.5.0", "dtype": "bfloat16", "quantization": "4bit", "decoding": "greedy (do_sample=False)", "batch_size": 1, "seed": 0, "max_new_tokens": 512, "scoring": "deterministic Python checks; no judge model" } ``` Two runs are comparable only if these match. `compare_results.py` checks that for you. ## Adjudicating failures Read the failures before believing the score. Deterministic string checks produce false negatives, and the honest response is to report both the raw number and the adjudication — not to quietly retune the grader. In the Piko-9b run, 3 of 5 failures were grading artefacts rather than model errors: | Case | Raw verdict | What actually happened | |---|---|---| | `hl-02` | FAIL | Model said the 2027 Nobel *"has not been awarded yet"* — correct abstention, phrase absent from the marker list | | `hl-05` | FAIL | Model said the standard library *"does not have"* that function — correct, marker list had "does not exist" | | `tab-05` | FAIL | JSON was correct but truncated at 512 tokens by the reasoning trace | | `hl-01` | FAIL | **Genuine hallucination** — invented a treaty wholesale | | `if-04` | FAIL | **Genuine miss** — "Rain falls from the sky" contains an 'e' in "the" | Both numbers belong in the record: 65/70 as measured, and the adjudication explaining what the grader got wrong. ## Cost Measured on an RTX 5070 Ti (17.1 GB), 4-bit NF4, weights on NVMe. | Stage | Time | |---|---| | Cold load, NVMe | ~100 s | | Cold load, external USB via WSL 9P | **10–25 min** | | Short text answer | 0.9–2.9 s | | Image + text answer | ~4 s | | 14.4K-token prompt | 3.5 s | | Custom suite, 70 cases | ~12 min after load | Everything doubles when you run the baseline, which you must. ## Benchmarks that were not run `evaluation/configs/piko_9b.yaml` lists IFEval, MMLU-Pro, GSM8K, HumanEval, OCRBench, DocVQA, ChartQA, TextVQA and MMMU with `enabled: false`. They are wired up and runnable; they were not executed here because each needs 1–3 hours per model on this hardware, and each needs a paired baseline run to mean anything. Enable one and re-run: ```yaml gsm8k: enabled: true limit: 200 ``` Anything disabled appears as **"Not run"** in `run_manifest_