evaluation/README.md

# Evaluation

Everything here exists to answer one question honestly: **how does Piko-9b compare to the model it
was built from, measured the same way, on the same hardware, on the same day?**

## Why the base-model comparison matters here

Piko-9b's published weights are a splice: the language backbone comes from a fine-tune of
`deepreinforce-ai/Ornith-1.0-9B`, the vision tower is copied verbatim from `Qwen/Qwen3.5-9B`
([lineage](../reports/lineage_analysis.md)). That makes two comparisons meaningful, and they
answer different questions:

| Comparison | Question it answers |
|---|---|
| Piko-9b vs `Qwen/Qwen3.5-9B` | Did the composition help or hurt relative to the model whose vision tower it borrowed? |
| Piko-9b vs `deepreinforce-ai/Ornith-1.0-9B` | What did the WraithFast fine-tuning chain change? |

The configured default is `Qwen/Qwen3.5-9B`, because that is the model whose vision tower Piko-9b
ships and therefore the fairest reference for any multimodal claim.

## Rules this harness enforces

1. **Identical everything.** Both models run with the same prompts, chat template application,
   precision, quantization, batch size, decoding parameters, seed, and dataset slice. The config
   files differ only in `model_id`.
2. **No placement shortcuts.** Every model is loaded fully resident on one GPU. CPU offload
   corrupts Piko-9b's linear-attention state and produces a constant token — a broken run that
   *looks* like a catastrophic benchmark score. See
   [troubleshooting](../docs/troubleshooting.md).
3. **Failures are recorded, never dropped.** Every result file has a `failures` list. A benchmark
   that could not run appears as `"Not run"` with a reason, never as a blank or a plausible-looking
   number.
4. **Provenance in every file.** Model id, revision, timestamp, hardware, OS, Python, torch,
   transformers, precision, quantization, batch size, generation parameters, dataset version,
   seed, example count, failure count, and any deviation from the standard benchmark protocol.

## Layout

```
evaluation/
├── README.md                  this file
├── requirements.txt           evaluation-only dependencies
├── run_all.py                 full sweep across both models
├── run_smoke_eval.py          ~5 minute sanity check
├── compare_results.py         builds the side-by-side table
├── configs/
│   ├── piko_9b.yaml
│   └── base_model.yaml
├── prompts/                   shared prompt templates and fixtures
├── results/                   JSON output, one file per model per suite
└── custom_suite/              65-case deterministic regression suite
```

## Running

Sanity check first — it catches a broken environment in minutes rather than hours:

```bash
python evaluation/run_smoke_eval.py --config evaluation/configs/piko_9b.yaml
```

The custom regression suite (no dataset downloads, fully deterministic):

```bash
python evaluation/custom_suite/build_assets.py
python evaluation/custom_suite/run_custom_eval.py \
  --model Dexy2/Piko-9b --quantization 4bit \
  --output evaluation/results/custom_suite_piko9b.json
```

Both models, then compare:

```bash
python evaluation/run_all.py --config evaluation/configs/piko_9b.yaml
python evaluation/run_all.py --config evaluation/configs/base_model.yaml
python evaluation/compare_results.py \
  --candidate evaluation/results/custom_suite_piko9b.json \
  --baseline  evaluation/results/custom_suite_qwen35.json \
  --output    evaluation/results/comparison.md
```

## Cost before you start

Measured on an RTX 5070 Ti (17.1 GB), 4-bit NF4, weights on NVMe.

| Suite | Examples | Approx. runtime per model | VRAM |
|---|---:|---|---:|
| `run_smoke_eval.py` | 8 | 3–6 min | 8 GB |
| `custom_suite` | 65 | 35–60 min | 8 GB |
| GSM8K (200-item slice) | 200 | 1.5–2.5 h | 8 GB |
| MMLU-Pro (200-item slice) | 200 | 1–2 h | 8 GB |
| IFEval (200-item slice) | 200 | 1–2 h | 8 GB |
| OCRBench / DocVQA / ChartQA slices | 200 each | 1.5–3 h each | 9 GB |

Add ~10 minutes of cold-load time per model per invocation, and **double everything** because
each number needs a baseline run to mean anything.

Loading from an external USB disk adds 10–20 minutes per load. Copy the weights to internal NVMe
first.

## Which benchmarks were actually run

See the results table in the [model card](../README.md). Anything not executed is labelled
**Not run** there and in `evaluation/results/`, with the reason. The scripts for unexecuted
benchmarks are present and runnable — they were not executed here for time and hardware reasons,
not because they are unfinished.

## A note on the previously published numbers

The nine benchmark scores in the original Piko-9b model card were measured on
`wraithfast-phase14-100k-full-ft` + `adapters/wraithfast-phase15-150k-qlora`, a text-only
checkpoint that predates the vision composition and contains none of the Piko training stages.
They are not results for the published model and are not reproduced here. Details:
[`reports/repository_audit.md`](../reports/repository_audit.md) §5.