evaluation/README.md

Evaluation

Everything here exists to answer one question honestly: how does Piko-9b compare to the model it was built from, measured the same way, on the same hardware, on the same day?

Why the base-model comparison matters here

Piko-9b's published weights are a splice: the language backbone comes from a fine-tune of deepreinforce-ai/Ornith-1.0-9B, the vision tower is copied verbatim from Qwen/Qwen3.5-9B (lineage). That makes two comparisons meaningful, and they answer different questions:

Comparison	Question it answers
Piko-9b vs Qwen/Qwen3.5-9B	Did the composition help or hurt relative to the model whose vision tower it borrowed?
Piko-9b vs deepreinforce-ai/Ornith-1.0-9B	What did the WraithFast fine-tuning chain change?

The configured default is Qwen/Qwen3.5-9B, because that is the model whose vision tower Piko-9b ships and therefore the fairest reference for any multimodal claim.

Rules this harness enforces

1. Identical everything. Both models run with the same prompts, chat template application, precision, quantization, batch size, decoding parameters, seed, and dataset slice. The config files differ only in model_id.
2. No placement shortcuts. Every model is loaded fully resident on one GPU. CPU offload corrupts Piko-9b's linear-attention state and produces a constant token — a broken run that looks like a catastrophic benchmark score. See troubleshooting.
3. Failures are recorded, never dropped. Every result file has a failures list. A benchmark that could not run appears as "Not run" with a reason, never as a blank or a plausible-looking number.
4. Provenance in every file. Model id, revision, timestamp, hardware, OS, Python, torch, transformers, precision, quantization, batch size, generation parameters, dataset version, seed, example count, failure count, and any deviation from the standard benchmark protocol.

Layout

evaluation/
├── README.md                  this file
├── requirements.txt           evaluation-only dependencies
├── run_all.py                 full sweep across both models
├── run_smoke_eval.py          ~5 minute sanity check
├── compare_results.py         builds the side-by-side table
├── configs/
│   ├── piko_9b.yaml
│   └── base_model.yaml
├── prompts/                   shared prompt templates and fixtures
├── results/                   JSON output, one file per model per suite
└── custom_suite/              65-case deterministic regression suite

Running

Sanity check first — it catches a broken environment in minutes rather than hours:

python evaluation/run_smoke_eval.py --config evaluation/configs/piko_9b.yaml

The custom regression suite (no dataset downloads, fully deterministic):

python evaluation/custom_suite/build_assets.py
python evaluation/custom_suite/run_custom_eval.py \
  --model Dexy2/Piko-9b --quantization 4bit \
  --output evaluation/results/custom_suite_piko9b.json

Both models, then compare:

python evaluation/run_all.py --config evaluation/configs/piko_9b.yaml
python evaluation/run_all.py --config evaluation/configs/base_model.yaml
python evaluation/compare_results.py \
  --candidate evaluation/results/custom_suite_piko9b.json \
  --baseline  evaluation/results/custom_suite_qwen35.json \
  --output    evaluation/results/comparison.md

Cost before you start

Measured on an RTX 5070 Ti (17.1 GB), 4-bit NF4, weights on NVMe.

Suite	Examples	Approx. runtime per model	VRAM
run_smoke_eval.py	8	3–6 min	8 GB
custom_suite	65	35–60 min	8 GB
GSM8K (200-item slice)	200	1.5–2.5 h	8 GB
MMLU-Pro (200-item slice)	200	1–2 h	8 GB
IFEval (200-item slice)	200	1–2 h	8 GB
OCRBench / DocVQA / ChartQA slices	200 each	1.5–3 h each	9 GB

Add ~10 minutes of cold-load time per model per invocation, and double everything because each number needs a baseline run to mean anything.

Loading from an external USB disk adds 10–20 minutes per load. Copy the weights to internal NVMe first.

Which benchmarks were actually run

See the results table in the model card. Anything not executed is labelled Not run there and in evaluation/results/, with the reason. The scripts for unexecuted benchmarks are present and runnable — they were not executed here for time and hardware reasons, not because they are unfinished.

A note on the previously published numbers

The nine benchmark scores in the original Piko-9b model card were measured on wraithfast-phase14-100k-full-ft + adapters/wraithfast-phase15-150k-qlora, a text-only checkpoint that predates the vision composition and contains none of the Piko training stages. They are not results for the published model and are not reproduced here. Details: reports/repository_audit.md §5.