| # Release Checklist — Piko-9b 1.0.0 |
|
|
| Run 2026-07-29. Every line is either **PASS** with evidence, **FAIL** with the reason, or |
| **NOT RUN** with why. Nothing is marked done on the strength of an assumption. |
|
|
| ## Model integrity |
|
|
| | Check | Status | Evidence | |
| |---|---|---| |
| | `config.json` parses | **PASS** | `scripts/audit_repository.py` | |
| | Tokenizer loads | **PASS** | `AutoTokenizer` — `TokenizersBackend` | |
| | Processor loads | **PASS** | `Qwen3VLProcessor`, **requires `torchvision`** | |
| | Model loads without `trust_remote_code` | **PASS** | `AutoModelForMultimodalLM`, no `auto_map`, no `.py` in repo | |
| | Parameter count verified | **PASS** | 9,653,104,368 from safetensors headers | |
| | Shard integrity | **PASS** | 11 shards, 775 tensors, all BF16, 0 unexplained keys | |
| | Weights unchanged from previous release | **PASS** | Byte-identical; this release changes documentation only | |
|
|
| ## Provenance |
|
|
| | Check | Status | Evidence | |
| |---|---|---| |
| | Base model identified | **PASS** | `deepreinforce-ai/Ornith-1.0-9B` + `Qwen/Qwen3.5-9B` | |
| | Lineage verified at tensor level | **PASS** | 427/427 language + 348/348 vision tensors matched bitwise | |
| | Adapter presence tested | **PASS** | Both probed adapters **not merged**; `\|cand−lang\| = 0` exactly | |
| | Ornith's own base model | **NOT VERIFIED** | Card declares none; structural identity to Qwen3.5-9B is an inference | |
| | Published checkpoint byte-compared to Hub | **NOT RUN** | Verified by file name and byte size only; 21 GB re-download not performed | |
|
|
| ## Documentation honesty |
|
|
| | Check | Status | Evidence | |
| |---|---|---| |
| | No fabricated numbers | **PASS** | `scripts/validate_model_card.py` cross-checks every table percentage | |
| | Unrun benchmarks labelled | **PASS** | 9 standard benchmarks marked **Not run** | |
| | Base-model scores not presented as Piko's | **PASS** | Misattributed scores removed; explicit warning added | |
| | Unsupported capabilities not advertised | **PASS** | Video/audio marked unsupported; identity caveat stated | |
| | Failures recorded, not omitted | **PASS** | 5 suite failures adjudicated in the card; 32K OOM recorded | |
| | Model card metadata validates | **PASS** | 0 errors, 0 warnings | |
| | Relative links resolve | **PASS** | Checked by `validate_model_card.py` | |
|
|
| ## Hygiene |
|
|
| | Check | Status | Evidence | |
| |---|---|---| |
| | No secrets | **PASS** | Token/key/private-key scan over all text files: 0 hits | |
| | No absolute local paths in shipped files | **PASS** | Result files normalised to repo ids; remaining mentions are deliberate documentation | |
| | `config.json` path leak fixed | **PASS (proposed)** | Correction specified in audit §8; **not yet applied to the Hub** | |
| | Regression test for path leak | **PASS** | `test_config_contains_no_absolute_paths` | |
| | `.gitignore` excludes weights | **PASS** | | |
|
|
| ## Code quality |
|
|
| | Check | Status | Evidence | |
| |---|---|---| |
| | Syntax check, all Python | **PASS** | `compileall` clean | |
| | Lint | **PASS** | `ruff check` — all checks passed | |
| | Format | **PASS** | `ruff format --check` — 27 files formatted | |
| | Fast tests | **PASS** | **10 passed**, 14 deselected, no weights or GPU needed | |
| | Slow tests | **NOT RUN** | 14 collected; require `PIKO_MODEL_PATH`. Behaviour they assert was verified directly by `scripts/validate_inference.py` | |
|
|
| ## Functional validation |
|
|
| | Check | Status | Evidence | |
| |---|---|---| |
| | Text generation | **PASS** | `reports/inference_validation.json` — 12/12 | |
| | Image / OCR | **PASS** | Read "NORTHGATE HARDWARE" and 22.75 correctly | |
| | Document JSON extraction | **PASS** | 10/10 on custom suite | |
| | Chart reading | **PASS** | Read all four bar values correctly | |
| | Multi-turn | **PASS** | | |
| | Batch inference | **PASS** | | |
| | Greedy determinism | **PASS** | Identical across repeats | |
| | Long context | **PASS** | Needle found at 14,429 tokens; suite passes at 2K/8K/32K | |
| | Streaming | **PASS (code)** | Implemented in `examples/inference_cli.py`; not separately measured | |
| | bfloat16 unquantized | **NOT RUN** | Needs ~22 GB; test GPU has 15.92 GB | |
| | float16 | **NOT RUN** | Supported by flag; not exercised | |
| | 8-bit | **NOT RUN** | | |
| | CPU loading | **NOT RUN** | ~38.6 GB float32 and no fast path; not viable | |
| | `device_map="auto"` | **FAIL — by design** | Produces `!!!!!` degenerate output; documented as unsupported | |
| | Video input | **NOT RUN** | No fixture built | |
|
|
| ## Evaluation |
|
|
| | Check | Status | Evidence | |
| |---|---|---| |
| | Smoke evaluation | **PASS** | **8/8 required** | |
| | Custom suite, Piko-9b | **PASS** | **65/70** | |
| | Custom suite, base model | **PASS** | **63/70**, identical settings | |
| | Like-for-like comparison | **PASS** | Settings matched; tool would have refused otherwise | |
| | Statistical honesty | **PASS** | Wilson intervals; every category reads *No significant difference* | |
| | Standard public benchmarks | **NOT RUN** | 1–3 h per model each, ×2 for baseline; scripts provided | |
|
|
| ## Performance |
|
|
| | Check | Status | Evidence | |
| |---|---|---| |
| | Cold load | **PASS** | 101–119 s NVMe | |
| | Peak VRAM | **PASS** | 7.37 GB resident, 13.4 GB at 8K×4 | |
| | Host RAM | **PASS** | 1.28 GB RSS | |
| | Throughput, 3 contexts × 3 batches | **PASS** | 9/9 configurations | |
| | Image preprocessing | **PASS** | 13 ms median | |
| | 32,768-token single forward | **FAIL — recorded** | OOM on 15.92 GB; 32K retrieval via `generate()` works | |
|
|
| ## Quantization |
|
|
| | Check | Status | Evidence | |
| |---|---|---| |
| | 4-bit NF4 | **PASS** | All results measured this way; vision path intact | |
| | 8-bit | **NOT RUN** | | |
| | GGUF / AWQ / GPTQ | **NOT PRODUCED** | Scripts provided, unexecuted, documented as unvalidated | |
| | Quantized artefact published | **NO** | Nothing to validate, nothing claimed | |
|
|
| ## Blocking items before publishing |
|
|
| 1. **Apply the `config.json` correction on the Hub** — remove the two absolute paths from |
| `piko_composition` and set `transformers_version` to `5.5.0`. This is the only change that |
| touches the model repository itself, and it is live right now. |
| 2. **Decide what the repository is for.** The evidence says Piko-9b performs the same as |
| `Qwen/Qwen3.5-9B` within measurement error. That is publishable, but only with the honesty this |
| release adopts. Do not restore the old benchmark table. |
|
|
| ## Not blocking, worth doing |
|
|
| * Run one standard benchmark pair (GSM8K 200 items is the cheapest useful one). |
| * Test the six unmerged Piko adapters against the composition — they may improve it, and nobody |
| has checked. |
| * Build a video fixture and either support or explicitly disable the video path. |
| * Test image inputs outside rendered documents. |
|
|
