CONTRIBUTING.md

Contributing

The rule that matters

No number enters the documentation unless a committed result file backs it.

The original Piko-9b release published nine benchmark scores that had been measured on a different checkpoint — a text-only model that predated the vision composition and contained none of the Piko training stages. That is the specific failure this repository is built to prevent.

So:

• If you did not run it, write "Not run" and say why.
• If it failed, record the failure. Every runner here has a failures list for exactly this.
• If a number came from an upstream model card, attribute it to that model, not to Piko-9b.
• If you cannot verify something, write "Could not be verified."

make validate-model-card enforces part of this automatically: it cross-references every percentage in a results table against the committed JSON under evaluation/results/ and benchmarks/results/.

Setup

python -m venv .venv && source .venv/bin/activate
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
make install-dev

Before opening a pull request

make check      # lint + fast tests + model-card validation, no weights needed

If your change touches inference, evaluation, or the checkpoint:

export PIKO_MODEL_PATH=/path/to/local/checkpoint
make test-all
make smoke-eval

Copy the checkpoint to internal NVMe first. Loading 21 GB from an external USB disk takes 10–20 minutes per run; from NVMe it takes about 100 seconds.

Test tiers

Tier	Marker	Needs	Runs in CI
Fast	(none)	config and tokenizer files only	Yes, every commit
Heavy	@pytest.mark.slow	the 9.65 B checkpoint and a CUDA GPU	Manual dispatch only

CI must never download the full model on an ordinary commit. Keep the fast tier fast and weight-free; put anything that loads weights behind @pytest.mark.slow.

Adding an evaluation case

1. Add a line to the right evaluation/custom_suite/cases/*.jsonl.
2. Prefer a deterministic check. The available types are listed in evaluation/custom_suite/README.md.
3. If you need an image, draw it in build_assets.py. Do not download fixtures — the original project's vision benchmark died permanently because a remote host's TLS certificate changed.
4. Re-run the category and commit the result file alongside the case.

If you change a grading rule, re-run and report both the old and new scores. Adjusting a grader after seeing results is how honest suites quietly become dishonest ones. Say what you changed and why.

Changing the checkpoint

Any change to weights or config.json requires:

1. make audit — regenerates reports/repository_audit.json; must report zero secrets and zero absolute paths.
2. make lineage — re-verifies provenance by tensor comparison.
3. make smoke-eval — the first check catches the degenerate-output failure mode.
4. An entry in CHANGELOG.md.

Style

ruff for linting and formatting; run make format. Beyond that: write comments that explain why, not what. The most valuable comments in this repository are the ones warning that device_map="auto" silently corrupts this architecture — that is not deducible from the code.

What not to do

• Do not add trust_remote_code=True to examples. It is unnecessary and teaches a bad habit.
• Do not use device_map="auto" anywhere. It is the single most likely way to break this model.
• Do not commit weights, tokens, or absolute local paths.
• Do not describe untested capabilities as supported. Video input, for instance, has inherited metadata and a preprocessor config but was never exercised — so it is documented as untested, not as a feature.