CONTRIBUTING.md

# Contributing

## The rule that matters

**No number enters the documentation unless a committed result file backs it.**

The original Piko-9b release published nine benchmark scores that had been measured on a
*different checkpoint* — a text-only model that predated the vision composition and contained none
of the Piko training stages. That is the specific failure this repository is built to prevent.

So:

* If you did not run it, write **"Not run"** and say why.
* If it failed, record the failure. Every runner here has a `failures` list for exactly this.
* If a number came from an upstream model card, attribute it to that model, not to Piko-9b.
* If you cannot verify something, write **"Could not be verified."**

`make validate-model-card` enforces part of this automatically: it cross-references every
percentage in a results table against the committed JSON under `evaluation/results/` and
`benchmarks/results/`.

## Setup

```bash
python -m venv .venv && source .venv/bin/activate
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
make install-dev
```

## Before opening a pull request

```bash
make check      # lint + fast tests + model-card validation, no weights needed
```

If your change touches inference, evaluation, or the checkpoint:

```bash
export PIKO_MODEL_PATH=/path/to/local/checkpoint
make test-all
make smoke-eval
```

**Copy the checkpoint to internal NVMe first.** Loading 21 GB from an external USB disk takes
10–20 minutes per run; from NVMe it takes about 100 seconds.

## Test tiers

| Tier | Marker | Needs | Runs in CI |
|---|---|---|---|
| Fast | *(none)* | config and tokenizer files only | Yes, every commit |
| Heavy | `@pytest.mark.slow` | the 9.65 B checkpoint and a CUDA GPU | Manual dispatch only |

CI must never download the full model on an ordinary commit. Keep the fast tier fast and
weight-free; put anything that loads weights behind `@pytest.mark.slow`.

## Adding an evaluation case

1. Add a line to the right `evaluation/custom_suite/cases/*.jsonl`.
2. Prefer a deterministic check. The available types are listed in
   [`evaluation/custom_suite/README.md`](evaluation/custom_suite/README.md).
3. If you need an image, draw it in `build_assets.py`. Do not download fixtures — the original
   project's vision benchmark died permanently because a remote host's TLS certificate changed.
4. Re-run the category and commit the result file alongside the case.

If you change a grading rule, **re-run and report both the old and new scores**. Adjusting a
grader after seeing results is how honest suites quietly become dishonest ones. Say what you
changed and why.

## Changing the checkpoint

Any change to weights or `config.json` requires:

1. `make audit` — regenerates `reports/repository_audit.json`; must report zero secrets and zero
   absolute paths.
2. `make lineage` — re-verifies provenance by tensor comparison.
3. `make smoke-eval` — the first check catches the degenerate-output failure mode.
4. An entry in `CHANGELOG.md`.

## Style

`ruff` for linting and formatting; run `make format`. Beyond that: write comments that explain
*why*, not *what*. The most valuable comments in this repository are the ones warning that
`device_map="auto"` silently corrupts this architecture — that is not deducible from the code.

## What not to do

* Do not add `trust_remote_code=True` to examples. It is unnecessary and teaches a bad habit.
* Do not use `device_map="auto"` anywhere. It is the single most likely way to break this model.
* Do not commit weights, tokens, or absolute local paths.
* Do not describe untested capabilities as supported. Video input, for instance, has inherited
  metadata and a preprocessor config but was never exercised — so it is documented as untested,
  not as a feature.