Contributing to PACTBench

Development setup

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt

Before submitting a change, run:

PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v
python3 pact/validate_pactbench.py --require_v5

Shell entry points must also pass bash -n.

Code and experiment changes

Put shared Python implementation in pact/.
Keep per-game files in games/<game>/ as thin wrappers and configuration.
Do not hand-edit published aggregate values in registry/registry.json.
A result is publishable only when its code, configuration, ordered rows, metrics, model identity, and source identities are available to the gate.
Keep unreviewed runs outside the published registry and README results. Follow the review-isolation procedure in docs/MAINTAINERS.md.

Repository hygiene

Do not commit:

API keys, access tokens, private endpoints, or credential files;
machine-specific absolute paths in code or documentation;
raw caches that can be rebuilt from documented inputs;
internal reviews, meeting notes, temporary reports, or backup copies;
new headline numbers that have not passed maintainer review.

Use environment variables for model paths, service URLs, and credentials. Keep one canonical implementation for every runnable tool; avoid copied scripts that can silently diverge.

Documentation

Update the narrowest relevant document. Dataset facts belong in docs/DATA.md, experiment definitions and results in docs/EXPERIMENTS.md, commands in docs/REPRODUCE.md, and game-specific entry points in the corresponding game README. Historical reasoning and future experiment wish lists do not belong in the release documentation.