| # Contributing to PACTBench |
|
|
| ## Development setup |
|
|
| ```bash |
| python3 -m venv .venv |
| source .venv/bin/activate |
| python3 -m pip install -r requirements.txt |
| ``` |
|
|
| Before submitting a change, run: |
|
|
| ```bash |
| PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v |
| python3 pact/validate_pactbench.py --require_v5 |
| ``` |
|
|
| Shell entry points must also pass `bash -n`. |
|
|
| ## Code and experiment changes |
|
|
| - Put shared Python implementation in `pact/`. |
| - Keep per-game files in `games/<game>/` as thin wrappers and configuration. |
| - Do not hand-edit published aggregate values in `registry/registry.json`. |
| - A result is publishable only when its code, configuration, ordered rows, |
| metrics, model identity, and source identities are available to the gate. |
| - Keep unreviewed runs outside the published registry and README results. Follow |
| the review-isolation procedure in `docs/MAINTAINERS.md`. |
|
|
| ## Repository hygiene |
|
|
| Do not commit: |
|
|
| - API keys, access tokens, private endpoints, or credential files; |
| - machine-specific absolute paths in code or documentation; |
| - raw caches that can be rebuilt from documented inputs; |
| - internal reviews, meeting notes, temporary reports, or backup copies; |
| - new headline numbers that have not passed maintainer review. |
|
|
| Use environment variables for model paths, service URLs, and credentials. |
| Keep one canonical implementation for every runnable tool; avoid copied scripts |
| that can silently diverge. |
|
|
| ## Documentation |
|
|
| Update the narrowest relevant document. Dataset facts belong in `docs/DATA.md`, |
| experiment definitions and results in `docs/EXPERIMENTS.md`, commands in |
| `docs/REPRODUCE.md`, and game-specific entry points in the corresponding game |
| README. Historical reasoning and future experiment wish lists do not belong in |
| the release documentation. |
|
|
