File size: 1,782 Bytes
f1fc3a0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | # Contributing to PACTBench
## Development setup
```bash
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt
```
Before submitting a change, run:
```bash
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v
python3 pact/validate_pactbench.py --require_v5
```
Shell entry points must also pass `bash -n`.
## Code and experiment changes
- Put shared Python implementation in `pact/`.
- Keep per-game files in `games/<game>/` as thin wrappers and configuration.
- Do not hand-edit published aggregate values in `registry/registry.json`.
- A result is publishable only when its code, configuration, ordered rows,
metrics, model identity, and source identities are available to the gate.
- Keep unreviewed runs outside the published registry and README results. Follow
the review-isolation procedure in `docs/MAINTAINERS.md`.
## Repository hygiene
Do not commit:
- API keys, access tokens, private endpoints, or credential files;
- machine-specific absolute paths in code or documentation;
- raw caches that can be rebuilt from documented inputs;
- internal reviews, meeting notes, temporary reports, or backup copies;
- new headline numbers that have not passed maintainer review.
Use environment variables for model paths, service URLs, and credentials.
Keep one canonical implementation for every runnable tool; avoid copied scripts
that can silently diverge.
## Documentation
Update the narrowest relevant document. Dataset facts belong in `docs/DATA.md`,
experiment definitions and results in `docs/EXPERIMENTS.md`, commands in
`docs/REPRODUCE.md`, and game-specific entry points in the corresponding game
README. Historical reasoning and future experiment wish lists do not belong in
the release documentation.
|