PACTBench
PACTBench evaluates a layered boss-control system across V Rising, Hollow Knight, and The Binding of Isaac:
- L1 converts causal RGB history into a compact, decision-facing belief.
- L2 selects among anonymous skill options from the belief and the visible skill-mechanics descriptions.
This directory is the release unit for code, checkpoints, published metrics,
and reproducibility metadata. Machine-readable publication state lives in
registry/registry.json; prose never overrides the
registry or the referenced metric files.
Current release
The published snapshot uses fight-disjoint splits, the compact belief interface, balanced-v3 L2 sampling, HP-free-v6 prompts, anonymous option IDs, Gemma-4-E2B-it at temperature zero, and deterministic sample seed 0.
L1 official evaluation
| Game | Test rows | Distance MAE | Angle MAE | Distance-bin accuracy | Angle-bin accuracy |
|---|---|---|---|---|---|
| V Rising | 8,075 | 0.2620 | 11.7599° | 0.8320 | 0.7201 |
| Hollow Knight | 8,360 | 0.4680 | 3.6527° | 0.7261 | 0.9646 |
| Isaac | 1,784 | 9.0263 px | 29.6388° | 0.9036 | 0.8352 |
Isaac distance is measured in pixels and is not directly comparable to the engine-unit distance used by the other two games.
L2 natural-ready semantic intervention
| Game | Valid paired states | Mean menu size | Menu-aware chance | Description-follow |
|---|---|---|---|---|
| V Rising | 200 | 4.875 | 21.93% | 67.0% |
| Hollow Knight | 200 | 5.370 | 18.78% | 79.5% |
| Isaac | 122 | 2.238 | 46.04% | 83.6% |
The 200 source states are fixed before L2 outputs are read. Isaac has 122
valid semantic-swap pairs because the predefined intervention requires at
least two tracker-ready options; singleton menus are excluded before outcome
aggregation. The fixed two-choice controls are reported separately in
docs/EXPERIMENTS.md.
The L1 visual comparison is a comparison of complete training schemes: PACT-L1 is task-trained end to end, whereas DINOv2-S and VideoMAE-B are frozen pretrained backbones with trained probes. It is not an isolated backbone claim.
V Rising also includes a separate accepted L1 comparison experiment that trains
DINOv2-S end to end on the same ordered 18,068/8,075 rows and the matched PACT
two-stage recipe. In this configuration it remains less accurate and 48.79%
slower at batch 1 than PACT-L1; the evidence and claim scope are recorded in
docs/EXPERIMENTS.md.
Quick start
cd pactbench
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt
python3 registry/status.py
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v
python3 pact/validate_pactbench.py --require_v5
The full artifact gate should end with:
PACTBench gate: PASS (7382 checks)
Large raw data and RGB caches are exposed through data -> ../data and
out -> ../out. See the reproducibility guide before rebuilding or moving
them.
Documentation
| Document | Purpose |
|---|---|
docs/DATA.md |
Data source, layout, schema, split, and audit rules |
docs/EXPERIMENTS.md |
Exact protocols, published results, and claim scope |
docs/REPRODUCE.md |
Environment, commands, expected artifacts, and release gate |
docs/MAINTAINERS.md |
Handoff, review isolation, registry update, and release procedure |
games/vrising/README.md |
V Rising entry points and artifacts |
games/hk/README.md |
Hollow Knight entry points and artifacts |
games/isaac/README.md |
Isaac entry points and artifacts |
pipelines/l2/llm_judge/README.md |
External decision-quality judge protocol |
Repository layout
pactbench/
├── games/{vrising,hk,isaac}/ # per-game configuration and thin entry points
├── pipelines/ # shared shell pipelines
├── pact/ # shared Python implementation
├── ckpt/ # published L1 checkpoints
├── results/ # published row-level and aggregate artifacts
├── registry/ # machine-readable publication snapshot
├── schemas/ # cross-game schemas
├── tests/ # regression and provenance tests
├── data -> ../data # large source/processed data
└── out -> ../out # rebuildable caches
The decision-quality LLM judge is intentionally run on a separate machine. Its absence from the local deterministic gate does not change the publication status of the local L1 and L2 intervention artifacts.