# PACTBench

PACTBench evaluates a layered boss-control system across **V Rising**,
**Hollow Knight**, and **The Binding of Isaac**:

- **L1** converts causal RGB history into a compact, decision-facing belief.
- **L2** selects among anonymous skill options from the belief and the visible
  skill-mechanics descriptions.

This directory is the release unit for code, checkpoints, published metrics,
and reproducibility metadata. Machine-readable publication state lives in
[`registry/registry.json`](registry/registry.json); prose never overrides the
registry or the referenced metric files.

## Current release

The published snapshot uses fight-disjoint splits, the compact belief
interface, balanced-v3 L2 sampling, HP-free-v6 prompts, anonymous option IDs,
Gemma-4-E2B-it at temperature zero, and deterministic sample seed 0.

### L1 official evaluation

| Game | Test rows | Distance MAE | Angle MAE | Distance-bin accuracy | Angle-bin accuracy |
|---|---:|---:|---:|---:|---:|
| V Rising | 8,075 | 0.2620 | 11.7599° | 0.8320 | 0.7201 |
| Hollow Knight | 8,360 | 0.4680 | 3.6527° | 0.7261 | 0.9646 |
| Isaac | 1,784 | 9.0263 px | 29.6388° | 0.9036 | 0.8352 |

Isaac distance is measured in pixels and is not directly comparable to the
engine-unit distance used by the other two games.

### L2 natural-ready semantic intervention

| Game | Valid paired states | Mean menu size | Menu-aware chance | Description-follow |
|---|---:|---:|---:|---:|
| V Rising | 200 | 4.875 | 21.93% | 67.0% |
| Hollow Knight | 200 | 5.370 | 18.78% | 79.5% |
| Isaac | 122 | 2.238 | 46.04% | 83.6% |

The 200 source states are fixed before L2 outputs are read. Isaac has 122
valid semantic-swap pairs because the predefined intervention requires at
least two tracker-ready options; singleton menus are excluded before outcome
aggregation. The fixed two-choice controls are reported separately in
[`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md).

The L1 visual comparison is a comparison of complete training schemes:
PACT-L1 is task-trained end to end, whereas DINOv2-S and VideoMAE-B are frozen
pretrained backbones with trained probes. It is not an isolated backbone claim.

V Rising also includes a separate accepted L1 comparison experiment that trains
DINOv2-S end to end on the same ordered 18,068/8,075 rows and the matched PACT
two-stage recipe. In this configuration it remains less accurate and 48.79%
slower at batch 1 than PACT-L1; the evidence and claim scope are recorded in
[`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md#v-rising-end-to-end-dinov2-s-l1-comparison-experiment).

## Quick start

```bash
cd pactbench
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt

python3 registry/status.py
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v
python3 pact/validate_pactbench.py --require_v5
```

The full artifact gate should end with:

```text
PACTBench gate: PASS (7382 checks)
```

Large raw data and RGB caches are exposed through `data -> ../data` and
`out -> ../out`. See the reproducibility guide before rebuilding or moving
them.

## Documentation

| Document | Purpose |
|---|---|
| [`docs/DATA.md`](docs/DATA.md) | Data source, layout, schema, split, and audit rules |
| [`docs/EXPERIMENTS.md`](docs/EXPERIMENTS.md) | Exact protocols, published results, and claim scope |
| [`docs/REPRODUCE.md`](docs/REPRODUCE.md) | Environment, commands, expected artifacts, and release gate |
| [`docs/MAINTAINERS.md`](docs/MAINTAINERS.md) | Handoff, review isolation, registry update, and release procedure |
| [`games/vrising/README.md`](games/vrising/README.md) | V Rising entry points and artifacts |
| [`games/hk/README.md`](games/hk/README.md) | Hollow Knight entry points and artifacts |
| [`games/isaac/README.md`](games/isaac/README.md) | Isaac entry points and artifacts |
| [`pipelines/l2/llm_judge/README.md`](pipelines/l2/llm_judge/README.md) | External decision-quality judge protocol |

## Repository layout

```text
pactbench/
├── games/{vrising,hk,isaac}/  # per-game configuration and thin entry points
├── pipelines/                 # shared shell pipelines
├── pact/                      # shared Python implementation
├── ckpt/                      # published L1 checkpoints
├── results/                   # published row-level and aggregate artifacts
├── registry/                  # machine-readable publication snapshot
├── schemas/                   # cross-game schemas
├── tests/                     # regression and provenance tests
├── data -> ../data            # large source/processed data
└── out  -> ../out             # rebuildable caches
```

The decision-quality LLM judge is intentionally run on a separate machine.
Its absence from the local deterministic gate does not change the publication
status of the local L1 and L2 intervention artifacts.