docs/REPRODUCE.md

# Reproducing the Published PACTBench Snapshot

Run all commands from the `pactbench` directory.

## 1. Environment

```bash
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
```

The checked-in environment was validated with Python 3.13 and CUDA 12.x. The
L1 path uses the versions pinned in `requirements.txt`.

Gemma serving uses a separate environment because the server requires a newer
Transformers release than the L1 environment:

```bash
python3 -m venv .venv-gemma
.venv-gemma/bin/python -m pip install \
  "transformers==5.13.0" "torch==2.5.1" accelerate pillow
```

## 2. Data and caches

Make `data` point to the directory containing the processed layouts documented
in `docs/DATA.md`. The repository currently uses:

```text
data -> ../data
out  -> ../out
```

Audit the available datasets before evaluation:

```bash
bash games/vrising/data/audit_dataset.sh
bash games/hk/data/audit_dataset.sh
bash games/isaac/data/audit_dataset.sh
```

RGB caches are rebuildable from processed videos:

```bash
python3 pact/build_rgb_frame_cache.py --workers 32
```

The canonical cache paths and their content identities are recorded in the
published registry. The full gate rejects missing, stale, or mismatched caches.

## 3. L1 evaluation

Evaluate the published checkpoints:

```bash
bash games/vrising/l1/eval_official_from_ckpt.sh
bash games/hk/l1/eval_official_from_ckpt.sh
bash games/isaac/l1/eval_official_from_ckpt.sh
```

Run the matched frozen-foundation comparisons:

```bash
bash games/vrising/l1/run_foundation_baselines.sh
bash games/hk/l1/run_foundation_baselines.sh
bash games/hk/l1/run_matched_foundation10k.sh
bash games/isaac/l1/run_foundation_baselines.sh
```

Run the accepted V Rising end-to-end DINOv2-S L1 comparison experiment:

```bash
bash games/vrising/l1/run_dinov2_end_to_end.sh
```

The wrapper preserves accepted artifacts unless `FORCE_RERUN=1` is set and
writes the canonical metrics, batch-1 latency, and comparison summary under
`results/vrising/eval_runs/L1-DINOV2-END-TO-END-MATCHED20/`.

Isaac's matched PACT checkpoint is produced by
`pipelines/l1/train/run_isaac_matched_after_weight_selection.sh` after the
validation-only distance-weight selection. Published checkpoints and metrics
can be validated without retraining them.

The image-input controls are:

```bash
bash games/vrising/l1/run_img_ablation.sh
bash games/hk/l1/run_img_ablation.sh
bash games/isaac/l1/run_img_ablation.sh
```

## 4. L2 service and evaluation

Start the pinned Gemma service with a local path or compatible model ID:

```bash
export GEMMA_MODEL_PATH=/path/to/gemma-4-E2B-it/revision-70af34e
export GEMMA_PORT=8018
.venv-gemma/bin/python pact/gemma_server.py
```

In another shell:

```bash
export GEMMA_OPENAI_BASE_URL=http://127.0.0.1:8018/v1
export GEMMA_MODEL=gemma-4-E2B-it
export GEMMA_TEMPERATURE=0

bash games/vrising/l2/run_semantic_views.sh
bash games/hk/l2/run_semantic_views.sh
bash games/isaac/l2/run_semantic_views.sh
```

Rationale correspondence and paired image-modality runs use:

```bash
bash games/vrising/l2/run_rationale_consistency.sh
bash games/hk/l2/run_rationale_consistency.sh
bash games/isaac/l2/run_rationale_consistency.sh

bash games/vrising/l2/run_image_modality_ablation.sh
bash games/hk/l2/run_image_modality_ablation.sh
bash games/isaac/l2/run_image_modality_ablation.sh
```

The Hollow Knight and Isaac fixed two-choice controls are selected through the
game wrapper environment variables documented in their game READMEs.

## 5. Verify the release

```bash
python3 registry/status.py
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v
python3 pact/validate_pactbench.py --require_v5
```

The current published snapshot passes 46 regression tests and 7,382 full-gate
checks. To rebuild the machine-readable registry after an accepted change:

```bash
python3 registry/build_registry.py
```

The builder reruns the full gate before atomically replacing
`registry/registry.json`. Do not edit the registry by hand.

## 6. External decision-quality judge

The external LLM judge is intentionally excluded from the local deterministic
queue. Use `pipelines/l2/llm_judge/README.md` with explicit rows, beliefs, and
skill-library paths. Judge outputs remain review-isolated until accepted by the
maintainer.