Hugging Face's logo

BBoran
/
pactbench

Model card Files
xet
 Community
pactbench / docs /REPRODUCE.md
BBoran's picture
BBoran
Publish current portable PACTBench release
f1fc3a0 verified
preview code
|
Raw
History Blame Contribute Delete
4.36 kB

Reproducing the Published PACTBench Snapshot

Run all commands from the pactbench directory.

1. Environment 

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt

The checked-in environment was validated with Python 3.13 and CUDA 12.x. The L1 path uses the versions pinned in requirements.txt.

Gemma serving uses a separate environment because the server requires a newer Transformers release than the L1 environment:

python3 -m venv .venv-gemma
.venv-gemma/bin/python -m pip install \
  "transformers==5.13.0" "torch==2.5.1" accelerate pillow

2. Data and caches

Make data point to the directory containing the processed layouts documented in docs/DATA.md. The repository currently uses:

data -> ../data
out  -> ../out

Audit the available datasets before evaluation:

bash games/vrising/data/audit_dataset.sh
bash games/hk/data/audit_dataset.sh
bash games/isaac/data/audit_dataset.sh

RGB caches are rebuildable from processed videos:

python3 pact/build_rgb_frame_cache.py --workers 32

The canonical cache paths and their content identities are recorded in the published registry. The full gate rejects missing, stale, or mismatched caches.

3. L1 evaluation

Evaluate the published checkpoints:

bash games/vrising/l1/eval_official_from_ckpt.sh
bash games/hk/l1/eval_official_from_ckpt.sh
bash games/isaac/l1/eval_official_from_ckpt.sh

Run the matched frozen-foundation comparisons:

bash games/vrising/l1/run_foundation_baselines.sh
bash games/hk/l1/run_foundation_baselines.sh
bash games/hk/l1/run_matched_foundation10k.sh
bash games/isaac/l1/run_foundation_baselines.sh

Run the accepted V Rising end-to-end DINOv2-S L1 comparison experiment:

bash games/vrising/l1/run_dinov2_end_to_end.sh

The wrapper preserves accepted artifacts unless FORCE_RERUN=1 is set and writes the canonical metrics, batch-1 latency, and comparison summary under results/vrising/eval_runs/L1-DINOV2-END-TO-END-MATCHED20/.

Isaac's matched PACT checkpoint is produced by pipelines/l1/train/run_isaac_matched_after_weight_selection.sh after the validation-only distance-weight selection. Published checkpoints and metrics can be validated without retraining them.

The image-input controls are:

bash games/vrising/l1/run_img_ablation.sh
bash games/hk/l1/run_img_ablation.sh
bash games/isaac/l1/run_img_ablation.sh

4. L2 service and evaluation

Start the pinned Gemma service with a local path or compatible model ID:

export GEMMA_MODEL_PATH=/path/to/gemma-4-E2B-it/revision-70af34e
export GEMMA_PORT=8018
.venv-gemma/bin/python pact/gemma_server.py

In another shell:

export GEMMA_OPENAI_BASE_URL=http://127.0.0.1:8018/v1
export GEMMA_MODEL=gemma-4-E2B-it
export GEMMA_TEMPERATURE=0

bash games/vrising/l2/run_semantic_views.sh
bash games/hk/l2/run_semantic_views.sh
bash games/isaac/l2/run_semantic_views.sh

Rationale correspondence and paired image-modality runs use:

bash games/vrising/l2/run_rationale_consistency.sh
bash games/hk/l2/run_rationale_consistency.sh
bash games/isaac/l2/run_rationale_consistency.sh

bash games/vrising/l2/run_image_modality_ablation.sh
bash games/hk/l2/run_image_modality_ablation.sh
bash games/isaac/l2/run_image_modality_ablation.sh

The Hollow Knight and Isaac fixed two-choice controls are selected through the game wrapper environment variables documented in their game READMEs.

5. Verify the release 

python3 registry/status.py
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v
python3 pact/validate_pactbench.py --require_v5

The current published snapshot passes 46 regression tests and 7,382 full-gate checks. To rebuild the machine-readable registry after an accepted change:

python3 registry/build_registry.py

The builder reruns the full gate before atomically replacing registry/registry.json. Do not edit the registry by hand.

6. External decision-quality judge

The external LLM judge is intentionally excluded from the local deterministic queue. Use pipelines/l2/llm_judge/README.md with explicit rows, beliefs, and skill-library paths. Judge outputs remain review-isolated until accepted by the maintainer.