File size: 4,355 Bytes
f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 544e392 f1fc3a0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | # Reproducing the Published PACTBench Snapshot
Run all commands from the `pactbench` directory.
## 1. Environment
```bash
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
```
The checked-in environment was validated with Python 3.13 and CUDA 12.x. The
L1 path uses the versions pinned in `requirements.txt`.
Gemma serving uses a separate environment because the server requires a newer
Transformers release than the L1 environment:
```bash
python3 -m venv .venv-gemma
.venv-gemma/bin/python -m pip install \
"transformers==5.13.0" "torch==2.5.1" accelerate pillow
```
## 2. Data and caches
Make `data` point to the directory containing the processed layouts documented
in `docs/DATA.md`. The repository currently uses:
```text
data -> ../data
out -> ../out
```
Audit the available datasets before evaluation:
```bash
bash games/vrising/data/audit_dataset.sh
bash games/hk/data/audit_dataset.sh
bash games/isaac/data/audit_dataset.sh
```
RGB caches are rebuildable from processed videos:
```bash
python3 pact/build_rgb_frame_cache.py --workers 32
```
The canonical cache paths and their content identities are recorded in the
published registry. The full gate rejects missing, stale, or mismatched caches.
## 3. L1 evaluation
Evaluate the published checkpoints:
```bash
bash games/vrising/l1/eval_official_from_ckpt.sh
bash games/hk/l1/eval_official_from_ckpt.sh
bash games/isaac/l1/eval_official_from_ckpt.sh
```
Run the matched frozen-foundation comparisons:
```bash
bash games/vrising/l1/run_foundation_baselines.sh
bash games/hk/l1/run_foundation_baselines.sh
bash games/hk/l1/run_matched_foundation10k.sh
bash games/isaac/l1/run_foundation_baselines.sh
```
Run the accepted V Rising end-to-end DINOv2-S L1 comparison experiment:
```bash
bash games/vrising/l1/run_dinov2_end_to_end.sh
```
The wrapper preserves accepted artifacts unless `FORCE_RERUN=1` is set and
writes the canonical metrics, batch-1 latency, and comparison summary under
`results/vrising/eval_runs/L1-DINOV2-END-TO-END-MATCHED20/`.
Isaac's matched PACT checkpoint is produced by
`pipelines/l1/train/run_isaac_matched_after_weight_selection.sh` after the
validation-only distance-weight selection. Published checkpoints and metrics
can be validated without retraining them.
The image-input controls are:
```bash
bash games/vrising/l1/run_img_ablation.sh
bash games/hk/l1/run_img_ablation.sh
bash games/isaac/l1/run_img_ablation.sh
```
## 4. L2 service and evaluation
Start the pinned Gemma service with a local path or compatible model ID:
```bash
export GEMMA_MODEL_PATH=/path/to/gemma-4-E2B-it/revision-70af34e
export GEMMA_PORT=8018
.venv-gemma/bin/python pact/gemma_server.py
```
In another shell:
```bash
export GEMMA_OPENAI_BASE_URL=http://127.0.0.1:8018/v1
export GEMMA_MODEL=gemma-4-E2B-it
export GEMMA_TEMPERATURE=0
bash games/vrising/l2/run_semantic_views.sh
bash games/hk/l2/run_semantic_views.sh
bash games/isaac/l2/run_semantic_views.sh
```
Rationale correspondence and paired image-modality runs use:
```bash
bash games/vrising/l2/run_rationale_consistency.sh
bash games/hk/l2/run_rationale_consistency.sh
bash games/isaac/l2/run_rationale_consistency.sh
bash games/vrising/l2/run_image_modality_ablation.sh
bash games/hk/l2/run_image_modality_ablation.sh
bash games/isaac/l2/run_image_modality_ablation.sh
```
The Hollow Knight and Isaac fixed two-choice controls are selected through the
game wrapper environment variables documented in their game READMEs.
## 5. Verify the release
```bash
python3 registry/status.py
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v
python3 pact/validate_pactbench.py --require_v5
```
The current published snapshot passes 46 regression tests and 7,382 full-gate
checks. To rebuild the machine-readable registry after an accepted change:
```bash
python3 registry/build_registry.py
```
The builder reruns the full gate before atomically replacing
`registry/registry.json`. Do not edit the registry by hand.
## 6. External decision-quality judge
The external LLM judge is intentionally excluded from the local deterministic
queue. Use `pipelines/l2/llm_judge/README.md` with explicit rows, beliefs, and
skill-library paths. Judge outputs remain review-isolated until accepted by the
maintainer.
|