docs/EXPERIMENTS.md

# Experiments and Published Results

This document describes the results published by
`registry/registry.json`. Aggregate numbers are recomputed from the referenced
artifacts by the full release gate.

## Common protocol

- Splits are fight-disjoint.
- L1 consumes five causal RGB frames at offsets `[-8,-4,-2,-1,0]`.
- L2 uses the compact belief interface and anonymous option IDs.
- The L2 model is Gemma-4-E2B-it revision `70af34e...`, temperature 0.
- balanced-v3 selects source states across boss, fight, and distance strata
  before any L2 response is read.
- The prompt conditions are full mechanics, no mechanics, and swapped
  mechanics. The word `full` in a run name refers to the full-description
  prompt, not the deprecated full belief view.

## L1 official evaluation

| Game | Test rows | Distance MAE | dp@2 | Angle MAE | angle@20° | Distance-bin acc. | Angle-bin acc. |
|---|---:|---:|---:|---:|---:|---:|---:|
| V Rising | 8,075 | 0.2620 | 0.9960 | 11.7599° | 0.8293 | 0.8320 | 0.7201 |
| Hollow Knight | 8,360 | 0.4680 | 0.9728 | 3.6527° | 0.9788 | 0.7261 | 0.9646 |
| Isaac | 1,784 | 9.0263 px | 0.1519 | 29.6388° | 0.6211 | 0.9036 | 0.8352 |

`dp@2` uses native distance units and is therefore not comparable between
Isaac and the other games.

## L1 matched-data visual comparison

| Game | Method | Train rows | Distance MAE | Angle MAE | Distance-bin acc. | Angle-bin acc. |
|---|---|---:|---:|---:|---:|---:|
| V Rising | PACT-L1 | 18,068 | **0.2620** | **11.7599°** | **0.8320** | **0.7201** |
|  | frozen DINOv2-S + probe | 18,068 | 0.6900 | 20.1532° | 0.6405 | 0.6041 |
|  | frozen VideoMAE-B + probe | 18,068 | 0.6672 | 17.6564° | 0.6821 | 0.5948 |
| Hollow Knight | PACT-L1 | 10,000 | **0.5671** | **4.0899°** | **0.6940** | **0.9560** |
|  | frozen DINOv2-S + probe | 10,000 | 0.8641 | 20.0433° | 0.5542 | 0.8276 |
|  | frozen VideoMAE-B + probe | 10,000 | 1.0017 | 27.5987° | 0.4544 | 0.7100 |
| Isaac | PACT-L1 | 9,484 | **15.0472** | **39.6855°** | **0.8358** | **0.7848** |
|  | frozen DINOv2-S + probe | 9,484 | 35.2852 | 77.7867° | 0.6015 | 0.6048 |
|  | frozen VideoMAE-B + probe | 9,484 | 47.5451 | 84.9615° | 0.4608 | 0.5067 |

Every three-way comparison uses the same ordered training rows, official test
rows, resolution, and five-frame history within a game. PACT-L1 is task-trained
end to end; the two foundation baselines freeze their pretrained backbone and
train task probes. These results compare complete training schemes rather than
isolating backbone architecture.

### V Rising end-to-end DINOv2-S L1 comparison experiment

This separate comparison makes the DINOv2-S backbone task-trainable and applies
the same published V Rising rows, five-frame history, task heads/losses, seed,
and two-stage 10+10 epoch schedule used by PACT-L1. Both methods are evaluated
on the same ordered 8,075 official test rows.

| Method | Parameters | Distance MAE | Angle MAE | Distance-bin acc. | Angle-bin acc. | A800 batch-1 latency |
|---|---:|---:|---:|---:|---:|---:|
| PACT-L1 | 13,040,742 | **0.2620** | **11.7599°** | **0.8320** | **0.7201** | **5.8497 ms** |
| DINOv2-S end to end | 23,443,622 | 0.8566 | 24.8192° | 0.5771 | 0.5845 | 8.7039 ms |

Under this matched recipe, DINOv2-S has 3.2695 times the distance error, 2.1105
times the angle error, and 48.79% higher model-forward latency. The latency
protocol uses AMP, 256 pre-decoded histories, 10 warm-up passes, and 20 measured
repeats; it includes CPU-to-GPU transfer and model-specific preprocessing but
excludes mp4 decoding. The measured 8.7039 ms is slower than PACT-L1 but does
not establish failure of a 60 Hz real-time budget on this A800.

The result does not mean that a larger pretrained model must be more accurate
after fine-tuning. In this implementation, the most likely explanation is a
training-recipe and spatial-interface mismatch:

- The PACT learning-rate schedule uses one shared learning rate for the
  pretrained DINO backbone and the newly initialized task heads. The frozen
  DINO probe obtains 0.6900 distance MAE and 20.1532° angle MAE, while the first
  end-to-end stage at learning rate 3e-4 degrades to 0.9128 and 24.7542°; the
  second stage recovers distance only to 0.8566. This is evidence that the
  copied schedule over-updates the current DINO adaptation, although it is not
  a one-factor causal ablation.
- The adapter retains one global CLS vector per frame and discards DINO patch
  tokens. That removes localized spatial evidence that is useful for distance
  and angle estimation.
- The native 192x336 frame is stretched to DINO's 224x224 square input, changing
  the geometry seen by the model.

Accordingly, this experiment supports the conclusion that the evaluated
end-to-end DINOv2-S configuration is worse than PACT-L1 under the matched PACT
recipe. It is not a general upper bound on a DINOv2 system with separately
validated backbone/head learning rates, patch-token readout, and
aspect-ratio-preserving preprocessing. The machine-readable record is
`results/vrising/eval_runs/L1-DINOV2-END-TO-END-MATCHED20/comparison.json`.

## L2 semantic intervention

Description-follow records whether the swapped-prompt decision follows the
mechanics moved to another anonymous option rather than retaining the original
option identity.

### Natural-ready menus

| Game | Valid pairs | Mean K | Chance | No-description change | Swap change | Description-follow |
|---|---:|---:|---:|---:|---:|---:|
| V Rising | 200 | 4.875 | 21.93% | 46.0% | 79.5% | **67.0%** |
| Hollow Knight | 200 | 5.370 | 18.78% | 90.0% | 95.0% | **79.5%** |
| Isaac | 122 | 2.238 | 46.04% | 56.6% | 85.2% | **83.6%** |

The natural-ready menu is a deterministic benchmark tracker rule based on
logged re-fire lower bounds and previous-action constraints. It is not an
engine legality oracle. Isaac has fewer skills and smaller natural menus than
the other games. Its 122 valid pairs are the predefined `menu>=2` subset of the
200 source states, not a selection based on model success.

### Fixed two-choice control

| Game | Pairs | Chance | No-description change | Swap change | Description-follow |
|---|---:|---:|---:|---:|---:|
| Hollow Knight | 200 | 50.0% | 31.5% | 60.5% | 60.5% |
| Isaac | 200 | 50.0% | 39.0% | 54.0% | 54.0% |

The two-choice experiment is a fixed high-chance stress control. Hollow Knight
is above its 50% reference; Isaac is close to it. The cross-game result is the
eligible natural-ready intervention, while the two-choice table shows that the
effect depends on candidate-space construction.

## Image-modality diagnostic

Text and text-plus-current-image conditions use the same states, prompt text,
and inference parameters.

| Game | Effective paired states | Text follow | Text + image follow | Original decision changed |
|---|---:|---:|---:|---:|
| V Rising | 60 | 60.0% | 65.0% | 21.7% |
| Hollow Knight | 60 | 78.3% | 80.0% | 3.3% |
| Isaac | 40 | 90.0% | 70.0% | 5.0% |

The observed direction is not uniform across games. This diagnostic measures
modality sensitivity and does not establish that adding an image always
improves semantic control.

## Prompt-visible rationale correspondence

The deterministic audit checks whether the visible reason and mechanics cards
support reconstruction of the model's selected option. It does not inspect a
hidden chain of thought.

| Game | Valid pairs | Normal support | No-description support | Shuffled-reason support |
|---|---:|---:|---:|---:|
| V Rising | 200 | 93.5% | 3.5% | 3.7% |
| Hollow Knight | 200 | 94.5% | 0.0% | 2.6% |
| Isaac | 122 | 74.6% | 31.1% | 48.2% |

## Game-specific completed analyses

- V Rising model ablation: description-follow is 67.0% for Gemma-4 E2B,
  64.5% for Gemma-3n E2B, and 56.0% for Gemma-4 E4B. Model size is not
  monotonic with this metric.
- V Rising batch-8 latency on pre-decoded histories: PACT-L1 1.1440 ms,
  frozen DINOv2-S + probe 4.3040 ms, and frozen VideoMAE-B + probe
  20.4469 ms per decision. This benchmark excludes video decoding.
- Hollow Knight current-frame ablation: on the fixed paired subset, distance
  MAE changes from 0.5160 with five frames to 0.5478 with the current frame
  only.

## Decision-quality judge

The semantic intervention measures whether decisions respond to visible skill
mechanics. Tactical decision quality is a different question and is evaluated
by the external menu-level judge described in
`pipelines/l2/llm_judge/README.md`. External judge artifacts are not part of
the local deterministic registry until explicitly reviewed and synchronized.