METHODOLOGY.md

# Evaluation methodology

## Paired experiment

For every model/question pair:

1. Generate the raw model response once.
2. Trial the verified expert fleet without exposing the target.
3. If an expert is verified, use its canonical answer and perform no additional
   model call.
4. If no expert is verified, reuse the current pair's raw response as the
   byte-identical bypass result.
5. Score raw and safe outputs with the same task scorer.

This produces four possible outcomes:

- **win**: raw wrong, safe correct;
- **loss**: raw correct, safe wrong;
- **tie-correct**;
- **tie-wrong**.

`GAIN` and `LOSS` require a two-sided exact paired test at alpha 0.05. Otherwise
the result is `INCONCLUSIVE`, regardless of the visible percentage difference.

## Freshness and leakage controls

- Models and questions are declared before the first generation.
- Question fingerprints are registered as consumed before inference.
- A consumed question cannot become a fresh benchmark again.
- Sealed targets are never present in model or expert prompts.
- Sealed tests cannot train, calibrate, or automatically promote a component.
- Checkpoints store hashes and scores, not raw questions, answers, or responses.

## Separate metrics

Correctness is reported separately from:

- strict output-envelope compliance;
- expert coverage and refusal rate;
- expert disagreement;
- generation-cap hits;
- latency and token use.

A formatting failure cannot turn a wrong answer into a correct one, and relaxed
answer extraction cannot erase the strict-format metric.