# Evaluation

This document defines an **offline** evaluation protocol for the PHPWind
captcha OCR model. It measures image-recognition quality on labelled captcha
images; it does not describe or endorse automated login attempts.

## Scope

Evaluate only images from PHPWind deployments you own or are explicitly
authorized to test. Keep the evaluation dataset separate from the training
dataset, and do not publish captcha images or labels that contain private data.

## Version boundary

The published checkpoint was trained on four-digit numeric captcha images from
one target deployment whose footer displayed `v0.7β`. This footer text is an
observed deployment label, not an official PHPWind release version claim.

| Deployment or version label | Status | Evidence | Notes |
|---|---|---|---|
| Target deployment — footer label `v0.7β` | Training and reference-evaluation scope | 997 manually labelled images; 88.61% held-out validation accuracy | The only visual configuration represented in the reported data. |
| Other PHPWind releases, forks, themes, or captcha generators | Unverified | No version-specific evaluation | Evaluate with authorized representative samples and fine-tune when the image distribution differs. |

## Reference metric

The published checkpoint reports the following held-out validation result:

| Metric | Value |
|---|---:|
| Validation accuracy | 88.61% |
| Labelled image count | 997 |
| Validation split | 8% |
| Task | Four-digit numeric captcha recognition for the target `v0.7β` footer-label deployment |

This metric is a historical reference for this checkpoint. It is not a
production guarantee: other PHPWind versions, themes, fonts, and generator
settings can change the image distribution.

## Reproducible protocol

1. Collect labelled PHPWind captcha images only from an authorized deployment.
2. Keep a fixed held-out split that is not used for training or tuning.
3. Apply the model's preprocessing exactly: RGB, resize to `160 × 64` with
   bilinear interpolation, then divide pixel values by `255`.
4. Decode each of the four output positions by argmax and compare the complete
   predicted code with its label.
5. Report exact-match accuracy together with dataset size, split method, model
   revision, and PHPWind version/theme details needed to interpret the result.

## Recommended report format

| Field | What to record |
|---|---|
| Model revision | Commit hash or file checksum |
| PHPWind environment | Version and relevant theme/customization details |
| Dataset provenance | Authorized source and collection date range |
| Split | Fixed seed or explicit file list |
| Metric | Complete four-digit exact-match accuracy |
| Limitations | Distribution differences, labelling issues, and known failures |

For adapting the checkpoint to a changed PHPWind generator, see the
[training guide](TRAINING.md).