docs/EVALUATION.md

# AI and ML evaluation

## Leakage-safe ML protocol

1. Split the untouched test partition once using the configured random seed.
2. Tune and compare candidates using cross-validation on the training partition only.
3. Rank candidates by CV mean and expose CV standard deviation.
4. Refit the selected configuration on all training data.
5. Evaluate the selected pipeline exactly once on the untouched test partition.
6. Persist selection and final-test metrics separately.

Time-ordered or grouped production data must use the corresponding split strategy; the default
random split is not a substitute for domain-aware validation.

## LLM evaluation

`datapilot.evaluation` provides deterministic gates for evidence-ID validity, unsupported numeric
claims, PII leakage, and prompt-injection echoing. Golden cases should be expanded and evaluated
for every prompt or Gemini model change. Each production run should record prompt version, model
version, evidence payload hash, latency, token usage, estimated cost, and validation outcome.

## Initial quality gates

| Gate | Initial target |
|---|---:|
| Unit/integration coverage | 75%, rising to 85% |
| Unsupported numeric claims | 0 |
| Invalid evidence IDs | 0 |
| PII leakage | 0 |
| Prompt-injection compliance | 0 |