docs/evaluation-protocol-v10.md

# v0.10 evaluation and device protocol

## Purpose and evidence states

This protocol compares candidate generators and finite-candidate rerankers without
turning a local smoke test into a mobile, provider, or public quality claim.

- `LOCAL_EVALUATION`: a pinned item artifact and prediction artifact were scored by the
  local CLI.
- `DEVICE_RECEIPT`: the public page downloaded a redacted timing/capability JSON on one
  named device. This is not a pass by itself.
- `DEVICE_PASS`: cold and warm procedures, device/OS/browser identity, memory pressure,
  tab survival, and thresholds were reviewed for that physical device.
- `PUBLIC_PASS`: the deployed revision was fetched anonymously and the public flow was
  exercised. It does not imply `DEVICE_PASS` for Safari.
- `HUMAN_GO`: remains a separate user decision.

## Reproducible finite-candidate scoring

Install the project, then run:

```powershell
uv run deberta-ime-evaluate `
  --items .\work\evaluation\items.json `
  --predictions .\work\evaluation\predictions.json `
  --format generic `
  --dataset-name private-real-typo-heldout `
  --dataset-revision 2026-08-11-v1 `
  --dataset-license not-redistributed `
  --candidate-limit 8 `
  --output-dir .\outputs `
  --stem typo_evaluation_v10
```

Generic item schema:

```json
[
  {
    "id": "stable-row-id",
    "input": "変換前または誤入力",
    "references": ["許容される正解"],
    "label": "clean"
  }
]
```

`label` is one of `clean`, `typo`, or `unspecified`. A `clean` row is valid only when
its input is one of its references. Prediction schema:

```json
[
  {
    "id": "stable-row-id",
    "candidates": ["有限候補1", "有限候補2"],
    "provenance": "deberta",
    "reason": "accepted",
    "margin": 1.25
  }
]
```

`provenance` (`provider` / `deberta` / `lfm` / `rule` / `unspecified`)、machine-readable
`reason`、有限数の `margin` は任意です。これらは prediction artifact の宣言値であり、評価 CLI が
モデル実行を再現して検証した事実ではありません。report schema v2 は宣言値の件数・平均 margin と、
正解が候補集合にない accepted candidate miss、正解は集合にあるが top-1 を誤った selection error を
別々に記録します。

An empty or missing candidate list is an abstention. Its effective output is the input
text, not an empty string. Prediction IDs must be unique and may not name rows absent
from the item artifact. Generic runs should name the dataset, revision, and license (or
non-redistribution boundary). Both input files are SHA-256 receipted in the report.

For pinned AJIMEE rows, use `--format ajimee`; the adapter reads `index`, `input`, and
`expected_output`. It deliberately assigns `unspecified`, so AJIMEE cannot silently
become the clean-control denominator for overcorrection. Official AJIMEE name, revision,
and license are attached only when the item bytes match the pinned SHA-256; merely using
the same JSON field names is reported as compatible but unverified input.

## Metric definitions

- Effective Acc@1: fraction whose top candidate, or preserved input on abstention, is an
  acceptable reference.
- Candidate Recall@k: fraction where any supplied candidate up to `k` is an acceptable
  reference. This is the hard ceiling a reranker can exploit at that boundary.
- MinCER: minimum character Levenshtein distance divided by reference length. Reports
  include baseline input MinCER and effective-output MinCER.
- Abstention rate: empty or missing prediction rows divided by all rows.
- Accepted accuracy: correct top candidates divided only by non-abstained rows; never
  substitute this conditional value for the all-row metric.
- Overcorrection rate: explicitly `clean` rows whose effective output differs from the
  input, divided by explicitly `clean` rows. It is `null` when the denominator is zero.
- Improvement/worsening: effective MinCER lower/higher than input MinCER on the same row.
- Accepted candidate miss: 非棄却行で、正解が top-k 候補集合に存在しない件数。
- Selection error: 正解が top-k 候補集合に存在するが、top-1 が正解でない件数。

AJIMEE is a 200-row kana-kanji benchmark reconstructed from erroneous-conversion cases
in JWTD v2. It is useful for Acc@1/CER, but it is not a physical keyboard log or a full
clean/typo paired corpus. The latter is still required before reporting the user's
requested real-typo overcorrection rate.

## Local candidate-generator prototype

The throwaway branch `codex/provider-prototype-v10`, commit
`c781530`, tests the pinned 48.1 MB Mozc SQLite dictionary as a sequence lattice. Rows
1-20 selected a declared segment-penalty grid; rows 21-200 were held out.

| Boundary | Acc@1 | Recall@8 | Mean MinCER | p95 local CPU |
| --- | ---: | ---: | ---: | ---: |
| 4 entries/reading, beam 64 | 18.89% | 27.78% | 22.88% | 47.2 ms |
| 8 entries/reading, beam 256 | 18.89% | 27.22% | 22.88% | 288.2 ms |

The dictionary lacks Mozc connection IDs and transition costs. Wider search cost about
6.1x at p95 without improving Recall@8. It is rejected as a Google replacement and is
not merged into the application. A licensed FST or audited decoder with real transition
scoring is the next candidate-generator comparison.

## Physical Safari cold/warm procedure

The public page now records model status transitions, conversion durations, navigation
timing, redacted resource timing, WebGPU/SharedArrayBuffer/cross-origin-isolation
capabilities, and any browser memory API that actually exists. It never records input
or converted text, strips query strings and fragments from resource URLs, and downloads
the JSON locally without telemetry. Cross-origin transfer/body sizes can be zero when
the remote server does not expose Resource Timing headers; zero is not interpreted as a
zero-byte model.

For each physical iPhone/iPad/macOS Safari target:

1. Record hardware model, RAM, OS build, Safari version, network, battery/thermal state,
   and the exact public Space revision outside the receipt.
2. Cold: close all tabs for the site, clear site data/cache, start a new browser process,
   open the public URL, run the fixed input once, and download the receipt.
3. Warm: on the same page run the same input again and download a second receipt.
4. Record wall-clock launch-to-result, tab reload/eviction, OS memory-pressure behavior,
   and whether the exact output remained correct. `performance.memory` absence on Safari
   is expected; do not estimate it from another browser.
5. Repeat at least five cold and twenty warm runs per target; report median and p95, not
   only the fastest run.

The current ONNX Runtime Web support table lists WASM CPU for Safari but not its WebGPU
execution provider. Browser exposure of `navigator.gpu` is therefore diagnostic only;
it does not promote the LFM WebGPU route to Safari support.

## LFM2.5 escalation gate

Keep candidate artifacts, tokenizer, prompt, candidate count/order, decoding policy,
context, and hardware route fixed. The model may return scores/ranks only; generated
free text may not enter the candidate list.

1. First make candidate Recall@k adequate. A larger LFM cannot repair missing candidates.
2. Evaluate 230M on the fixed artifact. Escalation is eligible only when errors are
   predominantly correct-candidate-present/reranker-choice errors, not candidate misses.
3. Compare 350M only on that same artifact and only if the predeclared 230M quality target
   is missed. Compare 1.2B-JP only if 350M demonstrates a paired improvement but still
   misses the target.
4. Promotion additionally requires physical-device cold/warm latency, memory/tab
   survival, license/NOTICE review, and no overcorrection regression. A browser or local
   PASS cannot stand in for the missing Safari device result.

The rejected local generator has only 27.78% Recall@8, so this iteration does not trigger
350M or 1.2B-JP testing.