docs/evaluation-protocol-v10.md

v0.10 evaluation and device protocol

Purpose and evidence states

This protocol compares candidate generators and finite-candidate rerankers without turning a local smoke test into a mobile, provider, or public quality claim.

• LOCAL_EVALUATION: a pinned item artifact and prediction artifact were scored by the local CLI.
• DEVICE_RECEIPT: the public page downloaded a redacted timing/capability JSON on one named device. This is not a pass by itself.
• DEVICE_PASS: cold and warm procedures, device/OS/browser identity, memory pressure, tab survival, and thresholds were reviewed for that physical device.
• PUBLIC_PASS: the deployed revision was fetched anonymously and the public flow was exercised. It does not imply DEVICE_PASS for Safari.
• HUMAN_GO: remains a separate user decision.

Reproducible finite-candidate scoring

Install the project, then run:

uv run deberta-ime-evaluate `
  --items .\work\evaluation\items.json `
  --predictions .\work\evaluation\predictions.json `
  --format generic `
  --dataset-name private-real-typo-heldout `
  --dataset-revision 2026-08-11-v1 `
  --dataset-license not-redistributed `
  --candidate-limit 8 `
  --output-dir .\outputs `
  --stem typo_evaluation_v10

Generic item schema:

[
  {
    "id": "stable-row-id",
    "input": "変換前または誤入力",
    "references": ["許容される正解"],
    "label": "clean"
  }
]

label is one of clean, typo, or unspecified. A clean row is valid only when its input is one of its references. Prediction schema:

[
  {
    "id": "stable-row-id",
    "candidates": ["有限候補1", "有限候補2"],
    "provenance": "deberta",
    "reason": "accepted",
    "margin": 1.25
  }
]

provenance (provider / deberta / lfm / rule / unspecified)、machine-readable reason、有限数の margin は任意です。これらは prediction artifact の宣言値であり、評価 CLI が モデル実行を再現して検証した事実ではありません。report schema v2 は宣言値の件数・平均 margin と、 正解が候補集合にない accepted candidate miss、正解は集合にあるが top-1 を誤った selection error を 別々に記録します。

An empty or missing candidate list is an abstention. Its effective output is the input text, not an empty string. Prediction IDs must be unique and may not name rows absent from the item artifact. Generic runs should name the dataset, revision, and license (or non-redistribution boundary). Both input files are SHA-256 receipted in the report.

For pinned AJIMEE rows, use --format ajimee; the adapter reads index, input, and expected_output. It deliberately assigns unspecified, so AJIMEE cannot silently become the clean-control denominator for overcorrection. Official AJIMEE name, revision, and license are attached only when the item bytes match the pinned SHA-256; merely using the same JSON field names is reported as compatible but unverified input.

Metric definitions

• Effective Acc@1: fraction whose top candidate, or preserved input on abstention, is an acceptable reference.
• Candidate Recall@k: fraction where any supplied candidate up to k is an acceptable reference. This is the hard ceiling a reranker can exploit at that boundary.
• MinCER: minimum character Levenshtein distance divided by reference length. Reports include baseline input MinCER and effective-output MinCER.
• Abstention rate: empty or missing prediction rows divided by all rows.
• Accepted accuracy: correct top candidates divided only by non-abstained rows; never substitute this conditional value for the all-row metric.
• Overcorrection rate: explicitly clean rows whose effective output differs from the input, divided by explicitly clean rows. It is null when the denominator is zero.
• Improvement/worsening: effective MinCER lower/higher than input MinCER on the same row.
• Accepted candidate miss: 非棄却行で、正解が top-k 候補集合に存在しない件数。
• Selection error: 正解が top-k 候補集合に存在するが、top-1 が正解でない件数。

AJIMEE is a 200-row kana-kanji benchmark reconstructed from erroneous-conversion cases in JWTD v2. It is useful for Acc@1/CER, but it is not a physical keyboard log or a full clean/typo paired corpus. The latter is still required before reporting the user's requested real-typo overcorrection rate.

Local candidate-generator prototype

The throwaway branch codex/provider-prototype-v10, commit c781530, tests the pinned 48.1 MB Mozc SQLite dictionary as a sequence lattice. Rows 1-20 selected a declared segment-penalty grid; rows 21-200 were held out.

Boundary	Acc@1	Recall@8	Mean MinCER	p95 local CPU
4 entries/reading, beam 64	18.89%	27.78%	22.88%	47.2 ms
8 entries/reading, beam 256	18.89%	27.22%	22.88%	288.2 ms

The dictionary lacks Mozc connection IDs and transition costs. Wider search cost about 6.1x at p95 without improving Recall@8. It is rejected as a Google replacement and is not merged into the application. A licensed FST or audited decoder with real transition scoring is the next candidate-generator comparison.

Physical Safari cold/warm procedure

The public page now records model status transitions, conversion durations, navigation timing, redacted resource timing, WebGPU/SharedArrayBuffer/cross-origin-isolation capabilities, and any browser memory API that actually exists. It never records input or converted text, strips query strings and fragments from resource URLs, and downloads the JSON locally without telemetry. Cross-origin transfer/body sizes can be zero when the remote server does not expose Resource Timing headers; zero is not interpreted as a zero-byte model.

For each physical iPhone/iPad/macOS Safari target:

1. Record hardware model, RAM, OS build, Safari version, network, battery/thermal state, and the exact public Space revision outside the receipt.
2. Cold: close all tabs for the site, clear site data/cache, start a new browser process, open the public URL, run the fixed input once, and download the receipt.
3. Warm: on the same page run the same input again and download a second receipt.
4. Record wall-clock launch-to-result, tab reload/eviction, OS memory-pressure behavior, and whether the exact output remained correct. performance.memory absence on Safari is expected; do not estimate it from another browser.
5. Repeat at least five cold and twenty warm runs per target; report median and p95, not only the fastest run.

The current ONNX Runtime Web support table lists WASM CPU for Safari but not its WebGPU execution provider. Browser exposure of navigator.gpu is therefore diagnostic only; it does not promote the LFM WebGPU route to Safari support.

LFM2.5 escalation gate

Keep candidate artifacts, tokenizer, prompt, candidate count/order, decoding policy, context, and hardware route fixed. The model may return scores/ranks only; generated free text may not enter the candidate list.

1. First make candidate Recall@k adequate. A larger LFM cannot repair missing candidates.
2. Evaluate 230M on the fixed artifact. Escalation is eligible only when errors are predominantly correct-candidate-present/reranker-choice errors, not candidate misses.
3. Compare 350M only on that same artifact and only if the predeclared 230M quality target is missed. Compare 1.2B-JP only if 350M demonstrates a paired improvement but still misses the target.
4. Promotion additionally requires physical-device cold/warm latency, memory/tab survival, license/NOTICE review, and no overcorrection regression. A browser or local PASS cannot stand in for the missing Safari device result.

The rejected local generator has only 27.78% Recall@8, so this iteration does not trigger 350M or 1.2B-JP testing.