CALIBRATION.md

Reliability, margin ordering & abstention notes

The four models return six-class score vectors, but the score vectors have different interpretations.

1. Naive Bayes: normalized conditional posterior from a bag-of-words likelihood model. Normalization is not calibration. Descriptive reliability bins and ECE are permitted but require appropriate cautions.
2. Word/character centroid: similarity scores, not class probabilities. Their values cannot be treated as P(correct), and any logistic mapping would require separately fitted and tested calibration.
3. Hybrid: a fixed mixture of cosine similarities, likewise not a probability.

For every model, the raw top-one minus top-two score gap can be sorted within that model to make a risk-versus-coverage curve on held-out evaluation examples. AURC is the simple average of observed cumulative risks; it is not a generalized deployment-risk integral. AUC under varying distribution or domain shift is unmeasured.

The application never treats a margin threshold as a deployment guarantee or safety certificate. The 54-example challenge has substantial sampling uncertainty and the labels are synthetic-curated.