Spaces:
Running
Running
File size: 1,179 Bytes
f4c6961 | 1 2 3 4 5 6 7 8 9 10 11 12 | # Reliability, margin ordering & abstention notes
The four models return six-class score vectors, but the score vectors have different interpretations.
1. Naive Bayes: normalized conditional posterior from a bag-of-words likelihood model. Normalization is **not calibration**. Descriptive reliability bins and ECE are permitted but require appropriate cautions.
2. Word/character centroid: similarity scores, not class probabilities. Their values cannot be treated as P(correct), and any logistic mapping would require separately fitted and tested calibration.
3. Hybrid: a fixed mixture of cosine similarities, likewise **not a probability**.
For every model, the raw top-one minus top-two score gap can be sorted **within that model** to make a risk-versus-coverage curve on held-out evaluation examples. `AURC` is the simple average of observed cumulative risks; it is not a generalized deployment-risk integral. AUC under varying distribution or domain shift is unmeasured.
The application never treats a margin threshold as a deployment guarantee or safety certificate. The 54-example challenge has substantial sampling uncertainty and the labels are synthetic-curated.
|