Spaces:
Running
Running
| # Methods and evaluation protocol | |
| ## Lexical models | |
| All models are deterministic and train in JavaScript on the 842 `train.json` rows. | |
| 1. **Multinomial Naive Bayes:** token count frequencies for Unicode-normalized word features (accent stripping for compatibility), per-class Laplace smoothing alpha=1, additive log-likelihoods and empirical smoothed class priors. | |
| 2. **Word TF-IDF centroid:** per-document TF=(1+log(count)) and IDF=(log((N+1)/(df+1))+1). L2-normalize each training document vector; sum by class; normalize the final class centroid; classify by cosine similarity. | |
| 3. **Character trigram TF-IDF centroid:** same TF-IDF scheme on normalized, space-padded character 3-grams. Language-neutral, lexical (not semantic). | |
| 4. **Weighted hybrid:** 0.55 × word-centroid cosine + 0.45 × character-centroid cosine. Fixed mixture weights; not learned from the held-out challenge. | |
| ## Evaluation metrics | |
| Accuracy = correct / total. Class F1=2PR/(P+R), P=TP/(TP+FP) and R=TP/(TP+FN), with division-by-zero returning 0. Macro F1 averages classes with positive true-label support. The overall test and challenge sets contain all six labels. Subgroup results depend on subgroup supports and can be unstable for small n. | |
| `engine.js` computes all predictions at runtime, then aggregates confusion matrices, language and domain slices, and example-level error rows. Historical Python-model results appear only as clearly separate context. | |
| ## Uncertainty calculation | |
| The `Experiments` tab uses a seeded Mulberry32 pseudo-random generator to draw 200 or 500 bootstrap resamples of held-out prediction rows, with replacement. The 2.5 and 97.5 percentiles form an *approximate sample-resampling interval* for Accuracy. This interval does not include uncertainty from training procedures, domain sampling, annotators, temporal changes or feature choices. Inference confidence scores in Error lab are uncalibrated similarity or posterior-like outputs, and must not be interpreted as reliable correctness probabilities. | |
| ## Release gates | |
| - Recompute both fixed splits; never train on evaluation rows. | |
| - Audit IDs and exact text overlaps; inspect sample diversity in future work. | |
| - Disclose large IID-to-novel-pattern generalization gaps. | |
| - Require a separate external human-labeled research benchmark before making claims of real-world utility. | |