METHODS.md

Methods and evaluation protocol

Lexical models

All models are deterministic and train in JavaScript on the 842 train.json rows.

1. Multinomial Naive Bayes: token count frequencies for Unicode-normalized word features (accent stripping for compatibility), per-class Laplace smoothing alpha=1, additive log-likelihoods and empirical smoothed class priors.
2. Word TF-IDF centroid: per-document TF=(1+log(count)) and IDF=(log((N+1)/(df+1))+1). L2-normalize each training document vector; sum by class; normalize the final class centroid; classify by cosine similarity.
3. Character trigram TF-IDF centroid: same TF-IDF scheme on normalized, space-padded character 3-grams. Language-neutral, lexical (not semantic).
4. Weighted hybrid: 0.55 × word-centroid cosine + 0.45 × character-centroid cosine. Fixed mixture weights; not learned from the held-out challenge.

Evaluation metrics

Accuracy = correct / total. Class F1=2PR/(P+R), P=TP/(TP+FP) and R=TP/(TP+FN), with division-by-zero returning 0. Macro F1 averages classes with positive true-label support. The overall test and challenge sets contain all six labels. Subgroup results depend on subgroup supports and can be unstable for small n.

engine.js computes all predictions at runtime, then aggregates confusion matrices, language and domain slices, and example-level error rows. Historical Python-model results appear only as clearly separate context.

Uncertainty calculation

The Experiments tab uses a seeded Mulberry32 pseudo-random generator to draw 200 or 500 bootstrap resamples of held-out prediction rows, with replacement. The 2.5 and 97.5 percentiles form an approximate sample-resampling interval for Accuracy. This interval does not include uncertainty from training procedures, domain sampling, annotators, temporal changes or feature choices. Inference confidence scores in Error lab are uncalibrated similarity or posterior-like outputs, and must not be interpreted as reliable correctness probabilities.

Release gates

• Recompute both fixed splits; never train on evaluation rows.
• Audit IDs and exact text overlaps; inspect sample diversity in future work.
• Disclose large IID-to-novel-pattern generalization gaps.
• Require a separate external human-labeled research benchmark before making claims of real-world utility.