METHODS.md

ModelGuard Observatory · Methods

Purpose

This tool supports structured auditing of evaluation snapshots for a small multilingual academic-intent classification benchmark. It is NOT a streaming production observability service, model safety certificate, or scientific external-validity study.

Metrics

• Accuracy = correct predictions / labeled samples.
• Macro-F1 = average of per-class F1 for labels with support > 0, calculated using precision = TP / (TP + FP) and recall = TP / (TP + FN). Zero denominators contribute 0.
• Generalization gap = controlled IID accuracy minus novel-pattern challenge accuracy (reported in percentage points). The splits differ in design: it is not a time-series degradation rate.
• Error confusion: row = true label, column = predicted label; raw counts.

Dataset distribution shift

• Jensen–Shannon divergence uses base-two logarithms and has range [0, 1] bits. It compares language distributions and true-label distributions across the two evaluation splits.
• Population Stability Index uses fixed bins [0,5,10,15,20,30,50,100001] on the number of unique lowercase token types per sentence (not the full word count). For baseline proportions p_i and challenge proportions q_i, PSI = Σ_i (q_i - p_i) * ln(q_i/p_i) with both proportions floored at 1e-4 before each term to avoid numerical singularities. This is a heuristic sensitive to binning and sample size.
• A difference in distributions indicates a diagnostic shift; it does not establish its cause or statistical significance.

Paired comparison and uncertainty

Paired differences are computed from identically identified samples across two methods within the same challenge split. A seeded nonparametric bootstrap resamples paired examples 500 times and reports descriptive percentile bounds. This interval is not an external population coverage guarantee, nor is it a confirmatory significance test. The package also contains an accuracy bootstrap helper for reproducibility.

Configurable release checklist

Default thresholds are deliberately explicit and adjustable. They are engineering heuristics, not peer-reviewed standards:

1. Challenge examples >= 50.
2. Challenge accuracy >= 75%.
3. Controlled-to-challenge gap <= 22 percentage points.
4. At least 10 challenge examples in each language slice.
5. Language-slice accuracy >= 58%.
6. Length histogram PSI <= 0.50.

The challenge length distribution shifts enough that all four baseline methods fail the default PSI criterion. The gate does not perform auto-retraining, auto-approval, or overrides. Lowering thresholds can yield a pass without improving the model; never interpret PASS as a safety or deployment certification.

Bias and fairness limitations

Language and academic-domain groupings are not demographic-protected classes. Aggregate or subgroup comparisons in this synthetic task cannot establish demographic fairness, equitable outcomes, privacy compliance, or production suitability.