Spaces:
Running
Methodology: RobustnessLab AI
Fixed research design
The source Academic Rhetorical Intent Mini v1.1 has 842 synthetic-curated training records, 130 controlled test records, and 54 novel-pattern challenge records, with six rhetorical classes in English, Spanish and Portuguese. No held-out row is used to fit the four baselines. The original prior project source_modelops_engine.js is retained as provided; generate_artifacts.js trains it once and evaluates every original source row under all conditions.
The evidence matrix contains 184 source examples × 4 methods × 6 conditions = 4,416 prediction records. Each row carries source ID, split, method, attack, language, domain, original source text, perturbed text, inherited class, clean prediction, transformed prediction, raw six-class output scores and top-two margin. SHA-256 hashes capture source artifacts. The app loads stored evidence and trains the bundled model locally for the custom-text sandbox.
Perturbations and how scores are interpreted
Six conditions: clean baseline; case/punctuation formatting; deterministic letter transposition; small dictionary replacement; appended contextual research sentence; and destructive truncation to approximately 65% of words.
A transformed text is considered changed only when its bytes differ from the original text. Dictionary substitution is sparse: if no word matches, its changed-row coverage is less than 100%.
Prediction flip rate = # transformed predictions differing from the original prediction / # all attempted evaluation items. A per-changed-only flip rate is available from summarize().changedFlipRate.
Modified-subset accuracy = # correct transformed predictions on actually changed rows / # actually changed rows. It assumes the inherited gold label remains applicable, which is not human verified. Matched baseline accuracy scores the unchanged original predictions on the same changed IDs. Therefore accuracyDelta is paired modified accuracy minus paired original accuracy, in percentage points.
Destructive deletion can change the true class, so accuracy, matched accuracy and accuracy differences are intentionally null and not rendered for that condition. Its flips are still descriptive.
Paired bootstrap
Using a deterministic PRNG and a fixed number of resampling iterations, sample changed IDs with replacement and calculate paired differences (transformed_correct - original_correct). Percentile boundaries are descriptive resampling intervals, not frequentist confidence guarantees for an external population. Small samples, especially the sparse dictionary transformation, are highly unstable.
Selective prediction
Each classifier ranks its own clean predictions by the difference between its largest and second-largest output scores. This margin is not a calibrated probability, and cannot be compared numerically across different baselines. For each top-k prefix, risk(k)=# errors in top-k/k, and coverage(k)=k/N. The app plots this risk-coverage curve and computes the arithmetic average of prefix risk (AURC). These measures do not establish safe decision-making or reliable abstention outside the synthetic data.
Reliability diagnostic
Only Multinomial Naive Bayes returns normalized model posterior values, which are not assumed calibrated. Eight equal-width bins group top-class posteriors; for each bin the app computes mean posterior and empirical accuracy. ECE = Σ (bin_n/N) × |mean_posterior − empirical_accuracy|. All bins with zero support contribute zero. Small-sample ECE can be misleading and changes with binning.
Word/character TF-IDF scores are cosine similarities, and hybrid output is a weighted similarity. No probability calibration or ECE is reported for these three baselines.
Research gates
A configurable checklist checks sample size, prediction flip ceilings and maximum matched accuracy deterioration across the four non-destructive perturbation families. The gate always lists failing checks. It is NOT a fairness, robustness, safety, compliance, privacy or deployment certification.
Nonclaims
This is NOT an adversarial red-team of an online foundation model, a naturally occurring error corpus, proof of semantic equivalence, test-time OOD detection, calibration transfer evaluation, fairness analysis or external academic-text performance. The source examples are synthetic. The model is no substitute for expert review.