# Methodology: RobustnessLab AI

## Fixed research design

The source Academic Rhetorical Intent Mini v1.1 has 842 synthetic-curated training records, 130 controlled test records, and 54 novel-pattern challenge records, with six rhetorical classes in English, Spanish and Portuguese. No held-out row is used to fit the four baselines. The original prior project `source_modelops_engine.js` is retained as provided; `generate_artifacts.js` trains it once and evaluates every original source row under all conditions.

The evidence matrix contains 184 source examples × 4 methods × 6 conditions = 4,416 prediction records. Each row carries source ID, split, method, attack, language, domain, original source text, perturbed text, inherited class, clean prediction, transformed prediction, raw six-class output scores and top-two margin. SHA-256 hashes capture source artifacts. The app loads stored evidence and trains the bundled model locally for the custom-text sandbox.

## Perturbations and how scores are interpreted

Six conditions: clean baseline; case/punctuation formatting; deterministic letter transposition; small dictionary replacement; appended contextual research sentence; and destructive truncation to approximately 65% of words.

A transformed text is considered **changed** only when its bytes differ from the original text. Dictionary substitution is sparse: if no word matches, its changed-row coverage is less than 100%.

**Prediction flip rate** = # transformed predictions differing from the original prediction / # all attempted evaluation items. A per-changed-only flip rate is available from `summarize().changedFlipRate`.

**Modified-subset accuracy** = # correct transformed predictions on *actually changed* rows / # actually changed rows. It assumes the inherited gold label remains applicable, which is not human verified. **Matched baseline accuracy** scores the unchanged original predictions on the **same changed IDs**. Therefore `accuracyDelta` is paired modified accuracy minus paired original accuracy, in percentage points.

Destructive deletion can change the true class, so accuracy, matched accuracy and accuracy differences are **intentionally null and not rendered** for that condition. Its flips are still descriptive.

## Paired bootstrap

Using a deterministic PRNG and a fixed number of resampling iterations, sample changed IDs with replacement and calculate paired differences `(transformed_correct - original_correct)`. Percentile boundaries are descriptive resampling intervals, not frequentist confidence guarantees for an external population. Small samples, especially the sparse dictionary transformation, are highly unstable.

## Selective prediction

Each classifier ranks its own clean predictions by the difference between its largest and second-largest output scores. This margin is **not a calibrated probability**, and cannot be compared numerically across different baselines. For each top-k prefix, `risk(k)=# errors in top-k/k`, and `coverage(k)=k/N`. The app plots this risk-coverage curve and computes the arithmetic average of prefix risk (AURC). These measures do not establish safe decision-making or reliable abstention outside the synthetic data.

## Reliability diagnostic

Only Multinomial Naive Bayes returns normalized model posterior values, which are **not assumed calibrated**. Eight equal-width bins group top-class posteriors; for each bin the app computes mean posterior and empirical accuracy. ECE = Σ (bin_n/N) × |mean_posterior − empirical_accuracy|. All bins with zero support contribute zero. Small-sample ECE can be misleading and changes with binning.

Word/character TF-IDF scores are cosine similarities, and hybrid output is a weighted similarity. No probability calibration or ECE is reported for these three baselines.

## Research gates

A configurable checklist checks sample size, prediction flip ceilings and maximum matched accuracy deterioration across the four **non-destructive** perturbation families. The gate always lists failing checks. It is NOT a fairness, robustness, safety, compliance, privacy or deployment certification.

## Nonclaims

This is NOT an adversarial red-team of an online foundation model, a naturally occurring error corpus, proof of semantic equivalence, test-time OOD detection, calibration transfer evaluation, fairness analysis or external academic-text performance. The source examples are synthetic. The model is no substitute for expert review.