docs/methodology.md

# Methodology

## Research Question

Do AI-generated-text detectors — built and validated almost entirely on English — work on
Hinglish (Hindi-English code-mixed, Roman script) text? And how does their accuracy change
as that text is paraphrased at increasing strength (a common evasion tactic)?

## Background / Gap

PADBen (Zha, Min & Sushmita, 2025, arXiv:2511.00416) is the most recent and comprehensive
benchmark for AI-text detector robustness against paraphrase attacks. It introduces a
five-type text taxonomy (Human Original, LLM-Generated, Human-Paraphrased, LLM-Paraphrased,
Iteratively-Paraphrased) and evaluates 11 detectors — but exclusively on English datasets
(MRPC, PAWS, HLPC). The paper's own Limitations section notes that testing beyond 3 paraphrase
iterations and adapting/fine-tuning detectors were both left as future work; multilingual or
code-mixed text is not addressed at all.

A search across the current literature (PADBen itself, MULTITUDE, CodeMirage, PAN 2026 tasks,
RAID, and industry reporting from GPTZero/Turnitin/Originality.ai) found no existing study
combining AI-text-detection-under-paraphrase-attack methodology with Hinglish/code-mixed text.

## Data Sources

- **PHINC** (Srivastava & Singh, WNUT 2020 @ EMNLP, arXiv:2004.09447) — 13,738 real,
  sentence-level Hinglish social media posts, HuggingFace: `LingoIITGN/PHINC`.
- **India Hate Speech Superset** (Tonneau et al., 2024, arXiv:2411.15462) — used as a
  secondary/supplementary source; HuggingFace: `manueltonneau/india-hate-speech-superset`.
  Note: this dataset is mostly pure Devanagari-script Hindi, not Roman-script Hinglish —
  only a small subset (~180 rows) was genuinely code-mixed and usable.

Two earlier candidate sources were tried and dropped:
- **SAIL 2017** — original host (`amitavadas.com`) has expired/been taken over by an
  unrelated site.
- **L3Cube-HingLID** — turned out to be a word-level language-identification dataset
  (single tokens tagged EN/HI), not sentence-level text, and so unsuitable for this pipeline.

## Pipeline (following PADBen's taxonomy)

1. **Type 1 — Human Original**: 9,736 cleaned, deduplicated (TF-IDF cosine similarity,
   threshold 0.85), quality-filtered (10–1000 chars, valid UTF-8) Roman-script Hinglish
   sentences from PHINC + the usable subset of the secondary source.
2. **Type 2 — LLM-Generated**: For each human sentence, the first ~20% of words is used as
   a seed prompt; Llama 3.1-8B-Instant (via Groq, free tier) is asked to continue it
   naturally in Hinglish. This keeps topic/opening matched between human and AI text.
3. **Type 4 — LLM-Paraphrased** / **Type 5 — Iteratively-Paraphrased**: Each Type-2 text is
   paraphrased once (1x), then the paraphrase is paraphrased again (2x), then again (3x),
   giving a full attack-strength gradient in one pass. Same model (Llama 3.1-8B-Instant) used
   throughout for consistency.
   - **Type 3 (Human-Paraphrased) was not collected** — no access to human paraphrasers at
     scale on a zero-budget solo project. This is a documented limitation, not an omission.

## Detectors Evaluated

1. **RADAR** (`TrustSafeAI/RADAR-Vicuna-7B`) — open-weight, RoBERTa-based (~355M params)
   classifier trained adversarially against paraphrasing, on English (Vicuna-7B) outputs.
2. **Zero-shot perplexity baseline** (GPT-2) — classic detection heuristic: lower perplexity
   (text is more "predictable" to the model) often correlates with AI-generated text.

*(Binoculars and Fast-DetectGPT were considered but not included in this pass — both require
loading two full LLMs simultaneously and more complex scoring logic, which was out of scope
for the free-tier compute budget of this stage. Documented as a direction for follow-up work.)*

## Evaluation

- Sample: 1,000 rows per text type (human_original, ai_generated, paraphrased 1x/2x/3x) =
  5,000 total evaluation rows, sampled from the full 9,736-row dataset to fit within Kaggle's
  free GPU quota.
- Metrics: AUROC (human vs. each AI text type) and accuracy at a Youden's-J-optimal threshold
  (calibrated on the ai_generated-vs-human split, then applied unchanged to the paraphrased
  levels — same practical-detection-rate approach PADBen itself uses alongside AUROC).

## Key Finding

RADAR AUROC and accuracy remained essentially flat (AUROC 0.84–0.85, accuracy ~77–78%)
across all paraphrase strength levels — no meaningful degradation, unlike the sharp drops
PADBen reports for English. The perplexity baseline showed the same pattern (AUROC
0.83–0.86, accuracy ~77–79%, roughly stable or even slightly improving under paraphrase).

Separately, RADAR's raw classification score was found to be **inverted** on this Hinglish
data relative to its intended direction (real human text scored as more "AI-like" than
actual AI text) — evidence the detector, trained purely on English OpenWebText-style data,
does not transfer cleanly to short, casual, code-mixed text.

## Limitations

- Evaluation sample (1,000/type) is a subset of the full generated dataset, constrained by
  free-tier GPU quota — not the full 9,736 rows.
- Only 2 detectors evaluated (RADAR, GPT-2 perplexity) vs. the 11 in PADBen's original study.
- The same underlying model (Llama 3.1-8B-Instant) was used for both AI-text generation and
  paraphrasing, which introduces some risk of self-consistency/circularity — ideally a
  different model family would be used for each step.
- No formal statistical significance testing (e.g., confidence intervals, p-values) was run
  on the metric differences across paraphrase levels — the "stable" finding is based on the
  observed AUROC/accuracy numbers, not a hypothesis test.
- Type 3 (Human-Paraphrased) is entirely absent from this dataset.

## Attribution

- PADBen: Zha, Y., Min, R., & Sushmita, S. (2025). *PADBen: A Comprehensive Benchmark for
  Evaluating AI Text Detectors Against Paraphrase Attacks.* arXiv:2511.00416.
- PHINC: Srivastava, V., & Singh, M. (2020). *PHINC: A Parallel Hinglish Social Media Code-Mixed
  Corpus for Machine Translation.* WNUT 2020 @ EMNLP. arXiv:2004.09447.
- India Hate Speech Superset: Tonneau, M., et al. (2024). arXiv:2411.15462.
- RADAR: TrustSafeAI/RADAR-Vicuna-7B, HuggingFace.