docs/methodology.md

Methodology

Research Question

Do AI-generated-text detectors — built and validated almost entirely on English — work on Hinglish (Hindi-English code-mixed, Roman script) text? And how does their accuracy change as that text is paraphrased at increasing strength (a common evasion tactic)?

Background / Gap

PADBen (Zha, Min & Sushmita, 2025, arXiv:2511.00416) is the most recent and comprehensive benchmark for AI-text detector robustness against paraphrase attacks. It introduces a five-type text taxonomy (Human Original, LLM-Generated, Human-Paraphrased, LLM-Paraphrased, Iteratively-Paraphrased) and evaluates 11 detectors — but exclusively on English datasets (MRPC, PAWS, HLPC). The paper's own Limitations section notes that testing beyond 3 paraphrase iterations and adapting/fine-tuning detectors were both left as future work; multilingual or code-mixed text is not addressed at all.

A search across the current literature (PADBen itself, MULTITUDE, CodeMirage, PAN 2026 tasks, RAID, and industry reporting from GPTZero/Turnitin/Originality.ai) found no existing study combining AI-text-detection-under-paraphrase-attack methodology with Hinglish/code-mixed text.

Data Sources

• PHINC (Srivastava & Singh, WNUT 2020 @ EMNLP, arXiv:2004.09447) — 13,738 real, sentence-level Hinglish social media posts, HuggingFace: LingoIITGN/PHINC.
• India Hate Speech Superset (Tonneau et al., 2024, arXiv:2411.15462) — used as a secondary/supplementary source; HuggingFace: manueltonneau/india-hate-speech-superset. Note: this dataset is mostly pure Devanagari-script Hindi, not Roman-script Hinglish — only a small subset (~180 rows) was genuinely code-mixed and usable.

Two earlier candidate sources were tried and dropped:

• SAIL 2017 — original host (amitavadas.com) has expired/been taken over by an unrelated site.
• L3Cube-HingLID — turned out to be a word-level language-identification dataset (single tokens tagged EN/HI), not sentence-level text, and so unsuitable for this pipeline.

Pipeline (following PADBen's taxonomy)

1. Type 1 — Human Original: 9,736 cleaned, deduplicated (TF-IDF cosine similarity, threshold 0.85), quality-filtered (10–1000 chars, valid UTF-8) Roman-script Hinglish sentences from PHINC + the usable subset of the secondary source.
2. Type 2 — LLM-Generated: For each human sentence, the first ~20% of words is used as a seed prompt; Llama 3.1-8B-Instant (via Groq, free tier) is asked to continue it naturally in Hinglish. This keeps topic/opening matched between human and AI text.
3. Type 4 — LLM-Paraphrased / Type 5 — Iteratively-Paraphrased: Each Type-2 text is paraphrased once (1x), then the paraphrase is paraphrased again (2x), then again (3x), giving a full attack-strength gradient in one pass. Same model (Llama 3.1-8B-Instant) used throughout for consistency.
   • Type 3 (Human-Paraphrased) was not collected — no access to human paraphrasers at scale on a zero-budget solo project. This is a documented limitation, not an omission.

Detectors Evaluated

1. RADAR (TrustSafeAI/RADAR-Vicuna-7B) — open-weight, RoBERTa-based (~355M params) classifier trained adversarially against paraphrasing, on English (Vicuna-7B) outputs.
2. Zero-shot perplexity baseline (GPT-2) — classic detection heuristic: lower perplexity (text is more "predictable" to the model) often correlates with AI-generated text.

(Binoculars and Fast-DetectGPT were considered but not included in this pass — both require loading two full LLMs simultaneously and more complex scoring logic, which was out of scope for the free-tier compute budget of this stage. Documented as a direction for follow-up work.)

Evaluation

• Sample: 1,000 rows per text type (human_original, ai_generated, paraphrased 1x/2x/3x) = 5,000 total evaluation rows, sampled from the full 9,736-row dataset to fit within Kaggle's free GPU quota.
• Metrics: AUROC (human vs. each AI text type) and accuracy at a Youden's-J-optimal threshold (calibrated on the ai_generated-vs-human split, then applied unchanged to the paraphrased levels — same practical-detection-rate approach PADBen itself uses alongside AUROC).

Key Finding

RADAR AUROC and accuracy remained essentially flat (AUROC 0.84–0.85, accuracy ~77–78%) across all paraphrase strength levels — no meaningful degradation, unlike the sharp drops PADBen reports for English. The perplexity baseline showed the same pattern (AUROC 0.83–0.86, accuracy ~77–79%, roughly stable or even slightly improving under paraphrase).

Separately, RADAR's raw classification score was found to be inverted on this Hinglish data relative to its intended direction (real human text scored as more "AI-like" than actual AI text) — evidence the detector, trained purely on English OpenWebText-style data, does not transfer cleanly to short, casual, code-mixed text.

Limitations

• Evaluation sample (1,000/type) is a subset of the full generated dataset, constrained by free-tier GPU quota — not the full 9,736 rows.
• Only 2 detectors evaluated (RADAR, GPT-2 perplexity) vs. the 11 in PADBen's original study.
• The same underlying model (Llama 3.1-8B-Instant) was used for both AI-text generation and paraphrasing, which introduces some risk of self-consistency/circularity — ideally a different model family would be used for each step.
• No formal statistical significance testing (e.g., confidence intervals, p-values) was run on the metric differences across paraphrase levels — the "stable" finding is based on the observed AUROC/accuracy numbers, not a hypothesis test.
• Type 3 (Human-Paraphrased) is entirely absent from this dataset.

Attribution

• PADBen: Zha, Y., Min, R., & Sushmita, S. (2025). PADBen: A Comprehensive Benchmark for Evaluating AI Text Detectors Against Paraphrase Attacks. arXiv:2511.00416.
• PHINC: Srivastava, V., & Singh, M. (2020). PHINC: A Parallel Hinglish Social Media Code-Mixed Corpus for Machine Translation. WNUT 2020 @ EMNLP. arXiv:2004.09447.
• India Hate Speech Superset: Tonneau, M., et al. (2024). arXiv:2411.15462.
• RADAR: TrustSafeAI/RADAR-Vicuna-7B, HuggingFace.