README.md

metadata

title: RM-DETECT
emoji: 🔍
colorFrom: red
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
license: mit

RM-DETECT

AI 작성 텍스트 탐지 웹 앱 · a transparent, self-hostable AI-vs-human text detector (Korean & English).

RM-DETECT is an independently built detector that estimates how likely a piece of text was written by a large language model. It combines token predictability (perplexity) with stylometric analysis — both well-established signals in the AI-text-detection literature — behind a FastAPI service and a single-page web UI. Paste text → get an overall AI-likelihood score, a verdict, and a per-paragraph 의심 영역 view where each paragraph is colour-graded from green (human) through yellow (mixed) to red (AI), with the top contributing signals shown on click.

Every score is explainable: the model reports which features pushed a verdict toward "AI" or "human", so it is not a black box.

📐 상세 내부 구조·데이터 흐름·피처 사양은 ARCHITECTURE.md 참고.

---

Detection approach

RM-DETECT fuses two independent, complementary signal families into one classifier:

1. Token predictability (perplexity). A small causal language model (GPT-2 for English, KoGPT2 for Korean) scores how predictable each token is. LLMs generate text auto-regressively by favouring high-probability next tokens, so machine-written text tends to be more predictable — lower perplexity and a higher fraction of tokens the model would itself have ranked at the top. This is the same principle behind academic detectors such as GLTR and DetectGPT.
2. Stylometry (model-free). Sentence-length burstiness, logical-connective density, punctuation / emoji / informal-marker density (ㅋㅋ, …, lol), lexical repetition and entropy, and Korean formal-vs-colloquial register endings (하였습니다 vs ~했음). Human writing varies far more; AI tends to a uniform, formal rhythm.

A logistic-regression classifier fuses 28 features in total. On the calibration set it reached 97.2% cross-validated accuracy (AUC 0.994). Every paragraph is scored independently, so mixed human/AI documents are broken down region by region.

---

Run it

# 1. install deps (CPU is fine)
pip install -r requirements.txt

# 2. models — unpack the two language models into ./models/
#    (from models_gpt2_kogpt2.tar.gz), giving ./models/gpt2 and ./models/kogpt2.
#    Or skip this and let transformers pull them from the HF hub on first run.
mkdir -p models && tar -xzf ../models_gpt2_kogpt2.tar.gz -C models

# 3. launch
./run.sh                       # -> http://127.0.0.1:8000
# or:  uvicorn app:app --host 127.0.0.1 --port 8000

Open http://127.0.0.1:8000 in a browser, paste text, and click 분석하기.

Prefer not to run a server? Open ../rmdetect_demo.html directly in a browser — it's the real UI rendered with a real detection response.

---

API

GET /health → {status, model_loaded, uptime_s, max_chars}

POST /detect

// request
{ "text": "검사할 글 ...", "lang": "ko" }   // lang optional: "ko" | "en" | omit for auto

// response (abridged)
{
  "language": "ko",
  "overall_ai_probability": 0.50,
  "verdict": "Mixed",
  "n_paragraphs": 3,
  "n_flagged": 1,
  "top_features": [ {"feature": "informal_per100", "contribution": -1.17, "direction": "human"}, ... ],
  "paragraphs": [
    { "index": 0, "start": 0, "end": 40, "text": "...",
      "ai_probability": 0.35, "verdict": "Likely human",
      "flagged": false, "low_confidence": false,
      "top_features": [ ... ] }
  ],
  "elapsed_ms": 2255.8
}

start/end are character offsets into the submitted text, so a client can highlight the exact source region of each paragraph.

Verdict bands: ≥0.85 AI-generated · ≥0.60 Likely AI · ≥0.40 Mixed · ≥0.15 Likely human · else Human-written. A paragraph is flagged at ai_probability ≥ 0.60.

---

Files

path	purpose
app.py	FastAPI backend (/health, /detect, serves UI)
static/index.html, style.css, app.js	single-page UI
aidetect/	detection engine (perplexity + stylometry)
ai_detector_model.joblib	fitted classifier
requirements.txt, run.sh	install + launch

---

Limitations (MVP)

RM-DETECT is a research prototype, not a definitive judgment tool. Carried over from the underlying engine:

• Small calibration set (n=36) and LLM-imitated "human" samples — real, especially formal, human writing (academic papers, polished 자기소개서) will over-flag more than the headline accuracy suggests: 인간 글이라도 너무 형식적이면 오탐지될 수 있음.
• Small reference LMs (GPT-2/KoGPT2, ~125M) — a larger or instruction-tuned LM would sharpen the perplexity signal.
• Paraphrase-vulnerable by design — the detector keys on burstiness and informality, so "humanizing" edits (adding colloquialisms, varying sentence length) lower the AI score.
• CPU latency — ~1–2 s per request on a multi-core machine; noticeably slower on shared free-tier CPU. Thresholds should be recalibrated per deployment and per acceptable false-positive rate.

Do not use RM-DETECT as sole grounds for a consequential decision (academic misconduct, hiring). Treat its output as one probabilistic signal among many.