README.md

---
language:
  - ko
license: gemma
library_name: peft
pipeline_tag: text-classification
base_model: google/embeddinggemma-300m
base_model_relation: adapter
tags:
  - text-classification
  - ai-text-detection
  - authorship-analysis
  - korean
  - fiction
  - stylometry
---

# ⚠️ Do not use this model as evidence of AI authorship.

**Munche-768-AI-Detector was built for dataset triage and proof-of-concept research. It cannot establish plagiarism, copyright infringement, misconduct, or whether a person or an AI wrote a text. Its output can be wrong.**

# Munche-768-AI-Detector

<p align="center">
  <img src="./assets/baragi-ai.png" width="128" alt="Baragi AI">
</p>

Munche-768-AI-Detector classifies Korean genre-fiction passages as `human`, `uncertain`, or `llm`. It starts from [Munche-768](https://huggingface.co/Baragi-AI/Munche-768), then jointly tunes LoRA weights in the top four Transformer layers and a 769-parameter linear classifier.

## Demo

[**Run Munche-768-AI-Detector in your browser**](https://huggingface.co/spaces/ij/Munche-768-AI-Detector)
![image](https://cdn-uploads.huggingface.co/production/uploads/63007fd0c1e149ceaff6e43a/bJrIwTjPuAr2XY9fZyc3O.png)

## Overall test result

The sealed test combines 277 human passages and 256 LLM passages from the independent-generation and content-preserving rewrite evaluations.

| Metric | Result |
|---|---:|
| AUROC | **98.59%** |
| Binary accuracy | **94.00%** |
| Balanced accuracy | **93.91%** |
| Human recall | **96.03%** |
| LLM recall | **91.80%** |
| Human false-positive rate | **3.97%** |

<p align="center">
  <img src="./assets/overall-confusion-matrix.svg" width="900" alt="Overall binary confusion matrix across independent generation and content-preserving rewrites">
</p>

## Independent-generation test

The sealed test contains 87 passages from human-written novels and 66 passages written directly by 11 language-model families. Content-preserving rewrites are excluded from this evaluation.

| Metric | Result |
|---|---:|
| Binary accuracy | **100.00%** |
| Balanced accuracy | **100.00%** |
| Human recall | **100.00%** |
| LLM recall | **100.00%** |
| Human false-positive rate | **0.00%** |

<p align="center">
  <img src="./assets/independent-confusion-matrix.svg" width="900" alt="Binary confusion matrix for independently written human and LLM fiction">
</p>

## Three-way decision

The thresholds were selected on validation data only. The classification score is not a calibrated probability that a passage was written by AI. Results below cover the full sealed test.

| Score | Output |
|---:|---|
| `≤ 0.5000` | Human |
| `0.5000 < score < 0.8697` | Uncertain |
| `≥ 0.8697` | LLM |

| Metric | Result |
|---|---:|
| Coverage | **93.62%** |
| Accuracy among classified passages | **95.19%** |
| Human classified as LLM | **1.08%** |
| Human classified as uncertain | **2.89%** |
| LLM classified as human | **8.20%** |
| LLM classified as uncertain | **10.16%** |

Within the independent-generation subset, all 87 human passages received a human decision. Of the 66 LLM passages, 64 received an LLM decision and two were uncertain.

<p align="center">
  <img src="./assets/three-way-outcomes.svg" width="900" alt="Human, uncertain, and LLM outcomes for the independent-generation test">
</p>

## Content-preserving rewrite test

This test contains 190 human passages and 190 LLM rewrites that preserve the source content. It is harder than distinguishing independently written human and LLM fiction.

| Metric | Binary decision | Three-way decision |
|---|---:|---:|
| AUROC | **97.47%** | N/A |
| Balanced accuracy | **91.58%** | N/A |
| Human recall | **94.21%** | N/A |
| LLM recall | **88.95%** | N/A |
| Human classified as LLM | 5.79% | **1.58%** |
| LLM classified as human | 11.05% | **11.05%** |
| Coverage | N/A | **91.58%** |
| Accuracy among classified passages | N/A | **93.10%** |

<p align="center">
  <img src="./assets/benchmark-comparison.svg" width="900" alt="Performance comparison between independent generation and content-preserving rewrites">
</p>

## Input length

Use passages between **384 and 2,048 EmbeddingGemma tokens**. Inputs shorter than 384 tokens are not supported as stable operating inputs. Inputs longer than 2,048 tokens must be divided into separate windows before classification.

The training and evaluation corpora covered short rewrite passages near 400 tokens and independent fiction passages near the 2,048-token model limit. Document-level aggregation across multiple windows has not been calibrated.

## Usage

Access to the gated EmbeddingGemma base model is required.

```bash
pip install torch numpy sentence-transformers peft safetensors
```

```python
from inference import MuncheAIDetector

detector = MuncheAIDetector(".")
result = detector.predict(korean_fiction_passage)

print(result)
# {"label": "human" | "uncertain" | "llm", "score": float, "tokens": int}
```

## Training data

| Split | Human | LLM |
|---|---:|---:|
| Train | 1,315 | 1,135 |
| Validation | 290 | 275 |
| Test | 277 | 256 |

The training set combines human-written Korean genre fiction, independently generated LLM fiction, and content-preserving LLM rewrites. GPT-5.6 Sol Medium contributes 24 training passages and six validation passages. No Sol Medium passage was added to test.

The detector was initialized from Munche-768. Only LoRA weights in Transformer layers 20-23 and the linear classifier were updated. The selected checkpoint is step 275. A preservation loss limited movement away from the original Munche-768 embedding during tuning.

Raw human fiction is not distributed with this repository.

## Limitations

- The model was trained and evaluated on Korean genre fiction.
- Generalization to language-model families absent from training remains unknown.
- The model cannot identify text jointly written or substantially edited by humans and AI.

## License

Munche-768-AI-Detector is derived from `google/embeddinggemma-300m` and Munche-768. Use is subject to the Gemma license and the access terms of the gated base model.