Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -2,125 +2,124 @@
|
|
| 2 |
library_name: detectorscope
|
| 3 |
license: gemma
|
| 4 |
base_model: google/gemma-2-2b
|
|
|
|
| 5 |
tags:
|
| 6 |
- ai-text-detection
|
| 7 |
- sparse-autoencoder
|
| 8 |
- interpretability
|
| 9 |
---
|
| 10 |
|
| 11 |
-
#
|
| 12 |
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
```python
|
| 23 |
-
from detectorscope import AuditableDetector
|
| 24 |
|
| 25 |
det = AuditableDetector.from_pretrained("liamdugan/detector-scope")
|
| 26 |
out = det("The authors propose a novel framework for ...")
|
| 27 |
-
|
| 28 |
-
out.
|
|
|
|
|
|
|
| 29 |
```
|
| 30 |
|
| 31 |
-
`
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
`
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
|
| 69 |
-
|
|
| 70 |
-
|
|
| 71 |
-
|
|
| 72 |
-
|
|
| 73 |
-
|
|
| 74 |
-
|
|
| 75 |
-
|
|
| 76 |
-
|
|
| 77 |
-
|
|
| 78 |
-
|
|
| 79 |
-
|
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
|
| 84 |
-
|
|
| 85 |
-
|
|
| 86 |
-
|
|
| 87 |
-
|
|
| 88 |
-
|
|
| 89 |
-
|
|
| 90 |
-
|
|
| 91 |
-
|
|
| 92 |
-
|
|
| 93 |
-
|
|
| 94 |
-
|
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
| 15058 | -0.0435 | wikipedia category and list formatting specifically numerical prefixes and lists |
|
| 121 |
-
| 7991 | +0.0381 | reads more or similar web page navigation links |
|
| 122 |
-
| 13528 | -0.0321 | tokenized components of medical or diagnostic phrasing and lists |
|
| 123 |
-
| 33943 | +0.0309 | repeated phrases or sequences in text |
|
| 124 |
-
| 25049 | -0.0302 | newline characters and line breaks within technical or academic documents |
|
| 125 |
-
| 6780 | -0.0298 | commas occurring within parenthetical or descriptive clauses |
|
| 126 |
-
| 27535 | -0.0296 | newline characters in mathematical or technical contexts |
|
|
|
|
| 2 |
library_name: detectorscope
|
| 3 |
license: gemma
|
| 4 |
base_model: google/gemma-2-2b
|
| 5 |
+
pipeline_tag: text-classification
|
| 6 |
tags:
|
| 7 |
- ai-text-detection
|
| 8 |
- sparse-autoencoder
|
| 9 |
- interpretability
|
| 10 |
---
|
| 11 |
|
| 12 |
+
# detector-scope
|
| 13 |
|
| 14 |
+
Detects AI-generated text across domains and generators (trained on the COLING 2025 GenAI Detection shared task: news, essays, reviews and more, written by people or by 40 language models). The unmodified Gemma-2-2b, sparse autoencoders fine-tuned on this data, and a sparse logistic-regression head over their features: the best general-domain detector of the paper. Every prediction is a sum of contributions from sparse-autoencoder
|
| 15 |
+
features, each with a validated natural-language description, so the detector
|
| 16 |
+
can say which properties of the text drove its decision and where they occur.
|
| 17 |
+
Paper: [Building Auditable AI Text Detectors with Sparse Autoencoders](https://arxiv.org/abs/2610.30006);
|
| 18 |
+
code: [google-deepmind/detectorscope](https://github.com/google-deepmind/detectorscope).
|
| 19 |
+
|
| 20 |
+
## Use
|
| 21 |
+
|
| 22 |
+
```bash
|
| 23 |
+
pip install detectorscope
|
| 24 |
+
```
|
| 25 |
|
| 26 |
```python
|
| 27 |
+
from detectorscope import AuditableDetector
|
| 28 |
|
| 29 |
det = AuditableDetector.from_pretrained("liamdugan/detector-scope")
|
| 30 |
out = det("The authors propose a novel framework for ...")
|
| 31 |
+
|
| 32 |
+
out.p_ai # probability that the text is AI-written
|
| 33 |
+
out.label # "ai" or "human" (logit threshold +0.50)
|
| 34 |
+
out.features # [Feature(id, importance, description, highlights), ...]
|
| 35 |
```
|
| 36 |
|
| 37 |
+
`out.features` lists every feature active in the text with a non-zero effect,
|
| 38 |
+
largest effect first. `importance` is how much the logit falls if that feature
|
| 39 |
+
is silenced: positive pushes toward AI, negative toward human, and
|
| 40 |
+
`out.logit == out.bias + sum(f.importance for f in out.features)`. A list of
|
| 41 |
+
texts returns a list of detections.
|
| 42 |
+
|
| 43 |
+
To see where a feature fires:
|
| 44 |
+
|
| 45 |
+
```python
|
| 46 |
+
out = det(text, return_highlights=True)
|
| 47 |
+
out.tokens # the text as a list of tokens
|
| 48 |
+
out.features[0].highlights # [(token index, activation), ...] of the top feature
|
| 49 |
+
```
|
| 50 |
+
|
| 51 |
+
The default head is logistic regression over the features whose
|
| 52 |
+
auto-interpretation score is at least 0.85. `head="xgboost"` or another
|
| 53 |
+
`tau` (0.00 to 0.95 in steps of 0.05), on `from_pretrained` or on the call,
|
| 54 |
+
selects one of the other 39 heads.
|
| 55 |
+
|
| 56 |
+
## Performance
|
| 57 |
+
|
| 58 |
+
Default head; the decision threshold is fixed on the development data.
|
| 59 |
+
|
| 60 |
+
| split | AUROC | F1 | accuracy | documents |
|
| 61 |
+
|---|---|---|---|---|
|
| 62 |
+
| COLING dev | 0.932 | 0.867 | 0.839 | 261,758 |
|
| 63 |
+
| COLING dev-test | 0.916 | 0.866 | 0.854 | 32,557 |
|
| 64 |
+
| COLING test | 0.857 | 0.751 | 0.770 | 73,941 |
|
| 65 |
+
|
| 66 |
+
## What it looks at
|
| 67 |
+
|
| 68 |
+
The features with the largest average contribution on the training data
|
| 69 |
+
(weight × mean activation when active; density = share of the documents of
|
| 70 |
+
each class in which the feature fires).
|
| 71 |
+
|
| 72 |
+
**Toward AI**
|
| 73 |
+
|
| 74 |
+
| feature | description | mean contribution | AI density | human density |
|
| 75 |
+
|---|---|---|---|---|
|
| 76 |
+
| 13_3738 | introductory phrases or filler text commonly used by ai language models when declining personal questions | +0.987 | 25% | 2% |
|
| 77 |
+
| 13_1545 | academic and scientific research methodology and technical analysis terminology | +0.909 | 13% | 3% |
|
| 78 |
+
| 17_12178 | critical commentary or argumentative analysis of student policies and social responsibility | +0.558 | 38% | 11% |
|
| 79 |
+
| 17_8638 | neuron activates on random text fragments and noisy character sequences often found in garbled or fragmented data | +0.372 | 13% | 13% |
|
| 80 |
+
| 17_1175 | repeated phrases or sequences in text | +0.363 | 41% | 31% |
|
| 81 |
+
| 13_2153 | exclamation points used at the end of emotional or emphatic sentences | +0.318 | 16% | 15% |
|
| 82 |
+
| 22_9579 | markdown formatting for headers or list items | +0.297 | 25% | 13% |
|
| 83 |
+
| 7_1076 | conversational filler phrases used in informal speech | +0.261 | 15% | 13% |
|
| 84 |
+
| 7_7991 | reads more or similar web page navigation links | +0.239 | 8% | 7% |
|
| 85 |
+
| 17_4337 | introductory transition words signaling a summary or final thought | +0.210 | 25% | 7% |
|
| 86 |
+
|
| 87 |
+
**Toward human**
|
| 88 |
+
|
| 89 |
+
| feature | description | mean contribution | AI density | human density |
|
| 90 |
+
|---|---|---|---|---|
|
| 91 |
+
| 7_14967 | comma punctuation following a newline or preceding sentence start | -1.563 | 3% | 14% |
|
| 92 |
+
| 7_10954 | neuron fires on the digit 0 inside url placeholders like url 0 | -0.951 | 1% | 5% |
|
| 93 |
+
| 7_9535 | newline characters in scientific or technical text | -0.842 | 1% | 11% |
|
| 94 |
+
| 7_1262 | contractions containing the letter n and apostrophe specifically focusing on the n't ending | -0.785 | 3% | 14% |
|
| 95 |
+
| 7_5896 | newline characters in scientific or technical text | -0.658 | 1% | 11% |
|
| 96 |
+
| 13_11151 | newline characters in mathematical or technical contexts | -0.473 | 2% | 12% |
|
| 97 |
+
| 7_14666 | wikipedia category and navigation section headers | -0.454 | 4% | 10% |
|
| 98 |
+
| 22_3894 | semicolon followed by a newline character | -0.394 | 3% | 10% |
|
| 99 |
+
| 17_15548 | opening double quotation marks following a closing single quote and comma | -0.374 | 1% | 4% |
|
| 100 |
+
| 13_15646 | markdown editing and formatting indicators | -0.366 | 14% | 29% |
|
| 101 |
+
|
| 102 |
+
## Variants
|
| 103 |
+
|
| 104 |
+
Branches of this repo hold the paper's other two settings on the same data: `ft-model-sae` (Gemma-2-2b fine-tuned as a classifier, with SAEs fine-tuned on it) and `base-sae` (the unmodified Gemma-2-2b with the Gemma Scope SAEs). Load one with `revision="ft-model-sae"` or `revision="base-sae"`. For reviews of machine-learning papers use [detector-scope-peer-review-gpt4o](https://huggingface.co/liamdugan/detector-scope-peer-review-gpt4o) or [detector-scope-peer-review-llama](https://huggingface.co/liamdugan/detector-scope-peer-review-llama).
|
| 105 |
+
|
| 106 |
+
## Licence
|
| 107 |
+
|
| 108 |
+
The language model under `extractor/` is Gemma-2-2b, provided under
|
| 109 |
+
and subject to the [Gemma Terms of Use](https://ai.google.dev/gemma/terms)
|
| 110 |
+
(see `NOTICE`). The sparse autoencoders derive from
|
| 111 |
+
[Gemma Scope](https://huggingface.co/google/gemma-scope-2b-pt-res) (CC-BY-4.0);
|
| 112 |
+
the heads and feature descriptions are CC-BY-4.0.
|
| 113 |
+
|
| 114 |
+
## Citation
|
| 115 |
+
|
| 116 |
+
```bibtex
|
| 117 |
+
@misc{dugan2026detectorscope,
|
| 118 |
+
title={Building Auditable AI Text Detectors with Sparse Autoencoders},
|
| 119 |
+
author={Liam Dugan and Callum McDougall and Matthew Denton and Phillip Pham and Christine Kaeser-Chen and Chris Callison-Burch},
|
| 120 |
+
year={2026},
|
| 121 |
+
eprint={2610.30006},
|
| 122 |
+
archivePrefix={arXiv},
|
| 123 |
+
url={https://arxiv.org/abs/2610.30006},
|
| 124 |
+
}
|
| 125 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|