Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,3 +1,87 @@
|
|
| 1 |
-
---
|
| 2 |
-
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
- es
|
| 5 |
+
pipeline_tag: text-classification
|
| 6 |
+
tags:
|
| 7 |
+
- ai-generated-text-detection
|
| 8 |
+
- text-classification
|
| 9 |
+
- ensemble
|
| 10 |
+
- deberta
|
| 11 |
+
- lightgbm
|
| 12 |
+
- tf-idf
|
| 13 |
+
- pytorch
|
| 14 |
+
- scikit-learn
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# EC_MODELS (AI vs Human Detection)
|
| 18 |
+
|
| 19 |
+
This repository contains **model artifacts** for AI‑generated text detection experiments, including a HOPE classifier and a stacked ensemble (DeBERTa + LightGBM + TF‑IDF/SGD). It is **not a single model**, but a collection of checkpoints and meta‑models produced during experiments.
|
| 20 |
+
|
| 21 |
+
## Model inventory
|
| 22 |
+
|
| 23 |
+
Artifacts are stored in `src/ai_vs_human/models/`:
|
| 24 |
+
|
| 25 |
+
- `hope/`
|
| 26 |
+
HOPE checkpoints (`fold_*/best_model.pt`) for a transformer‑based classifier with memory modules.
|
| 27 |
+
- `deberta_v3_base/`
|
| 28 |
+
DeBERTa base checkpoints used in the ensemble.
|
| 29 |
+
- `lightgbm/`, `lightgbm_numeric/`
|
| 30 |
+
LightGBM models on engineered features.
|
| 31 |
+
- `tfidf_sgd/`
|
| 32 |
+
TF‑IDF + SGD models.
|
| 33 |
+
- `stack_meta/meta_learner.joblib`
|
| 34 |
+
Logistic‑regression meta‑learner used for stacking.
|
| 35 |
+
|
| 36 |
+
Out‑of‑fold predictions used to compute metrics and train the stacker are in:
|
| 37 |
+
|
| 38 |
+
- `src/ai_vs_human/oof/` (e.g., `oof_stack.csv`, `oof_deberta.csv`, `oof_lgb.csv`, `oof_sgd.csv`)
|
| 39 |
+
|
| 40 |
+
## Intended use
|
| 41 |
+
|
| 42 |
+
These models are intended for **research and evaluation** of AI‑generated text detection. They can be used to:
|
| 43 |
+
|
| 44 |
+
- compare HOPE vs. ensemble baselines,
|
| 45 |
+
- reproduce experiments from the notebooks,
|
| 46 |
+
- evaluate domain‑shift and multilingual robustness.
|
| 47 |
+
|
| 48 |
+
## How to evaluate (metrics only)
|
| 49 |
+
|
| 50 |
+
Use the metrics‑only notebook to compute standard metrics without retraining:
|
| 51 |
+
|
| 52 |
+
- `src/ai_vs_human/metrics_only.ipynb`
|
| 53 |
+
|
| 54 |
+
This notebook loads `oof_stack.csv` and prints AUC‑ROC, PR‑AUC, Accuracy, Precision, Recall, F1, Brier, and ECE, plus a best‑F1 threshold.
|
| 55 |
+
|
| 56 |
+
## Training data
|
| 57 |
+
|
| 58 |
+
Training data and features were produced from the project’s datasets under `src/ai_vs_human/`, including:
|
| 59 |
+
|
| 60 |
+
- `merged_ai_human_multisocial_features*.csv`
|
| 61 |
+
|
| 62 |
+
See the dataset‑building and training notebooks for details:
|
| 63 |
+
|
| 64 |
+
- `src/ai_vs_human/ai_generated_text_detection.ipynb`
|
| 65 |
+
- `src/ai_vs_human/hope_train_distributed.py`
|
| 66 |
+
|
| 67 |
+
## Evaluation notes
|
| 68 |
+
|
| 69 |
+
Metrics depend on threshold selection and domain. This repo includes tools for:
|
| 70 |
+
|
| 71 |
+
- internal test evaluation,
|
| 72 |
+
- external/ood evaluation,
|
| 73 |
+
- calibration and threshold selection.
|
| 74 |
+
|
| 75 |
+
See:
|
| 76 |
+
|
| 77 |
+
- `src/ai_vs_human/evaluation_suite.ipynb`
|
| 78 |
+
|
| 79 |
+
## Limitations and biases
|
| 80 |
+
|
| 81 |
+
- Performance can degrade under **domain shift** (new sources or languages).
|
| 82 |
+
- False‑positive rates for **human multilingual text** can be high without careful calibration.
|
| 83 |
+
- These models are **not** a definitive AI‑detection system and should not be used for high‑stakes decisions without additional validation.
|
| 84 |
+
|
| 85 |
+
## License
|
| 86 |
+
|
| 87 |
+
No explicit license is included in this repo. Please add a license if you intend to distribute or reuse the models.
|