File size: 2,824 Bytes
3931802 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 |
---
language:
- en
- es
pipeline_tag: text-classification
tags:
- ai-generated-text-detection
- text-classification
- ensemble
- deberta
- lightgbm
- tf-idf
- pytorch
- scikit-learn
---
# EC_MODELS (AI vs Human Detection)
This repository contains **model artifacts** for AI‑generated text detection experiments, including a HOPE classifier and a stacked ensemble (DeBERTa + LightGBM + TF‑IDF/SGD). It is **not a single model**, but a collection of checkpoints and meta‑models produced during experiments.
## Model inventory
Artifacts are stored in `src/ai_vs_human/models/`:
- `hope/`
HOPE checkpoints (`fold_*/best_model.pt`) for a transformer‑based classifier with memory modules.
- `deberta_v3_base/`
DeBERTa base checkpoints used in the ensemble.
- `lightgbm/`, `lightgbm_numeric/`
LightGBM models on engineered features.
- `tfidf_sgd/`
TF‑IDF + SGD models.
- `stack_meta/meta_learner.joblib`
Logistic‑regression meta‑learner used for stacking.
Out‑of‑fold predictions used to compute metrics and train the stacker are in:
- `src/ai_vs_human/oof/` (e.g., `oof_stack.csv`, `oof_deberta.csv`, `oof_lgb.csv`, `oof_sgd.csv`)
## Intended use
These models are intended for **research and evaluation** of AI‑generated text detection. They can be used to:
- compare HOPE vs. ensemble baselines,
- reproduce experiments from the notebooks,
- evaluate domain‑shift and multilingual robustness.
## How to evaluate (metrics only)
Use the metrics‑only notebook to compute standard metrics without retraining:
- `src/ai_vs_human/metrics_only.ipynb`
This notebook loads `oof_stack.csv` and prints AUC‑ROC, PR‑AUC, Accuracy, Precision, Recall, F1, Brier, and ECE, plus a best‑F1 threshold.
## Training data
Training data and features were produced from the project’s datasets under `src/ai_vs_human/`, including:
- `merged_ai_human_multisocial_features*.csv`
See the dataset‑building and training notebooks for details:
- `src/ai_vs_human/ai_generated_text_detection.ipynb`
- `src/ai_vs_human/hope_train_distributed.py`
## Evaluation notes
Metrics depend on threshold selection and domain. This repo includes tools for:
- internal test evaluation,
- external/ood evaluation,
- calibration and threshold selection.
See:
- `src/ai_vs_human/evaluation_suite.ipynb`
## Limitations and biases
- Performance can degrade under **domain shift** (new sources or languages).
- False‑positive rates for **human multilingual text** can be high without careful calibration.
- These models are **not** a definitive AI‑detection system and should not be used for high‑stakes decisions without additional validation.
## License
No explicit license is included in this repo. Please add a license if you intend to distribute or reuse the models.
|