|
|
--- |
|
|
language: |
|
|
- en |
|
|
- es |
|
|
pipeline_tag: text-classification |
|
|
tags: |
|
|
- ai-generated-text-detection |
|
|
- text-classification |
|
|
- ensemble |
|
|
- deberta |
|
|
- lightgbm |
|
|
- tf-idf |
|
|
- pytorch |
|
|
- scikit-learn |
|
|
--- |
|
|
|
|
|
# EC_MODELS (AI vs Human Detection) |
|
|
|
|
|
This repository contains **model artifacts** for AI‑generated text detection experiments, including a HOPE classifier and a stacked ensemble (DeBERTa + LightGBM + TF‑IDF/SGD). It is **not a single model**, but a collection of checkpoints and meta‑models produced during experiments. |
|
|
|
|
|
## Model inventory |
|
|
|
|
|
Artifacts are stored in `src/ai_vs_human/models/`: |
|
|
|
|
|
- `hope/` |
|
|
HOPE checkpoints (`fold_*/best_model.pt`) for a transformer‑based classifier with memory modules. |
|
|
- `deberta_v3_base/` |
|
|
DeBERTa base checkpoints used in the ensemble. |
|
|
- `lightgbm/`, `lightgbm_numeric/` |
|
|
LightGBM models on engineered features. |
|
|
- `tfidf_sgd/` |
|
|
TF‑IDF + SGD models. |
|
|
- `stack_meta/meta_learner.joblib` |
|
|
Logistic‑regression meta‑learner used for stacking. |
|
|
|
|
|
Out‑of‑fold predictions used to compute metrics and train the stacker are in: |
|
|
|
|
|
- `src/ai_vs_human/oof/` (e.g., `oof_stack.csv`, `oof_deberta.csv`, `oof_lgb.csv`, `oof_sgd.csv`) |
|
|
|
|
|
## Intended use |
|
|
|
|
|
These models are intended for **research and evaluation** of AI‑generated text detection. They can be used to: |
|
|
|
|
|
- compare HOPE vs. ensemble baselines, |
|
|
- reproduce experiments from the notebooks, |
|
|
- evaluate domain‑shift and multilingual robustness. |
|
|
|
|
|
## How to evaluate (metrics only) |
|
|
|
|
|
Use the metrics‑only notebook to compute standard metrics without retraining: |
|
|
|
|
|
- `src/ai_vs_human/metrics_only.ipynb` |
|
|
|
|
|
This notebook loads `oof_stack.csv` and prints AUC‑ROC, PR‑AUC, Accuracy, Precision, Recall, F1, Brier, and ECE, plus a best‑F1 threshold. |
|
|
|
|
|
## Training data |
|
|
|
|
|
Training data and features were produced from the project’s datasets under `src/ai_vs_human/`, including: |
|
|
|
|
|
- `merged_ai_human_multisocial_features*.csv` |
|
|
|
|
|
See the dataset‑building and training notebooks for details: |
|
|
|
|
|
- `src/ai_vs_human/ai_generated_text_detection.ipynb` |
|
|
- `src/ai_vs_human/hope_train_distributed.py` |
|
|
|
|
|
## Evaluation notes |
|
|
|
|
|
Metrics depend on threshold selection and domain. This repo includes tools for: |
|
|
|
|
|
- internal test evaluation, |
|
|
- external/ood evaluation, |
|
|
- calibration and threshold selection. |
|
|
|
|
|
See: |
|
|
|
|
|
- `src/ai_vs_human/evaluation_suite.ipynb` |
|
|
|
|
|
## Limitations and biases |
|
|
|
|
|
- Performance can degrade under **domain shift** (new sources or languages). |
|
|
- False‑positive rates for **human multilingual text** can be high without careful calibration. |
|
|
- These models are **not** a definitive AI‑detection system and should not be used for high‑stakes decisions without additional validation. |
|
|
|
|
|
## License |
|
|
|
|
|
No explicit license is included in this repo. Please add a license if you intend to distribute or reuse the models. |
|
|
|
