| --- |
| library_name: transformers |
| pipeline_tag: text-classification |
| base_model: roberta-base |
| tags: |
| - text-classification |
| - media-bias |
| - roberta |
| datasets: |
| - mediabiasgroup/BABE |
| language: |
| - en |
| --- |
| |
| # roberta-babe-baseline |
|
|
| Best-fold checkpoint from a 5-fold RoBERTa-base reproduction of BABE sentence-level media bias classification. |
|
|
| - Training code: [https://github.com/vulonviing/babe-roberta-baseline](https://github.com/vulonviing/babe-roberta-baseline) |
| - Source dataset: [https://huggingface.co/datasets/mediabiasgroup/BABE](https://huggingface.co/datasets/mediabiasgroup/BABE) |
| - Released checkpoint: `models/fold_0/checkpoint-532` |
| - Selected checkpoint: `fold_0` with macro-F1 `0.876` |
| - Summary statement: trained on 80% of BABE, 5-fold CV mean: `0.857 +- 0.012` |
|
|
| ## Model details |
|
|
| | Item | Value | |
| |---|---| |
| | Base model | `roberta-base` | |
| | Task | Sentence-level media bias classification | |
| | Labels | `non-biased`, `biased` | |
| | Max sequence length | `128` | |
| | Epochs | `4` | |
| | Learning rate | `2e-05` | |
| | Batch size | `16` train / `32` eval | |
| | Weight decay | `0.01` | |
| | Warmup ratio | `0.1` | |
| | Random seed | `42` | |
|
|
| ## Cross-validation summary |
|
|
| | Metric | Mean +- Std | |
| |---|---| |
| | Macro-F1 | 0.857 +- 0.012 | |
| | Accuracy | 0.858 +- 0.012 | |
| | Precision (macro) | 0.856 +- 0.011 | |
| | Recall (macro) | 0.859 +- 0.012 | |
| | Biased F1 | 0.869 +- 0.011 | |
|
|
| Per-fold macro-F1 values in the repo: `0.876, 0.854, 0.845, 0.852, 0.856`. |
|
|
| ## Held-out quick-run reference |
|
|
| | Metric | Score | |
| |---|---| |
| | Macro-F1 | 0.870 | |
| | Accuracy | 0.872 | |
| | Precision (macro) | 0.870 | |
| | Recall (macro) | 0.872 | |
| | Biased F1 | 0.884 | |
|
|
| Confusion matrix from the held-out quick run (`n=468`): |
|
|
| | | Pred non-biased | Pred biased | |
| |---|---|---| |
| | True non-biased (207) | 180 | 27 | |
| | True biased (261) | 33 | 228 | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| |
| repo_id = 'vulonviing/roberta-babe-baseline' |
| tokenizer = AutoTokenizer.from_pretrained(repo_id) |
| model = AutoModelForSequenceClassification.from_pretrained(repo_id) |
| ``` |
|
|
