RoBERTa fine-tuned on LIAR + evaluation, charts, model card

README.md

@@ -0,0 +1,237 @@
+---
+language: en
+license: mit
+library_name: transformers
+pipeline_tag: text-classification
+base_model: roberta-base
+base_model_relation: finetune
+tags:
+  - text-classification
+  - fake-news-detection
+  - misinformation
+  - fact-checking
+  - roberta
+  - liar
+datasets:
+  - liar
+metrics:
+  - accuracy
+  - f1
+model-index:
+  - name: liar-roberta
+    results:
+      - task:
+          type: text-classification
+          name: Fake news detection
+        dataset:
+          name: LIAR
+          type: liar
+          split: test
+        metrics:
+          - type: accuracy
+            value: 0.6446
+          - type: f1
+            name: Macro F1
+            value: 0.6351
+widget:
+  - text: "The moon landing was filmed in a Hollywood studio."
+  - text: "The unemployment rate fell to 3.8 percent last quarter."
+---
+
+# liar-roberta
+
+**A binary classifier that labels a short political statement as REAL or FAKE.**
+Full fine-tune of [`roberta-base`](https://huggingface.co/roberta-base) on
+[LIAR](https://huggingface.co/datasets/liar), the PolitiFact fact-check corpus.
+
+On the LIAR test split it reaches **64.5% accuracy and 0.6351 macro-F1**, clearly ahead of
+a TF-IDF + logistic regression baseline trained on the same data (0.6025 macro-F1).
+Every number on this page comes from the single run that produced these weights; the raw
+values are in [`results.json`](results.json).
+
+```python
+from transformers import pipeline
+clf = pipeline("text-classification", model="emonHasan01/liar-roberta")
+clf("The moon landing was filmed in a Hollywood studio.")
+```
+
+## Label scheme
+
+LIAR ships six truthfulness classes, collapsed here into two:
+
+| Binary label | LIAR classes |
+|---|---|
+| `REAL` (0) | true, mostly-true |
+| `FAKE` (1) | false, half-true, barely-true, pants-fire |
+
+Worth flagging: LIAR's integer label order is **not** sorted by truthfulness
+(`4 = barely-true` and `5 = pants-fire` are the two least truthful, not the most).
+A fair amount of published code treats them as the truthful end and inverts the whole
+task. This model resolves the mapping from the dataset's label *names*, never indices.
+
+The collapse leaves the data imbalanced — 460 REAL against 823 FAKE in the
+test split — so training used class-weighted cross-entropy and the checkpoint was chosen
+on macro-F1 rather than accuracy.
+
+## Results
+
+![Model versus baselines](assets/baselines.png)
+
+| | Accuracy | Macro F1 | F1 (FAKE) | Precision (FAKE) | Recall (FAKE) | ROC AUC |
+|---|---|---|---|---|---|---|
+| Majority class | 0.6415 | 0.3908 | — | — | — | — |
+| TF-IDF + LogReg | 0.6220 | 0.6025 | 0.6905 | 0.7272 | 0.6574 | 0.6496 |
+| **RoBERTa (this model)** | **0.6446** | **0.6351** | **0.6940** | **0.7751** | **0.6282** | **0.6959** |
+
+The majority-class row is there on purpose. Accuracy alone on this dataset says very
+little, since always answering FAKE already scores 0.6415.
+
+![Confusion matrix](assets/confusion-matrix.png)
+
+## Where it succeeds and fails
+
+Accuracy across the six original LIAR classes, before the binary collapse:
+
+![Per-class accuracy](assets/per-class.png)
+
+| LIAR class | Mapped to | n | Accuracy | Mean P(FAKE) |
+|---|---|---|---|---|
+| false | FAKE | 250 | 0.652 | 0.586 |
+| half-true | FAKE | 267 | 0.532 | 0.534 |
+| mostly-true | REAL | 249 | 0.663 | 0.456 |
+| true | REAL | 211 | 0.687 | 0.449 |
+| barely-true | FAKE | 214 | 0.668 | 0.594 |
+| pants-fire | FAKE | 92 | 0.750 | 0.639 |
+
+The ends of the scale are separable; the middle is not. `half-true` and `mostly-true`
+differ by an editorial judgement that is largely unrecoverable from 18 words of text,
+and the model's scores reflect that.
+
+### Most confident mistakes
+
+| LIAR class | True | Predicted | Confidence | Statement |
+|---|---|---|---|---|
+| true | REAL | FAKE | 0.90 | "Obama has more czars than the Romanovs." |
+| true | REAL | FAKE | 0.88 | The words subhuman mongrel, which Ted Nugent called President Barack Obama, were used by the Na |
+| mostly-true | REAL | FAKE | 0.88 | Obama-Nelson economic record: Job creation ... at slowest post-recession rate since Great Depre |
+
+Of 456 errors, 150 are real statements flagged as fake and 306 are fake
+statements let through.
+
+## Choosing a threshold
+
+P(FAKE) is exposed, so the 0.5 cutoff is a starting point rather than a verdict:
+
+![Precision/recall trade-off](assets/threshold.png)
+
+| Threshold | Precision (FAKE) | Recall (FAKE) | F1 | Share flagged |
+|---|---|---|---|---|
+| 0.20 | 0.642 | 1.000 | 0.782 | 99.9% |
+| 0.30 | 0.669 | 0.939 | 0.782 | 90.0% |
+| 0.40 | 0.733 | 0.785 | 0.758 | 68.7% |
+| 0.50 | 0.775 | 0.628 | 0.694 | 52.0% |
+| 0.60 | 0.796 | 0.485 | 0.603 | 39.0% |
+| 0.70 | 0.835 | 0.301 | 0.443 | 23.1% |
+| 0.80 | 0.884 | 0.139 | 0.239 | 10.1% |
+
+At threshold 0.65 precision reaches 0.811, with recall falling to 0.385.
+
+## Is the confidence meaningful?
+
+![Reliability diagram](assets/calibration.png)
+
+Expected calibration error is **0.042** over 10 bins. Points below the diagonal mean
+overconfidence: when the model says 0.9 it is right less often than 90% of the time.
+Treat the score as a ranking signal, not as a probability of being lied to.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+tok = AutoTokenizer.from_pretrained("emonHasan01/liar-roberta")
+model = AutoModelForSequenceClassification.from_pretrained("emonHasan01/liar-roberta")
+
+inputs = tok("Your statement here", return_tensors="pt", truncation=True, max_length=128)
+with torch.no_grad():
+    probs = model(**inputs).logits.softmax(-1)[0]
+print("P(REAL)", probs[0].item(), "P(FAKE)", probs[1].item())
+```
+
+### Longer documents
+
+The model saw statements of about 22 tokens and has a 128-token window. Feeding
+a whole article truncates it. Chunk and average instead:
+
+```python
+def classify_long(text, tok, model, chunk_words=60):
+    w = text.split()
+    chunks = [" ".join(w[i:i+chunk_words]) for i in range(0, len(w), chunk_words)] or [text]
+    enc = tok(chunks[:32], return_tensors="pt", truncation=True, padding=True, max_length=128)
+    with torch.no_grad():
+        p = model(**enc).logits.softmax(-1).mean(0)
+    return {"REAL": p[0].item(), "FAKE": p[1].item()}
+```
+
+Results on full articles are out of distribution either way — see Limitations.
+
+## Training
+
+| | |
+|---|---|
+| Base model | `roberta-base`, 125M parameters, all trained |
+| Objective | Class-weighted cross-entropy (weights 1.41, 0.78) |
+| Max length | 128 tokens |
+| Learning rate | 2e-5, AdamW, 10% warmup |
+| Batch size | 16 |
+| Epochs | 3, best checkpoint by validation macro-F1 |
+| Weight decay | 0.01 |
+| Seed | 42 |
+| Hardware | Tesla T4 |
+| Training time | ~5 minutes |
+
+Statement length in the training set: median 22 tokens, 95th percentile
+40, so a 128-token window truncates almost nothing.
+
+## Limitations
+
+Please read this section before using the model for anything that matters.
+
+- **It does not check facts.** There is no retrieval and no knowledge base. The model
+  learns lexical and stylistic patterns that correlate with a PolitiFact verdict. A false
+  claim written in the register of a true one gets through, which the confident-mistakes
+  table above shows directly.
+- **The domain is narrow.** Short US political statements, roughly 2007–2016, selected
+  for fact-checking by PolitiFact. It has not learned to judge full news articles,
+  scientific claims, health advice, or non-US or non-political content, and accuracy
+  there is unmeasured and probably much worse.
+- **The binary collapse discards information.** Grouping `half-true` with `pants-fire`
+  throws away most of the distinction the annotators recorded.
+- **It inherits PolitiFact's selection.** Which speakers and claims get fact-checked at
+  all is an editorial decision, absorbed here along with the labels.
+- **It is overconfident** (ECE 0.042). Do not read the softmax score as a probability.
+- **Temporal drift.** Names, events, and political vocabulary have moved on since the
+  corpus was collected.
+- **The ceiling is low, and that is expected.** LIAR is a hard benchmark and published
+  binary results sit broadly in this range. A very high accuracy on LIAR usually means a
+  label-mapping bug or a leak rather than a better model.
+
+Do not use this as an automated arbiter of truth, for moderation at scale, or in any
+decision affecting a person without a human reading the output.
+
+## Citation
+
+```bibtex
+@inproceedings{wang2017liar,
+  title     = {"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection},
+  author    = {Wang, William Yang},
+  booktitle = {Proceedings of ACL},
+  year      = {2017}
+}
+```
+
+## Reproducing
+
+Everything on this page — training, both baselines, all five charts and this card — comes
+from one Colab notebook run on a Tesla T4. No metric was typed in by hand.

config.json

@@ -0,0 +1,36 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "RobertaForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "REAL",
+    "1": "FAKE"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "label2id": {
+    "FAKE": 1,
+    "REAL": 0
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.19.0",
+  "type_vocab_size": 1,
+  "use_cache": false,
+  "vocab_size": 50265
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba129a3b629e9ba1f56912e728b00eda36e164ccde6d5275b4c39b805aaeef47
+size 498612824

results.json

@@ -0,0 +1,257 @@
+{
+  "model": {
+    "accuracy": 0.6445830085736555,
+    "macro_f1": 0.6350839549911429,
+    "f1_fake": 0.6939597315436241,
+    "precision_fake": 0.775112443778111,
+    "recall_fake": 0.6281895504252734,
+    "roc_auc": 0.6959308468487506
+  },
+  "baseline_majority": {
+    "accuracy": 0.6414653156664069,
+    "macro_f1": 0.39078822412155745,
+    "f1_fake": 0.7815764482431149,
+    "precision_fake": 0.6414653156664069,
+    "recall_fake": 1.0
+  },
+  "baseline_tfidf_logreg": {
+    "accuracy": 0.6219797349961029,
+    "macro_f1": 0.6025029496631283,
+    "f1_fake": 0.6904913848117422,
+    "precision_fake": 0.7271505376344086,
+    "recall_fake": 0.6573511543134872,
+    "roc_auc": 0.6495694437107086
+  },
+  "confusion_matrix": [
+    [
+      310,
+      150
+    ],
+    [
+      306,
+      517
+    ]
+  ],
+  "ece": 0.04245237071162911,
+  "per_class": [
+    {
+      "label_name": "false",
+      "n": 250,
+      "accuracy": 0.652,
+      "mean_p_fake": 0.5863052606582642,
+      "binary": "FAKE"
+    },
+    {
+      "label_name": "half-true",
+      "n": 267,
+      "accuracy": 0.5318352059925093,
+      "mean_p_fake": 0.5340710878372192,
+      "binary": "FAKE"
+    },
+    {
+      "label_name": "mostly-true",
+      "n": 249,
+      "accuracy": 0.6626506024096386,
+      "mean_p_fake": 0.45594820380210876,
+      "binary": "REAL"
+    },
+    {
+      "label_name": "true",
+      "n": 211,
+      "accuracy": 0.6872037914691943,
+      "mean_p_fake": 0.44890156388282776,
+      "binary": "REAL"
+    },
+    {
+      "label_name": "barely-true",
+      "n": 214,
+      "accuracy": 0.6682242990654206,
+      "mean_p_fake": 0.5944609642028809,
+      "binary": "FAKE"
+    },
+    {
+      "label_name": "pants-fire",
+      "n": 92,
+      "accuracy": 0.75,
+      "mean_p_fake": 0.6390070915222168,
+      "binary": "FAKE"
+    }
+  ],
+  "thresholds": [
+    {
+      "threshold": 0.05,
+      "precision": 0.6414653156664069,
+      "recall": 1.0,
+      "f1": 0.7815764482431149,
+      "accuracy": 0.6414653156664069,
+      "flagged_rate": 1.0
+    },
+    {
+      "threshold": 0.1,
+      "precision": 0.6414653156664069,
+      "recall": 1.0,
+      "f1": 0.7815764482431149,
+      "accuracy": 0.6414653156664069,
+      "flagged_rate": 1.0
+    },
+    {
+      "threshold": 0.15,
+      "precision": 0.6414653156664069,
+      "recall": 1.0,
+      "f1": 0.7815764482431149,
+      "accuracy": 0.6414653156664069,
+      "flagged_rate": 1.0
+    },
+    {
+      "threshold": 0.2,
+      "precision": 0.641965678627145,
+      "recall": 1.0,
+      "f1": 0.7819477434679335,
+      "accuracy": 0.642244738893219,
+      "flagged_rate": 0.9992205767731879
+    },
+    {
+      "threshold": 0.25,
+      "precision": 0.649164677804296,
+      "recall": 0.991494532199271,
+      "f1": 0.7846153846153846,
+      "accuracy": 0.6508183943881528,
+      "flagged_rate": 0.9797349961028838
+    },
+    {
+      "threshold": 0.3,
+      "precision": 0.6692640692640692,
+      "recall": 0.9392466585662211,
+      "f1": 0.78159757330637,
+      "accuracy": 0.6632891660171473,
+      "flagged_rate": 0.9002338269680437
+    },
+    {
+      "threshold": 0.35,
+      "precision": 0.7057057057057057,
+      "recall": 0.8566221142162819,
+      "f1": 0.7738748627881449,
+      "accuracy": 0.6788776305533905,
+      "flagged_rate": 0.7786438035853468
+    },
+    {
+      "threshold": 0.4,
+      "precision": 0.7332576617480137,
+      "recall": 0.7849331713244229,
+      "f1": 0.7582159624413145,
+      "accuracy": 0.6788776305533905,
+      "flagged_rate": 0.686671862821512
+    },
+    {
+      "threshold": 0.45,
+      "precision": 0.7464239271781534,
+      "recall": 0.6974483596597812,
+      "f1": 0.7211055276381909,
+      "accuracy": 0.6539360872954014,
+      "flagged_rate": 0.5993764614185503
+    },
+    {
+      "threshold": 0.5,
+      "precision": 0.775112443778111,
+      "recall": 0.6281895504252734,
+      "f1": 0.6939597315436241,
+      "accuracy": 0.6445830085736555,
+      "flagged_rate": 0.5198752922837101
+    },
+    {
+      "threshold": 0.55,
+      "precision": 0.7837370242214533,
+      "recall": 0.5504252733900364,
+      "f1": 0.6466809421841542,
+      "accuracy": 0.6141855027279813,
+      "flagged_rate": 0.4505066250974279
+    },
+    {
+      "threshold": 0.6,
+      "precision": 0.7964071856287425,
+      "recall": 0.4848116646415553,
+      "f1": 0.6027190332326284,
+      "accuracy": 0.5900233826968043,
+      "flagged_rate": 0.39049103663289164
+    },
+    {
+      "threshold": 0.65,
+      "precision": 0.8107416879795396,
+      "recall": 0.38517618469015796,
+      "f1": 0.5222405271828665,
+      "accuracy": 0.5479345284489477,
+      "flagged_rate": 0.3047544816835542
+    },
+    {
+      "threshold": 0.7,
+      "precision": 0.835016835016835,
+      "recall": 0.30133657351154314,
+      "f1": 0.44285714285714284,
+      "accuracy": 0.5136399064692128,
+      "flagged_rate": 0.23148869836321123
+    },
+    {
+      "threshold": 0.75,
+      "precision": 0.8564356435643564,
+      "recall": 0.21020656136087484,
+      "f1": 0.3375609756097561,
+      "accuracy": 0.47077162899454406,
+      "flagged_rate": 0.15744349181605613
+    },
+    {
+      "threshold": 0.8,
+      "precision": 0.8837209302325582,
+      "recall": 0.1385176184690158,
+      "f1": 0.23949579831932774,
+      "accuracy": 0.4356975837879969,
+      "flagged_rate": 0.10054559625876851
+    },
+    {
+      "threshold": 0.85,
+      "precision": 0.873015873015873,
+      "recall": 0.06682867557715674,
+      "f1": 0.12415349887133183,
+      "accuracy": 0.3951675759937646,
+      "flagged_rate": 0.049103663289166016
+    },
+    {
+      "threshold": 0.9,
+      "precision": 1.0,
+      "recall": 0.014580801944106925,
+      "f1": 0.02874251497005988,
+      "accuracy": 0.36788776305533905,
+      "flagged_rate": 0.009353078721745909
+    },
+    {
+      "threshold": 0.95,
+      "precision": 0.0,
+      "recall": 0.0,
+      "f1": 0.0,
+      "accuracy": 0.35853468433359315,
+      "flagged_rate": 0.0
+    }
+  ],
+  "label_map": {
+    "false": 1,
+    "half-true": 1,
+    "mostly-true": 0,
+    "true": 0,
+    "barely-true": 1,
+    "pants-fire": 1
+  },
+  "config": {
+    "base_model": "roberta-base",
+    "max_len": 128,
+    "epochs": 3,
+    "lr": 2e-05,
+    "batch_size": 16,
+    "seed": 42,
+    "class_weights": [
+      1.4071,
+      0.7756
+    ],
+    "train_minutes": 5.5,
+    "hardware": "Tesla T4",
+    "data_source": "hub:ucsbnlp/liar"
+  }
+}

tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "errors": "replace",
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "RobertaTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}