README.md

---
language:
  - en
license: mit
tags:
  - text-classification
  - deception-detection
  - fake-news
  - spam-detection
  - modernbert
  - pytorch
  - multi-task-learning
  - cross-domain
base_model: answerdotai/ModernBERT-base
datasets:
  - difraud/difraud
metrics:
  - f1
  - roc_auc
pipeline_tag: text-classification
---

# Verite!: Cross-Domain Deception Detection

**Verite!** is a cross-domain deception detection classifier built on [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base). It detects deceptive content (fake news, phishing, spam, job scams, political misinformation, product review fraud, and Twitter rumours) with a single unified model.

> Code and training script: [github.com/Daxlia/Verite](https://github.com/Daxlia/Verite)

Evaluated on the [DIFrauD](https://huggingface.co/datasets/difraud/difraud) benchmark — 7 domains, ~103K samples.

---

## Results

![System comparison](comparison_chart.png)

Evaluated on the DIFrauD test set (macro-F1 and AUC-ROC, higher is better).

| System | Macro-F1 | AUC-ROC |
| ------ | -------- | ------- |
| Majority class | 0.3792 | 0.5000 |
| TF-IDF + LR | 0.8094 | 0.9079 |
| **Verite! (ours)** | **0.8512** | **0.9487** |
| SOTA (DIFrauD leaderboard) | 0.904 | — |

> Single seed (seed=42) on 2×NVIDIA T4. Multi-seed ensemble (3 seeds) expected to improve further.

### Per-Domain Breakdown

![Per-domain performance](per_domain_chart.png)

| Domain | Macro-F1 | AUC-ROC | Accuracy |
| ------ | -------- | ------- | -------- |
| Fake News | 0.9855 | 0.9990 | 0.9858 |
| Job Scams | 0.7584 | 0.9354 | 0.9636 |
| Phishing | 0.9708 | 0.9964 | 0.9719 |
| Political Statements | 0.4562 | 0.6418 | 0.6504 |
| Product Reviews | 0.6424 | 0.7653 | 0.6582 |
| SMS Spam | 0.9834 | 0.9998 | 0.9894 |
| Twitter Rumours | 0.7856 | 0.8800 | 0.7945 |

### Ablation Study

![Ablation study](ablation_chart.png)

| Variant | Macro-F1 | AUC-ROC |
| ------- | -------- | ------- |
| Verite! (full) | 0.8512 | 0.9487 |
| w/o Linguistic Features | 0.8544 | 0.9475 |
| TTA | 0.8515 | 0.9456 |

---

## Model Description

Verite! extends ModernBERT-base with four feature streams fused before classification:

| Component | Output | Role |
| --------- | ------ | ---- |
| **AttentionPooling + semantic_proj** | 512-d | Weighted token aggregation → projected semantic embedding |
| **ling_proj** | 512-d | 8 hand-crafted linguistic features (caps ratio, punctuation, TTR, hedge words, negation…) projected to hidden dim |
| **LocalConsistencyModule** | 256-d | Splits sequence into 4 segments; cross-attention detects internal contradictions |
| **SpectralFeatures** | 10-d | Top-8 FFT magnitudes + spectral centroid + entropy over the token sequence |

All four streams are concatenated (512+512+256+10 = **1290-d**), projected through `LayerNorm → Linear(512) → GELU`, then through 5× multi-sample dropout into a **HypersphericalHead** (prototype classifier on the unit hypersphere, scale=20).

**Architecture diagram:**

```text
Input Text ──────────────────────── Linguistic Features (8-d)
    │                                         │
    ▼                                         ▼
ModernBERT-base (149M params, fp32)     ling_proj → feat_emb (512-d)
    │                                         │
    ├── AttentionPooling ──► semantic_proj ──► sem_emb  (512-d) ──┐
    ├── LocalConsistencyModule (4 segs) ──► cons_emb (256-d) ─────┤
    └── SpectralFeatures (top-8 FFT + centroid + entropy) ► spec (10-d)
                                                                   │
             Concatenate [sem_emb | feat_emb | cons_emb | spec]  (1290-d)
                                         │
                               LayerNorm → Linear(512) → GELU
                                         │
                         Multi-sample Dropout (5×) → HypersphericalHead
                                         │
                                  Logits (2 classes)
```

---

## Training

Trained on 2×NVIDIA T4 (16 GB each) via Kaggle (~13 h, < 14 h total session). The encoder always runs in fp32 to avoid SDPA overflow on sm75 hardware; head layers use fp16 via PyTorch AMP.

**Training objectives:**

- Focal loss (γ=2.0) with class-balanced weights and label smoothing (ε=0.05)
- Supervised Contrastive loss (λ=0.1, τ=0.07) on semantic embeddings
- Domain MTL cross-entropy (λ=0.1) across 7 domain heads

**Regularization stack:**

- AdamW + LLRD (decay=0.9): encoder LR=1e-5, head LR=1e-4
- Cosine schedule with 8% linear warmup
- Gradient accumulation (×4), gradient clipping (0.7)
- EMA (decay=0.995) — used for final checkpoint selection
- Adversarial Weight Perturbation (AWP, ε=0.001) from epoch 4 onward

**Hyperparameters:**

| Parameter | Value |
| --------- | ----- |
| Base encoder | `answerdotai/ModernBERT-base` |
| Max sequence length | 256 |
| Effective batch size | 64 (8 × 2 GPUs × 4 accum steps) |
| Epochs | 5 |
| Encoder LR | 1e-5 |
| Head LR | 1e-4 |
| LLRD decay | 0.9 |
| Warmup | 8% |
| Weight decay | 0.02 |
| Focal γ | 2.0 |
| SupCon λ | 0.1 |
| Domain MTL λ | 0.1 |
| AWP start | Epoch 4 |
| EMA decay | 0.995 |
| Total runtime | ~13 h on 2×T4 (< 14 h) |

---

## Usage

> **Important:** This model uses a custom architecture defined in `VeriteTrainer.py`. You must have that file in your Python path before loading.

### Installation

```bash
pip install torch>=2.1.0 transformers>=4.47.0 safetensors sentencepiece
```

### Inference

```python
import torch
from torch.utils.data import DataLoader
from transformers import AutoTokenizer
from safetensors.torch import load_file

from VeriteTrainer import DeceptionReasoningModel, Config, DeceptionDataset, collate_fn

cfg       = Config()
tokenizer = AutoTokenizer.from_pretrained("Daxlia/verite")

model = DeceptionReasoningModel(cfg)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

texts = ["Congratulations! You've been selected to receive a $1,000 gift card. Click here to claim."]

dataset = DeceptionDataset(texts, [0] * len(texts), tokenizer, cfg)
loader  = DataLoader(dataset, batch_size=8, shuffle=False, collate_fn=collate_fn)

with torch.no_grad():
    for batch in loader:
        out  = model(input_ids=batch["input_ids"],
                     attention_mask=batch["attention_mask"],
                     ling_feats=batch["ling_feats"])
        prob = torch.softmax(out["logits"], dim=-1)[:, 1]
        for t, p in zip(texts, prob.tolist()):
            label = "DECEPTIVE" if p > 0.5 else "GENUINE"
            print(f"[{label}] P(deceptive) = {p:.4f} | {t}")
```

### Output format

| Key | Shape | Description |
| --- | ----- | ----------- |
| `logits` | `(B, 2)` | Raw class logits (genuine=0, deceptive=1) |
| `sem_emb` | `(B, 512)` | Semantic embedding (useful for downstream tasks) |
| `domain_logits` | `(B, 7)` | Domain classification logits |

---

## Domains

The model was trained jointly on all 7 DIFrauD domains:

| Domain | Description |
| ------ | ----------- |
| `fake_news` | News articles with fabricated or misleading content |
| `job_scams` | Fraudulent job postings |
| `phishing` | Phishing emails and URLs |
| `political_statements` | Fact-checked political claims |
| `product_reviews` | Fake/incentivised product reviews |
| `sms` | SMS spam messages |
| `twitter_rumours` | Unverified claims spread on Twitter/X |

---

## Limitations

- Max input length is 256 tokens; longer texts are truncated.
- Inference requires `VeriteTrainer.py` — no `AutoModel`-compatible registration yet.
- Performance is weakest on Political Statements (F1=0.456) and Product Reviews (F1=0.642) — domains with subtle, opinion-heavy deception signals.
- Results reported for a single training seed; variance across seeds has not been fully characterized.

---

## Citation

```bibtex
@misc{verite2026,
  author    = {Daxlia},
  title     = {Verite!: Cross-Domain Deception Detection with ModernBERT},
  year      = {2026},
  doi       = {10.5281/zenodo.20256648},
  url       = {https://doi.org/10.5281/zenodo.20256648}
}
```

---

## AI Disclosure

This project was developed with assistance from AI tools:

- **ChatGPT (OpenAI)** — Bug identification, algorithmic suggestions, and research ideation
- **Claude (Anthropic)** — Code implementation, debugging, and writing of documentation files

All AI-generated content was reviewed, validated, and adapted by the author.

---

## License

This model is released under the [MIT License](https://opensource.org/licenses/MIT).

The base encoder [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) is licensed under Apache 2.0.  
The [DIFrauD dataset](https://huggingface.co/datasets/difraud/difraud) is subject to its own license; refer to the dataset repository.