README.md

---
datasets:
- puyang2025/seven-phishing-email-datasets
- corbt/enron-emails
- kudzaiprichard/aura-phishing-email-corpus
language: en
library_name: pytorch
license: mit
pipeline_tag: text-classification
tags:
- phishing
- email-security
- text-classification
- from-scratch
- spam-detection
model-index:
- name: phishing-detector-encoder
  results:
  - task:
      type: text-classification
    dataset:
      name: TREC-06 holdout (puyang2025), binary roll-up
      type: puyang2025/seven-phishing-email-datasets
    metrics:
    - type: recall
      value: 0.9741
      name: Recall (unwanted)
    - type: precision
      value: 0.8794
      name: Precision (unwanted)
    - type: f1
      value: 0.9243
      name: F1 (unwanted)
    - type: accuracy
      value: 0.9619
      name: Accuracy
    - type: roc_auc
      value: 0.9946
      name: ROC-AUC
  - task:
      type: text-classification
    dataset:
      name: three-way holdout (Nazario + Nigerian Fraud phishing, SpamAssassin / Enron-Spam
        spam, CEAS-08 / TREC-07 / SpamAssassin ham)
      type: kudzaiprichard/aura-phishing-email-corpus
    metrics:
    - type: f1
      value: 0.9867
      name: Macro-F1 (legitimate / phishing / spam)
    - type: accuracy
      value: 0.9885
      name: Accuracy (3-way)
---

# phishing-detector-encoder: legitimate / phishing / spam email classifier trained from scratch

> **No external pretrained weights were used.** The encoder, the tokenizer and every parameter in `model.safetensors`
> were trained from random initialisation inside this project, in plain PyTorch, without the `transformers`
> library. The only external inputs are the public email corpora listed below.
> Initialisation of this checkpoint: `pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final`.

## What it is

A 337,444,867-parameter BERT-style **pre-LayerNorm Transformer encoder** (24 layers, d_model 1024,
16 heads, FFN 4096, byte-level BPE vocabulary of 32,768, 512 positions) with a
CLS-pooled classification head that maps an email (`subject + "\n\n" + body`) to three classes: `0 = Legitimate`, `1 = Phishing`, `2 = Spam`. The binary verdict shown by the demo is the roll-up **Unwanted (phishing or spam)** = P(phishing) + P(spam) (`config.json["binary_rollup"]`). Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class.
The architecture is defined in `model.py`, which ships with the weights. Weights are stored in **bf16** safetensors
(~675 MB); the header carries the architecture config and the label names as JSON.

## How it was trained

1. **Masked-language-model pretraining from scratch** on the deduplicated Enron email corpus (`corbt/enron-emails`: 517,401 rows -> 226,273 documents after exact/near-duplicate removal and quote stripping), 15% dynamic masking (80/10/10), sequence packing to 512 tokens, AdamW (betas (0.9, 0.98), weight decay 0.01), peak LR 0.00015, bf16 autocast on one RTX 3090 Ti. Run summary (from the pretraining run's `final/config.json`):
   - run_name: large-24L
   - status: complete
   - source_checkpoint: step-0009552
   - steps: 9552
   - total_steps_planned: 9552
   - steps_per_epoch: 796
   - epochs_planned: 12
   - epochs_completed: 12.0
   - tokens_seen: 1291124736
   - total_training_seconds: 54185.9
   - total_training_time: 15h 03m 05s
   - final_loss: 1.523698394948786
   - val_loss: 1.9862932266429436
   - val_masked_acc: 0.6291136684837473
   - peak_vram_reserved_gb: 17.939
   - peak_vram_allocated_gb: 17.494
   - micro_batch: 24
   - grad_accumulation: 11
   - effective_batch: 264
   - precision: bf16_autocast
   - optimizer: AdamW(fused) {"lr": 0.00015, "betas": [0.9, 0.98], "eps": 1e-06, "weight_decay": 0.01, "grad_clip": 1.0, "mlm_probability": 0.15, "mask_replace_probs": [0.8, 0.1, 0.1], "schedule": "cosine_to_10pct", "effective_batch_sequences": 256}
   - schedule: {'kind': 'cosine', 'peak_lr': 0.00015, 'warmup_steps': 573, 'total_steps': 9552, 'floor_ratio': 0.1}
   - mean_tokens_per_s: 23889
   - data: {'n_windows': 210368, 'n_val_windows': 1999, 'tokenizer_sha1': '9f50c5e775089d2ca3efc82e100cd6b3396ed7e7', 'n_docs': 226273, 'limit_docs': None, 'window': '[CLS] + 511 stream tokens'}
   - weights_dtype: float32
   - exported_at: 2026-10-09T03:31:14
2. **Fine-tuning** on `puyang2025/seven-phishing-email-datasets` with our own split: TREC-05, TREC-07 and CEAS-08 (positives trained as *unwanted* only), Enron-Spam and SpamAssassin (spam folders = `Spam`), the Nazario and Nigerian-fraud corpora of `kudzaiprichard/aura-phishing-email-corpus` (= `Phishing`) and synthetic BITS-style lures (= `Phishing`) form train (157,113) / val (17,643), grouped by
   near-duplicate cluster so no template straddles the two; **TREC-06 (16,302 rows, 23.9% positive) is held out
   entirely** as the unseen binary test set and never touched by the tokenizer, pretraining or fine-tuning (1,652 pool rows that
   near-duplicate TREC-06 templates were removed). A second holdout, **test_holdout3**, holds clean-provenance three-way rows never seen by the tokenizer or the pretraining corpus for the 3x3 confusion matrix. Counts come from `data/processed/splits_manifest.json`.
   Rows with a clean three-way label train with 3-class cross-entropy; TREC / CEAS positives train on the probability mass of {phishing, spam} (partial-label loss: -log(P(phishing) + P(spam))), so they shape the legitimate-vs-unwanted decision without asserting a kind they were never labelled with.
   Fine-tune dials (from the checkpoint's config.json):
   - run_name: ft-focal
   - init: pretrained:checkpoints/pretrain/large-24L/final
   - n_layers: 24
   - freeze_layers: 0
   - loss: focal
   - focal_gamma: 2.0
   - lr: 3e-05
   - batch_size: 32
   - micro_batch: 16
   - epochs: 2
   - max_steps: None
   - eval_every: 1250
   - log_every: 20
   - eval_batch_size: 64
   - warmup_fraction: 0.06
   - weight_decay: 0.01
   - betas: [0.9, 0.98]
   - eps: 1e-06
   - grad_clip: 1.0
   - patience: 2
   - schedule: linear_decay
   - selection_metric: val_f1_unwanted
   - pooling: cls
   - num_labels: 3
   - synthetic: off
   - limit_rows: None
   - seed: 42
   - cpu: False

## Metrics

### TREC-06 binary holdout (Unwanted (phishing or spam))

Evaluated by `src/evaluate.py` on the **test** split (`test_holdout.parquet`, 16,302 rows, threshold 0.5). Positive class = **Unwanted (phishing or spam)**, P(unwanted) = P(phishing) + P(spam); these definitions are identical to the binary-era runs, so the numbers are comparable.

| metric | value |
|---|---:|
| **Recall (unwanted)** | **0.9741** (101 missed of 3,894) |
| Precision (unwanted) | 0.8794 |
| F1 (unwanted) | 0.9243 |
| Accuracy | 0.9619 |
| Specificity | 0.9581 |
| ROC-AUC | 0.9946 |
| PR-AUC | 0.9848 |
| Recall at 1% FPR | 0.8955 |
| Recall at 0.1% FPR | 0.6497 |
| Credential-lure slice recall / precision (n=398) | 0.9959 / 0.9758 |
| Latency, batch 1, GPU bf16 (avg / p50) | 10.08 ms / 8.9 ms |
| Latency, batch 1, CPU fp32, 16 threads (avg / p50) | 458.57 ms / 445.01 ms |
| Instructor benchmark | 74.0 ms per sample |

Recall on the unwanted class is the headline number: in a defensive triage aid a missed phish is the costly error, while a false positive costs a human one glance.

TREC-06 carries no phishing-vs-spam ground truth, so the three-way head can only be *described* here: of the 3,793 true positives it flagged, the model called 974 phishing and 2,819 spam (informational, not a metric).

### Three-way holdout (legitimate / phishing / spam)

Evaluated by `src/evaluate.py --split test3` on `test_holdout3.parquet` (3,220 rows with a full three-way label, threshold 0.5). Decision rule `rollup_then_argmax`: a row is unwanted iff P(phishing) + P(spam) >= threshold, then the kind is argmax(P(phishing), P(spam)), so this table collapses exactly onto the binary roll-up.

| metric | value |
|---|---:|
| **Macro-F1 (3 classes)** | **0.9867** |
| Accuracy | 0.9885 |
| Legitimate: precision / recall / F1 (n=1,999) | 0.9900 / 0.9935 / 0.9918 |
| Phishing: precision / recall / F1 (n=521) | 0.9961 / 0.9866 / 0.9913 |
| Spam: precision / recall / F1 (n=700) | 0.9785 / 0.9757 / 0.9771 |
| Phishing-vs-spam accuracy on the 1,221 true unwanted rows (kind decision alone) | 0.9959 (4 phishing read as spam, 1 spam read as phishing) |
| Binary roll-up on this split: recall / precision / F1 | 0.9836 / 0.9893 / 0.9864 |

Confusion matrix (rows = true label by corpus provenance, columns = predicted):

| true \ predicted | Legitimate | Phishing | Spam |
|---|---:|---:|---:|
| **Legitimate** | 1,986 | 1 | 12 |
| **Phishing** | 4 | 514 | 3 |
| **Spam** | 16 | 1 | 683 |

| source | rows | Legitimate | Phishing | Spam | accuracy |
|---|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9534 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9905 |
| Enron | 628 | 0 | 0 | 628 | 0.9841 |
| Nazario | 222 | 0 | 222 | 0 | 0.9865 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9866 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9982 |

**Label provenance.** Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class. The three-way holdout is drawn from text the tokenizer never saw and the pretraining corpus does not contain (Nazario / Nigerian-fraud phishing, SpamAssassin / Enron-Spam spam, CEAS-08 / TREC-07 / SpamAssassin legitimate mail from the former validation split). Because the kind labels come from which corpus a message sits in, part of any phishing-vs-spam confusion measures corpus style rather than lure content.

## Intended use

A **defensive triage aid for small-business inboxes**: rank or flag incoming mail for a human to review, surface a
probability and the attention-highlighted tokens that drove it, and shorten the time between a phishing wave
landing and someone noticing. It is a classroom/final-project model and is not a substitute for a mail gateway.

## Limitations

- **Era**: the labelled data is 1998-2008 spam-track mail (TREC, CEAS, SpamAssassin, Enron), plus Nazario phishing (1995-2022) and Nigerian-fraud mail.
  Modern lures (OAuth consent phishing, QR codes, MFA fatigue, SaaS-brand impersonation) are under-represented; expect lower recall on them.
- **Labels follow corpus provenance**: Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class.
  A phishing-vs-spam confusion therefore partly measures corpus style, and the credential-lure slice remains the closest proxy for phishing-proper recall.
- **Length confound**: legitimate emails in these corpora average ~2x the tokens of positives. Inputs are truncated to
  512 tokens, and the error analysis checks whether errors follow length, but a short legitimate note can
  still look suspicious.
- **English only**; byte-level BPE will tokenize other scripts but the model has not seen them.
- **Headers are not used** (TREC rewrote them); the model sees subject and body text only. Sender reputation,
  SPF/DKIM and URL reputation are out of scope and should be combined with this score in any real deployment.
- Attention highlights are a signal of what the model attended to, not an explanation of why it decided.

## How to load

`model.py` from this repository is the only code dependency (plain PyTorch).

```python
import json, torch
from safetensors.torch import load_file
from tokenizers import Tokenizer
from model import EncoderConfig, EncoderForClassification   # model.py, shipped with the weights

cfg = json.load(open("config.json"))
model = EncoderForClassification(EncoderConfig.from_dict(cfg["model"]))   # num_labels = 3
model.load_state_dict(load_file("model.safetensors"), strict=True)   # bf16 weights cast into the fp32 module
model.eval()

tok = Tokenizer.from_file("tokenizer.json")
tok.enable_truncation(max_length=cfg["max_seq_len"])
enc = tok.encode("Your account has been suspended\n\nClick here to verify your identity within 24 hours.")
ids = torch.tensor([enc.ids])
mask = torch.ones_like(ids, dtype=torch.bool)
with torch.inference_mode():
    probs = model(ids, mask).float().softmax(-1)[0]
p_unwanted = sum(float(probs[i]) for i in cfg["binary_rollup"]["unwanted"])   # P(phishing) + P(spam)
print("unwanted" if p_unwanted >= 0.5 else "legitimate", {cfg["labels"][str(i)]: round(float(p), 4) for i, p in enumerate(probs)})
```

On a GPU wrap the forward in `torch.autocast("cuda", dtype=torch.bfloat16)`. Measured single-email latency (batch 1, tokenize + forward + softmax, see the metrics table): GPU bf16 10.08 ms avg on NVIDIA GeForce RTX 3090 Ti, CPU fp32 (16 threads) 458.57 ms avg.

## Files

- `model.safetensors` — bf16 weights, metadata `format=pt, dtype=bf16, architecture=EncoderForClassification, config=<json>, labels=<json>`
- `config.json` — model config, num_labels, labels, binary_rollup, label provenance, max_seq_len, export dtype, source checkpoint, fine-tune init, binary metrics (TREC-06), three-way metrics (test_holdout3), pretraining summary (null for scratch init)
- `tokenizer.json` — byte-level BPE, NFKC, case preserved, specials `[PAD]=0 [UNK]=1 [CLS]=2 [SEP]=3 [MASK]=4`
- `README.md` — this card

## Provenance

Built for the Skillspire AI & Machine Learning course final project (BITS, 2026). Data decisions, measured VRAM /
throughput / latency and every challenge hit along the way are recorded in the project's RUN_LOG.md.