| --- |
| datasets: |
| - puyang2025/seven-phishing-email-datasets |
| - corbt/enron-emails |
| - kudzaiprichard/aura-phishing-email-corpus |
| language: en |
| library_name: pytorch |
| license: mit |
| pipeline_tag: text-classification |
| tags: |
| - phishing |
| - email-security |
| - text-classification |
| - from-scratch |
| - spam-detection |
| model-index: |
| - name: phishing-detector-encoder |
| results: |
| - task: |
| type: text-classification |
| dataset: |
| name: TREC-06 holdout (puyang2025), binary roll-up |
| type: puyang2025/seven-phishing-email-datasets |
| metrics: |
| - type: recall |
| value: 0.9741 |
| name: Recall (unwanted) |
| - type: precision |
| value: 0.8794 |
| name: Precision (unwanted) |
| - type: f1 |
| value: 0.9243 |
| name: F1 (unwanted) |
| - type: accuracy |
| value: 0.9619 |
| name: Accuracy |
| - type: roc_auc |
| value: 0.9946 |
| name: ROC-AUC |
| - task: |
| type: text-classification |
| dataset: |
| name: three-way holdout (Nazario + Nigerian Fraud phishing, SpamAssassin / Enron-Spam |
| spam, CEAS-08 / TREC-07 / SpamAssassin ham) |
| type: kudzaiprichard/aura-phishing-email-corpus |
| metrics: |
| - type: f1 |
| value: 0.9867 |
| name: Macro-F1 (legitimate / phishing / spam) |
| - type: accuracy |
| value: 0.9885 |
| name: Accuracy (3-way) |
| --- |
| |
| # phishing-detector-encoder: legitimate / phishing / spam email classifier trained from scratch |
|
|
| > **No external pretrained weights were used.** The encoder, the tokenizer and every parameter in `model.safetensors` |
| > were trained from random initialisation inside this project, in plain PyTorch, without the `transformers` |
| > library. The only external inputs are the public email corpora listed below. |
| > Initialisation of this checkpoint: `pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final`. |
|
|
| ## What it is |
|
|
| A 337,444,867-parameter BERT-style **pre-LayerNorm Transformer encoder** (24 layers, d_model 1024, |
| 16 heads, FFN 4096, byte-level BPE vocabulary of 32,768, 512 positions) with a |
| CLS-pooled classification head that maps an email (`subject + "\n\n" + body`) to three classes: `0 = Legitimate`, `1 = Phishing`, `2 = Spam`. The binary verdict shown by the demo is the roll-up **Unwanted (phishing or spam)** = P(phishing) + P(spam) (`config.json["binary_rollup"]`). Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class. |
| The architecture is defined in `model.py`, which ships with the weights. Weights are stored in **bf16** safetensors |
| (~675 MB); the header carries the architecture config and the label names as JSON. |
|
|
| ## How it was trained |
|
|
| 1. **Masked-language-model pretraining from scratch** on the deduplicated Enron email corpus (`corbt/enron-emails`: 517,401 rows -> 226,273 documents after exact/near-duplicate removal and quote stripping), 15% dynamic masking (80/10/10), sequence packing to 512 tokens, AdamW (betas (0.9, 0.98), weight decay 0.01), peak LR 0.00015, bf16 autocast on one RTX 3090 Ti. Run summary (from the pretraining run's `final/config.json`): |
| - run_name: large-24L |
| - status: complete |
| - source_checkpoint: step-0009552 |
| - steps: 9552 |
| - total_steps_planned: 9552 |
| - steps_per_epoch: 796 |
| - epochs_planned: 12 |
| - epochs_completed: 12.0 |
| - tokens_seen: 1291124736 |
| - total_training_seconds: 54185.9 |
| - total_training_time: 15h 03m 05s |
| - final_loss: 1.523698394948786 |
| - val_loss: 1.9862932266429436 |
| - val_masked_acc: 0.6291136684837473 |
| - peak_vram_reserved_gb: 17.939 |
| - peak_vram_allocated_gb: 17.494 |
| - micro_batch: 24 |
| - grad_accumulation: 11 |
| - effective_batch: 264 |
| - precision: bf16_autocast |
| - optimizer: AdamW(fused) {"lr": 0.00015, "betas": [0.9, 0.98], "eps": 1e-06, "weight_decay": 0.01, "grad_clip": 1.0, "mlm_probability": 0.15, "mask_replace_probs": [0.8, 0.1, 0.1], "schedule": "cosine_to_10pct", "effective_batch_sequences": 256} |
| - schedule: {'kind': 'cosine', 'peak_lr': 0.00015, 'warmup_steps': 573, 'total_steps': 9552, 'floor_ratio': 0.1} |
| - mean_tokens_per_s: 23889 |
| - data: {'n_windows': 210368, 'n_val_windows': 1999, 'tokenizer_sha1': '9f50c5e775089d2ca3efc82e100cd6b3396ed7e7', 'n_docs': 226273, 'limit_docs': None, 'window': '[CLS] + 511 stream tokens'} |
| - weights_dtype: float32 |
| - exported_at: 2026-10-09T03:31:14 |
| 2. **Fine-tuning** on `puyang2025/seven-phishing-email-datasets` with our own split: TREC-05, TREC-07 and CEAS-08 (positives trained as *unwanted* only), Enron-Spam and SpamAssassin (spam folders = `Spam`), the Nazario and Nigerian-fraud corpora of `kudzaiprichard/aura-phishing-email-corpus` (= `Phishing`) and synthetic BITS-style lures (= `Phishing`) form train (157,113) / val (17,643), grouped by |
| near-duplicate cluster so no template straddles the two; **TREC-06 (16,302 rows, 23.9% positive) is held out |
| entirely** as the unseen binary test set and never touched by the tokenizer, pretraining or fine-tuning (1,652 pool rows that |
| near-duplicate TREC-06 templates were removed). A second holdout, **test_holdout3**, holds clean-provenance three-way rows never seen by the tokenizer or the pretraining corpus for the 3x3 confusion matrix. Counts come from `data/processed/splits_manifest.json`. |
| Rows with a clean three-way label train with 3-class cross-entropy; TREC / CEAS positives train on the probability mass of {phishing, spam} (partial-label loss: -log(P(phishing) + P(spam))), so they shape the legitimate-vs-unwanted decision without asserting a kind they were never labelled with. |
| Fine-tune dials (from the checkpoint's config.json): |
| - run_name: ft-focal |
| - init: pretrained:checkpoints/pretrain/large-24L/final |
| - n_layers: 24 |
| - freeze_layers: 0 |
| - loss: focal |
| - focal_gamma: 2.0 |
| - lr: 3e-05 |
| - batch_size: 32 |
| - micro_batch: 16 |
| - epochs: 2 |
| - max_steps: None |
| - eval_every: 1250 |
| - log_every: 20 |
| - eval_batch_size: 64 |
| - warmup_fraction: 0.06 |
| - weight_decay: 0.01 |
| - betas: [0.9, 0.98] |
| - eps: 1e-06 |
| - grad_clip: 1.0 |
| - patience: 2 |
| - schedule: linear_decay |
| - selection_metric: val_f1_unwanted |
| - pooling: cls |
| - num_labels: 3 |
| - synthetic: off |
| - limit_rows: None |
| - seed: 42 |
| - cpu: False |
|
|
| ## Metrics |
|
|
| ### TREC-06 binary holdout (Unwanted (phishing or spam)) |
|
|
| Evaluated by `src/evaluate.py` on the **test** split (`test_holdout.parquet`, 16,302 rows, threshold 0.5). Positive class = **Unwanted (phishing or spam)**, P(unwanted) = P(phishing) + P(spam); these definitions are identical to the binary-era runs, so the numbers are comparable. |
|
|
| | metric | value | |
| |---|---:| |
| | **Recall (unwanted)** | **0.9741** (101 missed of 3,894) | |
| | Precision (unwanted) | 0.8794 | |
| | F1 (unwanted) | 0.9243 | |
| | Accuracy | 0.9619 | |
| | Specificity | 0.9581 | |
| | ROC-AUC | 0.9946 | |
| | PR-AUC | 0.9848 | |
| | Recall at 1% FPR | 0.8955 | |
| | Recall at 0.1% FPR | 0.6497 | |
| | Credential-lure slice recall / precision (n=398) | 0.9959 / 0.9758 | |
| | Latency, batch 1, GPU bf16 (avg / p50) | 10.08 ms / 8.9 ms | |
| | Latency, batch 1, CPU fp32, 16 threads (avg / p50) | 458.57 ms / 445.01 ms | |
| | Instructor benchmark | 74.0 ms per sample | |
|
|
| Recall on the unwanted class is the headline number: in a defensive triage aid a missed phish is the costly error, while a false positive costs a human one glance. |
|
|
| TREC-06 carries no phishing-vs-spam ground truth, so the three-way head can only be *described* here: of the 3,793 true positives it flagged, the model called 974 phishing and 2,819 spam (informational, not a metric). |
|
|
| ### Three-way holdout (legitimate / phishing / spam) |
|
|
| Evaluated by `src/evaluate.py --split test3` on `test_holdout3.parquet` (3,220 rows with a full three-way label, threshold 0.5). Decision rule `rollup_then_argmax`: a row is unwanted iff P(phishing) + P(spam) >= threshold, then the kind is argmax(P(phishing), P(spam)), so this table collapses exactly onto the binary roll-up. |
|
|
| | metric | value | |
| |---|---:| |
| | **Macro-F1 (3 classes)** | **0.9867** | |
| | Accuracy | 0.9885 | |
| | Legitimate: precision / recall / F1 (n=1,999) | 0.9900 / 0.9935 / 0.9918 | |
| | Phishing: precision / recall / F1 (n=521) | 0.9961 / 0.9866 / 0.9913 | |
| | Spam: precision / recall / F1 (n=700) | 0.9785 / 0.9757 / 0.9771 | |
| | Phishing-vs-spam accuracy on the 1,221 true unwanted rows (kind decision alone) | 0.9959 (4 phishing read as spam, 1 spam read as phishing) | |
| | Binary roll-up on this split: recall / precision / F1 | 0.9836 / 0.9893 / 0.9864 | |
|
|
| Confusion matrix (rows = true label by corpus provenance, columns = predicted): |
|
|
| | true \ predicted | Legitimate | Phishing | Spam | |
| |---|---:|---:|---:| |
| | **Legitimate** | 1,986 | 1 | 12 | |
| | **Phishing** | 4 | 514 | 3 | |
| | **Spam** | 16 | 1 | 683 | |
|
|
| | source | rows | Legitimate | Phishing | Spam | accuracy | |
| |---|---:|---:|---:|---:|---:| |
| | Assassin | 236 | 164 | 0 | 72 | 0.9534 | |
| | CEAS-08 | 736 | 736 | 0 | 0 | 0.9905 | |
| | Enron | 628 | 0 | 0 | 628 | 0.9841 | |
| | Nazario | 222 | 0 | 222 | 0 | 0.9865 | |
| | Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9866 | |
| | TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9982 | |
| |
| **Label provenance.** Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class. The three-way holdout is drawn from text the tokenizer never saw and the pretraining corpus does not contain (Nazario / Nigerian-fraud phishing, SpamAssassin / Enron-Spam spam, CEAS-08 / TREC-07 / SpamAssassin legitimate mail from the former validation split). Because the kind labels come from which corpus a message sits in, part of any phishing-vs-spam confusion measures corpus style rather than lure content. |
| |
| ## Intended use |
| |
| A **defensive triage aid for small-business inboxes**: rank or flag incoming mail for a human to review, surface a |
| probability and the attention-highlighted tokens that drove it, and shorten the time between a phishing wave |
| landing and someone noticing. It is a classroom/final-project model and is not a substitute for a mail gateway. |
| |
| ## Limitations |
| |
| - **Era**: the labelled data is 1998-2008 spam-track mail (TREC, CEAS, SpamAssassin, Enron), plus Nazario phishing (1995-2022) and Nigerian-fraud mail. |
| Modern lures (OAuth consent phishing, QR codes, MFA fatigue, SaaS-brand impersonation) are under-represented; expect lower recall on them. |
| - **Labels follow corpus provenance**: Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class. |
| A phishing-vs-spam confusion therefore partly measures corpus style, and the credential-lure slice remains the closest proxy for phishing-proper recall. |
| - **Length confound**: legitimate emails in these corpora average ~2x the tokens of positives. Inputs are truncated to |
| 512 tokens, and the error analysis checks whether errors follow length, but a short legitimate note can |
| still look suspicious. |
| - **English only**; byte-level BPE will tokenize other scripts but the model has not seen them. |
| - **Headers are not used** (TREC rewrote them); the model sees subject and body text only. Sender reputation, |
| SPF/DKIM and URL reputation are out of scope and should be combined with this score in any real deployment. |
| - Attention highlights are a signal of what the model attended to, not an explanation of why it decided. |
| |
| ## How to load |
| |
| `model.py` from this repository is the only code dependency (plain PyTorch). |
| |
| ```python |
| import json, torch |
| from safetensors.torch import load_file |
| from tokenizers import Tokenizer |
| from model import EncoderConfig, EncoderForClassification # model.py, shipped with the weights |
|
|
| cfg = json.load(open("config.json")) |
| model = EncoderForClassification(EncoderConfig.from_dict(cfg["model"])) # num_labels = 3 |
| model.load_state_dict(load_file("model.safetensors"), strict=True) # bf16 weights cast into the fp32 module |
| model.eval() |
| |
| tok = Tokenizer.from_file("tokenizer.json") |
| tok.enable_truncation(max_length=cfg["max_seq_len"]) |
| enc = tok.encode("Your account has been suspended\n\nClick here to verify your identity within 24 hours.") |
| ids = torch.tensor([enc.ids]) |
| mask = torch.ones_like(ids, dtype=torch.bool) |
| with torch.inference_mode(): |
| probs = model(ids, mask).float().softmax(-1)[0] |
| p_unwanted = sum(float(probs[i]) for i in cfg["binary_rollup"]["unwanted"]) # P(phishing) + P(spam) |
| print("unwanted" if p_unwanted >= 0.5 else "legitimate", {cfg["labels"][str(i)]: round(float(p), 4) for i, p in enumerate(probs)}) |
| ``` |
| |
| On a GPU wrap the forward in `torch.autocast("cuda", dtype=torch.bfloat16)`. Measured single-email latency (batch 1, tokenize + forward + softmax, see the metrics table): GPU bf16 10.08 ms avg on NVIDIA GeForce RTX 3090 Ti, CPU fp32 (16 threads) 458.57 ms avg. |
|
|
| ## Files |
|
|
| - `model.safetensors` — bf16 weights, metadata `format=pt, dtype=bf16, architecture=EncoderForClassification, config=<json>, labels=<json>` |
| - `config.json` — model config, num_labels, labels, binary_rollup, label provenance, max_seq_len, export dtype, source checkpoint, fine-tune init, binary metrics (TREC-06), three-way metrics (test_holdout3), pretraining summary (null for scratch init) |
| - `tokenizer.json` — byte-level BPE, NFKC, case preserved, specials `[PAD]=0 [UNK]=1 [CLS]=2 [SEP]=3 [MASK]=4` |
| - `README.md` — this card |
| |
| ## Provenance |
| |
| Built for the Skillspire AI & Machine Learning course final project (BITS, 2026). Data decisions, measured VRAM / |
| throughput / latency and every challenge hit along the way are recorded in the project's RUN_LOG.md. |
|
|
