Upload BITS phishing detector

README.md

@@ -0,0 +1,251 @@
+---
+datasets:
+- puyang2025/seven-phishing-email-datasets
+- corbt/enron-emails
+- kudzaiprichard/aura-phishing-email-corpus
+language: en
+library_name: pytorch
+license: mit
+pipeline_tag: text-classification
+tags:
+- phishing
+- email-security
+- text-classification
+- from-scratch
+- spam-detection
+model-index:
+- name: phishing-detector-encoder
+  results:
+  - task:
+      type: text-classification
+    dataset:
+      name: TREC-06 holdout (puyang2025), binary roll-up
+      type: puyang2025/seven-phishing-email-datasets
+    metrics:
+    - type: recall
+      value: 0.9741
+      name: Recall (unwanted)
+    - type: precision
+      value: 0.8794
+      name: Precision (unwanted)
+    - type: f1
+      value: 0.9243
+      name: F1 (unwanted)
+    - type: accuracy
+      value: 0.9619
+      name: Accuracy
+    - type: roc_auc
+      value: 0.9946
+      name: ROC-AUC
+  - task:
+      type: text-classification
+    dataset:
+      name: three-way holdout (Nazario + Nigerian Fraud phishing, SpamAssassin / Enron-Spam
+        spam, CEAS-08 / TREC-07 / SpamAssassin ham)
+      type: kudzaiprichard/aura-phishing-email-corpus
+    metrics:
+    - type: f1
+      value: 0.9867
+      name: Macro-F1 (legitimate / phishing / spam)
+    - type: accuracy
+      value: 0.9885
+      name: Accuracy (3-way)
+---
+
+# phishing-detector-encoder: legitimate / phishing / spam email classifier trained from scratch
+
+> **No external pretrained weights were used.** The encoder, the tokenizer and every parameter in `model.safetensors`
+> were trained from random initialisation inside this project, in plain PyTorch, without the `transformers`
+> library. The only external inputs are the public email corpora listed below.
+> Initialisation of this checkpoint: `pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final`.
+
+## What it is
+
+A 337,444,867-parameter BERT-style **pre-LayerNorm Transformer encoder** (24 layers, d_model 1024,
+16 heads, FFN 4096, byte-level BPE vocabulary of 32,768, 512 positions) with a
+CLS-pooled classification head that maps an email (`subject + "\n\n" + body`) to three classes: `0 = Legitimate`, `1 = Phishing`, `2 = Spam`. The binary verdict shown by the demo is the roll-up **Unwanted (phishing or spam)** = P(phishing) + P(spam) (`config.json["binary_rollup"]`). Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class.
+The architecture is defined in `model.py`, which ships with the weights. Weights are stored in **bf16** safetensors
+(~675 MB); the header carries the architecture config and the label names as JSON.
+
+## How it was trained
+
+1. **Masked-language-model pretraining from scratch** on the deduplicated Enron email corpus (`corbt/enron-emails`: 517,401 rows -> 226,273 documents after exact/near-duplicate removal and quote stripping), 15% dynamic masking (80/10/10), sequence packing to 512 tokens, AdamW (betas (0.9, 0.98), weight decay 0.01), peak LR 0.00015, bf16 autocast on one RTX 3090 Ti. Run summary (from the pretraining run's `final/config.json`):
+   - run_name: large-24L
+   - status: complete
+   - source_checkpoint: step-0009552
+   - steps: 9552
+   - total_steps_planned: 9552
+   - steps_per_epoch: 796
+   - epochs_planned: 12
+   - epochs_completed: 12.0
+   - tokens_seen: 1291124736
+   - total_training_seconds: 54185.9
+   - total_training_time: 15h 03m 05s
+   - final_loss: 1.523698394948786
+   - val_loss: 1.9862932266429436
+   - val_masked_acc: 0.6291136684837473
+   - peak_vram_reserved_gb: 17.939
+   - peak_vram_allocated_gb: 17.494
+   - micro_batch: 24
+   - grad_accumulation: 11
+   - effective_batch: 264
+   - precision: bf16_autocast
+   - optimizer: AdamW(fused) {"lr": 0.00015, "betas": [0.9, 0.98], "eps": 1e-06, "weight_decay": 0.01, "grad_clip": 1.0, "mlm_probability": 0.15, "mask_replace_probs": [0.8, 0.1, 0.1], "schedule": "cosine_to_10pct", "effective_batch_sequences": 256}
+   - schedule: {'kind': 'cosine', 'peak_lr': 0.00015, 'warmup_steps': 573, 'total_steps': 9552, 'floor_ratio': 0.1}
+   - mean_tokens_per_s: 23889
+   - data: {'n_windows': 210368, 'n_val_windows': 1999, 'tokenizer_sha1': '9f50c5e775089d2ca3efc82e100cd6b3396ed7e7', 'n_docs': 226273, 'limit_docs': None, 'window': '[CLS] + 511 stream tokens'}
+   - weights_dtype: float32
+   - exported_at: 2026-10-09T03:31:14
+2. **Fine-tuning** on `puyang2025/seven-phishing-email-datasets` with our own split: TREC-05, TREC-07 and CEAS-08 (positives trained as *unwanted* only), Enron-Spam and SpamAssassin (spam folders = `Spam`), the Nazario and Nigerian-fraud corpora of `kudzaiprichard/aura-phishing-email-corpus` (= `Phishing`) and synthetic BITS-style lures (= `Phishing`) form train (157,113) / val (17,643), grouped by
+   near-duplicate cluster so no template straddles the two; **TREC-06 (16,302 rows, 23.9% positive) is held out
+   entirely** as the unseen binary test set and never touched by the tokenizer, pretraining or fine-tuning (1,652 pool rows that
+   near-duplicate TREC-06 templates were removed). A second holdout, **test_holdout3**, holds clean-provenance three-way rows never seen by the tokenizer or the pretraining corpus for the 3x3 confusion matrix. Counts come from `data/processed/splits_manifest.json`.
+   Rows with a clean three-way label train with 3-class cross-entropy; TREC / CEAS positives train on the probability mass of {phishing, spam} (partial-label loss: -log(P(phishing) + P(spam))), so they shape the legitimate-vs-unwanted decision without asserting a kind they were never labelled with.
+   Fine-tune dials (from the checkpoint's config.json):
+   - run_name: ft-focal
+   - init: pretrained:checkpoints/pretrain/large-24L/final
+   - n_layers: 24
+   - freeze_layers: 0
+   - loss: focal
+   - focal_gamma: 2.0
+   - lr: 3e-05
+   - batch_size: 32
+   - micro_batch: 16
+   - epochs: 2
+   - max_steps: None
+   - eval_every: 1250
+   - log_every: 20
+   - eval_batch_size: 64
+   - warmup_fraction: 0.06
+   - weight_decay: 0.01
+   - betas: [0.9, 0.98]
+   - eps: 1e-06
+   - grad_clip: 1.0
+   - patience: 2
+   - schedule: linear_decay
+   - selection_metric: val_f1_unwanted
+   - pooling: cls
+   - num_labels: 3
+   - synthetic: off
+   - limit_rows: None
+   - seed: 42
+   - cpu: False
+
+## Metrics
+
+### TREC-06 binary holdout (Unwanted (phishing or spam))
+
+Evaluated by `src/evaluate.py` on the **test** split (`test_holdout.parquet`, 16,302 rows, threshold 0.5). Positive class = **Unwanted (phishing or spam)**, P(unwanted) = P(phishing) + P(spam); these definitions are identical to the binary-era runs, so the numbers are comparable.
+
+| metric | value |
+|---|---:|
+| **Recall (unwanted)** | **0.9741** (101 missed of 3,894) |
+| Precision (unwanted) | 0.8794 |
+| F1 (unwanted) | 0.9243 |
+| Accuracy | 0.9619 |
+| Specificity | 0.9581 |
+| ROC-AUC | 0.9946 |
+| PR-AUC | 0.9848 |
+| Recall at 1% FPR | 0.8955 |
+| Recall at 0.1% FPR | 0.6497 |
+| Credential-lure slice recall / precision (n=398) | 0.9959 / 0.9758 |
+| Latency, batch 1, GPU bf16 (avg / p50) | 10.08 ms / 8.9 ms |
+| Latency, batch 1, CPU fp32, 16 threads (avg / p50) | 458.57 ms / 445.01 ms |
+| Instructor benchmark | 74.0 ms per sample |
+
+Recall on the unwanted class is the headline number: in a defensive triage aid a missed phish is the costly error, while a false positive costs a human one glance.
+
+TREC-06 carries no phishing-vs-spam ground truth, so the three-way head can only be *described* here: of the 3,793 true positives it flagged, the model called 974 phishing and 2,819 spam (informational, not a metric).
+
+### Three-way holdout (legitimate / phishing / spam)
+
+Evaluated by `src/evaluate.py --split test3` on `test_holdout3.parquet` (3,220 rows with a full three-way label, threshold 0.5). Decision rule `rollup_then_argmax`: a row is unwanted iff P(phishing) + P(spam) >= threshold, then the kind is argmax(P(phishing), P(spam)), so this table collapses exactly onto the binary roll-up.
+
+| metric | value |
+|---|---:|
+| **Macro-F1 (3 classes)** | **0.9867** |
+| Accuracy | 0.9885 |
+| Legitimate: precision / recall / F1 (n=1,999) | 0.9900 / 0.9935 / 0.9918 |
+| Phishing: precision / recall / F1 (n=521) | 0.9961 / 0.9866 / 0.9913 |
+| Spam: precision / recall / F1 (n=700) | 0.9785 / 0.9757 / 0.9771 |
+| Phishing-vs-spam accuracy on the 1,221 true unwanted rows (kind decision alone) | 0.9959 (4 phishing read as spam, 1 spam read as phishing) |
+| Binary roll-up on this split: recall / precision / F1 | 0.9836 / 0.9893 / 0.9864 |
+
+Confusion matrix (rows = true label by corpus provenance, columns = predicted):
+
+| true \ predicted | Legitimate | Phishing | Spam |
+|---|---:|---:|---:|
+| **Legitimate** | 1,986 | 1 | 12 |
+| **Phishing** | 4 | 514 | 3 |
+| **Spam** | 16 | 1 | 683 |
+
+| source | rows | Legitimate | Phishing | Spam | accuracy |
+|---|---:|---:|---:|---:|---:|
+| Assassin | 236 | 164 | 0 | 72 | 0.9534 |
+| CEAS-08 | 736 | 736 | 0 | 0 | 0.9905 |
+| Enron | 628 | 0 | 0 | 628 | 0.9841 |
+| Nazario | 222 | 0 | 222 | 0 | 0.9865 |
+| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9866 |
+| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9982 |
+
+**Label provenance.** Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class. The three-way holdout is drawn from text the tokenizer never saw and the pretraining corpus does not contain (Nazario / Nigerian-fraud phishing, SpamAssassin / Enron-Spam spam, CEAS-08 / TREC-07 / SpamAssassin legitimate mail from the former validation split). Because the kind labels come from which corpus a message sits in, part of any phishing-vs-spam confusion measures corpus style rather than lure content.
+
+## Intended use
+
+A **defensive triage aid for small-business inboxes**: rank or flag incoming mail for a human to review, surface a
+probability and the attention-highlighted tokens that drove it, and shorten the time between a phishing wave
+landing and someone noticing. It is a classroom/final-project model and is not a substitute for a mail gateway.
+
+## Limitations
+
+- **Era**: the labelled data is 1998-2008 spam-track mail (TREC, CEAS, SpamAssassin, Enron), plus Nazario phishing (1995-2022) and Nigerian-fraud mail.
+  Modern lures (OAuth consent phishing, QR codes, MFA fatigue, SaaS-brand impersonation) are under-represented; expect lower recall on them.
+- **Labels follow corpus provenance**: Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class.
+  A phishing-vs-spam confusion therefore partly measures corpus style, and the credential-lure slice remains the closest proxy for phishing-proper recall.
+- **Length confound**: legitimate emails in these corpora average ~2x the tokens of positives. Inputs are truncated to
+  512 tokens, and the error analysis checks whether errors follow length, but a short legitimate note can
+  still look suspicious.
+- **English only**; byte-level BPE will tokenize other scripts but the model has not seen them.
+- **Headers are not used** (TREC rewrote them); the model sees subject and body text only. Sender reputation,
+  SPF/DKIM and URL reputation are out of scope and should be combined with this score in any real deployment.
+- Attention highlights are a signal of what the model attended to, not an explanation of why it decided.
+
+## How to load
+
+`model.py` from this repository is the only code dependency (plain PyTorch).
+
+```python
+import json, torch
+from safetensors.torch import load_file
+from tokenizers import Tokenizer
+from model import EncoderConfig, EncoderForClassification   # model.py, shipped with the weights
+
+cfg = json.load(open("config.json"))
+model = EncoderForClassification(EncoderConfig.from_dict(cfg["model"]))   # num_labels = 3
+model.load_state_dict(load_file("model.safetensors"), strict=True)   # bf16 weights cast into the fp32 module
+model.eval()
+
+tok = Tokenizer.from_file("tokenizer.json")
+tok.enable_truncation(max_length=cfg["max_seq_len"])
+enc = tok.encode("Your account has been suspended\n\nClick here to verify your identity within 24 hours.")
+ids = torch.tensor([enc.ids])
+mask = torch.ones_like(ids, dtype=torch.bool)
+with torch.inference_mode():
+    probs = model(ids, mask).float().softmax(-1)[0]
+p_unwanted = sum(float(probs[i]) for i in cfg["binary_rollup"]["unwanted"])   # P(phishing) + P(spam)
+print("unwanted" if p_unwanted >= 0.5 else "legitimate", {cfg["labels"][str(i)]: round(float(p), 4) for i, p in enumerate(probs)})
+```
+
+On a GPU wrap the forward in `torch.autocast("cuda", dtype=torch.bfloat16)`. Measured single-email latency (batch 1, tokenize + forward + softmax, see the metrics table): GPU bf16 10.08 ms avg on NVIDIA GeForce RTX 3090 Ti, CPU fp32 (16 threads) 458.57 ms avg.
+
+## Files
+
+- `model.safetensors` — bf16 weights, metadata `format=pt, dtype=bf16, architecture=EncoderForClassification, config=<json>, labels=<json>`
+- `config.json` — model config, num_labels, labels, binary_rollup, label provenance, max_seq_len, export dtype, source checkpoint, fine-tune init, binary metrics (TREC-06), three-way metrics (test_holdout3), pretraining summary (null for scratch init)
+- `tokenizer.json` — byte-level BPE, NFKC, case preserved, specials `[PAD]=0 [UNK]=1 [CLS]=2 [SEP]=3 [MASK]=4`
+- `README.md` — this card
+
+## Provenance
+
+Built for the Skillspire AI & Machine Learning course final project (BITS, 2026). Data decisions, measured VRAM /
+throughput / latency and every challenge hit along the way are recorded in the project's RUN_LOG.md.

config.json

@@ -0,0 +1,467 @@
+{
+  "architecture": "EncoderForClassification",
+  "model": {
+    "vocab_size": 32768,
+    "max_seq_len": 512,
+    "d_model": 1024,
+    "n_layers": 24,
+    "n_heads": 16,
+    "d_ff": 4096,
+    "dropout": 0.1,
+    "attention_dropout": 0.1,
+    "layer_norm_eps": 1e-05,
+    "pad_token_id": 0,
+    "tie_mlm_head": true,
+    "grad_checkpointing": false,
+    "num_labels": 3
+  },
+  "num_labels": 3,
+  "labels": {
+    "0": "Legitimate",
+    "1": "Phishing",
+    "2": "Spam"
+  },
+  "binary_rollup": {
+    "unwanted": [
+      1,
+      2
+    ]
+  },
+  "unwanted_name": "Unwanted (phishing or spam)",
+  "label_provenance": "Phishing = Nazario phishing corpus + Nigerian-fraud (advance-fee) corpus + synthetic BITS-style credential/BEC lures; Spam = SpamAssassin and Enron-Spam spam folders; the TREC-05 / TREC-07 / CEAS-08 positives never separated spam from phishing and were trained as 'unwanted' only (probability mass of phishing + spam), never as either class.",
+  "max_seq_len": 512,
+  "export_dtype": "bf16",
+  "source_checkpoint": "/home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-focal/best",
+  "exported": "2026-10-12T11:28:39",
+  "tokenizer": "tokenizer.json",
+  "model_input": "subject + \"\\n\\n\" + body",
+  "finetune_checkpoint": {
+    "tokenizer": "tokenizer.json",
+    "finetune": {
+      "run_name": "ft-focal",
+      "init": "pretrained:checkpoints/pretrain/large-24L/final",
+      "n_layers": 24,
+      "freeze_layers": 0,
+      "loss": "focal",
+      "focal_gamma": 2.0,
+      "lr": 3e-05,
+      "batch_size": 32,
+      "micro_batch": 16,
+      "epochs": 2,
+      "max_steps": null,
+      "eval_every": 1250,
+      "log_every": 20,
+      "eval_batch_size": 64,
+      "warmup_fraction": 0.06,
+      "weight_decay": 0.01,
+      "betas": [
+        0.9,
+        0.98
+      ],
+      "eps": 1e-06,
+      "grad_clip": 1.0,
+      "patience": 2,
+      "schedule": "linear_decay",
+      "selection_metric": "val_f1_unwanted",
+      "pooling": "cls",
+      "num_labels": 3,
+      "synthetic": "off",
+      "limit_rows": null,
+      "seed": 42,
+      "cpu": false
+    },
+    "best_epoch": 1,
+    "best_step": 8750,
+    "num_labels": 3,
+    "label_names": [
+      "Legitimate",
+      "Phishing",
+      "Spam"
+    ],
+    "binary_rollup_name": "Unwanted (phishing or spam)",
+    "selection_metric": "val_f1_unwanted",
+    "selection_value": 0.9951084002657166,
+    "val_metrics": {
+      "accuracy": 0.9947003402250719,
+      "precision": 0.9954089646007007,
+      "recall": 0.9948080173871046,
+      "f1": 0.9951084002657166,
+      "roc_auc": 0.9998186946414248,
+      "pr_auc": 0.9998464119156345,
+      "tn": 6964,
+      "fp": 38,
+      "fn": 43,
+      "tp": 8239,
+      "threshold": 0.5,
+      "f1_optimal_threshold": 0.42074909806251526,
+      "f1_at_optimal_threshold": 0.9952995058454863,
+      "f1_unwanted": 0.9951084002657166,
+      "n_3way": 8105,
+      "accuracy_3way": 0.994694632942628,
+      "macro_f1_3way": 0.9868751212108838,
+      "precision_legitimate": 0.9988533753762362,
+      "recall_legitimate": 0.99528706083976,
+      "f1_legitimate": 0.9970670291151013,
+      "precision_phishing": 0.9853372434017595,
+      "recall_phishing": 0.9882352941176471,
+      "f1_phishing": 0.986784140969163,
+      "precision_spam": 0.9618805590851334,
+      "recall_spam": 0.9921363040629095,
+      "f1_spam": 0.9767741935483871,
+      "cm3_ll": 6969,
+      "cm3_lp": 4,
+      "cm3_ls": 29,
+      "cm3_pl": 3,
+      "cm3_pp": 336,
+      "cm3_ps": 1,
+      "cm3_sl": 5,
+      "cm3_sp": 1,
+      "cm3_ss": 757,
+      "n_unwanted_3way": 1103,
+      "pvs_phish_as_phish": 339,
+      "pvs_phish_as_spam": 1,
+      "pvs_spam_as_phish": 1,
+      "pvs_spam_as_spam": 762,
+      "val_loss": 0.030296484990695233,
+      "n": 15284,
+      "eval_seconds": 48.0
+    },
+    "val_metric_sets": {
+      "binary_unwanted": {
+        "accuracy": 0.9947003402250719,
+        "precision": 0.9954089646007007,
+        "recall": 0.9948080173871046,
+        "f1": 0.9951084002657166,
+        "roc_auc": 0.9998186946414248,
+        "pr_auc": 0.9998464119156345,
+        "tn": 6964,
+        "fp": 38,
+        "fn": 43,
+        "tp": 8239,
+        "threshold": 0.5,
+        "f1_optimal_threshold": 0.42074909806251526,
+        "f1_at_optimal_threshold": 0.9952995058454863,
+        "n": 15284,
+        "positive_class": "Unwanted (phishing or spam)"
+      },
+      "three_way": {
+        "n": 8105,
+        "accuracy": 0.994694632942628,
+        "macro_f1": 0.9868751212108838,
+        "per_class": {
+          "Legitimate": {
+            "precision": 0.9988533753762362,
+            "recall": 0.99528706083976,
+            "f1": 0.9970670291151013
+          },
+          "Phishing": {
+            "precision": 0.9853372434017595,
+            "recall": 0.9882352941176471,
+            "f1": 0.986784140969163
+          },
+          "Spam": {
+            "precision": 0.9618805590851334,
+            "recall": 0.9921363040629095,
+            "f1": 0.9767741935483871
+          }
+        },
+        "confusion": {
+          "order": [
+            "Legitimate",
+            "Phishing",
+            "Spam"
+          ],
+          "rows_true_cols_pred": [
+            [
+              6969,
+              4,
+              29
+            ],
+            [
+              3,
+              336,
+              1
+            ],
+            [
+              5,
+              1,
+              757
+            ]
+          ]
+        },
+        "phish_vs_spam_on_true_unwanted": {
+          "n": 1103,
+          "order": [
+            "Phishing",
+            "Spam"
+          ],
+          "rows_true_cols_pred": [
+            [
+              339,
+              1
+            ],
+            [
+              1,
+              762
+            ]
+          ]
+        }
+      }
+    },
+    "thresholds": {
+      "decision": 0.5,
+      "decision_applies_to": "P(unwanted) = sum of non-legitimate probabilities",
+      "val_f1_optimal": 0.42074909806251526,
+      "val_f1_at_optimal": 0.9952995058454863
+    },
+    "init": "pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final",
+    "params": {
+      "total": 337444867,
+      "trainable": 337444867,
+      "embeddings": 34080768,
+      "encoder_layers": 302309376,
+      "final_norm": 2048,
+      "head": 1052675
+    }
+  },
+  "finetune_init": "pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final",
+  "mlm_pretraining": "summary below",
+  "metrics": {
+    "split": "test",
+    "split_file": "/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet",
+    "n_rows": 16302,
+    "dev_run": false,
+    "threshold": 0.5,
+    "timestamp": "2026-10-09 10:46",
+    "num_labels": 3,
+    "positive_class": "Unwanted (phishing or spam)",
+    "accuracy": 0.9619065145380935,
+    "precision": 0.8794342684906098,
+    "recall": 0.9740626605033385,
+    "f1": 0.9243328865602534,
+    "specificity": 0.9580915538362347,
+    "confusion_matrix": {
+      "tn": 11888,
+      "fp": 520,
+      "fn": 101,
+      "tp": 3793
+    },
+    "roc_auc": 0.9946343971962354,
+    "pr_auc": 0.9847899987683836,
+    "recall_at_fpr_1pct": 0.8954802259887006,
+    "recall_at_fpr_0_1pct": 0.6497175141242938,
+    "best_f1_threshold": 0.6815109252929688,
+    "credential_lure_recall": 0.9958847736625515,
+    "credential_lure_precision": 0.9758064516129032,
+    "credential_lure_n": 398,
+    "predicted_kind_of_flagged_true_positives": {
+      "n": 3793,
+      "predicted_phishing": 974,
+      "predicted_spam": 2819
+    },
+    "latency_ms": {
+      "gpu_bf16_avg": 10.08,
+      "gpu_bf16_p50": 8.9,
+      "cpu_fp32_avg": 458.57,
+      "cpu_fp32_p50": 445.01,
+      "cpu_threads": 16,
+      "instructor_benchmark_ms": 74.0
+    }
+  },
+  "metrics_source": "/home/nathank/BITS/dev/Phishing Model/reports/metrics.json",
+  "metrics_checkpoint_matches": true,
+  "metrics_mismatch": null,
+  "metrics_threeway": {
+    "split": "test3",
+    "split_file": "/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout3.parquet",
+    "n_rows": 3220,
+    "dev_run": false,
+    "threshold": 0.5,
+    "timestamp": "2026-10-09 10:46",
+    "rows_with_full_label": 3220,
+    "label3_counts": {
+      "legit": 1999,
+      "phish": 521,
+      "spam": 700,
+      "unwanted": 0
+    },
+    "decision_rule": "rollup_then_argmax",
+    "accuracy": 0.9885093167701864,
+    "macro_f1": 0.9867305252015051,
+    "per_class": {
+      "Legitimate": {
+        "precision": 0.9900299102691924,
+        "recall": 0.9934967483741871,
+        "f1": 0.9917602996254682,
+        "support": 1999
+      },
+      "Phishing": {
+        "precision": 0.9961240310077519,
+        "recall": 0.9865642994241842,
+        "f1": 0.991321118611379,
+        "support": 521
+      },
+      "Spam": {
+        "precision": 0.9785100286532952,
+        "recall": 0.9757142857142858,
+        "f1": 0.977110157367668,
+        "support": 700
+      }
+    },
+    "confusion_matrix": {
+      "labels": [
+        "Legitimate",
+        "Phishing",
+        "Spam"
+      ],
+      "matrix": [
+        [
+          1986,
+          1,
+          12
+        ],
+        [
+          4,
+          514,
+          3
+        ],
+        [
+          16,
+          1,
+          683
+        ]
+      ],
+      "cells": {
+        "true_legitimate": {
+          "pred_legitimate": 1986,
+          "pred_phishing": 1,
+          "pred_spam": 12
+        },
+        "true_phishing": {
+          "pred_legitimate": 4,
+          "pred_phishing": 514,
+          "pred_spam": 3
+        },
+        "true_spam": {
+          "pred_legitimate": 16,
+          "pred_phishing": 1,
+          "pred_spam": 683
+        }
+      }
+    },
+    "kind_accuracy_on_true_unwanted": 0.9959049959049959,
+    "phishing_as_spam": 4,
+    "spam_as_phishing": 1,
+    "n_true_unwanted": 1221,
+    "per_source": {
+      "Assassin": {
+        "n": 236,
+        "counts": {
+          "Legitimate": 164,
+          "Phishing": 0,
+          "Spam": 72
+        },
+        "accuracy": 0.9533898305084746
+      },
+      "CEAS-08": {
+        "n": 736,
+        "counts": {
+          "Legitimate": 736,
+          "Phishing": 0,
+          "Spam": 0
+        },
+        "accuracy": 0.9904891304347826
+      },
+      "Enron": {
+        "n": 628,
+        "counts": {
+          "Legitimate": 0,
+          "Phishing": 0,
+          "Spam": 628
+        },
+        "accuracy": 0.9840764331210191
+      },
+      "Nazario": {
+        "n": 222,
+        "counts": {
+          "Legitimate": 0,
+          "Phishing": 222,
+          "Spam": 0
+        },
+        "accuracy": 0.9864864864864865
+      },
+      "Nigerian_Fraud": {
+        "n": 299,
+        "counts": {
+          "Legitimate": 0,
+          "Phishing": 299,
+          "Spam": 0
+        },
+        "accuracy": 0.9866220735785953
+      },
+      "TREC-07": {
+        "n": 1099,
+        "counts": {
+          "Legitimate": 1099,
+          "Phishing": 0,
+          "Spam": 0
+        },
+        "accuracy": 0.9981801637852593
+      }
+    },
+    "binary_rollup_on_this_split": {
+      "recall": 0.9836199836199836,
+      "precision": 0.9892915980230642,
+      "f1": 0.986447638603696,
+      "accuracy": 0.989751552795031
+    }
+  },
+  "metrics_threeway_source": "/home/nathank/BITS/dev/Phishing Model/reports/metrics_test3.json",
+  "metrics_threeway_reason": null,
+  "metrics_threeway_checkpoint_matches": true,
+  "metrics_threeway_mismatch": null,
+  "training": {
+    "run_name": "large-24L",
+    "status": "complete",
+    "source_checkpoint": "step-0009552",
+    "steps": 9552,
+    "total_steps_planned": 9552,
+    "steps_per_epoch": 796,
+    "epochs_planned": 12,
+    "epochs_completed": 12.0,
+    "tokens_seen": 1291124736,
+    "total_training_seconds": 54185.9,
+    "total_training_time": "15h 03m 05s",
+    "final_loss": 1.523698394948786,
+    "val_loss": 1.9862932266429436,
+    "val_masked_acc": 0.6291136684837473,
+    "peak_vram_reserved_gb": 17.939,
+    "peak_vram_allocated_gb": 17.494,
+    "micro_batch": 24,
+    "grad_accumulation": 11,
+    "effective_batch": 264,
+    "precision": "bf16_autocast",
+    "optimizer": "AdamW(fused) {\"lr\": 0.00015, \"betas\": [0.9, 0.98], \"eps\": 1e-06, \"weight_decay\": 0.01, \"grad_clip\": 1.0, \"mlm_probability\": 0.15, \"mask_replace_probs\": [0.8, 0.1, 0.1], \"schedule\": \"cosine_to_10pct\", \"effective_batch_sequences\": 256}",
+    "schedule": {
+      "kind": "cosine",
+      "peak_lr": 0.00015,
+      "warmup_steps": 573,
+      "total_steps": 9552,
+      "floor_ratio": 0.1
+    },
+    "mean_tokens_per_s": 23889,
+    "data": {
+      "n_windows": 210368,
+      "n_val_windows": 1999,
+      "tokenizer_sha1": "9f50c5e775089d2ca3efc82e100cd6b3396ed7e7",
+      "n_docs": 226273,
+      "limit_docs": null,
+      "window": "[CLS] + 511 stream tokens"
+    },
+    "weights_dtype": "float32",
+    "exported_at": "2026-10-09T03:31:14"
+  },
+  "training_source": "/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final/config.json",
+  "from_scratch": true,
+  "external_pretrained_weights_used": "none"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f23caf1836250c83ad5a29a4fd5931bcd33a3dae4877120a6254a1e465017ff
+size 674920638