File size: 4,904 Bytes
9dcb805 57703cb 9dcb805 57703cb 9dcb805 57703cb 8bbfa79 57703cb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 | ---
language:
- en
license: mit
library_name: transformers
tags:
- cybersecurity
- phishing-detection
- nlp
- bert
- social-engineering
datasets:
- phishing-dataset
- email-phishing-detection
metrics:
- accuracy
- precision
- recall
- f1
- roc_auc
pipeline_tag: text-classification
widget:
- text: "URGENT: Your bank account will be suspended in 24 hours. Click here to verify your identity."
example_title: "High Urgency Phishing Lure"
- text: "Hi team, please find attached the minutes of our quarterly project review meeting."
example_title: "Benign Communication"
---
# Ph5 Text Classifier: Social Engineering & Email Phishing Detector (BERT-base)
<p align="center">
<img src="https://img.shields.io/badge/Model-BERT--base-blue.svg" alt="Model">
<img src="https://img.shields.io/badge/Accuracy-97.22%25-brightgreen.svg" alt="Accuracy">
<img src="https://img.shields.io/badge/F1--Score-96.67%25-success.svg" alt="F1 Score">
<img src="https://img.shields.io/badge/ROC--AUC-0.9961-red.svg" alt="ROC-AUC">
<img src="https://img.shields.io/badge/Latency-34.2ms-orange.svg" alt="Latency">
<img src="https://img.shields.io/badge/License-MIT-purple.svg" alt="License">
</p>
## Model Summary
The **Ph5 Text Classifier** is a fine-tuned `bert-base-uncased` language model engineered to detect psychological manipulation, urgency cues, coercive authority claims, and credential-harvesting lures across emails, SMS messages, and HTML message payloads.
It functions as the core semantic inspection branch in the **Ph5 Multimodal Cybersecurity Framework**, and is directly coupled with an optical character recognition (OCR) fallback engine to analyze text embedded within attached images and QR-code lures.
---
## Performance & Evaluation Benchmark
Evaluated on an independent, strictly isolated held-out test split of **11,652 deduplicated messages** (6,747 Benign [57.90%], 4,905 Phishing [42.10%]):
| Metric | Measured Score | Standard Deviation (5-Fold CV) |
| :--- | :--- | :--- |
| **Accuracy** | **97.22%** | ± 0.10% |
| **Precision** | **97.51%** | ± 0.08% |
| **Recall** | **95.84%** | ± 0.12% |
| **F1-Score** | **96.67%** | ± 0.09% |
| **ROC-AUC** | **0.9961** | ± 0.0004 |
| **False Positive Rate** | **1.78%** (FP = 120 / 6747) | — |
| **False Negative Rate** | **4.16%** (FN = 204 / 4905) | — |
| **Mean Inference Time** | **34.2 ms** | Batch Size = 1 (RTX 3050 Laptop GPU) |
### Baseline Comparison
| Architecture | Accuracy | Precision | Recall | F1-Score |
| :--- | :--- | :--- | :--- | :--- |
| TF-IDF + Multinomial Naive Bayes | 86.85% | 88.32% | 79.25% | 83.54% |
| TF-IDF + Logistic Regression | 88.27% | 89.24% | 82.02% | 85.48% |
| **Ph5 BERT-base (This Model)** | **97.22%** | **97.51%** | **95.84%** | **96.67%** |
---
## Confusion Matrix (Test Split: N = 11,652)
```text
Predicted
Benign Phishing
Benign 6627 120
Phishing 204 4701
```
---
## Intended Use & Integration
### Python (Transformers & PyTorch)
```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
# Load model and tokenizer from Hugging Face Hub
repo_id = "xynaxdev/ph5-text-bert"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id)
model.eval()
# Sample message text
message = "Urgent: Your account credentials have expired. Verify immediately to avoid permanent lockout."
# Tokenize and run forward pass
inputs = tokenizer(message, return_tensors="pt", max_length=256, truncation=True)
with torch.no_grad():
logits = model(**inputs).logits
probabilities = torch.softmax(logits, dim=-1).squeeze().tolist()
p_benign, p_phish = probabilities[0], probabilities[1]
print(f"Benign Probability: {p_benign:.4f}")
print(f"Phishing Probability: {p_phish:.4f}")
# Output: Phishing Probability: 0.9872
```
---
## Training Configuration & Data Provenance
- **Pretrained Backbone:** `bert-base-uncased` (12 layers, 768 hidden dimensions, 12 attention heads, 110M parameters)
- **Classification Head:** Linear projection (768 -> 2)
- **Training Samples:** 77,677 messages total (54,374 train / 11,651 val / 11,652 test)
- Sourced from curated open-access corpora on Hugging Face: `ealvaradob/phishing-dataset` and `mrm8488/email-phishing-detection`
- **Optimizer:** AdamW (beta1=0.9, beta2=0.999, weight decay = 0.01)
- **Learning Rate:** 2e-5 with linear learning rate warmup
- **Batch Size:** 16 (Trained with PyTorch Automatic Mixed Precision)
- **Epochs:** 3 epochs (Early stopping checkpoint)
---
## Citation & Framework
```bibtex
@article{devi2026ph5,
title={Ph5: A Multimodal Framework for Cross-Modality Evasion-Resilient Phishing Detection in Multi-Vector Environments},
author={Ritu Devi, Deepti Singh and Akash Sah},
journal={Journal of Computer & Mathematical Sciences / JCMM},
year={2026}
}
```
|