mmBERT Multilingual Spam Detector
A multilingual spam detection model fine-tuned on jhu-clsp/mmBERT-small using the agentlans/common-crawl-spam dataset.
- Model Architecture: ModernBERT
- Problem Type: Regression (Single-output spam scoring)
- Number of Labels: 1
- Vocabulary Size: 256,000
- License: MIT
Quick Start & Usage
You can easily use this model for inference with the Hugging Face pipeline:
import torch
from transformers import pipeline
model_name = "agentlans/mmBERT-small-spam-detector"
regressor = pipeline(
"text-classification",
model=model_name,
tokenizer=model_name,
device=0 if torch.cuda.is_available() else -1,
function_to_apply="none",
)
messages = [
"The meeting starts at 10 AM tomorrow.",
"WIN $$$ NOW!!! FREE MONEY GUARANTEED!!! CLICK HERE!!!",
]
predictions = regressor(messages, batch_size=16, truncation=True)
for message, prediction in zip(messages, predictions):
score = prediction["score"] # logits for the "spam" label prediction
print(f"Message: {message}\nPredicted spam score: {score:.3f}\n")
Evaluation & Performance
Test Set Performance
- ROC-AUC: 0.9327 (Evaluated on n = 10,000 test samples from
agentlans/common-crawl-spam). - Decision Rule: In this dataset, a sample is classified as spam if its logit > 0. Points along the ROC curve represent alternative cutoff thresholds for the classifier's output.
Example Predictions & Qualitative Analysis
The dot plot below illustrates predicted spam scores across various categories and languages, highlighting the expected progression from informational messages to pure spam. Note the score variability observed within the three least spam-like categories.
The raw sample classification results are in the following table:
Click here for Table 1
| text | language | category | predicted_spam_score |
|---|---|---|---|
| The meeting starts at 10 AM tomorrow. | en | pure_information | -11.5533 |
| La biblioteca cierra a las seis de la tarde. | es | pure_information | -11.8742 |
| Le train pour Paris part ร 14 h 30. | fr | pure_information | -5.45476 |
| Die Prรผfung findet am Montag statt. | de | pure_information | -10.171 |
| ๆๅคฉ็ๅคฉๆฐ้ขๆฅๆพ็คบๆ้จใ | zh | pure_information | -7.34299 |
| เคเค เคธเฅเคเฅเคฒ เคธเฅเคฌเคน เคเค เคฌเคเฅ เคเฅเคฒเฅเคเคพใ | hi | pure_information | -3.78344 |
| ุณูุจุฏุฃ ุงูุงุฌุชู ุงุน ูู ุงูุณุงุนุฉ ุงูุชุงุณุนุฉ ุตุจุงุญูุง. | ar | pure_information | -7.88322 |
| A reuniรฃo foi transferida para sexta-feira. | pt | pure_information | -10.3215 |
| Your package will arrive tomorrow. Track it on the official website. | en | mostly_informational | -4.51236 |
| Tu factura vence maรฑana. Puedes consultarla en nuestra aplicaciรณn. | es | mostly_informational | 2.66382 |
| Votre abonnement sera renouvelรฉ le mois prochain. | fr | mostly_informational | -4.89603 |
| Ihr Kontoauszug ist jetzt online verfรผgbar. | de | mostly_informational | -1.92531 |
| Our store is open until 9 PM. Check out our new collection! | en | mixed_information_promotion | -5.87974 |
| ยกTu pedido estรก listo! Aprovecha un 10% de descuento en tu prรณxima compra. | es | mixed_information_promotion | 3.82188 |
| Votre livraison est arrivรฉe. Dรฉcouvrez รฉgalement nos offres de la semaine. | fr | mixed_information_promotion | -4.02641 |
| ่ฏพ็จๅทฒๆดๆฐ๏ผๆฌข่ฟๅๅ ๆไปฌ็้ซ็บงไป่ดนๅน่ฎญใ | zh | mixed_information_promotion | -1.41289 |
| Limited-time offer! Subscribe today to unlock exclusive benefits. | en | suspicious_promotion | 4.41937 |
| ยกOferta exclusiva! Compra ahora antes de que termine la promociรณn. | es | suspicious_promotion | 5.47664 |
| Gagnez des cadeaux en vous inscrivant dรจs aujourd'hui ! | fr | suspicious_promotion | 4.5007 |
| Herzlichen Glรผckwunsch! Sichern Sie sich jetzt Ihren Sonderrabatt! | de | suspicious_promotion | 2.1179 |
| เคเคชเคเฅ เคฒเคฟเค เคเคพเคธ เคเคซเคฐ! เค เคญเฅ เคเคฐเฅเคฆเฅเค เคเคฐ เคฌเคกเคผเฅ เคฌเคเคค เคเคฐเฅเคใ | hi | suspicious_promotion | 5.47138 |
| CONGRATULATIONS!!! You have won a FREE iPhone. Click now to claim! | en | strong_spam | 5.92358 |
| ยกFELICIDADES! Has ganado un premio. Haz clic aquรญ para reclamarlo YA. | es | strong_spam | 5.3937 |
| URGENT ! Vous avez gagnรฉ 10 000 โฌ. Cliquez immรฉdiatement pour recevoir votre prix ! | fr | strong_spam | 5.50725 |
| Sie haben einen Geldpreis gewonnen! Klicken Sie sofort auf diesen Link! | de | strong_spam | 6.06017 |
| ู ุจุฑูู! ุฑุจุญุช ุฌุงุฆุฒุฉ ููุฏูุฉ ุถุฎู ุฉ. ุงุถุบุท ููุง ูุงุณุชูุงู ูุง ููุฑูุง! | ar | strong_spam | 5.99319 |
| WIN $$$ NOW!!! FREE MONEY GUARANTEED!!! CLICK HERE!!! BUY NOW!!! | en | pure_spam | 6.16083 |
| ยกยกยกGANA DINERO FรCIL!!! ยกยกยกPREMIO GARANTIZADO!!! ยกยกยกHAZ CLIC YA!!! | es | pure_spam | 5.90183 |
| ARGENT GRATUIT !!! GAIN GARANTI !!! CLIQUEZ ICI MAINTENANT !!! | fr | pure_spam | 5.54298 |
| KOSTENLOS GELD VERDIENEN!!! 100% GARANTIERT!!! JETZT KLICKEN!!! | de | pure_spam | 5.97362 |
| ๅ ่ดน้ๅฝฉ้๏ผ๏ผ๏ผไฟ่ฏไธญๅฅ๏ผ๏ผ๏ผ็ซๅณ็นๅป้ขๅ๏ผ๏ผ๏ผ | zh | pure_spam | 5.99609 |
| เคฌเคฟเคจเคพ เคฎเฅเคนเคจเคค เคเฅ เคฒเคพเคเฅเค เคเคฎเคพเคเค!!! เค เคญเฅ เคเฅเคฒเคฟเค เคเคฐเฅเค!!! เคเคจเคพเคฎ เคชเคเฅเคเคพ!!! | hi | pure_spam | 6.00728 |
| ุงุฑุจุญ ุฃู ูุงููุง ู ุฌุงููุฉ ุงูุขู!!! ุฌุงุฆุฒุฉ ู ุถู ููุฉ!!! ุงุถุบุท ููุฑูุง!!! | ar | pure_spam | 6.23987 |
Training Details
Click here for training details
Hyperparameters
Learning Rate: 5e-05
Train Batch Size: 8
Eval Batch Size: 8
Optimizer:
AdamW(Fused)Number of Epochs: 3.0
Mixed Precision: BF16
Gradient Accumulation Steps: 1
Learning Rate Scheduler: Linear
Warmup Steps: 0
Warmup Ratio: 0.0
Weight Decay: 0.0
Max Gradient Norm: 1.0
Number of GPUs: 1
Seed: 42
Training Results & Evaluation Metrics
| Metric | Value |
|---|---|
| Train Loss | 2.8603 |
| Validation Loss | 3.9000 |
| Validation F1 Score | N/A (Regression setup) |
| Total FLOPs | $7.8037 \times 10^{15}$ |
Speed Performance
- Training Runtime: 1,302.34 seconds
- Train Samples per Second: 184.28
- Evaluation Runtime: 10.82 seconds
- Eval Samples per Second: 924.47
| Step | Epoch | Learning Rate | Training Loss | Validation Loss | Validation F1 |
|---|---|---|---|---|---|
| 500 | 0.05 | 4.9168e-05 | 7.3389 | N/A | N/A |
| 1000 | 0.1 | 4.8335e-05 | 6.0772 | N/A | N/A |
| 1500 | 0.15 | 4.7502e-05 | 5.6355 | N/A | N/A |
| 2000 | 0.2 | 4.6668e-05 | 5.4067 | N/A | N/A |
| 2500 | 0.25 | 4.5835e-05 | 5.2273 | N/A | N/A |
| 3000 | 0.3 | 4.5002e-05 | 4.9198 | N/A | N/A |
| 3500 | 0.35 | 4.4168e-05 | 5.2285 | N/A | N/A |
| 4000 | 0.4 | 4.3335e-05 | 5.0132 | N/A | N/A |
| 4500 | 0.45 | 4.2502e-05 | 4.8953 | N/A | N/A |
| 5000 | 0.5 | 4.1668e-05 | 4.5493 | N/A | N/A |
| 5500 | 0.55 | 4.0835e-05 | 4.5720 | N/A | N/A |
| 6000 | 0.6 | 4.0002e-05 | 4.5171 | N/A | N/A |
| 6500 | 0.65 | 3.9168e-05 | 4.6098 | N/A | N/A |
| 7000 | 0.7 | 3.8335e-05 | 4.5045 | N/A | N/A |
| 7500 | 0.75 | 3.7502e-05 | 4.6530 | N/A | N/A |
| 8000 | 0.8 | 3.6668e-05 | 4.2827 | N/A | N/A |
| 8500 | 0.85 | 3.5835e-05 | 4.3984 | N/A | N/A |
| 9000 | 0.9 | 3.5002e-05 | 4.4387 | N/A | N/A |
| 9500 | 0.95 | 3.4168e-05 | 4.1526 | N/A | N/A |
| 10000 | 1.0 | 3.3335e-05 | 4.5002 | 4.2595 | N/A |
| 10500 | 1.05 | 3.2502e-05 | 2.7414 | N/A | N/A |
| 11000 | 1.1 | 3.1668e-05 | 2.7818 | N/A | N/A |
| 11500 | 1.15 | 3.0835e-05 | 2.6185 | N/A | N/A |
| 12000 | 1.2 | 3.0002e-05 | 2.7448 | N/A | N/A |
| 12500 | 1.25 | 2.9168e-05 | 2.4649 | N/A | N/A |
| 13000 | 1.3 | 2.8335e-05 | 2.6484 | N/A | N/A |
| 13500 | 1.35 | 2.7502e-05 | 2.6997 | N/A | N/A |
| 14000 | 1.4 | 2.6668e-05 | 2.4067 | N/A | N/A |
| 14500 | 1.45 | 2.5835e-05 | 2.7081 | N/A | N/A |
| 15000 | 1.5 | 2.5002e-05 | 2.5956 | N/A | N/A |
| 15500 | 1.55 | 2.4168e-05 | 2.5097 | N/A | N/A |
| 16000 | 1.6 | 2.3335e-05 | 2.5163 | N/A | N/A |
| 16500 | 1.65 | 2.2502e-05 | 2.6180 | N/A | N/A |
| 17000 | 1.7 | 2.1668e-05 | 2.5386 | N/A | N/A |
| 17500 | 1.75 | 2.0835e-05 | 2.4471 | N/A | N/A |
| 18000 | 1.8 | 2.0002e-05 | 2.5227 | N/A | N/A |
| 18500 | 1.85 | 1.9168e-05 | 2.4538 | N/A | N/A |
| 19000 | 1.9 | 1.8335e-05 | 2.3036 | N/A | N/A |
| 19500 | 1.95 | 1.7502e-05 | 2.2903 | N/A | N/A |
| 20000 | 2.0 | 1.6668e-05 | 2.5138 | 3.9579 | N/A |
| 20500 | 2.05 | 1.5835e-05 | 1.1239 | N/A | N/A |
| 21000 | 2.1 | 1.5002e-05 | 1.1748 | N/A | N/A |
| 21500 | 2.15 | 1.4168e-05 | 1.0715 | N/A | N/A |
| 22000 | 2.2 | 1.3335e-05 | 1.1432 | N/A | N/A |
| 22500 | 2.25 | 1.2502e-05 | 1.1180 | N/A | N/A |
| 23000 | 2.3 | 1.1668e-05 | 1.1273 | N/A | N/A |
| 23500 | 2.35 | 1.0835e-05 | 1.1292 | N/A | N/A |
| 24000 | 2.4 | 1.0002e-05 | 1.1070 | N/A | N/A |
| 24500 | 2.45 | 9.1683e-06 | 1.0862 | N/A | N/A |
| 25000 | 2.5 | 8.3350e-06 | 1.0994 | N/A | N/A |
| 25500 | 2.55 | 7.5017e-06 | 1.1265 | N/A | N/A |
| 26000 | 2.6 | 6.6683e-06 | 1.0964 | N/A | N/A |
| 26500 | 2.65 | 5.8350e-06 | 1.0535 | N/A | N/A |
| 27000 | 2.7 | 5.0017e-06 | 1.0348 | N/A | N/A |
| 27500 | 2.75 | 4.1683e-06 | 1.0017 | N/A | N/A |
| 28000 | 2.8 | 3.3350e-06 | 1.0187 | N/A | N/A |
| 28500 | 2.85 | 2.5017e-06 | 1.0135 | N/A | N/A |
| 29000 | 2.9 | 1.6683e-06 | 0.9879 | N/A | N/A |
| 29500 | 2.95 | 8.3500e-07 | 1.0514 | N/A | N/A |
| 30000 | 3.0 | 1.6667e-09 | 1.0064 | 3.9000 | N/A |
Framework Versions
- Transformers: 5.17.0
- PyTorch: 2.13.0+cu130
Limitations & Bias
- Inherits biases and distributions present in the training dataset, which primarily focuses on webpage spam and features a strong English-language skew.
- May not generalize across all niche or emerging spam vectors.
- Users should carefully evaluate the downstream impacts of false positives and false negatives for their specific deployment use case and adjust classification thresholds accordingly.
- While ModernBERT offers efficiency gains over standard encoder architectures, transformer-based text scoring can have higher latency compared to lightweight linear/bag-of-words baselines.
License
This model is licensed under the MIT License
- Downloads last month
- 34
Model tree for agentlans/mmBERT-small-spam-detector
Base model
jhu-clsp/mmBERT-smallDataset used to train agentlans/mmBERT-small-spam-detector
Evaluation results
- ROC AUC on agentlans/common-crawl-spamself-reported0.933