Bulwark scam detector v1

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,147 @@
+---
+license: apache-2.0
+base_model: Horizon-Labs/phishing-scam-detector-base
+language:
+  - tr
+  - en
+  - multilingual
+pipeline_tag: text-classification
+tags:
+  - scam-detection
+  - phishing
+  - smishing
+  - fraud
+  - spam
+  - turkish
+  - modernbert
+  - mmbert
+datasets:
+  - SavienWalker/bulwark-bench-tr
+metrics:
+  - accuracy
+---
+
+# Bulwark Scam Detector
+
+A multilingual scam / spam / legitimate message classifier, **specialised for Turkish**.
+Fine-tuned from [Horizon-Labs/phishing-scam-detector-base](https://huggingface.co/Horizon-Labs/phishing-scam-detector-base)
+(mmBERT-base, 307M parameters) to fix one specific weakness: **false alarms on real Turkish institutional
+messages** (bank, cargo, e-Devlet, operator, OTP notifications).
+
+> 🇹🇷 **Türkçe özet:** Bulwark; SMS, e-posta ve kısa mesajları *dolandırıcılık / spam / temiz* olarak ayırır.
+> Gerçek Türk kurum mesajlarında mevcut modelin temiz mesajların yarısına verdiği yanlış alarmı %14'e indirir,
+> istenmeyen mesajları yakalama oranı ise %98'de kalır. Diğer dillerdeki temel yeteneği korunmuştur.
+
+## Headline result
+
+On a held-out half of **real Turkish SMS from a personal inbox** (opened once, after all model choices were made):
+
+| | Base model (threshold 0.5) | **Bulwark (threshold 0.6)** |
+|---|---|---|
+| False alarm on legitimate messages ↓ | 53% (54/102) | **14% (14/102)** |
+| Legitimate message labelled *fraud* ↓ | 26% | **9%** |
+| Unwanted (spam + fraud) caught ↑ | 100% (52/52) | 98% (51/52) |
+| 3-class accuracy ↑ | 64% | **87%** |
+
+The base model flags roughly every second real bank / cargo / e-Devlet notification as unwanted because it
+was trained on synthetic scams plus generic web text, and has rarely seen what a *real* institutional message
+looks like. Bulwark keeps the base model's recall while learning the difference.
+
+## Labels and usage
+
+| id | label | meaning |
+|---|---|---|
+| 0 | `legitimate` | genuine personal or institutional message |
+| 1 | `spam` | advertising from a real business |
+| 2 | `fraud` | tries to steal money, card data, passwords, codes, identity or trust |
+
+**Recommended decision rule:** treat a message as unwanted when `p(spam) + p(fraud) > 0.6`
+(chosen on the development split; stored in `bulwark_config.json`).
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+model_id = "SavienWalker/bulwark-scam-detector"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
+
+text = "PTT: Gonderiniz adres eksikligi nedeniyle bekletilmektedir. 0,99 TL odeme yapin: ptt-gonderitakip.com"
+with torch.no_grad():
+    p = torch.softmax(model(**tok(text, return_tensors="pt", truncation=True, max_length=256)).logits, -1)[0]
+probs = {model.config.id2label[i]: round(v.item(), 3) for i, v in enumerate(p)}
+unwanted = probs["spam"] + probs["fraud"] > 0.6
+print(probs, "unwanted" if unwanted else "ok")
+```
+
+## Evaluation
+
+All numbers compare the base model and Bulwark on the same messages. Sets are described in
+[bulwark-bench-tr](https://huggingface.co/datasets/SavienWalker/bulwark-bench-tr).
+Messages marked `ambiguous` (mostly illegal-betting ads) are excluded from metrics.
+
+| Set | Type | Metric | Base | Bulwark |
+|---|---|---|---|---|
+| Real SMS, test half (212) | real, held out | false alarm ↓ | 53% | **14%** |
+| | | unwanted caught ↑ | 100% | 98% |
+| Real e-mail (40) | real, **development** set | false alarm ↓ | 33% | **19%** |
+| Hard set v2 (60) | synthetic, human-reviewed | fraud caught ↑ | 82% | **95%** |
+| | | false alarm ↓ | 25% | **0%** |
+| | | contrast pairs fully correct ↑ | 9/13 | **12/13** |
+| Hard set v1 (42) | synthetic, human-reviewed | contrast pairs fully correct ↑ | 5/7 | **7/7** |
+| UCI SMS Spam, held-out half (2,787) | real, English | unwanted caught ↑ | 87% | 96% |
+| Multilingual probe (54; de, fr, es, pt, ru, ar) | synthetic, easy | 3-class accuracy | 54/54 | 54/54 |
+
+*Contrast pairs* are a fake message and the genuine message it imitates (e.g. fake vs. real HGS toll notice);
+a pair counts only if **both** are classified correctly — the clearest test of whether a model reads the
+signals (domain, request, tone) rather than the topic.
+
+How to read these numbers honestly:
+- **The only fully clean real-world number is the SMS test half.** The e-mail set was used for error analysis,
+  so it is reported as development data.
+- **Hard sets and training templates were written by the same author (an LLM, Claude).** A style overlap
+  could inflate those scores. They are reported separately and labelled synthetic.
+- **The SMS test half contains no non-ambiguous real fraud** — real fraud in a personal inbox is almost all
+  illegal betting. Fraud recall is therefore measured on the (synthetic) hard sets only.
+- **English:** half of the UCI SMS Spam Collection was used as replay data during training, so the English
+  gain is in-distribution; the honest claim is "no regression", not "better than the base model".
+- **Other languages:** a small, easy probe shows no regression. Hard cases outside Turkish were not measured.
+
+## Training
+
+- **Base:** Horizon-Labs/phishing-scam-detector-base (Apache-2.0), itself built on jhu-clsp/mmBERT-base (MIT).
+- **Data (~9k messages):**
+  - ~8k Turkish synthetic messages from 190 hand-written templates with slot filling (institution, amount,
+    date, code, look-alike domain). Covers real institutional notifications, real-business ads, link-based
+    phishing, and link-free social engineering. Patterns were drawn from public warnings (KGM, SGK, PTT, EGM)
+    and public complaint narratives; no complaint text was copied.
+  - ~1k English messages from the UCI SMS Spam Collection (CC BY 4.0) as replay against forgetting; spam vs.
+    fraud for these was taken from the base model's own prediction (distillation).
+- **Leakage control:** every training message was compared with every evaluation message (token-set
+  Jaccard); anything ≥ 0.5 was removed.
+- **Personalisation balancing:** scammers now use leaked personal data (name, card last 4 digits, plate,
+  old address). These details were injected into legitimate *and* fraudulent training messages at equal
+  rates, so that "the message knows my name" is **not** learned as a sign of legitimacy.
+- **Recipe:** word embeddings frozen (110M of 307M parameters trained), bf16, AdamW lr 2e-5, 2 epochs,
+  max length 256, class-weighted loss, fraud examples down-sampled to the legitimate count. Runs on a single
+  6 GB laptop GPU (RTX 4050) in about 6 minutes.
+- **Model selection:** on a development half of the real SMS, split by sender so the same institution never
+  appears in both halves.
+
+## Scope and limitations
+
+- **One message at a time.** Many real scams unfold over a phone call, a long WhatsApp conversation, a fake
+  website or an ad. Bulwark sees only the text it is given; the first message of a slow scam often looks
+  harmless.
+- **A message containing your real personal details is not proof that it is genuine.** Leaked databases make
+  this trivial for scammers.
+- Illegal-betting ads are labelled `fraud` but treated as ambiguous in evaluation.
+- Small real evaluation sets (hundreds, not thousands). Treat the numbers as indicative.
+- Not a replacement for a bank's or carrier's own fraud systems. Do not use it to make decisions about
+  people.
+
+## License and attribution
+
+Apache-2.0. Derived from [Horizon-Labs/phishing-scam-detector-base](https://huggingface.co/Horizon-Labs/phishing-scam-detector-base)
+(Apache-2.0) and [jhu-clsp/mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) (MIT).
+English replay data: Almeida, T.A. & Gómez Hidalgo, J.M., *SMS Spam Collection* (UCI, CC BY 4.0).

bulwark_config.json

@@ -0,0 +1,6 @@
+{
+  "recommended_threshold": 0.6,
+  "threshold_applies_to": "p_spam + p_fraud",
+  "base_model": "Horizon-Labs/phishing-scam-detector-base",
+  "checkpoint": "checkpoints/v5"
+}

config.json

@@ -0,0 +1,88 @@
+{
+  "architectures": [
+    "ModernBertForSequenceClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 1,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "id2label": {
+    "0": "legitimate",
+    "1": "spam",
+    "2": "fraud"
+  },
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "label2id": {
+    "fraud": 2,
+    "legitimate": 0,
+    "spam": 1
+  },
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "mask_token_id": 4,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 0,
+  "position_embedding_type": "sans_pos",
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 1,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.21.0",
+  "vocab_size": 256000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60bf6102bb35c62941b01eac2bc4d367d5395a81e6fdc13c8bfb793d92197351
+size 1230144500

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a5069feab370ece9f452dcda41c0ba2c216eebf6cb49b06893698f53cdc2b0b
+size 34363441

tokenizer_config.json

@@ -0,0 +1,29 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<bos>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "max_length": 384,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "sep_token": "<eos>",
+  "spaces_between_special_tokens": false,
+  "stride": 0,
+  "tokenizer_class": "TokenizersBackend",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "<unk>"
+}