BSIS Scam Classifier (XLM-RoBERTa base)

A scam message classifier for Indian SMS and WhatsApp traffic, covering 23 Indian languages in native scripts and Romanized code-mix. Fine-tuned from FacebookAI/xlm-roberta-base on the harshcasper/bsis dataset: 32,858 training windows of synthetic red-team conversations, each labelled by deterministic ground truth.

Trained on Tesla T4 in fp16 mixed precision, 3 epochs, best epoch selected by validation F1.

Labels: SAFE (0), SCAM (1).

Metrics

Tuned decision threshold from the validation split: 0.45 (best F1 0.8548). Lowest threshold with FPR <= 0.05: None.

BSIS test split (1883 windows)

threshold accuracy precision recall F1 FPR ROC-AUC
0.50 0.8147 0.8468 0.8861 0.866 0.3344 0.868
0.45 (tuned) 0.8173 0.8408 0.9002 0.8695 0.3557 0.868

Script modes

script n positives F1
native_script 1239 854 0.8671
romanized_codemix 644 419 0.8739

Hard examples

group n positives F1
false 1372 1021 0.8812
true 511 252 0.8233

Per language

language n positives F1
as-IN 89 72 0.9211
bn-IN 153 97 0.901
brx-IN 61 45 0.7848
doi-IN 58 41 0.8493
en-IN 118 80 0.8953
gu-IN 73 56 0.8908
hi-IN 224 135 0.8483
kn-IN 68 53 0.9174
kok-IN 60 45 0.8434
ks-IN 51 34 0.9412
mai-IN 56 38 0.9211
ml-IN 72 40 0.8211
mni-IN 50 33 0.8667
mr-IN 101 60 0.8406
ne-IN 51 38 0.9474
od-IN 25 23 0.9583
pa-IN 67 46 0.9
sa-IN 45 36 0.8254
sat-IN 59 45 0.7848
sd-IN 63 45 0.8642
ta-IN 128 79 0.8065
te-IN 142 89 0.8673
ur-IN 69 43 0.8571

Per vector

vector n positives F1
digital_arrest 270 235 0.8712
fake_courier_customs 114 93 0.9
fake_customer_care 232 198 0.8741
kyc_utility_panic 166 134 0.9078
legit_bank_alert 66 0 None
legit_delivery_otp 63 0 None
legit_electricity_reminder 73 0 None
legit_family_transfer 107 0 None
predatory_loan_app 94 74 0.8727
qr_code_refund_trap 145 117 0.8968
social_friend_emergency 105 68 0.8322
upi_collect_trap 197 149 0.891
whatsapp_stock_task 251 205 0.8072

Real-world held-out set (500 anonymized SMS)

Trained never on this set. English and Indian English messages only.

accuracy precision recall F1 FPR ROC-AUC
0.67 0.5517 0.4444 0.4923 0.2031 0.5582

Use

from transformers import pipeline

classifier = pipeline(
    "text-classification", model="harshcasper/bsis-xlmr-scam-classifier"
)
classifier("Bhai, naya number hai mera, airport phans gaya hu, 5000 bhej de")
# [{"label": "SCAM", "score": 0.98}]

Caveats

  • Training data is synthetic, generated by the BSIS red-team arena on Sarvam Token Factory. Style and phrasing are realistic; no real user messages were used.
  • Threshold choice depends on deployment. On-device warnings favour the low-FPR threshold (None); balanced detection favours 0.45.
  • The real-world eval covers English and Indian English only. Scores on other real-world languages will differ from the synthetic test numbers.
  • The classifier scores a message window, not a person. It is decision support, not a verdict.

License

Research and educational use in scam detection and cyber-safety training. Provided as is, with no warranty. Do not use it as the only signal to block, ban or accuse anyone.

Downloads last month
-
Safetensors
Model size
0.3B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for harshcasper/bsis-xlmr-scam-classifier

Finetuned
(4,325)
this model

Dataset used to train harshcasper/bsis-xlmr-scam-classifier