Instructions to use harshcasper/bsis-xlmr-scam-classifier with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use harshcasper/bsis-xlmr-scam-classifier with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="harshcasper/bsis-xlmr-scam-classifier")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("harshcasper/bsis-xlmr-scam-classifier") model = AutoModelForSequenceClassification.from_pretrained("harshcasper/bsis-xlmr-scam-classifier", device_map="auto") - Notebooks
- Google Colab
- Kaggle
BSIS Scam Classifier (XLM-RoBERTa base)
A scam message classifier for Indian SMS and WhatsApp traffic, covering 23
Indian languages in native scripts and Romanized code-mix. Fine-tuned from
FacebookAI/xlm-roberta-base on the harshcasper/bsis dataset: 32,858
training windows of synthetic red-team conversations, each labelled by
deterministic ground truth.
Trained on Tesla T4 in fp16 mixed precision, 3 epochs, best epoch selected by validation F1.
Labels: SAFE (0), SCAM (1).
Metrics
Tuned decision threshold from the validation split: 0.45 (best F1 0.8548). Lowest threshold with FPR <= 0.05: None.
BSIS test split (1883 windows)
| threshold | accuracy | precision | recall | F1 | FPR | ROC-AUC |
|---|---|---|---|---|---|---|
| 0.50 | 0.8147 | 0.8468 | 0.8861 | 0.866 | 0.3344 | 0.868 |
| 0.45 (tuned) | 0.8173 | 0.8408 | 0.9002 | 0.8695 | 0.3557 | 0.868 |
Script modes
| script | n | positives | F1 |
|---|---|---|---|
| native_script | 1239 | 854 | 0.8671 |
| romanized_codemix | 644 | 419 | 0.8739 |
Hard examples
| group | n | positives | F1 |
|---|---|---|---|
| false | 1372 | 1021 | 0.8812 |
| true | 511 | 252 | 0.8233 |
Per language
| language | n | positives | F1 |
|---|---|---|---|
| as-IN | 89 | 72 | 0.9211 |
| bn-IN | 153 | 97 | 0.901 |
| brx-IN | 61 | 45 | 0.7848 |
| doi-IN | 58 | 41 | 0.8493 |
| en-IN | 118 | 80 | 0.8953 |
| gu-IN | 73 | 56 | 0.8908 |
| hi-IN | 224 | 135 | 0.8483 |
| kn-IN | 68 | 53 | 0.9174 |
| kok-IN | 60 | 45 | 0.8434 |
| ks-IN | 51 | 34 | 0.9412 |
| mai-IN | 56 | 38 | 0.9211 |
| ml-IN | 72 | 40 | 0.8211 |
| mni-IN | 50 | 33 | 0.8667 |
| mr-IN | 101 | 60 | 0.8406 |
| ne-IN | 51 | 38 | 0.9474 |
| od-IN | 25 | 23 | 0.9583 |
| pa-IN | 67 | 46 | 0.9 |
| sa-IN | 45 | 36 | 0.8254 |
| sat-IN | 59 | 45 | 0.7848 |
| sd-IN | 63 | 45 | 0.8642 |
| ta-IN | 128 | 79 | 0.8065 |
| te-IN | 142 | 89 | 0.8673 |
| ur-IN | 69 | 43 | 0.8571 |
Per vector
| vector | n | positives | F1 |
|---|---|---|---|
| digital_arrest | 270 | 235 | 0.8712 |
| fake_courier_customs | 114 | 93 | 0.9 |
| fake_customer_care | 232 | 198 | 0.8741 |
| kyc_utility_panic | 166 | 134 | 0.9078 |
| legit_bank_alert | 66 | 0 | None |
| legit_delivery_otp | 63 | 0 | None |
| legit_electricity_reminder | 73 | 0 | None |
| legit_family_transfer | 107 | 0 | None |
| predatory_loan_app | 94 | 74 | 0.8727 |
| qr_code_refund_trap | 145 | 117 | 0.8968 |
| social_friend_emergency | 105 | 68 | 0.8322 |
| upi_collect_trap | 197 | 149 | 0.891 |
| whatsapp_stock_task | 251 | 205 | 0.8072 |
Real-world held-out set (500 anonymized SMS)
Trained never on this set. English and Indian English messages only.
| accuracy | precision | recall | F1 | FPR | ROC-AUC |
|---|---|---|---|---|---|
| 0.67 | 0.5517 | 0.4444 | 0.4923 | 0.2031 | 0.5582 |
Use
from transformers import pipeline
classifier = pipeline(
"text-classification", model="harshcasper/bsis-xlmr-scam-classifier"
)
classifier("Bhai, naya number hai mera, airport phans gaya hu, 5000 bhej de")
# [{"label": "SCAM", "score": 0.98}]
Caveats
- Training data is synthetic, generated by the BSIS red-team arena on Sarvam Token Factory. Style and phrasing are realistic; no real user messages were used.
- Threshold choice depends on deployment. On-device warnings favour the low-FPR threshold (None); balanced detection favours 0.45.
- The real-world eval covers English and Indian English only. Scores on other real-world languages will differ from the synthetic test numbers.
- The classifier scores a message window, not a person. It is decision support, not a verdict.
License
Research and educational use in scam detection and cyber-safety training. Provided as is, with no warranty. Do not use it as the only signal to block, ban or accuse anyone.
- Downloads last month
- -
Model tree for harshcasper/bsis-xlmr-scam-classifier
Base model
FacebookAI/xlm-roberta-base