Add v8 model card
Browse files
MODEL_CARD_MINICPM5_LORA_V8.md
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
base_model: openbmb/MiniCPM5-1B
|
| 4 |
+
library_name: peft
|
| 5 |
+
tags:
|
| 6 |
+
- build-small-hackathon
|
| 7 |
+
- openbmb
|
| 8 |
+
- minicpm
|
| 9 |
+
- minicpm5
|
| 10 |
+
- peft
|
| 11 |
+
- lora
|
| 12 |
+
- scam-defense
|
| 13 |
+
- tiny-titan
|
| 14 |
+
- well-tuned
|
| 15 |
+
datasets:
|
| 16 |
+
- build-small-hackathon/jawbreaker-scam-defense-data
|
| 17 |
+
---
|
| 18 |
+
|
| 19 |
+
# Jawbreaker MiniCPM5-1B LoRA v8
|
| 20 |
+
|
| 21 |
+
Jawbreaker MiniCPM5-1B LoRA v8 is the final small-model adapter used by the Jawbreaker Hugging Face Space.
|
| 22 |
+
|
| 23 |
+
- Base model: `openbmb/MiniCPM5-1B`
|
| 24 |
+
- Adapter: `build-small-hackathon/jawbreaker-minicpm5-1b-lora-v8`
|
| 25 |
+
- App: https://huggingface.co/spaces/build-small-hackathon/jawbreaker
|
| 26 |
+
- Dataset/eval bundle: https://huggingface.co/datasets/build-small-hackathon/jawbreaker-scam-defense-data
|
| 27 |
+
- GitHub: https://github.com/gowtham0992/jawbreaker
|
| 28 |
+
|
| 29 |
+
## Task
|
| 30 |
+
|
| 31 |
+
The model converts one suspicious text, email, or DM into strict JSON for a scam-safety card:
|
| 32 |
+
|
| 33 |
+
- risk level: `safe`, `needs_check`, `suspicious`, or `dangerous`
|
| 34 |
+
- scam type
|
| 35 |
+
- short summary
|
| 36 |
+
- safest next action
|
| 37 |
+
- warning tactics
|
| 38 |
+
- scam DNA fields for what the sender pretends to be, how they apply pressure, what they ask for, and what could happen
|
| 39 |
+
|
| 40 |
+
Jawbreaker is intentionally narrow. It is not a general chatbot. The goal is to help a non-expert pause before clicking, replying, sharing a code, or sending money.
|
| 41 |
+
|
| 42 |
+
## Training
|
| 43 |
+
|
| 44 |
+
The adapter was trained with PEFT/LoRA on Modal A100 using synthetic and sanitized scam-defense examples generated in the public repository.
|
| 45 |
+
|
| 46 |
+
The v8 pass is failure-driven calibration. It targeted two gaps found during earlier v7/fresh-pattern evals:
|
| 47 |
+
|
| 48 |
+
- wrong-number crypto / gold / trading grooming that could be under-called
|
| 49 |
+
- ordinary family, school, pharmacy, and logistics messages that should not be over-called as dangerous
|
| 50 |
+
|
| 51 |
+
Key training/eval files:
|
| 52 |
+
|
| 53 |
+
- `training/generate_v8_data.py`
|
| 54 |
+
- `training/data/train_v8.jsonl`
|
| 55 |
+
- `training/data/dev_v8.jsonl`
|
| 56 |
+
- `training/data/test_v8.jsonl`
|
| 57 |
+
- `eval/hard_v8_eval.jsonl`
|
| 58 |
+
- `eval/reports/jawbreaker-minicpm5-1b-lora-v8-hard632-safetyguard-v4.json`
|
| 59 |
+
|
| 60 |
+
## Final Eval
|
| 61 |
+
|
| 62 |
+
Final guarded Modal A100 eval on `eval/hard_v8_eval.jsonl`:
|
| 63 |
+
|
| 64 |
+
| Metric | Result |
|
| 65 |
+
| --- | ---: |
|
| 66 |
+
| Cases | 632 |
|
| 67 |
+
| Risk accuracy | 579/632, 91.61% |
|
| 68 |
+
| Scam type accuracy | 561/632, 88.77% |
|
| 69 |
+
| Mean tactic recall | 90.69% |
|
| 70 |
+
| Dangerous as safe | 0 |
|
| 71 |
+
| Dangerous as needs_check | 0 |
|
| 72 |
+
| Safe as dangerous or suspicious | 0 |
|
| 73 |
+
| Unsafe action violations | 0 |
|
| 74 |
+
| Invalid predictions | 0 |
|
| 75 |
+
| Model errors | 0 |
|
| 76 |
+
|
| 77 |
+
The final report is published in the dataset/eval bundle and in the app repository.
|
| 78 |
+
|
| 79 |
+
## Runtime Safety
|
| 80 |
+
|
| 81 |
+
The live app validates model output against a strict schema before rendering. It also applies a deterministic safety guard for obvious high-risk patterns, so a weak small-model response does not render an obvious scam as safe.
|
| 82 |
+
|
| 83 |
+
If the model fails, returns malformed JSON, or under-calls an obvious danger signal, Jawbreaker falls back to deterministic safety analysis and recommends verification through trusted official channels.
|
| 84 |
+
|
| 85 |
+
## Limitations
|
| 86 |
+
|
| 87 |
+
- This is a hackathon prototype, not legal, financial, or cybersecurity advice.
|
| 88 |
+
- Training data is synthetic/sanitized, not a proprietary corpus of private user messages.
|
| 89 |
+
- The model is optimized for short scam-like messages and may not generalize to long documents.
|
| 90 |
+
- The safest action should still be verified by the user through official channels or a trusted person.
|