gowtham0992 commited on
Commit
762a84b
·
verified ·
1 Parent(s): b9088a7

Add v8 model card

Browse files
Files changed (1) hide show
  1. MODEL_CARD_MINICPM5_LORA_V8.md +90 -0
MODEL_CARD_MINICPM5_LORA_V8.md ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ base_model: openbmb/MiniCPM5-1B
4
+ library_name: peft
5
+ tags:
6
+ - build-small-hackathon
7
+ - openbmb
8
+ - minicpm
9
+ - minicpm5
10
+ - peft
11
+ - lora
12
+ - scam-defense
13
+ - tiny-titan
14
+ - well-tuned
15
+ datasets:
16
+ - build-small-hackathon/jawbreaker-scam-defense-data
17
+ ---
18
+
19
+ # Jawbreaker MiniCPM5-1B LoRA v8
20
+
21
+ Jawbreaker MiniCPM5-1B LoRA v8 is the final small-model adapter used by the Jawbreaker Hugging Face Space.
22
+
23
+ - Base model: `openbmb/MiniCPM5-1B`
24
+ - Adapter: `build-small-hackathon/jawbreaker-minicpm5-1b-lora-v8`
25
+ - App: https://huggingface.co/spaces/build-small-hackathon/jawbreaker
26
+ - Dataset/eval bundle: https://huggingface.co/datasets/build-small-hackathon/jawbreaker-scam-defense-data
27
+ - GitHub: https://github.com/gowtham0992/jawbreaker
28
+
29
+ ## Task
30
+
31
+ The model converts one suspicious text, email, or DM into strict JSON for a scam-safety card:
32
+
33
+ - risk level: `safe`, `needs_check`, `suspicious`, or `dangerous`
34
+ - scam type
35
+ - short summary
36
+ - safest next action
37
+ - warning tactics
38
+ - scam DNA fields for what the sender pretends to be, how they apply pressure, what they ask for, and what could happen
39
+
40
+ Jawbreaker is intentionally narrow. It is not a general chatbot. The goal is to help a non-expert pause before clicking, replying, sharing a code, or sending money.
41
+
42
+ ## Training
43
+
44
+ The adapter was trained with PEFT/LoRA on Modal A100 using synthetic and sanitized scam-defense examples generated in the public repository.
45
+
46
+ The v8 pass is failure-driven calibration. It targeted two gaps found during earlier v7/fresh-pattern evals:
47
+
48
+ - wrong-number crypto / gold / trading grooming that could be under-called
49
+ - ordinary family, school, pharmacy, and logistics messages that should not be over-called as dangerous
50
+
51
+ Key training/eval files:
52
+
53
+ - `training/generate_v8_data.py`
54
+ - `training/data/train_v8.jsonl`
55
+ - `training/data/dev_v8.jsonl`
56
+ - `training/data/test_v8.jsonl`
57
+ - `eval/hard_v8_eval.jsonl`
58
+ - `eval/reports/jawbreaker-minicpm5-1b-lora-v8-hard632-safetyguard-v4.json`
59
+
60
+ ## Final Eval
61
+
62
+ Final guarded Modal A100 eval on `eval/hard_v8_eval.jsonl`:
63
+
64
+ | Metric | Result |
65
+ | --- | ---: |
66
+ | Cases | 632 |
67
+ | Risk accuracy | 579/632, 91.61% |
68
+ | Scam type accuracy | 561/632, 88.77% |
69
+ | Mean tactic recall | 90.69% |
70
+ | Dangerous as safe | 0 |
71
+ | Dangerous as needs_check | 0 |
72
+ | Safe as dangerous or suspicious | 0 |
73
+ | Unsafe action violations | 0 |
74
+ | Invalid predictions | 0 |
75
+ | Model errors | 0 |
76
+
77
+ The final report is published in the dataset/eval bundle and in the app repository.
78
+
79
+ ## Runtime Safety
80
+
81
+ The live app validates model output against a strict schema before rendering. It also applies a deterministic safety guard for obvious high-risk patterns, so a weak small-model response does not render an obvious scam as safe.
82
+
83
+ If the model fails, returns malformed JSON, or under-calls an obvious danger signal, Jawbreaker falls back to deterministic safety analysis and recommends verification through trusted official channels.
84
+
85
+ ## Limitations
86
+
87
+ - This is a hackathon prototype, not legal, financial, or cybersecurity advice.
88
+ - Training data is synthetic/sanitized, not a proprietary corpus of private user messages.
89
+ - The model is optimized for short scam-like messages and may not generalize to long documents.
90
+ - The safest action should still be verified by the user through official channels or a trusted person.