gowtham0992 commited on
Commit
73a1a03
·
1 Parent(s): 86b65d8

Add model selection evidence (#2)

Browse files

- Add model selection evidence (dc29f13b39a22e4c646f14632d0069d5138c9ef1)

Files changed (2) hide show
  1. README.md +13 -3
  2. docs/model-selection-chart.svg +34 -0
README.md CHANGED
@@ -154,10 +154,20 @@ Safety architecture:
154
  - The UI always recommends verification through official channels or a known phone number, never the suspicious link or number.
155
  - Session memory is local to the current Gradio session and helps show repeated scam patterns.
156
 
157
- Current eval results:
158
 
159
- - 632-case hard guarded eval, 1B v8: `579/632` risk accuracy (`91.61%`), **`0` dangerous-as-safe**, `0` dangerous-as-needs-check, `0` safe-as-dangerous-or-suspicious, `0` unsafe action violations, `0` invalid predictions, `0` model errors.
160
- - Earlier comparison evidence, 1B v4: 394-case hard guarded eval at `379/394` risk accuracy (`96.19%`) and 320-case hard guarded eval at `310/320` risk accuracy (`96.88%`), both with no dangerous undercalls.
 
 
 
 
 
 
 
 
 
 
161
 
162
  Training/eval artifacts:
163
 
 
154
  - The UI always recommends verification through official channels or a known phone number, never the suspicious link or number.
155
  - Session memory is local to the current Gradio session and helps show repeated scam patterns.
156
 
157
+ ## Model Selection Evidence
158
 
159
+ ![Risk accuracy comparison for Jawbreaker model candidates](docs/model-selection-chart.svg)
160
+
161
+ | Candidate | Size | Eval set | Risk accuracy | Dangerous -> safe | Dangerous -> needs check | Safe -> dangerous/suspicious | Invalid JSON | Unsafe actions | Why not final |
162
+ | --- | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
163
+ | Heuristic guard | none | 215 hard cases | 84.7% | 0 | 0 | 0 | 0 | 0 | Guard layer only, not model-led. |
164
+ | MiniCPM4.1 LoRA v3 | 8B | 215 hard cases | 97.7% | 0 | 0 | 0 | 0 | 0 | Strong, but not Tiny Titan. |
165
+ | MiniCPM5 LoRA v4 | 1B | 394 hard cases | 96.2% | 0 | 0 | 3 | 0 | 0 | Strong, narrower eval and a few safe-message overcalls. |
166
+ | MiniCPM5 LoRA v8 | 1B | 632 hard cases | 91.6% | 0 | 0 | 0 | 0 | 0 | **Final: broadest safety gate.** |
167
+
168
+ Jawbreaker ships the 1B v8 adapter not because it has the prettiest accuracy number, but because it cleared the broadest completed hard safety gate with zero dangerous undercalls, zero safe-message overcalls, zero unsafe actions, zero invalid JSON, and zero model errors.
169
+
170
+ `Qwen/Qwen3-0.6B` was an early ZeroGPU runtime prototype and remains a documented fallback path, but it is not included in the numeric comparison because the final committed reports are for the heuristic guard and MiniCPM LoRA candidates above. The judged path is MiniCPM5-1B + Jawbreaker LoRA v8.
171
 
172
  Training/eval artifacts:
173
 
docs/model-selection-chart.svg ADDED