Commit ·
73a1a03
1
Parent(s): 86b65d8
Add model selection evidence (#2)
Browse files- Add model selection evidence (dc29f13b39a22e4c646f14632d0069d5138c9ef1)
- README.md +13 -3
- docs/model-selection-chart.svg +34 -0
README.md
CHANGED
|
@@ -154,10 +154,20 @@ Safety architecture:
|
|
| 154 |
- The UI always recommends verification through official channels or a known phone number, never the suspicious link or number.
|
| 155 |
- Session memory is local to the current Gradio session and helps show repeated scam patterns.
|
| 156 |
|
| 157 |
-
|
| 158 |
|
| 159 |
-
|
| 160 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 161 |
|
| 162 |
Training/eval artifacts:
|
| 163 |
|
|
|
|
| 154 |
- The UI always recommends verification through official channels or a known phone number, never the suspicious link or number.
|
| 155 |
- Session memory is local to the current Gradio session and helps show repeated scam patterns.
|
| 156 |
|
| 157 |
+
## Model Selection Evidence
|
| 158 |
|
| 159 |
+

|
| 160 |
+
|
| 161 |
+
| Candidate | Size | Eval set | Risk accuracy | Dangerous -> safe | Dangerous -> needs check | Safe -> dangerous/suspicious | Invalid JSON | Unsafe actions | Why not final |
|
| 162 |
+
| --- | ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
|
| 163 |
+
| Heuristic guard | none | 215 hard cases | 84.7% | 0 | 0 | 0 | 0 | 0 | Guard layer only, not model-led. |
|
| 164 |
+
| MiniCPM4.1 LoRA v3 | 8B | 215 hard cases | 97.7% | 0 | 0 | 0 | 0 | 0 | Strong, but not Tiny Titan. |
|
| 165 |
+
| MiniCPM5 LoRA v4 | 1B | 394 hard cases | 96.2% | 0 | 0 | 3 | 0 | 0 | Strong, narrower eval and a few safe-message overcalls. |
|
| 166 |
+
| MiniCPM5 LoRA v8 | 1B | 632 hard cases | 91.6% | 0 | 0 | 0 | 0 | 0 | **Final: broadest safety gate.** |
|
| 167 |
+
|
| 168 |
+
Jawbreaker ships the 1B v8 adapter not because it has the prettiest accuracy number, but because it cleared the broadest completed hard safety gate with zero dangerous undercalls, zero safe-message overcalls, zero unsafe actions, zero invalid JSON, and zero model errors.
|
| 169 |
+
|
| 170 |
+
`Qwen/Qwen3-0.6B` was an early ZeroGPU runtime prototype and remains a documented fallback path, but it is not included in the numeric comparison because the final committed reports are for the heuristic guard and MiniCPM LoRA candidates above. The judged path is MiniCPM5-1B + Jawbreaker LoRA v8.
|
| 171 |
|
| 172 |
Training/eval artifacts:
|
| 173 |
|
docs/model-selection-chart.svg
ADDED
|
|