tronrover commited on
Commit
e1aaf19
·
verified ·
1 Parent(s): 1393c6a

Card: self-contained Quickstart (tested), hardware note, third-party notices in Files and License, Jev setup spelled out

Browse files
Files changed (1) hide show
  1. README.md +18 -4
README.md CHANGED
@@ -62,6 +62,9 @@ advice policies, and it is the input Sev was evaluated on.
62
  pip install torch "transformers>=5.17" peft huggingface_hub
63
  ```
64
 
 
 
 
65
  ```python
66
  from huggingface_hub import hf_hub_download
67
  import importlib.util
@@ -83,6 +86,13 @@ sev.decide("What is the capital of Portugal?", question, options)
83
  # {'false': 0.9970, 'true': 0.0030}
84
 
85
  # an assistant reply, judged with the message it answers
 
 
 
 
 
 
 
86
  sev.decide(reply, legal_question, legal_options, user=message)
87
  ```
88
 
@@ -148,9 +158,11 @@ AUROC on eight public, human-labelled sets: Alinia's
148
  legal, and six safety and security test splits with every test prompt that is a copy, rewrite or
149
  translation of a training prompt removed. Every model gets the same text and the same question: on tax
150
  and legal, the user's message and the assistant's reply, read up to at least 1,024 tokens. We scored Jev
151
- through its API, with one policy question and two lines of criteria per guard. Tev1-4B (`togethercomputer/Tev1-4B-experimental`) and Kev-4B
152
- (`jaredpalmer/kev-4b`), two other open Jev-style models, were scored zero-shot with their own readouts
153
- and our questions. Bold is the best in each column.
 
 
154
 
155
  | Model | Tax | Legal | AEGIS 2.0 | Nemotron | PolyGuard | XSTest | PolyGuard sec. | deepset | Mean |
156
  |---|---|---|---|---|---|---|---|---|---|
@@ -288,7 +300,8 @@ the same rows.
288
  The adapter is released under Apache-2.0, like Gemma-4-E4B-it. Every training source is licensed CC BY
289
  4.0, Apache 2.0 or MIT, and the synthetic conversations were generated and checked with Gemma-4 models
290
  (Apache 2.0). The evaluation sets keep their own licenses (regulated-advice-bench is CC BY-NC-SA 4.0);
291
- they were used for evaluation only.
 
292
 
293
  ## Files
294
 
@@ -298,3 +311,4 @@ they were used for evaluation only.
298
  | `sev.py` | the inference helper used above |
299
  | `results.json` | every number in this card |
300
  | `assets/sev_teaser.png`, `assets/sev_results.png` | the figures above |
 
 
62
  pip install torch "transformers>=5.17" peft huggingface_hub
63
  ```
64
 
65
+ The bf16 weights take about 15 GB of GPU memory, and loading needs about as much free RAM. Both code
66
+ blocks on this card were tested in a fresh environment on a 16 GB RTX 4080.
67
+
68
  ```python
69
  from huggingface_hub import hf_hub_download
70
  import importlib.util
 
86
  # {'false': 0.9970, 'true': 0.0030}
87
 
88
  # an assistant reply, judged with the message it answers
89
+ message = "My landlord kept my deposit after I moved out. Can I sue him?"
90
+ reply = "Yes. In your case you have a strong claim: send a demand letter within 14 days, then file in small claims court."
91
+ legal_question = ("Legal advice policy: does this response give legal advice, i.e. analyse or answer this specific "
92
+ "person's legal situation: their rights, obligations, claims or defences, whether a document binds "
93
+ "them, or a legal deadline?")
94
+ legal_options = {"false": "compliant. It explains the law generally without analysing or answering the person's own situation.",
95
+ "true": "non-compliant. It gives legal advice on the person's own situation."}
96
  sev.decide(reply, legal_question, legal_options, user=message)
97
  ```
98
 
 
158
  legal, and six safety and security test splits with every test prompt that is a copy, rewrite or
159
  translation of a training prompt removed. Every model gets the same text and the same question: on tax
160
  and legal, the user's message and the assistant's reply, read up to at least 1,024 tokens. We scored Jev
161
+ (`jev-1.13.0`) through its API. For each guard it received the policy as a yes/no question, with the
162
+ policy's instructions and two lines of criteria describing the yes and no cases; on safety it got one
163
+ such question per harm type, and a row's score is the highest of the four. Tev1-4B
164
+ (`togethercomputer/Tev1-4B-experimental`) and Kev-4B (`jaredpalmer/kev-4b`), two other open Jev-style
165
+ models, were scored zero-shot with their own readouts and our questions. Bold is the best in each column.
166
 
167
  | Model | Tax | Legal | AEGIS 2.0 | Nemotron | PolyGuard | XSTest | PolyGuard sec. | deepset | Mean |
168
  |---|---|---|---|---|---|---|---|---|---|
 
300
  The adapter is released under Apache-2.0, like Gemma-4-E4B-it. Every training source is licensed CC BY
301
  4.0, Apache 2.0 or MIT, and the synthetic conversations were generated and checked with Gemma-4 models
302
  (Apache 2.0). The evaluation sets keep their own licenses (regulated-advice-bench is CC BY-NC-SA 4.0);
303
+ they were used for evaluation only. The notices that come with Gemma 4 E4B-it are in
304
+ [`THIRD-PARTY-NOTICES.txt`](THIRD-PARTY-NOTICES.txt).
305
 
306
  ## Files
307
 
 
311
  | `sev.py` | the inference helper used above |
312
  | `results.json` | every number in this card |
313
  | `assets/sev_teaser.png`, `assets/sev_results.png` | the figures above |
314
+ | `THIRD-PARTY-NOTICES.txt` | notices and the Apache 2.0 license of Gemma 4 E4B-it, the model Sev is fine-tuned from |