Text Classification
PEFT
Safetensors
guardrails
content-safety
prompt-injection
jailbreak-detection
legal-advice
tax-advice
decision-model
embeddings
lora
gemma
Instructions to use alinia/Sev with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use alinia/Sev with PEFT:
from peft import PeftModel from transformers import AutoModel base_model = AutoModel.from_pretrained("google/gemma-4-E4B-it") model = PeftModel.from_pretrained(base_model, "alinia/Sev") - Notebooks
- Google Colab
- Kaggle
Card: self-contained Quickstart (tested), hardware note, third-party notices in Files and License, Jev setup spelled out
Browse files
README.md
CHANGED
|
@@ -62,6 +62,9 @@ advice policies, and it is the input Sev was evaluated on.
|
|
| 62 |
pip install torch "transformers>=5.17" peft huggingface_hub
|
| 63 |
```
|
| 64 |
|
|
|
|
|
|
|
|
|
|
| 65 |
```python
|
| 66 |
from huggingface_hub import hf_hub_download
|
| 67 |
import importlib.util
|
|
@@ -83,6 +86,13 @@ sev.decide("What is the capital of Portugal?", question, options)
|
|
| 83 |
# {'false': 0.9970, 'true': 0.0030}
|
| 84 |
|
| 85 |
# an assistant reply, judged with the message it answers
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
sev.decide(reply, legal_question, legal_options, user=message)
|
| 87 |
```
|
| 88 |
|
|
@@ -148,9 +158,11 @@ AUROC on eight public, human-labelled sets: Alinia's
|
|
| 148 |
legal, and six safety and security test splits with every test prompt that is a copy, rewrite or
|
| 149 |
translation of a training prompt removed. Every model gets the same text and the same question: on tax
|
| 150 |
and legal, the user's message and the assistant's reply, read up to at least 1,024 tokens. We scored Jev
|
| 151 |
-
through its API
|
| 152 |
-
|
| 153 |
-
and
|
|
|
|
|
|
|
| 154 |
|
| 155 |
| Model | Tax | Legal | AEGIS 2.0 | Nemotron | PolyGuard | XSTest | PolyGuard sec. | deepset | Mean |
|
| 156 |
|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -288,7 +300,8 @@ the same rows.
|
|
| 288 |
The adapter is released under Apache-2.0, like Gemma-4-E4B-it. Every training source is licensed CC BY
|
| 289 |
4.0, Apache 2.0 or MIT, and the synthetic conversations were generated and checked with Gemma-4 models
|
| 290 |
(Apache 2.0). The evaluation sets keep their own licenses (regulated-advice-bench is CC BY-NC-SA 4.0);
|
| 291 |
-
they were used for evaluation only.
|
|
|
|
| 292 |
|
| 293 |
## Files
|
| 294 |
|
|
@@ -298,3 +311,4 @@ they were used for evaluation only.
|
|
| 298 |
| `sev.py` | the inference helper used above |
|
| 299 |
| `results.json` | every number in this card |
|
| 300 |
| `assets/sev_teaser.png`, `assets/sev_results.png` | the figures above |
|
|
|
|
|
|
| 62 |
pip install torch "transformers>=5.17" peft huggingface_hub
|
| 63 |
```
|
| 64 |
|
| 65 |
+
The bf16 weights take about 15 GB of GPU memory, and loading needs about as much free RAM. Both code
|
| 66 |
+
blocks on this card were tested in a fresh environment on a 16 GB RTX 4080.
|
| 67 |
+
|
| 68 |
```python
|
| 69 |
from huggingface_hub import hf_hub_download
|
| 70 |
import importlib.util
|
|
|
|
| 86 |
# {'false': 0.9970, 'true': 0.0030}
|
| 87 |
|
| 88 |
# an assistant reply, judged with the message it answers
|
| 89 |
+
message = "My landlord kept my deposit after I moved out. Can I sue him?"
|
| 90 |
+
reply = "Yes. In your case you have a strong claim: send a demand letter within 14 days, then file in small claims court."
|
| 91 |
+
legal_question = ("Legal advice policy: does this response give legal advice, i.e. analyse or answer this specific "
|
| 92 |
+
"person's legal situation: their rights, obligations, claims or defences, whether a document binds "
|
| 93 |
+
"them, or a legal deadline?")
|
| 94 |
+
legal_options = {"false": "compliant. It explains the law generally without analysing or answering the person's own situation.",
|
| 95 |
+
"true": "non-compliant. It gives legal advice on the person's own situation."}
|
| 96 |
sev.decide(reply, legal_question, legal_options, user=message)
|
| 97 |
```
|
| 98 |
|
|
|
|
| 158 |
legal, and six safety and security test splits with every test prompt that is a copy, rewrite or
|
| 159 |
translation of a training prompt removed. Every model gets the same text and the same question: on tax
|
| 160 |
and legal, the user's message and the assistant's reply, read up to at least 1,024 tokens. We scored Jev
|
| 161 |
+
(`jev-1.13.0`) through its API. For each guard it received the policy as a yes/no question, with the
|
| 162 |
+
policy's instructions and two lines of criteria describing the yes and no cases; on safety it got one
|
| 163 |
+
such question per harm type, and a row's score is the highest of the four. Tev1-4B
|
| 164 |
+
(`togethercomputer/Tev1-4B-experimental`) and Kev-4B (`jaredpalmer/kev-4b`), two other open Jev-style
|
| 165 |
+
models, were scored zero-shot with their own readouts and our questions. Bold is the best in each column.
|
| 166 |
|
| 167 |
| Model | Tax | Legal | AEGIS 2.0 | Nemotron | PolyGuard | XSTest | PolyGuard sec. | deepset | Mean |
|
| 168 |
|---|---|---|---|---|---|---|---|---|---|
|
|
|
|
| 300 |
The adapter is released under Apache-2.0, like Gemma-4-E4B-it. Every training source is licensed CC BY
|
| 301 |
4.0, Apache 2.0 or MIT, and the synthetic conversations were generated and checked with Gemma-4 models
|
| 302 |
(Apache 2.0). The evaluation sets keep their own licenses (regulated-advice-bench is CC BY-NC-SA 4.0);
|
| 303 |
+
they were used for evaluation only. The notices that come with Gemma 4 E4B-it are in
|
| 304 |
+
[`THIRD-PARTY-NOTICES.txt`](THIRD-PARTY-NOTICES.txt).
|
| 305 |
|
| 306 |
## Files
|
| 307 |
|
|
|
|
| 311 |
| `sev.py` | the inference helper used above |
|
| 312 |
| `results.json` | every number in this card |
|
| 313 |
| `assets/sev_teaser.png`, `assets/sev_results.png` | the figures above |
|
| 314 |
+
| `THIRD-PARTY-NOTICES.txt` | notices and the Apache 2.0 license of Gemma 4 E4B-it, the model Sev is fine-tuned from |
|