Blame for README.md

---
license: apache-2.0
language:
- en
base_model:
- Qwen/Qwen3.5-9B
- meta-llama/Llama-3.1-8B-Instruct
tags:
- safety
- guardrail
- constitutional-classifiers
- jailbreak-detection
- content-moderation
library_name: peft
pipeline_tag: text-classification
---

# EuroSafeGuard

**EuroSafeGuard** is an open constitutional-classifier guardrail from EuroSafeAI. It screens the requests sent to a
language model and the replies the model produces, and it is trained on data generated from a written constitution of
eleven harm categories.

Paper: *EuroSafeGuard: Open Constitutional Classifiers* (arXiv: coming soon) · Code: coming soon ·
Training prompts: coming soon (gated access) · Project page: [guard.safe.eu](https://guard.safe.eu)

## Components

| Folder | What it is | Base model |
|---|---|---|
| `probe/` | MLP probe (hidden size 256) on the activations of all 32 blocks of the protected model, reading the chat-formatted prompt; per-token logits are smoothed with an exponential moving average and the prompt score is their maximum | Llama-3.1-8B-Instruct |
| `gate/seed{0..4}/` | Input classifier: LoRA adapter + one sigmoid head per harm category (11 heads); score = highest head | Qwen3.5-9B |
| `exchange/` | Exchange classifier: reads `User: <prompt>\n\nAssistant: <reply>`; LoRA + 11 heads | Qwen3.5-9B |
| `thresholds/seed{k}.json` | Alert mode: score normalisation and thresholds for each input-classifier seed | |
| `thresholds/full_seed{k}.json` | Full mode: normalisation (probe, gate, exchange), probe escalation cut, weights and threshold for each seed | |
| `eurosafeguard.py` | Single-file loader with both modes | |

The five `gate/seed*` folders are the five training runs (same data, different seeds) that the paper reports as a mean.
They vary, especially against search attacks such as PAIR, so evaluate the seed you deploy.

**Harm categories:** CBRN, Cyber, Harm to Minors, Harmful Manipulation, Hate Speech, Illegal Activities, Information
Integrity, Physical Harm, Privacy, Self-Harm, Sexual Content. `taxonomy_11heads_short.json` lists what each category
allows and disallows.

## Two modes

| Mode | Load with | Paper configuration | Paper tables |
|---|---|---|---|
| `alert` (default) | `mode="alert"` | EuroSafeGuard (Input) at the 5% WildChat-clean budget, plus alert-unless-CBRN and the exchange check | Held-out attacks (Input columns), alert mode ("Alert, strict exchange") |
| `full` | `mode="full"` | EuroSafeGuard (Full): probe escalation, weighted sum of probe, input and exchange scores | End-to-end table |

### Alert mode (input/alert, alert-unless-CBRN)

1. **Input stage.** The input score is the mean of the probe and input-classifier scores, each z-scored on
   WildChat-clean. Its threshold (`alert_input_thr`) flags 5% of WildChat-clean prompts.
2. A flagged request whose top category is **CBRN** is **refused**. Any other flagged request is **answered under an
   alert system note** (`system_note(category)`), which tells the protected model what it must not provide.
3. **Output stage.** The exchange classifier checks the reply. The reply is withheld if its score exceeds
   `exch_thr_20pct` for alerted requests (a stricter threshold, flagging 20% of WildChat-clean replies), or `exch_thr_1pct`
   for unflagged requests (flagging 1%).

```python
from eurosafeguard import EuroSafeGuard          # download eurosafeguard.py from this repo
g = EuroSafeGuard.from_pretrained("EuroSafeAI/EuroSafeGuard", seed=0, devices=("cuda:0", "cuda:0", "cuda:1"))
decision, category, score = g.route(prompt)      # "pass" | "alert" | "refuse"
messages = [{"role": "user", "content": prompt}]
if decision == "alert":
    messages.insert(0, {"role": "system", "content": g.system_note(category)})
reply = ...                                        # your protected model
if decision == "refuse" or g.check_reply(prompt, reply, decision):
    reply = "Sorry, I can't help with that."
```

### Full mode (the paper's end-to-end configuration)

1. **Probe screen.** The probe scores every request. Only requests whose probe z-score exceeds `probe_escalation_z`
   (1.1644, the 90th percentile on WildChat-clean) are escalated; the rest are never blocked.
2. **Escalated requests** are scored by the input classifier and, once the protected model has replied, by the exchange
   classifier. The exchange is blocked if `0.6·probe_z + 0.2·input_z + 0.2·exchange_z > full_thr`.
3. All three scores are z-scored on WildChat-clean (the exchange on Llama-3-8B-Instruct replies: μ = 0.02560,
   σ = 0.09032). `full_thr` is the smallest threshold that keeps every benign calibration set under its cap (WildChat-clean,
   MMLU, GSM8K, Alpaca, Dolly, OASST 0.5%; XSTest-safe, OR-Bench-80k 2%; OR-Bench-hard 5%; benign rows of styled
   jailbreak sets 40%): 3.382 / 3.571 / 3.132 / 3.097 / 3.318 for seeds 0–4. It was fitted once on the undefended
   Llama-3-8B-Instruct's traffic.

```python
g = EuroSafeGuard.from_pretrained("EuroSafeAI/EuroSafeGuard", seed=0, mode="full", devices=("cuda:0", "cuda:0", "cuda:1"))
reply = ...                                        # your protected model
if g.check_full(prompt, reply):                    # probe on every request; gate + exchange only if escalated
    reply = "Sorry, I can't help with that."
```

`full_block(probe, gate, exchange, cfg)` applies the same rule to precomputed raw scores. On the paper's stored scores it
reproduces the end-to-end attack success of every seed exactly (mean 3.88%).

**Requirements:** `torch`, `transformers` with Qwen3.5 support, `peft`, `safetensors`, `huggingface_hub`. About 16 GB of
GPU memory for Llama-3.1-8B (the probe) plus about 19 GB for each Qwen3.5-9B classifier (input and exchange). Use
`load_exchange=False` to run the input stage only. The probe reads Llama-3.1-8B-Instruct activations. If you protect
Llama-3.1-8B-Instruct itself, you can reuse the activations of its forward pass.

## Results (from the paper; every guard calibrated to the same false-positive budget on real chat traffic)

- **End to end** (Full mode, Llama-3-8B-Instruct, 1,974 Jailbreak Foundry attacks on held-out GuidedBench behaviors):
  attack success falls from 7.95% to 3.88% (EuroSafe-CC++, our reimplementation of Constitutional Classifiers++, 4.57%),
  adding at most 3.2 points of benign failure on any of six benign benchmarks. On gpt-oss-20b: 13.17% → 5.28%.
- **Compute:** only the 10% of requests the probe escalates reach the two classifiers, so the guardrail adds 15% of the
  protected model's FLOPs.
- **Held-out jailbreaks**, input stage at the 5% WildChat-clean budget: 95.9% recall on the BELLS-O jailbreak set
  (88.2% for HaloGuard, the strongest open guard); highest recall on three of four held-out jailbreak sets.
- **PAIR** (iterative black-box attack), input stage at the 5% WildChat-clean budget: 21% of the behaviors that break
  the unprotected model get through, against 49% for the best other open guard.
- **Chemical and biological goals** transferred to three protected models: the full guardrail lets through at most 1.4% of goals on any model.
- **Alert mode** (`mode="alert"`) vs refusing every flagged request: refusals of benign sensitive requests fall
  from 37% to 5–16% on six protected models, while at most 3.6% of chemical and biological goals get through.
- **Latency:** the full guardrail adds 1.5% to generation time on the requests it escalates.

Numbers are means over the five input-classifier seeds; see the paper for the full tables, judges and budgets.

## Limitations

- **Low-resource languages:** recall on multilingual harmful prompts (MSafeTemp) drops to 74% in low-resource
  languages, below several other guards.
- **Content moderation:** on general moderation benchmarks (ToxicChat, OpenAI Moderation and others), dedicated
  moderation models remain ahead.
- **Over-refusal:** the input stage flags a fraction of benign but sensitive requests. Alert mode answers most of them,
  but it is not a substitute for checking false positives on your own traffic.
- **Fixed policy:** the thresholds are calibrated on WildChat-clean. Recalibrate on your own benign traffic if your
  traffic differs.
- **Not a complete defence:** no guardrail stops every attack. Combine it with a safety-trained model, monitoring and
  rate limits.

## License

Our adapters, heads, probe and code are released under Apache-2.0. The base models keep their own licenses:
[Llama 3.1 Community License](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) for the probe's protected model,
and the [Qwen license](https://huggingface.co/Qwen/Qwen3.5-9B) for the classifier backbones. Some training data is
licensed for non-commercial use only.

## Citation

```bibtex
@article{waibl2026eurosafeguard,
  title   = {EuroSafeGuard: Open Constitutional Classifiers},
  author  = {Waibl, Leonhard and Chen, Yen-Shan and Simko, Samuel and Michalak, Felix and
             Diwan, Nirav and Singh, Arth and Thomas, Aditya and
             Sch{\"o}lkopf, Bernhard and Jin, Zhijing},
  journal = {arXiv preprint},
  year    = {2026}
}
```