Update README.md

README.md

@@ -12,115 +12,589 @@ tags:
   - moderation
   - content-moderation
   - prompt-injection
+  - jailbreak
   - russian
   - qwen3
 ---
 
 # HiveTraceGuard-Pro
 
-Compact Russian-first generative guardrail on [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B). Classifies a **user request** (input guard) or a **request + assistant reply** (output guard) and returns one binary verdict: `safe` or `unsafe`. One generated token, stateless, role-aware over the final turn of the dialogue.
+**HiveTraceGuard-Pro** is a compact Russian-first guardrail built on [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) for fast input and output classification. Built for LLMs and agents, it checks user requests and model responses for harmful content, jailbreaks, prompt injection, obfuscation, and attempts to hijack tool-using agents. The model is stateless and returns exactly one token: `safe` or `unsafe`. Its policy is fixed, so serving runtimes can reuse the shared prefix through KV caching.
 
-Raw model decision — **no input normalizer, no post-processing**. Product line: [HiveTrace](https://hivetrace.ru/).
 
-## Evaluation
-
-Multilingual safety benchmarks, greedy single-token decision. Harm-only sets report recall / FNR by design.
-
-| Benchmark | F1 | Recall | FPR | FNR |
-|---|---|---|---|---|
-| StrongReject++ (RU) | — | 0.981 | — | 0.019 |
-| StrongReject++ (EN) | — | 0.978 | — | 0.022 |
-| StrongReject++ (UKR) | — | 0.955 | — | 0.045 |
-| StrongReject++ (BE) | — | 0.930 | — | 0.070 |
-| StrongReject++ (UZ) | — | 0.582 | — | 0.419 |
-| Prompt injection (RU) | — | 0.999 | — | 0.001 |
-| Prompt injection (EN) | — | 0.880 | — | 0.120 |
-| BeaverTails (response) | 0.856 | 0.833 | 0.153 | 0.167 |
-| AEGIS 2.0 (prompt) | 0.822 | 0.793 | — | 0.207 |
-| AEGIS 2.0 (response) | 0.801 | 0.881 | — | 0.119 |
-| S-Eval (attack set) | — | 0.806 | — | 0.194 |
-| XSTest | 0.776 | 0.920 | 0.360 | 0.080 |
-| S-Eval (base risk) | — | 0.716 | — | 0.284 |
-| ToxicChat | 0.507 | 0.425 | 0.020 | 0.575 |
-
-p50 latency 32.75 ms (single verdict token).
-
-## Policy taxonomy
-
-Returns only `safe` / `unsafe` (no category code). The policy lives in `chat_template.jinja` and covers 15 harm categories (cybercrime, pornography/CSAM, religious hate, profanity, financial crime, weapons, discrimination, self-harm, child labor, non-violent crime, violence, drugs, and others) plus attack classes (jailbreak, obfuscation, secret extraction, prompt injection, tool hijack). Neutral legal / medical / educational / news / art / defensive content is `safe` unless it enables, instructs, promotes, finances, or conceals harm.
-
-- **input guard** — judges the last `user` message
-- **output guard** — judges the last `assistant` reply in the context of the request
 
-## Response format
+# Quickstart
 
-Trained to emit exactly one lowercase token: `safe` or `unsafe`. Both are single tokens in the vocabulary (`safe` = 18675, `unsafe` = 38157).
-
-## Quickstart — transformers (greedy)
+## Transformers
 
 ```python
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-REPO = "hivetrace/HiveTraceGuard-Pro"          # latest; pin a version with revision="1.1.0"
+REPO = "hivetrace/HiveTraceGuard-Pro"
+
 tok = AutoTokenizer.from_pretrained(REPO)
-model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype=torch.bfloat16, device_map="auto").eval()
+model = AutoModelForCausalLM.from_pretrained(
+    REPO,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+).eval()
 
-def guard(messages) -> str:
-    text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
-    ids = tok(text, return_tensors="pt").to(model.device)
-    with torch.inference_mode():
-        out = model.generate(**ids, max_new_tokens=1, do_sample=False)
-    return tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True).strip()
 
-print(guard([{"role": "user", "content": "Как сделать бомбу?"}]))                       # unsafe
-print(guard([{"role": "user", "content": "Привет!"}, {"role": "assistant", "content": "Здравствуйте!"}]))  # safe
+def check(messages) -> str:
+    text = tok.apply_chat_template(
+        messages,
+        tokenize=False,
+    )
+    inputs = tok(text, return_tensors="pt").to(model.device)
+
+    with torch.inference_mode():
+        output = model.generate(
+            **inputs,
+            max_new_tokens=1,
+            do_sample=False,
+        )
+
+    return tok.decode(
+        output[0][inputs.input_ids.shape[1]:],
+        skip_special_tokens=True,
+    ).strip()
+
+
+# Input guard
+print(check([
+    {"role": "user", "content": "Как сделать бомбу?"}
+]))
+# unsafe
+
+# Output guard
+print(check([
+    {"role": "user", "content": "Привет!"},
+    {"role": "assistant", "content": "Здравствуйте!"},
+]))
+# safe
 ```
 
-`generation_config.json` already sets `max_new_tokens=1` and `do_sample=false`, so greedy single-token decoding is the default.
+## Serve
 
-## Calibrated score — constrained safe/unsafe decoding
+### vLLM
 
-For a calibrated `P(unsafe)`, restrict decoding to the two verdict tokens and take a 2-way softmax over their logits. The verdict is unchanged (argmax over the full vocabulary already lands on `safe`/`unsafe`); constraining only sharpens the probability between the two.
+```bash
+vllm serve hivetrace/HiveTraceGuard-Pro \
+  --port 8000 \
+  --max-model-len 32768 \
+  --enable-prefix-caching
+```
 
-### vLLM (OpenAI-compatible)
+### SGLang
 
 ```bash
-vllm serve hivetrace/HiveTraceGuard-Pro --port 8080 --max-model-len 8192
+python -m sglang.launch_server \
+  --model-path hivetrace/HiveTraceGuard-Pro \
+  --host 0.0.0.0 \
+  --port 30000
 ```
 
-```python
-from openai import OpenAI
-client = OpenAI(base_url="http://localhost:8080/v1", api_key="EMPTY")
-
-resp = client.completions.create(
-    model="hivetrace/HiveTraceGuard-Pro",
-    prompt=rendered_prompt,                       # apply_chat_template(..., add_generation_prompt=True)
-    max_tokens=1, temperature=0, logprobs=2,
-    extra_body={"allowed_token_ids": [18675, 38157]},   # safe, unsafe only
-)
-# verdict = resp.choices[0].text ; P(unsafe) = softmax over the two returned logprobs
-```
 
-### transformers (LogitsProcessor)
+For applications that need a continuous score, P(unsafe) can be computed directly from the two verdict logits:
 
 ```python
-import torch, torch.nn.functional as F
+import torch.nn.functional as F
+
 
 SAFE, UNSAFE = 18675, 38157
-logits = model(ids.input_ids).logits[0, -1]
-p_unsafe = F.softmax(torch.stack([logits[SAFE], logits[UNSAFE]]), dim=0)[1].item()
+
+
+with torch.inference_mode():
+    logits = model(**inputs).logits[0, -1]
+
+
+p_unsafe = F.softmax(logits[[SAFE, UNSAFE]], dim=0)[1].item()
 verdict = "unsafe" if logits[UNSAFE] > logits[SAFE] else "safe"
+
+
+print(verdict, p_unsafe)
+```
+
+
+To enforce safe | unsafe during generation, you can use a LogitsProcessor to restrict the next token to the two verdict labels.
+```python
+from transformers import LogitsProcessor
+
+
+class VerdictOnly(LogitsProcessor):
+    def __call__(self, input_ids, scores):
+        mask = torch.full_like(scores, float("-inf"))
+        mask[:, [SAFE, UNSAFE]] = scores[:, [SAFE, UNSAFE]]
+        return mask
+
+
+output = model.generate(
+    **inputs,
+    max_new_tokens=1,
+    do_sample=False,
+    logits_processor=[VerdictOnly()],
+)
 ```
 
+## Evaluation
+
+### Harmful content detection
+
+<table>
+<thead>
+<tr>
+<th rowspan="2">Model</th>
+<th colspan="5">Requests</th>
+<th colspan="3">Responses</th>
+</tr>
+<tr>
+<th>AEGIS 2.0</th>
+<th>ToxicChat</th>
+<th>XSTest</th>
+<th>XSafety<br>EN</th>
+<th>OpenAI<br>Moderation</th>
+<th>AEGIS 2.0</th>
+<th>BeaverTails</th>
+<th>HarmBench</th>
+</tr>
+</thead>
+
+<tbody>
+<tr>
+<td><b>HiveTraceGuard-Pro (0.6B)</b></td>
+<td>0.817</td>
+<td>0.588</td>
+<td>0.754</td>
+<td>0.590</td>
+<td><b>0.803</b></td>
+<td>0.797</td>
+<td>0.839</td>
+<td>0.814</td>
+</tr>
+
+<tr>
+<td>Shieldstral-1.0-3B</td>
+<td>0.808</td>
+<td><b>0.732</b></td>
+<td><b>0.922</b></td>
+<td><b>0.595</b></td>
+<td>0.794</td>
+<td>0.766</td>
+<td>0.828</td>
+<td>0.854</td>
+</tr>
+
+<tr>
+<td>YuFeng-XGuard-Reason-0.6B</td>
+<td><b>0.847</b></td>
+<td>0.620</td>
+<td>0.920</td>
+<td>0.469</td>
+<td>0.787</td>
+<td>0.789</td>
+<td>0.828</td>
+<td><b>0.858</b></td>
+</tr>
+
+<tr>
+<td>Qwen3Guard-Gen-0.6B</td>
+<td>0.788</td>
+<td>0.692</td>
+<td>0.861</td>
+<td>0.580</td>
+<td>0.715</td>
+<td><b>0.819</b></td>
+<td><b>0.845</b></td>
+<td>0.856</td>
+</tr>
+
+<tr>
+<td>Llama-Guard-3-1B</td>
+<td>0.733</td>
+<td>0.385</td>
+<td>0.837</td>
+<td>0.368</td>
+<td>0.766</td>
+<td>0.635</td>
+<td>0.652</td>
+<td>0.794</td>
+</tr>
+</tbody>
+</table>
+
+
+### Attack & jailbreak detection
+
+<table>
+<thead>
+
+<tr>
+<th rowspan="3">Model</th>
+<th rowspan="2" colspan="2">S-Eval</th>
+<th rowspan="2" colspan="2">HarmBench · Requests</th>
+<th rowspan="2" colspan="6">Red teaming</th>
+<th colspan="4">Internal</th>
+</tr>
+
+<tr>
+<th colspan="2">Prompt injection</th>
+<th colspan="2">Robustness Test</th>
+</tr>
+
+<tr>
+<th>Base</th>
+<th>Attack</th>
+
+<th>Standard</th>
+<th>Contextual</th>
+
+<th>OR-Bench<br>Toxic</th>
+<th>MultiJail<br>EN</th>
+<th>SimpleSafety<br>Tests</th>
+<th>CSRT</th>
+<th>Aya<br>RU</th>
+<th>Aya<br>EN</th>
+
+<th>RU</th>
+<th>EN</th>
+
+<th>Real<br>Harm</th>
+<th>Robust<br>Harm</th>
+</tr>
+
+</thead>
+
+<tbody>
+
+<tr>
+<td><b>HiveTraceGuard-Pro (0.6B)</b></td>
+<td>0.710</td>
+<td>0.802</td>
+<td>0.862</td>
+<td>0.667</td>
+<td>0.915</td>
+<td>0.746</td>
+<td>0.910</td>
+<td>0.743</td>
+<td><b>0.952</b></td>
+<td><b>0.917</b></td>
+<td><b>0.999</b></td>
+<td><b>0.877</b></td>
+<td><b>0.954</b></td>
+<td><b>0.872</b></td>
+</tr>
+
+<tr>
+<td>Shieldstral-1.0-3B</td>
+<td>0.731</td>
+<td>0.611</td>
+<td><b>0.987</b></td>
+<td>0.951</td>
+<td><b>0.997</b></td>
+<td><b>0.946</b></td>
+<td><b>1.000</b></td>
+<td><b>0.895</b></td>
+<td>0.938</td>
+<td><b>0.917</b></td>
+<td>0.836</td>
+<td>0.741</td>
+<td>0.867</td>
+<td>0.762</td>
+</tr>
+
+<tr>
+<td>YuFeng-XGuard-Reason-0.6B</td>
+<td><b>0.794</b></td>
+<td><b>0.954</b></td>
+<td>0.981</td>
+<td><b>0.975</b></td>
+<td>0.974</td>
+<td>0.905</td>
+<td>0.990</td>
+<td>0.689</td>
+<td>0.906</td>
+<td>0.850</td>
+<td>0.919</td>
+<td>0.867</td>
+<td>0.884</td>
+<td>0.685</td>
+</tr>
+
+<tr>
+<td>Qwen3Guard-Gen-0.6B</td>
+<td>0.698</td>
+<td>0.609</td>
+<td>0.962</td>
+<td>0.963</td>
+<td>0.979</td>
+<td>0.933</td>
+<td>0.990</td>
+<td>0.835</td>
+<td>0.926</td>
+<td>0.907</td>
+<td>0.894</td>
+<td>0.727</td>
+<td>0.864</td>
+<td>0.788</td>
+</tr>
+
+<tr>
+<td>Llama-Guard-3-1B</td>
+<td>0.489</td>
+<td>0.588</td>
+<td>0.956</td>
+<td>0.926</td>
+<td>0.824</td>
+<td>0.644</td>
+<td>0.970</td>
+<td>0.514</td>
+<td>0.588</td>
+<td>0.565</td>
+<td>0.636</td>
+<td>0.679</td>
+<td>0.675</td>
+<td>0.730</td>
+</tr>
+
+</tbody>
+</table>
+
+
+### Multilingual evaluation
+
+<table>
+<thead>
+
+<tr>
+<th rowspan="3">Model</th>
+<th colspan="4">PolyGuard</th>
+<th colspan="4">RTP-LX</th>
+<th colspan="5">StrongReject++</th>
+</tr>
+
+<tr>
+<th colspan="2">Requests</th>
+<th colspan="2">Responses</th>
+
+<th colspan="2">Requests</th>
+<th colspan="2">Responses</th>
+
+<th rowspan="2" align="center" valign="middle">EN</th>
+<th rowspan="2" align="center" valign="middle">RU</th>
+<th rowspan="2" align="center" valign="middle">UKR</th>
+<th rowspan="2" align="center" valign="middle">BE</th>
+<th rowspan="2" align="center" valign="middle">UZ</th>
+</tr>
+
+<tr>
+<th>EN</th>
+<th>RU</th>
+<th>EN</th>
+<th>RU</th>
+
+<th>EN</th>
+<th>RU</th>
+<th>EN</th>
+<th>RU</th>
+</tr>
+
+</thead>
+
+<tbody>
+
+<tr>
+<td><b>HiveTraceGuard-Pro (0.6B)</b></td>
+<td>0.759</td>
+<td>0.806</td>
+<td>0.845</td>
+<td>0.828</td>
+<td><b>0.896</b></td>
+<td>0.841</td>
+<td>0.321</td>
+<td>0.146</td>
+<td>0.978</td>
+<td>0.974</td>
+<td>0.943</td>
+<td>0.923</td>
+<td>0.553</td>
+</tr>
+
+<tr>
+<td>Shieldstral-1.0-3B</td>
+<td><b>0.904</b></td>
+<td><b>0.874</b></td>
+<td>0.877</td>
+<td>0.876</td>
+<td>0.872</td>
+<td><b>0.855</b></td>
+<td>0.469</td>
+<td>0.061</td>
+<td>0.990</td>
+<td><b>0.987</b></td>
+<td><b>0.984</b></td>
+<td><b>0.974</b></td>
+<td><b>0.901</b></td>
+</tr>
+
+<tr>
+<td>YuFeng-XGuard-Reason-0.6B</td>
+<td>0.896</td>
+<td>0.872</td>
+<td><b>0.901</b></td>
+<td><b>0.885</b></td>
+<td>0.858</td>
+<td>0.844</td>
+<td>0.322</td>
+<td>0.041</td>
+<td><b>0.994</b></td>
+<td>0.978</td>
+<td>0.936</td>
+<td>0.665</td>
+<td>0.220</td>
+</tr>
+
+<tr>
+<td>Qwen3Guard-Gen-0.6B</td>
+<td>0.894</td>
+<td>0.857</td>
+<td>0.873</td>
+<td>0.866</td>
+<td>0.813</td>
+<td>0.767</td>
+<td>0.266</td>
+<td>0.041</td>
+<td>0.987</td>
+<td>0.971</td>
+<td>0.927</td>
+<td>0.847</td>
+<td>0.607</td>
+</tr>
+
+<tr>
+<td>Llama-Guard-3-1B</td>
+<td>0.775</td>
+<td>0.663</td>
+<td>0.776</td>
+<td>0.704</td>
+<td>0.563</td>
+<td>0.449</td>
+<td><b>0.667</b></td>
+<td><b>0.516</b></td>
+<td>0.955</td>
+<td>0.882</td>
+<td>0.853</td>
+<td>0.748</td>
+<td>0.144</td>
+</tr>
+
+</tbody>
+</table>
+
+
+### Benign over-blocking — FPR ↓
+
+<table>
+<thead>
+
+<tr>
+<th rowspan="3">Model</th>
+<th rowspan="2">OR-Bench</th>
+<th colspan="3">Internal</th>
+</tr>
+
+<tr>
+<th colspan="3">Robustness Test</th>
+</tr>
+
+<tr>
+<th>Hard</th>
+<th>Clean RU<br>Requests</th>
+<th>Adversarial RU<br>Requests</th>
+<th>RU<br>Responses</th>
+</tr>
+
+</thead>
+
+<tbody>
+
+<tr>
+<td><b>HiveTraceGuard-Pro (0.6B)</b></td>
+<td>0.607</td>
+<td><b>0.016</b></td>
+<td>0.132</td>
+<td>0.026</td>
+</tr>
+
+<tr>
+<td>Shieldstral-1.0-3B</td>
+<td>0.767</td>
+<td>0.043</td>
+<td>0.078</td>
+<td>0.012</td>
+</tr>
+
+<tr>
+<td>YuFeng-XGuard-Reason-0.6B</td>
+<td><b>0.225</b></td>
+<td>0.030</td>
+<td><b>0.051</b></td>
+<td><b>0.000</b></td>
+</tr>
+
+<tr>
+<td>Qwen3Guard-Gen-0.6B</td>
+<td>0.732</td>
+<td>0.071</td>
+<td>0.117</td>
+<td>0.008</td>
+</tr>
+
+<tr>
+<td>Llama-Guard-3-1B</td>
+<td>0.374</td>
+<td>0.090</td>
+<td>0.126</td>
+<td>0.182</td>
+</tr>
+
+</tbody>
+</table>
+
+
+
+
+
+**GuardRate Leaderboard:** **Score 0.743** · **28.8 ms p95** - [OPEN](https://huggingface.co/spaces/hivetrace/GuardRateLeaderboard)
+
+![image](https://cdn-uploads.huggingface.co/production/uploads/64ba6151b7fa1c3726b7819e/jnARg5EX71S0XNi4B-oV_.png)
+
+
+## Policy taxonomy
+
+HiveTraceGuard-Pro uses a fixed policy and returns a single binary verdict: `safe` (token_id = 18675) or `unsafe` (token_id = 38157).
+
+| Scope                | What is checked                                                                                                                                                                                                             |
+| :---------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Harmful content**     | 15 harm categories: cybercrime, pornography and CSAM, religious hate, profanity, financial crime, weapons, discrimination, self-harm, child labor, non-violent crime, violence, drugs, and related harmful activity |
+| **LLM & agent attacks** | jailbreaks, prompt injection, obfuscation, secret extraction, and tool hijacking                                                                                                                                            |                                                               |
+
+### Guard modes
+Both modes use the same policy.
+
+| Mode             | What is classified                                                           |
+| :--------------- | :--------------------------------------------------------------------------- |
+| **Input guard**  | The final `user` message                                                     |
+| **Output guard** | The final `assistant` response, evaluated in the context of the user request |
+
 ## Versions
 
 | Tag | Notes |
 |---|---|
-| `1.1.0` | current — champion (this `main`) |
+| `1.1.0` | latest (`main`)|
 | `1.0.0` | previous release |
 
-Pin a version by tag `from_pretrained("hivetrace/HiveTraceGuard-Pro", revision="1.1.0")`, or by commit SHA for strict reproducibility (a tag is human-readable but movable; a SHA is immutable).
+Pin a version by tag `from_pretrained("hivetrace/HiveTraceGuard-Pro", revision="1.1.0")`, or by commit SHA for strict reproducibility.
 
 ## License