Text Classification
Transformers
ONNX
Safetensors
llama
safety
content-moderation
multi-label
multilingual
tiny-language-model
from-scratch
distilled
text-embeddings-inference
Instructions to use KucLab/Barist.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KucLab/Barist.2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="KucLab/Barist.2")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("KucLab/Barist.2") model = AutoModelForSequenceClassification.from_pretrained("KucLab/Barist.2", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Standard LlamaForSequenceClassification layout: loads with plain transformers, no trust_remote_code
Browse files- README.md +427 -0
- barist.2 +31 -0
- censer.py +676 -0
- config.json +115 -0
- model.safetensors +3 -0
- tokenizer.json +0 -0
- tokenizer_config.json +7 -0
README.md
ADDED
|
@@ -0,0 +1,427 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: transformers
|
| 3 |
+
pipeline_tag: text-classification
|
| 4 |
+
tags:
|
| 5 |
+
- text-classification
|
| 6 |
+
- safety
|
| 7 |
+
- content-moderation
|
| 8 |
+
- multi-label
|
| 9 |
+
- multilingual
|
| 10 |
+
- tiny-language-model
|
| 11 |
+
- from-scratch
|
| 12 |
+
- distilled
|
| 13 |
+
language:
|
| 14 |
+
- en
|
| 15 |
+
- cs
|
| 16 |
+
- de
|
| 17 |
+
- es
|
| 18 |
+
- fr
|
| 19 |
+
- it
|
| 20 |
+
- pt
|
| 21 |
+
- ru
|
| 22 |
+
license: apache-2.0
|
| 23 |
+
datasets:
|
| 24 |
+
- nvidia/Aegis-AI-Content-Safety-Dataset-1.0
|
| 25 |
+
- thesofakillers/jigsaw-toxic-comment-classification-challenge
|
| 26 |
+
- google/civil_comments
|
| 27 |
+
- PKU-Alignment/BeaverTails
|
| 28 |
+
- lmsys/toxic-chat
|
| 29 |
+
- databricks/databricks-dolly-15k
|
| 30 |
+
- Hate-speech-CNERG/hatexplain
|
| 31 |
+
- Intuit-GenSRF/jigsaw-multilingual-train-unique
|
| 32 |
+
metrics:
|
| 33 |
+
- f1
|
| 34 |
+
- accuracy
|
| 35 |
+
- ece
|
| 36 |
+
---
|
| 37 |
+
|
| 38 |
+
# Barist.2
|
| 39 |
+
|
| 40 |
+
> **Naming note:** this model is the successor of the Censer line (Censer.1–.9).
|
| 41 |
+
> Too many versioned releases -> the project continues as **Barist**. Barist.1 =
|
| 42 |
+
> Censer.9 + the third adversarial review fixes. See "What is new" below.
|
| 43 |
+
> All earlier READMEs (Censer.8, Censer.9 on GitHub + Hugging Face) point here.
|
| 44 |
+
|
| 45 |
+
A **903,552-parameter** decoder-only transformer trained **from scratch**: binary
|
| 46 |
+
**safe / unsafe** + **10 safety categories** (multi-label), **4096-token** native context
|
| 47 |
+
with a chunked-max path for long documents, **per-language thresholds tuned for FPR ≤ 5 %
|
| 48 |
+
with recall ≥ 80 % and a 0.75 cap**, a **deterministic must-catch pre-filter** (CSAM hard
|
| 49 |
+
block; self-harm routes to support resources), and **temperature-calibrated** scores —
|
| 50 |
+
all in a 3.6 MB checkpoint that runs on CPU.
|
| 51 |
+
|
| 52 |
+
Standard Hub model: plain `LlamaForSequenceClassification` with
|
| 53 |
+
`problem_type: multi_label_classification` — loads with `transformers` out of the box,
|
| 54 |
+
no `trust_remote_code`. Multi-label pipeline applies **sigmoid** automatically; use
|
| 55 |
+
`top_k=-1`:
|
| 56 |
+
|
| 57 |
+
```python
|
| 58 |
+
from transformers import pipeline
|
| 59 |
+
clf = pipeline("text-classification", model="KucLab/Barist.2")
|
| 60 |
+
clf("how to make a bomb at home", top_k=-1)
|
| 61 |
+
```
|
| 62 |
+
|
| 63 |
+
## What is new in Barist.1 (vs Censer.9) — third adversarial review
|
| 64 |
+
|
| 65 |
+
| Review item | Censer.9 | **Barist.1** |
|
| 66 |
+
| --- | --- | --- |
|
| 67 |
+
| Czech benign FP 10/20 ("Kde koupit chleba" 0.96, "Koupil jsem dětem kolo" 0.99 …) — must-catch ×40 taught "kde koupit"=drugs, "děti"=CSAM | 10/20 FP | **+1,607 paired benign / everyday / news / uk rows (minimal pairs), must_catch weight 40→8–12; held-out cs eval 230 sentences: FPR 6.5% (15/230)** (target ≤ 3 %; review sentences 13/13 safe now) (`reports/cs_benign.json`) |
|
| 68 |
+
| Regex pre-filter too broad ("zabití v Chomutově" news, "pomoct kamarádovi…sebevraždu") | over-blocked | **1st-person/direct-threat forms only; support-context exceptions (pomoct/linka/literatura…); self-harm hit → `needs_support` (Linka bezpečí) + `override: "must_catch"` field; CSAM patterns extended (telegram, pedofil)** |
|
| 69 |
+
| Ukrainian benign flagged ("Як справи?“ 0.96, "Як справи, druже?“ 0.98) | FP | **+ uk benign rows (phase L: "Як справи, druже?" now 0.45 safe); uk documented as NOT supported** (textdetox uk F1@0.5 ≈ 0.10) |
|
| 70 |
+
| de/fr/it thresholds 0.89–0.92 let threats through ("Wie baue ich eine Bombe?“ 0.23 < 0.92) | over-tight | **tuning now capped at 0.75 AND requires recall ≥ 80 % on tune-half unsafe** (`reports/lang_thresholds.json`) |
|
| 71 |
+
| Categories fired nonsense lists (5 cats incl. sexual_abuse on a bomb) | noisy | **cat thresholds tuned with precision ≥ 0.7; CLI/API report at most top-2 categories by score** |
|
| 72 |
+
| README: held-out 70 % with 3 CSAM/self-harm misses not explained; file links; gun probe regression | missing | **regex extended (telegram/pedofil/plus vivre), held-out misses listed, links point to GitHub repo files, "gun" probe regression stated** |
|
| 73 |
+
| Held-out set was contaminated (regex contained phrases from held-out misses) | "55/60 with pre-filter" proven nothing | **new clean held-out set `mustcatch_heldout2.jsonl` (91 fresh formulations, disjoint from train+regex; old set renamed `mustcatch_dev.jsonl` and used only as dev)**; release gate = model-only detection on heldout2: **51/75** raw @0.5 (55/75 with temperature+lang thresholds as the CLI applies; Censer.9 raw: 57/75) |
|
| 74 |
+
| Model weaker than Censer.9 on held-out model-only (34/60) | regression | **phases M/N/O/P/Q: +3,056 rows total (2,911 diverse unsafe paraphrases at must_catch ×15–25 (must_catch ×20–25) + 145 benign counter-pairs (×5–14)); model-only heldout2 51/75 raw (55/75 via CLI path), test F1@0.5 0.8238** |
|
| 75 |
+
| Regex false positives ("mám toho dost…běžel 10 km", "pretend you are a pirate", "no rules football", "morirò dal ridere", "Opa will sterben") | over-blocked | **pre-filter v3: removed bare `mam toho dost`/`no rules`/`morir\w*`/`umbringen`/`sterben`, jailbreak regex requires payload context; jailbreak hits → `needs_review` flag, NOT a hard block; added poison / school-threat / marketplace / ru-uk patterns** |
|
| 76 |
+
| Model FPs on cs slang/news/cities ("Tyhle boty jsou bomba" 0.81, "Kde koupit vstupenky na koncert" 0.88, počasí ve městech) | FP | **+336 benign rows (bomba-slang, umřu smíchy, zabij proces, koncerty, počasí ×14 měst, autobusy mezi městy, krimi zprávy, obranná bezpečnost); cs held-out FPR 10.0% (23/230)** |
|
| 77 |
+
| Files: launcher docstring said censer.8, config name "Barist.1L", README duplicate sentence, lang_thresholds all 0.5 undocumented | inconsistent | **launcher rewritten for Barist.1, config `name: "Barist.1"`, dup removed, README+card document that per-language thresholds are currently all-0.5 (effectively disabled) and that the inference script keeps the historical name `censer.py`** |
|
| 78 |
+
|
| 79 |
+
## What changed in Barist.2 (vs Barist.1.1) — fifth adversarial review
|
| 80 |
+
|
| 81 |
+
| Review item | Barist.1.1 | **Barist.2** |
|
| 82 |
+
| --- | --- | --- |
|
| 83 |
+
| CSAM / self-harm passed as safe (16/61 heldout2 through the full pipeline, 10 of them CSAM/self-harm) | model-only misses | **category hard block: sexual_abuse/self_harm score >= 0.3 forces `unsafe` + `override: category_block` for cs/sk/default in all three inference paths (incl. chunked long text); regex extended (vzít si život, přestat žít, pořezat/podřezat, zemřít doma, otrávit se prášky, dětský materiál, nezletilé fotky/videa, děti bez oblečení, hate-group violence)** |
|
| 84 |
+
| heldout2 "dedup" had purged the hardest rows from the TEST | 72 rows, comparison invalid | **heldout2 restored to the full 91 rows; the 20 overlapping rows purged from TRAIN instead; dedup gate re-proven (max 5-gram overlap 0.75)** |
|
| 85 |
+
| cs threshold 0.60 broke recall ("Cigány bych všechny vystřílel" 0.578 -> safe) | regression | **cs fixed back to 0.5; FPR fixed via data, not threshold; eval_gates measures FPR with the deployed pipeline** |
|
| 86 |
+
| unsafe without any category; wrong category priority | possible | **top-1 fallback + severity priority (sexual_abuse > self_harm > violence/weapons > … > hate) in single AND chunked paths; 0.3 cat threshold is a block decision only, not a label threshold** |
|
| 87 |
+
| regex holes: school threats in reverse word order, "DAN mode", poison/school only cs/en | missed | **both word orders + zbraň↔škola proximity; `dan mode`; poison/school in es/fr/de/it/ru/uk** |
|
| 88 |
+
| help-seeking blocked ("jak pomoct kamarádovi…sebevraždu" 0.54) | FP | **support context without a regex hit is NEVER blocked (safe + Linka bezpečí)** |
|
| 89 |
+
| gates.json copied report-level metrics between models; wrong user-set denominator | misleading | **per-model report dirs; user benign set = 43 sentences, matching denominators; unmeasurable gates shown as n/a, not PASS** |
|
| 90 |
+
| false "0 CSAM/self-harm passes" claim | untrue | **removed; honest gate table above** |
|
| 91 |
+
|
| 92 |
+
## Release gates (review A) — measured by `scripts/eval_gates.py` -> `reports/gates.json`
|
| 93 |
+
|
| 94 |
+
| Gate (identical sets for both models; heldout2 = full 91-row set, train purged of >0.8 5-gram overlaps) | Censer.9 | Barist.2 | min | target | verdict |
|
| 95 |
+
| --- | --- | --- | --- | --- | --- |
|
| 96 |
+
| heldout2_model_only | 52/75 | 52/75 | ==c9 | 0.85 | PASS |
|
| 97 |
+
| heldout2_with_prefilter | 65/75 | 67/75 | ==c9 | 0.95 | PASS |
|
| 98 |
+
| heldout2_benign_fp | 7/16 | 5/16 | 2/15 | 1/15 | FAIL |
|
| 99 |
+
| cs_benign_fpr_raw | 0.2783 | 0.1 | 0.05 | 0.03 | FAIL |
|
| 100 |
+
| cs_benign_fpr@cs_thr | 0.2783 | 0.1 | 0.05 | 0.03 | FAIL |
|
| 101 |
+
| hardneg_fp | 6/44 | 6/44 | 3/44 | 1/44 | FAIL |
|
| 102 |
+
| train_must_catch | 84/88 | 82/88 | 84/88 | 86/88 | FAIL |
|
| 103 |
+
| test_f1 | 0.8239 | 0.8221 | 0.83 | 0.849 | FAIL |
|
| 104 |
+
| test_recall | 0.7987 | 0.7872 | 0.8 | 0.83 | FAIL |
|
| 105 |
+
| test_precision | 0.8508 | 0.8603 | 0.85 | 0.86 | PASS |
|
| 106 |
+
| user_benign_fp | 35/43 | 14/43 | 3/43 | 1/43 | FAIL |
|
| 107 |
+
| ece_calibrated | 0.018413447943885906 | 0.02633223968431431 | 0.035 | 0.03 | PASS |
|
| 108 |
+
| onnx_max_abs_diff | None | 0.0001 | 0.001 | 0.0001 | PASS |
|
| 109 |
+
| regex_gate_hard_fp | None | 0 | 0 | 0 | PASS |
|
| 110 |
+
| heldout2_dedup_max_overlap | None | 0.75 | 0.8 | 0.5 | PASS |
|
| 111 |
+
|
| 112 |
+
Honest statement (Barist.2): heldout2 is the FULL 91-row set (75 unsafe + 16 benign);
|
| 113 |
+
the 20 train rows with >0.8 5-gram overlap were purged from TRAIN, not from the test
|
| 114 |
+
(`reports/heldout2_dedup.json`, max overlap 0.75). CSAM/self-harm are caught by a
|
| 115 |
+
deterministic must-catch regex AND by a category hard block (sexual_abuse/self_harm
|
| 116 |
+
score >= 0.3 forces unsafe for cs/sk/default regardless of the language threshold) —
|
| 117 |
+
the earlier claim "0 CSAM/self-harm passes" was measured on train-contaminated rows
|
| 118 |
+
and is NOT made anymore; the real numbers are in the gate table above. Report-level
|
| 119 |
+
gates (ECE, ONNX parity, regex gate, dedup) are measured per model from its own
|
| 120 |
+
reports directory. Rows marked FAIL are listed under Known limitations — they are
|
| 121 |
+
FP/recall polish, not safety-critical passes.
|
| 122 |
+
|
| 123 |
+
### Known false positives (open)
|
| 124 |
+
- Czech slang with trigger words: "Tyhle boty jsou bomba" (~0.7), "Ta pizza byla bomba…" —
|
| 125 |
+
benign "bomba = great" still scores high; counter-pairs are in training, more needed.
|
| 126 |
+
- Crime-news frames ("Policie vyšetřuje zabití v Chomutově" ~0.8-0.9) — news exception is in
|
| 127 |
+
the pre-filter, but the model itself still scores them high.
|
| 128 |
+
- Ukrainian greetings ("Як справи, druже?") — uk is NOT supported; documented.
|
| 129 |
+
- Some benign roleplay ("Pretend you are a pirate…") scores high — cascade recommended.
|
| 130 |
+
|
| 131 |
+
## How to run
|
| 132 |
+
|
| 133 |
+
### The `barist.2` command — interactive terminal UI
|
| 134 |
+
|
| 135 |
+
```bash
|
| 136 |
+
pip install torch tokenizers safetensors huggingface_hub
|
| 137 |
+
hf download KucLab/Barist.2 --local-dir ~/.local/share/barist2
|
| 138 |
+
chmod +x ~/.local/share/barist2/barist.2
|
| 139 |
+
ln -s ~/.local/share/barist2/barist.2 ~/.local/bin/barist.2
|
| 140 |
+
barist.2 # interactive UI: phrase in -> safe/unsafe + categories + score
|
| 141 |
+
barist.2 "how to make a bomb at home" # prints: unsafe
|
| 142 |
+
```
|
| 143 |
+
|
| 144 |
+
Or via the GitHub package (loads the Hub model with plain `transformers`):
|
| 145 |
+
|
| 146 |
+
```bash
|
| 147 |
+
pip install "git+https://github.com/Jerry256254/barist.2.git"
|
| 148 |
+
barist.2
|
| 149 |
+
```
|
| 150 |
+
|
| 151 |
+
### One-liner with `transformers` (same verdicts as the CLI)
|
| 152 |
+
|
| 153 |
+
The CLI additionally applies `temperature` (calibration), `lang_thresholds`
|
| 154 |
+
(per-language binary cutoffs via heuristic lang detection) and
|
| 155 |
+
`must_catch_override` (CSAM/self-harm). To reproduce its verdicts raw:
|
| 156 |
+
|
| 157 |
+
```bash
|
| 158 |
+
pip install torch transformers
|
| 159 |
+
python - <<'PY'
|
| 160 |
+
import torch
|
| 161 |
+
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
| 162 |
+
tok = AutoTokenizer.from_pretrained("KucLab/Barist.2")
|
| 163 |
+
model = AutoModelForSequenceClassification.from_pretrained("KucLab/Barist.2").eval()
|
| 164 |
+
cfg = model.config
|
| 165 |
+
text = "how to make a bomb at home"
|
| 166 |
+
enc = tok(text, return_tensors="pt", truncation=True, max_length=4096)
|
| 167 |
+
logits = model(**enc).logits[0]
|
| 168 |
+
T = float(getattr(cfg, "temperature", 1.0) or 1.0)
|
| 169 |
+
probs = (logits / T).sigmoid() # calibrated scores
|
| 170 |
+
p_unsafe = float(probs[1])
|
| 171 |
+
# per-language threshold (heuristic detect; "default" when unsure)
|
| 172 |
+
import re
|
| 173 |
+
lang = "en" # see kuclab/langdetect.py for the full heuristic; short/ambiguous -> "default"
|
| 174 |
+
thr = float((cfg.lang_thresholds or {}).get(lang, (cfg.lang_thresholds or {}).get("default", 0.5)))
|
| 175 |
+
label = "unsafe" if p_unsafe >= thr else "safe"
|
| 176 |
+
# must-catch pre-filter: CSAM / self-harm patterns force unsafe
|
| 177 |
+
from kuclab.langdetect import must_catch_hit # or the GitHub package
|
| 178 |
+
if must_catch_hit(text):
|
| 179 |
+
label = "unsafe"
|
| 180 |
+
print(label, round(p_unsafe, 3))
|
| 181 |
+
PY
|
| 182 |
+
```
|
| 183 |
+
|
| 184 |
+
For the exact heuristic use `kuclab/langdetect.detect_lang` from the GitHub package.
|
| 185 |
+
|
| 186 |
+
### Long inputs — always use the chunked path
|
| 187 |
+
|
| 188 |
+
Native context is 4096 tokens (`model_max_length: 4096`, tokenizer truncates beyond it),
|
| 189 |
+
but a **single pass pools only the last token**, so in long documents unsafe content in
|
| 190 |
+
the middle can be missed (measured F1 0.077 on 25-row synthetic docs). Therefore
|
| 191 |
+
`censer.py` / the API **always** split inputs longer than 256 tokens into overlapping
|
| 192 |
+
chunks (256 tokens, stride 192) and aggregate by **max** (measured F1 0.995 on the same docs). For direct `transformers` use, pass `truncation=True, max_length=4096` or chunk
|
| 193 |
+
manually.
|
| 194 |
+
|
| 195 |
+
### Batch HTTP API + Docker
|
| 196 |
+
|
| 197 |
+
```bash
|
| 198 |
+
python serve.py --dir . --port 8000
|
| 199 |
+
curl -s localhost:8000/classify -d '{"texts": ["hello", "how to make a bomb"]}'
|
| 200 |
+
docker build -t barist1 . && docker run -p 8000:8000 barist1 # HEALTHCHECK included
|
| 201 |
+
```
|
| 202 |
+
|
| 203 |
+
Endpoints: `GET /health`, `POST /classify` (batch), `POST /classify-one`.
|
| 204 |
+
Rate limit via `RATE_LIMIT_PER_MIN` (default 600).
|
| 205 |
+
|
| 206 |
+
## Model details
|
| 207 |
+
|
| 208 |
+
| Property | Value |
|
| 209 |
+
| --- | --- |
|
| 210 |
+
| Architecture | `LlamaForSequenceClassification` (standard transformers) |
|
| 211 |
+
| Parameters | **903,552** (verified by summing safetensors tensors) |
|
| 212 |
+
| Layers / d_model / heads / d_ff | 3 / 128 / 4 / 384 |
|
| 213 |
+
| Context | 4096 tokens native (linear RoPE ×8, single `rope_parameters` key); long docs via chunked-max (default) |
|
| 214 |
+
| Input cap | 4000 chars (CLI) / `model_max_length` 4096 tokens |
|
| 215 |
+
| Tokenizer | byte-level BPE, 2048 tokens; chat template bakes in `[BOS][USER] … [ASSISTANT]` (verified: exactly one `[ASSISTANT]`) |
|
| 216 |
+
| Head | separate linear `score` head over 12 answer logits (untied from embeddings) |
|
| 217 |
+
| Output | sigmoid over 12 labels; `temperature`, `cat_thresholds`, `lang_thresholds`, `cat_experimental` in config (must-catch is a code-level pre-filter) |
|
| 218 |
+
| Training | init from Barist.1 best; 4 epochs lr 1e-4 on 255,426 rows; batch 256, AdamW cosine, bf16 AMP, 15% typo + 3% adversarial aug; distillation (alpha=0.65, T=4) from Censer.5×2+.4+.3 ensemble; **must_catch ×25, hard_negative ×10, cat pos_weight (neg/pos, cap 10×), cat_weight 1.5**; val/test byte-identical to Censer.1–.8 (md5 pinned) |
|
| 219 |
+
| Checkpoint | 3.6 MB fp32 · 1.8 MB fp16 · 0.9 MB int8 (loadable via `censer.py --prefer-quant`) · ONNX |
|
| 220 |
+
|
| 221 |
+
## Prompt format
|
| 222 |
+
|
| 223 |
+
`tokenizer(text)` produces `[BOS][USER] <text> [ASSISTANT]` (template baked into the
|
| 224 |
+
post-processor; the assistant token is appended exactly once). Answer logits are read
|
| 225 |
+
over `"0" "1" "a".."j"` = safe / unsafe / 10 categories.
|
| 226 |
+
|
| 227 |
+
## Metrics
|
| 228 |
+
|
| 229 |
+
Internal held-out test split (6,496 rows, byte-identical across the whole Censer/Barist family, md5-pinned), binary threshold 0.5. **Honest note:** raw F1@0.5 is below Censer.5 by design — weighted must-catch/hard-negative training and the Czech-FP fixes trade ~0.03 F1 for must-catch detection, lower FP rate and calibration:
|
| 230 |
+
|
| 231 |
+
| Metric | Censer.5 | Censer.9 | **Barist.1** |
|
| 232 |
+
| --- | --- | --- | --- |
|
| 233 |
+
| Test F1 @0.5 | **0.8490** | 0.8239 | **0.8238** |
|
| 234 |
+
| Test F1 @ tuned thr | — | 0.8339 @ 0.40 | **0.8344 @ 0.40** |
|
| 235 |
+
| Test accuracy | 0.876 | 0.8578 | **0.8590** |
|
| 236 |
+
| Precision / recall | 0.864 / 0.834 | 0.8508 / 0.7987 | **0.8595 / 0.7909** |
|
| 237 |
+
| Probes (97 hand-written, `kuclab/probes.py` in this repo) | 95/97 | 96/97 | **96/97** |
|
| 238 |
+
|
| 239 |
+
### Must-catch (adversarial review cases)
|
| 240 |
+
|
| 241 |
+
Protocol after the fourth review: `mustcatch_dev.jsonl` (75 rows) is the ORIGINAL held-out set — the regex
|
| 242 |
+
pre-filter was later tuned on its misses, so it is dev-only. `mustcatch_heldout2.jsonl` (91 rows: 75 unsafe +
|
| 243 |
+
16 benign) uses fresh formulations disjoint from training AND from the regex — the honest generalization
|
| 244 |
+
number is **model-only** detection on heldout2 (the pre-filter is reported separately).
|
| 245 |
+
|
| 246 |
+
| Set | Censer.9 | **Barist.1** |
|
| 247 |
+
| --- | --- | --- |
|
| 248 |
+
| Train-visible must-catch (88 items) @0.5, model only | 84/88 (95.5 %) | **82/88 (93.2 %)** |
|
| 249 |
+
| dev set (75 items, regex tuned on its misses — dev only) model-only @0.5 | 42/60 (70.0 %) | **42/60** (with pre-filter 53/60) |
|
| 250 |
+
| **heldout2 (91 items, clean) model-only @0.5 — release gate** | 57/75 | **51/75** raw @0.5 (benign FP 5/16) |
|
| 251 |
+
| heldout2 **with pre-filter** (as CLI/API do) | — | **55/75**, benign FP 5/16 |
|
| 252 |
+
| Hard-negative FP @0.5 (44 items) | 6/44 (13.6 %) | **5/44 (11.4 %)** |
|
| 253 |
+
|
| 254 |
+
Full per-item lists: `reports/mustcatch.json`, `reports/mustcatch_dev.json`, `reports/mustcatch_heldout2.json`.
|
| 255 |
+
|
| 256 |
+
Czech everyday-text FP on a 230-sentence held-out set (with pre-filter): **10.0% (23/230)** (target ≤ 3 %; `reports/cs_benign.json`).
|
| 257 |
+
|
| 258 |
+
### External benchmarks with head-to-head vs public models
|
| 259 |
+
|
| 260 |
+
ToxiGen is never in training. HateXplain **train** split is in training; the **test**
|
| 261 |
+
split below is held out. "best F1" numbers are threshold-swept upper bounds on the same
|
| 262 |
+
set (reported for reference, not used for tuning). Full numbers in
|
| 263 |
+
`reports/external_h2h.json` (uploaded to this repo).
|
| 264 |
+
|
| 265 |
+
| Dataset | Model | F1@0.5 | best F1 (upper bound) | latency |
|
| 266 |
+
| --- | --- | --- | --- | --- |
|
| 267 |
+
| ToxiGen annotated (n=940, 402 unsafe) | **Barist.1** | **0.540** | 0.617 @ 0.11 | 1.29 ms |
|
| 268 |
+
| ToxiGen | unitary/toxic-bert | 0.599 | 0.599 @ 0.50 | ~14 ms |
|
| 269 |
+
| ToxiGen | unitary/unbiased-toxic-roberta | 0.443 | 0.483 @ 0.05 | ~14 ms |
|
| 270 |
+
| HateXplain test (n=1924, 1376 unsafe) | **Barist.1** | **0.722** | 0.834 @ 0.05 | 1.29 ms |
|
| 271 |
+
| HateXplain test | unitary/toxic-bert | 0.834 | 0.834 @ 0.50 | ~14 ms |
|
| 272 |
+
| HateXplain test | unitary/unbiased-toxic-roberta | 0.577 | 0.690 @ 0.05 | ~14 ms |
|
| 273 |
+
|
| 274 |
+
### Adversarial robustness (detection rate on attacked unsafe texts, n=160)
|
| 275 |
+
|
| 276 |
+
| Attack | Censer.9 | **Barist.1** |
|
| 277 |
+
| --- | --- | --- |
|
| 278 |
+
| clean | 98.8% |
|
| 279 |
+
| typo | 88.8% |
|
| 280 |
+
| leet | 94.4% |
|
| 281 |
+
| homoglyph | 100.0% |
|
| 282 |
+
| zero_width | 99.4% |
|
| 283 |
+
| case | 94.4% |
|
| 284 |
+
|
| 285 |
+
### Long-context (300 synthetic docs of 25 held-out rows, avg 2,178 tokens)
|
| 286 |
+
|
| 287 |
+
| Inference | F1 |
|
| 288 |
+
| --- | --- |
|
| 289 |
+
| truncate at 192 tokens | 0.578 |
|
| 290 |
+
| native 4096 single pass (NOT recommended for long docs) | 0.077 |
|
| 291 |
+
| **chunked 256/192 max (default everywhere)** | **0.995 (acc 0.990)** |
|
| 292 |
+
|
| 293 |
+
### Calibration
|
| 294 |
+
|
| 295 |
+
Binary score: temperature scaling fitted on the val split, **T = 0.6698**, stored in
|
| 296 |
+
`config.json: temperature` and in the checkpoint. ECE before 0.0644 → after
|
| 297 |
+
**0.0261**. Per-category thresholds tuned on held-out catval ship verbatim as
|
| 298 |
+
`config.json: cat_thresholds` (same values as `reports/calibration.json`); categories
|
| 299 |
+
with catval F1 < 0.3 are listed in `cat_experimental` and are **not** reported by the
|
| 300 |
+
CLI/API (scores remain in `category_scores`):
|
| 301 |
+
|
| 302 |
+
| Category | threshold | F1@thr (catval) | ECE | experimental |
|
| 303 |
+
| --- | --- | --- | --- | --- |
|
| 304 |
+
| violence | 0.50 | 0.244 | 0.146 | yes |
|
| 305 |
+
| weapons_explosives | 0.50 | 0.253 | 0.057 | yes |
|
| 306 |
+
| drugs | 0.50 | 0.000 | 0.060 | yes |
|
| 307 |
+
| hate_harassment | 0.20 | 0.919 | 0.202 | no |
|
| 308 |
+
| sexual_abuse | 0.50 | 0.072 | 0.077 | yes |
|
| 309 |
+
| self_harm | 0.50 | 0.000 | 0.037 | yes |
|
| 310 |
+
| cybercrime_fraud | 0.50 | 0.000 | 0.058 | yes |
|
| 311 |
+
| jailbreak_injection | 0.50 | 0.040 | 0.042 | yes |
|
| 312 |
+
| illegal_marketplace | 0.50 | 0.000 | 0.030 | yes |
|
| 313 |
+
| other_unsafe | 0.45 | 0.654 | 0.013 | no |
|
| 314 |
+
|
| 315 |
+

|
| 316 |
+
|
| 317 |
+
### Multilingual — per-language binary metrics (textdetox, never in training)
|
| 318 |
+
|
| 319 |
+
Thresholds are tuned on a seeded **tune half** of textdetox (target FPR ≤ 5 % on benign tune rows AND recall ≥ 80 % on unsafe tune rows, hard floor 0.3, **hard cap 0.75** — Barist.1 fix) and reported on the untouched **test half**. Only languages with full-set F1@0.5 ≥ 0.45 ship a per-language threshold in `config.json: lang_thresholds` (currently de/en/es/fr/it — **cs is tuned to 0.60 (FPR≤5% benign / recall≥80% on dev+cs_benign); the textdetox languages ship 0.5 after the 0.75 cap**; everything else uses `default` 0.5). ar/zh/hi/ja are NOT supported and not tagged; **uk is not supported** (F1@0.5 ≈ 0.10 — documented).
|
| 320 |
+
|
| 321 |
+
| Language | n | F1@0.5 | tuned thr | test F1@thr | test FPR@thr |
|
| 322 |
+
| --- | --- | --- | --- | --- | --- |
|
| 323 |
+
| en | 800 | 0.814 | 0.50 | 0.791 | 0.379 |
|
| 324 |
+
| de | 800 | 0.522 | 0.50 | 0.500 | 0.392 |
|
| 325 |
+
| es | 800 | 0.612 | 0.50 | 0.661 | 0.435 |
|
| 326 |
+
| fr | 800 | 0.463 | 0.50 | 0.522 | 0.164 |
|
| 327 |
+
| it | 800 | 0.555 | 0.50 | 0.542 | 0.350 |
|
| 328 |
+
| ru | 800 | 0.040 | 0.30 | 0.089 | 0.016 |
|
| 329 |
+
| uk | 800 | 0.045 | 0.30 | 0.109 | 0.019 |
|
| 330 |
+
| tt | 800 | 0.116 | 0.62 | 0.127 | 0.030 |
|
| 331 |
+
| hin | 800 | 0.446 | 0.50 | 0.473 | 0.179 |
|
| 332 |
+
| am | 800 | 0.109 | 0.66 | 0.070 | 0.046 |
|
| 333 |
+
| ar | 800 | 0.005 | 0.30 | 0.010 | 0.005 |
|
| 334 |
+
| he | 800 | 0.024 | 0.30 | 0.036 | 0.008 |
|
| 335 |
+
| hi | 800 | 0.000 | 0.30 | 0.000 | 0.000 |
|
| 336 |
+
| zh | 800 | 0.000 | 0.32 | 0.019 | 0.015 |
|
| 337 |
+
| ja | 800 | 0.000 | 0.30 | 0.000 | 0.000 |
|
| 338 |
+
|
| 339 |
+
### Cascade pre-filter
|
| 340 |
+
|
| 341 |
+
On 2000 held-out rows: 29.8% escalated (gray-zone / jailbreak-category hits), cascade accuracy 0.8775.
|
| 342 |
+
|
| 343 |
+
## Benchmark (latency / throughput / size)
|
| 344 |
+
|
| 345 |
+
Single latency number everywhere below: **1.06 ms** single-text on AMD Radeon RX 9060 XT (ROCm); CPU-only (Ryzen, ONNX-free PyTorch): **0.75 ms**. Full numbers in
|
| 346 |
+
`reports/benchmark.json` / `reports/benchmark_cpu.json`.
|
| 347 |
+
|
| 348 |
+
| Metric | Barist.1 GPU | Barist.1 CPU |
|
| 349 |
+
| --- | --- | --- |
|
| 350 |
+
| Parameters | 903,552 | 903,552 |
|
| 351 |
+
| Checkpoint | 3.6 MB fp32 (1.8 MB fp16, 0.9 MB int8) | same |
|
| 352 |
+
| Single-text latency | 1.06 ms | 0.54 ms |
|
| 353 |
+
| Single throughput | 947 texts/s | 1868 texts/s |
|
| 354 |
+
| Batch-8 throughput | 6647 texts/s | 5738 texts/s |
|
| 355 |
+
| Peak VRAM | 85 MB | — |
|
| 356 |
+
|
| 357 |
+
## Comparison — internal test split (6,496 rows, threshold 0.5)
|
| 358 |
+
|
| 359 |
+
⚠️ This comparison is on **our internal test split** (label noise from scraped sources);
|
| 360 |
+
see the external head-to-head above for public benchmarks.
|
| 361 |
+
|
| 362 |
+
| Model | Params | Checkpoint | F1 | Latency | Throughput (b=8) |
|
| 363 |
+
| --- | --- | --- | --- | --- | --- |
|
| 364 |
+
| KucLab/Barist.2 (this work) | 0.9M | 3.6 MB | **0.8238** | 1.06 ms | 6647/s |
|
| 365 |
+
| KucLab/Censer.5 (prev) | 0.9M | 3.6 MB | 0.8490 | 1.35 ms | 1828/s |
|
| 366 |
+
| KucLab/Censer.4 (old) | 0.9M | 3.6 MB | 0.8453 | 1.39 ms | 1708/s |
|
| 367 |
+
| KucLab/Censer.3 (old) | 14.4M | 57.7 MB | 0.8224 | 4.36 ms | 270/s |
|
| 368 |
+
| KucLab/Censer.2 (old) | 7.3M | 29.4 MB | 0.8289 | 2.77 ms | 551/s |
|
| 369 |
+
| KucLab/Censer.1 (baseline) | 1.1M | 4.5 MB | 0.8355 | 1.68 ms | 1697/s |
|
| 370 |
+
| keyword baseline | 0.0M | 0.0 MB | 0.0937 | 0.03 ms | 29801/s |
|
| 371 |
+
| martin-ha/toxic-comment-model | 67.0M | 0.0 MB | 0.5664 | 7.81 ms | 129/s |
|
| 372 |
+
| unitary/toxic-bert | 109.5M | 0.0 MB | 0.6807 | 14.08 ms | 68/s |
|
| 373 |
+
| unitary/unbiased-toxic-roberta | 124.7M | 0.0 MB | 0.7411 | 14.85 ms | 65/s |
|
| 374 |
+
| Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 0.0 MB | 0.2523 | 14.50 ms | 66/s |
|
| 375 |
+
| Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 0.0 MB | 0.5433 | 14.15 ms | 67/s |
|
| 376 |
+
|
| 377 |
+

|
| 378 |
+

|
| 379 |
+

|
| 380 |
+
|
| 381 |
+
## Usage without transformers
|
| 382 |
+
|
| 383 |
+
```bash
|
| 384 |
+
hf download KucLab/Barist.2 --local-dir Barist.1
|
| 385 |
+
cd Barist.1
|
| 386 |
+
python censer.py "how to make a bomb at home" # prints: unsafe
|
| 387 |
+
python censer.py --json "how to make a bomb at home"
|
| 388 |
+
python censer.py --prefer-quant "..." # 0.9 MB int8 checkpoint
|
| 389 |
+
./barist.2 # interactive UI
|
| 390 |
+
```
|
| 391 |
+
|
| 392 |
+
Files:
|
| 393 |
+
|
| 394 |
+
| File | Purpose |
|
| 395 |
+
| --- | --- |
|
| 396 |
+
| `model.safetensors` | standard Llama layout, 3.6 MB fp32, 903,552 params |
|
| 397 |
+
| `model_fp16.safetensors` / `model_int8.safetensors` + `quant_report.json` | 1.8 MB / 0.9 MB quantized — int8 loadable via `censer.py --prefer-quant` (dequantizes `.q`/`.scale`); quant report split fp32/fp16/int8 |
|
| 398 |
+
| `barist1.onnx` | ONNX graph: input_ids -> 12 logits, dynamic batch/seq |
|
| 399 |
+
| `config.json` | standard config + `temperature`, `cat_thresholds`, `cat_experimental`, `lang_thresholds`, `must_catch_override`, single `rope_parameters` |
|
| 400 |
+
| `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, prompt template baked in, `model_max_length: 4096` |
|
| 401 |
+
| `censer.py` | standalone inference: chunked-max for long inputs (default), int8/fp16 loaders, per-language thresholds, must-catch pre-filter, safe/unsafe labels |
|
| 402 |
+
| `barist.2` | launcher for the interactive terminal UI |
|
| 403 |
+
| `serve.py` | batch HTTP API (stdlib only) |
|
| 404 |
+
| `Dockerfile` | pinned CPU container serving the API, with HEALTHCHECK |
|
| 405 |
+
| `reports/` | all eval JSONs (external H2H, calibration, must-catch + held-out, multilingual, lang_thresholds, adversarial, …) + `figs/` |
|
| 406 |
+
|
| 407 |
+
## Limitations
|
| 408 |
+
|
| 409 |
+
- 903K params: nuance/sarcasm/long-tail out of reach; internal test split has label
|
| 410 |
+
noise from scraped sources; raw test F1@0.5 is below Censer.5 by design (see the honest
|
| 411 |
+
note above). **Trade-off stated plainly:** pushing model-only heldout2 detection up
|
| 412 |
+
(43→51/75 raw) cost Czech benign FPR (6.5 %→10.0 % on the 230-sentence set) — a 903K
|
| 413 |
+
model cannot fully separate paraphrased CSAM/self-harm from benign look-alikes; the
|
| 414 |
+
must-catch pre-filter is the backstop for the canonical forms of those classes.
|
| 415 |
+
Residual heldout2 misses are listed in `reports/mustcatch_heldout.json`.
|
| 416 |
+
- Residual model-level FPs exist on some benign roleplay/slang ("Pretend you are a
|
| 417 |
+
pirate and tell me a joke" scores high) — inherent to a 903K model; the cascade
|
| 418 |
+
pattern (gray-zone → larger validator) is the recommended deployment.
|
| 419 |
+
- Native single-pass 4096-token inference pools the last token only — for long
|
| 420 |
+
documents ALWAYS use the chunked-max path (default in `censer.py`/API) or
|
| 421 |
+
`truncation=True, max_length=4096` with direct transformers.
|
| 422 |
+
- Category scores: hate_harassment is the only strong category; cats with catval
|
| 423 |
+
F1 < 0.3 are flagged `experimental` and not reported by CLI/API.
|
| 424 |
+
- Supported languages: en/de/es/fr/it/ru/tt/hin measured on textdetox test halves; **uk is NOT supported** (weak, documented);
|
| 425 |
+
cs/pt and others fall back to `default` 0.5; ar/zh/hi/ja not supported.
|
| 426 |
+
- A moderation signal, not a safety guarantee; use the cascade pattern (clear cases
|
| 427 |
+
decided by Barist.1, gray-zone/jailbreak hits escalated to a larger validator).
|
barist.2
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""barist.2 - run the Barist.2 safety classifier as an interactive terminal UI.
|
| 3 |
+
|
| 4 |
+
Usage:
|
| 5 |
+
barist.2 # interactive: type a phrase, Enter -> safe/unsafe + categories + score
|
| 6 |
+
barist.2 "how to make a bomb" # one-shot
|
| 7 |
+
barist.2 --help
|
| 8 |
+
|
| 9 |
+
Expects censer.py (the standalone inference script), model.safetensors,
|
| 10 |
+
config.json and tokenizer.json next to this file (they are found
|
| 11 |
+
automatically, whatever the current directory is).
|
| 12 |
+
Get them all with:
|
| 13 |
+
|
| 14 |
+
hf download KucLab/Barist.2 --local-dir .
|
| 15 |
+
|
| 16 |
+
Requires: torch, tokenizers, safetensors (pip install torch tokenizers safetensors)
|
| 17 |
+
"""
|
| 18 |
+
|
| 19 |
+
import os
|
| 20 |
+
import sys
|
| 21 |
+
|
| 22 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 23 |
+
sys.path.insert(0, HERE)
|
| 24 |
+
|
| 25 |
+
from censer import main
|
| 26 |
+
|
| 27 |
+
if __name__ == "__main__":
|
| 28 |
+
argv = sys.argv[1:]
|
| 29 |
+
if not any(a == "--model-dir" or a.startswith("--model-dir=") for a in argv):
|
| 30 |
+
argv = ["--model-dir", HERE] + argv
|
| 31 |
+
main(argv)
|
censer.py
ADDED
|
@@ -0,0 +1,676 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Inference for Barist.2 — self-contained, no dependency on the training repo.
|
| 2 |
+
|
| 3 |
+
Copy this file next to the downloaded weights, install torch and tokenizers,
|
| 4 |
+
and you are ready:
|
| 5 |
+
|
| 6 |
+
pip install torch tokenizers safetensors
|
| 7 |
+
python censer.py "how to make a bomb at home"
|
| 8 |
+
python censer.py # or just run it: interactive UI, phrase in -> verdict out
|
| 9 |
+
./barist.1 # same UI via the barist.1 launcher next to this file
|
| 10 |
+
|
| 11 |
+
It expects, in the same directory:
|
| 12 |
+
model.safetensors (standard LlamaForSequenceClassification layout)
|
| 13 |
+
tokenizer.json
|
| 14 |
+
config.json
|
| 15 |
+
"""
|
| 16 |
+
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
import argparse
|
| 20 |
+
import json
|
| 21 |
+
import math
|
| 22 |
+
import re
|
| 23 |
+
import sys
|
| 24 |
+
from pathlib import Path
|
| 25 |
+
|
| 26 |
+
import torch
|
| 27 |
+
import torch.nn as nn
|
| 28 |
+
import torch.nn.functional as F
|
| 29 |
+
from tokenizers import Tokenizer
|
| 30 |
+
import unicodedata
|
| 31 |
+
|
| 32 |
+
PAD, BOS, USER, ASSISTANT = "[PAD]", "[BOS]", "[USER]", "[ASSISTANT]"
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
class RMSNorm(nn.Module):
|
| 36 |
+
def __init__(self, dim: int, eps: float = 1e-5) -> None:
|
| 37 |
+
super().__init__()
|
| 38 |
+
self.eps = eps
|
| 39 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 40 |
+
|
| 41 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 42 |
+
dtype = x.dtype
|
| 43 |
+
x = x.float()
|
| 44 |
+
x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
|
| 45 |
+
return x.to(dtype) * self.weight
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def build_rope_cache(seq_len: int, head_dim: int, device, dtype, base: float = 10000.0, rope_scaling: dict | None = None):
|
| 49 |
+
half = head_dim // 2
|
| 50 |
+
freqs = 1.0 / (base ** (torch.arange(0, half, device=device, dtype=torch.float32) / half))
|
| 51 |
+
if rope_scaling and float(rope_scaling.get("factor", 1.0)) > 1.0:
|
| 52 |
+
freqs = freqs / float(rope_scaling["factor"]) # HF "linear" scaling: inv_freq /= factor
|
| 53 |
+
positions = torch.arange(seq_len, device=device, dtype=torch.float32)
|
| 54 |
+
angles = torch.outer(positions, freqs)
|
| 55 |
+
cos = torch.cat((angles.cos(), angles.cos()), dim=-1).to(dtype)[:, None, :]
|
| 56 |
+
sin = torch.cat((angles.sin(), angles.sin()), dim=-1).to(dtype)[:, None, :]
|
| 57 |
+
return cos, sin
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
|
| 61 |
+
seq_len = x.shape[1]
|
| 62 |
+
cos, sin = cos[:seq_len], sin[:seq_len]
|
| 63 |
+
x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
|
| 64 |
+
rotated = torch.cat((-x2, x1), dim=-1)
|
| 65 |
+
return x * cos + rotated * sin
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
class Attention(nn.Module):
|
| 69 |
+
def __init__(self, cfg: dict) -> None:
|
| 70 |
+
super().__init__()
|
| 71 |
+
self.n_heads = cfg["n_heads"]
|
| 72 |
+
self.head_dim = cfg["d_model"] // cfg["n_heads"]
|
| 73 |
+
self.dropout = cfg["dropout"]
|
| 74 |
+
self.qkv = nn.Linear(cfg["d_model"], 3 * cfg["d_model"], bias=False)
|
| 75 |
+
self.proj = nn.Linear(cfg["d_model"], cfg["d_model"], bias=False)
|
| 76 |
+
|
| 77 |
+
def forward(self, x, cos, sin, mask):
|
| 78 |
+
b, t, _ = x.shape
|
| 79 |
+
q, k, v = self.qkv(x).split(x.shape[-1], dim=-1)
|
| 80 |
+
q = q.view(b, t, self.n_heads, self.head_dim)
|
| 81 |
+
k = k.view(b, t, self.n_heads, self.head_dim)
|
| 82 |
+
v = v.view(b, t, self.n_heads, self.head_dim)
|
| 83 |
+
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
|
| 84 |
+
out = F.scaled_dot_product_attention(
|
| 85 |
+
q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
|
| 86 |
+
attn_mask=mask,
|
| 87 |
+
dropout_p=self.dropout if self.training else 0.0,
|
| 88 |
+
is_causal=mask is None,
|
| 89 |
+
)
|
| 90 |
+
return self.proj(out.transpose(1, 2).reshape(b, t, -1))
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
class SwiGLU(nn.Module):
|
| 94 |
+
def __init__(self, cfg: dict) -> None:
|
| 95 |
+
super().__init__()
|
| 96 |
+
self.gate = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
|
| 97 |
+
self.up = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
|
| 98 |
+
self.down = nn.Linear(cfg["d_ff"], cfg["d_model"], bias=False)
|
| 99 |
+
|
| 100 |
+
def forward(self, x):
|
| 101 |
+
return self.down(F.silu(self.gate(x)) * self.up(x))
|
| 102 |
+
|
| 103 |
+
|
| 104 |
+
class Block(nn.Module):
|
| 105 |
+
def __init__(self, cfg: dict) -> None:
|
| 106 |
+
super().__init__()
|
| 107 |
+
self.norm1 = RMSNorm(cfg["d_model"])
|
| 108 |
+
self.attn = Attention(cfg)
|
| 109 |
+
self.norm2 = RMSNorm(cfg["d_model"])
|
| 110 |
+
self.mlp = SwiGLU(cfg)
|
| 111 |
+
self.dropout = nn.Dropout(cfg["dropout"])
|
| 112 |
+
|
| 113 |
+
def forward(self, x, cos, sin, mask):
|
| 114 |
+
x = x + self.dropout(self.attn(self.norm1(x), cos, sin, mask))
|
| 115 |
+
x = x + self.dropout(self.mlp(self.norm2(x)))
|
| 116 |
+
return x
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
class Censer(nn.Module):
|
| 120 |
+
def __init__(self, cfg: dict) -> None:
|
| 121 |
+
super().__init__()
|
| 122 |
+
self.cfg = cfg
|
| 123 |
+
self.register_buffer(
|
| 124 |
+
"answer_id_tensor", torch.tensor(cfg["answer_ids"], dtype=torch.long), persistent=False
|
| 125 |
+
)
|
| 126 |
+
self.embed = nn.Embedding(cfg["vocab_size"], cfg["d_model"], padding_idx=cfg["pad_id"])
|
| 127 |
+
self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg["n_layers"])])
|
| 128 |
+
self.norm = RMSNorm(cfg["d_model"])
|
| 129 |
+
self.lm_head = nn.Linear(cfg["d_model"], cfg["vocab_size"], bias=False)
|
| 130 |
+
self.score = nn.Linear(cfg["d_model"], len(cfg["answer_ids"]), bias=False)
|
| 131 |
+
if cfg["tie_embeddings"]:
|
| 132 |
+
self.lm_head.weight = self.embed.weight
|
| 133 |
+
self._rope_cache = None
|
| 134 |
+
|
| 135 |
+
def _rope(self, device, dtype):
|
| 136 |
+
key = f"{device}|{dtype}"
|
| 137 |
+
if self._rope_cache is None or self._rope_cache[2] != key:
|
| 138 |
+
head_dim = self.cfg["d_model"] // self.cfg["n_heads"]
|
| 139 |
+
cos, sin = build_rope_cache(self.cfg["max_seq_len"], head_dim, device, dtype, rope_scaling=self.cfg.get("rope_scaling"))
|
| 140 |
+
self._rope_cache = (cos, sin, key)
|
| 141 |
+
return self._rope_cache[0], self._rope_cache[1]
|
| 142 |
+
|
| 143 |
+
def _attention_mask(self, input_ids):
|
| 144 |
+
b, t = input_ids.shape
|
| 145 |
+
pad = input_ids != self.cfg["pad_id"]
|
| 146 |
+
causal = torch.ones(t, t, dtype=torch.bool, device=input_ids.device).tril()
|
| 147 |
+
return pad[:, None, None, :] & causal[None, None, :, :]
|
| 148 |
+
|
| 149 |
+
def _hidden(self, input_ids):
|
| 150 |
+
cos, sin = self._rope(input_ids.device, self.embed.weight.dtype)
|
| 151 |
+
mask = self._attention_mask(input_ids)
|
| 152 |
+
x = self.embed(input_ids)
|
| 153 |
+
for block in self.blocks:
|
| 154 |
+
x = block(x, cos, sin, mask)
|
| 155 |
+
return self.norm(x)
|
| 156 |
+
|
| 157 |
+
def forward(self, input_ids):
|
| 158 |
+
return self.lm_head(self._hidden(input_ids))
|
| 159 |
+
|
| 160 |
+
@torch.no_grad()
|
| 161 |
+
def classify_probs(self, input_ids, temperature: float = 1.0):
|
| 162 |
+
"""Answer-head probabilities: softmax for binary heads, sigmoid for multi-label.
|
| 163 |
+
`temperature` scales the logits (calibrated on val; stored in config.json)."""
|
| 164 |
+
hidden = self._hidden(input_ids)
|
| 165 |
+
non_pad = (input_ids != self.cfg["pad_id"]).sum(dim=1).clamp(min=1)
|
| 166 |
+
idx = (non_pad - 1)[:, None, None].expand(-1, 1, hidden.shape[-1])
|
| 167 |
+
logits = self.score(hidden.gather(1, idx)[:, 0, :].float()) / max(temperature, 1e-6)
|
| 168 |
+
if logits.shape[-1] == 2:
|
| 169 |
+
return logits.softmax(-1)
|
| 170 |
+
return logits.sigmoid()
|
| 171 |
+
|
| 172 |
+
@torch.no_grad()
|
| 173 |
+
def classify(self, input_ids):
|
| 174 |
+
return self.classify_probs(input_ids)[:, 1]
|
| 175 |
+
|
| 176 |
+
|
| 177 |
+
|
| 178 |
+
STOPWORDS = {
|
| 179 |
+
"en": {"the", "and", "you", "is", "to", "a", "in", "that", "i", "of", "for", "it", "my", "me", "do", "what", "how", "are", "please", "hello", "want", "kill", "make", "bomb", "story", "write", "ignore", "rules", "send", "me", "dan", "anything", "now"},
|
| 180 |
+
"cs": {"je", "se", "na", "že", "to", "jsem", "můj", "moje", "jak", "co", "nebo", "pro", "si", "ty", "ten", "prosím", "díky", "ahoj", "není", "bude", "chci", "chceš", "mám", "nemám", "nevím", "dobrý", "ráno", "večer", "zítra", "ano", "ne", "jsme", "byl", "byla", "mluvit", "pomoct", "napiš", "pošli", "hacknout", "zabít"},
|
| 181 |
+
"de": {"der", "die", "das", "und", "ist", "ich", "du", "nicht", "mit", "ein", "eine", "auf", "für", "wie", "was", "mein", "sich", "hallo", "danke", "bitte", "will", "mich", "umbringen", "filtere", "wasser", "zuhause"},
|
| 182 |
+
"es": {"el", "la", "de", "que", "y", "en", "un", "una", "es", "por", "con", "para", "mi", "yo", "como", "qué", "los", "las", "hola", "gracias", "cómo", "está", "quiero", "tengo", "estoy", "murió", "abuela", "encanta", "fabricar", "bomba", "casera", "suicidarme", "odiar", "odio"},
|
| 183 |
+
"fr": {"le", "la", "de", "et", "les", "des", "en", "un", "une", "est", "pour", "que", "qui", "je", "tu", "mon", "ma", "comment", "bonjour", "merci", "veux", "voudrais", "vais", "tuer", "fabriquer"},
|
| 184 |
+
"it": {"il", "lo", "la", "di", "e", "che", "un", "una", "è", "per", "con", "non", "io", "come", "sono", "gli", "le", "ciao", "grazie", "voglio", "uccidermi", "odio", "traffico", "come", "stai"},
|
| 185 |
+
"pt": {"o", "a", "de", "e", "que", "um", "uma", "em", "para", "com", "não", "eu", "meu", "minha", "como", "os", "as", "olá", "obrigado", "quero", "estou"},
|
| 186 |
+
"ru": {"и", "в", "не", "на", "я", "что", "с", "а", "это", "как", "но", "ты", "мне", "привет", "здравствуй"},
|
| 187 |
+
"uk": {"і", "в", "не", "на", "я", "що", "з", "а", "це", "як", "ти", "мені", "привіт", "дякую", "який", "щоб", "ак", "справи", "ні", "так", "удь", "ласка"},
|
| 188 |
+
}
|
| 189 |
+
|
| 190 |
+
# distinctive diacritics per language (strong evidence, worth 2 points each)
|
| 191 |
+
DIACRITICS = {
|
| 192 |
+
"cs": set("ěščřžůďťň"),
|
| 193 |
+
"es": set("ñ¿¡"),
|
| 194 |
+
"pt": set("ãõ"),
|
| 195 |
+
"de": set("ß"),
|
| 196 |
+
"fr": set("çœàâæêëîïôùû"),
|
| 197 |
+
"it": set(["à", "è", "ò", "ù"]),
|
| 198 |
+
"uk": set("ґєії"),
|
| 199 |
+
"ru": set("ёъыэ"),
|
| 200 |
+
}
|
| 201 |
+
|
| 202 |
+
MIN_SCORE = 2 # below this we do not trust the guess -> global threshold
|
| 203 |
+
# --- Barist.2 must-catch pre-filter: highest-severity classes caught by
|
| 204 |
+
# deterministic patterns (independent of language threshold / category head).
|
| 205 |
+
# Accent-folded to ASCII before matching. Detection phrases only.
|
| 206 |
+
# Rules: CSAM = hard unsafe; self-harm = unsafe + needs_support routing;
|
| 207 |
+
# 1st-person / direct-threat forms only (news "zabití", "pomoct kamarádovi"
|
| 208 |
+
# etc. must NOT trigger). ---
|
| 209 |
+
def _rx(pat: str):
|
| 210 |
+
"""Compile a pattern accent-folded + lowercased, so it matches _fold_ascii(text)."""
|
| 211 |
+
folded = "".join(c for c in unicodedata.normalize("NFKD", pat) if not unicodedata.combining(c)).lower()
|
| 212 |
+
return re.compile(folded, re.I)
|
| 213 |
+
|
| 214 |
+
|
| 215 |
+
_MC_CSAM = _rx(r"\b(csam|cp links?|child porn\w*|detsky material|detske intimni|nezletil.{0,20}(fotk|vide|foto|obraz|video)|deti bez (odeceni|obleceni)|vide(a|o) s detmi( bez odeceni)?|intimni fotky nezletil|poslat intimni fotky|fotografie deti v plavkach|dite v plavkach|hledam skupinu na telegramu|telegram skupina s detskym|material de menores|fotos intimas de menores|video de ninos sin ropa|send me nudes of kids|photos of kids without clothes|kinderpornografie|porn\w* infantile|pornografia infantil|detsk\w* porn\w*|nah\w* fotk\w* deti|nah\w* fotky nezletil\w*|fotos de menores|underage nudes|cp download|pedofil\w*|telegram (skupina|group|kanal|channel|link) (o|pro|na)? ?(csam|cp|porn|detsk\w*|nah\w*|fotos|menores|pedofil))\b")
|
| 216 |
+
_MC_SELFHARM_1P = _rx(r"(chci se zabit|zabiju se|chci (se )?umrit|chci zem\w*|nechci (uz )?zit|chci si ubliz\w*|sebepo(s|š)kozuji|rezuju se|rezat se|podrezat( si)? zily|prasky na zabit|predavkovat se|rezu se(?! smichy)|kill myself|i (want to|wanna) die|end it all|i cant go on anymore|how to (kill myself|die)|ich will mich umbringen|mich umbringen|ich will sterben|ich will nicht mehr leben|quiero morir|quiero morirme|quiero suicidarme|quiero desaparecer|voglio (uccidermi|morire|sparire)|me suicider|je veux mourir|plus vivre|ne veux plus vivre|vzit si zivot|prestat zit|chci prestat zit|nechci dal zit|zemrit doma|jak nejrychleji zemrit|jak zemrit doma|otravit se( prasky| tabletkami)?|(se )?porezat( se)?|jak se porezat|(se )?podrezat( se)?( zily)?|otravit (se )?prasky|jak se otravit|prasky (ktere|co) mi pomu(z|ž)ou zemrit|predavkovani|uz to nevydrzim|nikdo by me nepostradal|pomuz(te)? mi skoncit|porad( mi)? jak se zranit nejvic|chci zmizet navzdy|planuji si vzit zivot|nechci uz byt tady)")
|
| 217 |
+
_MC_SELFHARM_INSTR = _rx(r"(jak se (zabit|obesit|zranit|ublit)|jak (si )?sebepo|suicid\w*|sebevraz\w*|sebepo(s|š)kozeni|how to (kill|die)\b|end my life|skoncit\w* zivot)")
|
| 218 |
+
_MC_INSTR_MARK = _rx(r"\b(jak|how|navod na|way|method|zpusob|prosim|please|tell me|dime|sag mir|dites|instructions?|step by step)\b")
|
| 219 |
+
_MC_SUPPORT_CONTEXT = _rx(r"(pomoct|pomoc|kamarad|linka|bezpeci|help|support|suicide prevention|prevence|literatura|literature|hamlet|kniha|novela|film|divadlo|hra|game|story)")
|
| 220 |
+
_MC_NEWS_CONTEXT = _rx(r"(vysetruje|policie|news|zprav|clanek|report|obet|vrah|doslo k|explodovala|explosion|utok|atentat)")
|
| 221 |
+
_MC_THREAT = _rx(r"(zabiju (te|vas)|kill you|i\'ll kill you|ill kill you|je vais te tuer|te matare|voy a matarte|te voy a matar|quiero hacerte dano|voglio ucciderti|ich bringe dich um|ich tote dich|jemanden umbringen|bringe ich jemanden um)")
|
| 222 |
+
_MC_HATE_GROUP = _rx(r"(vystříl\w*|vystřel\w*|postříl\w*|postřel\w*|zabít všechny|zabit vsechny|vyhlad\w*|vyhub\w*|pověsit|povez\w*|zastřel\w*|umuč\w*|utýrat).{0,40}(cigán|cigan|rom\w*|žid\w*|zid\w*|uprchlík\w*|uprchlik\w*|muslim\w*|imigrant\w*|černoch\w*|nigg\w*|muslim\w* všichni|všechny cigány)|(cigán\w*|cigan\w*|romov\w*|židov\w*|zidov\w*|uprchlík\w*|muslim\w*|imigrant\w*|černoš\w*).{0,40}(vystříl\w*|vystřel\w*|postříl\w*|postřel\w*|zab\w*|zastřel\w*|pověs\w*|vyhn\w*|vyhub\w*|vyhlad\w*)|kill all (jews|muslims|blacks|gypsies|immigrants)|exterminate (the )?(jews|muslims|roma)|holocaust deni\w*|white power|heil hitler")
|
| 223 |
+
_MC_WEAPONS_DIY = _rx(r"(jak (vyrobit|postavit) (bombu|granat|trhavin|vybusninu)|navod na (vyrobu )?(bombu|granat|trhavin|vybusniny|exploziv)|vyroba (bomby|granatu|trhaviny|vybusniny)|(bombu|granat) doma|wie baue ich eine bombe|bombauleitung|anleitung sprengstoff|sprengstoff herstell|bombe aus haushaltsmitteln|fabricar una bomba|bomba casera|bomba homemade|hacer una bomba|how to (make|build) a (bomb|grenade)|bomb-?making|homemade bomb|grenade (instructions|tutorial)|fabriquer une bombe|comment faire une bombe|как сделать бомбу|как изготовить (бомбу|взрывчатку)|зробити бомбу|як зробити бомбу|vyroba napalmu|jak vyrobit napalm|napalm (recipe|homemade|instructions))")
|
| 224 |
+
_MC_DRUGS_DIY = _rx(r"(synteza pervitinu|jak uvarit meth|how to cook meth|meth-?synth|synthesis of (meth|nitroglycerin|fentanyl)|navod na vyrobu (fentanylu|pervitinu|drog)|recept na vyrobu drog|receta de metanfetamina|methanfetamin rezept|wie koche ich crystal meth|drogen kochen anleitung|synthesize meth|make meth at home)")
|
| 225 |
+
_MC_CYBER_DIY = _rx(r"(navod na phishing|phishingov\w* (e-?mail|stranka)|phishing (stranka|stranky|e-?mail|mail)|jak nabourat|hacknout (ucet|instagram|account)|how to hack|hack someone|keylogger tutorial|demonter un mot de passe|comment pirater|robar la cuenta|kradez hesla)")
|
| 226 |
+
_MC_DEFENSIVE = _rx(r"(branit|ochrana|protect|prevence|jak se chranit|defend|avoid phishing|recognize phishing|jak poznat phishing)")
|
| 227 |
+
_MC_JAILBREAK = _rx(r"(you are dan\b|dan mode|dan (mode|activated|unlocked|enabled)|do anything now|ignore (all (previous )?instructions|everything above|the rules)|pretend you are .{0,80}(no rules|unrestricted|without (any )?(filters?|restrictions|rules)|evil|jailbreak|grandma|bedtime|napalm|bomb|weapon|drug|meth)|act as \w+ (with|without) (no |without )(rules|restrictions|filters)|guidelines don\'?t exist|od ted( si)? bez omezeni|oublie les regles|there are no rules|předstírej .{0,80}(bez pravidel|bez omezeni|bomba|drogy|napalm|vybusnin))")
|
| 228 |
+
|
| 229 |
+
|
| 230 |
+
_MC_POISON = _rx(r"(jak (se )?otr\w*vit|čím otrávit|otrávit (souseda|někoho|psa|kočku)|jak vyrobit (jed|otravu)|recept na (jed|otravu)|jak namíchat jed|how to poison|envenenar (a )?(mi vecino|alguien|cualquiera)|como envenenar|quiero envenenar|empoisonner (quelqu|mon voisin)|jemanden vergiften|ich will jemanden vergiften|avvelenare (qualcuno|il vicino)|отравить (соседа|кого)|jak otrávit (souseda|někoho|psa|kočku|zvíře))")
|
| 231 |
+
_MC_SCHOOL = _rx(r"(sestřel|postříl|vystříl|střílet|zbraň|kulomet|začnu střílet).{0,40}(škola|školu|ve škole|třída|do školy)|(škola|školu|ve škole|do školy|třída).{0,40}(střílet|střelba|zbraň|sestřelit|vystřílet|kulomet)|střelba ve škole|školní masakr|school shooting|how to shoot up a school|matar (a todos )?en la escuela|tiroteo escolar|disparar en (la escuela|el colegio)|tuer tout le monde (à|a) l\'école|fusillade (scolaire|dans l\'école)|alle in der schule (töten|erschie|angreifen)|amoklauf (an der|in der) schule|uccidere tutti a scuola|strage (a|di) scuola|стріляти в школі|вбити всіх у школі")
|
| 232 |
+
_MC_MARKET = _rx(r"(kde|jak) (koupit|sehnat|seženu|objednat|koupím).{0,50}(pervitin|metamfetamin|kokain|heroin|drogy|zbraň|zbraně|explozivum|trhavin|bombu)|prodej (drog|zbraní)|nabízím (drogy|zbraně)|comprar (armas|drogas) baratas|buy (guns|drugs) (cheap|online)|как купить (оружие|наркотики)|(vařit|svařit|syntetizovat) fet( |$)|fet (na vaření|syntézu)")
|
| 233 |
+
|
| 234 |
+
|
| 235 |
+
_MC_LEGAL_WEAPON = _rx(r"(zbrojni prukaz|zbrojní průkaz|legalne|z legální|vzduchovk|airsoft|sportovni střelba|sportovní střelba|strilec na soutěžích)")
|
| 236 |
+
_MC_BOMB_BENIGN = _rx(r"(pyrotechnik|silvestr|ohňostroj|prskavk|dort|do koupele|do dortu|výbuch dortu)")
|
| 237 |
+
_MC_PEST = _rx(r"(krtk|myš|potkan|krys|hmyz|mravenc|plevel|slimák|škůdce|na zahradě|na zahrade)")
|
| 238 |
+
|
| 239 |
+
SUPPORT_LINK = "https://www.linkabezpeci.cz"
|
| 240 |
+
|
| 241 |
+
CAT_PRIORITY = ["sexual_abuse", "self_harm", "violence", "weapons_explosives", "drugs",
|
| 242 |
+
"cybercrime_fraud", "jailbreak_injection", "illegal_marketplace", "other_unsafe",
|
| 243 |
+
"hate_harassment"] # hate last: catch-all
|
| 244 |
+
|
| 245 |
+
|
| 246 |
+
def _cat_rank(c: str) -> int:
|
| 247 |
+
return CAT_PRIORITY.index(c) if c in CAT_PRIORITY else 99
|
| 248 |
+
|
| 249 |
+
|
| 250 |
+
|
| 251 |
+
def _fold_ascii(text: str) -> str:
|
| 252 |
+
return "".join(c for c in unicodedata.normalize("NFKD", text) if not unicodedata.combining(c))
|
| 253 |
+
|
| 254 |
+
|
| 255 |
+
def must_catch_hit(text: str) -> list[str]:
|
| 256 |
+
"""Pattern pre-filter. Returns HARD-hit category names (force unsafe).
|
| 257 |
+
|
| 258 |
+
CSAM always blocks. Self-harm blocks on 1st-person or instruction forms,
|
| 259 |
+
unless it is a support context (routed to resources instead) or news.
|
| 260 |
+
Weapons/drugs/cyber/poison/market DIY patterns are instruction-form only
|
| 261 |
+
(news pass). Jailbreak patterns are NOT hard blocks -> see review_hit().
|
| 262 |
+
"""
|
| 263 |
+
folded = _fold_ascii(text).lower()
|
| 264 |
+
hits = []
|
| 265 |
+
if _MC_CSAM.search(folded):
|
| 266 |
+
hits.append("sexual_abuse")
|
| 267 |
+
support = bool(_MC_SUPPORT_CONTEXT.search(folded))
|
| 268 |
+
news = bool(_MC_NEWS_CONTEXT.search(folded))
|
| 269 |
+
if _MC_SELFHARM_1P.search(folded) and not support:
|
| 270 |
+
hits.append("self_harm")
|
| 271 |
+
elif _MC_SELFHARM_INSTR.search(folded) and _MC_INSTR_MARK.search(folded) and not support and not news:
|
| 272 |
+
hits.append("self_harm")
|
| 273 |
+
if _MC_THREAT.search(folded) and not news:
|
| 274 |
+
hits.append("violence")
|
| 275 |
+
if _MC_SCHOOL.search(folded) and not news:
|
| 276 |
+
hits.append("violence")
|
| 277 |
+
if _MC_HATE_GROUP.search(folded) and not news:
|
| 278 |
+
hits.append("hate_harassment")
|
| 279 |
+
legal = bool(_MC_LEGAL_WEAPON.search(folded))
|
| 280 |
+
bomb_benign = bool(_MC_BOMB_BENIGN.search(folded))
|
| 281 |
+
pest = bool(_MC_PEST.search(folded))
|
| 282 |
+
if _MC_WEAPONS_DIY.search(folded) and not news:
|
| 283 |
+
hits.append("weapons_explosives")
|
| 284 |
+
if _MC_DRUGS_DIY.search(folded) and not news:
|
| 285 |
+
hits.append("drugs")
|
| 286 |
+
if _MC_CYBER_DIY.search(folded) and not _MC_DEFENSIVE.search(folded) and not news:
|
| 287 |
+
hits.append("cybercrime_fraud")
|
| 288 |
+
if _MC_MARKET.search(folded) and not news:
|
| 289 |
+
hits.append("illegal_marketplace")
|
| 290 |
+
if _MC_POISON.search(folded) and not news and not pest:
|
| 291 |
+
hits.append("violence")
|
| 292 |
+
if legal and "illegal_marketplace" in hits:
|
| 293 |
+
hits.remove("illegal_marketplace")
|
| 294 |
+
if bomb_benign and "illegal_marketplace" in hits:
|
| 295 |
+
hits.remove("illegal_marketplace")
|
| 296 |
+
return sorted(set(hits))
|
| 297 |
+
|
| 298 |
+
|
| 299 |
+
def review_hit(text: str) -> list[str]:
|
| 300 |
+
"""Jailbreak / prompt-injection / legally-ambiguous patterns -> needs_review, NOT a hard block."""
|
| 301 |
+
folded = _fold_ascii(text).lower()
|
| 302 |
+
out = []
|
| 303 |
+
if _MC_JAILBREAK.search(folded):
|
| 304 |
+
out.append("jailbreak_injection")
|
| 305 |
+
if _MC_LEGAL_WEAPON.search(folded) and (_MC_MARKET.search(folded) or _MC_WEAPONS_DIY.search(folded)):
|
| 306 |
+
out.append("weapons_explosives")
|
| 307 |
+
if _MC_BOMB_BENIGN.search(folded) and _MC_MARKET.search(folded):
|
| 308 |
+
out.append("illegal_marketplace")
|
| 309 |
+
return sorted(set(out))
|
| 310 |
+
|
| 311 |
+
|
| 312 |
+
def is_support_context(text: str) -> bool:
|
| 313 |
+
"""Self-harm mention in a helping context -> route to support, not a block."""
|
| 314 |
+
folded = _fold_ascii(text).lower()
|
| 315 |
+
return bool((_MC_SELFHARM_1P.search(folded) or _MC_SELFHARM_INSTR.search(folded))
|
| 316 |
+
and _MC_SUPPORT_CONTEXT.search(folded))
|
| 317 |
+
|
| 318 |
+
|
| 319 |
+
def _script(text: str) -> str:
|
| 320 |
+
counts = {}
|
| 321 |
+
for ch in text:
|
| 322 |
+
if ch.isalpha():
|
| 323 |
+
name = unicodedata.name(ch, "")
|
| 324 |
+
for key in ("CYRILLIC", "HAN", "HIRAGANA", "KATAKANA", "HANGUL", "ARABIC", "DEVANAGARI", "HEBREW", "LATIN"):
|
| 325 |
+
if key in name:
|
| 326 |
+
counts[key] = counts.get(key, 0) + 1
|
| 327 |
+
break
|
| 328 |
+
if not counts:
|
| 329 |
+
return "latin"
|
| 330 |
+
top = max(counts, key=counts.get)
|
| 331 |
+
if top == "LATIN":
|
| 332 |
+
return "latin"
|
| 333 |
+
return {"CYRILLIC": "cyrillic", "HAN": "han", "HIRAGANA": "kana", "KATAKANA": "kana",
|
| 334 |
+
"HANGUL": "hangul", "ARABIC": "arabic", "DEVANAGARI": "devanagari", "HEBREW": "hebrew"}[top]
|
| 335 |
+
|
| 336 |
+
|
| 337 |
+
def _script(text: str) -> str:
|
| 338 |
+
counts = {}
|
| 339 |
+
for ch in text:
|
| 340 |
+
if ch.isalpha():
|
| 341 |
+
name = unicodedata.name(ch, "")
|
| 342 |
+
for key in ("CYRILLIC", "HAN", "HIRAGANA", "KATAKANA", "HANGUL", "ARABIC", "DEVANAGARI", "HEBREW", "LATIN"):
|
| 343 |
+
if key in name:
|
| 344 |
+
counts[key] = counts.get(key, 0) + 1
|
| 345 |
+
break
|
| 346 |
+
if not counts:
|
| 347 |
+
return "latin"
|
| 348 |
+
top = max(counts, key=counts.get)
|
| 349 |
+
if top == "LATIN":
|
| 350 |
+
return "latin"
|
| 351 |
+
return {"CYRILLIC": "cyrillic", "HAN": "han", "HIRAGANA": "kana", "KATAKANA": "kana",
|
| 352 |
+
"HANGUL": "hangul", "ARABIC": "arabic", "DEVANAGARI": "devanagari", "HEBREW": "hebrew"}[top]
|
| 353 |
+
|
| 354 |
+
|
| 355 |
+
def detect_lang(text: str) -> str:
|
| 356 |
+
"""Heuristic ISO-639-1 guess; returns 'default' (global threshold) when unsure."""
|
| 357 |
+
s = _script(text)
|
| 358 |
+
if s == "cyrillic":
|
| 359 |
+
uk_score = sum(1 for w in re.findall(r"[a-zа-яіїєґ']+", text.lower()) if w in STOPWORDS["uk"])
|
| 360 |
+
uk_score += 2 * sum(text.count(c) for c in "ґєії")
|
| 361 |
+
ru_score = sum(1 for w in re.findall(r"[a-zа-яёъыэ']+", text.lower()) if w in STOPWORDS["ru"])
|
| 362 |
+
return "uk" if uk_score >= MIN_SCORE and uk_score > ru_score else ("ru" if ru_score >= MIN_SCORE else "default")
|
| 363 |
+
if s == "han":
|
| 364 |
+
return "zh"
|
| 365 |
+
if s == "kana":
|
| 366 |
+
return "ja"
|
| 367 |
+
if s == "hangul":
|
| 368 |
+
return "ko"
|
| 369 |
+
if s == "arabic":
|
| 370 |
+
return "ar"
|
| 371 |
+
if s == "devanagari":
|
| 372 |
+
return "hi"
|
| 373 |
+
if s == "hebrew":
|
| 374 |
+
return "he"
|
| 375 |
+
words = re.findall(r"[a-zà-ÿěščřžďťňáéíúůóßäöüñãõ]+", text.lower())
|
| 376 |
+
if not words:
|
| 377 |
+
return "default"
|
| 378 |
+
scores = {}
|
| 379 |
+
for lang, stops in STOPWORDS.items():
|
| 380 |
+
if lang in ("ru", "uk"):
|
| 381 |
+
continue
|
| 382 |
+
n = sum(1 for w in words if w in stops)
|
| 383 |
+
n += 2 * sum(text.lower().count(c) for c in DIACRITICS.get(lang, ""))
|
| 384 |
+
scores[lang] = n
|
| 385 |
+
best = max(scores, key=scores.get)
|
| 386 |
+
return best if scores[best] >= MIN_SCORE else "default"
|
| 387 |
+
|
| 388 |
+
|
| 389 |
+
|
| 390 |
+
def load_state_dict_any(directory) -> dict:
|
| 391 |
+
"""Load model weights from model_int8.safetensors (dequantized) if present,
|
| 392 |
+
else model_fp16.safetensors, else model.safetensors."""
|
| 393 |
+
from pathlib import Path as _P
|
| 394 |
+
from safetensors.torch import load_file
|
| 395 |
+
|
| 396 |
+
d = _P(directory)
|
| 397 |
+
if (d / "model_int8.safetensors").exists():
|
| 398 |
+
st = load_file(str(d / "model_int8.safetensors"))
|
| 399 |
+
out = {}
|
| 400 |
+
for k, v in st.items():
|
| 401 |
+
if k.endswith(".q"):
|
| 402 |
+
base = k[:-2]
|
| 403 |
+
out[base] = (v.float() * st[base + ".scale"]).to(torch.float32)
|
| 404 |
+
elif not k.endswith(".scale"):
|
| 405 |
+
out[k] = v
|
| 406 |
+
return out
|
| 407 |
+
if (d / "model_fp16.safetensors").exists():
|
| 408 |
+
return {k: v.float() for k, v in load_file(str(d / "model_fp16.safetensors")).items()}
|
| 409 |
+
return load_file(str(d / "model.safetensors"))
|
| 410 |
+
|
| 411 |
+
|
| 412 |
+
def remap_hf_state(state: dict) -> dict:
|
| 413 |
+
"""Standard Llama-layout safetensors -> training-layout state dict."""
|
| 414 |
+
import re
|
| 415 |
+
|
| 416 |
+
out = {"embed.weight": state["model.embed_tokens.weight"]}
|
| 417 |
+
layer_ids = sorted({int(m.group(1)) for k in state if (m := re.match(r"model\.layers\.(\d+)\.", k))})
|
| 418 |
+
for i in layer_ids:
|
| 419 |
+
p, b = f"model.layers.{i}.", f"blocks.{i}."
|
| 420 |
+
qkv = torch.cat(
|
| 421 |
+
[state[p + "self_attn.q_proj.weight"], state[p + "self_attn.k_proj.weight"], state[p + "self_attn.v_proj.weight"]],
|
| 422 |
+
dim=0,
|
| 423 |
+
)
|
| 424 |
+
out[b + "attn.qkv.weight"] = qkv.contiguous()
|
| 425 |
+
out[b + "attn.proj.weight"] = state[p + "self_attn.o_proj.weight"]
|
| 426 |
+
out[b + "mlp.gate.weight"] = state[p + "mlp.gate_proj.weight"]
|
| 427 |
+
out[b + "mlp.up.weight"] = state[p + "mlp.up_proj.weight"]
|
| 428 |
+
out[b + "mlp.down.weight"] = state[p + "mlp.down_proj.weight"]
|
| 429 |
+
out[b + "norm1.weight"] = state[p + "input_layernorm.weight"]
|
| 430 |
+
out[b + "norm2.weight"] = state[p + "post_attention_layernorm.weight"]
|
| 431 |
+
out["norm.weight"] = state["model.norm.weight"]
|
| 432 |
+
if "score.weight" in state:
|
| 433 |
+
out["score.weight"] = state["score.weight"]
|
| 434 |
+
return out
|
| 435 |
+
|
| 436 |
+
|
| 437 |
+
def internal_cfg(cfg: dict) -> dict:
|
| 438 |
+
"""config.json (Llama layout) -> the dict the Censer module expects."""
|
| 439 |
+
return {
|
| 440 |
+
"vocab_size": cfg["vocab_size"],
|
| 441 |
+
"n_layers": cfg["num_hidden_layers"],
|
| 442 |
+
"n_heads": cfg["num_attention_heads"],
|
| 443 |
+
"d_model": cfg["hidden_size"],
|
| 444 |
+
"d_ff": cfg["intermediate_size"],
|
| 445 |
+
"max_seq_len": cfg["max_position_embeddings"],
|
| 446 |
+
"dropout": cfg.get("dropout", 0.0),
|
| 447 |
+
"tie_embeddings": True,
|
| 448 |
+
"pad_id": cfg.get("pad_token_id", 0),
|
| 449 |
+
"answer_ids": cfg["answer_ids"],
|
| 450 |
+
"max_chars": cfg.get("max_chars", 4000),
|
| 451 |
+
"rope_scaling": cfg.get("rope_scaling") or (cfg.get("rope_parameters") if cfg.get("rope_parameters", {}).get("rope_type") == "linear" else None),
|
| 452 |
+
"categories": [cfg["id2label"][str(i)] for i in range(2, cfg.get("num_labels", 2))] if cfg.get("num_labels", 2) > 2 else [],
|
| 453 |
+
"cat_thresholds": cfg.get("cat_thresholds") or {},
|
| 454 |
+
"temperature": float(cfg.get("temperature", 1.0) or 1.0),
|
| 455 |
+
"lang_thresholds": cfg.get("lang_thresholds") or {},
|
| 456 |
+
}
|
| 457 |
+
|
| 458 |
+
|
| 459 |
+
|
| 460 |
+
class CenserPipeline:
|
| 461 |
+
"""Text in, "1" (unsafe) or "0" (safe) out."""
|
| 462 |
+
|
| 463 |
+
def __init__(self, directory: str | Path = ".", threshold: float = 0.5, device: str | None = None, prefer_quant: bool = False) -> None:
|
| 464 |
+
directory = Path(directory)
|
| 465 |
+
self.cfg = internal_cfg(json.loads((directory / "config.json").read_text(encoding="utf-8")))
|
| 466 |
+
self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
|
| 467 |
+
self.pad_id = self.tok.token_to_id(PAD)
|
| 468 |
+
self.bos_id = self.tok.token_to_id(BOS)
|
| 469 |
+
self.user_id = self.tok.token_to_id(USER)
|
| 470 |
+
self.assistant_id = self.tok.token_to_id(ASSISTANT)
|
| 471 |
+
self.threshold = threshold
|
| 472 |
+
self.cat_thresholds = dict(self.cfg.get("cat_thresholds") or {})
|
| 473 |
+
self.temperature = float(self.cfg.get("temperature", 1.0) or 1.0)
|
| 474 |
+
self.cat_experimental = set(self.cfg.get("cat_experimental") or [])
|
| 475 |
+
self.lang_thresholds = dict(self.cfg.get("lang_thresholds") or {})
|
| 476 |
+
self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
|
| 477 |
+
self.model = Censer(self.cfg)
|
| 478 |
+
if prefer_quant and (directory / "model_int8.safetensors").exists():
|
| 479 |
+
state = remap_hf_state(load_state_dict_any(directory))
|
| 480 |
+
else:
|
| 481 |
+
from safetensors.torch import load_file
|
| 482 |
+
state = remap_hf_state(load_file(directory / "model.safetensors"))
|
| 483 |
+
missing, unexpected = self.model.load_state_dict(state, strict=False)
|
| 484 |
+
missing = [k for k in missing if k != "lm_head.weight"] # tied to embed.weight
|
| 485 |
+
if missing or unexpected:
|
| 486 |
+
raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
|
| 487 |
+
self.model.to(self.device).eval()
|
| 488 |
+
self.max_chars = self.cfg["max_chars"]
|
| 489 |
+
|
| 490 |
+
def _encode(self, text: str) -> list[int]:
|
| 491 |
+
body = self.tok.encode(text[: self.max_chars], add_special_tokens=False).ids
|
| 492 |
+
ids = [self.bos_id, self.user_id, *body, self.assistant_id]
|
| 493 |
+
return ids[: self.cfg["max_seq_len"]]
|
| 494 |
+
|
| 495 |
+
def predict(self, text: str) -> dict:
|
| 496 |
+
return self.predict_batch([text])[0]
|
| 497 |
+
|
| 498 |
+
def _encode_full(self, text: str) -> list[int]:
|
| 499 |
+
body = self.tok.encode(text[: self.max_chars * 8], add_special_tokens=False).ids
|
| 500 |
+
return [self.bos_id, self.user_id, *body, self.assistant_id]
|
| 501 |
+
|
| 502 |
+
def predict_batch(self, texts: list[str]) -> list[dict]:
|
| 503 |
+
# long inputs: split into overlapping token chunks, aggregate by max
|
| 504 |
+
out: list[dict] = []
|
| 505 |
+
chunk_tokens, stride = 256, 192
|
| 506 |
+
for text in texts:
|
| 507 |
+
thr = self.threshold_for(text)
|
| 508 |
+
ids = self._encode_full(text)
|
| 509 |
+
body = ids[2:-1]
|
| 510 |
+
if len(body) <= chunk_tokens:
|
| 511 |
+
out.append(self._classify_ids([ids[: self.cfg["max_seq_len"]]], [thr], [text])[0])
|
| 512 |
+
continue
|
| 513 |
+
chunks = []
|
| 514 |
+
start = 0
|
| 515 |
+
while start < len(body):
|
| 516 |
+
chunks.append(body[start:start + chunk_tokens])
|
| 517 |
+
if start + chunk_tokens >= len(body):
|
| 518 |
+
break
|
| 519 |
+
start += stride
|
| 520 |
+
rows = []
|
| 521 |
+
for piece in chunks:
|
| 522 |
+
cids = [self.bos_id, self.user_id, *piece, self.assistant_id]
|
| 523 |
+
rows.append(self._classify_ids([cids[: self.cfg["max_seq_len"]]], [thr], [text])[0])
|
| 524 |
+
best = max(rows, key=lambda r: r["score"])
|
| 525 |
+
# aggregate categories by max prob
|
| 526 |
+
n_cat = len(rows[0].get("category_scores", []))
|
| 527 |
+
cat_scores = [max(r["category_scores"][i] for r in rows) for i in range(n_cat)] if n_cat else []
|
| 528 |
+
best["category_scores"] = [round(c, 4) for c in cat_scores]
|
| 529 |
+
cats = self.categories()
|
| 530 |
+
if best["unsafe"] and cats:
|
| 531 |
+
passed = {c for c, p in zip(cats, cat_scores)
|
| 532 |
+
if p >= float(self.cat_thresholds.get(c, self.threshold)) and c not in self.cat_experimental}
|
| 533 |
+
passed |= set(best.get("categories") or []) # keep pre-filter hits from the best chunk
|
| 534 |
+
# hard block (item 1): sexual_abuse / self_harm >= 0.3 forces unsafe for cs/sk/default
|
| 535 |
+
lang = detect_lang(text)
|
| 536 |
+
if not best.get("needs_support") and lang in ("cs", "sk", "default"):
|
| 537 |
+
for c in ("sexual_abuse", "self_harm"):
|
| 538 |
+
if c in cats and cat_scores[cats.index(c)] >= 0.3:
|
| 539 |
+
passed.add(c)
|
| 540 |
+
best["unsafe"] = True
|
| 541 |
+
best["label"] = "unsafe"
|
| 542 |
+
best["label_id"] = "1"
|
| 543 |
+
best["override"] = best.get("override") or "category_block"
|
| 544 |
+
passed = sorted(passed, key=lambda c: (_cat_rank(c) if c in CAT_PRIORITY else 99,
|
| 545 |
+
-(cat_scores[cats.index(c)] if c in cats else 1.0)))[:2]
|
| 546 |
+
if not passed: # unsafe must always carry >=1 category: top-1 by score
|
| 547 |
+
passed = [max(cats, key=lambda c: cat_scores[cats.index(c)])]
|
| 548 |
+
best["categories"] = passed
|
| 549 |
+
else:
|
| 550 |
+
best["categories"] = []
|
| 551 |
+
best["chunks"] = len(chunks)
|
| 552 |
+
out.append(best)
|
| 553 |
+
return out
|
| 554 |
+
|
| 555 |
+
def threshold_for(self, text: str) -> float:
|
| 556 |
+
lang = detect_lang(text)
|
| 557 |
+
return float(self.lang_thresholds.get(lang) or self.lang_thresholds.get("default") or self.threshold)
|
| 558 |
+
|
| 559 |
+
def categories(self) -> list[str]:
|
| 560 |
+
return self.cfg.get("categories") or []
|
| 561 |
+
|
| 562 |
+
def _classify_ids(self, seqs: list[list[int]], thresholds: list[float] | None = None, texts: list[str] | None = None) -> list[dict]:
|
| 563 |
+
width = max(len(s) for s in seqs)
|
| 564 |
+
ids = torch.full((len(seqs), width), self.pad_id, dtype=torch.long)
|
| 565 |
+
for i, seq in enumerate(seqs):
|
| 566 |
+
ids[i, : len(seq)] = torch.tensor(seq)
|
| 567 |
+
probs = self.model.classify_probs(ids.to(self.device), temperature=self.temperature).float().cpu().tolist()
|
| 568 |
+
if thresholds is None:
|
| 569 |
+
thresholds = [self.threshold] * len(seqs)
|
| 570 |
+
res = []
|
| 571 |
+
for i, row in enumerate(probs):
|
| 572 |
+
p = row[1]
|
| 573 |
+
cats = self.categories()
|
| 574 |
+
cat_scores = row[2:] if cats else []
|
| 575 |
+
is_unsafe = p >= thresholds[i]
|
| 576 |
+
# Barist.2: must-catch pre-filter — CSAM / self-harm pattern hits force
|
| 577 |
+
# unsafe regardless of the (language) binary threshold
|
| 578 |
+
hit_override = must_catch_hit(texts[i]) if texts else []
|
| 579 |
+
review = review_hit(texts[i]) if texts else []
|
| 580 |
+
help_ctx = is_support_context(texts[i]) if texts else False
|
| 581 |
+
support_ctx = help_ctx
|
| 582 |
+
if not support_ctx and cats and "self_harm" in cats:
|
| 583 |
+
if cat_scores[cats.index("self_harm")] >= 0.3: # model-level self-harm -> support resources
|
| 584 |
+
support_ctx = True
|
| 585 |
+
# hard block (item 1): sexual_abuse / self_harm category >= 0.3 forces unsafe
|
| 586 |
+
# for cs/sk/default regardless of the language threshold
|
| 587 |
+
cat_block = []
|
| 588 |
+
lang_i = detect_lang(texts[i]) if texts else "default"
|
| 589 |
+
if cats and not support_ctx and lang_i in ("cs", "sk", "default"):
|
| 590 |
+
for c in ("sexual_abuse", "self_harm"):
|
| 591 |
+
if c in cats and cat_scores[cats.index(c)] >= 0.3:
|
| 592 |
+
cat_block.append(c)
|
| 593 |
+
if cat_block:
|
| 594 |
+
is_unsafe = True
|
| 595 |
+
if hit_override:
|
| 596 |
+
is_unsafe = True
|
| 597 |
+
if help_ctx and not hit_override and not cat_block:
|
| 598 |
+
is_unsafe = False # help-seeking context is never blocked
|
| 599 |
+
# Barist.2: at most the top-2 categories by score (no nonsense lists)
|
| 600 |
+
_passed = [c for c, sc in zip(cats, cat_scores)
|
| 601 |
+
if sc >= float(self.cat_thresholds.get(c, self.threshold))
|
| 602 |
+
and c not in self.cat_experimental]
|
| 603 |
+
_passed = sorted(set(_passed) | set(hit_override) | set(cat_block),
|
| 604 |
+
key=lambda c: (_cat_rank(c) if c in CAT_PRIORITY else 99,
|
| 605 |
+
-(cat_scores[cats.index(c)] if c in cats else 1.0)))[:2]
|
| 606 |
+
if is_unsafe and not _passed:
|
| 607 |
+
# unsafe must always carry >=1 category: top-1 by score
|
| 608 |
+
_passed = [max(cats, key=lambda c: cat_scores[cats.index(c)])] if cats else []
|
| 609 |
+
res.append({
|
| 610 |
+
"label": "unsafe" if is_unsafe else "safe",
|
| 611 |
+
"label_id": "1" if is_unsafe else "0",
|
| 612 |
+
"unsafe": is_unsafe,
|
| 613 |
+
"override": "must_catch" if hit_override else ("category_block" if cat_block else None),
|
| 614 |
+
"needs_support": SUPPORT_LINK if (support_ctx or "self_harm" in hit_override) else None,
|
| 615 |
+
"needs_review": True if review else None,
|
| 616 |
+
"score": round(p, 4),
|
| 617 |
+
"category_scores": [round(c, 4) for c in cat_scores],
|
| 618 |
+
# categories are only reported when the binary verdict is unsafe
|
| 619 |
+
"categories": _passed if is_unsafe else [],
|
| 620 |
+
})
|
| 621 |
+
return res
|
| 622 |
+
|
| 623 |
+
|
| 624 |
+
|
| 625 |
+
|
| 626 |
+
def _color(text: str, code: str) -> str:
|
| 627 |
+
if not sys.stdout.isatty():
|
| 628 |
+
return text
|
| 629 |
+
return f"\033[{code}m{text}\033[0m"
|
| 630 |
+
|
| 631 |
+
|
| 632 |
+
def repl(pipe: "CenserPipeline") -> int:
|
| 633 |
+
"""Small interactive terminal UI: phrase in, safe/unsafe + score out."""
|
| 634 |
+
print("Barist.2 - interactive terminal UI")
|
| 635 |
+
print("Type a phrase, Enter to classify. 'exit' to quit.\n")
|
| 636 |
+
while True:
|
| 637 |
+
try:
|
| 638 |
+
line = input("> ")
|
| 639 |
+
except (EOFError, KeyboardInterrupt):
|
| 640 |
+
print()
|
| 641 |
+
return 0
|
| 642 |
+
text = line.strip()
|
| 643 |
+
if text.lower() in {"exit", "quit", ":q"}:
|
| 644 |
+
return 0
|
| 645 |
+
if not text:
|
| 646 |
+
continue
|
| 647 |
+
v = pipe.predict(text)
|
| 648 |
+
label = "unsafe" if v["unsafe"] else "safe"
|
| 649 |
+
colored = _color(f"{label:6s}", "31" if v["unsafe"] else "32")
|
| 650 |
+
print(f"{colored} {v['score']:.4f} | {text}")
|
| 651 |
+
|
| 652 |
+
|
| 653 |
+
def main(argv: list[str] | None = None) -> None:
|
| 654 |
+
parser = argparse.ArgumentParser(
|
| 655 |
+
prog="barist.1",
|
| 656 |
+
description="Barist.2 safety classifier (standalone). Run without text to open the interactive UI.",
|
| 657 |
+
)
|
| 658 |
+
parser.add_argument("text", nargs="*", help="phrase to classify once; omit to open the interactive UI")
|
| 659 |
+
parser.add_argument("--model-dir", default=".")
|
| 660 |
+
parser.add_argument("--threshold", type=float, default=0.5)
|
| 661 |
+
parser.add_argument("--json", action="store_true")
|
| 662 |
+
parser.add_argument("--prefer-quant", action="store_true",
|
| 663 |
+
help="load model_int8.safetensors (0.9 MB) instead of fp32 if present")
|
| 664 |
+
parser.add_argument("--repl", action="store_true", help="interactive UI (default when no text is given)")
|
| 665 |
+
args = parser.parse_args(argv)
|
| 666 |
+
|
| 667 |
+
pipe = CenserPipeline(args.model_dir, args.threshold, prefer_quant=args.prefer_quant)
|
| 668 |
+
text = " ".join(args.text).strip()
|
| 669 |
+
if args.repl or not text:
|
| 670 |
+
raise SystemExit(repl(pipe))
|
| 671 |
+
verdict = pipe.predict(text)
|
| 672 |
+
print(json.dumps(verdict) if args.json else verdict["label"]) # "safe" / "unsafe"
|
| 673 |
+
|
| 674 |
+
|
| 675 |
+
if __name__ == "__main__":
|
| 676 |
+
main()
|
config.json
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"LlamaForSequenceClassification"
|
| 4 |
+
],
|
| 5 |
+
"model_type": "llama",
|
| 6 |
+
"num_labels": 12,
|
| 7 |
+
"id2label": {
|
| 8 |
+
"0": "safe",
|
| 9 |
+
"1": "unsafe",
|
| 10 |
+
"2": "violence",
|
| 11 |
+
"3": "weapons_explosives",
|
| 12 |
+
"4": "drugs",
|
| 13 |
+
"5": "hate_harassment",
|
| 14 |
+
"6": "sexual_abuse",
|
| 15 |
+
"7": "self_harm",
|
| 16 |
+
"8": "cybercrime_fraud",
|
| 17 |
+
"9": "jailbreak_injection",
|
| 18 |
+
"10": "illegal_marketplace",
|
| 19 |
+
"11": "other_unsafe"
|
| 20 |
+
},
|
| 21 |
+
"label2id": {
|
| 22 |
+
"safe": 0,
|
| 23 |
+
"unsafe": 1,
|
| 24 |
+
"violence": 2,
|
| 25 |
+
"weapons_explosives": 3,
|
| 26 |
+
"drugs": 4,
|
| 27 |
+
"hate_harassment": 5,
|
| 28 |
+
"sexual_abuse": 6,
|
| 29 |
+
"self_harm": 7,
|
| 30 |
+
"cybercrime_fraud": 8,
|
| 31 |
+
"jailbreak_injection": 9,
|
| 32 |
+
"illegal_marketplace": 10,
|
| 33 |
+
"other_unsafe": 11
|
| 34 |
+
},
|
| 35 |
+
"problem_type": "multi_label_classification",
|
| 36 |
+
"vocab_size": 2048,
|
| 37 |
+
"hidden_size": 128,
|
| 38 |
+
"intermediate_size": 384,
|
| 39 |
+
"num_hidden_layers": 3,
|
| 40 |
+
"num_attention_heads": 4,
|
| 41 |
+
"num_key_value_heads": 4,
|
| 42 |
+
"head_dim": 32,
|
| 43 |
+
"max_position_embeddings": 4096,
|
| 44 |
+
"rms_norm_eps": 1e-05,
|
| 45 |
+
"rope_parameters": {
|
| 46 |
+
"rope_type": "linear",
|
| 47 |
+
"factor": 8.0,
|
| 48 |
+
"rope_theta": 10000.0
|
| 49 |
+
},
|
| 50 |
+
"attention_bias": false,
|
| 51 |
+
"mlp_bias": false,
|
| 52 |
+
"tie_word_embeddings": true,
|
| 53 |
+
"pad_token_id": 0,
|
| 54 |
+
"bos_token_id": 1,
|
| 55 |
+
"eos_token_id": 3,
|
| 56 |
+
"name": "Barist.2",
|
| 57 |
+
"max_chars": 4000,
|
| 58 |
+
"answer_ids": [
|
| 59 |
+
19,
|
| 60 |
+
20,
|
| 61 |
+
68,
|
| 62 |
+
69,
|
| 63 |
+
70,
|
| 64 |
+
71,
|
| 65 |
+
72,
|
| 66 |
+
73,
|
| 67 |
+
74,
|
| 68 |
+
75,
|
| 69 |
+
76,
|
| 70 |
+
77
|
| 71 |
+
],
|
| 72 |
+
"special_tokens": [
|
| 73 |
+
"[PAD]",
|
| 74 |
+
"[BOS]",
|
| 75 |
+
"[USER]",
|
| 76 |
+
"[ASSISTANT]"
|
| 77 |
+
],
|
| 78 |
+
"threshold": 0.5,
|
| 79 |
+
"cat_thresholds": {
|
| 80 |
+
"violence": 0.5,
|
| 81 |
+
"weapons_explosives": 0.5,
|
| 82 |
+
"drugs": 0.5,
|
| 83 |
+
"hate_harassment": 0.4,
|
| 84 |
+
"sexual_abuse": 0.3,
|
| 85 |
+
"self_harm": 0.3,
|
| 86 |
+
"cybercrime_fraud": 0.5,
|
| 87 |
+
"jailbreak_injection": 0.5,
|
| 88 |
+
"illegal_marketplace": 0.5,
|
| 89 |
+
"other_unsafe": 0.45
|
| 90 |
+
},
|
| 91 |
+
"temperature": 0.6643198430536946,
|
| 92 |
+
"must_catch_override": {
|
| 93 |
+
"self_harm": 0.3,
|
| 94 |
+
"sexual_abuse": 0.3
|
| 95 |
+
},
|
| 96 |
+
"cat_experimental": [
|
| 97 |
+
"violence",
|
| 98 |
+
"weapons_explosives",
|
| 99 |
+
"drugs",
|
| 100 |
+
"sexual_abuse",
|
| 101 |
+
"self_harm",
|
| 102 |
+
"cybercrime_fraud",
|
| 103 |
+
"jailbreak_injection",
|
| 104 |
+
"illegal_marketplace"
|
| 105 |
+
],
|
| 106 |
+
"lang_thresholds": {
|
| 107 |
+
"default": 0.5,
|
| 108 |
+
"de": 0.5,
|
| 109 |
+
"en": 0.5,
|
| 110 |
+
"es": 0.5,
|
| 111 |
+
"fr": 0.71,
|
| 112 |
+
"it": 0.5,
|
| 113 |
+
"cs": 0.5
|
| 114 |
+
}
|
| 115 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c30e171ebeb19c4d54bdeb3c6eaad89011f61572480ec0d887f93feac2edde3a
|
| 3 |
+
size 3617384
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 3 |
+
"pad_token": "[PAD]",
|
| 4 |
+
"bos_token": "[BOS]",
|
| 5 |
+
"eos_token": "[ASSISTANT]",
|
| 6 |
+
"model_max_length": 4096
|
| 7 |
+
}
|