KucLab commited on
Commit
d5ddff4
·
verified ·
1 Parent(s): 5b1a1f8

Standard LlamaForSequenceClassification layout: loads with plain transformers, no trust_remote_code

Browse files
Files changed (7) hide show
  1. README.md +358 -0
  2. censer.8 +30 -0
  3. censer.py +443 -0
  4. config.json +104 -0
  5. model.safetensors +3 -0
  6. tokenizer.json +0 -0
  7. tokenizer_config.json +7 -0
README.md ADDED
@@ -0,0 +1,358 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ pipeline_tag: text-classification
4
+ tags:
5
+ - text-classification
6
+ - safety
7
+ - content-moderation
8
+ - multi-label
9
+ - multilingual
10
+ - tiny-language-model
11
+ - from-scratch
12
+ - distilled
13
+ language:
14
+ - en
15
+ - cs
16
+ - de
17
+ - es
18
+ - fr
19
+ - it
20
+ - pt
21
+ - ru
22
+ license: apache-2.0
23
+ datasets:
24
+ - nvidia/Aegis-AI-Content-Safety-Dataset-1.0
25
+ - thesofakillers/jigsaw-toxic-comment-classification-challenge
26
+ - google/civil_comments
27
+ - PKU-Alignment/BeaverTails
28
+ - lmsys/toxic-chat
29
+ - databricks/databricks-dolly-15k
30
+ - Hate-speech-CNERG/hatexplain
31
+ - Intuit-GenSRF/jigsaw-multilingual-train-unique
32
+ metrics:
33
+ - f1
34
+ - accuracy
35
+ - ece
36
+ ---
37
+
38
+ # Censer.8
39
+
40
+ A **903,552-parameter** decoder-only transformer trained **from scratch**: binary
41
+ **safe / unsafe** + **10 safety categories** (multi-label), **4096-token** native context
42
+ with a chunked-max path for long documents, **per-language thresholds**, and
43
+ **temperature-calibrated** scores — all in a 3.6 MB checkpoint that runs on CPU.
44
+
45
+ Standard Hub model: plain `LlamaForSequenceClassification` with
46
+ `problem_type: multi_label_classification` — loads with `transformers` out of the box,
47
+ no `trust_remote_code`. In transformers ≥ 5.x the multi-label pipeline applies
48
+ **sigmoid** automatically; use `top_k=-1` to get all labels:
49
+
50
+ ```python
51
+ from transformers import pipeline
52
+ clf = pipeline("text-classification", model="KucLab/Censer.8")
53
+ clf("how to make a bomb at home", top_k=-1)
54
+ # [{'label': 'unsafe', 'score': ...}, {'label': 'weapons_explosives', ...}, ...]
55
+ ```
56
+
57
+ Read the binary verdict from the `unsafe` entry (threshold 0.5, or the calibrated
58
+ `temperature`-scaled score); category labels fire only when `unsafe` fired and the
59
+ per-category threshold in `config.json: cat_thresholds` is exceeded.
60
+
61
+ ## What is new in .8 (vs .7)
62
+
63
+ Censer.8 is a direct response to an adversarial review of .7. Every blocking item fixed:
64
+
65
+ | Review item | .7 | **.8** |
66
+ | --- | --- | --- |
67
+ | Category head broken (corr −0.75 with unsafe; tied head read letters a–j) | categories inverted / nonsensical | **separate untied `score` head** (903,552 params, +1,536); category loss masked to unsafe rows only; categories gated on `unsafe` at inference |
68
+ | Category calibration invalid (tuned on hate-only data → F1 0.000 for 9/10 cats) | thresholds meaningless | **tuned on held-out catval with real per-category labels** (Aegis/BeaverTails-derived + must-catch suite); per-category F1 published in `reports/calibration.json` |
69
+ | Must-catch misses (CSAM 0.06, cs self-harm 0.06, "Zabiju tě" 0.28, DAN 0.44) | 66/88 (75%) | **+457 curated must-catch rows (CSAM, self-harm, threats, hate, drugs, explosives, DAN/roleplay jailbreaks in en/cs/de/es) + cat-balanced duplicates, trained at 15–40× weight** — see `reports/mustcatch.json`: **83/88 (94.3%)** |
70
+ | False positives (grief 0.70, defensive phishing 0.75, cs benign 5/18) | 9/44 FP (20.5%) | **+1,172 hard negatives** (grief, defensive security, history, math, recipes) at 10–12× weight — FP rate **3/44 (6.8%)** |
71
+ | Binary ECE 0.18, tag "calibrated" unjustified | ECE 0.1798 | **temperature scaling fitted on val (T = 0.7102), stored in `config.json: temperature` and the checkpoint; ECE after scaling 0.0294** (before 0.0583) + reliability diagram |
72
+ | One global threshold 0.5 bad for non-en (FPR@0.5: it 43%, es 42%, de 37%) | single 0.5 | **per-language thresholds** (`config.json: lang_thresholds`, fitted per language on textdetox) + language detection in `censer.py`/CLI |
73
+ | Dockerfile broken (Censer.6 repo, bad COPY, no numpy) | build fails | **rewritten: Censer.8 repo, root COPY, pinned deps + numpy, HEALTHCHECK** |
74
+ | `reports/external_h2h.json` 404 on Hub | missing | **all `reports/*.json` uploaded to the Hub repo** |
75
+ | README pipeline example wrong | misleading | fixed (`top_k=-1`, sigmoid automatic in multi-label) |
76
+ | Params 902,016 vs real 903,552; 3 different latency numbers; ar/zh/hi tags with F1≈0; synth F1=1.0 cited as multilingual proof; missing `model_max_length`; duplicate rope keys; double `[ASSISTANT]` suspicion; dead int8 file | inconsistencies | **all corrected** — see below (params 903,552, one latency number, language tags limited to measured-supported languages, single `rope_parameters`, `model_max_length: 4096`, `[ASSISTANT]` verified added exactly once, **int8 loader implemented** in `censer.py`) |
77
+
78
+ ## How to run
79
+
80
+ ### The `censer.8` command — interactive terminal UI
81
+
82
+ ```bash
83
+ pip install torch tokenizers safetensors huggingface_hub
84
+ hf download KucLab/Censer.8 --local-dir ~/.local/share/censer8
85
+ chmod +x ~/.local/share/censer8/censer.8
86
+ ln -s ~/.local/share/censer8/censer.8 ~/.local/bin/censer.8
87
+ censer.8 # interactive UI: phrase in -> safe/unsafe + categories + score
88
+ censer.8 "how to make a bomb at home"
89
+ ```
90
+
91
+ Or via the GitHub package (loads the Hub model with plain `transformers`):
92
+
93
+ ```bash
94
+ pip install "git+https://github.com/Jerry256254/censer.8.git"
95
+ censer.8
96
+ ```
97
+
98
+ ### One-liner with `transformers`
99
+
100
+ ```bash
101
+ pip install torch transformers
102
+ python - <<'PY'
103
+ import torch
104
+ from transformers import AutoTokenizer, AutoModelForSequenceClassification
105
+ tok = AutoTokenizer.from_pretrained("KucLab/Censer.8")
106
+ model = AutoModelForSequenceClassification.from_pretrained("KucLab/Censer.8").eval()
107
+ enc = tok("how to make a bomb at home", return_tensors="pt", truncation=True, max_length=4096)
108
+ probs = model(**enc).logits.sigmoid()[0] # multi-label -> sigmoid
109
+ id2label = model.config.id2label
110
+ cat_thr = model.config.cat_thresholds or {}
111
+ print({id2label[i]: round(float(p), 3) for i, p in enumerate(probs) if p > cat_thr.get(id2label[i], 0.5)})
112
+ PY
113
+ ```
114
+
115
+ - Binary unsafe score = label `unsafe` (index 1); threshold 0.5.
116
+ - Category labels use per-category calibrated thresholds (`config.json: cat_thresholds`)
117
+ and are only meaningful when `unsafe` fired.
118
+ - Scores are temperature-calibrated (`config.json: temperature`); apply
119
+ `p_cal = sigmoid(logit(p) / T)` if you need calibrated probabilities from raw logits.
120
+ - Runs on CPU out of the box; GPU via `device=0`.
121
+
122
+ ### Long inputs — always use the chunked path
123
+
124
+ Native context is 4096 tokens (`model_max_length: 4096`, tokenizer truncates beyond it),
125
+ but a **single pass pools only the last token**, so in long documents unsafe content in
126
+ the middle can be missed (measured F1 0.052 on 25-row synthetic docs). Therefore
127
+ `censer.py` / the API **always** split inputs longer than 256 tokens into overlapping
128
+ chunks (256 tokens, stride 192) and aggregate by **max** — this is the default and only
129
+ recommended path for long documents (measured F1 0.998 (acc 0.997) on the same docs):
130
+
131
+ ```python
132
+ from censer import CenserPipeline
133
+ pipe = CenserPipeline(".")
134
+ pipe.predict("very long text ...") # {"label": "1", "unsafe": true, "score": ..., "categories": [...], "chunks": N}
135
+ ```
136
+
137
+ For direct `transformers` use, pass `truncation=True, max_length=4096` (as above) or
138
+ chunk manually for documents where the middle matters.
139
+
140
+ ### Batch HTTP API + Docker
141
+
142
+ ```bash
143
+ python serve.py --dir . --port 8000
144
+ curl -s localhost:8000/classify -d '{"texts": ["hello", "how to make a bomb"]}'
145
+ docker build -t censer8 . && docker run -p 8000:8000 censer8 # includes HEALTHCHECK
146
+ ```
147
+
148
+ Endpoints: `GET /health`, `POST /classify` (batch), `POST /classify-one`.
149
+ Rate limit via `RATE_LIMIT_PER_MIN` (default 600).
150
+
151
+ ## Model details
152
+
153
+ | Property | Value |
154
+ | --- | --- |
155
+ | Architecture | `LlamaForSequenceClassification` (standard transformers) |
156
+ | Parameters | **903,552** (verified by summing safetensors tensors) |
157
+ | Layers / d_model / heads / d_ff | 3 / 128 / 4 / 384 |
158
+ | Context | 4096 tokens native (linear RoPE ×8, single `rope_parameters` key); long docs via chunked-max (default) |
159
+ | Input cap | 4000 chars (CLI) / `model_max_length` 4096 tokens (tokenizer truncates) |
160
+ | Tokenizer | byte-level BPE, 2048 tokens (multilingual corpus); chat template bakes in `[BOS][USER] … [ASSISTANT]` (verified: exactly one `[ASSISTANT]`) |
161
+ | Head | separate linear `score` head over 12 answer logits `"0" "1" "a".."j"` (untied from embeddings) |
162
+ | Output | sigmoid over 12 labels; `temperature` + `cat_thresholds` + `lang_thresholds` in config |
163
+ | Training | init from Censer.7 best; phases A–F (binary → categories → weighted must-catch/cat-balance polish, lr 1.5e-4→5e-5) on 253,258 rows; batch 256, AdamW cosine, bf16 AMP, 15% typo + 3% adversarial aug; distillation (alpha=0.65, T=4, curated boost 0.75) from Censer.5×2+.4+.3 ensemble; **must_catch rows ×40, hard_negative rows ×12**; category BCE masked to rows with category labels; val/test byte-identical to Censer.1–.7 (md5 pinned) |
164
+ | Checkpoint | 3.6 MB fp32 · 1.8 MB fp16 · 0.9 MB int8 (loadable via `censer.py --prefer-quant`) · ONNX |
165
+
166
+ ## Prompt format
167
+
168
+ `tokenizer(text)` produces `[BOS][USER] <text> [ASSISTANT]` (template baked into the
169
+ post-processor; the assistant token is appended exactly once). Answer logits are read
170
+ over `"0" "1" "a".."j"` = safe / unsafe / 10 categories.
171
+
172
+ ## Metrics
173
+
174
+ Internal held-out test split (6,496 rows, byte-identical across Censer.1–.8), binary
175
+ threshold 0.5:
176
+
177
+ | Metric | Censer.5 | Censer.6 | Censer.7 | **Censer.8** |
178
+ | --- | --- | --- | --- | --- |
179
+ | Test F1 | **0.8490** | 0.8289 | 0.8348 | **0.8198 @0.5 (best 0.8361 @ tuned thr 0.36)** |
180
+ | Test accuracy | 0.876 | 0.858 | 0.8655 | **0.8568** |
181
+ | Precision / recall | 0.864 / 0.834 | 0.833 / 0.825 | 0.855 / 0.816 | **0.8622 / 0.7813** |
182
+ | Val F1 | 0.8495 | 0.8358 | 0.8368 | **0.8259** |
183
+ | Probes (97 hand-written, published in `kuclab/probes.py`) | 95/97 | 90/97 | 95/97 | **97/97** |
184
+
185
+ Per-source test F1: civil_comments 0.750, jigsaw 0.851, aegis 0.892, beavertails 0.871, dolly 0.000, toxic_chat 0.293
186
+
187
+ ### Must-catch suite (adversarial review cases + curated cs/de/es/en)
188
+
189
+ 88 must-catch texts (CSAM requests, self-harm, murder threats, hate, drug/explosive
190
+ synthesis, DAN/roleplay jailbreaks) vs 44 hard negatives (grief, defensive security,
191
+ history, math, recipes) — `reports/mustcatch.json`:
192
+
193
+ | Metric | .7 | **.8** |
194
+ | --- | --- | --- |
195
+ | Must-catch detection @0.5 | 66/88 (75%) | **83/88 (94.3%)** |
196
+ | Hard-negative FP rate @0.5 | 9/44 (20.5%) | **3/44 (6.8%)** |
197
+
198
+ ### External benchmarks with head-to-head vs public models
199
+
200
+ ToxiGen is never in training. HateXplain **train** split is in training; the **test**
201
+ split below is held out. Full numbers in `reports/external_h2h.json` (uploaded to this
202
+ repo).
203
+
204
+ | Dataset | Model | F1@0.5 | best F1 | latency |
205
+ | --- | --- | --- | --- | --- |
206
+ | ToxiGen annotated (n=940, 402 unsafe) | **Censer.8** | **0.552** | 0.619 @ 0.17 | 1.27 ms |
207
+ | ToxiGen | unitary/unbiased-toxic-roberta | 0.443 | — | ~14 ms |
208
+ | HateXplain test (n=1,924, 1,376 unsafe) | **Censer.8** | **0.730** | 0.834 @ 0.05 | 1.27 ms |
209
+ | HateXplain test | unitary/unbiased-toxic-roberta | 0.577 | — | ~14 ms |
210
+
211
+ ### Adversarial robustness (detection rate on attacked unsafe texts, n=160)
212
+
213
+ | Attack | .7 | **.8** |
214
+ | --- | --- | --- |
215
+ | clean | 93.8% | **100.0%** |
216
+ | typo | 88.8% | **92.5%** |
217
+ | leetspeak | 93.1% | **96.9%** |
218
+ | homoglyph | 98.1% | **100.0%** |
219
+ | zero-width | 97.5% | **100.0%** |
220
+ | case flip | 91.9% | **96.2%** |
221
+
222
+ ### Long-context (300 synthetic docs of 25 held-out rows, avg 2,178 tokens)
223
+
224
+ | Inference | F1 |
225
+ | --- | --- |
226
+ | truncate at 192 tokens | 0.571 |
227
+ | native 4096 single pass (NOT recommended for long docs) | 0.052 |
228
+ | **chunked 256/192 max (default everywhere)** | **0.998 (acc 0.997)** |
229
+
230
+ ### Calibration
231
+
232
+ Binary score: temperature scaling fitted on the val split, **T = 0.7102**, stored in
233
+ `config.json: temperature` and in the checkpoint. ECE before 0.0583 → after
234
+ **0.0294**. Per-category thresholds tuned on held-out catval (rows with real
235
+ per-category labels; NOT on hate-only data) ship as `config.json: cat_thresholds`:
236
+
237
+ | Category | threshold | F1@thr (catval) |
238
+ | --- | --- | --- |
239
+ | violence | 0.20 | 0.235 | 0.039 |
240
+ | weapons_explosives | 0.22 | 0.237 | 0.034 |
241
+ | drugs | 0.22 | 0.029 | 0.033 |
242
+ | hate_harassment | 0.20 | 0.880 | 0.357 |
243
+ | sexual_abuse | 0.20 | 0.000 | 0.023 |
244
+ | self_harm | 0.20 | 0.000 | 0.010 |
245
+ | cybercrime_fraud | 0.20 | 0.000 | 0.030 |
246
+ | jailbreak_injection | 0.20 | 0.000 | 0.014 |
247
+ | illegal_marketplace | 0.20 | 0.000 | 0.010 |
248
+ | other_unsafe | 0.20 | 0.593 | 0.038 |
249
+
250
+ ![Reliability diagram](figs/reliability.png)
251
+
252
+ ### Multilingual — per-language binary F1 (textdetox multilingual toxicity, never in training)
253
+
254
+ Per-language thresholds (`config.json: lang_thresholds`, fitted per language on
255
+ textdetox) are applied automatically by `censer.py` / the CLI. Supported languages
256
+ (en/cs/de/es/fr/it/pt/ru) are tagged in the card metadata; **ar/zh/hi/ja are NOT
257
+ supported** (F1 ≈ 0 measured) and are deliberately not tagged.
258
+
259
+ | Language | n | F1@0.5 | best F1 @ best thr |
260
+ | --- | --- | --- | --- |
261
+ | en | 800 | 0.814 | 0.837 @ 0.71 |
262
+ | de | 800 | 0.558 | 0.677 @ 0.17 |
263
+ | es | 800 | 0.619 | 0.673 @ 0.09 |
264
+ | fr | 800 | 0.484 | 0.676 @ 0.08 |
265
+ | it | 800 | 0.568 | 0.677 @ 0.05 |
266
+ | ru | 800 | 0.223 | 0.636 @ 0.05 |
267
+ | uk | 800 | 0.247 | 0.630 @ 0.05 |
268
+ | tt | 800 | 0.433 | 0.639 @ 0.05 |
269
+ | hin | 800 | 0.503 | 0.639 @ 0.10 |
270
+ | am | 800 | 0.102 | 0.423 @ 0.05 |
271
+ | ar | 800 | 0.005 | 0.100 @ 0.05 |
272
+ | he | 800 | 0.029 | 0.199 @ 0.05 |
273
+ | hi | 800 | 0.000 | 0.010 @ 0.05 |
274
+ | zh | 800 | 0.000 | 0.209 @ 0.05 |
275
+ | ja | 800 | 0.000 | 0.000 @ 0.50 |
276
+
277
+ ### Cascade pre-filter
278
+
279
+ On 2,000 held-out rows: 30.2% escalated (gray-zone / jailbreak-category
280
+ hits), cascade accuracy 0.8750.
281
+
282
+ ## Benchmark (latency / throughput / size)
283
+
284
+ Single latency number everywhere below: **1.27 ms** single-text p50 on
285
+ AMD Radeon RX 9060 XT (ROCm); CPU-only: **0.55 ms** p50. Full distributions in
286
+ `reports/benchmark.json` / `reports/benchmark_cpu.json`.
287
+
288
+ | Metric | .8 GPU | .8 CPU |
289
+ | --- | --- | --- |
290
+ | Parameters | 903,552 | 903,552 |
291
+ | Checkpoint | 3.6 MB fp32 (1.8 MB fp16, 0.9 MB int8) | same |
292
+ | Single-text latency p50 | 1.27 ms | 0.55 ms |
293
+ | Single throughput | 790 texts/s | 1818 texts/s |
294
+ | Batch-8 throughput | 5659 texts/s | 5691 texts/s |
295
+ | Peak VRAM | 85 MB | — |
296
+
297
+ ## Comparison — internal test split (6,496 rows, threshold 0.5)
298
+
299
+ ⚠️ This comparison is on **our internal test split** (label noise from scraped sources);
300
+ see the external head-to-head above for public benchmarks.
301
+
302
+ | Model | Params | Checkpoint | F1 | Latency | Throughput (b=8) |
303
+ | --- | --- | --- | --- | --- | --- |
304
+ | KucLab/Censer.8 (this work) | 0.9M | 3.6 MB | 0.8198 | 1.37 ms | 1843/s |
305
+ | KucLab/Censer.5 (prev) | 0.9M | 3.6 MB | 0.8490 | 1.29 ms | 1875/s |
306
+ | KucLab/Censer.4 (old) | 0.9M | 3.6 MB | 0.8453 | 1.32 ms | 1818/s |
307
+ | KucLab/Censer.3 (old) | 14.4M | 57.7 MB | 0.8224 | 4.49 ms | 272/s |
308
+ | KucLab/Censer.2 (old) | 7.3M | 29.4 MB | 0.8289 | 2.72 ms | 549/s |
309
+ | KucLab/Censer.1 (baseline) | 1.1M | 4.5 MB | 0.8355 | 1.72 ms | 1682/s |
310
+ | keyword baseline | 0.0M | 0.0 MB | 0.0937 | 0.03 ms | 29866/s |
311
+ | martin-ha/toxic-comment-model | 67.0M | 0.0 MB | 0.5664 | 7.79 ms | 129/s |
312
+ | unitary/toxic-bert | 109.5M | 0.0 MB | 0.6807 | 14.04 ms | 68/s |
313
+ | unitary/unbiased-toxic-roberta | 124.7M | 0.0 MB | 0.7411 | 14.81 ms | 65/s |
314
+ | Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 0.0 MB | 0.2523 | 14.52 ms | 66/s |
315
+ | Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 0.0 MB | 0.5433 | 14.22 ms | 68/s |
316
+
317
+ ![Comparison overview](figs/comparison_overview.png)
318
+ ![F1 per megabyte](figs/f1_vs_size.png)
319
+ ![Metric bars](figs/metrics_bars.png)
320
+
321
+ ## Usage without transformers
322
+
323
+ ```bash
324
+ hf download KucLab/Censer.8 --local-dir Censer.8
325
+ cd Censer.8
326
+ python censer.py "how to make a bomb at home" --json
327
+ python censer.py --prefer-quant "..." # loads model_int8.safetensors (0.9 MB)
328
+ ./censer.8 # interactive UI
329
+ ```
330
+
331
+ Files:
332
+
333
+ | File | Purpose |
334
+ | --- | --- |
335
+ | `model.safetensors` | standard Llama layout, 3.6 MB fp32, 903,552 params |
336
+ | `model_fp16.safetensors` / `model_int8.safetensors` + `quant_report.json` | 1.8 MB / 0.9 MB quantized — **int8 is loadable**: `censer.py` dequantizes `.q`/`.scale` tensors automatically (`--prefer-quant`); `quant_report.json` reports fp32/fp16/int8 separately |
337
+ | `censer8.onnx` | ONNX graph: input_ids -> 12 logits, dynamic batch/seq |
338
+ | `config.json` | standard config + `temperature`, `cat_thresholds`, `lang_thresholds`, single `rope_parameters` |
339
+ | `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, prompt template baked in, `model_max_length: 4096` |
340
+ | `censer.py` | standalone inference, chunked-max for long inputs (default), int8/fp16 loaders, per-language thresholds |
341
+ | `censer.8` | launcher for the interactive terminal UI |
342
+ | `serve.py` | batch HTTP API (stdlib only) |
343
+ | `Dockerfile` | pinned CPU container serving the API, with HEALTHCHECK |
344
+ | `reports/` | all eval JSONs (external H2H, calibration, must-catch, multilingual, adversarial, …) + `figs/` |
345
+
346
+ ## Limitations
347
+
348
+ - 903K params: nuance/sarcasm/long-tail out of reach; internal test split has label
349
+ noise from scraped sources.
350
+ - Native single-pass 4096-token inference pools the last token only — for long
351
+ documents ALWAYS use the chunked-max path (default in `censer.py`/API) or
352
+ `truncation=True, max_length=4096` with direct transformers.
353
+ - Category scores use calibrated per-category thresholds, but remain less reliable
354
+ than the binary verdict; they are only reported when `unsafe` fired.
355
+ - Supported languages: en/cs/de/es/fr/it/pt/ru (measured). Other languages (ar/zh/hi/ja
356
+ …) are not supported and not claimed.
357
+ - A moderation signal, not a safety guarantee; use the cascade pattern (clear cases
358
+ decided by Censer.8, gray-zone/jailbreak hits escalated to a larger validator).
censer.8 ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """censer.8 - run the Censer.8 safety classifier as an interactive terminal UI.
3
+
4
+ Usage:
5
+ censer.8 # interactive: type a phrase, Enter -> safe/unsafe + categories + score
6
+ censer.8 "how to make a bomb" # one-shot
7
+ censer.8 --help
8
+
9
+ Expects censer.py, model.safetensors, config.json and tokenizer.json next to
10
+ this file (they are found automatically, whatever the current directory is).
11
+ Get them all with:
12
+
13
+ hf download KucLab/Censer.8 --local-dir .
14
+
15
+ Requires: torch, tokenizers, safetensors (pip install torch tokenizers safetensors)
16
+ """
17
+
18
+ import os
19
+ import sys
20
+
21
+ HERE = os.path.dirname(os.path.abspath(__file__))
22
+ sys.path.insert(0, HERE)
23
+
24
+ from censer import main
25
+
26
+ if __name__ == "__main__":
27
+ argv = sys.argv[1:]
28
+ if not any(a == "--model-dir" or a.startswith("--model-dir=") for a in argv):
29
+ argv = ["--model-dir", HERE] + argv
30
+ main(argv)
censer.py ADDED
@@ -0,0 +1,443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Inference for Censer.8 — self-contained, no dependency on the training repo.
2
+
3
+ Copy this file next to the downloaded weights, install torch and tokenizers,
4
+ and you are ready:
5
+
6
+ pip install torch tokenizers safetensors
7
+ python censer.py "how to make a bomb at home"
8
+ python censer.py # or just run it: interactive UI, phrase in -> verdict out
9
+ ./censer.8 # same UI via the censer.8 launcher next to this file
10
+
11
+ It expects, in the same directory:
12
+ model.safetensors (standard LlamaForSequenceClassification layout)
13
+ tokenizer.json
14
+ config.json
15
+ """
16
+
17
+ from __future__ import annotations
18
+
19
+ import argparse
20
+ import json
21
+ import math
22
+ import sys
23
+ from pathlib import Path
24
+
25
+ import torch
26
+ import torch.nn as nn
27
+ import torch.nn.functional as F
28
+ from tokenizers import Tokenizer
29
+
30
+ PAD, BOS, USER, ASSISTANT = "[PAD]", "[BOS]", "[USER]", "[ASSISTANT]"
31
+
32
+
33
+ class RMSNorm(nn.Module):
34
+ def __init__(self, dim: int, eps: float = 1e-5) -> None:
35
+ super().__init__()
36
+ self.eps = eps
37
+ self.weight = nn.Parameter(torch.ones(dim))
38
+
39
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
40
+ dtype = x.dtype
41
+ x = x.float()
42
+ x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
43
+ return x.to(dtype) * self.weight
44
+
45
+
46
+ def build_rope_cache(seq_len: int, head_dim: int, device, dtype, base: float = 10000.0, rope_scaling: dict | None = None):
47
+ half = head_dim // 2
48
+ freqs = 1.0 / (base ** (torch.arange(0, half, device=device, dtype=torch.float32) / half))
49
+ if rope_scaling and float(rope_scaling.get("factor", 1.0)) > 1.0:
50
+ freqs = freqs / float(rope_scaling["factor"]) # HF "linear" scaling: inv_freq /= factor
51
+ positions = torch.arange(seq_len, device=device, dtype=torch.float32)
52
+ angles = torch.outer(positions, freqs)
53
+ cos = torch.cat((angles.cos(), angles.cos()), dim=-1).to(dtype)[:, None, :]
54
+ sin = torch.cat((angles.sin(), angles.sin()), dim=-1).to(dtype)[:, None, :]
55
+ return cos, sin
56
+
57
+
58
+ def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
59
+ seq_len = x.shape[1]
60
+ cos, sin = cos[:seq_len], sin[:seq_len]
61
+ x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
62
+ rotated = torch.cat((-x2, x1), dim=-1)
63
+ return x * cos + rotated * sin
64
+
65
+
66
+ class Attention(nn.Module):
67
+ def __init__(self, cfg: dict) -> None:
68
+ super().__init__()
69
+ self.n_heads = cfg["n_heads"]
70
+ self.head_dim = cfg["d_model"] // cfg["n_heads"]
71
+ self.dropout = cfg["dropout"]
72
+ self.qkv = nn.Linear(cfg["d_model"], 3 * cfg["d_model"], bias=False)
73
+ self.proj = nn.Linear(cfg["d_model"], cfg["d_model"], bias=False)
74
+
75
+ def forward(self, x, cos, sin, mask):
76
+ b, t, _ = x.shape
77
+ q, k, v = self.qkv(x).split(x.shape[-1], dim=-1)
78
+ q = q.view(b, t, self.n_heads, self.head_dim)
79
+ k = k.view(b, t, self.n_heads, self.head_dim)
80
+ v = v.view(b, t, self.n_heads, self.head_dim)
81
+ q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
82
+ out = F.scaled_dot_product_attention(
83
+ q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
84
+ attn_mask=mask,
85
+ dropout_p=self.dropout if self.training else 0.0,
86
+ is_causal=mask is None,
87
+ )
88
+ return self.proj(out.transpose(1, 2).reshape(b, t, -1))
89
+
90
+
91
+ class SwiGLU(nn.Module):
92
+ def __init__(self, cfg: dict) -> None:
93
+ super().__init__()
94
+ self.gate = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
95
+ self.up = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
96
+ self.down = nn.Linear(cfg["d_ff"], cfg["d_model"], bias=False)
97
+
98
+ def forward(self, x):
99
+ return self.down(F.silu(self.gate(x)) * self.up(x))
100
+
101
+
102
+ class Block(nn.Module):
103
+ def __init__(self, cfg: dict) -> None:
104
+ super().__init__()
105
+ self.norm1 = RMSNorm(cfg["d_model"])
106
+ self.attn = Attention(cfg)
107
+ self.norm2 = RMSNorm(cfg["d_model"])
108
+ self.mlp = SwiGLU(cfg)
109
+ self.dropout = nn.Dropout(cfg["dropout"])
110
+
111
+ def forward(self, x, cos, sin, mask):
112
+ x = x + self.dropout(self.attn(self.norm1(x), cos, sin, mask))
113
+ x = x + self.dropout(self.mlp(self.norm2(x)))
114
+ return x
115
+
116
+
117
+ class Censer(nn.Module):
118
+ def __init__(self, cfg: dict) -> None:
119
+ super().__init__()
120
+ self.cfg = cfg
121
+ self.register_buffer(
122
+ "answer_id_tensor", torch.tensor(cfg["answer_ids"], dtype=torch.long), persistent=False
123
+ )
124
+ self.embed = nn.Embedding(cfg["vocab_size"], cfg["d_model"], padding_idx=cfg["pad_id"])
125
+ self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg["n_layers"])])
126
+ self.norm = RMSNorm(cfg["d_model"])
127
+ self.lm_head = nn.Linear(cfg["d_model"], cfg["vocab_size"], bias=False)
128
+ self.score = nn.Linear(cfg["d_model"], len(cfg["answer_ids"]), bias=False)
129
+ if cfg["tie_embeddings"]:
130
+ self.lm_head.weight = self.embed.weight
131
+ self._rope_cache = None
132
+
133
+ def _rope(self, device, dtype):
134
+ key = f"{device}|{dtype}"
135
+ if self._rope_cache is None or self._rope_cache[2] != key:
136
+ head_dim = self.cfg["d_model"] // self.cfg["n_heads"]
137
+ cos, sin = build_rope_cache(self.cfg["max_seq_len"], head_dim, device, dtype, rope_scaling=self.cfg.get("rope_scaling"))
138
+ self._rope_cache = (cos, sin, key)
139
+ return self._rope_cache[0], self._rope_cache[1]
140
+
141
+ def _attention_mask(self, input_ids):
142
+ b, t = input_ids.shape
143
+ pad = input_ids != self.cfg["pad_id"]
144
+ causal = torch.ones(t, t, dtype=torch.bool, device=input_ids.device).tril()
145
+ return pad[:, None, None, :] & causal[None, None, :, :]
146
+
147
+ def _hidden(self, input_ids):
148
+ cos, sin = self._rope(input_ids.device, self.embed.weight.dtype)
149
+ mask = self._attention_mask(input_ids)
150
+ x = self.embed(input_ids)
151
+ for block in self.blocks:
152
+ x = block(x, cos, sin, mask)
153
+ return self.norm(x)
154
+
155
+ def forward(self, input_ids):
156
+ return self.lm_head(self._hidden(input_ids))
157
+
158
+ @torch.no_grad()
159
+ def classify_probs(self, input_ids, temperature: float = 1.0):
160
+ """Answer-head probabilities: softmax for binary heads, sigmoid for multi-label.
161
+ `temperature` scales the logits (calibrated on val; stored in config.json)."""
162
+ hidden = self._hidden(input_ids)
163
+ non_pad = (input_ids != self.cfg["pad_id"]).sum(dim=1).clamp(min=1)
164
+ idx = (non_pad - 1)[:, None, None].expand(-1, 1, hidden.shape[-1])
165
+ logits = self.score(hidden.gather(1, idx)[:, 0, :].float()) / max(temperature, 1e-6)
166
+ if logits.shape[-1] == 2:
167
+ return logits.softmax(-1)
168
+ return logits.sigmoid()
169
+
170
+ @torch.no_grad()
171
+ def classify(self, input_ids):
172
+ return self.classify_probs(input_ids)[:, 1]
173
+
174
+
175
+
176
+ _STOPWORDS = {
177
+ "en": {"the","and","you","is","to","a","in","that","i","of","for","it","my","me","do","what","how","are"},
178
+ "cs": {"je","se","na","že","to","s","v","jsem","můj","moje","jak","co","nebo","pro","si","ty","ten"},
179
+ "de": {"der","die","das","und","ist","ich","du","nicht","mit","ein","eine","auf","für","wie","was","mein","sich"},
180
+ "es": {"el","la","de","que","y","en","un","una","es","por","con","para","mi","yo","como","qué","los","las"},
181
+ "fr": {"le","la","de","et","les","des","en","un","une","est","pour","que","qui","je","tu","mon","ma","comment"},
182
+ "it": {"il","lo","la","di","e","che","un","una","è","per","con","non","mi","io","come","sono","gli","le"},
183
+ "pt": {"o","a","de","e","que","um","uma","em","para","com","não","eu","meu","minha","como","os","as"},
184
+ }
185
+
186
+ def detect_lang(text: str) -> str:
187
+ """Heuristic language guess (scripts first, then Latin stopwords)."""
188
+ import unicodedata, re as _re
189
+ for ch in text:
190
+ if ch.isalpha():
191
+ name = unicodedata.name(ch, "")
192
+ if "CYRILLIC" in name:
193
+ return "uk" if any(c in text for c in "їєії") else "ru"
194
+ if "HAN" in name:
195
+ return "zh"
196
+ if "HIRAGANA" in name or "KATAKANA" in name:
197
+ return "ja"
198
+ if "HANGUL" in name:
199
+ return "ko"
200
+ if "ARABIC" in name:
201
+ return "ar"
202
+ if "DEVANAGARI" in name:
203
+ return "hi"
204
+ if "HEBREW" in name:
205
+ return "he"
206
+ break
207
+ words = _re.findall(r"[a-zà-ÿ']+", text.lower())
208
+ best, bn = "en", 0
209
+ for lang, stops in _STOPWORDS.items():
210
+ n = sum(1 for w in words if w in stops)
211
+ if n > bn:
212
+ best, bn = lang, n
213
+ return best if bn > 0 else "en"
214
+
215
+
216
+
217
+ def load_state_dict_any(directory) -> dict:
218
+ """Load model weights from model_int8.safetensors (dequantized) if present,
219
+ else model_fp16.safetensors, else model.safetensors."""
220
+ from pathlib import Path as _P
221
+ from safetensors.torch import load_file
222
+
223
+ d = _P(directory)
224
+ if (d / "model_int8.safetensors").exists():
225
+ st = load_file(str(d / "model_int8.safetensors"))
226
+ out = {}
227
+ for k, v in st.items():
228
+ if k.endswith(".q"):
229
+ base = k[:-2]
230
+ out[base] = (v.float() * st[base + ".scale"]).to(torch.float32)
231
+ elif not k.endswith(".scale"):
232
+ out[k] = v
233
+ return out
234
+ if (d / "model_fp16.safetensors").exists():
235
+ return {k: v.float() for k, v in load_file(str(d / "model_fp16.safetensors")).items()}
236
+ return load_file(str(d / "model.safetensors"))
237
+
238
+
239
+ def remap_hf_state(state: dict) -> dict:
240
+ """Standard Llama-layout safetensors -> training-layout state dict."""
241
+ import re
242
+
243
+ out = {"embed.weight": state["model.embed_tokens.weight"]}
244
+ layer_ids = sorted({int(m.group(1)) for k in state if (m := re.match(r"model\.layers\.(\d+)\.", k))})
245
+ for i in layer_ids:
246
+ p, b = f"model.layers.{i}.", f"blocks.{i}."
247
+ qkv = torch.cat(
248
+ [state[p + "self_attn.q_proj.weight"], state[p + "self_attn.k_proj.weight"], state[p + "self_attn.v_proj.weight"]],
249
+ dim=0,
250
+ )
251
+ out[b + "attn.qkv.weight"] = qkv.contiguous()
252
+ out[b + "attn.proj.weight"] = state[p + "self_attn.o_proj.weight"]
253
+ out[b + "mlp.gate.weight"] = state[p + "mlp.gate_proj.weight"]
254
+ out[b + "mlp.up.weight"] = state[p + "mlp.up_proj.weight"]
255
+ out[b + "mlp.down.weight"] = state[p + "mlp.down_proj.weight"]
256
+ out[b + "norm1.weight"] = state[p + "input_layernorm.weight"]
257
+ out[b + "norm2.weight"] = state[p + "post_attention_layernorm.weight"]
258
+ out["norm.weight"] = state["model.norm.weight"]
259
+ if "score.weight" in state:
260
+ out["score.weight"] = state["score.weight"]
261
+ return out
262
+
263
+
264
+ def internal_cfg(cfg: dict) -> dict:
265
+ """config.json (Llama layout) -> the dict the Censer module expects."""
266
+ return {
267
+ "vocab_size": cfg["vocab_size"],
268
+ "n_layers": cfg["num_hidden_layers"],
269
+ "n_heads": cfg["num_attention_heads"],
270
+ "d_model": cfg["hidden_size"],
271
+ "d_ff": cfg["intermediate_size"],
272
+ "max_seq_len": cfg["max_position_embeddings"],
273
+ "dropout": cfg.get("dropout", 0.0),
274
+ "tie_embeddings": True,
275
+ "pad_id": cfg.get("pad_token_id", 0),
276
+ "answer_ids": cfg["answer_ids"],
277
+ "max_chars": cfg.get("max_chars", 4000),
278
+ "rope_scaling": cfg.get("rope_scaling") or (cfg.get("rope_parameters") if cfg.get("rope_parameters", {}).get("rope_type") == "linear" else None),
279
+ "categories": [cfg["id2label"][str(i)] for i in range(2, cfg.get("num_labels", 2))] if cfg.get("num_labels", 2) > 2 else [],
280
+ "cat_thresholds": cfg.get("cat_thresholds") or {},
281
+ "temperature": float(cfg.get("temperature", 1.0) or 1.0),
282
+ "lang_thresholds": cfg.get("lang_thresholds") or {},
283
+ }
284
+
285
+
286
+ class CenserPipeline:
287
+ """Text in, "1" (unsafe) or "0" (safe) out."""
288
+
289
+ def __init__(self, directory: str | Path = ".", threshold: float = 0.5, device: str | None = None, prefer_quant: bool = False) -> None:
290
+ directory = Path(directory)
291
+ self.cfg = internal_cfg(json.loads((directory / "config.json").read_text(encoding="utf-8")))
292
+ self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
293
+ self.pad_id = self.tok.token_to_id(PAD)
294
+ self.bos_id = self.tok.token_to_id(BOS)
295
+ self.user_id = self.tok.token_to_id(USER)
296
+ self.assistant_id = self.tok.token_to_id(ASSISTANT)
297
+ self.threshold = threshold
298
+ self.cat_thresholds = dict(self.cfg.get("cat_thresholds") or {})
299
+ self.temperature = float(self.cfg.get("temperature", 1.0) or 1.0)
300
+ self.lang_thresholds = dict(self.cfg.get("lang_thresholds") or {})
301
+ self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
302
+ self.model = Censer(self.cfg)
303
+ if prefer_quant and (directory / "model_int8.safetensors").exists():
304
+ state = remap_hf_state(load_state_dict_any(directory))
305
+ else:
306
+ from safetensors.torch import load_file
307
+ state = remap_hf_state(load_file(directory / "model.safetensors"))
308
+ missing, unexpected = self.model.load_state_dict(state, strict=False)
309
+ missing = [k for k in missing if k != "lm_head.weight"] # tied to embed.weight
310
+ if missing or unexpected:
311
+ raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
312
+ self.model.to(self.device).eval()
313
+ self.max_chars = self.cfg["max_chars"]
314
+
315
+ def _encode(self, text: str) -> list[int]:
316
+ body = self.tok.encode(text[: self.max_chars], add_special_tokens=False).ids
317
+ ids = [self.bos_id, self.user_id, *body, self.assistant_id]
318
+ return ids[: self.cfg["max_seq_len"]]
319
+
320
+ def predict(self, text: str) -> dict:
321
+ return self.predict_batch([text])[0]
322
+
323
+ def _encode_full(self, text: str) -> list[int]:
324
+ body = self.tok.encode(text[: self.max_chars * 8], add_special_tokens=False).ids
325
+ return [self.bos_id, self.user_id, *body, self.assistant_id]
326
+
327
+ def predict_batch(self, texts: list[str]) -> list[dict]:
328
+ # long inputs: split into overlapping token chunks, aggregate by max
329
+ out: list[dict] = []
330
+ chunk_tokens, stride = 256, 192
331
+ for text in texts:
332
+ thr = self.threshold_for(text)
333
+ ids = self._encode_full(text)
334
+ body = ids[2:-1]
335
+ if len(body) <= chunk_tokens:
336
+ out.append(self._classify_ids([ids[: self.cfg["max_seq_len"]]], [thr])[0])
337
+ continue
338
+ chunks = []
339
+ start = 0
340
+ while start < len(body):
341
+ chunks.append(body[start:start + chunk_tokens])
342
+ if start + chunk_tokens >= len(body):
343
+ break
344
+ start += stride
345
+ rows = []
346
+ for piece in chunks:
347
+ cids = [self.bos_id, self.user_id, *piece, self.assistant_id]
348
+ rows.append(self._classify_ids([cids[: self.cfg["max_seq_len"]]], [thr])[0])
349
+ best = max(rows, key=lambda r: r["score"])
350
+ # aggregate categories by max prob
351
+ n_cat = len(rows[0].get("category_scores", []))
352
+ cat_scores = [max(r["category_scores"][i] for r in rows) for i in range(n_cat)] if n_cat else []
353
+ best["category_scores"] = [round(c, 4) for c in cat_scores]
354
+ cats = self.categories()
355
+ best["categories"] = ([c for c, p in zip(cats, cat_scores) if p >= float(self.cat_thresholds.get(c, self.threshold))]
356
+ if best["unsafe"] else [])
357
+ best["chunks"] = len(chunks)
358
+ out.append(best)
359
+ return out
360
+
361
+ def threshold_for(self, text: str) -> float:
362
+ lang = detect_lang(text)
363
+ return float(self.lang_thresholds.get(lang, self.threshold))
364
+
365
+ def categories(self) -> list[str]:
366
+ return self.cfg.get("categories") or []
367
+
368
+ def _classify_ids(self, seqs: list[list[int]], thresholds: list[float] | None = None) -> list[dict]:
369
+ width = max(len(s) for s in seqs)
370
+ ids = torch.full((len(seqs), width), self.pad_id, dtype=torch.long)
371
+ for i, seq in enumerate(seqs):
372
+ ids[i, : len(seq)] = torch.tensor(seq)
373
+ probs = self.model.classify_probs(ids.to(self.device), temperature=self.temperature).float().cpu().tolist()
374
+ if thresholds is None:
375
+ thresholds = [self.threshold] * len(seqs)
376
+ res = []
377
+ for i, row in enumerate(probs):
378
+ p = row[1]
379
+ cats = self.categories()
380
+ cat_scores = row[2:] if cats else []
381
+ is_unsafe = p >= thresholds[i]
382
+ res.append({
383
+ "label": "1" if is_unsafe else "0",
384
+ "unsafe": is_unsafe,
385
+ "score": round(p, 4),
386
+ "category_scores": [round(c, 4) for c in cat_scores],
387
+ # categories are only reported when the binary verdict is unsafe
388
+ "categories": ([c for c, sc in zip(cats, cat_scores)
389
+ if sc >= float(self.cat_thresholds.get(c, self.threshold))]
390
+ if is_unsafe else []),
391
+ })
392
+ return res
393
+
394
+
395
+ def _color(text: str, code: str) -> str:
396
+ if not sys.stdout.isatty():
397
+ return text
398
+ return f"\033[{code}m{text}\033[0m"
399
+
400
+
401
+ def repl(pipe: "CenserPipeline") -> int:
402
+ """Small interactive terminal UI: phrase in, safe/unsafe + score out."""
403
+ print("Censer.8 - interactive terminal UI")
404
+ print("Type a phrase, Enter to classify. 'exit' to quit.\n")
405
+ while True:
406
+ try:
407
+ line = input("> ")
408
+ except (EOFError, KeyboardInterrupt):
409
+ print()
410
+ return 0
411
+ text = line.strip()
412
+ if text.lower() in {"exit", "quit", ":q"}:
413
+ return 0
414
+ if not text:
415
+ continue
416
+ v = pipe.predict(text)
417
+ label = "unsafe" if v["unsafe"] else "safe"
418
+ colored = _color(f"{label:6s}", "31" if v["unsafe"] else "32")
419
+ print(f"{colored} {v['score']:.4f} | {text}")
420
+
421
+
422
+ def main(argv: list[str] | None = None) -> None:
423
+ parser = argparse.ArgumentParser(
424
+ prog="censer.8",
425
+ description="Censer.8 safety classifier (standalone). Run without text to open the interactive UI.",
426
+ )
427
+ parser.add_argument("text", nargs="*", help="phrase to classify once; omit to open the interactive UI")
428
+ parser.add_argument("--model-dir", default=".")
429
+ parser.add_argument("--threshold", type=float, default=0.5)
430
+ parser.add_argument("--json", action="store_true")
431
+ parser.add_argument("--repl", action="store_true", help="interactive UI (default when no text is given)")
432
+ args = parser.parse_args(argv)
433
+
434
+ pipe = CenserPipeline(args.model_dir, args.threshold)
435
+ text = " ".join(args.text).strip()
436
+ if args.repl or not text:
437
+ raise SystemExit(repl(pipe))
438
+ verdict = pipe.predict(text)
439
+ print(json.dumps(verdict) if args.json else verdict["label"])
440
+
441
+
442
+ if __name__ == "__main__":
443
+ main()
config.json ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForSequenceClassification"
4
+ ],
5
+ "model_type": "llama",
6
+ "num_labels": 12,
7
+ "id2label": {
8
+ "0": "safe",
9
+ "1": "unsafe",
10
+ "2": "violence",
11
+ "3": "weapons_explosives",
12
+ "4": "drugs",
13
+ "5": "hate_harassment",
14
+ "6": "sexual_abuse",
15
+ "7": "self_harm",
16
+ "8": "cybercrime_fraud",
17
+ "9": "jailbreak_injection",
18
+ "10": "illegal_marketplace",
19
+ "11": "other_unsafe"
20
+ },
21
+ "label2id": {
22
+ "safe": 0,
23
+ "unsafe": 1,
24
+ "violence": 2,
25
+ "weapons_explosives": 3,
26
+ "drugs": 4,
27
+ "hate_harassment": 5,
28
+ "sexual_abuse": 6,
29
+ "self_harm": 7,
30
+ "cybercrime_fraud": 8,
31
+ "jailbreak_injection": 9,
32
+ "illegal_marketplace": 10,
33
+ "other_unsafe": 11
34
+ },
35
+ "problem_type": "multi_label_classification",
36
+ "vocab_size": 2048,
37
+ "hidden_size": 128,
38
+ "intermediate_size": 384,
39
+ "num_hidden_layers": 3,
40
+ "num_attention_heads": 4,
41
+ "num_key_value_heads": 4,
42
+ "head_dim": 32,
43
+ "max_position_embeddings": 4096,
44
+ "rms_norm_eps": 1e-05,
45
+ "rope_parameters": {
46
+ "rope_type": "linear",
47
+ "factor": 8.0,
48
+ "rope_theta": 10000.0,
49
+ "original_max_position_embeddings": 512
50
+ },
51
+ "attention_bias": false,
52
+ "mlp_bias": false,
53
+ "tie_word_embeddings": true,
54
+ "pad_token_id": 0,
55
+ "bos_token_id": 1,
56
+ "eos_token_id": 3,
57
+ "name": "Censer.8",
58
+ "max_chars": 4000,
59
+ "answer_ids": [
60
+ 19,
61
+ 20,
62
+ 68,
63
+ 69,
64
+ 70,
65
+ 71,
66
+ 72,
67
+ 73,
68
+ 74,
69
+ 75,
70
+ 76,
71
+ 77
72
+ ],
73
+ "special_tokens": [
74
+ "[PAD]",
75
+ "[BOS]",
76
+ "[USER]",
77
+ "[ASSISTANT]"
78
+ ],
79
+ "threshold": 0.5,
80
+ "cat_thresholds": {
81
+ "violence": 0.2,
82
+ "weapons_explosives": 0.22,
83
+ "drugs": 0.22,
84
+ "hate_harassment": 0.2,
85
+ "sexual_abuse": 0.5,
86
+ "self_harm": 0.5,
87
+ "cybercrime_fraud": 0.5,
88
+ "jailbreak_injection": 0.5,
89
+ "illegal_marketplace": 0.5,
90
+ "other_unsafe": 0.2
91
+ },
92
+ "temperature": 0.7102126181125441,
93
+ "lang_thresholds": {
94
+ "de": 0.17,
95
+ "en": 0.71,
96
+ "es": 0.09,
97
+ "fr": 0.08,
98
+ "hin": 0.1,
99
+ "it": 0.05,
100
+ "ru": 0.05,
101
+ "tt": 0.05,
102
+ "uk": 0.05
103
+ }
104
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eb82a7824aedcec3deadc1dbf928b0cb10ad48157e4bb41e8d6274581f83bc49
3
+ size 3617384
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "pad_token": "[PAD]",
4
+ "bos_token": "[BOS]",
5
+ "eos_token": "[ASSISTANT]",
6
+ "model_max_length": 4096
7
+ }