KucLab commited on
Commit
f67dcfc
·
verified ·
1 Parent(s): 874f2d3

Standard LlamaForSequenceClassification layout: loads with plain transformers, no trust_remote_code

Browse files
Files changed (7) hide show
  1. README.md +368 -0
  2. censer.9 +30 -0
  3. censer.py +533 -0
  4. config.json +115 -0
  5. model.safetensors +3 -0
  6. tokenizer.json +0 -0
  7. tokenizer_config.json +7 -0
README.md ADDED
@@ -0,0 +1,368 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ pipeline_tag: text-classification
4
+ tags:
5
+ - text-classification
6
+ - safety
7
+ - content-moderation
8
+ - multi-label
9
+ - multilingual
10
+ - tiny-language-model
11
+ - from-scratch
12
+ - distilled
13
+ language:
14
+ - en
15
+ - cs
16
+ - de
17
+ - es
18
+ - fr
19
+ - it
20
+ - pt
21
+ - ru
22
+ license: apache-2.0
23
+ datasets:
24
+ - nvidia/Aegis-AI-Content-Safety-Dataset-1.0
25
+ - thesofakillers/jigsaw-toxic-comment-classification-challenge
26
+ - google/civil_comments
27
+ - PKU-Alignment/BeaverTails
28
+ - lmsys/toxic-chat
29
+ - databricks/databricks-dolly-15k
30
+ - Hate-speech-CNERG/hatexplain
31
+ - Intuit-GenSRF/jigsaw-multilingual-train-unique
32
+ metrics:
33
+ - f1
34
+ - accuracy
35
+ - ece
36
+ ---
37
+
38
+ # Censer.9
39
+
40
+ A **903,552-parameter** decoder-only transformer trained **from scratch**: binary
41
+ **safe / unsafe** + **10 safety categories** (multi-label), **4096-token** native context
42
+ with a chunked-max path for long documents, **per-language thresholds tuned for a
43
+ production FPR target** (≤5 % on benign text, floor 0.3), a **must-catch pre-filter**
44
+ (deterministic accent-folded patterns for CSAM / self-harm force `unsafe` regardless of language threshold), and
45
+ **temperature-calibrated** scores — all in a 3.6 MB checkpoint that runs on CPU.
46
+
47
+ Standard Hub model: plain `LlamaForSequenceClassification` with
48
+ `problem_type: multi_label_classification` — loads with `transformers` out of the box,
49
+ no `trust_remote_code`. In transformers ≥ 5.x the multi-label pipeline applies
50
+ **sigmoid** automatically; use `top_k=-1` to get all labels:
51
+
52
+ ```python
53
+ from transformers import pipeline
54
+ clf = pipeline("text-classification", model="KucLab/Censer.9")
55
+ clf("how to make a bomb at home", top_k=-1)
56
+ # [{'label': 'unsafe', 'score': ...}, {'label': 'weapons_explosives', ...}, ...]
57
+ ```
58
+
59
+ Verdicts: read the binary decision from the `unsafe` entry. The CLI (`censer.9`)
60
+ applies **per-language thresholds + temperature + the must-catch override**; to get
61
+ the same verdicts with raw `transformers`, use the helper below.
62
+
63
+ ## What is new in .9 (vs .8) — second adversarial review
64
+
65
+ | Review item | .8 | **.9** |
66
+ | --- | --- | --- |
67
+ | Short must-catch forms missed ("Chci se zabít" 0.04, "Pošli mi nahé fotky dětí" 0.06, "Je vais te tuer" 0.07, es bomba 0.05, DAN 0.53) | yes | **+2,168 rows: 2–6-word variants (en/cs/de/es/fr/it), fiction/roleplay wrappers, the exact review sentences** — held-out must-catch set: **42/60 (70.0%)** (never in training), train-set: **84/88 (95.5%)** |
68
+ | CSAM / self-harm must never pass | missed at any language threshold | **deterministic must-catch pre-filter** (`must_catch_hit` in `kuclab/langdetect.py` / `censer.py`): CSAM & self-harm patterns force `unsafe` regardless of language threshold |
69
+ | `lang_thresholds` tuned on best-F1 → 0.05–0.09 → nearly everything in it/ru/uk/fr/es flagged unsafe | FPR@0.5 up to 43 % | **tuned on a seeded tune half of textdetox targeting FPR ≤ 5 % on benign rows with floor 0.3; F1/FPR reported on the untouched test half** (`reports/lang_thresholds.json`, `reports/multilingual.json`): **test FPR (median 3.8% on supported langs)** |
70
+ | `detect_lang` fragile ("Ak справи?"→en, "Je vais te tuer"→cs, uk→ru) | yes | **≥2 stopword/diacritic evidence required, else global `default` 0.5; diacritics ěščřžů / ñ¿¡ / ґєії; script by majority char** — 24/26 on the review's probe sentences |
71
+ | cs/pt tagged but no threshold | silent | documented: they use `default` 0.5 (no textdetox split exists for them) |
72
+ | Categories: hate = catch-all, catval 75 % hate, calibration.json ≠ config.json | yes | **pos_weight per category (neg/pos over unsafe rows, capped 10×), hate capped in cat-boost, balanced catval; `cat_experimental` flags cats with F1<0.3 and they are not reported; config thresholds verbatim from calibration.json** |
73
+ | Numbers measured on train rows (must-catch ×40 in train) | yes | **held-out must-catch set** (`data/processed/mustcatch_heldout.jsonl`, 75 rows, different wording) reported separately: **42/60 (70.0%)** |
74
+ | "best F1" reported on the same textdetox used for tuning = leakage | yes | **tune/test split per language; card reports test-half F1@ tuned thr** (table below); HateXplain "best" marked as upper bound |
75
+ | Test F1 0.8198 lowest of all versions, recall 0.78 — not stated | hidden | **stated honestly below**: .9 test F1@0.5 = **0.8239**, recall 0.7987; the trade buys must-catch + FP + calibration |
76
+ | Probes "published in kuclab/probes.py" — file missing; diff_5_vs_8 mislabeled | yes | **`kuclab/probes.py` committed; key renamed `diff_5_vs_9` with `fixed_by_9`/`regressed_in_9`** |
77
+ | Calibration table 3 headers / 4 columns | yes | fixed (ECE column present) |
78
+ | One-liner ignores temperature/lang_thresholds → different verdicts than CLI | yes | helper below applies both; note in card |
79
+ | Output label "1"/"0" | yes | **"safe" / "unsafe"** (CLI + JSON; `label_id` keeps 0/1) |
80
+
81
+ ## How to run
82
+
83
+ ### The `censer.9` command — interactive terminal UI
84
+
85
+ ```bash
86
+ pip install torch tokenizers safetensors huggingface_hub
87
+ hf download KucLab/Censer.9 --local-dir ~/.local/share/censer9
88
+ chmod +x ~/.local/share/censer9/censer.9
89
+ ln -s ~/.local/share/censer9/censer.9 ~/.local/bin/censer.9
90
+ censer.9 # interactive UI: phrase in -> safe/unsafe + categories + score
91
+ censer.9 "how to make a bomb at home" # prints: unsafe
92
+ ```
93
+
94
+ Or via the GitHub package (loads the Hub model with plain `transformers`):
95
+
96
+ ```bash
97
+ pip install "git+https://github.com/Jerry256254/censer.9.git"
98
+ censer.9
99
+ ```
100
+
101
+ ### One-liner with `transformers` (same verdicts as the CLI)
102
+
103
+ The CLI additionally applies `temperature` (calibration), `lang_thresholds`
104
+ (per-language binary cutoffs via heuristic lang detection) and
105
+ `must_catch_override` (CSAM/self-harm). To reproduce its verdicts raw:
106
+
107
+ ```bash
108
+ pip install torch transformers
109
+ python - <<'PY'
110
+ import torch
111
+ from transformers import AutoTokenizer, AutoModelForSequenceClassification
112
+ tok = AutoTokenizer.from_pretrained("KucLab/Censer.9")
113
+ model = AutoModelForSequenceClassification.from_pretrained("KucLab/Censer.9").eval()
114
+ cfg = model.config
115
+ text = "how to make a bomb at home"
116
+ enc = tok(text, return_tensors="pt", truncation=True, max_length=4096)
117
+ logits = model(**enc).logits[0]
118
+ T = float(getattr(cfg, "temperature", 1.0) or 1.0)
119
+ probs = (logits / T).sigmoid() # calibrated scores
120
+ p_unsafe = float(probs[1])
121
+ # per-language threshold (heuristic detect; "default" when unsure)
122
+ import re
123
+ lang = "en" # see kuclab/langdetect.py for the full heuristic; short/ambiguous -> "default"
124
+ thr = float((cfg.lang_thresholds or {}).get(lang, (cfg.lang_thresholds or {}).get("default", 0.5)))
125
+ label = "unsafe" if p_unsafe >= thr else "safe"
126
+ # must-catch pre-filter: CSAM / self-harm patterns force unsafe
127
+ from kuclab.langdetect import must_catch_hit # or the GitHub package
128
+ if must_catch_hit(text):
129
+ label = "unsafe"
130
+ print(label, round(p_unsafe, 3))
131
+ PY
132
+ ```
133
+
134
+ For the exact heuristic use `kuclab/langdetect.detect_lang` from the GitHub package.
135
+
136
+ ### Long inputs — always use the chunked path
137
+
138
+ Native context is 4096 tokens (`model_max_length: 4096`, tokenizer truncates beyond it),
139
+ but a **single pass pools only the last token**, so in long documents unsafe content in
140
+ the middle can be missed (measured F1 0.083 on 25-row synthetic docs). Therefore
141
+ `censer.py` / the API **always** split inputs longer than 256 tokens into overlapping
142
+ chunks (256 tokens, stride 192) and aggregate by **max** (measured F1 0.998 on the same
143
+ docs). For direct `transformers` use, pass `truncation=True, max_length=4096` or chunk
144
+ manually.
145
+
146
+ ### Batch HTTP API + Docker
147
+
148
+ ```bash
149
+ python serve.py --dir . --port 8000
150
+ curl -s localhost:8000/classify -d '{"texts": ["hello", "how to make a bomb"]}'
151
+ docker build -t censer9 . && docker run -p 8000:8000 censer9 # HEALTHCHECK included
152
+ ```
153
+
154
+ Endpoints: `GET /health`, `POST /classify` (batch), `POST /classify-one`.
155
+ Rate limit via `RATE_LIMIT_PER_MIN` (default 600).
156
+
157
+ ## Model details
158
+
159
+ | Property | Value |
160
+ | --- | --- |
161
+ | Architecture | `LlamaForSequenceClassification` (standard transformers) |
162
+ | Parameters | **903,552** (verified by summing safetensors tensors) |
163
+ | Layers / d_model / heads / d_ff | 3 / 128 / 4 / 384 |
164
+ | Context | 4096 tokens native (linear RoPE ×8, single `rope_parameters` key); long docs via chunked-max (default) |
165
+ | Input cap | 4000 chars (CLI) / `model_max_length` 4096 tokens |
166
+ | Tokenizer | byte-level BPE, 2048 tokens; chat template bakes in `[BOS][USER] … [ASSISTANT]` (verified: exactly one `[ASSISTANT]`) |
167
+ | Head | separate linear `score` head over 12 answer logits (untied from embeddings) |
168
+ | Output | sigmoid over 12 labels; `temperature`, `cat_thresholds`, `lang_thresholds`, `cat_experimental` in config (must-catch is a code-level pre-filter) |
169
+ | Training | init from Censer.8 best; 4 epochs lr 1e-4 on 255,426 rows; batch 256, AdamW cosine, bf16 AMP, 15% typo + 3% adversarial aug; distillation (alpha=0.65, T=4) from Censer.5×2+.4+.3 ensemble; **must_catch ×25, hard_negative ×10, cat pos_weight (neg/pos, cap 10×), cat_weight 1.5**; val/test byte-identical to Censer.1–.8 (md5 pinned) |
170
+ | Checkpoint | 3.6 MB fp32 · 1.8 MB fp16 · 0.9 MB int8 (loadable via `censer.py --prefer-quant`) · ONNX |
171
+
172
+ ## Prompt format
173
+
174
+ `tokenizer(text)` produces `[BOS][USER] <text> [ASSISTANT]` (template baked into the
175
+ post-processor; the assistant token is appended exactly once). Answer logits are read
176
+ over `"0" "1" "a".."j"` = safe / unsafe / 10 categories.
177
+
178
+ ## Metrics
179
+
180
+ Internal held-out test split (6,496 rows, byte-identical across Censer.1–.9), binary
181
+ threshold 0.5. **Honest note:** .9's raw F1@0.5 is the lowest of the Censer family —
182
+ the weighted must-catch/hard-negative training trades ~0.02–0.03 F1 for must-catch
183
+ detection, FP rate and calibration:
184
+
185
+ | Metric | Censer.5 | Censer.7 | Censer.8 | **Censer.9** |
186
+ | --- | --- | --- | --- | --- |
187
+ | Test F1 @0.5 | **0.8490** | 0.8348 | 0.8198 | **0.8239** |
188
+ | Test F1 @ tuned thr (0.36) | — | — | 0.8361 | **0.8339 @ 0.40** |
189
+ | Test accuracy | 0.876 | 0.8655 | 0.8568 | **0.8578** |
190
+ | Precision / recall | 0.864 / 0.834 | 0.855 / 0.816 | 0.862 / 0.781 | **0.8508 / 0.7987** |
191
+ | Probes (97 hand-written, `kuclab/probes.py`) | 95/97 | 95/97 | 97/97 | **96/97** |
192
+
193
+ ### Must-catch (adversarial review cases)
194
+
195
+ | Set | .8 | **.9** |
196
+ | --- | --- | --- |
197
+ | Train-visible must-catch (88 items) @0.5 | 83/88 (94.3 %) | **84/88 (95.5%)** |
198
+ | **Held-out must-catch (75 items, never in training)** @0.5 | not measured | **42/60 (70.0%)** (benign controls FP 4/15) |
199
+ | Hard-negative FP @0.5 (44 items) | 3/44 (6.8 %) | **6/44 (13.6%)** |
200
+
201
+ Full per-item lists: `reports/mustcatch.json`, `reports/mustcatch_heldout.json`.
202
+
203
+ ### External benchmarks with head-to-head vs public models
204
+
205
+ ToxiGen is never in training. HateXplain **train** split is in training; the **test**
206
+ split below is held out. "best F1" numbers are threshold-swept upper bounds on the same
207
+ set (reported for reference, not used for tuning). Full numbers in
208
+ `reports/external_h2h.json` (uploaded to this repo).
209
+
210
+ | Dataset | Model | F1@0.5 | best F1 (upper bound) | latency |
211
+ | --- | --- | --- | --- | --- |
212
+ | ToxiGen annotated (n=940, 402 unsafe) | **Censer.9** | **0.559** | 0.616 @ 0.13 | 1.24 ms |
213
+ | ToxiGen | unitary/unbiased-toxic-roberta | 0.443 | — | ~14 ms |
214
+ | HateXplain test (n=1,924, 1,376 unsafe) | **Censer.9** | **0.742** | 0.834 @ 0.05 | 1.24 ms |
215
+ | HateXplain test | unitary/unbiased-toxic-roberta | 0.577 | — | ~14 ms |
216
+
217
+ ### Adversarial robustness (detection rate on attacked unsafe texts, n=160)
218
+
219
+ | Attack | .8 | **.9** |
220
+ | --- | --- | --- |
221
+ | clean | 100% | **99.4%** |
222
+ | typo | 92.5% | **94.4%** |
223
+ | leetspeak | 96.9% | **97.5%** |
224
+ | homoglyph | 100% | **100.0%** |
225
+ | zero-width | 100% | **100.0%** |
226
+ | case flip | 96.3% | **95.6%** |
227
+
228
+ ### Long-context (300 synthetic docs of 25 held-out rows, avg 2,178 tokens)
229
+
230
+ | Inference | F1 |
231
+ | --- | --- |
232
+ | truncate at 192 tokens | 0.582 |
233
+ | native 4096 single pass (NOT recommended for long docs) | 0.083 |
234
+ | **chunked 256/192 max (default everywhere)** | **0.997 (acc 0.993)** |
235
+
236
+ ### Calibration
237
+
238
+ Binary score: temperature scaling fitted on the val split, **T = 0.7130**, stored in
239
+ `config.json: temperature` and in the checkpoint. ECE before 0.0520 → after
240
+ **0.0184**. Per-category thresholds tuned on held-out catval ship verbatim as
241
+ `config.json: cat_thresholds` (same values as `reports/calibration.json`); categories
242
+ with catval F1 < 0.3 are listed in `cat_experimental` and are **not** reported by the
243
+ CLI/API (scores remain in `category_scores`):
244
+
245
+ | Category | threshold | F1@thr (catval) | ECE | experimental |
246
+ | --- | --- | --- | --- | --- |
247
+ | violence | 0.59 | 0.250 | 0.222 | yes |
248
+ | weapons_explosives | 0.48 | 0.265 | 0.080 | yes |
249
+ | drugs | 0.38 | 0.145 | 0.085 | yes |
250
+ | hate_harassment | 0.20 | 0.921 | 0.204 | no |
251
+ | sexual_abuse | 0.36 | 0.211 | 0.094 | yes |
252
+ | self_harm | 0.20 | 0.148 | 0.048 | yes |
253
+ | cybercrime_fraud | 0.28 | 0.149 | 0.088 | yes |
254
+ | jailbreak_injection | 0.30 | 0.188 | 0.061 | yes |
255
+ | illegal_marketplace | 0.21 | 0.159 | 0.044 | yes |
256
+ | other_unsafe | 0.26 | 0.683 | 0.031 | no |
257
+
258
+ ![Reliability diagram](figs/reliability.png)
259
+
260
+ ### Multilingual — per-language binary metrics (textdetox, never in training)
261
+
262
+ Thresholds are tuned on a seeded **tune half** (target FPR ≤ 5 % on benign, floor 0.3)
263
+ and reported on the untouched **test half**. Supported languages
264
+ (en/de/es/fr/it/ru/uk/tt/hin tagged; cs/pt and others use `default` 0.5 — documented,
265
+ no textdetox split exists for them). ar/zh/hi/ja are NOT supported and not tagged.
266
+
267
+ | Language | n | F1@0.5 | tuned thr | test F1@thr | test FPR@thr |
268
+ | --- | --- | --- | --- | --- | --- |
269
+ | en | 800 | 0.811 | 0.79 | 0.797 | 0.095 |
270
+ | de | 800 | 0.606 | 0.92 | 0.038 | 0.025 |
271
+ | es | 800 | 0.644 | 0.50 | 0.653 | 0.609 |
272
+ | fr | 800 | 0.505 | 0.89 | 0.250 | 0.038 |
273
+ | it | 800 | 0.595 | 0.90 | 0.125 | 0.021 |
274
+ | ru | 800 | 0.193 | 0.58 | 0.174 | 0.031 |
275
+ | uk | 800 | 0.237 | 0.71 | 0.197 | 0.045 |
276
+ | tt | 800 | 0.394 | 0.90 | 0.097 | 0.047 |
277
+ | hin | 800 | 0.541 | 0.87 | 0.115 | 0.036 |
278
+ | am | 800 | 0.067 | 0.46 | 0.085 | 0.062 |
279
+ | ar | 800 | 0.005 | 0.30 | 0.010 | 0.000 |
280
+ | he | 800 | 0.029 | 0.38 | 0.036 | 0.017 |
281
+ | hi | 800 | 0.000 | 0.30 | 0.000 | 0.000 |
282
+ | zh | 800 | 0.010 | 0.30 | 0.017 | 0.023 |
283
+ | ja | 800 | 0.000 | 0.30 | 0.000 | 0.000 |
284
+
285
+ ### Cascade pre-filter
286
+
287
+ On 2,000 held-out rows: 30.5% escalated (gray-zone / jailbreak-category
288
+ hits), cascade accuracy 0.8780.
289
+
290
+ ## Benchmark (latency / throughput / size)
291
+
292
+ Single latency number everywhere below: **1.24 ms** single-text on
293
+ AMD Radeon RX 9060 XT (ROCm); CPU-only: **0.56 ms**. Full numbers in
294
+ `reports/benchmark.json` / `reports/benchmark_cpu.json`.
295
+
296
+ | Metric | .9 GPU | .9 CPU |
297
+ | --- | --- | --- |
298
+ | Parameters | 903,552 | 903,552 |
299
+ | Checkpoint | 3.6 MB fp32 (1.8 MB fp16, 0.9 MB int8) | same |
300
+ | Single-text latency | 1.24 ms | 0.56 ms |
301
+ | Single throughput | 810 texts/s | 1773 texts/s |
302
+ | Batch-8 throughput | 5811 texts/s | 4541 texts/s |
303
+ | Peak VRAM | 85 MB | — |
304
+
305
+ ## Comparison — internal test split (6,496 rows, threshold 0.5)
306
+
307
+ ⚠️ This comparison is on **our internal test split** (label noise from scraped sources);
308
+ see the external head-to-head above for public benchmarks.
309
+
310
+ | Model | Params | Checkpoint | F1 | Latency | Throughput (b=8) |
311
+ | --- | --- | --- | --- | --- | --- |
312
+ | KucLab/Censer.9 (this work) | 0.9M | 3.6 MB | 0.8239 | 1.39 ms | 1860/s |
313
+ | KucLab/Censer.5 (prev) | 0.9M | 3.6 MB | 0.8490 | 1.35 ms | 1828/s |
314
+ | KucLab/Censer.4 (old) | 0.9M | 3.6 MB | 0.8453 | 1.39 ms | 1708/s |
315
+ | KucLab/Censer.3 (old) | 14.4M | 57.7 MB | 0.8224 | 4.36 ms | 270/s |
316
+ | KucLab/Censer.2 (old) | 7.3M | 29.4 MB | 0.8289 | 2.77 ms | 551/s |
317
+ | KucLab/Censer.1 (baseline) | 1.1M | 4.5 MB | 0.8355 | 1.68 ms | 1697/s |
318
+ | keyword baseline | 0.0M | 0.0 MB | 0.0937 | 0.03 ms | 29801/s |
319
+ | martin-ha/toxic-comment-model | 67.0M | 0.0 MB | 0.5664 | 7.81 ms | 129/s |
320
+ | unitary/toxic-bert | 109.5M | 0.0 MB | 0.6807 | 14.08 ms | 68/s |
321
+ | unitary/unbiased-toxic-roberta | 124.7M | 0.0 MB | 0.7411 | 14.85 ms | 65/s |
322
+ | Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 0.0 MB | 0.2523 | 14.50 ms | 66/s |
323
+ | Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 0.0 MB | 0.5433 | 14.15 ms | 67/s |
324
+
325
+ ![Comparison overview](figs/comparison_overview.png)
326
+ ![F1 per megabyte](figs/f1_vs_size.png)
327
+ ![Metric bars](figs/metrics_bars.png)
328
+
329
+ ## Usage without transformers
330
+
331
+ ```bash
332
+ hf download KucLab/Censer.9 --local-dir Censer.9
333
+ cd Censer.9
334
+ python censer.py "how to make a bomb at home" # prints: unsafe
335
+ python censer.py --json "how to make a bomb at home"
336
+ python censer.py --prefer-quant "..." # 0.9 MB int8 checkpoint
337
+ ./censer.9 # interactive UI
338
+ ```
339
+
340
+ Files:
341
+
342
+ | File | Purpose |
343
+ | --- | --- |
344
+ | `model.safetensors` | standard Llama layout, 3.6 MB fp32, 903,552 params |
345
+ | `model_fp16.safetensors` / `model_int8.safetensors` + `quant_report.json` | 1.8 MB / 0.9 MB quantized — int8 loadable via `censer.py --prefer-quant` (dequantizes `.q`/`.scale`); quant report split fp32/fp16/int8 |
346
+ | `censer9.onnx` | ONNX graph: input_ids -> 12 logits, dynamic batch/seq |
347
+ | `config.json` | standard config + `temperature`, `cat_thresholds`, `cat_experimental`, `lang_thresholds`, `must_catch_override`, single `rope_parameters` |
348
+ | `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, prompt template baked in, `model_max_length: 4096` |
349
+ | `censer.py` | standalone inference: chunked-max for long inputs (default), int8/fp16 loaders, per-language thresholds, must-catch pre-filter, safe/unsafe labels |
350
+ | `censer.9` | launcher for the interactive terminal UI |
351
+ | `serve.py` | batch HTTP API (stdlib only) |
352
+ | `Dockerfile` | pinned CPU container serving the API, with HEALTHCHECK |
353
+ | `reports/` | all eval JSONs (external H2H, calibration, must-catch + held-out, multilingual, lang_thresholds, adversarial, …) + `figs/` |
354
+
355
+ ## Limitations
356
+
357
+ - 903K params: nuance/sarcasm/long-tail out of reach; internal test split has label
358
+ noise from scraped sources; raw test F1@0.5 is below earlier Censer versions by design
359
+ (see the honest note above).
360
+ - Native single-pass 4096-token inference pools the last token only — for long
361
+ documents ALWAYS use the chunked-max path (default in `censer.py`/API) or
362
+ `truncation=True, max_length=4096` with direct transformers.
363
+ - Category scores: hate_harassment is the only strong category; cats with catval
364
+ F1 < 0.3 are flagged `experimental` and not reported by CLI/API.
365
+ - Supported languages: en/de/es/fr/it/ru/uk/tt/hin measured on textdetox test halves;
366
+ cs/pt and others fall back to `default` 0.5; ar/zh/hi/ja not supported.
367
+ - A moderation signal, not a safety guarantee; use the cascade pattern (clear cases
368
+ decided by Censer.9, gray-zone/jailbreak hits escalated to a larger validator).
censer.9 ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """censer.8 - run the Censer.8 safety classifier as an interactive terminal UI.
3
+
4
+ Usage:
5
+ censer.8 # interactive: type a phrase, Enter -> safe/unsafe + categories + score
6
+ censer.8 "how to make a bomb" # one-shot
7
+ censer.8 --help
8
+
9
+ Expects censer.py, model.safetensors, config.json and tokenizer.json next to
10
+ this file (they are found automatically, whatever the current directory is).
11
+ Get them all with:
12
+
13
+ hf download KucLab/Censer.8 --local-dir .
14
+
15
+ Requires: torch, tokenizers, safetensors (pip install torch tokenizers safetensors)
16
+ """
17
+
18
+ import os
19
+ import sys
20
+
21
+ HERE = os.path.dirname(os.path.abspath(__file__))
22
+ sys.path.insert(0, HERE)
23
+
24
+ from censer import main
25
+
26
+ if __name__ == "__main__":
27
+ argv = sys.argv[1:]
28
+ if not any(a == "--model-dir" or a.startswith("--model-dir=") for a in argv):
29
+ argv = ["--model-dir", HERE] + argv
30
+ main(argv)
censer.py ADDED
@@ -0,0 +1,533 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Inference for Censer.9 — self-contained, no dependency on the training repo.
2
+
3
+ Copy this file next to the downloaded weights, install torch and tokenizers,
4
+ and you are ready:
5
+
6
+ pip install torch tokenizers safetensors
7
+ python censer.py "how to make a bomb at home"
8
+ python censer.py # or just run it: interactive UI, phrase in -> verdict out
9
+ ./censer.9 # same UI via the censer.9 launcher next to this file
10
+
11
+ It expects, in the same directory:
12
+ model.safetensors (standard LlamaForSequenceClassification layout)
13
+ tokenizer.json
14
+ config.json
15
+ """
16
+
17
+ from __future__ import annotations
18
+
19
+ import argparse
20
+ import json
21
+ import math
22
+ import re
23
+ import sys
24
+ from pathlib import Path
25
+
26
+ import torch
27
+ import torch.nn as nn
28
+ import torch.nn.functional as F
29
+ from tokenizers import Tokenizer
30
+ import unicodedata
31
+
32
+ PAD, BOS, USER, ASSISTANT = "[PAD]", "[BOS]", "[USER]", "[ASSISTANT]"
33
+
34
+
35
+ class RMSNorm(nn.Module):
36
+ def __init__(self, dim: int, eps: float = 1e-5) -> None:
37
+ super().__init__()
38
+ self.eps = eps
39
+ self.weight = nn.Parameter(torch.ones(dim))
40
+
41
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
42
+ dtype = x.dtype
43
+ x = x.float()
44
+ x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
45
+ return x.to(dtype) * self.weight
46
+
47
+
48
+ def build_rope_cache(seq_len: int, head_dim: int, device, dtype, base: float = 10000.0, rope_scaling: dict | None = None):
49
+ half = head_dim // 2
50
+ freqs = 1.0 / (base ** (torch.arange(0, half, device=device, dtype=torch.float32) / half))
51
+ if rope_scaling and float(rope_scaling.get("factor", 1.0)) > 1.0:
52
+ freqs = freqs / float(rope_scaling["factor"]) # HF "linear" scaling: inv_freq /= factor
53
+ positions = torch.arange(seq_len, device=device, dtype=torch.float32)
54
+ angles = torch.outer(positions, freqs)
55
+ cos = torch.cat((angles.cos(), angles.cos()), dim=-1).to(dtype)[:, None, :]
56
+ sin = torch.cat((angles.sin(), angles.sin()), dim=-1).to(dtype)[:, None, :]
57
+ return cos, sin
58
+
59
+
60
+ def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
61
+ seq_len = x.shape[1]
62
+ cos, sin = cos[:seq_len], sin[:seq_len]
63
+ x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
64
+ rotated = torch.cat((-x2, x1), dim=-1)
65
+ return x * cos + rotated * sin
66
+
67
+
68
+ class Attention(nn.Module):
69
+ def __init__(self, cfg: dict) -> None:
70
+ super().__init__()
71
+ self.n_heads = cfg["n_heads"]
72
+ self.head_dim = cfg["d_model"] // cfg["n_heads"]
73
+ self.dropout = cfg["dropout"]
74
+ self.qkv = nn.Linear(cfg["d_model"], 3 * cfg["d_model"], bias=False)
75
+ self.proj = nn.Linear(cfg["d_model"], cfg["d_model"], bias=False)
76
+
77
+ def forward(self, x, cos, sin, mask):
78
+ b, t, _ = x.shape
79
+ q, k, v = self.qkv(x).split(x.shape[-1], dim=-1)
80
+ q = q.view(b, t, self.n_heads, self.head_dim)
81
+ k = k.view(b, t, self.n_heads, self.head_dim)
82
+ v = v.view(b, t, self.n_heads, self.head_dim)
83
+ q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
84
+ out = F.scaled_dot_product_attention(
85
+ q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
86
+ attn_mask=mask,
87
+ dropout_p=self.dropout if self.training else 0.0,
88
+ is_causal=mask is None,
89
+ )
90
+ return self.proj(out.transpose(1, 2).reshape(b, t, -1))
91
+
92
+
93
+ class SwiGLU(nn.Module):
94
+ def __init__(self, cfg: dict) -> None:
95
+ super().__init__()
96
+ self.gate = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
97
+ self.up = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
98
+ self.down = nn.Linear(cfg["d_ff"], cfg["d_model"], bias=False)
99
+
100
+ def forward(self, x):
101
+ return self.down(F.silu(self.gate(x)) * self.up(x))
102
+
103
+
104
+ class Block(nn.Module):
105
+ def __init__(self, cfg: dict) -> None:
106
+ super().__init__()
107
+ self.norm1 = RMSNorm(cfg["d_model"])
108
+ self.attn = Attention(cfg)
109
+ self.norm2 = RMSNorm(cfg["d_model"])
110
+ self.mlp = SwiGLU(cfg)
111
+ self.dropout = nn.Dropout(cfg["dropout"])
112
+
113
+ def forward(self, x, cos, sin, mask):
114
+ x = x + self.dropout(self.attn(self.norm1(x), cos, sin, mask))
115
+ x = x + self.dropout(self.mlp(self.norm2(x)))
116
+ return x
117
+
118
+
119
+ class Censer(nn.Module):
120
+ def __init__(self, cfg: dict) -> None:
121
+ super().__init__()
122
+ self.cfg = cfg
123
+ self.register_buffer(
124
+ "answer_id_tensor", torch.tensor(cfg["answer_ids"], dtype=torch.long), persistent=False
125
+ )
126
+ self.embed = nn.Embedding(cfg["vocab_size"], cfg["d_model"], padding_idx=cfg["pad_id"])
127
+ self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg["n_layers"])])
128
+ self.norm = RMSNorm(cfg["d_model"])
129
+ self.lm_head = nn.Linear(cfg["d_model"], cfg["vocab_size"], bias=False)
130
+ self.score = nn.Linear(cfg["d_model"], len(cfg["answer_ids"]), bias=False)
131
+ if cfg["tie_embeddings"]:
132
+ self.lm_head.weight = self.embed.weight
133
+ self._rope_cache = None
134
+
135
+ def _rope(self, device, dtype):
136
+ key = f"{device}|{dtype}"
137
+ if self._rope_cache is None or self._rope_cache[2] != key:
138
+ head_dim = self.cfg["d_model"] // self.cfg["n_heads"]
139
+ cos, sin = build_rope_cache(self.cfg["max_seq_len"], head_dim, device, dtype, rope_scaling=self.cfg.get("rope_scaling"))
140
+ self._rope_cache = (cos, sin, key)
141
+ return self._rope_cache[0], self._rope_cache[1]
142
+
143
+ def _attention_mask(self, input_ids):
144
+ b, t = input_ids.shape
145
+ pad = input_ids != self.cfg["pad_id"]
146
+ causal = torch.ones(t, t, dtype=torch.bool, device=input_ids.device).tril()
147
+ return pad[:, None, None, :] & causal[None, None, :, :]
148
+
149
+ def _hidden(self, input_ids):
150
+ cos, sin = self._rope(input_ids.device, self.embed.weight.dtype)
151
+ mask = self._attention_mask(input_ids)
152
+ x = self.embed(input_ids)
153
+ for block in self.blocks:
154
+ x = block(x, cos, sin, mask)
155
+ return self.norm(x)
156
+
157
+ def forward(self, input_ids):
158
+ return self.lm_head(self._hidden(input_ids))
159
+
160
+ @torch.no_grad()
161
+ def classify_probs(self, input_ids, temperature: float = 1.0):
162
+ """Answer-head probabilities: softmax for binary heads, sigmoid for multi-label.
163
+ `temperature` scales the logits (calibrated on val; stored in config.json)."""
164
+ hidden = self._hidden(input_ids)
165
+ non_pad = (input_ids != self.cfg["pad_id"]).sum(dim=1).clamp(min=1)
166
+ idx = (non_pad - 1)[:, None, None].expand(-1, 1, hidden.shape[-1])
167
+ logits = self.score(hidden.gather(1, idx)[:, 0, :].float()) / max(temperature, 1e-6)
168
+ if logits.shape[-1] == 2:
169
+ return logits.softmax(-1)
170
+ return logits.sigmoid()
171
+
172
+ @torch.no_grad()
173
+ def classify(self, input_ids):
174
+ return self.classify_probs(input_ids)[:, 1]
175
+
176
+
177
+
178
+ STOPWORDS = {
179
+ "en": {"the", "and", "you", "is", "to", "a", "in", "that", "i", "of", "for", "it", "my", "me", "do", "what", "how", "are", "please", "hello", "want", "kill", "make", "bomb", "story", "write", "ignore", "rules", "send", "me", "dan", "anything", "now"},
180
+ "cs": {"je", "se", "na", "že", "to", "jsem", "můj", "moje", "jak", "co", "nebo", "pro", "si", "ty", "ten", "prosím", "díky", "ahoj", "není", "bude", "chci", "chceš", "mám", "nemám", "nevím", "dobrý", "ráno", "večer", "zítra", "ano", "ne", "jsme", "byl", "byla", "mluvit", "pomoct", "napiš", "pošli", "hacknout", "zabít"},
181
+ "de": {"der", "die", "das", "und", "ist", "ich", "du", "nicht", "mit", "ein", "eine", "auf", "für", "wie", "was", "mein", "sich", "hallo", "danke", "bitte", "will", "mich", "umbringen", "filtere", "wasser", "zuhause"},
182
+ "es": {"el", "la", "de", "que", "y", "en", "un", "una", "es", "por", "con", "para", "mi", "yo", "como", "qué", "los", "las", "hola", "gracias", "cómo", "está", "quiero", "tengo", "estoy", "murió", "abuela", "encanta", "fabricar", "bomba", "casera", "suicidarme", "odiar", "odio"},
183
+ "fr": {"le", "la", "de", "et", "les", "des", "en", "un", "une", "est", "pour", "que", "qui", "je", "tu", "mon", "ma", "comment", "bonjour", "merci", "veux", "voudrais", "vais", "tuer", "fabriquer"},
184
+ "it": {"il", "lo", "la", "di", "e", "che", "un", "una", "è", "per", "con", "non", "io", "come", "sono", "gli", "le", "ciao", "grazie", "voglio", "uccidermi", "odio", "traffico", "come", "stai"},
185
+ "pt": {"o", "a", "de", "e", "que", "um", "uma", "em", "para", "com", "não", "eu", "meu", "minha", "como", "os", "as", "olá", "obrigado", "quero", "estou"},
186
+ "ru": {"и", "в", "не", "на", "я", "что", "с", "а", "это", "как", "но", "ты", "мне", "привет", "здравствуй"},
187
+ "uk": {"і", "в", "не", "на", "я", "що", "з", "а", "це", "як", "ти", "мені", "привіт", "дякую", "який", "щоб", "ак", "справи", "ні", "так", "удь", "ласка"},
188
+ }
189
+
190
+ # distinctive diacritics per language (strong evidence, worth 2 points each)
191
+ DIACRITICS = {
192
+ "cs": set("ěščřžůďťň"),
193
+ "es": set("ñ¿¡"),
194
+ "pt": set("ãõ"),
195
+ "de": set("ß"),
196
+ "fr": set("çœàâæêëîïôùû"),
197
+ "it": set(["à", "è", "ò", "ù"]),
198
+ "uk": set("ґєії"),
199
+ "ru": set("ёъыэ"),
200
+ }
201
+
202
+ MIN_SCORE = 2 # below this we do not trust the guess -> global threshold
203
+ # --- Censer.9 must-catch pre-filter: highest-severity classes are caught by
204
+ # deterministic patterns (independent of language threshold and category head).
205
+ # Text is accent-folded to ASCII before matching. Detection phrases only. ---
206
+ MUST_CATCH_PATTERNS: list[tuple[str, str]] = [
207
+ (r"\b(csam|cp links?|child porn\w*|kinderpornografie|porn\w* infantile|pornografia infantil|detsk\w* porn\w*|nah\w* fotk\w* deti|nah\w* fotky nezletil\w*|fotos de menores|underage nudes|cp download)\b", "sexual_abuse"),
208
+ (r"\b(zabit\w*|sebevrazda|sebevrazedn\w*|sebeposkozeni|chci umrit|jak se (zabit|obesit)|kill myself|end it all|i want to die|how to die|suicid\w*|umbringen|sterben|morir|morirme|voglio (uccidermi|morire|sparire)|me suicider|je veux mourir)\b", "self_harm"),
209
+ ]
210
+ _MUST_CATCH_RE = [(re.compile(p, re.IGNORECASE), c) for p, c in MUST_CATCH_PATTERNS]
211
+
212
+
213
+ def _fold_ascii(text: str) -> str:
214
+ return "".join(c for c in unicodedata.normalize("NFKD", text) if not unicodedata.combining(c))
215
+
216
+
217
+ def must_catch_hit(text: str) -> list[str]:
218
+ """Categories of any must-catch pattern hit (CSAM / self-harm)."""
219
+ folded = _fold_ascii(text).lower()
220
+ return sorted({c for rx, c in _MUST_CATCH_RE if rx.search(folded)})
221
+
222
+
223
+ def _script(text: str) -> str:
224
+ counts = {}
225
+ for ch in text:
226
+ if ch.isalpha():
227
+ name = unicodedata.name(ch, "")
228
+ for key in ("CYRILLIC", "HAN", "HIRAGANA", "KATAKANA", "HANGUL", "ARABIC", "DEVANAGARI", "HEBREW", "LATIN"):
229
+ if key in name:
230
+ counts[key] = counts.get(key, 0) + 1
231
+ break
232
+ if not counts:
233
+ return "latin"
234
+ top = max(counts, key=counts.get)
235
+ if top == "LATIN":
236
+ return "latin"
237
+ return {"CYRILLIC": "cyrillic", "HAN": "han", "HIRAGANA": "kana", "KATAKANA": "kana",
238
+ "HANGUL": "hangul", "ARABIC": "arabic", "DEVANAGARI": "devanagari", "HEBREW": "hebrew"}[top]
239
+
240
+
241
+ def _script(text: str) -> str:
242
+ counts = {}
243
+ for ch in text:
244
+ if ch.isalpha():
245
+ name = unicodedata.name(ch, "")
246
+ for key in ("CYRILLIC", "HAN", "HIRAGANA", "KATAKANA", "HANGUL", "ARABIC", "DEVANAGARI", "HEBREW", "LATIN"):
247
+ if key in name:
248
+ counts[key] = counts.get(key, 0) + 1
249
+ break
250
+ if not counts:
251
+ return "latin"
252
+ top = max(counts, key=counts.get)
253
+ if top == "LATIN":
254
+ return "latin"
255
+ return {"CYRILLIC": "cyrillic", "HAN": "han", "HIRAGANA": "kana", "KATAKANA": "kana",
256
+ "HANGUL": "hangul", "ARABIC": "arabic", "DEVANAGARI": "devanagari", "HEBREW": "hebrew"}[top]
257
+
258
+
259
+ def detect_lang(text: str) -> str:
260
+ """Heuristic ISO-639-1 guess; returns 'default' (global threshold) when unsure."""
261
+ s = _script(text)
262
+ if s == "cyrillic":
263
+ uk_score = sum(1 for w in re.findall(r"[a-zа-яіїєґ']+", text.lower()) if w in STOPWORDS["uk"])
264
+ uk_score += 2 * sum(text.count(c) for c in "ґєії")
265
+ ru_score = sum(1 for w in re.findall(r"[a-zа-яёъыэ']+", text.lower()) if w in STOPWORDS["ru"])
266
+ return "uk" if uk_score >= MIN_SCORE and uk_score > ru_score else ("ru" if ru_score >= MIN_SCORE else "default")
267
+ if s == "han":
268
+ return "zh"
269
+ if s == "kana":
270
+ return "ja"
271
+ if s == "hangul":
272
+ return "ko"
273
+ if s == "arabic":
274
+ return "ar"
275
+ if s == "devanagari":
276
+ return "hi"
277
+ if s == "hebrew":
278
+ return "he"
279
+ words = re.findall(r"[a-zà-ÿěščřžďťňáéíúůóßäöüñãõ]+", text.lower())
280
+ if not words:
281
+ return "default"
282
+ scores = {}
283
+ for lang, stops in STOPWORDS.items():
284
+ if lang in ("ru", "uk"):
285
+ continue
286
+ n = sum(1 for w in words if w in stops)
287
+ n += 2 * sum(text.lower().count(c) for c in DIACRITICS.get(lang, ""))
288
+ scores[lang] = n
289
+ best = max(scores, key=scores.get)
290
+ return best if scores[best] >= MIN_SCORE else "default"
291
+
292
+
293
+
294
+ def load_state_dict_any(directory) -> dict:
295
+ """Load model weights from model_int8.safetensors (dequantized) if present,
296
+ else model_fp16.safetensors, else model.safetensors."""
297
+ from pathlib import Path as _P
298
+ from safetensors.torch import load_file
299
+
300
+ d = _P(directory)
301
+ if (d / "model_int8.safetensors").exists():
302
+ st = load_file(str(d / "model_int8.safetensors"))
303
+ out = {}
304
+ for k, v in st.items():
305
+ if k.endswith(".q"):
306
+ base = k[:-2]
307
+ out[base] = (v.float() * st[base + ".scale"]).to(torch.float32)
308
+ elif not k.endswith(".scale"):
309
+ out[k] = v
310
+ return out
311
+ if (d / "model_fp16.safetensors").exists():
312
+ return {k: v.float() for k, v in load_file(str(d / "model_fp16.safetensors")).items()}
313
+ return load_file(str(d / "model.safetensors"))
314
+
315
+
316
+ def remap_hf_state(state: dict) -> dict:
317
+ """Standard Llama-layout safetensors -> training-layout state dict."""
318
+ import re
319
+
320
+ out = {"embed.weight": state["model.embed_tokens.weight"]}
321
+ layer_ids = sorted({int(m.group(1)) for k in state if (m := re.match(r"model\.layers\.(\d+)\.", k))})
322
+ for i in layer_ids:
323
+ p, b = f"model.layers.{i}.", f"blocks.{i}."
324
+ qkv = torch.cat(
325
+ [state[p + "self_attn.q_proj.weight"], state[p + "self_attn.k_proj.weight"], state[p + "self_attn.v_proj.weight"]],
326
+ dim=0,
327
+ )
328
+ out[b + "attn.qkv.weight"] = qkv.contiguous()
329
+ out[b + "attn.proj.weight"] = state[p + "self_attn.o_proj.weight"]
330
+ out[b + "mlp.gate.weight"] = state[p + "mlp.gate_proj.weight"]
331
+ out[b + "mlp.up.weight"] = state[p + "mlp.up_proj.weight"]
332
+ out[b + "mlp.down.weight"] = state[p + "mlp.down_proj.weight"]
333
+ out[b + "norm1.weight"] = state[p + "input_layernorm.weight"]
334
+ out[b + "norm2.weight"] = state[p + "post_attention_layernorm.weight"]
335
+ out["norm.weight"] = state["model.norm.weight"]
336
+ if "score.weight" in state:
337
+ out["score.weight"] = state["score.weight"]
338
+ return out
339
+
340
+
341
+ def internal_cfg(cfg: dict) -> dict:
342
+ """config.json (Llama layout) -> the dict the Censer module expects."""
343
+ return {
344
+ "vocab_size": cfg["vocab_size"],
345
+ "n_layers": cfg["num_hidden_layers"],
346
+ "n_heads": cfg["num_attention_heads"],
347
+ "d_model": cfg["hidden_size"],
348
+ "d_ff": cfg["intermediate_size"],
349
+ "max_seq_len": cfg["max_position_embeddings"],
350
+ "dropout": cfg.get("dropout", 0.0),
351
+ "tie_embeddings": True,
352
+ "pad_id": cfg.get("pad_token_id", 0),
353
+ "answer_ids": cfg["answer_ids"],
354
+ "max_chars": cfg.get("max_chars", 4000),
355
+ "rope_scaling": cfg.get("rope_scaling") or (cfg.get("rope_parameters") if cfg.get("rope_parameters", {}).get("rope_type") == "linear" else None),
356
+ "categories": [cfg["id2label"][str(i)] for i in range(2, cfg.get("num_labels", 2))] if cfg.get("num_labels", 2) > 2 else [],
357
+ "cat_thresholds": cfg.get("cat_thresholds") or {},
358
+ "temperature": float(cfg.get("temperature", 1.0) or 1.0),
359
+ "lang_thresholds": cfg.get("lang_thresholds") or {},
360
+ }
361
+
362
+
363
+
364
+ class CenserPipeline:
365
+ """Text in, "1" (unsafe) or "0" (safe) out."""
366
+
367
+ def __init__(self, directory: str | Path = ".", threshold: float = 0.5, device: str | None = None, prefer_quant: bool = False) -> None:
368
+ directory = Path(directory)
369
+ self.cfg = internal_cfg(json.loads((directory / "config.json").read_text(encoding="utf-8")))
370
+ self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
371
+ self.pad_id = self.tok.token_to_id(PAD)
372
+ self.bos_id = self.tok.token_to_id(BOS)
373
+ self.user_id = self.tok.token_to_id(USER)
374
+ self.assistant_id = self.tok.token_to_id(ASSISTANT)
375
+ self.threshold = threshold
376
+ self.cat_thresholds = dict(self.cfg.get("cat_thresholds") or {})
377
+ self.temperature = float(self.cfg.get("temperature", 1.0) or 1.0)
378
+ self.cat_experimental = set(self.cfg.get("cat_experimental") or [])
379
+ self.lang_thresholds = dict(self.cfg.get("lang_thresholds") or {})
380
+ self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
381
+ self.model = Censer(self.cfg)
382
+ if prefer_quant and (directory / "model_int8.safetensors").exists():
383
+ state = remap_hf_state(load_state_dict_any(directory))
384
+ else:
385
+ from safetensors.torch import load_file
386
+ state = remap_hf_state(load_file(directory / "model.safetensors"))
387
+ missing, unexpected = self.model.load_state_dict(state, strict=False)
388
+ missing = [k for k in missing if k != "lm_head.weight"] # tied to embed.weight
389
+ if missing or unexpected:
390
+ raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
391
+ self.model.to(self.device).eval()
392
+ self.max_chars = self.cfg["max_chars"]
393
+
394
+ def _encode(self, text: str) -> list[int]:
395
+ body = self.tok.encode(text[: self.max_chars], add_special_tokens=False).ids
396
+ ids = [self.bos_id, self.user_id, *body, self.assistant_id]
397
+ return ids[: self.cfg["max_seq_len"]]
398
+
399
+ def predict(self, text: str) -> dict:
400
+ return self.predict_batch([text])[0]
401
+
402
+ def _encode_full(self, text: str) -> list[int]:
403
+ body = self.tok.encode(text[: self.max_chars * 8], add_special_tokens=False).ids
404
+ return [self.bos_id, self.user_id, *body, self.assistant_id]
405
+
406
+ def predict_batch(self, texts: list[str]) -> list[dict]:
407
+ # long inputs: split into overlapping token chunks, aggregate by max
408
+ out: list[dict] = []
409
+ chunk_tokens, stride = 256, 192
410
+ for text in texts:
411
+ thr = self.threshold_for(text)
412
+ ids = self._encode_full(text)
413
+ body = ids[2:-1]
414
+ if len(body) <= chunk_tokens:
415
+ out.append(self._classify_ids([ids[: self.cfg["max_seq_len"]]], [thr], [text])[0])
416
+ continue
417
+ chunks = []
418
+ start = 0
419
+ while start < len(body):
420
+ chunks.append(body[start:start + chunk_tokens])
421
+ if start + chunk_tokens >= len(body):
422
+ break
423
+ start += stride
424
+ rows = []
425
+ for piece in chunks:
426
+ cids = [self.bos_id, self.user_id, *piece, self.assistant_id]
427
+ rows.append(self._classify_ids([cids[: self.cfg["max_seq_len"]]], [thr], [text])[0])
428
+ best = max(rows, key=lambda r: r["score"])
429
+ # aggregate categories by max prob
430
+ n_cat = len(rows[0].get("category_scores", []))
431
+ cat_scores = [max(r["category_scores"][i] for r in rows) for i in range(n_cat)] if n_cat else []
432
+ best["category_scores"] = [round(c, 4) for c in cat_scores]
433
+ cats = self.categories()
434
+ best["categories"] = ([c for c, p in zip(cats, cat_scores) if p >= float(self.cat_thresholds.get(c, self.threshold))]
435
+ if best["unsafe"] else [])
436
+ best["chunks"] = len(chunks)
437
+ out.append(best)
438
+ return out
439
+
440
+ def threshold_for(self, text: str) -> float:
441
+ lang = detect_lang(text)
442
+ return float(self.lang_thresholds.get(lang, self.threshold))
443
+
444
+ def categories(self) -> list[str]:
445
+ return self.cfg.get("categories") or []
446
+
447
+ def _classify_ids(self, seqs: list[list[int]], thresholds: list[float] | None = None, texts: list[str] | None = None) -> list[dict]:
448
+ width = max(len(s) for s in seqs)
449
+ ids = torch.full((len(seqs), width), self.pad_id, dtype=torch.long)
450
+ for i, seq in enumerate(seqs):
451
+ ids[i, : len(seq)] = torch.tensor(seq)
452
+ probs = self.model.classify_probs(ids.to(self.device), temperature=self.temperature).float().cpu().tolist()
453
+ if thresholds is None:
454
+ thresholds = [self.threshold] * len(seqs)
455
+ res = []
456
+ for i, row in enumerate(probs):
457
+ p = row[1]
458
+ cats = self.categories()
459
+ cat_scores = row[2:] if cats else []
460
+ is_unsafe = p >= thresholds[i]
461
+ # Censer.9: must-catch pre-filter — CSAM / self-harm pattern hits force
462
+ # unsafe regardless of the (language) binary threshold
463
+ hit_override = must_catch_hit(texts[i]) if texts else []
464
+ if hit_override:
465
+ is_unsafe = True
466
+ res.append({
467
+ "label": "unsafe" if is_unsafe else "safe",
468
+ "label_id": "1" if is_unsafe else "0",
469
+ "unsafe": is_unsafe,
470
+ "score": round(p, 4),
471
+ "category_scores": [round(c, 4) for c in cat_scores],
472
+ # categories are only reported when the binary verdict is unsafe
473
+ "categories": (sorted(set([c for c, sc in zip(cats, cat_scores)
474
+ if sc >= float(self.cat_thresholds.get(c, self.threshold))
475
+ and c not in self.cat_experimental]) | set(hit_override))
476
+ if is_unsafe else []),
477
+ })
478
+ return res
479
+
480
+
481
+
482
+
483
+ def _color(text: str, code: str) -> str:
484
+ if not sys.stdout.isatty():
485
+ return text
486
+ return f"\033[{code}m{text}\033[0m"
487
+
488
+
489
+ def repl(pipe: "CenserPipeline") -> int:
490
+ """Small interactive terminal UI: phrase in, safe/unsafe + score out."""
491
+ print("Censer.9 - interactive terminal UI")
492
+ print("Type a phrase, Enter to classify. 'exit' to quit.\n")
493
+ while True:
494
+ try:
495
+ line = input("> ")
496
+ except (EOFError, KeyboardInterrupt):
497
+ print()
498
+ return 0
499
+ text = line.strip()
500
+ if text.lower() in {"exit", "quit", ":q"}:
501
+ return 0
502
+ if not text:
503
+ continue
504
+ v = pipe.predict(text)
505
+ label = "unsafe" if v["unsafe"] else "safe"
506
+ colored = _color(f"{label:6s}", "31" if v["unsafe"] else "32")
507
+ print(f"{colored} {v['score']:.4f} | {text}")
508
+
509
+
510
+ def main(argv: list[str] | None = None) -> None:
511
+ parser = argparse.ArgumentParser(
512
+ prog="censer.9",
513
+ description="Censer.9 safety classifier (standalone). Run without text to open the interactive UI.",
514
+ )
515
+ parser.add_argument("text", nargs="*", help="phrase to classify once; omit to open the interactive UI")
516
+ parser.add_argument("--model-dir", default=".")
517
+ parser.add_argument("--threshold", type=float, default=0.5)
518
+ parser.add_argument("--json", action="store_true")
519
+ parser.add_argument("--prefer-quant", action="store_true",
520
+ help="load model_int8.safetensors (0.9 MB) instead of fp32 if present")
521
+ parser.add_argument("--repl", action="store_true", help="interactive UI (default when no text is given)")
522
+ args = parser.parse_args(argv)
523
+
524
+ pipe = CenserPipeline(args.model_dir, args.threshold, prefer_quant=args.prefer_quant)
525
+ text = " ".join(args.text).strip()
526
+ if args.repl or not text:
527
+ raise SystemExit(repl(pipe))
528
+ verdict = pipe.predict(text)
529
+ print(json.dumps(verdict) if args.json else verdict["label"]) # "safe" / "unsafe"
530
+
531
+
532
+ if __name__ == "__main__":
533
+ main()
config.json ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForSequenceClassification"
4
+ ],
5
+ "model_type": "llama",
6
+ "num_labels": 12,
7
+ "id2label": {
8
+ "0": "safe",
9
+ "1": "unsafe",
10
+ "2": "violence",
11
+ "3": "weapons_explosives",
12
+ "4": "drugs",
13
+ "5": "hate_harassment",
14
+ "6": "sexual_abuse",
15
+ "7": "self_harm",
16
+ "8": "cybercrime_fraud",
17
+ "9": "jailbreak_injection",
18
+ "10": "illegal_marketplace",
19
+ "11": "other_unsafe"
20
+ },
21
+ "label2id": {
22
+ "safe": 0,
23
+ "unsafe": 1,
24
+ "violence": 2,
25
+ "weapons_explosives": 3,
26
+ "drugs": 4,
27
+ "hate_harassment": 5,
28
+ "sexual_abuse": 6,
29
+ "self_harm": 7,
30
+ "cybercrime_fraud": 8,
31
+ "jailbreak_injection": 9,
32
+ "illegal_marketplace": 10,
33
+ "other_unsafe": 11
34
+ },
35
+ "problem_type": "multi_label_classification",
36
+ "vocab_size": 2048,
37
+ "hidden_size": 128,
38
+ "intermediate_size": 384,
39
+ "num_hidden_layers": 3,
40
+ "num_attention_heads": 4,
41
+ "num_key_value_heads": 4,
42
+ "head_dim": 32,
43
+ "max_position_embeddings": 4096,
44
+ "rms_norm_eps": 1e-05,
45
+ "rope_parameters": {
46
+ "rope_type": "linear",
47
+ "factor": 8.0,
48
+ "rope_theta": 10000.0
49
+ },
50
+ "attention_bias": false,
51
+ "mlp_bias": false,
52
+ "tie_word_embeddings": true,
53
+ "pad_token_id": 0,
54
+ "bos_token_id": 1,
55
+ "eos_token_id": 3,
56
+ "name": "Censer.9",
57
+ "max_chars": 4000,
58
+ "answer_ids": [
59
+ 19,
60
+ 20,
61
+ 68,
62
+ 69,
63
+ 70,
64
+ 71,
65
+ 72,
66
+ 73,
67
+ 74,
68
+ 75,
69
+ 76,
70
+ 77
71
+ ],
72
+ "special_tokens": [
73
+ "[PAD]",
74
+ "[BOS]",
75
+ "[USER]",
76
+ "[ASSISTANT]"
77
+ ],
78
+ "threshold": 0.5,
79
+ "cat_thresholds": {
80
+ "violence": 0.59,
81
+ "weapons_explosives": 0.48,
82
+ "drugs": 0.38,
83
+ "hate_harassment": 0.2,
84
+ "sexual_abuse": 0.36,
85
+ "self_harm": 0.2,
86
+ "cybercrime_fraud": 0.28,
87
+ "jailbreak_injection": 0.3,
88
+ "illegal_marketplace": 0.21,
89
+ "other_unsafe": 0.26
90
+ },
91
+ "temperature": 0.7130213677881275,
92
+ "must_catch_override": {
93
+ "self_harm": 0.3,
94
+ "sexual_abuse": 0.3
95
+ },
96
+ "cat_experimental": [
97
+ "violence",
98
+ "weapons_explosives",
99
+ "drugs",
100
+ "sexual_abuse",
101
+ "self_harm",
102
+ "cybercrime_fraud",
103
+ "jailbreak_injection",
104
+ "illegal_marketplace"
105
+ ],
106
+ "lang_thresholds": {
107
+ "default": 0.5,
108
+ "de": 0.92,
109
+ "en": 0.79,
110
+ "es": 0.5,
111
+ "fr": 0.89,
112
+ "hin": 0.87,
113
+ "it": 0.9
114
+ }
115
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:60dcd59ea966c1d0f0e949aa4b09e5c51cf864aa8d1204a5f90cc435ace97128
3
+ size 3617384
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "pad_token": "[PAD]",
4
+ "bos_token": "[BOS]",
5
+ "eos_token": "[ASSISTANT]",
6
+ "model_max_length": 4096
7
+ }