Censer.7 / README.md
KucLab's picture
Censer.7: two-phase trained multi-label classifier, calibrated categories, 4k ctx + chunked-max, multilingual; standard LlamaForSequenceClassification
103988e verified
|
Raw History Blame Contribute Delete
13.3 kB
metadata
library_name: transformers
pipeline_tag: text-classification
tags:
  - text-classification
  - safety
  - content-moderation
  - multi-label
  - multilingual
  - calibrated
  - tiny-language-model
  - from-scratch
  - distilled
language:
  - en
  - cs
  - de
  - es
  - fr
  - it
  - pt
  - ru
  - ar
  - zh
  - hi
license: apache-2.0
datasets:
  - nvidia/Aegis-AI-Content-Safety-Dataset-1.0
  - thesofakillers/jigsaw-toxic-comment-classification-challenge
  - google/civil_comments
  - PKU-Alignment/BeaverTails
  - lmsys/toxic-chat
  - databricks/databricks-dolly-15k
  - Hate-speech-CNERG/hatexplain
  - Intuit-GenSRF/jigsaw-multilingual-train-unique
metrics:
  - f1
  - accuracy
  - ece

Censer.7

A 902K parameter decoder-only transformer trained from scratch: binary safe / unsafe (F1 restored to .5 level) + 10 calibrated safety categories, 4096-token native context with chunked-max inference for long documents, and multilingual coverage (trained on en + cs/de/es/fr synthetic + Jigsaw-multilingual fr/it/pt/es/ru/tr) β€” at the same tiny size and speed.

Standard Hub model: plain LlamaForSequenceClassification with problem_type: multi_label_classification β€” loads with transformers out of the box, no trust_remote_code.

from transformers import pipeline
clf = pipeline("text-classification", model="KucLab/Censer.7")
clf("how to make a bomb at home", top_k=-1)
# [{'label': 'unsafe', ...}, {'label': 'weapons_explosives', ...}, ...]

What is new in .7 (vs .6)

Fix .6 .7
Binary test F1 0.829 (regressed from .5's 0.849) 0.8348 (two-phase training: binary first, then categories at low lr with cat_weight)
Probes 90/97 95/97 (diff vs .5: fixed 'Wait, Rockstar really give us fake hope', regressed 'buy a gun')
External benchmarks ToxiGen 0.546, HateXplain 0.714, no head-to-head ToxiGen 0.561 / HateXplain 0.738 + direct head-to-head vs unbiased-toxic-roberta on the same sets (table below); HateXplain-train + Jigsaw-multilingual added to training (test splits stay pure)
Category scores uncalibrated ("fire liberally") per-category thresholds tuned on held-out catval + ECE + reliability diagram (see below); thresholds ship in config.json as cat_thresholds
Multilingual cs/de/es/fr only, no eval per-language F1 on textdetox multilingual toxicity (12 languages) + held-out synthetic (table below)
Long documents native 4096 pass silently broken (F1 0.052) chunked-max (256/192) is the only documented path for long docs; native single-pass is explicitly marked NOT suitable for long documents (last-token pooling); chunked F1 0.998 (acc 0.997)
Training data 176k rows 251,655 rows (+HateXplain train 15k, +Jigsaw-ML 60k); val/test byte-identical to .1–.6

How to run

The censer.7 command β€” interactive terminal UI

pip install torch tokenizers safetensors huggingface_hub
hf download KucLab/Censer.7 --local-dir ~/.local/share/censer7
chmod +x ~/.local/share/censer7/censer.7
ln -s ~/.local/share/censer7/censer.7 ~/.local/bin/censer.7
censer.7                          # interactive UI: phrase in -> safe/unsafe + categories + score
censer.7 "how to make a bomb at home"

Or via the GitHub package (loads the Hub model with plain transformers):

pip install "git+https://github.com/Jerry256254/censer.7.git"
censer.7

One-liner with transformers

pip install torch transformers
python - <<'PY'
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("KucLab/Censer.7")
model = AutoModelForSequenceClassification.from_pretrained("KucLab/Censer.7").eval()
enc = tok("how to make a bomb at home", return_tensors="pt")
probs = model(**enc).logits.sigmoid()[0]
id2label = model.config.id2label
cat_thr = model.config.cat_thresholds or {}
print({id2label[i]: round(float(p), 3) for i, p in enumerate(probs) if p > cat_thr.get(id2label[i], 0.5)})
PY
  • Binary unsafe score = index 1; threshold 0.5. Category labels use per-category calibrated thresholds (config.json: cat_thresholds).
  • Runs on CPU out of the box; GPU via device=0.

Long inputs β€” always use the chunked path

Native context is 4096 tokens, but a single pass pools only the last token, so in long documents unsafe content in the middle can be missed (measured F1 0.052 on 25-row synthetic docs). Therefore censer.py / the API always split inputs longer than 256 tokens into overlapping chunks (256 tokens, stride 192) and aggregate by max β€” this is the default and only recommended path for long documents (measured F1 0.998 (acc 0.997) on the same docs):

from censer import CenserPipeline
pipe = CenserPipeline(".")
pipe.predict("very long text ...")   # {"label": "1", "unsafe": true, "score": ..., "categories": [...], "chunks": N}

Batch HTTP API + Docker

python serve.py --dir . --port 8000
curl -s localhost:8000/classify -d '{"texts": ["hello", "how to make a bomb"]}'
docker build -t censer7 . && docker run -p 8000:8000 censer7

Endpoints: GET /health, POST /classify (batch), POST /classify-one. Rate limit via RATE_LIMIT_PER_MIN (default 600).

Model details

Property Value
Architecture LlamaForSequenceClassification (standard transformers)
Parameters 902,016
Layers / d_model / heads / d_ff 3 / 128 / 4 / 384
Context 4096 tokens native (linear RoPE Γ—8); long docs via chunked-max (default)
Input cap 4000 chars
Tokenizer byte-level BPE, 2048 tokens (multilingual corpus)
Head score = embedding rows of the 12 answer tokens 0,1,a..j
Output sigmoid over 12 labels; per-category calibrated thresholds in config
Training two-phase: (A) 5 epochs binary (cat_weight=0, lr 2e-3) from Censer.6 init, (B) 3 epochs categories (cat_weight=0.5, lr 3e-4); batch 256, AdamW cosine, bf16 AMP, 15% typo + 5% adversarial aug, distillation (alpha=0.65, T=4, boost 0.75) from Censer.5Γ—2+.4+.3 ensemble on 251,655 rows
Checkpoint 3.6 MB fp32 Β· 1.8 MB fp16 Β· 0.9 MB int8 Β· ONNX

Prompt format

tokenizer(text) produces [BOS][USER] <text> [ASSISTANT] -> answer logits over "0" "1" "a".."j" (template baked into the post-processor).

Metrics

Internal held-out test split (6,496 rows, byte-identical across Censer.1–.7), binary threshold 0.5:

Metric Censer.5 Censer.6 Censer.7
Test F1 0.8490 0.8289 0.8348
Test accuracy 0.876 0.858 0.8655
Precision / recall 0.864 / 0.834 0.833 / 0.825 0.855 / 0.816
Val F1 0.8495 0.8358 0.8368
Probes (97 hand-written) 95/97 90/97 95/97

Per-source test F1: aegis 0.912, BeaverTails 0.881, jigsaw 0.856, civil_comments 0.776.

External benchmarks (never in training) β€” with head-to-head vs public models

Dataset Model F1@0.5 best F1 latency
ToxiGen annotated (n=940, 402 unsafe) Censer.7 0.561 0.619 @ 0.27 1.24 ms
ToxiGen unitary/unbiased-toxic-roberta 0.443 0.483 ~14 ms
HateXplain test (n=1,924, 1,376 unsafe) Censer.7 0.738 0.834 @ 0.05 1.24 ms
HateXplain test unitary/unbiased-toxic-roberta 0.577 0.690 ~14 ms

(full numbers in reports/external_h2h.json)

Adversarial robustness (detection rate on attacked unsafe texts, n=160)

Attack .6 .7
clean 91.3% 93.8%
typo 89.4% 88.8%
leetspeak 92.5% 93.1%
homoglyph 98.1% 98.1%
zero-width 95.0% 97.5%
case flip 92.5% 91.9%

Long-context (300 synthetic docs of 25 held-out rows, avg 2,178 tokens)

Inference F1
truncate at 192 tokens 0.571
native 4096 single pass (NOT recommended for long docs) 0.052
chunked 256/192 max (default everywhere) 0.998 (acc 0.997)

Category calibration (held-out HateXplain val, n=1,922)

Binary-score ECE = 0.1798. Per-category tuned thresholds (F1-maximizing) ship in config.json: cat_thresholds:

Category threshold F1@thr ECE
violence 0.20 0.000 0.195
weapons_explosives 0.20 0.000 0.140
drugs 0.20 0.000 0.103
hate_harassment 0.20 0.834 0.112
sexual_abuse 0.20 0.000 0.124
self_harm 0.20 0.000 0.095
cybercrime_fraud 0.20 0.000 0.130
jailbreak_injection 0.20 0.000 0.088
illegal_marketplace 0.20 0.000 0.083
other_unsafe 0.20 0.000 0.147

Reliability diagram

Multilingual β€” per-language binary F1 (textdetox multilingual toxicity, never in training; + held-out synthetic)

Language n unsafe F1@0.5 best F1
synth_cs 300 150 1.000 1.000
synth_de 300 150 1.000 1.000
synth_es 300 150 1.000 1.000
synth_fr 300 150 1.000 1.000
textdetox_am 800 385 0.141 0.646
textdetox_ar 800 387 0.000 0.049
textdetox_de 800 397 0.596 0.690
textdetox_en 800 391 0.820 0.842
textdetox_es 800 408 0.543 0.674
textdetox_fr 800 391 0.512 0.676
textdetox_he 800 332 0.063 0.305
textdetox_hi 800 409 0.000 0.000
textdetox_hin 800 394 0.543 0.639
textdetox_it 800 409 0.607 0.677
textdetox_ja 800 409 0.000 0.010
textdetox_ru 800 391 0.399 0.655
textdetox_tt 800 403 0.590 0.671
textdetox_uk 800 391 0.393 0.657
textdetox_zh 800 409 0.010 0.190

Cascade pre-filter

On 2,000 held-out rows: 42.8% escalated (gray-zone / jailbreak-category hits), cascade accuracy 0.8600.

Benchmark (latency / throughput / size)

AMD Radeon RX 9060 XT (ROCm) and CPU-only:

Metric .7 GPU .7 CPU .5 GPU
Parameters 902,016 902,016 902,016
Checkpoint 3.6 MB 3.6 MB (int8 0.9 MB) 3.6 MB
Single-text latency 1.24 ms 0.61 ms 1.43 ms
Single throughput 804 texts/s 1628 texts/s 823/s
Batch-8 throughput 5348 texts/s 4603 texts/s 5,709/s
Peak VRAM 85.1 MB β€” 105 MB

Comparison on the internal test split (6,496 rows, threshold 0.5)

Model Params Checkpoint F1 Latency Throughput (b=8)
Censer.7 (ours) 0.9M 3.6 MB 0.8348 1.39 ms 1761/s
KucLab/Censer.5 0.9M 3.6 MB 0.849 1.27 ms 1,978/s
KucLab/Censer.6 0.9M 3.6 MB 0.829 1.22 ms 1,960/s
KucLab/Censer.4 0.9M 3.6 MB 0.845 1.17 ms 2,229/s
KucLab/Censer.1 1.1M 4.5 MB 0.836 1.43 ms 1,853/s
KucLab/Censer.2 7.3M 29.4 MB 0.829 2.70 ms 508/s
KucLab/Censer.3 14.4M 57.7 MB 0.822 4.27 ms 263/s
unitary/unbiased-toxic-roberta 124.7M 498.7 MB 0.741 14.7 ms 67/s
unitary/toxic-bert 109.5M 876.0 MB 0.681 14.2 ms 68/s
martin-ha/toxic-comment-model 67.0M 267.9 MB 0.566 7.2 ms 133/s
Hate-speech-CNERG/bert-base-uncased-hatexplain 109.5M 438.0 MB 0.543 14.7 ms 67/s
Hate-speech-CNERG/dehatebert-mono-english 167.4M 669.5 MB 0.252 14.3 ms 68/s
mistralai/Shieldstral-1.0-3B 3.85B 7.7 GB 0.318 51.6 ms 24/s
keyword baseline β€” β€” 0.094 0.03 ms 34,784/s

Comparison overview F1 per megabyte Metric bars

Usage without transformers

hf download KucLab/Censer.7 --local-dir Censer.7
cd Censer.7
python censer.py "how to make a bomb at home" --json
./censer.7                                # interactive UI

Files:

File Purpose
model.safetensors standard Llama layout, 3.6 MB
model_fp16.safetensors / model_int8.safetensors + quant_report.json 1.8 MB / 0.9 MB quantized
censer7.onnx ONNX graph: input_ids -> 12 logits, dynamic batch/seq
config.json standard config + cat_thresholds (calibrated category cutoffs)
tokenizer.json / tokenizer_config.json byte-level BPE, prompt template baked in
censer.py standalone inference, chunked-max for long inputs (default)
censer.7 launcher for the interactive terminal UI
serve.py batch HTTP API (stdlib only)
Dockerfile CPU container serving the API

Limitations

  • 902K params: nuance/sarcasm/long-tail out of reach; internal test split has label noise from scraped sources.
  • Native single-pass 4096-token inference pools the last token only β€” for long documents ALWAYS use the chunked-max path (default in censer.py/API).
  • Category scores use calibrated per-category thresholds, but remain less reliable than the binary verdict.
  • Multilingual accuracy is best for the trained languages (en/cs/de/es/fr + Jigsaw-ML languages); see the per-language table β€” low-resource languages are weaker.
  • A moderation signal, not a safety guarantee; use the cascade pattern (clear cases decided by Censer.7, gray-zone/jailbreak hits escalated to a larger validator).