--- library_name: transformers pipeline_tag: text-classification tags: - text-classification - safety - content-moderation - multi-label - multilingual - calibrated - tiny-language-model - from-scratch - distilled language: - en - cs - de - es - fr - it - pt - ru - ar - zh - hi license: apache-2.0 datasets: - nvidia/Aegis-AI-Content-Safety-Dataset-1.0 - thesofakillers/jigsaw-toxic-comment-classification-challenge - google/civil_comments - PKU-Alignment/BeaverTails - lmsys/toxic-chat - databricks/databricks-dolly-15k - Hate-speech-CNERG/hatexplain - Intuit-GenSRF/jigsaw-multilingual-train-unique metrics: - f1 - accuracy - ece --- # Censer.7 A **902K parameter** decoder-only transformer trained **from scratch**: binary **safe / unsafe** (F1 restored to .5 level) **+ 10 calibrated safety categories**, **4096-token** native context with chunked-max inference for long documents, and **multilingual** coverage (trained on en + cs/de/es/fr synthetic + Jigsaw-multilingual fr/it/pt/es/ru/tr) — at the same tiny size and speed. Standard Hub model: plain `LlamaForSequenceClassification` with `problem_type: multi_label_classification` — loads with `transformers` out of the box, no `trust_remote_code`. ```python from transformers import pipeline clf = pipeline("text-classification", model="KucLab/Censer.7") clf("how to make a bomb at home", top_k=-1) # [{'label': 'unsafe', ...}, {'label': 'weapons_explosives', ...}, ...] ``` ## What is new in .7 (vs .6) | Fix | .6 | **.7** | | --- | --- | --- | | Binary test F1 | 0.829 (regressed from .5's 0.849) | **0.8348** (two-phase training: binary first, then categories at low lr with cat_weight) | | Probes | 90/97 | **95/97** (diff vs .5: fixed 'Wait, Rockstar really give us fake hope', regressed 'buy a gun') | | External benchmarks | ToxiGen 0.546, HateXplain 0.714, no head-to-head | **ToxiGen 0.561 / HateXplain 0.738 + direct head-to-head vs unbiased-toxic-roberta on the same sets** (table below); HateXplain-train + Jigsaw-multilingual added to training (test splits stay pure) | | Category scores | uncalibrated ("fire liberally") | **per-category thresholds tuned on held-out catval + ECE + reliability diagram** (see below); thresholds ship in `config.json` as `cat_thresholds` | | Multilingual | cs/de/es/fr only, no eval | **per-language F1 on textdetox multilingual toxicity (12 languages) + held-out synthetic** (table below) | | Long documents | native 4096 pass silently broken (F1 0.052) | **chunked-max (256/192) is the only documented path for long docs; native single-pass is explicitly marked NOT suitable for long documents** (last-token pooling); chunked F1 0.998 (acc 0.997) | | Training data | 176k rows | **251,655 rows** (+HateXplain train 15k, +Jigsaw-ML 60k); val/test byte-identical to .1–.6 | ## How to run ### The `censer.7` command — interactive terminal UI ```bash pip install torch tokenizers safetensors huggingface_hub hf download KucLab/Censer.7 --local-dir ~/.local/share/censer7 chmod +x ~/.local/share/censer7/censer.7 ln -s ~/.local/share/censer7/censer.7 ~/.local/bin/censer.7 censer.7 # interactive UI: phrase in -> safe/unsafe + categories + score censer.7 "how to make a bomb at home" ``` Or via the GitHub package (loads the Hub model with plain `transformers`): ```bash pip install "git+https://github.com/Jerry256254/censer.7.git" censer.7 ``` ### One-liner with `transformers` ```bash pip install torch transformers python - <<'PY' import torch from transformers import AutoTokenizer, AutoModelForSequenceClassification tok = AutoTokenizer.from_pretrained("KucLab/Censer.7") model = AutoModelForSequenceClassification.from_pretrained("KucLab/Censer.7").eval() enc = tok("how to make a bomb at home", return_tensors="pt") probs = model(**enc).logits.sigmoid()[0] id2label = model.config.id2label cat_thr = model.config.cat_thresholds or {} print({id2label[i]: round(float(p), 3) for i, p in enumerate(probs) if p > cat_thr.get(id2label[i], 0.5)}) PY ``` - Binary unsafe score = index 1; threshold 0.5. Category labels use per-category calibrated thresholds (`config.json: cat_thresholds`). - Runs on CPU out of the box; GPU via `device=0`. ### Long inputs — always use the chunked path Native context is 4096 tokens, but a **single pass pools only the last token**, so in long documents unsafe content in the middle can be missed (measured F1 0.052 on 25-row synthetic docs). Therefore `censer.py` / the API **always** split inputs longer than 256 tokens into overlapping chunks (256 tokens, stride 192) and aggregate by **max** — this is the default and only recommended path for long documents (measured F1 0.998 (acc 0.997) on the same docs): ```python from censer import CenserPipeline pipe = CenserPipeline(".") pipe.predict("very long text ...") # {"label": "1", "unsafe": true, "score": ..., "categories": [...], "chunks": N} ``` ### Batch HTTP API + Docker ```bash python serve.py --dir . --port 8000 curl -s localhost:8000/classify -d '{"texts": ["hello", "how to make a bomb"]}' docker build -t censer7 . && docker run -p 8000:8000 censer7 ``` Endpoints: `GET /health`, `POST /classify` (batch), `POST /classify-one`. Rate limit via `RATE_LIMIT_PER_MIN` (default 600). ## Model details | Property | Value | | --- | --- | | Architecture | `LlamaForSequenceClassification` (standard transformers) | | Parameters | 902,016 | | Layers / d_model / heads / d_ff | 3 / 128 / 4 / 384 | | Context | 4096 tokens native (linear RoPE ×8); long docs via chunked-max (default) | | Input cap | 4000 chars | | Tokenizer | byte-level BPE, 2048 tokens (multilingual corpus) | | Head | `score` = embedding rows of the 12 answer tokens `0`,`1`,`a`..`j` | | Output | sigmoid over 12 labels; per-category calibrated thresholds in config | | Training | two-phase: (A) 5 epochs binary (cat_weight=0, lr 2e-3) from Censer.6 init, (B) 3 epochs categories (cat_weight=0.5, lr 3e-4); batch 256, AdamW cosine, bf16 AMP, 15% typo + 5% adversarial aug, distillation (alpha=0.65, T=4, boost 0.75) from Censer.5×2+.4+.3 ensemble on 251,655 rows | | Checkpoint | 3.6 MB fp32 · 1.8 MB fp16 · 0.9 MB int8 · ONNX | ## Prompt format `tokenizer(text)` produces `[BOS][USER] [ASSISTANT] ->` answer logits over `"0" "1" "a".."j"` (template baked into the post-processor). ## Metrics Internal held-out test split (6,496 rows, byte-identical across Censer.1–.7), binary threshold 0.5: | Metric | Censer.5 | Censer.6 | **Censer.7** | | --- | --- | --- | --- | | Test F1 | 0.8490 | 0.8289 | **0.8348** | | Test accuracy | 0.876 | 0.858 | **0.8655** | | Precision / recall | 0.864 / 0.834 | 0.833 / 0.825 | **0.855 / 0.816** | | Val F1 | 0.8495 | 0.8358 | **0.8368** | | Probes (97 hand-written) | 95/97 | 90/97 | **95/97** | Per-source test F1: aegis 0.912, BeaverTails 0.881, jigsaw 0.856, civil_comments 0.776. ### External benchmarks (never in training) — with head-to-head vs public models | Dataset | Model | F1@0.5 | best F1 | latency | | --- | --- | --- | --- | --- | | ToxiGen annotated (n=940, 402 unsafe) | **Censer.7** | **0.561** | 0.619 @ 0.27 | 1.24 ms | | ToxiGen | unitary/unbiased-toxic-roberta | 0.443 | 0.483 | ~14 ms | | HateXplain test (n=1,924, 1,376 unsafe) | **Censer.7** | **0.738** | 0.834 @ 0.05 | 1.24 ms | | HateXplain test | unitary/unbiased-toxic-roberta | 0.577 | 0.690 | ~14 ms | (full numbers in `reports/external_h2h.json`) ### Adversarial robustness (detection rate on attacked unsafe texts, n=160) | Attack | .6 | **.7** | | --- | --- | --- | | clean | 91.3% | 93.8% | | typo | 89.4% | 88.8% | | leetspeak | 92.5% | 93.1% | | homoglyph | 98.1% | 98.1% | | zero-width | 95.0% | 97.5% | | case flip | 92.5% | 91.9% | ### Long-context (300 synthetic docs of 25 held-out rows, avg 2,178 tokens) | Inference | F1 | | --- | --- | | truncate at 192 tokens | 0.571 | | native 4096 single pass (NOT recommended for long docs) | 0.052 | | **chunked 256/192 max (default everywhere)** | **0.998 (acc 0.997)** | ### Category calibration (held-out HateXplain val, n=1,922) Binary-score ECE = **0.1798**. Per-category tuned thresholds (F1-maximizing) ship in `config.json: cat_thresholds`: | Category | threshold | F1@thr | ECE | | --- | --- | --- | --- | | violence | 0.20 | 0.000 | 0.195 | | weapons_explosives | 0.20 | 0.000 | 0.140 | | drugs | 0.20 | 0.000 | 0.103 | | hate_harassment | 0.20 | 0.834 | 0.112 | | sexual_abuse | 0.20 | 0.000 | 0.124 | | self_harm | 0.20 | 0.000 | 0.095 | | cybercrime_fraud | 0.20 | 0.000 | 0.130 | | jailbreak_injection | 0.20 | 0.000 | 0.088 | | illegal_marketplace | 0.20 | 0.000 | 0.083 | | other_unsafe | 0.20 | 0.000 | 0.147 | ![Reliability diagram](figs/reliability.png) ### Multilingual — per-language binary F1 (textdetox multilingual toxicity, never in training; + held-out synthetic) | Language | n | unsafe | F1@0.5 | best F1 | | --- | --- | --- | --- | --- | | synth_cs | 300 | 150 | 1.000 | 1.000 | | synth_de | 300 | 150 | 1.000 | 1.000 | | synth_es | 300 | 150 | 1.000 | 1.000 | | synth_fr | 300 | 150 | 1.000 | 1.000 | | textdetox_am | 800 | 385 | 0.141 | 0.646 | | textdetox_ar | 800 | 387 | 0.000 | 0.049 | | textdetox_de | 800 | 397 | 0.596 | 0.690 | | textdetox_en | 800 | 391 | 0.820 | 0.842 | | textdetox_es | 800 | 408 | 0.543 | 0.674 | | textdetox_fr | 800 | 391 | 0.512 | 0.676 | | textdetox_he | 800 | 332 | 0.063 | 0.305 | | textdetox_hi | 800 | 409 | 0.000 | 0.000 | | textdetox_hin | 800 | 394 | 0.543 | 0.639 | | textdetox_it | 800 | 409 | 0.607 | 0.677 | | textdetox_ja | 800 | 409 | 0.000 | 0.010 | | textdetox_ru | 800 | 391 | 0.399 | 0.655 | | textdetox_tt | 800 | 403 | 0.590 | 0.671 | | textdetox_uk | 800 | 391 | 0.393 | 0.657 | | textdetox_zh | 800 | 409 | 0.010 | 0.190 | ### Cascade pre-filter On 2,000 held-out rows: 42.8% escalated (gray-zone / jailbreak-category hits), cascade accuracy 0.8600. ## Benchmark (latency / throughput / size) AMD Radeon RX 9060 XT (ROCm) and CPU-only: | Metric | .7 GPU | .7 CPU | .5 GPU | | --- | --- | --- | --- | | Parameters | 902,016 | 902,016 | 902,016 | | Checkpoint | 3.6 MB | 3.6 MB (int8 0.9 MB) | 3.6 MB | | Single-text latency | 1.24 ms | 0.61 ms | 1.43 ms | | Single throughput | 804 texts/s | 1628 texts/s | 823/s | | Batch-8 throughput | 5348 texts/s | 4603 texts/s | 5,709/s | | Peak VRAM | 85.1 MB | — | 105 MB | ## Comparison on the internal test split (6,496 rows, threshold 0.5) | Model | Params | Checkpoint | F1 | Latency | Throughput (b=8) | | --- | --- | --- | --- | --- | --- | | **Censer.7 (ours)** | **0.9M** | **3.6 MB** | **0.8348** | **1.39 ms** | **1761/s** | | KucLab/Censer.5 | 0.9M | 3.6 MB | 0.849 | 1.27 ms | 1,978/s | | KucLab/Censer.6 | 0.9M | 3.6 MB | 0.829 | 1.22 ms | 1,960/s | | KucLab/Censer.4 | 0.9M | 3.6 MB | 0.845 | 1.17 ms | 2,229/s | | KucLab/Censer.1 | 1.1M | 4.5 MB | 0.836 | 1.43 ms | 1,853/s | | KucLab/Censer.2 | 7.3M | 29.4 MB | 0.829 | 2.70 ms | 508/s | | KucLab/Censer.3 | 14.4M | 57.7 MB | 0.822 | 4.27 ms | 263/s | | unitary/unbiased-toxic-roberta | 124.7M | 498.7 MB | 0.741 | 14.7 ms | 67/s | | unitary/toxic-bert | 109.5M | 876.0 MB | 0.681 | 14.2 ms | 68/s | | martin-ha/toxic-comment-model | 67.0M | 267.9 MB | 0.566 | 7.2 ms | 133/s | | Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 438.0 MB | 0.543 | 14.7 ms | 67/s | | Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 669.5 MB | 0.252 | 14.3 ms | 68/s | | mistralai/Shieldstral-1.0-3B | 3.85B | 7.7 GB | 0.318 | 51.6 ms | 24/s | | keyword baseline | — | — | 0.094 | 0.03 ms | 34,784/s | ![Comparison overview](figs/comparison_overview.png) ![F1 per megabyte](figs/f1_vs_size.png) ![Metric bars](figs/metrics_bars.png) ## Usage without transformers ```bash hf download KucLab/Censer.7 --local-dir Censer.7 cd Censer.7 python censer.py "how to make a bomb at home" --json ./censer.7 # interactive UI ``` Files: | File | Purpose | | --- | --- | | `model.safetensors` | standard Llama layout, 3.6 MB | | `model_fp16.safetensors` / `model_int8.safetensors` + `quant_report.json` | 1.8 MB / 0.9 MB quantized | | `censer7.onnx` | ONNX graph: input_ids -> 12 logits, dynamic batch/seq | | `config.json` | standard config + `cat_thresholds` (calibrated category cutoffs) | | `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, prompt template baked in | | `censer.py` | standalone inference, chunked-max for long inputs (default) | | `censer.7` | launcher for the interactive terminal UI | | `serve.py` | batch HTTP API (stdlib only) | | `Dockerfile` | CPU container serving the API | ## Limitations - 902K params: nuance/sarcasm/long-tail out of reach; internal test split has label noise from scraped sources. - Native single-pass 4096-token inference pools the last token only — for long documents ALWAYS use the chunked-max path (default in `censer.py`/API). - Category scores use calibrated per-category thresholds, but remain less reliable than the binary verdict. - Multilingual accuracy is best for the trained languages (en/cs/de/es/fr + Jigsaw-ML languages); see the per-language table — low-resource languages are weaker. - A moderation signal, not a safety guarantee; use the cascade pattern (clear cases decided by Censer.7, gray-zone/jailbreak hits escalated to a larger validator).