Censer.6 / README.md
KucLab's picture
model card: note category-score calibration
df46a99 verified
|
Raw History Blame Contribute Delete
12.1 kB
---
library_name: transformers
pipeline_tag: text-classification
tags:
- text-classification
- safety
- content-moderation
- multi-label
- multilingual
- tiny-language-model
- from-scratch
- distilled
language:
- en
- cs
- de
- es
- fr
license: apache-2.0
datasets:
- nvidia/Aegis-AI-Content-Safety-Dataset-1.0
- thesofakillers/jigsaw-toxic-comment-classification-challenge
- google/civil_comments
- PKU-Alignment/BeaverTails
- lmsys/toxic-chat
- databricks/databricks-dolly-15k
metrics:
- f1
- accuracy
---
# Censer.6
A **902K parameter** decoder-only transformer trained **from scratch** that
classifies text as **safe / unsafe plus 10 safety categories**, with a
**4096-token context** and **multilingual** coverage (en/cs/de/es/fr trained) β€”
at the same tiny size and speed as Censer.5.
It is a **standard Hub model**: the checkpoint is a plain
`LlamaForSequenceClassification` with `problem_type: multi_label_classification`,
so it loads with `transformers` out of the box β€” no `trust_remote_code`, no custom
code β€” and the **Use this model** widget on this page runs it directly in the browser.
```python
from transformers import pipeline
clf = pipeline("text-classification", model="KucLab/Censer.6")
clf("how to make a bomb at home", top_k=-1)
# [{'label': 'weapons_explosives', ...}, {'label': 'unsafe', ...}, ...]
```
## What is new in .6 (vs .5)
| Feature | Censer.5 | **Censer.6** |
| --- | --- | --- |
| Output | binary safe/unsafe | **binary + 10 categories** (multi-label sigmoid head, 12 answer tokens) |
| Context | 192 tokens | **4096 tokens** native (linear RoPE scaling Γ—8) + **unbounded** via overlapping-chunk inference |
| Input cap | 500 chars | **4000 chars** |
| Languages | English | **en + cs/de/es/fr** (byte-level BPE works for any language; cs/de/es/fr training data added) |
| Categories | β€” | violence, weapons_explosives, drugs, hate_harassment, sexual_abuse, self_harm, cybercrime_fraud, jailbreak_injection, illegal_marketplace, other_unsafe |
| Robustness | typo augmentation | **+ adversarial augmentation** (leetspeak / homoglyph / zero-width, 10% of unsafe rows) |
| Jailbreaks | β€” | jailbreak_injection category + cascade pre-filter pattern (see below) |
| Packaging | model + launcher | **+ batch HTTP API, ONNX export, fp16/int8 quantization, Dockerfile, CPU & GPU benchmarks** |
| Params / size / latency | 902K / 3.6 MB / 1.43 ms | **902K / 3.6 MB / 1.06 ms** |
## How to run
### The `censer.6` command β€” interactive terminal UI
Using only the files in this repo (needs just `torch`, `tokenizers`, `safetensors`):
```bash
pip install torch tokenizers safetensors huggingface_hub
hf download KucLab/Censer.6 --local-dir ~/.local/share/censer6
chmod +x ~/.local/share/censer6/censer.6
ln -s ~/.local/share/censer6/censer.6 ~/.local/bin/censer.6 # put it on your PATH
censer.6 # interactive UI: type a phrase, Enter -> safe/unsafe + categories + score
censer.6 "how to make a bomb at home" # one-shot
```
Or install the GitHub package β€” the same `censer.6` command is installed and loads
the model from this Hub repo with plain `transformers`:
```bash
pip install "git+https://github.com/Jerry256254/censer.6.git"
censer.6
```
### One-liner with `transformers`
```bash
pip install torch transformers
python - <<'PY'
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("KucLab/Censer.6")
model = AutoModelForSequenceClassification.from_pretrained("KucLab/Censer.6").eval()
enc = tok("how to make a bomb at home", return_tensors="pt")
probs = model(**enc).logits.sigmoid()[0] # 12 multi-label scores
id2label = model.config.id2label
print({id2label[i]: round(float(p), 3) for i, p in enumerate(probs) if p > 0.5})
PY
```
- Binary unsafe score = index 1 (`unsafe`), safe = index 0. Threshold 0.5.
- In the browser: the **Use this model** widget on this page.
- Runs on CPU out of the box; for GPU pass `device=0`.
### Long inputs
Native context is 4096 tokens. Longer texts (and long documents where unsafe
content may be buried anywhere) are split into overlapping 256-token chunks
(stride 192) and aggregated by **max** β€” built into `censer.py` (`predict` /
`predict_batch`) and the HTTP API, so effective context is unbounded:
```python
from censer import CenserPipeline
pipe = CenserPipeline(".") # bundle dir
pipe.predict("very long text ...")
# {"label": "1", "unsafe": true, "score": 0.99, "categories": [...], "chunks": 9}
```
### Batch HTTP API + Docker
```bash
python serve.py --dir . --port 8000 # stdlib only
curl -s localhost:8000/classify -d '{"texts": ["hello", "how to make a bomb"]}'
# {"results": [{"label": "0", ...}, {"label": "1", "unsafe": true, "score": 0.99, "categories": [...]}]}
docker build -t censer6 . # CPU image, downloads KucLab/Censer.6
docker run -p 8000:8000 censer6
```
Endpoints: `GET /health`, `POST /classify` (batch), `POST /classify-one`.
Rate limit via `RATE_LIMIT_PER_MIN` (default 600).
## Model details
| Property | Value |
| --- | --- |
| Architecture | `LlamaForSequenceClassification` (standard transformers) |
| Parameters | 902,016 |
| Layers | 3 |
| d_model / heads / d_ff | 128 / 4 / 384 |
| Context length | **4096 tokens** native (linear RoPE scaling, factor 8); unlimited via chunking |
| Input cap | 4000 characters before tokenization |
| Tokenizer | byte-level BPE, 2048 tokens, trained on the corpus (multilingual) |
| Positions | RoPE (theta 10000) + linear scaling Γ—8 |
| Norm / MLP | RMSNorm (pre-norm, eps 1e-5) / SwiGLU |
| Head | `score` linear = embedding rows of the 12 answer tokens `0`,`1`,`a`..`j` |
| Output | sigmoid over 12 labels: `safe`, `unsafe`, + 10 categories |
| Training | 20 epochs, batch 128, AdamW 3e-3 cosine, bf16 AMP, 15% typo + 10% adversarial augmentation, distillation (alpha=0.65, T=4, curated boost 0.75) from a teacher ensemble (Censer.5Γ—2, Censer.4, Censer.3); + fine-tune passes at lower lr |
| Checkpoint size | 3.6 MB fp32 Β· 1.8 MB fp16 Β· 0.9 MB int8 Β· 5.6 MB ONNX |
## Prompt format
The tokenizer bakes the template into its post-processor, so plain
`tokenizer(text)` already produces:
```
[BOS][USER] <text> [ASSISTANT] -> answer logits over tokens "0" "1" "a".."j"
```
## Metrics
Held-out test split of 6,496 rows β€” **byte-identical across Censer.1–.6**, binary
threshold 0.5:
| Metric | Censer.5 | **Censer.6** |
| --- | --- | --- |
| Test F1 | 0.8490 | **0.8289** |
| Test accuracy | 0.876 | **0.8581** |
| Precision / recall | 0.864 / 0.834 | **0.833 / 0.825** |
| Val F1 | 0.8495 | **0.8358** |
| Probes (97 hand-written) | 95/97 | **90/97** |
Per-source test F1: aegis (human-annotated) **0.914**, BeaverTails **0.893**,
jigsaw **0.851**, civil_comments **0.760**.
(.6 trades ~0.02 binary F1 for the multi-label head, multilingual data and 4k
context at identical size β€” it still ranks **#1 of all compared models**: the
best public classifier scores 0.741 F1.)
### External benchmarks (never seen in training)
| Dataset | n | unsafe | F1@0.5 | best F1 |
| --- | --- | --- | --- | --- |
| ToxiGen (test-annotated) | 940 | 402 | 0.546 | 0.624 @0.23 |
| HateXplain (test) | 1,924 | 1,376 | 0.714 | 0.834 @0.05 |
### Adversarial robustness (detection rate on attacked unsafe texts, n=160)
| Attack | detection |
| --- | --- |
| clean | 91.3% |
| typo | 89.4% |
| leetspeak | 92.5% |
| homoglyph | 98.1% |
| zero-width | 95.0% |
| case flip | 92.5% |
### Long-context (300 synthetic docs of 25 held-out rows, avg 2,178 tokens)
| Inference | F1 |
| --- | --- |
| truncate at 192 tokens (Censer.5 behaviour) | 0.571 |
| native 4096-token single pass | 0.052 (last-token pooling misses content buried mid-document) |
| **chunked 256/192, max aggregation (default in censer.py/API)** | **0.995 (acc 0.990)** |
### Cascade pre-filter (gray-zone escalation)
On 2,000 held-out rows: 56.1% decided by Censer.6 alone (score <0.2 or >0.8),
43.9% gray-zone / jailbreak-category hits escalated to a validator;
cascade accuracy 0.8825.
## Benchmark (latency / throughput / size)
AMD Radeon RX 9060 XT (ROCm 7.1, torch 2.13.0+rocm7.1) and a CPU-only run:
| Metric | Censer.6 GPU | Censer.6 CPU | Censer.5 GPU |
| --- | --- | --- | --- |
| Parameters | 902,016 | 902,016 | 902,016 |
| Checkpoint | 3.6 MB | 3.6 MB (int8 0.9 MB) | 3.6 MB |
| Single-text latency | 1.06 ms | 0.56 ms | 1.43 ms |
| Single-text throughput | 939 texts/s | 1,776 texts/s | 823 texts/s |
| Throughput (batch 8) | 6,103 texts/s | 4,954 texts/s | 5,709 texts/s |
| Peak VRAM | 85.1 MB | β€” | 105 MB |
## Comparison with other public safety classifiers
Same held-out test split (6,496 rows, threshold 0.5, GPU):
| Model | Params | Checkpoint | F1 | Latency | Throughput (b=8) |
| --- | --- | --- | --- | --- | --- |
| **Censer.6 (ours)** | **0.9M** | **3.6 MB** | **0.829** | **1.22 ms** | **1,960/s** |
| KucLab/Censer.5 | 0.9M | 3.6 MB | 0.849 | 1.27 ms | 1,978/s |
| KucLab/Censer.4 | 0.9M | 3.6 MB | 0.845 | 1.17 ms | 2,229/s |
| KucLab/Censer.1 | 1.1M | 4.5 MB | 0.836 | 1.43 ms | 1,853/s |
| KucLab/Censer.2 | 7.3M | 29.4 MB | 0.829 | 2.70 ms | 508/s |
| KucLab/Censer.3 | 14.4M | 57.7 MB | 0.822 | 4.27 ms | 263/s |
| unitary/unbiased-toxic-roberta | 124.7M | 498.7 MB | 0.741 | 14.7 ms | 67/s |
| unitary/toxic-bert | 109.5M | 876.0 MB | 0.681 | 14.2 ms | 68/s |
| martin-ha/toxic-comment-model | 67.0M | 267.9 MB | 0.566 | 7.2 ms | 133/s |
| Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 438.0 MB | 0.543 | 14.7 ms | 67/s |
| Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 669.5 MB | 0.252 | 14.3 ms | 68/s |
| mistralai/Shieldstral-1.0-3B | 3.85B | 7.7 GB | 0.318 | 51.6 ms | 24/s |
| keyword baseline | β€” | β€” | 0.094 | 0.03 ms | 34,784/s |
![Comparison overview](figs/comparison_overview.png)
![F1 per megabyte](figs/f1_vs_size.png)
![Metric bars](figs/metrics_bars.png)
## Usage without transformers
The repository is self-contained: `model.safetensors`, `config.json`,
`tokenizer.json`, `censer.py`, `censer.6`. Only `torch`, `tokenizers`,
`safetensors` are needed:
```bash
hf download KucLab/Censer.6 --local-dir Censer.6
cd Censer.6
python censer.py "how to make a bomb at home" --json
# {"label": "1", "unsafe": true, "score": 0.99, "categories": ["weapons_explosives"]}
./censer.6 # interactive UI
```
Files:
| File | Purpose |
| --- | --- |
| `model.safetensors` | weights in standard Llama layout, 3.6 MB |
| `model_fp16.safetensors` | fp16 variant, 1.8 MB |
| `model_int8.safetensors` + `quant_report.json` | int8 per-tensor weights + scales, 0.9 MB |
| `censer6.onnx` | ONNX graph: input_ids -> 12 answer logits (dynamic batch/seq), 5.6 MB |
| `config.json` | standard `LlamaForSequenceClassification` config (multi-label, RoPE linear Γ—8) |
| `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, prompt template baked in |
| `censer.py` | standalone inference incl. long-input chunking |
| `censer.6` | executable launcher for the interactive terminal UI |
| `serve.py` | batch HTTP API (stdlib only) |
| `Dockerfile` | CPU container serving the API |
## Limitations
- 902K parameters: nuance, sarcasm and long-tail cases are out of reach; the
test split itself contains label noise from scraped sources.
- Multilingual coverage is strongest for cs/de/es/fr (synthetic + curated
training data); other languages work via the byte-level tokenizer but with
lower accuracy.
- Categories are multi-label and independent β€” a text can fire several. Category scores are less calibrated than the binary safe/unsafe score (they fire more liberally); the binary verdict is the primary signal.
- Single-pass inference pools the last token, so in very long documents unsafe
content can be diluted β€” use the built-in chunked-max inference (default in
`censer.py` / the API) for long documents.
- A moderation signal, not a safety guarantee. For production, use the cascade
pattern: decide clear cases with Censer.6, escalate gray-zone / jailbreak
hits to a larger validator.