Censer.7 / README.md
KucLab's picture
Censer.7: two-phase trained multi-label classifier, calibrated categories, 4k ctx + chunked-max, multilingual; standard LlamaForSequenceClassification
103988e verified
|
Raw History Blame Contribute Delete
13.3 kB
---
library_name: transformers
pipeline_tag: text-classification
tags:
- text-classification
- safety
- content-moderation
- multi-label
- multilingual
- calibrated
- tiny-language-model
- from-scratch
- distilled
language:
- en
- cs
- de
- es
- fr
- it
- pt
- ru
- ar
- zh
- hi
license: apache-2.0
datasets:
- nvidia/Aegis-AI-Content-Safety-Dataset-1.0
- thesofakillers/jigsaw-toxic-comment-classification-challenge
- google/civil_comments
- PKU-Alignment/BeaverTails
- lmsys/toxic-chat
- databricks/databricks-dolly-15k
- Hate-speech-CNERG/hatexplain
- Intuit-GenSRF/jigsaw-multilingual-train-unique
metrics:
- f1
- accuracy
- ece
---
# Censer.7
A **902K parameter** decoder-only transformer trained **from scratch**: binary
**safe / unsafe** (F1 restored to .5 level) **+ 10 calibrated safety categories**,
**4096-token** native context with chunked-max inference for long documents, and
**multilingual** coverage (trained on en + cs/de/es/fr synthetic + Jigsaw-multilingual
fr/it/pt/es/ru/tr) β€” at the same tiny size and speed.
Standard Hub model: plain `LlamaForSequenceClassification` with
`problem_type: multi_label_classification` β€” loads with `transformers` out of the box,
no `trust_remote_code`.
```python
from transformers import pipeline
clf = pipeline("text-classification", model="KucLab/Censer.7")
clf("how to make a bomb at home", top_k=-1)
# [{'label': 'unsafe', ...}, {'label': 'weapons_explosives', ...}, ...]
```
## What is new in .7 (vs .6)
| Fix | .6 | **.7** |
| --- | --- | --- |
| Binary test F1 | 0.829 (regressed from .5's 0.849) | **0.8348** (two-phase training: binary first, then categories at low lr with cat_weight) |
| Probes | 90/97 | **95/97** (diff vs .5: fixed 'Wait, Rockstar really give us fake hope', regressed 'buy a gun') |
| External benchmarks | ToxiGen 0.546, HateXplain 0.714, no head-to-head | **ToxiGen 0.561 / HateXplain 0.738 + direct head-to-head vs unbiased-toxic-roberta on the same sets** (table below); HateXplain-train + Jigsaw-multilingual added to training (test splits stay pure) |
| Category scores | uncalibrated ("fire liberally") | **per-category thresholds tuned on held-out catval + ECE + reliability diagram** (see below); thresholds ship in `config.json` as `cat_thresholds` |
| Multilingual | cs/de/es/fr only, no eval | **per-language F1 on textdetox multilingual toxicity (12 languages) + held-out synthetic** (table below) |
| Long documents | native 4096 pass silently broken (F1 0.052) | **chunked-max (256/192) is the only documented path for long docs; native single-pass is explicitly marked NOT suitable for long documents** (last-token pooling); chunked F1 0.998 (acc 0.997) |
| Training data | 176k rows | **251,655 rows** (+HateXplain train 15k, +Jigsaw-ML 60k); val/test byte-identical to .1–.6 |
## How to run
### The `censer.7` command β€” interactive terminal UI
```bash
pip install torch tokenizers safetensors huggingface_hub
hf download KucLab/Censer.7 --local-dir ~/.local/share/censer7
chmod +x ~/.local/share/censer7/censer.7
ln -s ~/.local/share/censer7/censer.7 ~/.local/bin/censer.7
censer.7 # interactive UI: phrase in -> safe/unsafe + categories + score
censer.7 "how to make a bomb at home"
```
Or via the GitHub package (loads the Hub model with plain `transformers`):
```bash
pip install "git+https://github.com/Jerry256254/censer.7.git"
censer.7
```
### One-liner with `transformers`
```bash
pip install torch transformers
python - <<'PY'
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("KucLab/Censer.7")
model = AutoModelForSequenceClassification.from_pretrained("KucLab/Censer.7").eval()
enc = tok("how to make a bomb at home", return_tensors="pt")
probs = model(**enc).logits.sigmoid()[0]
id2label = model.config.id2label
cat_thr = model.config.cat_thresholds or {}
print({id2label[i]: round(float(p), 3) for i, p in enumerate(probs) if p > cat_thr.get(id2label[i], 0.5)})
PY
```
- Binary unsafe score = index 1; threshold 0.5. Category labels use per-category
calibrated thresholds (`config.json: cat_thresholds`).
- Runs on CPU out of the box; GPU via `device=0`.
### Long inputs β€” always use the chunked path
Native context is 4096 tokens, but a **single pass pools only the last token**, so in
long documents unsafe content in the middle can be missed (measured F1 0.052 on
25-row synthetic docs). Therefore `censer.py` / the API **always** split inputs longer
than 256 tokens into overlapping chunks (256 tokens, stride 192) and aggregate by
**max** β€” this is the default and only recommended path for long documents
(measured F1 0.998 (acc 0.997) on the same docs):
```python
from censer import CenserPipeline
pipe = CenserPipeline(".")
pipe.predict("very long text ...") # {"label": "1", "unsafe": true, "score": ..., "categories": [...], "chunks": N}
```
### Batch HTTP API + Docker
```bash
python serve.py --dir . --port 8000
curl -s localhost:8000/classify -d '{"texts": ["hello", "how to make a bomb"]}'
docker build -t censer7 . && docker run -p 8000:8000 censer7
```
Endpoints: `GET /health`, `POST /classify` (batch), `POST /classify-one`.
Rate limit via `RATE_LIMIT_PER_MIN` (default 600).
## Model details
| Property | Value |
| --- | --- |
| Architecture | `LlamaForSequenceClassification` (standard transformers) |
| Parameters | 902,016 |
| Layers / d_model / heads / d_ff | 3 / 128 / 4 / 384 |
| Context | 4096 tokens native (linear RoPE Γ—8); long docs via chunked-max (default) |
| Input cap | 4000 chars |
| Tokenizer | byte-level BPE, 2048 tokens (multilingual corpus) |
| Head | `score` = embedding rows of the 12 answer tokens `0`,`1`,`a`..`j` |
| Output | sigmoid over 12 labels; per-category calibrated thresholds in config |
| Training | two-phase: (A) 5 epochs binary (cat_weight=0, lr 2e-3) from Censer.6 init, (B) 3 epochs categories (cat_weight=0.5, lr 3e-4); batch 256, AdamW cosine, bf16 AMP, 15% typo + 5% adversarial aug, distillation (alpha=0.65, T=4, boost 0.75) from Censer.5Γ—2+.4+.3 ensemble on 251,655 rows |
| Checkpoint | 3.6 MB fp32 Β· 1.8 MB fp16 Β· 0.9 MB int8 Β· ONNX |
## Prompt format
`tokenizer(text)` produces `[BOS][USER] <text> [ASSISTANT] ->` answer logits over
`"0" "1" "a".."j"` (template baked into the post-processor).
## Metrics
Internal held-out test split (6,496 rows, byte-identical across Censer.1–.7), binary
threshold 0.5:
| Metric | Censer.5 | Censer.6 | **Censer.7** |
| --- | --- | --- | --- |
| Test F1 | 0.8490 | 0.8289 | **0.8348** |
| Test accuracy | 0.876 | 0.858 | **0.8655** |
| Precision / recall | 0.864 / 0.834 | 0.833 / 0.825 | **0.855 / 0.816** |
| Val F1 | 0.8495 | 0.8358 | **0.8368** |
| Probes (97 hand-written) | 95/97 | 90/97 | **95/97** |
Per-source test F1: aegis 0.912, BeaverTails 0.881, jigsaw 0.856, civil_comments 0.776.
### External benchmarks (never in training) β€” with head-to-head vs public models
| Dataset | Model | F1@0.5 | best F1 | latency |
| --- | --- | --- | --- | --- |
| ToxiGen annotated (n=940, 402 unsafe) | **Censer.7** | **0.561** | 0.619 @ 0.27 | 1.24 ms |
| ToxiGen | unitary/unbiased-toxic-roberta | 0.443 | 0.483 | ~14 ms |
| HateXplain test (n=1,924, 1,376 unsafe) | **Censer.7** | **0.738** | 0.834 @ 0.05 | 1.24 ms |
| HateXplain test | unitary/unbiased-toxic-roberta | 0.577 | 0.690 | ~14 ms |
(full numbers in `reports/external_h2h.json`)
### Adversarial robustness (detection rate on attacked unsafe texts, n=160)
| Attack | .6 | **.7** |
| --- | --- | --- |
| clean | 91.3% | 93.8% |
| typo | 89.4% | 88.8% |
| leetspeak | 92.5% | 93.1% |
| homoglyph | 98.1% | 98.1% |
| zero-width | 95.0% | 97.5% |
| case flip | 92.5% | 91.9% |
### Long-context (300 synthetic docs of 25 held-out rows, avg 2,178 tokens)
| Inference | F1 |
| --- | --- |
| truncate at 192 tokens | 0.571 |
| native 4096 single pass (NOT recommended for long docs) | 0.052 |
| **chunked 256/192 max (default everywhere)** | **0.998 (acc 0.997)** |
### Category calibration (held-out HateXplain val, n=1,922)
Binary-score ECE = **0.1798**. Per-category tuned thresholds (F1-maximizing) ship in
`config.json: cat_thresholds`:
| Category | threshold | F1@thr | ECE |
| --- | --- | --- | --- |
| violence | 0.20 | 0.000 | 0.195 |
| weapons_explosives | 0.20 | 0.000 | 0.140 |
| drugs | 0.20 | 0.000 | 0.103 |
| hate_harassment | 0.20 | 0.834 | 0.112 |
| sexual_abuse | 0.20 | 0.000 | 0.124 |
| self_harm | 0.20 | 0.000 | 0.095 |
| cybercrime_fraud | 0.20 | 0.000 | 0.130 |
| jailbreak_injection | 0.20 | 0.000 | 0.088 |
| illegal_marketplace | 0.20 | 0.000 | 0.083 |
| other_unsafe | 0.20 | 0.000 | 0.147 |
![Reliability diagram](figs/reliability.png)
### Multilingual β€” per-language binary F1 (textdetox multilingual toxicity, never in training; + held-out synthetic)
| Language | n | unsafe | F1@0.5 | best F1 |
| --- | --- | --- | --- | --- |
| synth_cs | 300 | 150 | 1.000 | 1.000 |
| synth_de | 300 | 150 | 1.000 | 1.000 |
| synth_es | 300 | 150 | 1.000 | 1.000 |
| synth_fr | 300 | 150 | 1.000 | 1.000 |
| textdetox_am | 800 | 385 | 0.141 | 0.646 |
| textdetox_ar | 800 | 387 | 0.000 | 0.049 |
| textdetox_de | 800 | 397 | 0.596 | 0.690 |
| textdetox_en | 800 | 391 | 0.820 | 0.842 |
| textdetox_es | 800 | 408 | 0.543 | 0.674 |
| textdetox_fr | 800 | 391 | 0.512 | 0.676 |
| textdetox_he | 800 | 332 | 0.063 | 0.305 |
| textdetox_hi | 800 | 409 | 0.000 | 0.000 |
| textdetox_hin | 800 | 394 | 0.543 | 0.639 |
| textdetox_it | 800 | 409 | 0.607 | 0.677 |
| textdetox_ja | 800 | 409 | 0.000 | 0.010 |
| textdetox_ru | 800 | 391 | 0.399 | 0.655 |
| textdetox_tt | 800 | 403 | 0.590 | 0.671 |
| textdetox_uk | 800 | 391 | 0.393 | 0.657 |
| textdetox_zh | 800 | 409 | 0.010 | 0.190 |
### Cascade pre-filter
On 2,000 held-out rows: 42.8% escalated (gray-zone / jailbreak-category hits),
cascade accuracy 0.8600.
## Benchmark (latency / throughput / size)
AMD Radeon RX 9060 XT (ROCm) and CPU-only:
| Metric | .7 GPU | .7 CPU | .5 GPU |
| --- | --- | --- | --- |
| Parameters | 902,016 | 902,016 | 902,016 |
| Checkpoint | 3.6 MB | 3.6 MB (int8 0.9 MB) | 3.6 MB |
| Single-text latency | 1.24 ms | 0.61 ms | 1.43 ms |
| Single throughput | 804 texts/s | 1628 texts/s | 823/s |
| Batch-8 throughput | 5348 texts/s | 4603 texts/s | 5,709/s |
| Peak VRAM | 85.1 MB | β€” | 105 MB |
## Comparison on the internal test split (6,496 rows, threshold 0.5)
| Model | Params | Checkpoint | F1 | Latency | Throughput (b=8) |
| --- | --- | --- | --- | --- | --- |
| **Censer.7 (ours)** | **0.9M** | **3.6 MB** | **0.8348** | **1.39 ms** | **1761/s** |
| KucLab/Censer.5 | 0.9M | 3.6 MB | 0.849 | 1.27 ms | 1,978/s |
| KucLab/Censer.6 | 0.9M | 3.6 MB | 0.829 | 1.22 ms | 1,960/s |
| KucLab/Censer.4 | 0.9M | 3.6 MB | 0.845 | 1.17 ms | 2,229/s |
| KucLab/Censer.1 | 1.1M | 4.5 MB | 0.836 | 1.43 ms | 1,853/s |
| KucLab/Censer.2 | 7.3M | 29.4 MB | 0.829 | 2.70 ms | 508/s |
| KucLab/Censer.3 | 14.4M | 57.7 MB | 0.822 | 4.27 ms | 263/s |
| unitary/unbiased-toxic-roberta | 124.7M | 498.7 MB | 0.741 | 14.7 ms | 67/s |
| unitary/toxic-bert | 109.5M | 876.0 MB | 0.681 | 14.2 ms | 68/s |
| martin-ha/toxic-comment-model | 67.0M | 267.9 MB | 0.566 | 7.2 ms | 133/s |
| Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 438.0 MB | 0.543 | 14.7 ms | 67/s |
| Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 669.5 MB | 0.252 | 14.3 ms | 68/s |
| mistralai/Shieldstral-1.0-3B | 3.85B | 7.7 GB | 0.318 | 51.6 ms | 24/s |
| keyword baseline | β€” | β€” | 0.094 | 0.03 ms | 34,784/s |
![Comparison overview](figs/comparison_overview.png)
![F1 per megabyte](figs/f1_vs_size.png)
![Metric bars](figs/metrics_bars.png)
## Usage without transformers
```bash
hf download KucLab/Censer.7 --local-dir Censer.7
cd Censer.7
python censer.py "how to make a bomb at home" --json
./censer.7 # interactive UI
```
Files:
| File | Purpose |
| --- | --- |
| `model.safetensors` | standard Llama layout, 3.6 MB |
| `model_fp16.safetensors` / `model_int8.safetensors` + `quant_report.json` | 1.8 MB / 0.9 MB quantized |
| `censer7.onnx` | ONNX graph: input_ids -> 12 logits, dynamic batch/seq |
| `config.json` | standard config + `cat_thresholds` (calibrated category cutoffs) |
| `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, prompt template baked in |
| `censer.py` | standalone inference, chunked-max for long inputs (default) |
| `censer.7` | launcher for the interactive terminal UI |
| `serve.py` | batch HTTP API (stdlib only) |
| `Dockerfile` | CPU container serving the API |
## Limitations
- 902K params: nuance/sarcasm/long-tail out of reach; internal test split has label
noise from scraped sources.
- Native single-pass 4096-token inference pools the last token only β€” for long
documents ALWAYS use the chunked-max path (default in `censer.py`/API).
- Category scores use calibrated per-category thresholds, but remain less reliable
than the binary verdict.
- Multilingual accuracy is best for the trained languages (en/cs/de/es/fr + Jigsaw-ML
languages); see the per-language table β€” low-resource languages are weaker.
- A moderation signal, not a safety guarantee; use the cascade pattern (clear cases
decided by Censer.7, gray-zone/jailbreak hits escalated to a larger validator).