Text Classification
Transformers
ONNX
Safetensors
llama
safety
content-moderation
multi-label
multilingual
tiny-language-model
from-scratch
distilled
text-embeddings-inference
Instructions to use KucLab/Censer.6 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KucLab/Censer.6 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="KucLab/Censer.6")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("KucLab/Censer.6") model = AutoModelForSequenceClassification.from_pretrained("KucLab/Censer.6", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Censer.6: multi-label 10 categories + 4k context + multilingual, standard LlamaForSequenceClassification (no trust_remote_code)
Browse files- Dockerfile +27 -0
- README.md +295 -0
- censer.6 +30 -0
- censer.py +352 -0
- censer6.onnx +3 -0
- config.json +84 -0
- model.safetensors +3 -0
- model_fp16.safetensors +3 -0
- model_int8.safetensors +3 -0
- quant_report.json +6 -0
- serve.py +115 -0
- tokenizer.json +0 -0
- tokenizer_config.json +6 -0
Dockerfile
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Censer.6 — self-contained moderation API image (CPU-only torch by default).
|
| 2 |
+
#
|
| 3 |
+
# docker build -t censer6 .
|
| 4 |
+
# docker run -p 8000:8000 -e RATE_LIMIT_PER_MIN=600 censer6
|
| 5 |
+
# curl -s localhost:8000/classify -d '{"texts":["how to make a bomb at home","hello"]}'
|
| 6 |
+
#
|
| 7 |
+
# The image downloads the model from the Hub at build time (or copy a local
|
| 8 |
+
# bundle with: docker build --build-arg CENSER_DIR=./bundle . — then it is baked in).
|
| 9 |
+
|
| 10 |
+
FROM python:3.12-slim
|
| 11 |
+
|
| 12 |
+
ARG CENSER_REPO=KucLab/Censer.6
|
| 13 |
+
ENV CENSER_DIR=/model \
|
| 14 |
+
RATE_LIMIT_PER_MIN=600 \
|
| 15 |
+
PYTHONUNBUFFERED=1
|
| 16 |
+
|
| 17 |
+
RUN pip install --no-cache-dir \
|
| 18 |
+
torch --index-url https://download.pytorch.org/whl/cpu \
|
| 19 |
+
tokenizers safetensors huggingface_hub
|
| 20 |
+
|
| 21 |
+
RUN hf download ${CENSER_REPO} --local-dir /model
|
| 22 |
+
|
| 23 |
+
WORKDIR /app
|
| 24 |
+
COPY scripts/censer.py scripts/serve.py /app/
|
| 25 |
+
|
| 26 |
+
EXPOSE 8000
|
| 27 |
+
CMD ["python", "serve.py", "--dir", "/model", "--port", "8000", "--host", "0.0.0.0"]
|
README.md
ADDED
|
@@ -0,0 +1,295 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: transformers
|
| 3 |
+
pipeline_tag: text-classification
|
| 4 |
+
tags:
|
| 5 |
+
- text-classification
|
| 6 |
+
- safety
|
| 7 |
+
- content-moderation
|
| 8 |
+
- multi-label
|
| 9 |
+
- multilingual
|
| 10 |
+
- tiny-language-model
|
| 11 |
+
- from-scratch
|
| 12 |
+
- distilled
|
| 13 |
+
language:
|
| 14 |
+
- en
|
| 15 |
+
- cs
|
| 16 |
+
- de
|
| 17 |
+
- es
|
| 18 |
+
- fr
|
| 19 |
+
license: apache-2.0
|
| 20 |
+
datasets:
|
| 21 |
+
- nvidia/Aegis-AI-Content-Safety-Dataset-1.0
|
| 22 |
+
- thesofakillers/jigsaw-toxic-comment-classification-challenge
|
| 23 |
+
- google/civil_comments
|
| 24 |
+
- PKU-Alignment/BeaverTails
|
| 25 |
+
- lmsys/toxic-chat
|
| 26 |
+
- databricks/databricks-dolly-15k
|
| 27 |
+
metrics:
|
| 28 |
+
- f1
|
| 29 |
+
- accuracy
|
| 30 |
+
---
|
| 31 |
+
|
| 32 |
+
# Censer.6
|
| 33 |
+
|
| 34 |
+
A **902K parameter** decoder-only transformer trained **from scratch** that
|
| 35 |
+
classifies text as **safe / unsafe plus 10 safety categories**, with a
|
| 36 |
+
**4096-token context** and **multilingual** coverage (en/cs/de/es/fr trained) —
|
| 37 |
+
at the same tiny size and speed as Censer.5.
|
| 38 |
+
|
| 39 |
+
It is a **standard Hub model**: the checkpoint is a plain
|
| 40 |
+
`LlamaForSequenceClassification` with `problem_type: multi_label_classification`,
|
| 41 |
+
so it loads with `transformers` out of the box — no `trust_remote_code`, no custom
|
| 42 |
+
code — and the **Use this model** widget on this page runs it directly in the browser.
|
| 43 |
+
|
| 44 |
+
```python
|
| 45 |
+
from transformers import pipeline
|
| 46 |
+
|
| 47 |
+
clf = pipeline("text-classification", model="KucLab/Censer.6")
|
| 48 |
+
clf("how to make a bomb at home", top_k=-1)
|
| 49 |
+
# [{'label': 'weapons_explosives', ...}, {'label': 'unsafe', ...}, ...]
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
## What is new in .6 (vs .5)
|
| 53 |
+
|
| 54 |
+
| Feature | Censer.5 | **Censer.6** |
|
| 55 |
+
| --- | --- | --- |
|
| 56 |
+
| Output | binary safe/unsafe | **binary + 10 categories** (multi-label sigmoid head, 12 answer tokens) |
|
| 57 |
+
| Context | 192 tokens | **4096 tokens** native (linear RoPE scaling ×8) + **unbounded** via overlapping-chunk inference |
|
| 58 |
+
| Input cap | 500 chars | **4000 chars** |
|
| 59 |
+
| Languages | English | **en + cs/de/es/fr** (byte-level BPE works for any language; cs/de/es/fr training data added) |
|
| 60 |
+
| Categories | — | violence, weapons_explosives, drugs, hate_harassment, sexual_abuse, self_harm, cybercrime_fraud, jailbreak_injection, illegal_marketplace, other_unsafe |
|
| 61 |
+
| Robustness | typo augmentation | **+ adversarial augmentation** (leetspeak / homoglyph / zero-width, 10% of unsafe rows) |
|
| 62 |
+
| Jailbreaks | — | jailbreak_injection category + cascade pre-filter pattern (see below) |
|
| 63 |
+
| Packaging | model + launcher | **+ batch HTTP API, ONNX export, fp16/int8 quantization, Dockerfile, CPU & GPU benchmarks** |
|
| 64 |
+
| Params / size / latency | 902K / 3.6 MB / 1.43 ms | **902K / 3.6 MB / 1.06 ms** |
|
| 65 |
+
|
| 66 |
+
## How to run
|
| 67 |
+
|
| 68 |
+
### The `censer.6` command — interactive terminal UI
|
| 69 |
+
|
| 70 |
+
Using only the files in this repo (needs just `torch`, `tokenizers`, `safetensors`):
|
| 71 |
+
|
| 72 |
+
```bash
|
| 73 |
+
pip install torch tokenizers safetensors huggingface_hub
|
| 74 |
+
hf download KucLab/Censer.6 --local-dir ~/.local/share/censer6
|
| 75 |
+
chmod +x ~/.local/share/censer6/censer.6
|
| 76 |
+
ln -s ~/.local/share/censer6/censer.6 ~/.local/bin/censer.6 # put it on your PATH
|
| 77 |
+
censer.6 # interactive UI: type a phrase, Enter -> safe/unsafe + categories + score
|
| 78 |
+
censer.6 "how to make a bomb at home" # one-shot
|
| 79 |
+
```
|
| 80 |
+
|
| 81 |
+
Or install the GitHub package — the same `censer.6` command is installed and loads
|
| 82 |
+
the model from this Hub repo with plain `transformers`:
|
| 83 |
+
|
| 84 |
+
```bash
|
| 85 |
+
pip install "git+https://github.com/Jerry256254/censer.6.git"
|
| 86 |
+
censer.6
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
### One-liner with `transformers`
|
| 90 |
+
|
| 91 |
+
```bash
|
| 92 |
+
pip install torch transformers
|
| 93 |
+
python - <<'PY'
|
| 94 |
+
import torch
|
| 95 |
+
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
| 96 |
+
tok = AutoTokenizer.from_pretrained("KucLab/Censer.6")
|
| 97 |
+
model = AutoModelForSequenceClassification.from_pretrained("KucLab/Censer.6").eval()
|
| 98 |
+
enc = tok("how to make a bomb at home", return_tensors="pt")
|
| 99 |
+
probs = model(**enc).logits.sigmoid()[0] # 12 multi-label scores
|
| 100 |
+
id2label = model.config.id2label
|
| 101 |
+
print({id2label[i]: round(float(p), 3) for i, p in enumerate(probs) if p > 0.5})
|
| 102 |
+
PY
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
- Binary unsafe score = index 1 (`unsafe`), safe = index 0. Threshold 0.5.
|
| 106 |
+
- In the browser: the **Use this model** widget on this page.
|
| 107 |
+
- Runs on CPU out of the box; for GPU pass `device=0`.
|
| 108 |
+
|
| 109 |
+
### Long inputs
|
| 110 |
+
|
| 111 |
+
Native context is 4096 tokens. Longer texts (and long documents where unsafe
|
| 112 |
+
content may be buried anywhere) are split into overlapping 256-token chunks
|
| 113 |
+
(stride 192) and aggregated by **max** — built into `censer.py` (`predict` /
|
| 114 |
+
`predict_batch`) and the HTTP API, so effective context is unbounded:
|
| 115 |
+
|
| 116 |
+
```python
|
| 117 |
+
from censer import CenserPipeline
|
| 118 |
+
pipe = CenserPipeline(".") # bundle dir
|
| 119 |
+
pipe.predict("very long text ...")
|
| 120 |
+
# {"label": "1", "unsafe": true, "score": 0.99, "categories": [...], "chunks": 9}
|
| 121 |
+
```
|
| 122 |
+
|
| 123 |
+
### Batch HTTP API + Docker
|
| 124 |
+
|
| 125 |
+
```bash
|
| 126 |
+
python serve.py --dir . --port 8000 # stdlib only
|
| 127 |
+
curl -s localhost:8000/classify -d '{"texts": ["hello", "how to make a bomb"]}'
|
| 128 |
+
# {"results": [{"label": "0", ...}, {"label": "1", "unsafe": true, "score": 0.99, "categories": [...]}]}
|
| 129 |
+
|
| 130 |
+
docker build -t censer6 . # CPU image, downloads KucLab/Censer.6
|
| 131 |
+
docker run -p 8000:8000 censer6
|
| 132 |
+
```
|
| 133 |
+
|
| 134 |
+
Endpoints: `GET /health`, `POST /classify` (batch), `POST /classify-one`.
|
| 135 |
+
Rate limit via `RATE_LIMIT_PER_MIN` (default 600).
|
| 136 |
+
|
| 137 |
+
## Model details
|
| 138 |
+
|
| 139 |
+
| Property | Value |
|
| 140 |
+
| --- | --- |
|
| 141 |
+
| Architecture | `LlamaForSequenceClassification` (standard transformers) |
|
| 142 |
+
| Parameters | 902,016 |
|
| 143 |
+
| Layers | 3 |
|
| 144 |
+
| d_model / heads / d_ff | 128 / 4 / 384 |
|
| 145 |
+
| Context length | **4096 tokens** native (linear RoPE scaling, factor 8); unlimited via chunking |
|
| 146 |
+
| Input cap | 4000 characters before tokenization |
|
| 147 |
+
| Tokenizer | byte-level BPE, 2048 tokens, trained on the corpus (multilingual) |
|
| 148 |
+
| Positions | RoPE (theta 10000) + linear scaling ×8 |
|
| 149 |
+
| Norm / MLP | RMSNorm (pre-norm, eps 1e-5) / SwiGLU |
|
| 150 |
+
| Head | `score` linear = embedding rows of the 12 answer tokens `0`,`1`,`a`..`j` |
|
| 151 |
+
| Output | sigmoid over 12 labels: `safe`, `unsafe`, + 10 categories |
|
| 152 |
+
| Training | 20 epochs, batch 128, AdamW 3e-3 cosine, bf16 AMP, 15% typo + 10% adversarial augmentation, distillation (alpha=0.65, T=4, curated boost 0.75) from a teacher ensemble (Censer.5×2, Censer.4, Censer.3); + fine-tune passes at lower lr |
|
| 153 |
+
| Checkpoint size | 3.6 MB fp32 · 1.8 MB fp16 · 0.9 MB int8 · 5.6 MB ONNX |
|
| 154 |
+
|
| 155 |
+
## Prompt format
|
| 156 |
+
|
| 157 |
+
The tokenizer bakes the template into its post-processor, so plain
|
| 158 |
+
`tokenizer(text)` already produces:
|
| 159 |
+
|
| 160 |
+
```
|
| 161 |
+
[BOS][USER] <text> [ASSISTANT] -> answer logits over tokens "0" "1" "a".."j"
|
| 162 |
+
```
|
| 163 |
+
|
| 164 |
+
## Metrics
|
| 165 |
+
|
| 166 |
+
Held-out test split of 6,496 rows — **byte-identical across Censer.1–.6**, binary
|
| 167 |
+
threshold 0.5:
|
| 168 |
+
|
| 169 |
+
| Metric | Censer.5 | **Censer.6** |
|
| 170 |
+
| --- | --- | --- |
|
| 171 |
+
| Test F1 | 0.8490 | **0.8289** |
|
| 172 |
+
| Test accuracy | 0.876 | **0.8581** |
|
| 173 |
+
| Precision / recall | 0.864 / 0.834 | **0.833 / 0.825** |
|
| 174 |
+
| Val F1 | 0.8495 | **0.8358** |
|
| 175 |
+
| Probes (97 hand-written) | 95/97 | **90/97** |
|
| 176 |
+
|
| 177 |
+
Per-source test F1: aegis (human-annotated) **0.914**, BeaverTails **0.893**,
|
| 178 |
+
jigsaw **0.851**, civil_comments **0.760**.
|
| 179 |
+
|
| 180 |
+
(.6 trades ~0.02 binary F1 for the multi-label head, multilingual data and 4k
|
| 181 |
+
context at identical size — it still ranks **#1 of all compared models**: the
|
| 182 |
+
best public classifier scores 0.741 F1.)
|
| 183 |
+
|
| 184 |
+
### External benchmarks (never seen in training)
|
| 185 |
+
|
| 186 |
+
| Dataset | n | unsafe | F1@0.5 | best F1 |
|
| 187 |
+
| --- | --- | --- | --- | --- |
|
| 188 |
+
| ToxiGen (test-annotated) | 940 | 402 | 0.546 | 0.624 @0.23 |
|
| 189 |
+
| HateXplain (test) | 1,924 | 1,376 | 0.714 | 0.834 @0.05 |
|
| 190 |
+
|
| 191 |
+
### Adversarial robustness (detection rate on attacked unsafe texts, n=160)
|
| 192 |
+
|
| 193 |
+
| Attack | detection |
|
| 194 |
+
| --- | --- |
|
| 195 |
+
| clean | 91.3% |
|
| 196 |
+
| typo | 89.4% |
|
| 197 |
+
| leetspeak | 92.5% |
|
| 198 |
+
| homoglyph | 98.1% |
|
| 199 |
+
| zero-width | 95.0% |
|
| 200 |
+
| case flip | 92.5% |
|
| 201 |
+
|
| 202 |
+
### Long-context (300 synthetic docs of 25 held-out rows, avg 2,178 tokens)
|
| 203 |
+
|
| 204 |
+
| Inference | F1 |
|
| 205 |
+
| --- | --- |
|
| 206 |
+
| truncate at 192 tokens (Censer.5 behaviour) | 0.571 |
|
| 207 |
+
| native 4096-token single pass | 0.052 (last-token pooling misses content buried mid-document) |
|
| 208 |
+
| **chunked 256/192, max aggregation (default in censer.py/API)** | **0.995 (acc 0.990)** |
|
| 209 |
+
|
| 210 |
+
### Cascade pre-filter (gray-zone escalation)
|
| 211 |
+
|
| 212 |
+
On 2,000 held-out rows: 56.1% decided by Censer.6 alone (score <0.2 or >0.8),
|
| 213 |
+
43.9% gray-zone / jailbreak-category hits escalated to a validator;
|
| 214 |
+
cascade accuracy 0.8825.
|
| 215 |
+
|
| 216 |
+
## Benchmark (latency / throughput / size)
|
| 217 |
+
|
| 218 |
+
AMD Radeon RX 9060 XT (ROCm 7.1, torch 2.13.0+rocm7.1) and a CPU-only run:
|
| 219 |
+
|
| 220 |
+
| Metric | Censer.6 GPU | Censer.6 CPU | Censer.5 GPU |
|
| 221 |
+
| --- | --- | --- | --- |
|
| 222 |
+
| Parameters | 902,016 | 902,016 | 902,016 |
|
| 223 |
+
| Checkpoint | 3.6 MB | 3.6 MB (int8 0.9 MB) | 3.6 MB |
|
| 224 |
+
| Single-text latency | 1.06 ms | 0.56 ms | 1.43 ms |
|
| 225 |
+
| Single-text throughput | 939 texts/s | 1,776 texts/s | 823 texts/s |
|
| 226 |
+
| Throughput (batch 8) | 6,103 texts/s | 4,954 texts/s | 5,709 texts/s |
|
| 227 |
+
| Peak VRAM | 85.1 MB | — | 105 MB |
|
| 228 |
+
|
| 229 |
+
## Comparison with other public safety classifiers
|
| 230 |
+
|
| 231 |
+
Same held-out test split (6,496 rows, threshold 0.5, GPU):
|
| 232 |
+
|
| 233 |
+
| Model | Params | Checkpoint | F1 | Latency | Throughput (b=8) |
|
| 234 |
+
| --- | --- | --- | --- | --- | --- |
|
| 235 |
+
| **Censer.6 (ours)** | **0.9M** | **3.6 MB** | **0.829** | **1.22 ms** | **1,960/s** |
|
| 236 |
+
| KucLab/Censer.5 | 0.9M | 3.6 MB | 0.849 | 1.27 ms | 1,978/s |
|
| 237 |
+
| KucLab/Censer.4 | 0.9M | 3.6 MB | 0.845 | 1.17 ms | 2,229/s |
|
| 238 |
+
| KucLab/Censer.1 | 1.1M | 4.5 MB | 0.836 | 1.43 ms | 1,853/s |
|
| 239 |
+
| KucLab/Censer.2 | 7.3M | 29.4 MB | 0.829 | 2.70 ms | 508/s |
|
| 240 |
+
| KucLab/Censer.3 | 14.4M | 57.7 MB | 0.822 | 4.27 ms | 263/s |
|
| 241 |
+
| unitary/unbiased-toxic-roberta | 124.7M | 498.7 MB | 0.741 | 14.7 ms | 67/s |
|
| 242 |
+
| unitary/toxic-bert | 109.5M | 876.0 MB | 0.681 | 14.2 ms | 68/s |
|
| 243 |
+
| martin-ha/toxic-comment-model | 67.0M | 267.9 MB | 0.566 | 7.2 ms | 133/s |
|
| 244 |
+
| Hate-speech-CNERG/bert-base-uncased-hatexplain | 109.5M | 438.0 MB | 0.543 | 14.7 ms | 67/s |
|
| 245 |
+
| Hate-speech-CNERG/dehatebert-mono-english | 167.4M | 669.5 MB | 0.252 | 14.3 ms | 68/s |
|
| 246 |
+
| mistralai/Shieldstral-1.0-3B | 3.85B | 7.7 GB | 0.318 | 51.6 ms | 24/s |
|
| 247 |
+
| keyword baseline | — | — | 0.094 | 0.03 ms | 34,784/s |
|
| 248 |
+
|
| 249 |
+

|
| 250 |
+

|
| 251 |
+

|
| 252 |
+
|
| 253 |
+
## Usage without transformers
|
| 254 |
+
|
| 255 |
+
The repository is self-contained: `model.safetensors`, `config.json`,
|
| 256 |
+
`tokenizer.json`, `censer.py`, `censer.6`. Only `torch`, `tokenizers`,
|
| 257 |
+
`safetensors` are needed:
|
| 258 |
+
|
| 259 |
+
```bash
|
| 260 |
+
hf download KucLab/Censer.6 --local-dir Censer.6
|
| 261 |
+
cd Censer.6
|
| 262 |
+
python censer.py "how to make a bomb at home" --json
|
| 263 |
+
# {"label": "1", "unsafe": true, "score": 0.99, "categories": ["weapons_explosives"]}
|
| 264 |
+
./censer.6 # interactive UI
|
| 265 |
+
```
|
| 266 |
+
|
| 267 |
+
Files:
|
| 268 |
+
|
| 269 |
+
| File | Purpose |
|
| 270 |
+
| --- | --- |
|
| 271 |
+
| `model.safetensors` | weights in standard Llama layout, 3.6 MB |
|
| 272 |
+
| `model_fp16.safetensors` | fp16 variant, 1.8 MB |
|
| 273 |
+
| `model_int8.safetensors` + `quant_report.json` | int8 per-tensor weights + scales, 0.9 MB |
|
| 274 |
+
| `censer6.onnx` | ONNX graph: input_ids -> 12 answer logits (dynamic batch/seq), 5.6 MB |
|
| 275 |
+
| `config.json` | standard `LlamaForSequenceClassification` config (multi-label, RoPE linear ×8) |
|
| 276 |
+
| `tokenizer.json` / `tokenizer_config.json` | byte-level BPE, prompt template baked in |
|
| 277 |
+
| `censer.py` | standalone inference incl. long-input chunking |
|
| 278 |
+
| `censer.6` | executable launcher for the interactive terminal UI |
|
| 279 |
+
| `serve.py` | batch HTTP API (stdlib only) |
|
| 280 |
+
| `Dockerfile` | CPU container serving the API |
|
| 281 |
+
|
| 282 |
+
## Limitations
|
| 283 |
+
|
| 284 |
+
- 902K parameters: nuance, sarcasm and long-tail cases are out of reach; the
|
| 285 |
+
test split itself contains label noise from scraped sources.
|
| 286 |
+
- Multilingual coverage is strongest for cs/de/es/fr (synthetic + curated
|
| 287 |
+
training data); other languages work via the byte-level tokenizer but with
|
| 288 |
+
lower accuracy.
|
| 289 |
+
- Categories are multi-label and independent — a text can fire several.
|
| 290 |
+
- Single-pass inference pools the last token, so in very long documents unsafe
|
| 291 |
+
content can be diluted — use the built-in chunked-max inference (default in
|
| 292 |
+
`censer.py` / the API) for long documents.
|
| 293 |
+
- A moderation signal, not a safety guarantee. For production, use the cascade
|
| 294 |
+
pattern: decide clear cases with Censer.6, escalate gray-zone / jailbreak
|
| 295 |
+
hits to a larger validator.
|
censer.6
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""censer.6 - run the Censer.6 safety classifier as an interactive terminal UI.
|
| 3 |
+
|
| 4 |
+
Usage:
|
| 5 |
+
censer.6 # interactive: type a phrase, Enter -> safe/unsafe + categories + score
|
| 6 |
+
censer.6 "how to make a bomb" # one-shot
|
| 7 |
+
censer.6 --help
|
| 8 |
+
|
| 9 |
+
Expects censer.py, model.safetensors, config.json and tokenizer.json next to
|
| 10 |
+
this file (they are found automatically, whatever the current directory is).
|
| 11 |
+
Get them all with:
|
| 12 |
+
|
| 13 |
+
hf download KucLab/Censer.6 --local-dir .
|
| 14 |
+
|
| 15 |
+
Requires: torch, tokenizers, safetensors (pip install torch tokenizers safetensors)
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
import os
|
| 19 |
+
import sys
|
| 20 |
+
|
| 21 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 22 |
+
sys.path.insert(0, HERE)
|
| 23 |
+
|
| 24 |
+
from censer import main
|
| 25 |
+
|
| 26 |
+
if __name__ == "__main__":
|
| 27 |
+
argv = sys.argv[1:]
|
| 28 |
+
if not any(a == "--model-dir" or a.startswith("--model-dir=") for a in argv):
|
| 29 |
+
argv = ["--model-dir", HERE] + argv
|
| 30 |
+
main(argv)
|
censer.py
ADDED
|
@@ -0,0 +1,352 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Inference for Censer.6 — self-contained, no dependency on the training repo.
|
| 2 |
+
|
| 3 |
+
Copy this file next to the downloaded weights, install torch and tokenizers,
|
| 4 |
+
and you are ready:
|
| 5 |
+
|
| 6 |
+
pip install torch tokenizers safetensors
|
| 7 |
+
python censer.py "how to make a bomb at home"
|
| 8 |
+
python censer.py # or just run it: interactive UI, phrase in -> verdict out
|
| 9 |
+
./censer.5 # same UI via the censer.5 launcher next to this file
|
| 10 |
+
|
| 11 |
+
It expects, in the same directory:
|
| 12 |
+
model.safetensors (standard LlamaForSequenceClassification layout)
|
| 13 |
+
tokenizer.json
|
| 14 |
+
config.json
|
| 15 |
+
"""
|
| 16 |
+
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
import argparse
|
| 20 |
+
import json
|
| 21 |
+
import math
|
| 22 |
+
import sys
|
| 23 |
+
from pathlib import Path
|
| 24 |
+
|
| 25 |
+
import torch
|
| 26 |
+
import torch.nn as nn
|
| 27 |
+
import torch.nn.functional as F
|
| 28 |
+
from tokenizers import Tokenizer
|
| 29 |
+
|
| 30 |
+
PAD, BOS, USER, ASSISTANT = "[PAD]", "[BOS]", "[USER]", "[ASSISTANT]"
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
class RMSNorm(nn.Module):
|
| 34 |
+
def __init__(self, dim: int, eps: float = 1e-5) -> None:
|
| 35 |
+
super().__init__()
|
| 36 |
+
self.eps = eps
|
| 37 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 38 |
+
|
| 39 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 40 |
+
dtype = x.dtype
|
| 41 |
+
x = x.float()
|
| 42 |
+
x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
|
| 43 |
+
return x.to(dtype) * self.weight
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def build_rope_cache(seq_len: int, head_dim: int, device, dtype, base: float = 10000.0, rope_scaling: dict | None = None):
|
| 47 |
+
half = head_dim // 2
|
| 48 |
+
freqs = 1.0 / (base ** (torch.arange(0, half, device=device, dtype=torch.float32) / half))
|
| 49 |
+
if rope_scaling and float(rope_scaling.get("factor", 1.0)) > 1.0:
|
| 50 |
+
freqs = freqs / float(rope_scaling["factor"]) # HF "linear" scaling: inv_freq /= factor
|
| 51 |
+
positions = torch.arange(seq_len, device=device, dtype=torch.float32)
|
| 52 |
+
angles = torch.outer(positions, freqs)
|
| 53 |
+
cos = torch.cat((angles.cos(), angles.cos()), dim=-1).to(dtype)[:, None, :]
|
| 54 |
+
sin = torch.cat((angles.sin(), angles.sin()), dim=-1).to(dtype)[:, None, :]
|
| 55 |
+
return cos, sin
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
def apply_rope(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
|
| 59 |
+
seq_len = x.shape[1]
|
| 60 |
+
cos, sin = cos[:seq_len], sin[:seq_len]
|
| 61 |
+
x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2 :]
|
| 62 |
+
rotated = torch.cat((-x2, x1), dim=-1)
|
| 63 |
+
return x * cos + rotated * sin
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
class Attention(nn.Module):
|
| 67 |
+
def __init__(self, cfg: dict) -> None:
|
| 68 |
+
super().__init__()
|
| 69 |
+
self.n_heads = cfg["n_heads"]
|
| 70 |
+
self.head_dim = cfg["d_model"] // cfg["n_heads"]
|
| 71 |
+
self.dropout = cfg["dropout"]
|
| 72 |
+
self.qkv = nn.Linear(cfg["d_model"], 3 * cfg["d_model"], bias=False)
|
| 73 |
+
self.proj = nn.Linear(cfg["d_model"], cfg["d_model"], bias=False)
|
| 74 |
+
|
| 75 |
+
def forward(self, x, cos, sin, mask):
|
| 76 |
+
b, t, _ = x.shape
|
| 77 |
+
q, k, v = self.qkv(x).split(x.shape[-1], dim=-1)
|
| 78 |
+
q = q.view(b, t, self.n_heads, self.head_dim)
|
| 79 |
+
k = k.view(b, t, self.n_heads, self.head_dim)
|
| 80 |
+
v = v.view(b, t, self.n_heads, self.head_dim)
|
| 81 |
+
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
|
| 82 |
+
out = F.scaled_dot_product_attention(
|
| 83 |
+
q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2),
|
| 84 |
+
attn_mask=mask,
|
| 85 |
+
dropout_p=self.dropout if self.training else 0.0,
|
| 86 |
+
is_causal=mask is None,
|
| 87 |
+
)
|
| 88 |
+
return self.proj(out.transpose(1, 2).reshape(b, t, -1))
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
class SwiGLU(nn.Module):
|
| 92 |
+
def __init__(self, cfg: dict) -> None:
|
| 93 |
+
super().__init__()
|
| 94 |
+
self.gate = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
|
| 95 |
+
self.up = nn.Linear(cfg["d_model"], cfg["d_ff"], bias=False)
|
| 96 |
+
self.down = nn.Linear(cfg["d_ff"], cfg["d_model"], bias=False)
|
| 97 |
+
|
| 98 |
+
def forward(self, x):
|
| 99 |
+
return self.down(F.silu(self.gate(x)) * self.up(x))
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
class Block(nn.Module):
|
| 103 |
+
def __init__(self, cfg: dict) -> None:
|
| 104 |
+
super().__init__()
|
| 105 |
+
self.norm1 = RMSNorm(cfg["d_model"])
|
| 106 |
+
self.attn = Attention(cfg)
|
| 107 |
+
self.norm2 = RMSNorm(cfg["d_model"])
|
| 108 |
+
self.mlp = SwiGLU(cfg)
|
| 109 |
+
self.dropout = nn.Dropout(cfg["dropout"])
|
| 110 |
+
|
| 111 |
+
def forward(self, x, cos, sin, mask):
|
| 112 |
+
x = x + self.dropout(self.attn(self.norm1(x), cos, sin, mask))
|
| 113 |
+
x = x + self.dropout(self.mlp(self.norm2(x)))
|
| 114 |
+
return x
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
class Censer(nn.Module):
|
| 118 |
+
def __init__(self, cfg: dict) -> None:
|
| 119 |
+
super().__init__()
|
| 120 |
+
self.cfg = cfg
|
| 121 |
+
self.register_buffer(
|
| 122 |
+
"answer_id_tensor", torch.tensor(cfg["answer_ids"], dtype=torch.long), persistent=False
|
| 123 |
+
)
|
| 124 |
+
self.embed = nn.Embedding(cfg["vocab_size"], cfg["d_model"], padding_idx=cfg["pad_id"])
|
| 125 |
+
self.blocks = nn.ModuleList([Block(cfg) for _ in range(cfg["n_layers"])])
|
| 126 |
+
self.norm = RMSNorm(cfg["d_model"])
|
| 127 |
+
self.lm_head = nn.Linear(cfg["d_model"], cfg["vocab_size"], bias=False)
|
| 128 |
+
if cfg["tie_embeddings"]:
|
| 129 |
+
self.lm_head.weight = self.embed.weight
|
| 130 |
+
self._rope_cache = None
|
| 131 |
+
|
| 132 |
+
def _rope(self, device, dtype):
|
| 133 |
+
key = f"{device}|{dtype}"
|
| 134 |
+
if self._rope_cache is None or self._rope_cache[2] != key:
|
| 135 |
+
head_dim = self.cfg["d_model"] // self.cfg["n_heads"]
|
| 136 |
+
cos, sin = build_rope_cache(self.cfg["max_seq_len"], head_dim, device, dtype, rope_scaling=self.cfg.get("rope_scaling"))
|
| 137 |
+
self._rope_cache = (cos, sin, key)
|
| 138 |
+
return self._rope_cache[0], self._rope_cache[1]
|
| 139 |
+
|
| 140 |
+
def _attention_mask(self, input_ids):
|
| 141 |
+
b, t = input_ids.shape
|
| 142 |
+
pad = input_ids != self.cfg["pad_id"]
|
| 143 |
+
causal = torch.ones(t, t, dtype=torch.bool, device=input_ids.device).tril()
|
| 144 |
+
return pad[:, None, None, :] & causal[None, None, :, :]
|
| 145 |
+
|
| 146 |
+
def forward(self, input_ids):
|
| 147 |
+
cos, sin = self._rope(input_ids.device, self.embed.weight.dtype)
|
| 148 |
+
mask = self._attention_mask(input_ids)
|
| 149 |
+
x = self.embed(input_ids)
|
| 150 |
+
for block in self.blocks:
|
| 151 |
+
x = block(x, cos, sin, mask)
|
| 152 |
+
return self.lm_head(self.norm(x))
|
| 153 |
+
|
| 154 |
+
@torch.no_grad()
|
| 155 |
+
def classify_probs(self, input_ids):
|
| 156 |
+
"""Answer-token probabilities: softmax for binary heads, sigmoid for multi-label."""
|
| 157 |
+
hidden = self.forward(input_ids)
|
| 158 |
+
non_pad = (input_ids != self.cfg["pad_id"]).sum(dim=1).clamp(min=1)
|
| 159 |
+
idx = (non_pad - 1)[:, None, None].expand(-1, 1, hidden.shape[-1])
|
| 160 |
+
logits = hidden.gather(1, idx)[:, 0, :].float()
|
| 161 |
+
logits = logits.index_select(-1, self.answer_id_tensor)
|
| 162 |
+
if logits.shape[-1] == 2:
|
| 163 |
+
return logits.softmax(-1)
|
| 164 |
+
return logits.sigmoid()
|
| 165 |
+
|
| 166 |
+
@torch.no_grad()
|
| 167 |
+
def classify(self, input_ids):
|
| 168 |
+
return self.classify_probs(input_ids)[:, 1]
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
def remap_hf_state(state: dict) -> dict:
|
| 172 |
+
"""Standard Llama-layout safetensors -> training-layout state dict."""
|
| 173 |
+
import re
|
| 174 |
+
|
| 175 |
+
out = {"embed.weight": state["model.embed_tokens.weight"]}
|
| 176 |
+
layer_ids = sorted({int(m.group(1)) for k in state if (m := re.match(r"model\.layers\.(\d+)\.", k))})
|
| 177 |
+
for i in layer_ids:
|
| 178 |
+
p, b = f"model.layers.{i}.", f"blocks.{i}."
|
| 179 |
+
qkv = torch.cat(
|
| 180 |
+
[state[p + "self_attn.q_proj.weight"], state[p + "self_attn.k_proj.weight"], state[p + "self_attn.v_proj.weight"]],
|
| 181 |
+
dim=0,
|
| 182 |
+
)
|
| 183 |
+
out[b + "attn.qkv.weight"] = qkv.contiguous()
|
| 184 |
+
out[b + "attn.proj.weight"] = state[p + "self_attn.o_proj.weight"]
|
| 185 |
+
out[b + "mlp.gate.weight"] = state[p + "mlp.gate_proj.weight"]
|
| 186 |
+
out[b + "mlp.up.weight"] = state[p + "mlp.up_proj.weight"]
|
| 187 |
+
out[b + "mlp.down.weight"] = state[p + "mlp.down_proj.weight"]
|
| 188 |
+
out[b + "norm1.weight"] = state[p + "input_layernorm.weight"]
|
| 189 |
+
out[b + "norm2.weight"] = state[p + "post_attention_layernorm.weight"]
|
| 190 |
+
out["norm.weight"] = state["model.norm.weight"]
|
| 191 |
+
return out
|
| 192 |
+
|
| 193 |
+
|
| 194 |
+
def internal_cfg(cfg: dict) -> dict:
|
| 195 |
+
"""config.json (Llama layout) -> the dict the Censer module expects."""
|
| 196 |
+
return {
|
| 197 |
+
"vocab_size": cfg["vocab_size"],
|
| 198 |
+
"n_layers": cfg["num_hidden_layers"],
|
| 199 |
+
"n_heads": cfg["num_attention_heads"],
|
| 200 |
+
"d_model": cfg["hidden_size"],
|
| 201 |
+
"d_ff": cfg["intermediate_size"],
|
| 202 |
+
"max_seq_len": cfg["max_position_embeddings"],
|
| 203 |
+
"dropout": cfg.get("dropout", 0.0),
|
| 204 |
+
"tie_embeddings": True,
|
| 205 |
+
"pad_id": cfg.get("pad_token_id", 0),
|
| 206 |
+
"answer_ids": cfg["answer_ids"],
|
| 207 |
+
"max_chars": cfg.get("max_chars", 4000),
|
| 208 |
+
"rope_scaling": cfg.get("rope_scaling") or (cfg.get("rope_parameters") if cfg.get("rope_parameters", {}).get("rope_type") == "linear" else None),
|
| 209 |
+
"categories": [cfg["id2label"][str(i)] for i in range(2, cfg.get("num_labels", 2))] if cfg.get("num_labels", 2) > 2 else [],
|
| 210 |
+
}
|
| 211 |
+
|
| 212 |
+
|
| 213 |
+
class CenserPipeline:
|
| 214 |
+
"""Text in, "1" (unsafe) or "0" (safe) out."""
|
| 215 |
+
|
| 216 |
+
def __init__(self, directory: str | Path = ".", threshold: float = 0.5, device: str | None = None) -> None:
|
| 217 |
+
from safetensors.torch import load_file
|
| 218 |
+
|
| 219 |
+
directory = Path(directory)
|
| 220 |
+
self.cfg = internal_cfg(json.loads((directory / "config.json").read_text(encoding="utf-8")))
|
| 221 |
+
self.tok = Tokenizer.from_file(str(directory / "tokenizer.json"))
|
| 222 |
+
self.pad_id = self.tok.token_to_id(PAD)
|
| 223 |
+
self.bos_id = self.tok.token_to_id(BOS)
|
| 224 |
+
self.user_id = self.tok.token_to_id(USER)
|
| 225 |
+
self.assistant_id = self.tok.token_to_id(ASSISTANT)
|
| 226 |
+
self.threshold = threshold
|
| 227 |
+
self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
|
| 228 |
+
self.model = Censer(self.cfg)
|
| 229 |
+
state = remap_hf_state(load_file(directory / "model.safetensors"))
|
| 230 |
+
missing, unexpected = self.model.load_state_dict(state, strict=False)
|
| 231 |
+
missing = [k for k in missing if k != "lm_head.weight"] # tied to embed.weight
|
| 232 |
+
if missing or unexpected:
|
| 233 |
+
raise RuntimeError(f"weight mismatch: missing={missing} unexpected={unexpected}")
|
| 234 |
+
self.model.to(self.device).eval()
|
| 235 |
+
self.max_chars = self.cfg["max_chars"]
|
| 236 |
+
|
| 237 |
+
def _encode(self, text: str) -> list[int]:
|
| 238 |
+
body = self.tok.encode(text[: self.max_chars], add_special_tokens=False).ids
|
| 239 |
+
ids = [self.bos_id, self.user_id, *body, self.assistant_id]
|
| 240 |
+
return ids[: self.cfg["max_seq_len"]]
|
| 241 |
+
|
| 242 |
+
def predict(self, text: str) -> dict:
|
| 243 |
+
return self.predict_batch([text])[0]
|
| 244 |
+
|
| 245 |
+
def _encode_full(self, text: str) -> list[int]:
|
| 246 |
+
body = self.tok.encode(text[: self.max_chars * 8], add_special_tokens=False).ids
|
| 247 |
+
return [self.bos_id, self.user_id, *body, self.assistant_id]
|
| 248 |
+
|
| 249 |
+
def predict_batch(self, texts: list[str]) -> list[dict]:
|
| 250 |
+
# long inputs: split into overlapping token chunks, aggregate by max
|
| 251 |
+
out: list[dict] = []
|
| 252 |
+
chunk_tokens, stride = 256, 192
|
| 253 |
+
for text in texts:
|
| 254 |
+
ids = self._encode_full(text)
|
| 255 |
+
body = ids[2:-1]
|
| 256 |
+
if len(body) <= chunk_tokens:
|
| 257 |
+
out.append(self._classify_ids([ids[: self.cfg["max_seq_len"]]])[0])
|
| 258 |
+
continue
|
| 259 |
+
chunks = []
|
| 260 |
+
start = 0
|
| 261 |
+
while start < len(body):
|
| 262 |
+
chunks.append(body[start:start + chunk_tokens])
|
| 263 |
+
if start + chunk_tokens >= len(body):
|
| 264 |
+
break
|
| 265 |
+
start += stride
|
| 266 |
+
rows = []
|
| 267 |
+
for piece in chunks:
|
| 268 |
+
cids = [self.bos_id, self.user_id, *piece, self.assistant_id]
|
| 269 |
+
rows.append(self._classify_ids([cids[: self.cfg["max_seq_len"]]])[0])
|
| 270 |
+
best = max(rows, key=lambda r: r["score"])
|
| 271 |
+
# aggregate categories by max prob
|
| 272 |
+
n_cat = len(rows[0].get("category_scores", []))
|
| 273 |
+
cat_scores = [max(r["category_scores"][i] for r in rows) for i in range(n_cat)] if n_cat else []
|
| 274 |
+
best["category_scores"] = [round(c, 4) for c in cat_scores]
|
| 275 |
+
best["categories"] = [c for c, p in zip(self.categories(), cat_scores) if p >= self.threshold]
|
| 276 |
+
best["chunks"] = len(chunks)
|
| 277 |
+
out.append(best)
|
| 278 |
+
return out
|
| 279 |
+
|
| 280 |
+
def categories(self) -> list[str]:
|
| 281 |
+
return self.cfg.get("categories") or []
|
| 282 |
+
|
| 283 |
+
def _classify_ids(self, seqs: list[list[int]]) -> list[dict]:
|
| 284 |
+
width = max(len(s) for s in seqs)
|
| 285 |
+
ids = torch.full((len(seqs), width), self.pad_id, dtype=torch.long)
|
| 286 |
+
for i, seq in enumerate(seqs):
|
| 287 |
+
ids[i, : len(seq)] = torch.tensor(seq)
|
| 288 |
+
probs = self.model.classify_probs(ids.to(self.device)).float().cpu().tolist()
|
| 289 |
+
res = []
|
| 290 |
+
for row in probs:
|
| 291 |
+
p = row[1]
|
| 292 |
+
cats = self.categories()
|
| 293 |
+
cat_scores = row[2:] if cats else []
|
| 294 |
+
res.append({
|
| 295 |
+
"label": "1" if p >= self.threshold else "0",
|
| 296 |
+
"unsafe": p >= self.threshold,
|
| 297 |
+
"score": round(p, 4),
|
| 298 |
+
"category_scores": [round(c, 4) for c in cat_scores],
|
| 299 |
+
"categories": [c for c, s in zip(cats, cat_scores) if s >= self.threshold],
|
| 300 |
+
})
|
| 301 |
+
return res
|
| 302 |
+
|
| 303 |
+
|
| 304 |
+
def _color(text: str, code: str) -> str:
|
| 305 |
+
if not sys.stdout.isatty():
|
| 306 |
+
return text
|
| 307 |
+
return f"\033[{code}m{text}\033[0m"
|
| 308 |
+
|
| 309 |
+
|
| 310 |
+
def repl(pipe: "CenserPipeline") -> int:
|
| 311 |
+
"""Small interactive terminal UI: phrase in, safe/unsafe + score out."""
|
| 312 |
+
print("Censer.6 - interactive terminal UI")
|
| 313 |
+
print("Type a phrase, Enter to classify. 'exit' to quit.\n")
|
| 314 |
+
while True:
|
| 315 |
+
try:
|
| 316 |
+
line = input("> ")
|
| 317 |
+
except (EOFError, KeyboardInterrupt):
|
| 318 |
+
print()
|
| 319 |
+
return 0
|
| 320 |
+
text = line.strip()
|
| 321 |
+
if text.lower() in {"exit", "quit", ":q"}:
|
| 322 |
+
return 0
|
| 323 |
+
if not text:
|
| 324 |
+
continue
|
| 325 |
+
v = pipe.predict(text)
|
| 326 |
+
label = "unsafe" if v["unsafe"] else "safe"
|
| 327 |
+
colored = _color(f"{label:6s}", "31" if v["unsafe"] else "32")
|
| 328 |
+
print(f"{colored} {v['score']:.4f} | {text}")
|
| 329 |
+
|
| 330 |
+
|
| 331 |
+
def main(argv: list[str] | None = None) -> None:
|
| 332 |
+
parser = argparse.ArgumentParser(
|
| 333 |
+
prog="censer.5",
|
| 334 |
+
description="Censer.6 safety classifier (standalone). Run without text to open the interactive UI.",
|
| 335 |
+
)
|
| 336 |
+
parser.add_argument("text", nargs="*", help="phrase to classify once; omit to open the interactive UI")
|
| 337 |
+
parser.add_argument("--model-dir", default=".")
|
| 338 |
+
parser.add_argument("--threshold", type=float, default=0.5)
|
| 339 |
+
parser.add_argument("--json", action="store_true")
|
| 340 |
+
parser.add_argument("--repl", action="store_true", help="interactive UI (default when no text is given)")
|
| 341 |
+
args = parser.parse_args(argv)
|
| 342 |
+
|
| 343 |
+
pipe = CenserPipeline(args.model_dir, args.threshold)
|
| 344 |
+
text = " ".join(args.text).strip()
|
| 345 |
+
if args.repl or not text:
|
| 346 |
+
raise SystemExit(repl(pipe))
|
| 347 |
+
verdict = pipe.predict(text)
|
| 348 |
+
print(json.dumps(verdict) if args.json else verdict["label"])
|
| 349 |
+
|
| 350 |
+
|
| 351 |
+
if __name__ == "__main__":
|
| 352 |
+
main()
|
censer6.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:74f3c4ee3ac3e00b3e3f467e795310c91d23d23fc870a47887fc4d75850ec393
|
| 3 |
+
size 5778265
|
config.json
ADDED
|
@@ -0,0 +1,84 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"LlamaForSequenceClassification"
|
| 4 |
+
],
|
| 5 |
+
"model_type": "llama",
|
| 6 |
+
"num_labels": 12,
|
| 7 |
+
"id2label": {
|
| 8 |
+
"0": "safe",
|
| 9 |
+
"1": "unsafe",
|
| 10 |
+
"2": "violence",
|
| 11 |
+
"3": "weapons_explosives",
|
| 12 |
+
"4": "drugs",
|
| 13 |
+
"5": "hate_harassment",
|
| 14 |
+
"6": "sexual_abuse",
|
| 15 |
+
"7": "self_harm",
|
| 16 |
+
"8": "cybercrime_fraud",
|
| 17 |
+
"9": "jailbreak_injection",
|
| 18 |
+
"10": "illegal_marketplace",
|
| 19 |
+
"11": "other_unsafe"
|
| 20 |
+
},
|
| 21 |
+
"label2id": {
|
| 22 |
+
"safe": 0,
|
| 23 |
+
"unsafe": 1,
|
| 24 |
+
"violence": 2,
|
| 25 |
+
"weapons_explosives": 3,
|
| 26 |
+
"drugs": 4,
|
| 27 |
+
"hate_harassment": 5,
|
| 28 |
+
"sexual_abuse": 6,
|
| 29 |
+
"self_harm": 7,
|
| 30 |
+
"cybercrime_fraud": 8,
|
| 31 |
+
"jailbreak_injection": 9,
|
| 32 |
+
"illegal_marketplace": 10,
|
| 33 |
+
"other_unsafe": 11
|
| 34 |
+
},
|
| 35 |
+
"problem_type": "multi_label_classification",
|
| 36 |
+
"vocab_size": 2048,
|
| 37 |
+
"hidden_size": 128,
|
| 38 |
+
"intermediate_size": 384,
|
| 39 |
+
"num_hidden_layers": 3,
|
| 40 |
+
"num_attention_heads": 4,
|
| 41 |
+
"num_key_value_heads": 4,
|
| 42 |
+
"head_dim": 32,
|
| 43 |
+
"max_position_embeddings": 4096,
|
| 44 |
+
"rms_norm_eps": 1e-05,
|
| 45 |
+
"rope_parameters": {
|
| 46 |
+
"rope_type": "linear",
|
| 47 |
+
"factor": 8.0,
|
| 48 |
+
"rope_theta": 10000.0,
|
| 49 |
+
"original_max_position_embeddings": 512
|
| 50 |
+
},
|
| 51 |
+
"rope_scaling": {
|
| 52 |
+
"rope_type": "linear",
|
| 53 |
+
"factor": 8.0
|
| 54 |
+
},
|
| 55 |
+
"attention_bias": false,
|
| 56 |
+
"mlp_bias": false,
|
| 57 |
+
"tie_word_embeddings": true,
|
| 58 |
+
"pad_token_id": 0,
|
| 59 |
+
"bos_token_id": 1,
|
| 60 |
+
"eos_token_id": 3,
|
| 61 |
+
"name": "Censer.6",
|
| 62 |
+
"max_chars": 4000,
|
| 63 |
+
"answer_ids": [
|
| 64 |
+
19,
|
| 65 |
+
20,
|
| 66 |
+
68,
|
| 67 |
+
69,
|
| 68 |
+
70,
|
| 69 |
+
71,
|
| 70 |
+
72,
|
| 71 |
+
73,
|
| 72 |
+
74,
|
| 73 |
+
75,
|
| 74 |
+
76,
|
| 75 |
+
77
|
| 76 |
+
],
|
| 77 |
+
"special_tokens": [
|
| 78 |
+
"[PAD]",
|
| 79 |
+
"[BOS]",
|
| 80 |
+
"[USER]",
|
| 81 |
+
"[ASSISTANT]"
|
| 82 |
+
],
|
| 83 |
+
"threshold": 0.5
|
| 84 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:76c59e7055b052a60a05fbe377dfc019df644575c4ec5b632be96023372ebe60
|
| 3 |
+
size 3617384
|
model_fp16.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8667eddd3cd57162b5916cbda987024aec220051aed619990de36c52a73e5184
|
| 3 |
+
size 1810256
|
model_int8.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ad8c4273f36618a08986fbcd6bfb0e3380887333f09904f820cda33c108bbaaa
|
| 3 |
+
size 911740
|
quant_report.json
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_safetensors_mb": 3.62,
|
| 3 |
+
"model_fp16_mb": 1.81,
|
| 4 |
+
"model_int8_mb": 0.91,
|
| 5 |
+
"max_rel_roundtrip_error": 0.003937006462365389
|
| 6 |
+
}
|
serve.py
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Batch classification API for the Censer.6 Hub bundle (stdlib only, no FastAPI).
|
| 2 |
+
|
| 3 |
+
python scripts/serve.py --port 8000 # serves the local HF bundle dir
|
| 4 |
+
CENSER_DIR=/path/to/bundle python scripts/serve.py
|
| 5 |
+
|
| 6 |
+
Endpoints:
|
| 7 |
+
GET /health -> {"status": "ok", "model": ...}
|
| 8 |
+
POST /classify -> {"texts": [...]} => {"results": [{label, unsafe, score, categories, chunks}]}
|
| 9 |
+
POST /classify-one -> {"text": "..."} => single result
|
| 10 |
+
|
| 11 |
+
Features: batching, long-input chunking (chunk_tokens/stride), token-bucket
|
| 12 |
+
rate limit (env RATE_LIMIT_PER_MIN, default 600), max body size guard.
|
| 13 |
+
|
| 14 |
+
.venv/bin/python scripts/serve.py --dir bundle --port 8000
|
| 15 |
+
"""
|
| 16 |
+
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
import argparse
|
| 20 |
+
import json
|
| 21 |
+
import os
|
| 22 |
+
import time
|
| 23 |
+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 24 |
+
from pathlib import Path
|
| 25 |
+
|
| 26 |
+
MAX_BODY = 2 * 1024 * 1024 # 2 MB
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
class RateLimiter:
|
| 30 |
+
def __init__(self, per_min: float) -> None:
|
| 31 |
+
self.per_min = per_min
|
| 32 |
+
self.tokens = per_min
|
| 33 |
+
self.updated = time.monotonic()
|
| 34 |
+
self.last_deny = 0.0
|
| 35 |
+
|
| 36 |
+
def allow(self, cost: float = 1.0) -> bool:
|
| 37 |
+
now = time.monotonic()
|
| 38 |
+
self.tokens = min(self.per_min, self.tokens + (now - self.updated) * self.per_min / 60.0)
|
| 39 |
+
self.updated = now
|
| 40 |
+
if self.tokens >= cost:
|
| 41 |
+
self.tokens -= cost
|
| 42 |
+
return True
|
| 43 |
+
return False
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def make_handler(pipe, limiter: RateLimiter, model_name: str):
|
| 47 |
+
class Handler(BaseHTTPRequestHandler):
|
| 48 |
+
def _json(self, code: int, payload: dict) -> None:
|
| 49 |
+
body = json.dumps(payload, ensure_ascii=False).encode("utf-8")
|
| 50 |
+
self.send_response(code)
|
| 51 |
+
self.send_header("Content-Type", "application/json; charset=utf-8")
|
| 52 |
+
self.send_header("Content-Length", str(len(body)))
|
| 53 |
+
self.end_headers()
|
| 54 |
+
self.wfile.write(body)
|
| 55 |
+
|
| 56 |
+
def log_message(self, fmt, *args): # quiet
|
| 57 |
+
pass
|
| 58 |
+
|
| 59 |
+
def do_GET(self):
|
| 60 |
+
if self.path == "/health":
|
| 61 |
+
self._json(200, {"status": "ok", "model": model_name, "device": str(pipe.device)})
|
| 62 |
+
else:
|
| 63 |
+
self._json(404, {"error": "not found"})
|
| 64 |
+
|
| 65 |
+
def do_POST(self):
|
| 66 |
+
if not limiter.allow():
|
| 67 |
+
self._json(429, {"error": "rate limit exceeded"})
|
| 68 |
+
return
|
| 69 |
+
length = int(self.headers.get("Content-Length") or 0)
|
| 70 |
+
if length > MAX_BODY:
|
| 71 |
+
self._json(413, {"error": "body too large"})
|
| 72 |
+
return
|
| 73 |
+
try:
|
| 74 |
+
payload = json.loads(self.rfile.read(length) or b"{}")
|
| 75 |
+
except Exception:
|
| 76 |
+
self._json(400, {"error": "invalid json"})
|
| 77 |
+
return
|
| 78 |
+
if self.path == "/classify-one":
|
| 79 |
+
text = str(payload.get("text", ""))
|
| 80 |
+
self._json(200, pipe.predict(text))
|
| 81 |
+
elif self.path == "/classify":
|
| 82 |
+
texts = payload.get("texts")
|
| 83 |
+
if not isinstance(texts, list) or not texts or len(texts) > 256:
|
| 84 |
+
self._json(400, {"error": "texts must be a list of 1..256 strings"})
|
| 85 |
+
return
|
| 86 |
+
self._json(200, {"results": pipe.predict_batch([str(t) for t in texts])})
|
| 87 |
+
else:
|
| 88 |
+
self._json(404, {"error": "not found"})
|
| 89 |
+
|
| 90 |
+
return Handler
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def main() -> None:
|
| 94 |
+
parser = argparse.ArgumentParser()
|
| 95 |
+
parser.add_argument("--dir", default=os.environ.get("CENSER_DIR", "bundle"))
|
| 96 |
+
parser.add_argument("--port", type=int, default=8000)
|
| 97 |
+
parser.add_argument("--host", default="0.0.0.0")
|
| 98 |
+
parser.add_argument("--device", default=None)
|
| 99 |
+
args = parser.parse_args()
|
| 100 |
+
|
| 101 |
+
import sys
|
| 102 |
+
|
| 103 |
+
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
| 104 |
+
from censer import CenserPipeline
|
| 105 |
+
|
| 106 |
+
pipe = CenserPipeline(args.dir, device=args.device)
|
| 107 |
+
limiter = RateLimiter(float(os.environ.get("RATE_LIMIT_PER_MIN", "600")))
|
| 108 |
+
server = ThreadingHTTPServer((args.host, args.port), make_handler(pipe, limiter, args.dir))
|
| 109 |
+
print(f"censer serve: model={args.dir} device={pipe.device} on http://{args.host}:{args.port}")
|
| 110 |
+
print(" POST /classify {\"texts\": [...]} POST /classify-one {\"text\": ...} GET /health")
|
| 111 |
+
server.serve_forever()
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
if __name__ == "__main__":
|
| 115 |
+
main()
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 3 |
+
"pad_token": "[PAD]",
|
| 4 |
+
"bos_token": "[BOS]",
|
| 5 |
+
"eos_token": "[ASSISTANT]"
|
| 6 |
+
}
|