README.md

---
license: apache-2.0
co2_eq_emissions:
  emissions: 50
  source: estimated from instance power draw, runtime, and grid intensity
  training_type: pretraining
  hardware_used: 1 AWS c7i CPU instance, about 2 hours, CPU only
tags:
  - fela
  - fourier-neural-operator
  - fno
  - gated-linear-attention
  - cpu
  - on-device
  - content-moderation
  - toxicity
  - pii
  - byte-level
library_name: transformers
pipeline_tag: text-classification
---

> **N.B.** To use this model with a convenient interface [check us out on Github](https://github.com/Lowdown-Labs/fela_moderator)!

# FELA-Moderation: CPU native content moderation and PII detection

FELA-Moderation reads a piece of text and does two things with it. It scores how toxic the text
is across a set of moderation categories, and it points out the spans that look like personal
information: names, emails, phone numbers, credit cards, and the like. It is small enough to run
on a plain CPU, so that whole moderation and PII pass can live inside your own service or app
instead of going out to a moderation API and billing you per call.

It reads text as raw bytes rather than words. That has a nice side effect: there is no
vocabulary file to ship, and the model is not locked to any one language.

The model covers a 19 category moderation taxonomy, the 6 native Jigsaw labels, byte level PII over
56 entity types, and four extra classifier heads for spam, jailbreak, NSFW severity, and targeted
identity. It is about 18.9M parameters and ships as `model.safetensors` (fp32) plus an int8 export
of roughly 19 MB for on device and WASM serving. We ship weights only and never redistribute the
source text.

# What goes in, what comes out

- Input: any UTF-8 text string. It is byte tokenized to `[CLS] b0 b1 ...` (each byte is its own
  token, ids 0 to 255, plus 256=PAD, 257=CLS, 258=SEP) and truncated to 512 tokens. The
  `encode_text` and `pad_batch` helpers in `modeling.py` do this for you.
- Output 1 (taxonomy): 19 independent probabilities over the moderation taxonomy (the 11 OpenAI
  aligned categories plus identity_attack, insult, profanity_obscene, threat, sexual_explicit,
  severe_toxicity, dehumanize, incitement_violence). Full list in `config.json`. Sigmoid, multi
  label.
- Output 2 (jigsaw): 6 independent probabilities over the native Jigsaw labels (toxic,
  severe_toxic, obscene, threat, insult, identity_hate). Sigmoid, multi label.
- Output 3 (pii): per token BIO tags over 56 entity types (113 tags: O plus B- and I- per entity).
- Output 4 (safety heads): spam / scam / phishing, jailbreak / prompt injection, NSFW severity, and
  targeted identity.

The head is chosen with `task=` in the forward call (`taxonomy`, `jigsaw`, `pii`, `spam`,
`jailbreak`, `nsfw`, `identity`, or `both`).

# Why we built it this way

The model is built on the same Fourier Neural Operator the rest of the FELA family uses: a
filter that mixes the whole sequence at once in the frequency domain, with no attention matrix
that balloons as the text gets longer.

Alongside it rides a gated linear attention layer that picks up the content dependent details, and
its working memory stays a fixed size. Because none of this leans on the usual all pairs attention,
a moderation plus PII pass stays fast on an ordinary CPU. That speed is the whole cost argument
against paying a per call API.

It is deliberately small, about 18.9M parameters. One shared backbone feeds several light heads
(taxonomy, Jigsaw, PII, spam, jailbreak, NSFW, identity), and the byte vocabulary is only 259
symbols, which quantizes to int8 well.

# Performance

Format and footprint. The fp32 weights ship as `model.safetensors` (about 75 MB). INT8 dynamic
quantization of the Linear layers shrinks the model to roughly 19 MB while the FFT and GLA einsum
paths stay in float; the int8 export holds accuracy on the held out slices (see below). Because the
encoder carries no KV cache and a fixed size state, working memory does not grow with input length.

# Accuracy

Held out numbers are on permissively licensed splits: Jigsaw on a fixed leak free slice of the
Jigsaw training data, Civil Comments validation for taxonomy, and nemotron plus synthetic for PII.
Headline held out:

| head | AUROC / acc | note |
|---|---|---|
| PII (byte token acc) | 0.96 | |
| Jigsaw (mean AUROC) | 0.94 | |
| taxonomy: hate / harassment | 0.84 / 0.77 | |
| taxonomy: sexual | 0.67 | data limited, explicit data not in the training set |
| spam / jailbreak / nsfw | 0.94 / 0.88 / 0.82 | |
| identity | 0.64 | weaker, use as a soft signal only |

The int8 export holds these numbers. Final training loss 0.69. The model's advantage is privacy on
local hardware, speed, and clean provenance at roughly comparable quality, not topping a specialist
leaderboard.

The one honest gap is the sexual category (0.67). It does not close with more training: it is a data
limitation. Explicit sexual data was deliberately kept out of the training set, so that category is
supplied only by synthetic examples. We report it rather than hide it.

# How to run it

```python
import torch
from modeling import load_model, encode_text, pad_batch

m = load_model("model.safetensors")
ids, _ = encode_text("You are awful. Email me at jane.doe@example.com", m.cfg.max_len)
input_ids, mask = pad_batch([ids], m.cfg.max_len)
out = m(input_ids, mask, task="both")

tax = torch.sigmoid(out["taxonomy"][0])   # 19 taxonomy probabilities
jig = torch.sigmoid(out["jigsaw"][0])     # 6 Jigsaw probabilities
pii = out["pii"][0].argmax(-1)            # per token BIO tag id
```

## Loading with standard tooling

The repo ships `config.json` (architecture hyperparameters plus the full label and tag maps and the
per category toxicity thresholds) and a self contained `modeling.py` with a `load_model` /
`from_pretrained` entry point:

```python
from modeling import load_model
m = load_model("/path/to/weights_dir")  # OR
m = load_model("lowdown-labs/fela-moderator")
```

The fp32 weights are shipped as `model.safetensors`.

## Serving artifacts

- `model.safetensors` plus `config.json` for the safetensors load path (fp32).
- `tier_full_int8.safetensors` plus `tier_full_scales.json` for the roughly 19 MB int8 export.
- `manifest.json` and `NOTICE.txt` describe the tiers and the training source attributions.
- `verify.py` runs a fixed sample input and checks the output shapes and a verification value.

For serving at scale, use the separate CPU native FELA server
(`https://github.com/Lowdown-Labs/fela_server`). It runs this model on CPU with no GPU required.

# Training data

We train on the sources below and distribute model weights only. No source text or derived dataset
is redistributed, so the CC-BY-SA-3.0 status of Wikipedia derived comment text (under Jigsaw and
Civil Comments) never triggers share alike. `data/licenses.py` carries the verified license of every
source, and `NOTICE.txt` records the attributions.

- PII: `nvidia/nemotron-pii` (CC-BY-4.0, 55+ types, character offset spans) plus
  `gretelai/synthetic_pii_finance_multilingual` (Apache-2.0) plus synthetic generation via
  Faker (MIT) and the Presidio generator (MIT, driven from Faker).
- Toxicity and taxonomy: `google/civil_comments` (CC0-1.0), `google/jigsaw_toxicity_pred` (CC0-1.0
  annotations), `ucberkeley-dlab/measuring-hate-speech` (CC-BY-4.0),
  `allenai/real-toxicity-prompts` (Apache-2.0).
- Spam: `redasers/difraud` (MIT) and `ucirvine/sms_spam` (CC-BY-4.0).
- Jailbreak and prompt injection: `cyberec/llm-prompt-injection-attacks` (Apache-2.0),
  `deepset/prompt-injections` (Apache-2.0), `jackhhao/jailbreak-classification` (Apache-2.0),
  `Lakera/gandalf_ignore_instructions` (MIT).
- NSFW: `google/civil_comments` sexual_explicit signal, with
  `mmathys/openai-moderation-api-evaluation` (MIT) used for evaluation only.
- Targeted identity: `ucberkeley-dlab/measuring-hate-speech` (CC-BY-4.0).

Full split definitions, size assertions, and per source license verdicts are reproduced in
`train.py` and `data/licenses.py`.

**Expanded taxonomy (19).** Beyond the 11 OpenAI aligned categories the model adds identity_attack,
insult, profanity_obscene, threat, sexual_explicit, severe_toxicity, dehumanize, and
incitement_violence, each sourced from a permissive dataset. Growing the head does not change the
byte level backbone, so the int8 size is unchanged. Five categories (self_harm, self_harm_intent,
self_harm_instructions, sexual_minors, violence_graphic) have no permissive source: they are masked
from real data and supplied by synthetic and reviewed examples, and reported separately so coverage
is never overclaimed.

**Byte level PII BIO with exact spans.** Character offsets from the real sources and byte offsets
from the synthetic generator are aligned to byte tokens in `data/bio.py` (verified 100% correct on
1,251 spans). The head is trained across context lengths (128 to 8192 bytes) with PII needles buried
in benign carrier text, so it finds a single email or one toxic span inside a long document.

# How it was trained

The model was trained CPU only via DiLoCo (data parallel with infrequent parameter averaging
coordinated through S3), on commodity CPU instances with no GPU. Weights checkpoint to S3
periodically, so a spot interruption resumes rather than restarts. The taxonomy and Jigsaw heads
train every step; the spam, jailbreak, NSFW, and identity heads rotate. Run
`python train.py --smoke` for an offline end to end check that trains a few steps on synthetic data
and writes the full artifact set.

The model took about two hours on a single CPU instance, on the order of 0.05 kg CO2e - a few phone
charges of energy, and millions of times less than training a frontier language model.

# Intended use, limitations, and safety

What it is for: the moderation and PII inference core inside an on premises or on device text
pipeline, where sending user text to a third party API is a cost or privacy problem. It flags toxic
content and personal information for a downstream policy layer.

What it is not for: this is a small model and a decision support signal, not a final arbiter. Do not
use it as the sole gate on user safety without human review and per category threshold tuning
(sensible starting thresholds ship in `config.json`).

Known limitations:

- The taxonomy head is scored on validation, not a held out competition test; treat those numbers as
  in distribution. The sexual category is data limited (0.67), and the targeted identity head is
  weaker (0.64) and should be used as a soft signal only.
- English centric supervision: the byte tokenizer is multilingual by construction, but the training
  labels are predominantly English, so non English accuracy is not established.
- The PII head reflects the nemotron and Presidio aligned entity catalog and its distribution;
  entity types and formats outside that corpus may be missed.

# How to cite

```bibtex
@misc{lowdownlabs_felamoderation,
  title  = {FELA-Moderation: CPU native byte level content moderation and PII detection},
  author = {Lowdown Labs},
  year   = {2026},
  note   = {Model card}
}
```

You should also cite the datasets and baselines used (Jigsaw / Conversation AI, nemotron,
civil_comments, measuring-hate-speech, real-toxicity-prompts, and Detoxify / Presidio); full
references and licenses are in `data/licenses.py` and the references below.

# Acknowledgements and references

- Fourier Neural Operator: Li, Z., et al. (2021). Fourier Neural Operator for Parametric Partial
  Differential Equations. ICLR. https://arxiv.org/abs/2010.08895
- Gated Linear Attention: Yang, S., et al. (2024). Gated Linear Attention Transformers with
  Hardware-Efficient Training. https://arxiv.org/abs/2312.06635
- Byte level modeling: Xue, L., et al. (2022). ByT5: Towards a Token-Free Future with Pre-trained
  Byte-to-Byte Models. TACL. https://arxiv.org/abs/2105.13626
- PyTorch: Paszke, A., et al. (2019). NeurIPS. https://arxiv.org/abs/1912.01703

# Model family

This is part of the FELA family from Lowdown Labs: one Fourier Neural Operator architecture across
many modalities, all CPU native and subquadratic. This repo is pushed as `lowdown-labs/fela-moderator`.
Sibling repos share no weights, so none carries a `base_model` link.

# License

Apache-2.0. The model weights and the code are both under Apache-2.0 (see LICENSE and NOTICE.txt): no
separate model license and no non commercial restriction. Every training source is permissively
licensed and commercial safe. We distribute weights only, never the source text.