--- license: apache-2.0 co2_eq_emissions: emissions: 50 source: estimated from instance power draw, runtime, and grid intensity training_type: pretraining hardware_used: 1 AWS c7i CPU instance, about 2 hours, CPU only tags: - fela - fourier-neural-operator - fno - gated-linear-attention - cpu - on-device - content-moderation - toxicity - pii - byte-level library_name: transformers pipeline_tag: text-classification --- > **N.B.** To use this model with a convenient interface [check us out on Github](https://github.com/Lowdown-Labs/fela_moderator)! # FELA-Moderation: CPU native content moderation and PII detection FELA-Moderation reads a piece of text and does two things with it. It scores how toxic the text is across a set of moderation categories, and it points out the spans that look like personal information: names, emails, phone numbers, credit cards, and the like. It is small enough to run on a plain CPU, so that whole moderation and PII pass can live inside your own service or app instead of going out to a moderation API and billing you per call. It reads text as raw bytes rather than words. That has a nice side effect: there is no vocabulary file to ship, and the model is not locked to any one language. The model covers a 19 category moderation taxonomy, the 6 native Jigsaw labels, byte level PII over 56 entity types, and four extra classifier heads for spam, jailbreak, NSFW severity, and targeted identity. It is about 18.9M parameters and ships as `model.safetensors` (fp32) plus an int8 export of roughly 19 MB for on device and WASM serving. We ship weights only and never redistribute the source text. # What goes in, what comes out - Input: any UTF-8 text string. It is byte tokenized to `[CLS] b0 b1 ...` (each byte is its own token, ids 0 to 255, plus 256=PAD, 257=CLS, 258=SEP) and truncated to 512 tokens. The `encode_text` and `pad_batch` helpers in `modeling.py` do this for you. - Output 1 (taxonomy): 19 independent probabilities over the moderation taxonomy (the 11 OpenAI aligned categories plus identity_attack, insult, profanity_obscene, threat, sexual_explicit, severe_toxicity, dehumanize, incitement_violence). Full list in `config.json`. Sigmoid, multi label. - Output 2 (jigsaw): 6 independent probabilities over the native Jigsaw labels (toxic, severe_toxic, obscene, threat, insult, identity_hate). Sigmoid, multi label. - Output 3 (pii): per token BIO tags over 56 entity types (113 tags: O plus B- and I- per entity). - Output 4 (safety heads): spam / scam / phishing, jailbreak / prompt injection, NSFW severity, and targeted identity. The head is chosen with `task=` in the forward call (`taxonomy`, `jigsaw`, `pii`, `spam`, `jailbreak`, `nsfw`, `identity`, or `both`). # Why we built it this way The model is built on the same Fourier Neural Operator the rest of the FELA family uses: a filter that mixes the whole sequence at once in the frequency domain, with no attention matrix that balloons as the text gets longer. Alongside it rides a gated linear attention layer that picks up the content dependent details, and its working memory stays a fixed size. Because none of this leans on the usual all pairs attention, a moderation plus PII pass stays fast on an ordinary CPU. That speed is the whole cost argument against paying a per call API. It is deliberately small, about 18.9M parameters. One shared backbone feeds several light heads (taxonomy, Jigsaw, PII, spam, jailbreak, NSFW, identity), and the byte vocabulary is only 259 symbols, which quantizes to int8 well. # Performance Format and footprint. The fp32 weights ship as `model.safetensors` (about 75 MB). INT8 dynamic quantization of the Linear layers shrinks the model to roughly 19 MB while the FFT and GLA einsum paths stay in float; the int8 export holds accuracy on the held out slices (see below). Because the encoder carries no KV cache and a fixed size state, working memory does not grow with input length. # Accuracy Held out numbers are on permissively licensed splits: Jigsaw on a fixed leak free slice of the Jigsaw training data, Civil Comments validation for taxonomy, and nemotron plus synthetic for PII. Headline held out: | head | AUROC / acc | note | |---|---|---| | PII (byte token acc) | 0.96 | | | Jigsaw (mean AUROC) | 0.94 | | | taxonomy: hate / harassment | 0.84 / 0.77 | | | taxonomy: sexual | 0.67 | data limited, explicit data not in the training set | | spam / jailbreak / nsfw | 0.94 / 0.88 / 0.82 | | | identity | 0.64 | weaker, use as a soft signal only | The int8 export holds these numbers. Final training loss 0.69. The model's advantage is privacy on local hardware, speed, and clean provenance at roughly comparable quality, not topping a specialist leaderboard. The one honest gap is the sexual category (0.67). It does not close with more training: it is a data limitation. Explicit sexual data was deliberately kept out of the training set, so that category is supplied only by synthetic examples. We report it rather than hide it. # How to run it ```python import torch from modeling import load_model, encode_text, pad_batch m = load_model("model.safetensors") ids, _ = encode_text("You are awful. Email me at jane.doe@example.com", m.cfg.max_len) input_ids, mask = pad_batch([ids], m.cfg.max_len) out = m(input_ids, mask, task="both") tax = torch.sigmoid(out["taxonomy"][0]) # 19 taxonomy probabilities jig = torch.sigmoid(out["jigsaw"][0]) # 6 Jigsaw probabilities pii = out["pii"][0].argmax(-1) # per token BIO tag id ``` ## Loading with standard tooling The repo ships `config.json` (architecture hyperparameters plus the full label and tag maps and the per category toxicity thresholds) and a self contained `modeling.py` with a `load_model` / `from_pretrained` entry point: ```python from modeling import load_model m = load_model("/path/to/weights_dir") # OR m = load_model("lowdown-labs/fela-moderator") ``` The fp32 weights are shipped as `model.safetensors`. ## Serving artifacts - `model.safetensors` plus `config.json` for the safetensors load path (fp32). - `tier_full_int8.safetensors` plus `tier_full_scales.json` for the roughly 19 MB int8 export. - `manifest.json` and `NOTICE.txt` describe the tiers and the training source attributions. - `verify.py` runs a fixed sample input and checks the output shapes and a verification value. For serving at scale, use the separate CPU native FELA server (`https://github.com/Lowdown-Labs/fela_server`). It runs this model on CPU with no GPU required. # Training data We train on the sources below and distribute model weights only. No source text or derived dataset is redistributed, so the CC-BY-SA-3.0 status of Wikipedia derived comment text (under Jigsaw and Civil Comments) never triggers share alike. `data/licenses.py` carries the verified license of every source, and `NOTICE.txt` records the attributions. - PII: `nvidia/nemotron-pii` (CC-BY-4.0, 55+ types, character offset spans) plus `gretelai/synthetic_pii_finance_multilingual` (Apache-2.0) plus synthetic generation via Faker (MIT) and the Presidio generator (MIT, driven from Faker). - Toxicity and taxonomy: `google/civil_comments` (CC0-1.0), `google/jigsaw_toxicity_pred` (CC0-1.0 annotations), `ucberkeley-dlab/measuring-hate-speech` (CC-BY-4.0), `allenai/real-toxicity-prompts` (Apache-2.0). - Spam: `redasers/difraud` (MIT) and `ucirvine/sms_spam` (CC-BY-4.0). - Jailbreak and prompt injection: `cyberec/llm-prompt-injection-attacks` (Apache-2.0), `deepset/prompt-injections` (Apache-2.0), `jackhhao/jailbreak-classification` (Apache-2.0), `Lakera/gandalf_ignore_instructions` (MIT). - NSFW: `google/civil_comments` sexual_explicit signal, with `mmathys/openai-moderation-api-evaluation` (MIT) used for evaluation only. - Targeted identity: `ucberkeley-dlab/measuring-hate-speech` (CC-BY-4.0). Full split definitions, size assertions, and per source license verdicts are reproduced in `train.py` and `data/licenses.py`. **Expanded taxonomy (19).** Beyond the 11 OpenAI aligned categories the model adds identity_attack, insult, profanity_obscene, threat, sexual_explicit, severe_toxicity, dehumanize, and incitement_violence, each sourced from a permissive dataset. Growing the head does not change the byte level backbone, so the int8 size is unchanged. Five categories (self_harm, self_harm_intent, self_harm_instructions, sexual_minors, violence_graphic) have no permissive source: they are masked from real data and supplied by synthetic and reviewed examples, and reported separately so coverage is never overclaimed. **Byte level PII BIO with exact spans.** Character offsets from the real sources and byte offsets from the synthetic generator are aligned to byte tokens in `data/bio.py` (verified 100% correct on 1,251 spans). The head is trained across context lengths (128 to 8192 bytes) with PII needles buried in benign carrier text, so it finds a single email or one toxic span inside a long document. # How it was trained The model was trained CPU only via DiLoCo (data parallel with infrequent parameter averaging coordinated through S3), on commodity CPU instances with no GPU. Weights checkpoint to S3 periodically, so a spot interruption resumes rather than restarts. The taxonomy and Jigsaw heads train every step; the spam, jailbreak, NSFW, and identity heads rotate. Run `python train.py --smoke` for an offline end to end check that trains a few steps on synthetic data and writes the full artifact set. The model took about two hours on a single CPU instance, on the order of 0.05 kg CO2e - a few phone charges of energy, and millions of times less than training a frontier language model. # Intended use, limitations, and safety What it is for: the moderation and PII inference core inside an on premises or on device text pipeline, where sending user text to a third party API is a cost or privacy problem. It flags toxic content and personal information for a downstream policy layer. What it is not for: this is a small model and a decision support signal, not a final arbiter. Do not use it as the sole gate on user safety without human review and per category threshold tuning (sensible starting thresholds ship in `config.json`). Known limitations: - The taxonomy head is scored on validation, not a held out competition test; treat those numbers as in distribution. The sexual category is data limited (0.67), and the targeted identity head is weaker (0.64) and should be used as a soft signal only. - English centric supervision: the byte tokenizer is multilingual by construction, but the training labels are predominantly English, so non English accuracy is not established. - The PII head reflects the nemotron and Presidio aligned entity catalog and its distribution; entity types and formats outside that corpus may be missed. # How to cite ```bibtex @misc{lowdownlabs_felamoderation, title = {FELA-Moderation: CPU native byte level content moderation and PII detection}, author = {Lowdown Labs}, year = {2026}, note = {Model card} } ``` You should also cite the datasets and baselines used (Jigsaw / Conversation AI, nemotron, civil_comments, measuring-hate-speech, real-toxicity-prompts, and Detoxify / Presidio); full references and licenses are in `data/licenses.py` and the references below. # Acknowledgements and references - Fourier Neural Operator: Li, Z., et al. (2021). Fourier Neural Operator for Parametric Partial Differential Equations. ICLR. https://arxiv.org/abs/2010.08895 - Gated Linear Attention: Yang, S., et al. (2024). Gated Linear Attention Transformers with Hardware-Efficient Training. https://arxiv.org/abs/2312.06635 - Byte level modeling: Xue, L., et al. (2022). ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. TACL. https://arxiv.org/abs/2105.13626 - PyTorch: Paszke, A., et al. (2019). NeurIPS. https://arxiv.org/abs/1912.01703 # Model family This is part of the FELA family from Lowdown Labs: one Fourier Neural Operator architecture across many modalities, all CPU native and subquadratic. This repo is pushed as `lowdown-labs/fela-moderator`. Sibling repos share no weights, so none carries a `base_model` link. # License Apache-2.0. The model weights and the code are both under Apache-2.0 (see LICENSE and NOTICE.txt): no separate model license and no non commercial restriction. Every training source is permissively licensed and commercial safe. We distribute weights only, never the source text.