README.md

---
license: mit
language: [en, tr]
pipeline_tag: text-classification
base_model: [BAAI/bge-m3, FacebookAI/xlm-roberta-large]
tags: [email, triage, local, cpu]
---

# Bengu-1

Bengu-1 is a local, multilingual mail triage model from Orkhon. For each incoming message it gives three answers:

- **category**: one of ten labels;
- **protected**: mail that must never be auto-archived or deleted;
- **needs_reply**: mail that waits for an answer from you.

It runs on the CPU. The measured median is about 420 ms per mail on a Mac, batch 1. It costs 0 USD per mail, and mail never leaves the device.

On the public triage track it leads both API systems we compared on group macro-F1 and on protected recall and precision. It is not a spam filter: run it behind your existing spam filter.

## Results: public triage track

| System | Group macro-F1 | Protected recall | Protected precision |
|---|---|---|---|
| Bengu-1 | **0.963** | **0.990** | **0.986** |
| OpenAI Decisions (gpt-6-luna) | 0.915 | 0.978 | 0.879 |
| TypeSafe Jev 1.13 | 0.895 | 0.954 | 0.836 |

These are the clean subset's numbers: 1,473 rows, 416 of them protected. On the full track (1,587 rows, 437 protected), Bengu-1 scores 0.964 group macro-F1 and 0.991 / 0.984 protected recall / precision.

- **Rows.** 1,587 messages sampled with seed 20261010:
  - SpamAssassin: 850;
  - Enron, UC Berkeley annotated subset: 437;
  - Nazario phishing 2024-2025: 300.
- **Gold labels.** They come from each dataset's own label plus header facts, never from a model.
- **Groups.** Junk, bulk, person and other legitimate mail. A predicted category counts as its group.
- **Same input.** Every system got the same input view. The two API systems answered zero-shot.

**Footnote.** The test rows come from the same public sources as part of the training data. Exact and near-duplicate rows (word 5-gram Jaccard >= 0.7) were removed, so no test row has such a copy in training. The clean subset also drops every test row with Jaccard >= 0.5 to any training row.

`eval/bengu_public_eval.py` rebuilds the rows from the upstream files and scores any predictions:
- `eval/public-track-rows.json` holds the row ids, hashes and source pointers;
- `eval/results-public-track.json` holds Bengu-1's numbers.

No corpus text ships.

## Intended use

Bengu-1 sorts incoming e-mail for one person. It is a classifier only and never acts on mail.

Do not use it to:
- make decisions about people;
- filter other people's mail without their consent;
- make security decisions on its own.

## Labels

| Category | Meaning |
|---|---|
| marketing_newsletter | promotional or newsletter mail sent in bulk: offers, sales, product news or content digests |
| bulk_notification | automated bulk notifications that are not a conversation: activity digests, platform announcements, generic service updates |
| spam | unsolicited junk that is not a deception attempt: cold mass pitches, link offers, unrelated bulk solicitations |
| phishing_or_scam | a deception attempt: fake login, account or delivery warnings, credential or payment harvesting, impersonation, prize or crypto scams, fake invoices |
| person_or_business | a real person or business writing to you directly, or a reply in a conversation |
| transactional_money_legal_security | legitimate mail about money, payments, invoices, receipts, taxes, contracts, legal matters, account security, sign-in alerts or codes |
| school_university | school or university matters |
| vendor_or_supplier | a supplier or vendor of your business: orders, quotes, deliveries, price lists |
| other | legitimate mail that fits none of the other categories |
| review | no category clearly fits, or the mail is too unclear to decide |

Spam and phishing are never protected.

## Input

`inference/bengu_view.py` builds the text the model reads from the raw RFC 822 bytes. The view holds:
- a header-facts line;
- the sender and Reply-To;
- the subject;
- the decoded body, with HTML reduced to text.

The view is capped at about 12,000 characters. For mail larger than 256 KiB, only the headers are read.

```
python inference/bengu_infer.py . message.eml
```

The model has two member encoders: bge-m3 and xlm-roberta-large, about 2.2 GB of weights each. A member's probabilities are temperature-scaled and averaged; protected takes the maximum of the two members. A stacker then gives the category. Thresholds are in `ensemble.json`.

## Training data

Public corpora. No text from any of them ships:

- **SpamAssassin public corpus (2002/2005).** https://spamassassin.apache.org/old/publiccorpus/. The corpus readme states that copyright for the text in the messages remains with the original senders.
- **Enron e-mail.** CMU release of 2015-05-07, https://www.cs.cmu.edu/~enron/, plus the UC Berkeley annotated subset. It was released publicly by the US Federal Energy Regulatory Commission (FERC) during its investigation.
- **Nazario phishing corpus (2015-2025, phishing0-3).** https://monkey.org/~jose/phishing/. CC-BY-4.0; attribution: Jose Nazario.

Plus a private set of the author's own mail, labelled privately; not released.

Training labels come from the source datasets and from open-weight Qwen models used as labellers:
- **Qwen3.8-Flash-Next**: `dealignai/Qwen3.8-Flash-Next-ABLITERATED-FP8` @ e6b9e45, and a local 4-bit MTPLX build of an abliterated Qwen3.8-Flash-Next.
- **Qwen3.8-27B**: `chimingw/Qwen3.8-27B-Uncensored-OrcaRouter`, GGUF @ 58ebd12 and an MLX 8-bit copy. It gave second opinions and the calibration labels.

No output of a commercial or proprietary classification API was used for training, row selection, calibration or model selection. Commercial APIs were used only to evaluate baselines.

## Limitations

- Not a spam filter. Bengu-1 does not reliably catch spam; run it behind your existing spam filter.
- **Era bias.** The public legitimate mail is from 2001-2002 (SpamAssassin, Enron), while the public phishing is from 2015-2025. A model can learn the age of a message instead of its content, so expect more mistakes on modern legitimate mail unlike anything it saw.
- **CPU latency.** About 420 ms per mail at the median on a Mac CPU, batch 1, and slower when other work shares the CPU.
- **Language mix.** The private training mail is about 94 % English and 5 % Turkish. The public corpora are English. Other languages are untested.
- **No authentication signals.** Bengu-1 does not use DKIM, SPF, DMARC or ARC. A forged sender is judged on content alone.
- **One person's inbox.** The categories were learned from one person's inbox. school_university and vendor_or_supplier reflect that person's school and suppliers, so expect them to transfer poorly.
- **Privacy audit of these weights:**
  - Membership inference, loss attack, per member: AUC 0.488 and 0.493; true-positive rate at 1 % false-positive rate 0.006 and 0.017.
  - Membership inference, shadow-model attack, per member: AUC 0.539 and 0.538; true-positive rate at 1 % false-positive rate 0.014 and 0.007.
  - Memorisation: 40 canary strings were planted in training. None of those planted once or four times was recovered. One of the ten planted 16 times was ranked first by one member.
  - Verdict: PASS against limits written before the audit (AUC < 0.60; true-positive rate at 1 % false-positive rate < 5 %).

## Licences and attribution

- **Weights, inference code and eval code:** MIT (`LICENSE`), matching the MIT base models.
- **Base models:**
  - BAAI/bge-m3 @ 5617a9f6 (MIT);
  - FacebookAI/xlm-roberta-large @ c23d21b0 (MIT).
- **Training corpora:** as listed above. Nazario phishing corpus by Jose Nazario, CC-BY-4.0.
- **Labeller:** `dealignai/Qwen3.8-Flash-Next-ABLITERATED-FP8` @ e6b9e45, Qwen Community License 1.0; its attribution notice is in `NOTICE`.
- **Notices:** `NOTICE` carries the base-model, labeller and corpus notices.