Bengu-1

Bengu-1 is a local, multilingual mail triage model from Orkhon. For each incoming message it gives three answers:

• category: one of ten labels;
• protected: mail that must never be auto-archived or deleted;
• needs_reply: mail that waits for an answer from you.

It runs on the CPU. The measured median is about 420 ms per mail on a Mac, batch 1. It costs 0 USD per mail, and mail never leaves the device.

On the public triage track it leads both API systems we compared on group macro-F1 and on protected recall and precision. It is not a spam filter: run it behind your existing spam filter.

Results: public triage track

System	Group macro-F1	Protected recall	Protected precision
Bengu-1	0.963	0.990	0.986
OpenAI Decisions (gpt-6-luna)	0.915	0.978	0.879
TypeSafe Jev 1.13	0.895	0.954	0.836

These are the clean subset's numbers: 1,473 rows, 416 of them protected. On the full track (1,587 rows, 437 protected), Bengu-1 scores 0.964 group macro-F1 and 0.991 / 0.984 protected recall / precision.

• Rows. 1,587 messages sampled with seed 20261010:
  • SpamAssassin: 850;
  • Enron, UC Berkeley annotated subset: 437;
  • Nazario phishing 2024-2025: 300.
• Gold labels. They come from each dataset's own label plus header facts, never from a model.
• Groups. Junk, bulk, person and other legitimate mail. A predicted category counts as its group.
• Same input. Every system got the same input view. The two API systems answered zero-shot.

Footnote. The test rows come from the same public sources as part of the training data. Exact and near-duplicate rows (word 5-gram Jaccard >= 0.7) were removed, so no test row has such a copy in training. The clean subset also drops every test row with Jaccard >= 0.5 to any training row.

eval/bengu_public_eval.py rebuilds the rows from the upstream files and scores any predictions:

• eval/public-track-rows.json holds the row ids, hashes and source pointers;
• eval/results-public-track.json holds Bengu-1's numbers.

No corpus text ships.

Intended use

Bengu-1 sorts incoming e-mail for one person. It is a classifier only and never acts on mail.

Do not use it to:

• make decisions about people;
• filter other people's mail without their consent;
• make security decisions on its own.

Labels

Category	Meaning
marketing_newsletter	promotional or newsletter mail sent in bulk: offers, sales, product news or content digests
bulk_notification	automated bulk notifications that are not a conversation: activity digests, platform announcements, generic service updates
spam	unsolicited junk that is not a deception attempt: cold mass pitches, link offers, unrelated bulk solicitations
phishing_or_scam	a deception attempt: fake login, account or delivery warnings, credential or payment harvesting, impersonation, prize or crypto scams, fake invoices
person_or_business	a real person or business writing to you directly, or a reply in a conversation
transactional_money_legal_security	legitimate mail about money, payments, invoices, receipts, taxes, contracts, legal matters, account security, sign-in alerts or codes
school_university	school or university matters
vendor_or_supplier	a supplier or vendor of your business: orders, quotes, deliveries, price lists
other	legitimate mail that fits none of the other categories
review	no category clearly fits, or the mail is too unclear to decide

Spam and phishing are never protected.

Input

inference/bengu_view.py builds the text the model reads from the raw RFC 822 bytes. The view holds:

• a header-facts line;
• the sender and Reply-To;
• the subject;
• the decoded body, with HTML reduced to text.

The view is capped at about 12,000 characters. For mail larger than 256 KiB, only the headers are read.

python inference/bengu_infer.py . message.eml

The model has two member encoders: bge-m3 and xlm-roberta-large, about 2.2 GB of weights each. A member's probabilities are temperature-scaled and averaged; protected takes the maximum of the two members. A stacker then gives the category. Thresholds are in ensemble.json.

Training data

Public corpora. No text from any of them ships:

• SpamAssassin public corpus (2002/2005). https://spamassassin.apache.org/old/publiccorpus/. The corpus readme states that copyright for the text in the messages remains with the original senders.
• Enron e-mail. CMU release of 2015-05-07, https://www.cs.cmu.edu/~enron/, plus the UC Berkeley annotated subset. It was released publicly by the US Federal Energy Regulatory Commission (FERC) during its investigation.
• Nazario phishing corpus (2015-2025, phishing0-3). https://monkey.org/~jose/phishing/. CC-BY-4.0; attribution: Jose Nazario.

Plus a private set of the author's own mail, labelled privately; not released.

Training labels come from the source datasets and from open-weight Qwen models used as labellers:

• Qwen3.8-Flash-Next: dealignai/Qwen3.8-Flash-Next-ABLITERATED-FP8 @ e6b9e45, and a local 4-bit MTPLX build of an abliterated Qwen3.8-Flash-Next.
• Qwen3.8-27B: chimingw/Qwen3.8-27B-Uncensored-OrcaRouter, GGUF @ 58ebd12 and an MLX 8-bit copy. It gave second opinions and the calibration labels.

No output of a commercial or proprietary classification API was used for training, row selection, calibration or model selection. Commercial APIs were used only to evaluate baselines.

Limitations

• Not a spam filter. Bengu-1 does not reliably catch spam; run it behind your existing spam filter.
• Era bias. The public legitimate mail is from 2001-2002 (SpamAssassin, Enron), while the public phishing is from 2015-2025. A model can learn the age of a message instead of its content, so expect more mistakes on modern legitimate mail unlike anything it saw.
• CPU latency. About 420 ms per mail at the median on a Mac CPU, batch 1, and slower when other work shares the CPU.
• Language mix. The private training mail is about 94 % English and 5 % Turkish. The public corpora are English. Other languages are untested.
• No authentication signals. Bengu-1 does not use DKIM, SPF, DMARC or ARC. A forged sender is judged on content alone.
• One person's inbox. The categories were learned from one person's inbox. school_university and vendor_or_supplier reflect that person's school and suppliers, so expect them to transfer poorly.
• Privacy audit of these weights:
  • Membership inference, loss attack, per member: AUC 0.488 and 0.493; true-positive rate at 1 % false-positive rate 0.006 and 0.017.
  • Membership inference, shadow-model attack, per member: AUC 0.539 and 0.538; true-positive rate at 1 % false-positive rate 0.014 and 0.007.
  • Memorisation: 40 canary strings were planted in training. None of those planted once or four times was recovered. One of the ten planted 16 times was ranked first by one member.
  • Verdict: PASS against limits written before the audit (AUC < 0.60; true-positive rate at 1 % false-positive rate < 5 %).

Licences and attribution

• Weights, inference code and eval code: MIT (LICENSE), matching the MIT base models.
• Base models:
  • BAAI/bge-m3 @ 5617a9f6 (MIT);
  • FacebookAI/xlm-roberta-large @ c23d21b0 (MIT).
• Training corpora: as listed above. Nazario phishing corpus by Jose Nazario, CC-BY-4.0.
• Labeller: dealignai/Qwen3.8-Flash-Next-ABLITERATED-FP8 @ e6b9e45, Qwen Community License 1.0; its attribution notice is in NOTICE.
• Notices: NOTICE carries the base-model, labeller and corpus notices.