bengu-1 / README.md

Bengu-1 package r4 (bengu repo 1456a0f)

e79f133 verified 2 days ago

7.84 kB

---
license: mit
language: [en, tr]
pipeline_tag: text-classification
base_model: [BAAI/bge-m3, FacebookAI/xlm-roberta-large]
tags: [email, triage, local, cpu]
---
# Bengu-1

Bengu-1 is a local, multilingual mail triage model from Orkhon. For each incoming message it gives three answers:

- **category**: one of ten labels;
- **protected**: mail that must never be auto-archived or deleted;
- **needs_reply**: mail that waits for an answer from you.
It runs on the CPU. The measured median is about 420 ms per mail on a Mac, batch 1. It costs 0 USD per mail, and mail never leaves the device.
On the public triage track it leads both API systems we compared on group macro-F1 and on protected recall and precision. It is not a spam filter: run it behind your existing spam filter.
## Results: public triage track
| System | Group macro-F1 | Protected recall | Protected precision |
|---|---|---|---|
| Bengu-1 | **0.963** | **0.990** | **0.986** |
| OpenAI Decisions (gpt-6-luna) | 0.915 | 0.978 | 0.879 |
| TypeSafe Jev 1.13 | 0.895 | 0.954 | 0.836 |
These are the clean subset's numbers: 1,473 rows, 416 of them protected. On the full track (1,587 rows, 437 protected), Bengu-1 scores 0.964 group macro-F1 and 0.991 / 0.984 protected recall / precision.
- **Rows.** 1,587 messages sampled with seed 20261010:
  - SpamAssassin: 850;
  - Enron, UC Berkeley annotated subset: 437;
  - Nazario phishing 2024-2025: 300.
- **Gold labels.** They come from each dataset's own label plus header facts, never from a model.
- **Groups.** Junk, bulk, person and other legitimate mail. A predicted category counts as its group.
- **Same input.** Every system got the same input view. The two API systems answered zero-shot.
**Footnote.** The test rows come from the same public sources as part of the training data. Exact and near-duplicate rows (word 5-gram Jaccard >= 0.7) were removed, so no test row has such a copy in training. The clean subset also drops every test row with Jaccard >= 0.5 to any training row.
`eval/bengu_public_eval.py` rebuilds the rows from the upstream files and scores any predictions:
- `eval/public-track-rows.json` holds the row ids, hashes and source pointers;
- `eval/results-public-track.json` holds Bengu-1's numbers.
No corpus text ships.
## Intended use
Bengu-1 sorts incoming e-mail for one person. It is a classifier only and never acts on mail.
Do not use it to:
- make decisions about people;
- filter other people's mail without their consent;
- make security decisions on its own.
## Labels
| Category | Meaning |
|---|---|
| marketing_newsletter | promotional or newsletter mail sent in bulk: offers, sales, product news or content digests |
| bulk_notification | automated bulk notifications that are not a conversation: activity digests, platform announcements, generic service updates |
| spam | unsolicited junk that is not a deception attempt: cold mass pitches, link offers, unrelated bulk solicitations |
| phishing_or_scam | a deception attempt: fake login, account or delivery warnings, credential or payment harvesting, impersonation, prize or crypto scams, fake invoices |
| person_or_business | a real person or business writing to you directly, or a reply in a conversation |
| transactional_money_legal_security | legitimate mail about money, payments, invoices, receipts, taxes, contracts, legal matters, account security, sign-in alerts or codes |
| school_university | school or university matters |
| vendor_or_supplier | a supplier or vendor of your business: orders, quotes, deliveries, price lists |
| other | legitimate mail that fits none of the other categories |
| review | no category clearly fits, or the mail is too unclear to decide |
Spam and phishing are never protected.
## Input
`inference/bengu_view.py` builds the text the model reads from the raw RFC 822 bytes. The view holds:
- a header-facts line;
- the sender and Reply-To;
- the subject;
- the decoded body, with HTML reduced to text.
The view is capped at about 12,000 characters. For mail larger than 256 KiB, only the headers are read.
```
python inference/bengu_infer.py . message.eml
```
The model has two member encoders: bge-m3 and xlm-roberta-large, about 2.2 GB of weights each. A member's probabilities are temperature-scaled and averaged; protected takes the maximum of the two members. A stacker then gives the category. Thresholds are in `ensemble.json`.
## Training data
Public corpora. No text from any of them ships:
- **SpamAssassin public corpus (2002/2005).** https://spamassassin.apache.org/old/publiccorpus/. The corpus readme states that copyright for the text in the messages remains with the original senders.
- **Enron e-mail.** CMU release of 2015-05-07, https://www.cs.cmu.edu/~enron/, plus the UC Berkeley annotated subset. It was released publicly by the US Federal Energy Regulatory Commission (FERC) during its investigation.
- **Nazario phishing corpus (2015-2025, phishing0-3).** https://monkey.org/~jose/phishing/. CC-BY-4.0; attribution: Jose Nazario.
Plus a private set of the author's own mail, labelled privately; not released.
Training labels come from the source datasets and from open-weight Qwen models used as labellers:
- **Qwen3.8-Flash-Next**: `dealignai/Qwen3.8-Flash-Next-ABLITERATED-FP8` @ e6b9e45, and a local 4-bit MTPLX build of an abliterated Qwen3.8-Flash-Next.
- **Qwen3.8-27B**: `chimingw/Qwen3.8-27B-Uncensored-OrcaRouter`, GGUF @ 58ebd12 and an MLX 8-bit copy. It gave second opinions and the calibration labels.
No output of a commercial or proprietary classification API was used for training, row selection, calibration or model selection. Commercial APIs were used only to evaluate baselines.
## Limitations
- Not a spam filter. Bengu-1 does not reliably catch spam; run it behind your existing spam filter.
- **Era bias.** The public legitimate mail is from 2001-2002 (SpamAssassin, Enron), while the public phishing is from 2015-2025. A model can learn the age of a message instead of its content, so expect more mistakes on modern legitimate mail unlike anything it saw.
- **CPU latency.** About 420 ms per mail at the median on a Mac CPU, batch 1, and slower when other work shares the CPU.
- **Language mix.** The private training mail is about 94 % English and 5 % Turkish. The public corpora are English. Other languages are untested.
- **No authentication signals.** Bengu-1 does not use DKIM, SPF, DMARC or ARC. A forged sender is judged on content alone.
- **One person's inbox.** The categories were learned from one person's inbox. school_university and vendor_or_supplier reflect that person's school and suppliers, so expect them to transfer poorly.
- **Privacy audit of these weights:**
  - Membership inference, loss attack, per member: AUC 0.488 and 0.493; true-positive rate at 1 % false-positive rate 0.006 and 0.017.
  - Membership inference, shadow-model attack, per member: AUC 0.539 and 0.538; true-positive rate at 1 % false-positive rate 0.014 and 0.007.
  - Memorisation: 40 canary strings were planted in training. None of those planted once or four times was recovered. One of the ten planted 16 times was ranked first by one member.
  - Verdict: PASS against limits written before the audit (AUC < 0.60; true-positive rate at 1 % false-positive rate < 5 %).

## Licences and attribution

- **Weights, inference code and eval code:** MIT (`LICENSE`), matching the MIT base models.
- **Base models:**
  - BAAI/bge-m3 @ 5617a9f6 (MIT);
  - FacebookAI/xlm-roberta-large @ c23d21b0 (MIT).
- **Training corpora:** as listed above. Nazario phishing corpus by Jose Nazario, CC-BY-4.0.
- **Labeller:** `dealignai/Qwen3.8-Flash-Next-ABLITERATED-FP8` @ e6b9e45, Qwen Community License 1.0; its attribution notice is in `NOTICE`.
- **Notices:** `NOTICE` carries the base-model, labeller and corpus notices.

	---
	license: mit
	language: [en, tr]
	pipeline_tag: text-classification
	base_model: [BAAI/bge-m3, FacebookAI/xlm-roberta-large]
	tags: [email, triage, local, cpu]
	---

	# Bengu-1

	Bengu-1 is a local, multilingual mail triage model from Orkhon. For each incoming message it gives three answers:

	- category: one of ten labels;
	- protected: mail that must never be auto-archived or deleted;
	- needs_reply: mail that waits for an answer from you.

	It runs on the CPU. The measured median is about 420 ms per mail on a Mac, batch 1. It costs 0 USD per mail, and mail never leaves the device.

	On the public triage track it leads both API systems we compared on group macro-F1 and on protected recall and precision. It is not a spam filter: run it behind your existing spam filter.

	## Results: public triage track

	\| System \| Group macro-F1 \| Protected recall \| Protected precision \|
	\|---\|---\|---\|---\|
	\| Bengu-1 \| 0.963 \| 0.990 \| 0.986 \|
	\| OpenAI Decisions (gpt-6-luna) \| 0.915 \| 0.978 \| 0.879 \|
	\| TypeSafe Jev 1.13 \| 0.895 \| 0.954 \| 0.836 \|

	These are the clean subset's numbers: 1,473 rows, 416 of them protected. On the full track (1,587 rows, 437 protected), Bengu-1 scores 0.964 group macro-F1 and 0.991 / 0.984 protected recall / precision.

	- Rows. 1,587 messages sampled with seed 20261010:
	- SpamAssassin: 850;
	- Enron, UC Berkeley annotated subset: 437;
	- Nazario phishing 2024-2025: 300.
	- Gold labels. They come from each dataset's own label plus header facts, never from a model.
	- Groups. Junk, bulk, person and other legitimate mail. A predicted category counts as its group.
	- Same input. Every system got the same input view. The two API systems answered zero-shot.

	Footnote. The test rows come from the same public sources as part of the training data. Exact and near-duplicate rows (word 5-gram Jaccard >= 0.7) were removed, so no test row has such a copy in training. The clean subset also drops every test row with Jaccard >= 0.5 to any training row.

	`eval/bengu_public_eval.py` rebuilds the rows from the upstream files and scores any predictions:
	- `eval/public-track-rows.json` holds the row ids, hashes and source pointers;
	- `eval/results-public-track.json` holds Bengu-1's numbers.

	No corpus text ships.

	## Intended use

	Bengu-1 sorts incoming e-mail for one person. It is a classifier only and never acts on mail.

	Do not use it to:
	- make decisions about people;
	- filter other people's mail without their consent;
	- make security decisions on its own.

	## Labels

	\| Category \| Meaning \|
	\|---\|---\|
	\| marketing_newsletter \| promotional or newsletter mail sent in bulk: offers, sales, product news or content digests \|
	\| bulk_notification \| automated bulk notifications that are not a conversation: activity digests, platform announcements, generic service updates \|
	\| spam \| unsolicited junk that is not a deception attempt: cold mass pitches, link offers, unrelated bulk solicitations \|
	\| phishing_or_scam \| a deception attempt: fake login, account or delivery warnings, credential or payment harvesting, impersonation, prize or crypto scams, fake invoices \|
	\| person_or_business \| a real person or business writing to you directly, or a reply in a conversation \|
	\| transactional_money_legal_security \| legitimate mail about money, payments, invoices, receipts, taxes, contracts, legal matters, account security, sign-in alerts or codes \|
	\| school_university \| school or university matters \|
	\| vendor_or_supplier \| a supplier or vendor of your business: orders, quotes, deliveries, price lists \|
	\| other \| legitimate mail that fits none of the other categories \|
	\| review \| no category clearly fits, or the mail is too unclear to decide \|

	Spam and phishing are never protected.

	## Input

	`inference/bengu_view.py` builds the text the model reads from the raw RFC 822 bytes. The view holds:
	- a header-facts line;
	- the sender and Reply-To;
	- the subject;
	- the decoded body, with HTML reduced to text.

	The view is capped at about 12,000 characters. For mail larger than 256 KiB, only the headers are read.

	```
	python inference/bengu_infer.py . message.eml
	```

	The model has two member encoders: bge-m3 and xlm-roberta-large, about 2.2 GB of weights each. A member's probabilities are temperature-scaled and averaged; protected takes the maximum of the two members. A stacker then gives the category. Thresholds are in `ensemble.json`.

	## Training data

	Public corpora. No text from any of them ships:

	- SpamAssassin public corpus (2002/2005). https://spamassassin.apache.org/old/publiccorpus/. The corpus readme states that copyright for the text in the messages remains with the original senders.
	- Enron e-mail. CMU release of 2015-05-07, https://www.cs.cmu.edu/~enron/, plus the UC Berkeley annotated subset. It was released publicly by the US Federal Energy Regulatory Commission (FERC) during its investigation.
	- Nazario phishing corpus (2015-2025, phishing0-3). https://monkey.org/~jose/phishing/. CC-BY-4.0; attribution: Jose Nazario.

	Plus a private set of the author's own mail, labelled privately; not released.

	Training labels come from the source datasets and from open-weight Qwen models used as labellers:
	- Qwen3.8-Flash-Next: `dealignai/Qwen3.8-Flash-Next-ABLITERATED-FP8` @ e6b9e45, and a local 4-bit MTPLX build of an abliterated Qwen3.8-Flash-Next.
	- Qwen3.8-27B: `chimingw/Qwen3.8-27B-Uncensored-OrcaRouter`, GGUF @ 58ebd12 and an MLX 8-bit copy. It gave second opinions and the calibration labels.

	No output of a commercial or proprietary classification API was used for training, row selection, calibration or model selection. Commercial APIs were used only to evaluate baselines.

	## Limitations

	- Not a spam filter. Bengu-1 does not reliably catch spam; run it behind your existing spam filter.
	- Era bias. The public legitimate mail is from 2001-2002 (SpamAssassin, Enron), while the public phishing is from 2015-2025. A model can learn the age of a message instead of its content, so expect more mistakes on modern legitimate mail unlike anything it saw.
	- CPU latency. About 420 ms per mail at the median on a Mac CPU, batch 1, and slower when other work shares the CPU.
	- Language mix. The private training mail is about 94 % English and 5 % Turkish. The public corpora are English. Other languages are untested.
	- No authentication signals. Bengu-1 does not use DKIM, SPF, DMARC or ARC. A forged sender is judged on content alone.
	- One person's inbox. The categories were learned from one person's inbox. school_university and vendor_or_supplier reflect that person's school and suppliers, so expect them to transfer poorly.
	- Privacy audit of these weights:
	- Membership inference, loss attack, per member: AUC 0.488 and 0.493; true-positive rate at 1 % false-positive rate 0.006 and 0.017.
	- Membership inference, shadow-model attack, per member: AUC 0.539 and 0.538; true-positive rate at 1 % false-positive rate 0.014 and 0.007.
	- Memorisation: 40 canary strings were planted in training. None of those planted once or four times was recovered. One of the ten planted 16 times was ranked first by one member.
	- Verdict: PASS against limits written before the audit (AUC < 0.60; true-positive rate at 1 % false-positive rate < 5 %).

	## Licences and attribution

	- Weights, inference code and eval code: MIT (`LICENSE`), matching the MIT base models.
	- Base models:
	- BAAI/bge-m3 @ 5617a9f6 (MIT);
	- FacebookAI/xlm-roberta-large @ c23d21b0 (MIT).
	- Training corpora: as listed above. Nazario phishing corpus by Jose Nazario, CC-BY-4.0.
	- Labeller: `dealignai/Qwen3.8-Flash-Next-ABLITERATED-FP8` @ e6b9e45, Qwen Community License 1.0; its attribution notice is in `NOTICE`.
	- Notices: `NOTICE` carries the base-model, labeller and corpus notices.