Add presentation bundle (reports, curves, run log, code) for the Colab notebook

439e7b1 verified 15 days ago

18.1 kB

# Error analysis — test split (TREC-06 binary holdout), run `ft-focal` (2026-10-09 10:46)

All quoted emails are rows of public corpora (source TREC-06); nothing here is private mail. Threshold 0.5. Binary positive class = Unwanted (phishing or spam), P(unwanted) = P(phishing) + P(spam).

## Counts (binary roll-up)

- rows 16,302 (positives 3,894, 23.9%); label3 counts legit 12,408, unwanted 3,894
- **recall (unwanted) 0.9741** — 101 missed unwanted emails (false negatives) out of 3,894
- precision 0.8794 — 520 false alarms out of 4,313 flagged
- F1 0.9243, accuracy 0.9619, specificity 0.9581
- ROC-AUC 0.9946, PR-AUC 0.9848; recall at 1% FPR 0.8955, at 0.1% FPR 0.6497
- credential-lure slice: 398 rows, 243 positive (61.1% label purity: slice precision cannot exceed what the phrase list selects), recall 0.9959, precision 0.9758, 1 missed

## Top-25 false negatives (missed unwanted mail, most confident first)

| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---|---|---|
| 1 | `primary:052869` | 0.0322 | 630 | TREC-06 | unwanted | READ THIS! │ >>Read this! >>> >> > >> > > >>> >> > >> > > ---------- >>> >> > >> > > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > > > Subject: Lets see if this... |
| 2 | `primary:080991` | 0.0428 | 75 | TREC-06 | unwanted | prepare urself for this │ It is been awhile. Hope everythign with you is goign well. How is work going also? I've just started with this new firm, www.andwhatis... |
| 3 | `primary:086987` | 0.0506 | 188 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 4 | `primary:056767` | 0.0564 | 181 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 5 | `primary:121268` | 0.0564 | 185 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 6 | `primary:173636` | 0.0567 | 55 | TREC-06 | unwanted | everything ok │ Looking to increase your johnson. Vist at http://www.mydkillsanddoc.com/taw/ to settle that. that that Hardly were there is conquering Thanks. B... |
| 7 | `primary:020947` | 0.0629 | 113 | TREC-06 | unwanted | everything kool │ Hi Reg Here's some pretty hard facts to deal with that the majority of women are unhappy with the size of there signif other's thing. Don't be... |
| 8 | `primary:072993` | 0.0710 | 186 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 9 | `primary:044992` | 0.0726 | 76 | TREC-06 | unwanted | FW:any new news? i got some │ is hand the language care to cruel. must plate to design tax but fiction. if snow but question great the separate. but comparison ... |
| 10 | `primary:024565` | 0.0748 | 190 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 11 | `primary:076023` | 0.0867 | 54 | TREC-06 | unwanted | everything kool │ Hey Mostyn, Ramon wanted to me to send you an email about the store I went to buy my goods, http://www.fabloulusil.com/tpn/. Carlos |
| 12 | `primary:129745` | 0.0946 | 184 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 13 | `primary:071577` | 0.0961 | 162 | TREC-06 | unwanted | (no subject) │ Re: Student Packets Welcome to winter term student packet preparations! First: Fax (485-6253) or email (Copy-Shop@worldnet.att.net) JUST your bib... |
| 14 | `primary:173938` | 0.0961 | 185 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 15 | `primary:030013` | 0.0966 | 195 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 16 | `primary:063077` | 0.0982 | 344 | TREC-06 | unwanted | Party for Prisoners of Conscience - PLEASE SEND TO YOUR ENTIRE CLUB │ Amnesty International and the Intercultural Resource Center are holding a party to raise m... |
| 17 | `primary:145723` | 0.1007 | 25 | TREC-06 | unwanted | refresh my memory and mental energy │ may revision a golf on appendage or eagle try parsimony |
| 18 | `primary:154489` | 0.1026 | 187 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 19 | `primary:186523` | 0.1041 | 185 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 20 | `primary:147887` | 0.1150 | 181 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 21 | `primary:033014` | 0.1157 | 77 | TREC-06 | unwanted | isn't it time │ Allen mentioned me that u were seeking for a place to get ur stuff for ur sickness. http://www.whorunoverforitsl.org/tpn/ the place then for sur... |
| 22 | `primary:156328` | 0.1157 | 77 | TREC-06 | unwanted | isn't it time │ Allen mentioned me that u were seeking for a place to get ur stuff for ur sickness. http://www.whorunoverforitsl.org/tpn/ the place then for sur... |
| 23 | `primary:143065` | 0.1308 | 102 | TREC-06 | unwanted | everything kool │ Heya Alexa, Apologize for how long its been since we've talked. I take blame for that. I just wanted to share with you the place that made me ... |
| 24 | `primary:147193` | 0.1323 | 260 | TREC-06 | unwanted | ADV Textbooks, Tuesday & Wednesday, University of Oregon │ Dear Professor XXXXXXXX., I am going to be on your campus to buy textbooks this Tuesday, June 1st and... |
| 25 | `primary:101618` | 0.1387 | 117 | TREC-06 | unwanted | RE: If you are studing or working │ Dear Mr(Mrs), Iris!!. Our company Barcelo Travel Inc. have a great job offer for peoples from Australia. Your salary per wee... |

## Top-25 false positives (legitimate flagged, most confident first)

| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---|---|---|
| 1 | `primary:015304` | 0.9874 | 3712 | TREC-06 | legit | I made $230,000 in two months │ You owe it to yourself and your wealth to read this. AS SEEN ON NATIONAL TV: "Making over half a million dollars every 4 to 5 mo... |
| 2 | `primary:040048` | 0.9871 | 80 | TREC-06 | legit | Enter to win at mary-kateandashley.com! │ Dear val: Have you heard about the sweepstakes at mary-kateandashley.com? I entered to win and you can too! Check it o... |
| 3 | `primary:194751` | 0.9842 | 48 | TREC-06 | legit | Thank you │ Turns out I just needed a common ground between servos and HB Thanx all! :) Alex ______________________________________________________ Get Your Pri... |
| 4 | `primary:072489` | 0.9831 | 58 | TREC-06 | legit | Re: Junk email │ Please REMOVE this address from your list!!!! ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you b... |
| 5 | `primary:194063` | 0.9781 | 32 | TREC-06 | legit | info │ Here's an affordable robot that anyone might be interested: http://www.personalrobots.com/ joema |
| 6 | `primary:033174` | 0.9781 | 602 | TREC-06 | legit | You are Approved for Merchant Account!! │ INCREASE SALES UP TO 1500% ACCEPT CREDIT CARDS OVER THE INTERNET *** NO SETUP FEES Good Credit / Bad Credit/ No Credit... |
| 7 | `primary:101560` | 0.9769 | 59 | TREC-06 | legit |  │ Please, please, please remove us from your mailing list. Thank you. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can l... |
| 8 | `primary:159590` | 0.9764 | 56 | TREC-06 | legit | remove me │ Please remove Brandeis University from your list ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you bec... |
| 9 | `primary:184838` | 0.9757 | 66 | TREC-06 | legit | unwanted mail │ how cani stop all the mail from these sex sites i dont want them but when i try to e-mail asking them to stop i keep getting a message that say ... |
| 10 | `primary:011014` | 0.9736 | 46 | TREC-06 | legit | composer-wrap-column │ hi, anyone knows how to entirely turn off composer wrapping in pine? thanks! viinikala kauas pilvet karkaavat. |
| 11 | `primary:193401` | 0.9733 | 870 | TREC-06 | legit | [9fans] 57 Million Email Addresses - ONLY $149!! │ 57 MILLION EMAIL ADDRESSES FOR ONLY $149 You want to make some money? I can put you in touch with over 50 mil... |
| 12 | `primary:072275` | 0.9729 | 41 | TREC-06 | legit | HB weight │ does anyone know the weight of a handy board? thanks, -Brian --------------------------------------------------- Get free personalized email at http... |
| 13 | `primary:170605` | 0.9729 | 55 | TREC-06 | legit | List │ Please remove my name from your list serve. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you become anybod... |
| 14 | `primary:175533` | 0.9726 | 62 | TREC-06 | legit | Test │ I'm not sure why we are receiving mail addressed to cslist@fudan.edu ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We ... |
| 15 | `primary:131182` | 0.9719 | 31 | TREC-06 | legit |  │ unsubscribe --------------------------------------------------------------- Octopus traps summer's moonspun dreams soon fade away. Basho 1644 |
| 16 | `primary:106445` | 0.9717 | 61 | TREC-06 | legit | REMOVE Clemson University │ PLEASE REMOVE Clemson University from your mailing lists. ________________ Email Forwarding Service Brought To You By: http://www.bn... |
| 17 | `primary:078484` | 0.9706 | 77 | TREC-06 | legit | remove from list │ Please remove Brandeis University from your graduate school mailing list. maf@cs.brandeis.edu thank you ________________ Email Forwarding Ser... |
| 18 | `primary:196901` | 0.9698 | 65 | TREC-06 | legit | CS Lists │ PLEASE REMOVE THE UNIVERSITY OF VIRGINIA FROM YOUR MAILING LISTS. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We... |
| 19 | `primary:143547` | 0.9695 | 82 | TREC-06 | legit | How to avoid the result of ODEs prone to blow-up │ Dear Friends, I want to solve ODEs use Interval Arithmeric, but the reult prone to blow-up. Is there any meth... |
| 20 | `primary:129925` | 0.9694 | 255 | TREC-06 | legit | See where your Competion is advertising │ Are your clients: Travelers, Vacationers or Timeshare Owners? TVC...the Timeshare Vacation Club is where YOUR ADS shou... |
| 21 | `primary:092561` | 0.9690 | 1174 | TREC-06 | legit | We Need Homeworkers!! │ Dear Future Associate, You Can Work At Home & Set Your Own Hours. Start earning Big Money in a short time NO Newspaper Advertising! Your... |
| 22 | `primary:182736` | 0.9689 | 68 | TREC-06 | legit | Vesit rover │ Dear friends, You know that we at Vesit, Bombay were making a rover using the handy board. Learn more about it at http://members.tripod.com/ieeeve... |
| 23 | `primary:158244` | 0.9681 | 327 | TREC-06 | legit | Proposition │ HEALTH: A NEW NATURAL HELP To the product Manager The customers demand for natural products is increasing steadily, and Aromatherapy is becoming a... |
| 24 | `primary:152569` | 0.9677 | 171 | TREC-06 | legit | Reliable Computing, Vol.11, issue 3, 2005 │ Reliable Computing Volume 11, issue 3, 2005 Mathematical Research Enclosing Solutions of Singular Interval Systems I... |
| 25 | `primary:011118` | 0.9674 | 143 | TREC-06 | legit |  │ Dear Sir, I am very mingled by the pine system. MY inbox massage are always eatten up without pior notification. All my in-coming mag disappear and nothing i... |
## Three-way (legitimate / phishing / spam)
No three-way ground truth on this split: no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth).
### Predicted kind of the flagged rows (informational — no phishing-vs-spam ground truth on this split)
| group | n | predicted Phishing | predicted Spam |
|---|---:|---:|---:|
| true positives flagged | 3,793 | 974 | 2,819 |
| false positives flagged | 520 | 175 | 345 |
| missed positives (kind the model would have given) | 101 | 35 | 66 |
## Patterns (simple lexical statistics, not a model explanation)
### Tokens over-represented in false negatives vs true positives (document frequency, smoothed ratio)
| token | share of FN docs | share of TP docs | ratio |
|---|---:|---:|---:|
| `smailex` | 0.119 | 0.000 | 934.36 |
| `blocki` | 0.119 | 0.000 | 934.36 |
| `scanmail` | 0.119 | 0.000 | 934.36 |
| `progra` | 0.119 | 0.000 | 934.36 |
| `scanning` | 0.119 | 0.000 | 934.36 |
| `matched` | 0.119 | 0.000 | 311.45 |
| `tex-fonts` | 0.079 | 0.000 | 211.79 |
| `pattern` | 0.129 | 0.001 | 201.82 |
| `blocking` | 0.129 | 0.001 | 201.82 |
| `tpn` | 0.069 | 0.001 | 112.12 |
| `setting` | 0.129 | 0.001 | 91.74 |
| `pif` | 0.079 | 0.001 | 70.60 |
| `detected` | 0.119 | 0.002 | 62.29 |
| `utah` | 0.119 | 0.002 | 62.29 |
| `attachment` | 0.149 | 0.003 | 50.37 |
| `math` | 0.129 | 0.003 | 48.05 |
| `enroll` | 0.050 | 0.001 | 45.68 |
| `seismic` | 0.050 | 0.001 | 45.68 |
| `settings` | 0.129 | 0.003 | 37.37 |
| `moved` | 0.119 | 0.003 | 37.37 |
### Tokens over-represented in false positives vs true negatives
| token | share of FP docs | share of TN docs | ratio |
|---|---:|---:|---:|
| `irc-list-web` | 0.054 | 0.000 | 1301.91 |
| `quarantine` | 0.054 | 0.000 | 118.36 |
| `foundstone` | 0.050 | 0.001 | 80.70 |
| `skidmore` | 0.054 | 0.001 | 76.58 |
| `monty` | 0.052 | 0.001 | 66.12 |
| `bn3` | 0.060 | 0.002 | 27.15 |
| `ips` | 0.054 | 0.002 | 26.57 |
| `anyplace` | 0.060 | 0.002 | 26.16 |
| `noc's` | 0.054 | 0.002 | 23.67 |
| `https` | 0.108 | 0.006 | 19.12 |
| `cgi` | 0.060 | 0.004 | 16.17 |
| `procexp` | 0.054 | 0.003 | 15.69 |
| `tcpview` | 0.054 | 0.003 | 15.69 |
| `sysinternals` | 0.054 | 0.004 | 15.32 |
| `filtered` | 0.052 | 0.003 | 15.14 |
| `atkins` | 0.054 | 0.004 | 14.63 |
| `rogue` | 0.056 | 0.004 | 14.49 |
| `firewall` | 0.071 | 0.005 | 13.93 |
| `inbox` | 0.058 | 0.004 | 13.79 |
| `manage` | 0.079 | 0.006 | 13.64 |
### Composition of the error groups (share of rows)
| group | n | mailing-list / digest | HTML-heavy | very short (<20 content tokens) | truncated (>512) | lure phrase |
|---|---:|---:|---:|---:|---:|---:|
| FP | 520 | 19.6% | 0.0% | 2.3% | 29.0% | 1.2% |
| TN | 11,888 | 29.7% | 0.2% | 0.4% | 23.8% | 1.3% |
| FN | 101 | 8.9% | 0.0% | 0.0% | 12.9% | 1.0% |
| TP | 3,793 | 2.8% | 0.1% | 0.2% | 15.3% | 6.4% |
### Length confound (RUN_LOG entry 2)
| group | n | mean tokens (full) | mean tokens fed | median (full) | % truncated |
|---|---:|---:|---:|---:|---:|
| TP | 3,793 | 283.9 | 232.2 | 207 | 15.3 |
| TN | 11,888 | 827.0 | 294.4 | 269 | 23.8 |
| FP | 520 | 1237.2 | 276.6 | 242 | 29.0 |
| FN | 101 | 321.7 | 220.4 | 181 | 12.9 |
| CORRECT | 15,681 | 695.6 | 279.4 | 247 | 21.8 |
| WRONG | 621 | 1088.3 | 267.5 | 213 | 26.4 |
| all legitimate | 12,408 | 844.2 | 293.7 | 268 | 24.0 |
| all unwanted | 3,894 | 284.9 | 231.9 | 206 | 15.3 |
| all label3=unwanted | 3,894 | 284.9 | 231.9 | 206 | 15.3 |
| length bucket (full tokens) | n | positive rate | recall | FPR | accuracy |
|---|---:|---:|---:|---:|---:|
| <=64 | 1,363 | 0.431 | 0.9744 | 0.1070 | 0.9281 |
| 65-128 | 2,538 | 0.290 | 0.9620 | 0.0394 | 0.9610 |
| 129-256 | 4,594 | 0.260 | 0.9766 | 0.0341 | 0.9687 |
| 257-512 | 4,232 | 0.184 | 0.9782 | 0.0287 | 0.9726 |
| >512 | 3,575 | 0.166 | 0.9782 | 0.0507 | 0.9541 |
## Summary
- The classifier catches **97.41% of unwanted mail (phishing or spam)** at threshold 0.5 (101 of 3,894 missed) while flagging 4.19% of legitimate mail (520 false alarms). Recall is the headline because a missed phish is the costly error for a small-business inbox.
- Holding false alarms to 1 in 100 legitimate emails keeps recall at 89.55% (threshold 0.866, flag when p >= threshold); at 1 in 1,000 it is 64.97%. The best-F1 threshold is 0.682 (F1 0.9397).
- Three-way: this split has no phishing-vs-spam ground truth; of the 3,793 true positives the model flagged, it called 974 phishing and 2,819 spam (informational only).
- On the 398-row credential-lure slice (37 documented phrases, 61.1% of its rows are labelled positive) recall is 99.59% with precision 97.58%: the highest-cost phish are caught at least as well as the average.
- Length confound: missed unwanted mail averages 322 tokens vs 284 for caught; false positives average 1237 tokens vs 827 for correct legitimate mail (all legitimate 844, all unwanted 285).
  Legitimate mail is ~3.0x longer than unwanted mail on average; the error groups do not follow the length gap, so length is not the main shortcut.
- Recall by length is lowest in the 65-128-token bucket (96.2%, 736 positives).
- Of the 520 false positives, 19.6% look like mailing-list or digest traffic, 0.0% are HTML-heavy and 2.3% are very short (TN reference: 29.7% / 0.2% / 0.4%).
- Tokens most over-represented in missed unwanted mail vs caught: `smailex`, `blocki`, `scanmail`, `progra`, `scanning`, `matched`, `tex-fonts`, `pattern`, `blocking`, `tpn`.
- Tokens most over-represented in false positives vs correct legitimate mail: `irc-list-web`, `quarantine`, `foundstone`, `skidmore`, `monty`, `bn3`, `ips`, `anyplace`, `noc's`, `https`.

	# Error analysis — test split (TREC-06 binary holdout), run `ft-focal` (2026-10-09 10:46)

	All quoted emails are rows of public corpora (source TREC-06); nothing here is private mail. Threshold 0.5. Binary positive class = Unwanted (phishing or spam), P(unwanted) = P(phishing) + P(spam).

	## Counts (binary roll-up)

	- rows 16,302 (positives 3,894, 23.9%); label3 counts legit 12,408, unwanted 3,894
	- recall (unwanted) 0.9741 — 101 missed unwanted emails (false negatives) out of 3,894
	- precision 0.8794 — 520 false alarms out of 4,313 flagged
	- F1 0.9243, accuracy 0.9619, specificity 0.9581
	- ROC-AUC 0.9946, PR-AUC 0.9848; recall at 1% FPR 0.8955, at 0.1% FPR 0.6497
	- credential-lure slice: 398 rows, 243 positive (61.1% label purity: slice precision cannot exceed what the phrase list selects), recall 0.9959, precision 0.9758, 1 missed

	## Top-25 false negatives (missed unwanted mail, most confident first)

	\| # \| doc_id \| P(unwanted) \| tokens (full) \| source \| label_set \| preview (subject │ body, 160 chars) \|
	\|---:\|---\|---:\|---:\|---\|---\|---\|
	\| 1 \| `primary:052869` \| 0.0322 \| 630 \| TREC-06 \| unwanted \| READ THIS! │ >>Read this! >>> >> > >> > > >>> >> > >> > > ---------- >>> >> > >> > > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > > > Subject: Lets see if this... \|
	\| 2 \| `primary:080991` \| 0.0428 \| 75 \| TREC-06 \| unwanted \| prepare urself for this │ It is been awhile. Hope everythign with you is goign well. How is work going also? I've just started with this new firm, www.andwhatis... \|
	\| 3 \| `primary:086987` \| 0.0506 \| 188 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 4 \| `primary:056767` \| 0.0564 \| 181 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 5 \| `primary:121268` \| 0.0564 \| 185 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 6 \| `primary:173636` \| 0.0567 \| 55 \| TREC-06 \| unwanted \| everything ok │ Looking to increase your johnson. Vist at http://www.mydkillsanddoc.com/taw/ to settle that. that that Hardly were there is conquering Thanks. B... \|
	\| 7 \| `primary:020947` \| 0.0629 \| 113 \| TREC-06 \| unwanted \| everything kool │ Hi Reg Here's some pretty hard facts to deal with that the majority of women are unhappy with the size of there signif other's thing. Don't be... \|
	\| 8 \| `primary:072993` \| 0.0710 \| 186 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 9 \| `primary:044992` \| 0.0726 \| 76 \| TREC-06 \| unwanted \| FW:any new news? i got some │ is hand the language care to cruel. must plate to design tax but fiction. if snow but question great the separate. but comparison ... \|
	\| 10 \| `primary:024565` \| 0.0748 \| 190 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 11 \| `primary:076023` \| 0.0867 \| 54 \| TREC-06 \| unwanted \| everything kool │ Hey Mostyn, Ramon wanted to me to send you an email about the store I went to buy my goods, http://www.fabloulusil.com/tpn/. Carlos \|
	\| 12 \| `primary:129745` \| 0.0946 \| 184 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 13 \| `primary:071577` \| 0.0961 \| 162 \| TREC-06 \| unwanted \| (no subject) │ Re: Student Packets Welcome to winter term student packet preparations! First: Fax (485-6253) or email (Copy-Shop@worldnet.att.net) JUST your bib... \|
	\| 14 \| `primary:173938` \| 0.0961 \| 185 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 15 \| `primary:030013` \| 0.0966 \| 195 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 16 \| `primary:063077` \| 0.0982 \| 344 \| TREC-06 \| unwanted \| Party for Prisoners of Conscience - PLEASE SEND TO YOUR ENTIRE CLUB │ Amnesty International and the Intercultural Resource Center are holding a party to raise m... \|
	\| 17 \| `primary:145723` \| 0.1007 \| 25 \| TREC-06 \| unwanted \| refresh my memory and mental energy │ may revision a golf on appendage or eagle try parsimony \|
	\| 18 \| `primary:154489` \| 0.1026 \| 187 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 19 \| `primary:186523` \| 0.1041 \| 185 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 20 \| `primary:147887` \| 0.1150 \| 181 \| TREC-06 \| unwanted \| ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... \|
	\| 21 \| `primary:033014` \| 0.1157 \| 77 \| TREC-06 \| unwanted \| isn't it time │ Allen mentioned me that u were seeking for a place to get ur stuff for ur sickness. http://www.whorunoverforitsl.org/tpn/ the place then for sur... \|
	\| 22 \| `primary:156328` \| 0.1157 \| 77 \| TREC-06 \| unwanted \| isn't it time │ Allen mentioned me that u were seeking for a place to get ur stuff for ur sickness. http://www.whorunoverforitsl.org/tpn/ the place then for sur... \|
	\| 23 \| `primary:143065` \| 0.1308 \| 102 \| TREC-06 \| unwanted \| everything kool │ Heya Alexa, Apologize for how long its been since we've talked. I take blame for that. I just wanted to share with you the place that made me ... \|
	\| 24 \| `primary:147193` \| 0.1323 \| 260 \| TREC-06 \| unwanted \| ADV Textbooks, Tuesday & Wednesday, University of Oregon │ Dear Professor XXXXXXXX., I am going to be on your campus to buy textbooks this Tuesday, June 1st and... \|
	\| 25 \| `primary:101618` \| 0.1387 \| 117 \| TREC-06 \| unwanted \| RE: If you are studing or working │ Dear Mr(Mrs), Iris!!. Our company Barcelo Travel Inc. have a great job offer for peoples from Australia. Your salary per wee... \|

	## Top-25 false positives (legitimate flagged, most confident first)

	\| # \| doc_id \| P(unwanted) \| tokens (full) \| source \| label_set \| preview (subject │ body, 160 chars) \|
	\|---:\|---\|---:\|---:\|---\|---\|---\|
	\| 1 \| `primary:015304` \| 0.9874 \| 3712 \| TREC-06 \| legit \| I made $230,000 in two months │ You owe it to yourself and your wealth to read this. AS SEEN ON NATIONAL TV: "Making over half a million dollars every 4 to 5 mo... \|
	\| 2 \| `primary:040048` \| 0.9871 \| 80 \| TREC-06 \| legit \| Enter to win at mary-kateandashley.com! │ Dear val: Have you heard about the sweepstakes at mary-kateandashley.com? I entered to win and you can too! Check it o... \|
	\| 3 \| `primary:194751` \| 0.9842 \| 48 \| TREC-06 \| legit \| Thank you │ Turns out I just needed a common ground between servos and HB Thanx all! :) Alex ______________________________________________________ Get Your Pri... \|
	\| 4 \| `primary:072489` \| 0.9831 \| 58 \| TREC-06 \| legit \| Re: Junk email │ Please REMOVE this address from your list!!!! ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you b... \|
	\| 5 \| `primary:194063` \| 0.9781 \| 32 \| TREC-06 \| legit \| info │ Here's an affordable robot that anyone might be interested: http://www.personalrobots.com/ joema \|
	\| 6 \| `primary:033174` \| 0.9781 \| 602 \| TREC-06 \| legit \| You are Approved for Merchant Account!! │ INCREASE SALES UP TO 1500% ACCEPT CREDIT CARDS OVER THE INTERNET *** NO SETUP FEES Good Credit / Bad Credit/ No Credit... \|
	\| 7 \| `primary:101560` \| 0.9769 \| 59 \| TREC-06 \| legit \| │ Please, please, please remove us from your mailing list. Thank you. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can l... \|
	\| 8 \| `primary:159590` \| 0.9764 \| 56 \| TREC-06 \| legit \| remove me │ Please remove Brandeis University from your list ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you bec... \|
	\| 9 \| `primary:184838` \| 0.9757 \| 66 \| TREC-06 \| legit \| unwanted mail │ how cani stop all the mail from these sex sites i dont want them but when i try to e-mail asking them to stop i keep getting a message that say ... \|
	\| 10 \| `primary:011014` \| 0.9736 \| 46 \| TREC-06 \| legit \| composer-wrap-column │ hi, anyone knows how to entirely turn off composer wrapping in pine? thanks! viinikala kauas pilvet karkaavat. \|
	\| 11 \| `primary:193401` \| 0.9733 \| 870 \| TREC-06 \| legit \| [9fans] 57 Million Email Addresses - ONLY $149!! │ 57 MILLION EMAIL ADDRESSES FOR ONLY $149 You want to make some money? I can put you in touch with over 50 mil... \|
	\| 12 \| `primary:072275` \| 0.9729 \| 41 \| TREC-06 \| legit \| HB weight │ does anyone know the weight of a handy board? thanks, -Brian --------------------------------------------------- Get free personalized email at http... \|
	\| 13 \| `primary:170605` \| 0.9729 \| 55 \| TREC-06 \| legit \| List │ Please remove my name from your list serve. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you become anybod... \|
	\| 14 \| `primary:175533` \| 0.9726 \| 62 \| TREC-06 \| legit \| Test │ I'm not sure why we are receiving mail addressed to cslist@fudan.edu ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We ... \|
	\| 15 \| `primary:131182` \| 0.9719 \| 31 \| TREC-06 \| legit \| │ unsubscribe --------------------------------------------------------------- Octopus traps summer's moonspun dreams soon fade away. Basho 1644 \|
	\| 16 \| `primary:106445` \| 0.9717 \| 61 \| TREC-06 \| legit \| REMOVE Clemson University │ PLEASE REMOVE Clemson University from your mailing lists. ________________ Email Forwarding Service Brought To You By: http://www.bn... \|
	\| 17 \| `primary:078484` \| 0.9706 \| 77 \| TREC-06 \| legit \| remove from list │ Please remove Brandeis University from your graduate school mailing list. maf@cs.brandeis.edu thank you ________________ Email Forwarding Ser... \|
	\| 18 \| `primary:196901` \| 0.9698 \| 65 \| TREC-06 \| legit \| CS Lists │ PLEASE REMOVE THE UNIVERSITY OF VIRGINIA FROM YOUR MAILING LISTS. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We... \|
	\| 19 \| `primary:143547` \| 0.9695 \| 82 \| TREC-06 \| legit \| How to avoid the result of ODEs prone to blow-up │ Dear Friends, I want to solve ODEs use Interval Arithmeric, but the reult prone to blow-up. Is there any meth... \|
	\| 20 \| `primary:129925` \| 0.9694 \| 255 \| TREC-06 \| legit \| See where your Competion is advertising │ Are your clients: Travelers, Vacationers or Timeshare Owners? TVC...the Timeshare Vacation Club is where YOUR ADS shou... \|
	\| 21 \| `primary:092561` \| 0.9690 \| 1174 \| TREC-06 \| legit \| We Need Homeworkers!! │ Dear Future Associate, You Can Work At Home & Set Your Own Hours. Start earning Big Money in a short time NO Newspaper Advertising! Your... \|
	\| 22 \| `primary:182736` \| 0.9689 \| 68 \| TREC-06 \| legit \| Vesit rover │ Dear friends, You know that we at Vesit, Bombay were making a rover using the handy board. Learn more about it at http://members.tripod.com/ieeeve... \|
	\| 23 \| `primary:158244` \| 0.9681 \| 327 \| TREC-06 \| legit \| Proposition │ HEALTH: A NEW NATURAL HELP To the product Manager The customers demand for natural products is increasing steadily, and Aromatherapy is becoming a... \|
	\| 24 \| `primary:152569` \| 0.9677 \| 171 \| TREC-06 \| legit \| Reliable Computing, Vol.11, issue 3, 2005 │ Reliable Computing Volume 11, issue 3, 2005 Mathematical Research Enclosing Solutions of Singular Interval Systems I... \|
	\| 25 \| `primary:011118` \| 0.9674 \| 143 \| TREC-06 \| legit \| │ Dear Sir, I am very mingled by the pine system. MY inbox massage are always eatten up without pior notification. All my in-coming mag disappear and nothing i... \|

	## Three-way (legitimate / phishing / spam)

	No three-way ground truth on this split: no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth).

	### Predicted kind of the flagged rows (informational — no phishing-vs-spam ground truth on this split)

	\| group \| n \| predicted Phishing \| predicted Spam \|
	\|---\|---:\|---:\|---:\|
	\| true positives flagged \| 3,793 \| 974 \| 2,819 \|
	\| false positives flagged \| 520 \| 175 \| 345 \|
	\| missed positives (kind the model would have given) \| 101 \| 35 \| 66 \|

	## Patterns (simple lexical statistics, not a model explanation)

	### Tokens over-represented in false negatives vs true positives (document frequency, smoothed ratio)

	\| token \| share of FN docs \| share of TP docs \| ratio \|
	\|---\|---:\|---:\|---:\|
	\| `smailex` \| 0.119 \| 0.000 \| 934.36 \|
	\| `blocki` \| 0.119 \| 0.000 \| 934.36 \|
	\| `scanmail` \| 0.119 \| 0.000 \| 934.36 \|
	\| `progra` \| 0.119 \| 0.000 \| 934.36 \|
	\| `scanning` \| 0.119 \| 0.000 \| 934.36 \|
	\| `matched` \| 0.119 \| 0.000 \| 311.45 \|
	\| `tex-fonts` \| 0.079 \| 0.000 \| 211.79 \|
	\| `pattern` \| 0.129 \| 0.001 \| 201.82 \|
	\| `blocking` \| 0.129 \| 0.001 \| 201.82 \|
	\| `tpn` \| 0.069 \| 0.001 \| 112.12 \|
	\| `setting` \| 0.129 \| 0.001 \| 91.74 \|
	\| `pif` \| 0.079 \| 0.001 \| 70.60 \|
	\| `detected` \| 0.119 \| 0.002 \| 62.29 \|
	\| `utah` \| 0.119 \| 0.002 \| 62.29 \|
	\| `attachment` \| 0.149 \| 0.003 \| 50.37 \|
	\| `math` \| 0.129 \| 0.003 \| 48.05 \|
	\| `enroll` \| 0.050 \| 0.001 \| 45.68 \|
	\| `seismic` \| 0.050 \| 0.001 \| 45.68 \|
	\| `settings` \| 0.129 \| 0.003 \| 37.37 \|
	\| `moved` \| 0.119 \| 0.003 \| 37.37 \|

	### Tokens over-represented in false positives vs true negatives

	\| token \| share of FP docs \| share of TN docs \| ratio \|
	\|---\|---:\|---:\|---:\|
	\| `irc-list-web` \| 0.054 \| 0.000 \| 1301.91 \|
	\| `quarantine` \| 0.054 \| 0.000 \| 118.36 \|
	\| `foundstone` \| 0.050 \| 0.001 \| 80.70 \|
	\| `skidmore` \| 0.054 \| 0.001 \| 76.58 \|
	\| `monty` \| 0.052 \| 0.001 \| 66.12 \|
	\| `bn3` \| 0.060 \| 0.002 \| 27.15 \|
	\| `ips` \| 0.054 \| 0.002 \| 26.57 \|
	\| `anyplace` \| 0.060 \| 0.002 \| 26.16 \|
	\| `noc's` \| 0.054 \| 0.002 \| 23.67 \|
	\| `https` \| 0.108 \| 0.006 \| 19.12 \|
	\| `cgi` \| 0.060 \| 0.004 \| 16.17 \|
	\| `procexp` \| 0.054 \| 0.003 \| 15.69 \|
	\| `tcpview` \| 0.054 \| 0.003 \| 15.69 \|
	\| `sysinternals` \| 0.054 \| 0.004 \| 15.32 \|
	\| `filtered` \| 0.052 \| 0.003 \| 15.14 \|
	\| `atkins` \| 0.054 \| 0.004 \| 14.63 \|
	\| `rogue` \| 0.056 \| 0.004 \| 14.49 \|
	\| `firewall` \| 0.071 \| 0.005 \| 13.93 \|
	\| `inbox` \| 0.058 \| 0.004 \| 13.79 \|
	\| `manage` \| 0.079 \| 0.006 \| 13.64 \|

	### Composition of the error groups (share of rows)

	\| group \| n \| mailing-list / digest \| HTML-heavy \| very short (<20 content tokens) \| truncated (>512) \| lure phrase \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| FP \| 520 \| 19.6% \| 0.0% \| 2.3% \| 29.0% \| 1.2% \|
	\| TN \| 11,888 \| 29.7% \| 0.2% \| 0.4% \| 23.8% \| 1.3% \|
	\| FN \| 101 \| 8.9% \| 0.0% \| 0.0% \| 12.9% \| 1.0% \|
	\| TP \| 3,793 \| 2.8% \| 0.1% \| 0.2% \| 15.3% \| 6.4% \|

	### Length confound (RUN_LOG entry 2)

	\| group \| n \| mean tokens (full) \| mean tokens fed \| median (full) \| % truncated \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| TP \| 3,793 \| 283.9 \| 232.2 \| 207 \| 15.3 \|
	\| TN \| 11,888 \| 827.0 \| 294.4 \| 269 \| 23.8 \|
	\| FP \| 520 \| 1237.2 \| 276.6 \| 242 \| 29.0 \|
	\| FN \| 101 \| 321.7 \| 220.4 \| 181 \| 12.9 \|
	\| CORRECT \| 15,681 \| 695.6 \| 279.4 \| 247 \| 21.8 \|
	\| WRONG \| 621 \| 1088.3 \| 267.5 \| 213 \| 26.4 \|
	\| all legitimate \| 12,408 \| 844.2 \| 293.7 \| 268 \| 24.0 \|
	\| all unwanted \| 3,894 \| 284.9 \| 231.9 \| 206 \| 15.3 \|
	\| all label3=unwanted \| 3,894 \| 284.9 \| 231.9 \| 206 \| 15.3 \|

	\| length bucket (full tokens) \| n \| positive rate \| recall \| FPR \| accuracy \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| <=64 \| 1,363 \| 0.431 \| 0.9744 \| 0.1070 \| 0.9281 \|
	\| 65-128 \| 2,538 \| 0.290 \| 0.9620 \| 0.0394 \| 0.9610 \|
	\| 129-256 \| 4,594 \| 0.260 \| 0.9766 \| 0.0341 \| 0.9687 \|
	\| 257-512 \| 4,232 \| 0.184 \| 0.9782 \| 0.0287 \| 0.9726 \|
	\| >512 \| 3,575 \| 0.166 \| 0.9782 \| 0.0507 \| 0.9541 \|

	## Summary

	- The classifier catches 97.41% of unwanted mail (phishing or spam) at threshold 0.5 (101 of 3,894 missed) while flagging 4.19% of legitimate mail (520 false alarms). Recall is the headline because a missed phish is the costly error for a small-business inbox.
	- Holding false alarms to 1 in 100 legitimate emails keeps recall at 89.55% (threshold 0.866, flag when p >= threshold); at 1 in 1,000 it is 64.97%. The best-F1 threshold is 0.682 (F1 0.9397).
	- Three-way: this split has no phishing-vs-spam ground truth; of the 3,793 true positives the model flagged, it called 974 phishing and 2,819 spam (informational only).
	- On the 398-row credential-lure slice (37 documented phrases, 61.1% of its rows are labelled positive) recall is 99.59% with precision 97.58%: the highest-cost phish are caught at least as well as the average.
	- Length confound: missed unwanted mail averages 322 tokens vs 284 for caught; false positives average 1237 tokens vs 827 for correct legitimate mail (all legitimate 844, all unwanted 285).
	Legitimate mail is ~3.0x longer than unwanted mail on average; the error groups do not follow the length gap, so length is not the main shortcut.
	- Recall by length is lowest in the 65-128-token bucket (96.2%, 736 positives).
	- Of the 520 false positives, 19.6% look like mailing-list or digest traffic, 0.0% are HTML-heavy and 2.3% are very short (TN reference: 29.7% / 0.2% / 0.4%).
	- Tokens most over-represented in missed unwanted mail vs caught: `smailex`, `blocki`, `scanmail`, `progra`, `scanning`, `matched`, `tex-fonts`, `pattern`, `blocking`, `tpn`.
	- Tokens most over-represented in false positives vs correct legitimate mail: `irc-list-web`, `quarantine`, `foundstone`, `skidmore`, `monty`, `bn3`, `ips`, `anyplace`, `noc's`, `https`.