Error analysis — test split (TREC-06 binary holdout), run ft-focal (2026-10-09 10:46)
All quoted emails are rows of public corpora (source TREC-06); nothing here is private mail. Threshold 0.5. Binary positive class = Unwanted (phishing or spam), P(unwanted) = P(phishing) + P(spam).
Counts (binary roll-up)
- rows 16,302 (positives 3,894, 23.9%); label3 counts legit 12,408, unwanted 3,894
- recall (unwanted) 0.9741 — 101 missed unwanted emails (false negatives) out of 3,894
- precision 0.8794 — 520 false alarms out of 4,313 flagged
- F1 0.9243, accuracy 0.9619, specificity 0.9581
- ROC-AUC 0.9946, PR-AUC 0.9848; recall at 1% FPR 0.8955, at 0.1% FPR 0.6497
- credential-lure slice: 398 rows, 243 positive (61.1% label purity: slice precision cannot exceed what the phrase list selects), recall 0.9959, precision 0.9758, 1 missed
Top-25 false negatives (missed unwanted mail, most confident first)
| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---|---|---|---|---|---|---|
| 1 | primary:052869 |
0.0322 | 630 | TREC-06 | unwanted | READ THIS! │ >>Read this! >>> >> > >> > > >>> >> > >> > > ---------- >>> >> > >> > > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > > > Subject: Lets see if this... |
| 2 | primary:080991 |
0.0428 | 75 | TREC-06 | unwanted | prepare urself for this │ It is been awhile. Hope everythign with you is goign well. How is work going also? I've just started with this new firm, www.andwhatis... |
| 3 | primary:086987 |
0.0506 | 188 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 4 | primary:056767 |
0.0564 | 181 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 5 | primary:121268 |
0.0564 | 185 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 6 | primary:173636 |
0.0567 | 55 | TREC-06 | unwanted | everything ok │ Looking to increase your johnson. Vist at http://www.mydkillsanddoc.com/taw/ to settle that. that that Hardly were there is conquering Thanks. B... |
| 7 | primary:020947 |
0.0629 | 113 | TREC-06 | unwanted | everything kool │ Hi Reg Here's some pretty hard facts to deal with that the majority of women are unhappy with the size of there signif other's thing. Don't be... |
| 8 | primary:072993 |
0.0710 | 186 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 9 | primary:044992 |
0.0726 | 76 | TREC-06 | unwanted | FW:any new news? i got some │ is hand the language care to cruel. must plate to design tax but fiction. if snow but question great the separate. but comparison ... |
| 10 | primary:024565 |
0.0748 | 190 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 11 | primary:076023 |
0.0867 | 54 | TREC-06 | unwanted | everything kool │ Hey Mostyn, Ramon wanted to me to send you an email about the store I went to buy my goods, http://www.fabloulusil.com/tpn/. Carlos |
| 12 | primary:129745 |
0.0946 | 184 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 13 | primary:071577 |
0.0961 | 162 | TREC-06 | unwanted | (no subject) │ Re: Student Packets Welcome to winter term student packet preparations! First: Fax (485-6253) or email (Copy-Shop@worldnet.att.net) JUST your bib... |
| 14 | primary:173938 |
0.0961 | 185 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 15 | primary:030013 |
0.0966 | 195 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 16 | primary:063077 |
0.0982 | 344 | TREC-06 | unwanted | Party for Prisoners of Conscience - PLEASE SEND TO YOUR ENTIRE CLUB │ Amnesty International and the Intercultural Resource Center are holding a party to raise m... |
| 17 | primary:145723 |
0.1007 | 25 | TREC-06 | unwanted | refresh my memory and mental energy │ may revision a golf on appendage or eagle try parsimony |
| 18 | primary:154489 |
0.1026 | 187 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 19 | primary:186523 |
0.1041 | 185 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 20 | primary:147887 |
0.1150 | 181 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 21 | primary:033014 |
0.1157 | 77 | TREC-06 | unwanted | isn't it time │ Allen mentioned me that u were seeking for a place to get ur stuff for ur sickness. http://www.whorunoverforitsl.org/tpn/ the place then for sur... |
| 22 | primary:156328 |
0.1157 | 77 | TREC-06 | unwanted | isn't it time │ Allen mentioned me that u were seeking for a place to get ur stuff for ur sickness. http://www.whorunoverforitsl.org/tpn/ the place then for sur... |
| 23 | primary:143065 |
0.1308 | 102 | TREC-06 | unwanted | everything kool │ Heya Alexa, Apologize for how long its been since we've talked. I take blame for that. I just wanted to share with you the place that made me ... |
| 24 | primary:147193 |
0.1323 | 260 | TREC-06 | unwanted | ADV Textbooks, Tuesday & Wednesday, University of Oregon │ Dear Professor XXXXXXXX., I am going to be on your campus to buy textbooks this Tuesday, June 1st and... |
| 25 | primary:101618 |
0.1387 | 117 | TREC-06 | unwanted | RE: If you are studing or working │ Dear Mr(Mrs), Iris!!. Our company Barcelo Travel Inc. have a great job offer for peoples from Australia. Your salary per wee... |
Top-25 false positives (legitimate flagged, most confident first)
| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---|---|---|---|---|---|---|
| 1 | primary:015304 |
0.9874 | 3712 | TREC-06 | legit | I made $230,000 in two months │ You owe it to yourself and your wealth to read this. AS SEEN ON NATIONAL TV: "Making over half a million dollars every 4 to 5 mo... |
| 2 | primary:040048 |
0.9871 | 80 | TREC-06 | legit | Enter to win at mary-kateandashley.com! │ Dear val: Have you heard about the sweepstakes at mary-kateandashley.com? I entered to win and you can too! Check it o... |
| 3 | primary:194751 |
0.9842 | 48 | TREC-06 | legit | Thank you │ Turns out I just needed a common ground between servos and HB Thanx all! :) Alex ______________________________________________________ Get Your Pri... |
| 4 | primary:072489 |
0.9831 | 58 | TREC-06 | legit | Re: Junk email │ Please REMOVE this address from your list!!!! ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you b... |
| 5 | primary:194063 |
0.9781 | 32 | TREC-06 | legit | info │ Here's an affordable robot that anyone might be interested: http://www.personalrobots.com/ joema |
| 6 | primary:033174 |
0.9781 | 602 | TREC-06 | legit | You are Approved for Merchant Account!! │ INCREASE SALES UP TO 1500% ACCEPT CREDIT CARDS OVER THE INTERNET *** NO SETUP FEES Good Credit / Bad Credit/ No Credit... |
| 7 | primary:101560 |
0.9769 | 59 | TREC-06 | legit | │ Please, please, please remove us from your mailing list. Thank you. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can l... |
| 8 | primary:159590 |
0.9764 | 56 | TREC-06 | legit | remove me │ Please remove Brandeis University from your list ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you bec... |
| 9 | primary:184838 |
0.9757 | 66 | TREC-06 | legit | unwanted mail │ how cani stop all the mail from these sex sites i dont want them but when i try to e-mail asking them to stop i keep getting a message that say ... |
| 10 | primary:011014 |
0.9736 | 46 | TREC-06 | legit | composer-wrap-column │ hi, anyone knows how to entirely turn off composer wrapping in pine? thanks! viinikala kauas pilvet karkaavat. |
| 11 | primary:193401 |
0.9733 | 870 | TREC-06 | legit | [9fans] 57 Million Email Addresses - ONLY $149!! │ 57 MILLION EMAIL ADDRESSES FOR ONLY $149 You want to make some money? I can put you in touch with over 50 mil... |
| 12 | primary:072275 |
0.9729 | 41 | TREC-06 | legit | HB weight │ does anyone know the weight of a handy board? thanks, -Brian --------------------------------------------------- Get free personalized email at http... |
| 13 | primary:170605 |
0.9729 | 55 | TREC-06 | legit | List │ Please remove my name from your list serve. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you become anybod... |
| 14 | primary:175533 |
0.9726 | 62 | TREC-06 | legit | Test │ I'm not sure why we are receiving mail addressed to cslist@fudan.edu ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We ... |
| 15 | primary:131182 |
0.9719 | 31 | TREC-06 | legit | │ unsubscribe --------------------------------------------------------------- Octopus traps summer's moonspun dreams soon fade away. Basho 1644 |
| 16 | primary:106445 |
0.9717 | 61 | TREC-06 | legit | REMOVE Clemson University │ PLEASE REMOVE Clemson University from your mailing lists. ________________ Email Forwarding Service Brought To You By: http://www.bn... |
| 17 | primary:078484 |
0.9706 | 77 | TREC-06 | legit | remove from list │ Please remove Brandeis University from your graduate school mailing list. maf@cs.brandeis.edu thank you ________________ Email Forwarding Ser... |
| 18 | primary:196901 |
0.9698 | 65 | TREC-06 | legit | CS Lists │ PLEASE REMOVE THE UNIVERSITY OF VIRGINIA FROM YOUR MAILING LISTS. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We... |
| 19 | primary:143547 |
0.9695 | 82 | TREC-06 | legit | How to avoid the result of ODEs prone to blow-up │ Dear Friends, I want to solve ODEs use Interval Arithmeric, but the reult prone to blow-up. Is there any meth... |
| 20 | primary:129925 |
0.9694 | 255 | TREC-06 | legit | See where your Competion is advertising │ Are your clients: Travelers, Vacationers or Timeshare Owners? TVC...the Timeshare Vacation Club is where YOUR ADS shou... |
| 21 | primary:092561 |
0.9690 | 1174 | TREC-06 | legit | We Need Homeworkers!! │ Dear Future Associate, You Can Work At Home & Set Your Own Hours. Start earning Big Money in a short time NO Newspaper Advertising! Your... |
| 22 | primary:182736 |
0.9689 | 68 | TREC-06 | legit | Vesit rover │ Dear friends, You know that we at Vesit, Bombay were making a rover using the handy board. Learn more about it at http://members.tripod.com/ieeeve... |
| 23 | primary:158244 |
0.9681 | 327 | TREC-06 | legit | Proposition │ HEALTH: A NEW NATURAL HELP To the product Manager The customers demand for natural products is increasing steadily, and Aromatherapy is becoming a... |
| 24 | primary:152569 |
0.9677 | 171 | TREC-06 | legit | Reliable Computing, Vol.11, issue 3, 2005 │ Reliable Computing Volume 11, issue 3, 2005 Mathematical Research Enclosing Solutions of Singular Interval Systems I... |
| 25 | primary:011118 |
0.9674 | 143 | TREC-06 | legit | │ Dear Sir, I am very mingled by the pine system. MY inbox massage are always eatten up without pior notification. All my in-coming mag disappear and nothing i... |
Three-way (legitimate / phishing / spam)
No three-way ground truth on this split: no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth).
Predicted kind of the flagged rows (informational — no phishing-vs-spam ground truth on this split)
| group | n | predicted Phishing | predicted Spam |
|---|---|---|---|
| true positives flagged | 3,793 | 974 | 2,819 |
| false positives flagged | 520 | 175 | 345 |
| missed positives (kind the model would have given) | 101 | 35 | 66 |
Patterns (simple lexical statistics, not a model explanation)
Tokens over-represented in false negatives vs true positives (document frequency, smoothed ratio)
| token | share of FN docs | share of TP docs | ratio |
|---|---|---|---|
smailex |
0.119 | 0.000 | 934.36 |
blocki |
0.119 | 0.000 | 934.36 |
scanmail |
0.119 | 0.000 | 934.36 |
progra |
0.119 | 0.000 | 934.36 |
scanning |
0.119 | 0.000 | 934.36 |
matched |
0.119 | 0.000 | 311.45 |
tex-fonts |
0.079 | 0.000 | 211.79 |
pattern |
0.129 | 0.001 | 201.82 |
blocking |
0.129 | 0.001 | 201.82 |
tpn |
0.069 | 0.001 | 112.12 |
setting |
0.129 | 0.001 | 91.74 |
pif |
0.079 | 0.001 | 70.60 |
detected |
0.119 | 0.002 | 62.29 |
utah |
0.119 | 0.002 | 62.29 |
attachment |
0.149 | 0.003 | 50.37 |
math |
0.129 | 0.003 | 48.05 |
enroll |
0.050 | 0.001 | 45.68 |
seismic |
0.050 | 0.001 | 45.68 |
settings |
0.129 | 0.003 | 37.37 |
moved |
0.119 | 0.003 | 37.37 |
Tokens over-represented in false positives vs true negatives
| token | share of FP docs | share of TN docs | ratio |
|---|---|---|---|
irc-list-web |
0.054 | 0.000 | 1301.91 |
quarantine |
0.054 | 0.000 | 118.36 |
foundstone |
0.050 | 0.001 | 80.70 |
skidmore |
0.054 | 0.001 | 76.58 |
monty |
0.052 | 0.001 | 66.12 |
bn3 |
0.060 | 0.002 | 27.15 |
ips |
0.054 | 0.002 | 26.57 |
anyplace |
0.060 | 0.002 | 26.16 |
noc's |
0.054 | 0.002 | 23.67 |
https |
0.108 | 0.006 | 19.12 |
cgi |
0.060 | 0.004 | 16.17 |
procexp |
0.054 | 0.003 | 15.69 |
tcpview |
0.054 | 0.003 | 15.69 |
sysinternals |
0.054 | 0.004 | 15.32 |
filtered |
0.052 | 0.003 | 15.14 |
atkins |
0.054 | 0.004 | 14.63 |
rogue |
0.056 | 0.004 | 14.49 |
firewall |
0.071 | 0.005 | 13.93 |
inbox |
0.058 | 0.004 | 13.79 |
manage |
0.079 | 0.006 | 13.64 |
Composition of the error groups (share of rows)
| group | n | mailing-list / digest | HTML-heavy | very short (<20 content tokens) | truncated (>512) | lure phrase |
|---|---|---|---|---|---|---|
| FP | 520 | 19.6% | 0.0% | 2.3% | 29.0% | 1.2% |
| TN | 11,888 | 29.7% | 0.2% | 0.4% | 23.8% | 1.3% |
| FN | 101 | 8.9% | 0.0% | 0.0% | 12.9% | 1.0% |
| TP | 3,793 | 2.8% | 0.1% | 0.2% | 15.3% | 6.4% |
Length confound (RUN_LOG entry 2)
| group | n | mean tokens (full) | mean tokens fed | median (full) | % truncated |
|---|---|---|---|---|---|
| TP | 3,793 | 283.9 | 232.2 | 207 | 15.3 |
| TN | 11,888 | 827.0 | 294.4 | 269 | 23.8 |
| FP | 520 | 1237.2 | 276.6 | 242 | 29.0 |
| FN | 101 | 321.7 | 220.4 | 181 | 12.9 |
| CORRECT | 15,681 | 695.6 | 279.4 | 247 | 21.8 |
| WRONG | 621 | 1088.3 | 267.5 | 213 | 26.4 |
| all legitimate | 12,408 | 844.2 | 293.7 | 268 | 24.0 |
| all unwanted | 3,894 | 284.9 | 231.9 | 206 | 15.3 |
| all label3=unwanted | 3,894 | 284.9 | 231.9 | 206 | 15.3 |
| length bucket (full tokens) | n | positive rate | recall | FPR | accuracy |
|---|---|---|---|---|---|
| <=64 | 1,363 | 0.431 | 0.9744 | 0.1070 | 0.9281 |
| 65-128 | 2,538 | 0.290 | 0.9620 | 0.0394 | 0.9610 |
| 129-256 | 4,594 | 0.260 | 0.9766 | 0.0341 | 0.9687 |
| 257-512 | 4,232 | 0.184 | 0.9782 | 0.0287 | 0.9726 |
| >512 | 3,575 | 0.166 | 0.9782 | 0.0507 | 0.9541 |
Summary
- The classifier catches 97.41% of unwanted mail (phishing or spam) at threshold 0.5 (101 of 3,894 missed) while flagging 4.19% of legitimate mail (520 false alarms). Recall is the headline because a missed phish is the costly error for a small-business inbox.
- Holding false alarms to 1 in 100 legitimate emails keeps recall at 89.55% (threshold 0.866, flag when p >= threshold); at 1 in 1,000 it is 64.97%. The best-F1 threshold is 0.682 (F1 0.9397).
- Three-way: this split has no phishing-vs-spam ground truth; of the 3,793 true positives the model flagged, it called 974 phishing and 2,819 spam (informational only).
- On the 398-row credential-lure slice (37 documented phrases, 61.1% of its rows are labelled positive) recall is 99.59% with precision 97.58%: the highest-cost phish are caught at least as well as the average.
- Length confound: missed unwanted mail averages 322 tokens vs 284 for caught; false positives average 1237 tokens vs 827 for correct legitimate mail (all legitimate 844, all unwanted 285). Legitimate mail is ~3.0x longer than unwanted mail on average; the error groups do not follow the length gap, so length is not the main shortcut.
- Recall by length is lowest in the 65-128-token bucket (96.2%, 736 positives).
- Of the 520 false positives, 19.6% look like mailing-list or digest traffic, 0.0% are HTML-heavy and 2.3% are very short (TN reference: 29.7% / 0.2% / 0.4%).
- Tokens most over-represented in missed unwanted mail vs caught:
smailex,blocki,scanmail,progra,scanning,matched,tex-fonts,pattern,blocking,tpn. - Tokens most over-represented in false positives vs correct legitimate mail:
irc-list-web,quarantine,foundstone,skidmore,monty,bn3,ips,anyplace,noc's,https.