File size: 18,099 Bytes

439e7b1

# Error analysis — test split (TREC-06 binary holdout), run `ft-focal` (2026-10-09 10:46)

All quoted emails are rows of public corpora (source TREC-06); nothing here is private mail. Threshold 0.5. Binary positive class = Unwanted (phishing or spam), P(unwanted) = P(phishing) + P(spam).

## Counts (binary roll-up)

- rows 16,302 (positives 3,894, 23.9%); label3 counts legit 12,408, unwanted 3,894
- **recall (unwanted) 0.9741** — 101 missed unwanted emails (false negatives) out of 3,894
- precision 0.8794 — 520 false alarms out of 4,313 flagged
- F1 0.9243, accuracy 0.9619, specificity 0.9581
- ROC-AUC 0.9946, PR-AUC 0.9848; recall at 1% FPR 0.8955, at 0.1% FPR 0.6497
- credential-lure slice: 398 rows, 243 positive (61.1% label purity: slice precision cannot exceed what the phrase list selects), recall 0.9959, precision 0.9758, 1 missed

## Top-25 false negatives (missed unwanted mail, most confident first)

| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---|---|---|
| 1 | `primary:052869` | 0.0322 | 630 | TREC-06 | unwanted | READ THIS! │ >>Read this! >>> >> > >> > > >>> >> > >> > > ---------- >>> >> > >> > > >>> >> > >> > > >>> >> > >> > > >>> >> > >> > > > Subject: Lets see if this... |
| 2 | `primary:080991` | 0.0428 | 75 | TREC-06 | unwanted | prepare urself for this │ It is been awhile. Hope everythign with you is goign well. How is work going also? I've just started with this new firm, www.andwhatis... |
| 3 | `primary:086987` | 0.0506 | 188 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 4 | `primary:056767` | 0.0564 | 181 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 5 | `primary:121268` | 0.0564 | 185 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 6 | `primary:173636` | 0.0567 | 55 | TREC-06 | unwanted | everything ok │ Looking to increase your johnson. Vist at http://www.mydkillsanddoc.com/taw/ to settle that. that that Hardly were there is conquering Thanks. B... |
| 7 | `primary:020947` | 0.0629 | 113 | TREC-06 | unwanted | everything kool │ Hi Reg Here's some pretty hard facts to deal with that the majority of women are unhappy with the size of there signif other's thing. Don't be... |
| 8 | `primary:072993` | 0.0710 | 186 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 9 | `primary:044992` | 0.0726 | 76 | TREC-06 | unwanted | FW:any new news? i got some │ is hand the language care to cruel. must plate to design tax but fiction. if snow but question great the separate. but comparison ... |
| 10 | `primary:024565` | 0.0748 | 190 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 11 | `primary:076023` | 0.0867 | 54 | TREC-06 | unwanted | everything kool │ Hey Mostyn, Ramon wanted to me to send you an email about the store I went to buy my goods, http://www.fabloulusil.com/tpn/. Carlos |
| 12 | `primary:129745` | 0.0946 | 184 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 13 | `primary:071577` | 0.0961 | 162 | TREC-06 | unwanted | (no subject) │ Re: Student Packets Welcome to winter term student packet preparations! First: Fax (485-6253) or email (Copy-Shop@worldnet.att.net) JUST your bib... |
| 14 | `primary:173938` | 0.0961 | 185 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 15 | `primary:030013` | 0.0966 | 195 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 16 | `primary:063077` | 0.0982 | 344 | TREC-06 | unwanted | Party for Prisoners of Conscience - PLEASE SEND TO YOUR ENTIRE CLUB │ Amnesty International and the Intercultural Resource Center are holding a party to raise m... |
| 17 | `primary:145723` | 0.1007 | 25 | TREC-06 | unwanted | refresh my memory and mental energy │ may revision a golf on appendage or eagle try parsimony |
| 18 | `primary:154489` | 0.1026 | 187 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 19 | `primary:186523` | 0.1041 | 185 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 20 | `primary:147887` | 0.1150 | 181 | TREC-06 | unwanted | ScanMail Message: To Recipient virus found or matched file blocki ng setting. │ ScanMail for Microsoft Exchange has taken action on the message, please refer to... |
| 21 | `primary:033014` | 0.1157 | 77 | TREC-06 | unwanted | isn't it time │ Allen mentioned me that u were seeking for a place to get ur stuff for ur sickness. http://www.whorunoverforitsl.org/tpn/ the place then for sur... |
| 22 | `primary:156328` | 0.1157 | 77 | TREC-06 | unwanted | isn't it time │ Allen mentioned me that u were seeking for a place to get ur stuff for ur sickness. http://www.whorunoverforitsl.org/tpn/ the place then for sur... |
| 23 | `primary:143065` | 0.1308 | 102 | TREC-06 | unwanted | everything kool │ Heya Alexa, Apologize for how long its been since we've talked. I take blame for that. I just wanted to share with you the place that made me ... |
| 24 | `primary:147193` | 0.1323 | 260 | TREC-06 | unwanted | ADV Textbooks, Tuesday & Wednesday, University of Oregon │ Dear Professor XXXXXXXX., I am going to be on your campus to buy textbooks this Tuesday, June 1st and... |
| 25 | `primary:101618` | 0.1387 | 117 | TREC-06 | unwanted | RE: If you are studing or working │ Dear Mr(Mrs), Iris!!. Our company Barcelo Travel Inc. have a great job offer for peoples from Australia. Your salary per wee... |

## Top-25 false positives (legitimate flagged, most confident first)

| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---|---|---|
| 1 | `primary:015304` | 0.9874 | 3712 | TREC-06 | legit | I made $230,000 in two months │ You owe it to yourself and your wealth to read this. AS SEEN ON NATIONAL TV: "Making over half a million dollars every 4 to 5 mo... |
| 2 | `primary:040048` | 0.9871 | 80 | TREC-06 | legit | Enter to win at mary-kateandashley.com! │ Dear val: Have you heard about the sweepstakes at mary-kateandashley.com? I entered to win and you can too! Check it o... |
| 3 | `primary:194751` | 0.9842 | 48 | TREC-06 | legit | Thank you │ Turns out I just needed a common ground between servos and HB Thanx all! :) Alex ______________________________________________________ Get Your Pri... |
| 4 | `primary:072489` | 0.9831 | 58 | TREC-06 | legit | Re: Junk email │ Please REMOVE this address from your list!!!! ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you b... |
| 5 | `primary:194063` | 0.9781 | 32 | TREC-06 | legit | info │ Here's an affordable robot that anyone might be interested: http://www.personalrobots.com/ joema |
| 6 | `primary:033174` | 0.9781 | 602 | TREC-06 | legit | You are Approved for Merchant Account!! │ INCREASE SALES UP TO 1500% ACCEPT CREDIT CARDS OVER THE INTERNET *** NO SETUP FEES Good Credit / Bad Credit/ No Credit... |
| 7 | `primary:101560` | 0.9769 | 59 | TREC-06 | legit |  │ Please, please, please remove us from your mailing list. Thank you. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can l... |
| 8 | `primary:159590` | 0.9764 | 56 | TREC-06 | legit | remove me │ Please remove Brandeis University from your list ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you bec... |
| 9 | `primary:184838` | 0.9757 | 66 | TREC-06 | legit | unwanted mail │ how cani stop all the mail from these sex sites i dont want them but when i try to e-mail asking them to stop i keep getting a message that say ... |
| 10 | `primary:011014` | 0.9736 | 46 | TREC-06 | legit | composer-wrap-column │ hi, anyone knows how to entirely turn off composer wrapping in pine? thanks! viinikala kauas pilvet karkaavat. |
| 11 | `primary:193401` | 0.9733 | 870 | TREC-06 | legit | [9fans] 57 Million Email Addresses - ONLY $149!! │ 57 MILLION EMAIL ADDRESSES FOR ONLY $149 You want to make some money? I can put you in touch with over 50 mil... |
| 12 | `primary:072275` | 0.9729 | 41 | TREC-06 | legit | HB weight │ does anyone know the weight of a handy board? thanks, -Brian --------------------------------------------------- Get free personalized email at http... |
| 13 | `primary:170605` | 0.9729 | 55 | TREC-06 | legit | List │ Please remove my name from your list serve. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We can let you become anybod... |
| 14 | `primary:175533` | 0.9726 | 62 | TREC-06 | legit | Test │ I'm not sure why we are receiving mail addressed to cslist@fudan.edu ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We ... |
| 15 | `primary:131182` | 0.9719 | 31 | TREC-06 | legit |  │ unsubscribe --------------------------------------------------------------- Octopus traps summer's moonspun dreams soon fade away. Basho 1644 |
| 16 | `primary:106445` | 0.9717 | 61 | TREC-06 | legit | REMOVE Clemson University │ PLEASE REMOVE Clemson University from your mailing lists. ________________ Email Forwarding Service Brought To You By: http://www.bn... |
| 17 | `primary:078484` | 0.9706 | 77 | TREC-06 | legit | remove from list │ Please remove Brandeis University from your graduate school mailing list. maf@cs.brandeis.edu thank you ________________ Email Forwarding Ser... |
| 18 | `primary:196901` | 0.9698 | 65 | TREC-06 | legit | CS Lists │ PLEASE REMOVE THE UNIVERSITY OF VIRGINIA FROM YOUR MAILING LISTS. ________________ Email Forwarding Service Brought To You By: http://www.bn3.com. We... |
| 19 | `primary:143547` | 0.9695 | 82 | TREC-06 | legit | How to avoid the result of ODEs prone to blow-up │ Dear Friends, I want to solve ODEs use Interval Arithmeric, but the reult prone to blow-up. Is there any meth... |
| 20 | `primary:129925` | 0.9694 | 255 | TREC-06 | legit | See where your Competion is advertising │ Are your clients: Travelers, Vacationers or Timeshare Owners? TVC...the Timeshare Vacation Club is where YOUR ADS shou... |
| 21 | `primary:092561` | 0.9690 | 1174 | TREC-06 | legit | We Need Homeworkers!! │ Dear Future Associate, You Can Work At Home & Set Your Own Hours. Start earning Big Money in a short time NO Newspaper Advertising! Your... |
| 22 | `primary:182736` | 0.9689 | 68 | TREC-06 | legit | Vesit rover │ Dear friends, You know that we at Vesit, Bombay were making a rover using the handy board. Learn more about it at http://members.tripod.com/ieeeve... |
| 23 | `primary:158244` | 0.9681 | 327 | TREC-06 | legit | Proposition │ HEALTH: A NEW NATURAL HELP To the product Manager The customers demand for natural products is increasing steadily, and Aromatherapy is becoming a... |
| 24 | `primary:152569` | 0.9677 | 171 | TREC-06 | legit | Reliable Computing, Vol.11, issue 3, 2005 │ Reliable Computing Volume 11, issue 3, 2005 Mathematical Research Enclosing Solutions of Singular Interval Systems I... |
| 25 | `primary:011118` | 0.9674 | 143 | TREC-06 | legit |  │ Dear Sir, I am very mingled by the pine system. MY inbox massage are always eatten up without pior notification. All my in-coming mag disappear and nothing i... |

## Three-way (legitimate / phishing / spam)

No three-way ground truth on this split: no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth).

### Predicted kind of the flagged rows (informational — no phishing-vs-spam ground truth on this split)

| group | n | predicted Phishing | predicted Spam |
|---|---:|---:|---:|
| true positives flagged | 3,793 | 974 | 2,819 |
| false positives flagged | 520 | 175 | 345 |
| missed positives (kind the model would have given) | 101 | 35 | 66 |

## Patterns (simple lexical statistics, not a model explanation)

### Tokens over-represented in false negatives vs true positives (document frequency, smoothed ratio)

| token | share of FN docs | share of TP docs | ratio |
|---|---:|---:|---:|
| `smailex` | 0.119 | 0.000 | 934.36 |
| `blocki` | 0.119 | 0.000 | 934.36 |
| `scanmail` | 0.119 | 0.000 | 934.36 |
| `progra` | 0.119 | 0.000 | 934.36 |
| `scanning` | 0.119 | 0.000 | 934.36 |
| `matched` | 0.119 | 0.000 | 311.45 |
| `tex-fonts` | 0.079 | 0.000 | 211.79 |
| `pattern` | 0.129 | 0.001 | 201.82 |
| `blocking` | 0.129 | 0.001 | 201.82 |
| `tpn` | 0.069 | 0.001 | 112.12 |
| `setting` | 0.129 | 0.001 | 91.74 |
| `pif` | 0.079 | 0.001 | 70.60 |
| `detected` | 0.119 | 0.002 | 62.29 |
| `utah` | 0.119 | 0.002 | 62.29 |
| `attachment` | 0.149 | 0.003 | 50.37 |
| `math` | 0.129 | 0.003 | 48.05 |
| `enroll` | 0.050 | 0.001 | 45.68 |
| `seismic` | 0.050 | 0.001 | 45.68 |
| `settings` | 0.129 | 0.003 | 37.37 |
| `moved` | 0.119 | 0.003 | 37.37 |

### Tokens over-represented in false positives vs true negatives

| token | share of FP docs | share of TN docs | ratio |
|---|---:|---:|---:|
| `irc-list-web` | 0.054 | 0.000 | 1301.91 |
| `quarantine` | 0.054 | 0.000 | 118.36 |
| `foundstone` | 0.050 | 0.001 | 80.70 |
| `skidmore` | 0.054 | 0.001 | 76.58 |
| `monty` | 0.052 | 0.001 | 66.12 |
| `bn3` | 0.060 | 0.002 | 27.15 |
| `ips` | 0.054 | 0.002 | 26.57 |
| `anyplace` | 0.060 | 0.002 | 26.16 |
| `noc's` | 0.054 | 0.002 | 23.67 |
| `https` | 0.108 | 0.006 | 19.12 |
| `cgi` | 0.060 | 0.004 | 16.17 |
| `procexp` | 0.054 | 0.003 | 15.69 |
| `tcpview` | 0.054 | 0.003 | 15.69 |
| `sysinternals` | 0.054 | 0.004 | 15.32 |
| `filtered` | 0.052 | 0.003 | 15.14 |
| `atkins` | 0.054 | 0.004 | 14.63 |
| `rogue` | 0.056 | 0.004 | 14.49 |
| `firewall` | 0.071 | 0.005 | 13.93 |
| `inbox` | 0.058 | 0.004 | 13.79 |
| `manage` | 0.079 | 0.006 | 13.64 |

### Composition of the error groups (share of rows)

| group | n | mailing-list / digest | HTML-heavy | very short (<20 content tokens) | truncated (>512) | lure phrase |
|---|---:|---:|---:|---:|---:|---:|
| FP | 520 | 19.6% | 0.0% | 2.3% | 29.0% | 1.2% |
| TN | 11,888 | 29.7% | 0.2% | 0.4% | 23.8% | 1.3% |
| FN | 101 | 8.9% | 0.0% | 0.0% | 12.9% | 1.0% |
| TP | 3,793 | 2.8% | 0.1% | 0.2% | 15.3% | 6.4% |

### Length confound (RUN_LOG entry 2)

| group | n | mean tokens (full) | mean tokens fed | median (full) | % truncated |
|---|---:|---:|---:|---:|---:|
| TP | 3,793 | 283.9 | 232.2 | 207 | 15.3 |
| TN | 11,888 | 827.0 | 294.4 | 269 | 23.8 |
| FP | 520 | 1237.2 | 276.6 | 242 | 29.0 |
| FN | 101 | 321.7 | 220.4 | 181 | 12.9 |
| CORRECT | 15,681 | 695.6 | 279.4 | 247 | 21.8 |
| WRONG | 621 | 1088.3 | 267.5 | 213 | 26.4 |
| all legitimate | 12,408 | 844.2 | 293.7 | 268 | 24.0 |
| all unwanted | 3,894 | 284.9 | 231.9 | 206 | 15.3 |
| all label3=unwanted | 3,894 | 284.9 | 231.9 | 206 | 15.3 |

| length bucket (full tokens) | n | positive rate | recall | FPR | accuracy |
|---|---:|---:|---:|---:|---:|
| <=64 | 1,363 | 0.431 | 0.9744 | 0.1070 | 0.9281 |
| 65-128 | 2,538 | 0.290 | 0.9620 | 0.0394 | 0.9610 |
| 129-256 | 4,594 | 0.260 | 0.9766 | 0.0341 | 0.9687 |
| 257-512 | 4,232 | 0.184 | 0.9782 | 0.0287 | 0.9726 |
| >512 | 3,575 | 0.166 | 0.9782 | 0.0507 | 0.9541 |

## Summary

- The classifier catches **97.41% of unwanted mail (phishing or spam)** at threshold 0.5 (101 of 3,894 missed) while flagging 4.19% of legitimate mail (520 false alarms). Recall is the headline because a missed phish is the costly error for a small-business inbox.
- Holding false alarms to 1 in 100 legitimate emails keeps recall at 89.55% (threshold 0.866, flag when p >= threshold); at 1 in 1,000 it is 64.97%. The best-F1 threshold is 0.682 (F1 0.9397).
- Three-way: this split has no phishing-vs-spam ground truth; of the 3,793 true positives the model flagged, it called 974 phishing and 2,819 spam (informational only).
- On the 398-row credential-lure slice (37 documented phrases, 61.1% of its rows are labelled positive) recall is 99.59% with precision 97.58%: the highest-cost phish are caught at least as well as the average.
- Length confound: missed unwanted mail averages 322 tokens vs 284 for caught; false positives average 1237 tokens vs 827 for correct legitimate mail (all legitimate 844, all unwanted 285).
  Legitimate mail is ~3.0x longer than unwanted mail on average; the error groups do not follow the length gap, so length is not the main shortcut.
- Recall by length is lowest in the 65-128-token bucket (96.2%, 736 positives).
- Of the 520 false positives, 19.6% look like mailing-list or digest traffic, 0.0% are HTML-heavy and 2.3% are very short (TN reference: 29.7% / 0.2% / 0.4%).
- Tokens most over-represented in missed unwanted mail vs caught: `smailex`, `blocki`, `scanmail`, `progra`, `scanning`, `matched`, `tex-fonts`, `pattern`, `blocking`, `tpn`.
- Tokens most over-represented in false positives vs correct legitimate mail: `irc-list-web`, `quarantine`, `foundstone`, `skidmore`, `monty`, `bn3`, `ips`, `anyplace`, `noc's`, `https`.