presentation/reports/error_analysis_test3.md

# Error analysis — test3 split (three-way holdout), run `ft-focal` (2026-10-09 10:46)

All quoted emails are rows of public corpora (source Assassin, CEAS-08, Enron, Nazario, Nigerian_Fraud, TREC-07); nothing here is private mail. Threshold 0.5. Binary positive class = Unwanted (phishing or spam), P(unwanted) = P(phishing) + P(spam).

## Counts (binary roll-up)

- rows 3,220 (positives 1,221, 37.9%); label3 counts legit 1,999, phish 521, spam 700
- **recall (unwanted) 0.9836** — 20 missed unwanted emails (false negatives) out of 1,221
- precision 0.9893 — 13 false alarms out of 1,214 flagged
- F1 0.9864, accuracy 0.9898, specificity 0.9935
- ROC-AUC 0.9993, PR-AUC 0.9989; recall at 1% FPR 0.9869, at 0.1% FPR 0.8788
- credential-lure slice: 275 rows, 268 positive (97.5% label purity: slice precision cannot exceed what the phrase list selects), recall 1.0000, precision 1.0000, 0 missed

## Top-25 false negatives (missed unwanted mail, most confident first)

| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---|---|---|
| 1 | `primary:191966` | 0.0224 | 86 | Enron | spam | re : saturday . here new and │ good day : i tried to call your 3 three time but your phone is not available . i think you did a mistake during filling the form ... |
| 2 | `primary:148887` | 0.0467 | 1097 | Enron | spam | southern california demographic analysis workshop announcement ( january charset = windows - 1252 " > │ analyzing southern california communities : demographic ... |
| 3 | `aura:Nazario:000097` | 0.1782 | 39 | Nazario | phish | Oil Payment Approval │ Find the attached document via Google secure to view closing statement. All parties have completed Documents. please sign these at your e... |
| 4 | `primary:010013` | 0.2112 | 135 | Enron | spam | new birthday calendar │ hi i ' m setting up a birthday calendar and need your help . just click the link below and enter your birthday details . ( it ' s quick ... |
| 5 | `primary:007176` | 0.2300 | 281 | Assassin | spam | TOMORROW -- Password: Soleil -- TV Casting │ Tomorrow will be huge. Password: Soleil has become the premiere Thursday night in New York - with a chill after-wor... |
| 6 | `aura:Nazario:000643` | 0.2584 | 368 | Nazario | phish | Phishing Email Corpus │ Good Morning, Jose, My name is Michael Kotson, and I am a Cybersecurity researcher at MIT Lincoln Laboratory. I am developing a lecture ... |
| 7 | `primary:181246` | 0.2858 | 1051 | Assassin | spam | [ILUG] (no subject) │ Suppliers of Computers, Printers, etc. & Consumables Central Data Supplies 24, Middleton Road, Banbury, Oxon, OX16 4QJ. Tel: 01295 271666 ... |
| 8 | `primary:095405` | 0.2967 | 40 | Enron | spam | pocket micro scope id : sxo 683 │ re : ppql 308243023 not interested , please let us know : powerlinepromo @ yahoo . com |
| 9 | `primary:014300` | 0.3203 | 301 | Enron | spam | cecil county networking lunch │ networking lunch chesapeake bay golf club in north east wednesday , november 17 th 11 : 15 am smallcompanies . com in conjunctio... |
| 10 | `primary:001939` | 0.3348 | 1153 | Assassin | spam | Invite: Content Management Summit, Oct. 10th New York City │ ![CMS](http://www.cmsummit.com/images/right_02.gif) ![CMS](http://www.cmsummit.com/images/cms_title... |
| 11 | `primary:033220` | 0.3678 | 28444 | Assassin | spam | Wir suchen Erotikdarsteller! │ This is a Multipart MIME message. Since your mail reader does not understand this format, some or all of this message may not be ... |
| 12 | `primary:025244` | 0.3789 | 8 | Enron | spam | message subject │ goodbye |
| 13 | `primary:133074` | 0.3791 | 86 | Enron | spam | his lips , his nose , │ good day : i tried to call your 3 three time but your phone is not available . i think you did a mistake during filling the form . anywa... |
| 14 | `aura:Nigerian_Fraud:003049` | 0.3823 | 516 | Nigerian_Fraud | phish | Invitation from aditya monu │ This is a multi-part message in MIME format. --incrowd_6506e2e6636163656626f7892e636f26d05b3743 Content-Type: text/plain; charset=... |
| 15 | `aura:Nazario:000299` | 0.3973 | 46 | Nazario | phish | RE: IT Administrator │ IT Administrator desk: has currently upgraded our mailbox service quote to 150GB space Faculty/Staff are hereby advice to click on WEBMAI... |
| 16 | `primary:046569` | 0.4039 | 1078 | Enron | spam | invite : content management summit , oct . 10 th new york city │ dear friends , the online content business is doing fantastic . sound crazy ? perhaps . with ad... |
| 17 | `primary:135631` | 0.4338 | 18 | Enron | spam | reply : shy mom wants to date you │ hello there ! sode |
| 18 | `primary:148857` | 0.4750 | 180 | Assassin | spam | send REAL PAPER GREETING CARDS on-line! │ Cards In Advance Birthday Anniversary Get Well Soon Thinking of you Congratulations Sympathy Engagement Good Luck New ... |
| 19 | `primary:129677` | 0.4929 | 510 | Assassin | spam | Proposal for cooperation │ Hello. I meant to contact http://taint.org webmaster, Am I correct, I hope? Actually, I am not quite sure... Did I or Natasha Fedorov... |
| 20 | `primary:075760` | 0.4969 | 719 | Enron | spam | greece based professionals and organizations │ you areinvited to register on training consortiumjoin us by registering for free as a trainer , consultant , or s... |

## Top-25 false positives (legitimate flagged, most confident first)

| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---|---|---|
| 1 | `primary:144048` | 0.9853 | 46 | CEAS-08 | legit | Whiskas Scratching Post │ HTML Message - =?windows-1252?B?V2hpc2thcyBTY3JhdGNoaW5nIFBvc3QA?= |
| 2 | `primary:199386` | 0.9471 | 2206 | Assassin | legit | Back to School with Buy4Now │ This is a multi-part message in MIME format. ------=_NextPart_000_22A2C_01C23F09.09760420 Content-Type: text/plain; charset="iso-8... |
| 3 | `primary:091801` | 0.9436 | 66 | CEAS-08 | legit | Lots Of SPAM │ Hi List, i have posted my RAW email in http://pastebin.ca/918849 , i'm receiving 1000 to 4000 per day this king of mesages. SA also skipping this... |
| 4 | `primary:152763` | 0.8427 | 1228 | CEAS-08 | legit | SAVE 30% on Best Recipe Cookbooks │ Untitled Document We don't want you to miss out on our e-mail offers! Please add our e-mail address, arfldpykmjrzc@americast... |
| 5 | `primary:165228` | 0.8246 | 4917 | Assassin | legit | United We Fall? │ This is a multi-part message in MIME format... ------------=_1029244856-11739-4 Content-Type: text/plain; charset="iso-8859-1" Content-Disposi... |
| 6 | `primary:041713` | 0.7668 | 340 | TREC-07 | legit | Electric Transmission Week complimentary trial │ 1105_ETW_Trial View this message as a Web page. FREE Trial Offer You're invited to receive a FREE trial subscri... |
| 7 | `primary:103927` | 0.7592 | 1587 | CEAS-08 | legit | FREE Shipping on "America's Best Lost Recipes" Cookbook - Plus Free OXO(R) Tongs │ Untitled Document We don't want you to miss out on our e-mail offers! Please ... |
| 8 | `primary:055965` | 0.7247 | 5197 | Assassin | legit | Brother, Can You Spare a Jet? │ This is a multi-part message in MIME format... ------------=_1029157891-23956-4 Content-Type: text/plain; charset="iso-8859-1" C... |
| 9 | `primary:170237` | 0.6660 | 88 | CEAS-08 | legit | The jet set │ Well, it appears you have all taken an idea and run with it. My point was that "Knocked Up" appeared quite the good film, and the Williams one so ... |
| 10 | `primary:117398` | 0.6510 | 5959 | Assassin | legit | AOL Keyword: Fraud Probe │ This is a multi-part message in MIME format... ------------=_1033046883-23957-4 Content-Type: text/plain; charset="iso-8859-1" Conten... |
| 11 | `primary:168120` | 0.5796 | 268 | TREC-07 | legit | Please confirm your request to join Deficiv │ Hello deficient@speedy.uwaterloo.ca, We have received your request to join the Deficiv group hosted by Yahoo! Grou... |
| 12 | `primary:057514` | 0.5173 | 809 | CEAS-08 | legit | WIN a $6,000 Cook Islands getaway │ HotelClub Newsletter Dear Tony, Subscribe to any free NEWS.com.au newsletter for your chance to WIN a $6,000 Cook Islands ge... |
| 13 | `primary:021911` | 0.5051 | 875 | CEAS-08 | legit | It's here. Dell's first tablet PC. │ Dell would like to introduce our first tablet PC, the Latitude XT. Part of the trusted Latitude line, the XT gives you all ... |

## Three-way (legitimate / phishing / spam)

Decision rule: unwanted iff P(phishing)+P(spam) >= threshold, then kind = argmax(P(phishing), P(spam)); the 3x3 matrix therefore collapses exactly onto the binary confusion matrix. argmax-of-3 reported alongside.

Rows with a full label: 3,220; unwanted-but-unsplit rows (binary roll-up only): 0.

| true \ predicted | Legitimate | Phishing | Spam |
|---|---:|---:|---:|
| **Legitimate** | 1,986 | 1 | 12 |
| **Phishing** | 4 | 514 | 3 |
| **Spam** | 16 | 1 | 683 |

- accuracy 0.9885, **macro-F1 0.9867** (over 3 classes; over the 3 present: 0.9867); argmax-of-3 rule: macro-F1 0.9858, 6 rows differ between the rules

| class | precision | recall | F1 | support |
|---|---:|---:|---:|---:|
| Legitimate | 0.9900 | 0.9935 | 0.9918 | 1,999 |
| Phishing | 0.9961 | 0.9866 | 0.9913 | 521 |
| Spam | 0.9785 | 0.9757 | 0.9771 | 700 |

### Phishing vs spam on the 1,221 true unwanted rows

Kind decision alone (given the row is unwanted, argmax(P(phishing), P(spam))): accuracy 0.9959; phishing read as spam 4, spam read as phishing 1.

| true \ predicted (roll-up rule) | Legitimate (missed) | Phishing | Spam |
|---|---:|---:|---:|
| **Phishing** | 4 | 514 | 3 |
| **Spam** | 16 | 1 | 683 |

### Per source (three-way rows)

| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9534 | 0.9458 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9905 | 0.9952 |
| Enron | 628 | 0 | 0 | 628 | 0.9841 | 0.9920 |
| Nazario | 222 | 0 | 222 | 0 | 0.9865 | 0.9932 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9866 | 0.9933 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9982 | 0.9991 |

### Top-25 phishing -> spam confusions (3 total, most confident first)

| # | doc_id | P(legit) | P(phish) | P(spam) | tokens (full) | source | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---:|---:|---|---|
| 1 | `aura:Nigerian_Fraud:001698` | 0.0949 | 0.0320 | 0.8730 | 2061 | Nigerian_Fraud | " Please add me to your mailing list. " │ "How to turn 3 into 10,000" This is not a get rich quick scheme, but with a little effort will make a nice little retu... |
| 2 | `aura:Nigerian_Fraud:000337` | 0.1620 | 0.3029 | 0.5352 | 330 | Nigerian_Fraud | Dddddddd R. Rrrrr Message from Vasil Nanchev. │ Hi, This is Vasil Nanchev. I just wanted to inform you about something which is really time sensitive. For the f... |
| 3 | `aura:Nigerian_Fraud:000824` | 0.1349 | 0.3997 | 0.4654 | 1089 | Nigerian_Fraud | Seeking joint-venture partners for ERW oil casing project │ Dear Sir or Madam: We are keenly interested in setting up a new factory to produce 120,000 ERW oil c... |

### Top-25 spam -> phishing confusions (1 total, most confident first)

| # | doc_id | P(legit) | P(phish) | P(spam) | tokens (full) | source | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---:|---:|---|---|
| 1 | `primary:077835` | 0.0080 | 0.8669 | 0.1251 | 1179 | Assassin | URGENT BUSINESS ASSISTANCE. │ --0-1118764524-1030968705=:44128 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit KEVIN GEORGE ABIDJAN... |

### Tokens over-represented in phishing read as spam vs phishing read correctly

| token | share of phish->spam docs | share of phish->phish docs | ratio |
|---|---:|---:|---:|
| `join` | 1.000 | 0.019 | 49.00 |
| `could` | 1.000 | 0.076 | 13.03 |
| `www` | 1.000 | 0.150 | 6.64 |
| `new` | 1.000 | 0.160 | 6.24 |
| `http` | 1.000 | 0.193 | 5.17 |
| `years` | 1.000 | 0.196 | 5.07 |
| `get` | 1.000 | 0.202 | 4.92 |
| `about` | 1.000 | 0.239 | 4.17 |
| `after` | 1.000 | 0.278 | 3.59 |
| `who` | 1.000 | 0.317 | 3.15 |
| `into` | 1.000 | 0.323 | 3.09 |
| `com` | 1.000 | 0.339 | 2.95 |
| `which` | 1.000 | 0.362 | 2.76 |
| `now` | 1.000 | 0.379 | 2.63 |
| `can` | 1.000 | 0.467 | 2.14 |
| `are` | 1.000 | 0.562 | 1.78 |
| `from` | 1.000 | 0.644 | 1.55 |
| `will` | 1.000 | 0.714 | 1.40 |
| `for` | 1.000 | 0.800 | 1.25 |
| `this` | 1.000 | 0.815 | 1.23 |

### Tokens over-represented in spam read as phishing vs spam read correctly

(too few rows in one of the groups)

### Predicted kind of the flagged rows (informational — includes the unsplit rows, which have no kind label)

| group | n | predicted Phishing | predicted Spam |
|---|---:|---:|---:|
| true positives flagged | 1,201 | 515 | 686 |
| false positives flagged | 13 | 1 | 12 |
| missed positives (kind the model would have given) | 20 | 3 | 17 |

## Patterns (simple lexical statistics, not a model explanation)

### Tokens over-represented in false negatives vs true positives (document frequency, smoothed ratio)

| token | share of FN docs | share of TP docs | ratio |
|---|---:|---:|---:|
| `invite` | 0.250 | 0.002 | 128.94 |
| `cut` | 0.150 | 0.002 | 82.05 |
| `practical` | 0.150 | 0.002 | 82.05 |
| `networking` | 0.200 | 0.002 | 75.36 |
| `audience` | 0.150 | 0.003 | 45.59 |
| `oct` | 0.150 | 0.003 | 45.59 |
| `innovative` | 0.150 | 0.003 | 45.59 |
| `event` | 0.200 | 0.005 | 40.58 |
| `angeles` | 0.150 | 0.004 | 37.30 |
| `anyway` | 0.150 | 0.004 | 37.30 |
| `feature` | 0.150 | 0.005 | 31.56 |
| `york` | 0.150 | 0.005 | 31.56 |
| `subscription` | 0.200 | 0.007 | 27.76 |
| `technologies` | 0.150 | 0.006 | 27.35 |
| `entertainment` | 0.150 | 0.007 | 24.13 |
| `staff` | 0.200 | 0.011 | 19.54 |
| `association` | 0.150 | 0.008 | 19.54 |
| `markets` | 0.150 | 0.009 | 17.84 |
| `hosting` | 0.150 | 0.009 | 17.84 |
| `mailman` | 0.150 | 0.009 | 17.84 |

### Tokens over-represented in false positives vs true negatives

| token | share of FP docs | share of TN docs | ratio |
|---|---:|---:|---:|
| `outlets` | 0.231 | 0.001 | 206.01 |
| `zzz-unspun` | 0.231 | 0.001 | 206.01 |
| `czzz-unspun` | 0.231 | 0.001 | 206.01 |
| `cook` | 0.231 | 0.001 | 206.01 |
| `full-color` | 0.385 | 0.002 | 179.85 |
| `muehlbauer` | 0.231 | 0.002 | 147.15 |
| `vanderkolk` | 0.231 | 0.002 | 147.15 |
| `eletra` | 0.231 | 0.002 | 147.15 |
| `loripatel` | 0.231 | 0.002 | 147.15 |
| `copyedited` | 0.231 | 0.002 | 147.15 |
| `jimduffy86` | 0.231 | 0.002 | 147.15 |
| `asbrand` | 0.231 | 0.002 | 147.15 |
| `patel` | 0.231 | 0.002 | 147.15 |
| `authoritatively` | 0.231 | 0.002 | 147.15 |
| `four-week` | 0.231 | 0.002 | 147.15 |
| `duffy` | 0.231 | 0.002 | 147.15 |
| `mediaunspun` | 0.231 | 0.002 | 147.15 |
| `savviest` | 0.231 | 0.002 | 147.15 |
| `dasbrand` | 0.231 | 0.002 | 147.15 |
| `sendashtml` | 0.231 | 0.002 | 147.15 |

### Composition of the error groups (share of rows)

| group | n | mailing-list / digest | HTML-heavy | very short (<20 content tokens) | truncated (>512) | lure phrase |
|---|---:|---:|---:|---:|---:|---:|
| FP | 13 | 46.2% | 0.0% | 0.0% | 61.5% | 0.0% |
| TN | 1,986 | 56.3% | 0.0% | 0.0% | 38.3% | 0.4% |
| FN | 20 | 30.0% | 0.0% | 10.0% | 35.0% | 0.0% |
| TP | 1,201 | 8.3% | 0.0% | 0.6% | 26.6% | 22.3% |

### Length confound (RUN_LOG entry 2)

| group | n | mean tokens (full) | mean tokens fed | median (full) | % truncated |
|---|---:|---:|---:|---:|---:|
| TP | 1,201 | 978.0 | 279.8 | 243 | 26.6 |
| TN | 1,986 | 732.6 | 358.4 | 392 | 38.3 |
| FP | 13 | 1814.3 | 377.2 | 875 | 61.5 |
| FN | 20 | 1807.8 | 284.1 | 291 | 35.0 |
| CORRECT | 3,187 | 825.1 | 328.8 | 339 | 33.9 |
| WRONG | 33 | 1810.4 | 320.8 | 368 | 45.5 |
| all legitimate | 1,999 | 739.7 | 358.6 | 393 | 38.5 |
| all unwanted | 1,221 | 991.6 | 279.9 | 243 | 26.7 |
| all label3=phish | 521 | 1349.4 | 355.1 | 412 | 39.9 |
| all label3=spam | 700 | 725.3 | 223.9 | 159 | 16.9 |

| length bucket (full tokens) | n | positive rate | recall | FPR | accuracy |
|---|---:|---:|---:|---:|---:|
| <=64 | 129 | 0.767 | 0.9495 | 0.0333 | 0.9535 |
| 65-128 | 427 | 0.590 | 0.9921 | 0.0114 | 0.9906 |
| 129-256 | 699 | 0.395 | 0.9928 | 0.0000 | 0.9971 |
| 257-512 | 870 | 0.308 | 0.9851 | 0.0033 | 0.9931 |
| >512 | 1,095 | 0.298 | 0.9785 | 0.0104 | 0.9863 |

## Summary

- The classifier catches **98.36% of unwanted mail (phishing or spam)** at threshold 0.5 (20 of 1,221 missed) while flagging 0.65% of legitimate mail (13 false alarms). Recall is the headline because a missed phish is the costly error for a small-business inbox.
- Holding false alarms to 1 in 100 legitimate emails keeps recall at 98.69% (threshold 0.434, flag when p >= threshold); at 1 in 1,000 it is 87.88%. The best-F1 threshold is 0.475 (F1 0.9877).
- Three-way: macro-F1 **0.9867** over 3,220 rows with a full label (recall legitimate 99.3%, phishing 98.7%, spam 97.6%). Given that an email is unwanted, the model tells phishing from spam 99.6% of the time (4 phishing read as spam, 1 spam read as phishing). The phishing/spam labels follow corpus provenance (Nazario + Nigerian fraud + synthetic lures = phishing; SpamAssassin + Enron-Spam spam folders = spam), so part of this confusion is corpus style, not lure content.
- On the 275-row credential-lure slice (37 documented phrases, 97.5% of its rows are labelled positive) recall is 100.00% with precision 100.00%: the highest-cost phish are caught at least as well as the average.
- Length confound: missed unwanted mail averages 1808 tokens vs 978 for caught; false positives average 1814 tokens vs 733 for correct legitimate mail (all legitimate 740, all unwanted 992).
  Legitimate mail is ~0.7x longer than unwanted mail on average; the error groups do not follow the length gap, so length is not the main shortcut.
  Phishing rows average 1349 tokens vs spam 725: a length gap between the two unwanted kinds is a second shortcut to watch in the phishing-vs-spam confusion.
- Recall by length is lowest in the <=64-token bucket (94.9%, 99 positives).
- Of the 13 false positives, 46.2% look like mailing-list or digest traffic, 0.0% are HTML-heavy and 0.0% are very short (TN reference: 56.3% / 0.0% / 0.0%).
- Tokens most over-represented in missed unwanted mail vs caught: `invite`, `cut`, `practical`, `networking`, `audience`, `oct`, `innovative`, `event`, `angeles`, `anyway`.
- Tokens most over-represented in false positives vs correct legitimate mail: `outlets`, `zzz-unspun`, `czzz-unspun`, `cook`, `full-color`, `muehlbauer`, `vanderkolk`, `eletra`, `loripatel`, `copyedited`.