Add presentation bundle (reports, curves, run log, code) for the Colab notebook

439e7b1 verified 15 days ago

18.6 kB

# Error analysis — test3 split (three-way holdout), run `ft-focal` (2026-10-09 10:46)

All quoted emails are rows of public corpora (source Assassin, CEAS-08, Enron, Nazario, Nigerian_Fraud, TREC-07); nothing here is private mail. Threshold 0.5. Binary positive class = Unwanted (phishing or spam), P(unwanted) = P(phishing) + P(spam).
## Counts (binary roll-up)
- rows 3,220 (positives 1,221, 37.9%); label3 counts legit 1,999, phish 521, spam 700
- **recall (unwanted) 0.9836** — 20 missed unwanted emails (false negatives) out of 1,221
- precision 0.9893 — 13 false alarms out of 1,214 flagged
- F1 0.9864, accuracy 0.9898, specificity 0.9935
- ROC-AUC 0.9993, PR-AUC 0.9989; recall at 1% FPR 0.9869, at 0.1% FPR 0.8788
- credential-lure slice: 275 rows, 268 positive (97.5% label purity: slice precision cannot exceed what the phrase list selects), recall 1.0000, precision 1.0000, 0 missed
## Top-25 false negatives (missed unwanted mail, most confident first)
| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---|---|---|
| 1 | `primary:191966` | 0.0224 | 86 | Enron | spam | re : saturday . here new and │ good day : i tried to call your 3 three time but your phone is not available . i think you did a mistake during filling the form ... |
| 2 | `primary:148887` | 0.0467 | 1097 | Enron | spam | southern california demographic analysis workshop announcement ( january charset = windows - 1252 " > │ analyzing southern california communities : demographic ... |
| 3 | `aura:Nazario:000097` | 0.1782 | 39 | Nazario | phish | Oil Payment Approval │ Find the attached document via Google secure to view closing statement. All parties have completed Documents. please sign these at your e... |
| 4 | `primary:010013` | 0.2112 | 135 | Enron | spam | new birthday calendar │ hi i ' m setting up a birthday calendar and need your help . just click the link below and enter your birthday details . ( it ' s quick ... |
| 5 | `primary:007176` | 0.2300 | 281 | Assassin | spam | TOMORROW -- Password: Soleil -- TV Casting │ Tomorrow will be huge. Password: Soleil has become the premiere Thursday night in New York - with a chill after-wor... |
| 6 | `aura:Nazario:000643` | 0.2584 | 368 | Nazario | phish | Phishing Email Corpus │ Good Morning, Jose, My name is Michael Kotson, and I am a Cybersecurity researcher at MIT Lincoln Laboratory. I am developing a lecture ... |
| 7 | `primary:181246` | 0.2858 | 1051 | Assassin | spam | [ILUG] (no subject) │ Suppliers of Computers, Printers, etc. & Consumables Central Data Supplies 24, Middleton Road, Banbury, Oxon, OX16 4QJ. Tel: 01295 271666 ... |
| 8 | `primary:095405` | 0.2967 | 40 | Enron | spam | pocket micro scope id : sxo 683 │ re : ppql 308243023 not interested , please let us know : powerlinepromo @ yahoo . com |
| 9 | `primary:014300` | 0.3203 | 301 | Enron | spam | cecil county networking lunch │ networking lunch chesapeake bay golf club in north east wednesday , november 17 th 11 : 15 am smallcompanies . com in conjunctio... |
| 10 | `primary:001939` | 0.3348 | 1153 | Assassin | spam | Invite: Content Management Summit, Oct. 10th New York City │ ![CMS](http://www.cmsummit.com/images/right_02.gif) ![CMS](http://www.cmsummit.com/images/cms_title... |
| 11 | `primary:033220` | 0.3678 | 28444 | Assassin | spam | Wir suchen Erotikdarsteller! │ This is a Multipart MIME message. Since your mail reader does not understand this format, some or all of this message may not be ... |
| 12 | `primary:025244` | 0.3789 | 8 | Enron | spam | message subject │ goodbye |
| 13 | `primary:133074` | 0.3791 | 86 | Enron | spam | his lips , his nose , │ good day : i tried to call your 3 three time but your phone is not available . i think you did a mistake during filling the form . anywa... |
| 14 | `aura:Nigerian_Fraud:003049` | 0.3823 | 516 | Nigerian_Fraud | phish | Invitation from aditya monu │ This is a multi-part message in MIME format. --incrowd_6506e2e6636163656626f7892e636f26d05b3743 Content-Type: text/plain; charset=... |
| 15 | `aura:Nazario:000299` | 0.3973 | 46 | Nazario | phish | RE: IT Administrator │ IT Administrator desk: has currently upgraded our mailbox service quote to 150GB space Faculty/Staff are hereby advice to click on WEBMAI... |
| 16 | `primary:046569` | 0.4039 | 1078 | Enron | spam | invite : content management summit , oct . 10 th new york city │ dear friends , the online content business is doing fantastic . sound crazy ? perhaps . with ad... |
| 17 | `primary:135631` | 0.4338 | 18 | Enron | spam | reply : shy mom wants to date you │ hello there ! sode |
| 18 | `primary:148857` | 0.4750 | 180 | Assassin | spam | send REAL PAPER GREETING CARDS on-line! │ Cards In Advance Birthday Anniversary Get Well Soon Thinking of you Congratulations Sympathy Engagement Good Luck New ... |
| 19 | `primary:129677` | 0.4929 | 510 | Assassin | spam | Proposal for cooperation │ Hello. I meant to contact http://taint.org webmaster, Am I correct, I hope? Actually, I am not quite sure... Did I or Natasha Fedorov... |
| 20 | `primary:075760` | 0.4969 | 719 | Enron | spam | greece based professionals and organizations │ you areinvited to register on training consortiumjoin us by registering for free as a trainer , consultant , or s... |

## Top-25 false positives (legitimate flagged, most confident first)

| # | doc_id | P(unwanted) | tokens (full) | source | label_set | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---|---|---|
| 1 | `primary:144048` | 0.9853 | 46 | CEAS-08 | legit | Whiskas Scratching Post │ HTML Message - =?windows-1252?B?V2hpc2thcyBTY3JhdGNoaW5nIFBvc3QA?= |
| 2 | `primary:199386` | 0.9471 | 2206 | Assassin | legit | Back to School with Buy4Now │ This is a multi-part message in MIME format. ------=_NextPart_000_22A2C_01C23F09.09760420 Content-Type: text/plain; charset="iso-8... |
| 3 | `primary:091801` | 0.9436 | 66 | CEAS-08 | legit | Lots Of SPAM │ Hi List, i have posted my RAW email in http://pastebin.ca/918849 , i'm receiving 1000 to 4000 per day this king of mesages. SA also skipping this... |
| 4 | `primary:152763` | 0.8427 | 1228 | CEAS-08 | legit | SAVE 30% on Best Recipe Cookbooks │ Untitled Document We don't want you to miss out on our e-mail offers! Please add our e-mail address, arfldpykmjrzc@americast... |
| 5 | `primary:165228` | 0.8246 | 4917 | Assassin | legit | United We Fall? │ This is a multi-part message in MIME format... ------------=_1029244856-11739-4 Content-Type: text/plain; charset="iso-8859-1" Content-Disposi... |
| 6 | `primary:041713` | 0.7668 | 340 | TREC-07 | legit | Electric Transmission Week complimentary trial │ 1105_ETW_Trial View this message as a Web page. FREE Trial Offer You're invited to receive a FREE trial subscri... |
| 7 | `primary:103927` | 0.7592 | 1587 | CEAS-08 | legit | FREE Shipping on "America's Best Lost Recipes" Cookbook - Plus Free OXO(R) Tongs │ Untitled Document We don't want you to miss out on our e-mail offers! Please ... |
| 8 | `primary:055965` | 0.7247 | 5197 | Assassin | legit | Brother, Can You Spare a Jet? │ This is a multi-part message in MIME format... ------------=_1029157891-23956-4 Content-Type: text/plain; charset="iso-8859-1" C... |
| 9 | `primary:170237` | 0.6660 | 88 | CEAS-08 | legit | The jet set │ Well, it appears you have all taken an idea and run with it. My point was that "Knocked Up" appeared quite the good film, and the Williams one so ... |
| 10 | `primary:117398` | 0.6510 | 5959 | Assassin | legit | AOL Keyword: Fraud Probe │ This is a multi-part message in MIME format... ------------=_1033046883-23957-4 Content-Type: text/plain; charset="iso-8859-1" Conten... |
| 11 | `primary:168120` | 0.5796 | 268 | TREC-07 | legit | Please confirm your request to join Deficiv │ Hello deficient@speedy.uwaterloo.ca, We have received your request to join the Deficiv group hosted by Yahoo! Grou... |
| 12 | `primary:057514` | 0.5173 | 809 | CEAS-08 | legit | WIN a $6,000 Cook Islands getaway │ HotelClub Newsletter Dear Tony, Subscribe to any free NEWS.com.au newsletter for your chance to WIN a $6,000 Cook Islands ge... |
| 13 | `primary:021911` | 0.5051 | 875 | CEAS-08 | legit | It's here. Dell's first tablet PC. │ Dell would like to introduce our first tablet PC, the Latitude XT. Part of the trusted Latitude line, the XT gives you all ... |
## Three-way (legitimate / phishing / spam)
Decision rule: unwanted iff P(phishing)+P(spam) >= threshold, then kind = argmax(P(phishing), P(spam)); the 3x3 matrix therefore collapses exactly onto the binary confusion matrix. argmax-of-3 reported alongside.
Rows with a full label: 3,220; unwanted-but-unsplit rows (binary roll-up only): 0.
| true \ predicted | Legitimate | Phishing | Spam |
|---|---:|---:|---:|
| **Legitimate** | 1,986 | 1 | 12 |
| **Phishing** | 4 | 514 | 3 |
| **Spam** | 16 | 1 | 683 |
- accuracy 0.9885, **macro-F1 0.9867** (over 3 classes; over the 3 present: 0.9867); argmax-of-3 rule: macro-F1 0.9858, 6 rows differ between the rules
| class | precision | recall | F1 | support |
|---|---:|---:|---:|---:|
| Legitimate | 0.9900 | 0.9935 | 0.9918 | 1,999 |
| Phishing | 0.9961 | 0.9866 | 0.9913 | 521 |
| Spam | 0.9785 | 0.9757 | 0.9771 | 700 |
### Phishing vs spam on the 1,221 true unwanted rows
Kind decision alone (given the row is unwanted, argmax(P(phishing), P(spam))): accuracy 0.9959; phishing read as spam 4, spam read as phishing 1.
| true \ predicted (roll-up rule) | Legitimate (missed) | Phishing | Spam |
|---|---:|---:|---:|
| **Phishing** | 4 | 514 | 3 |
| **Spam** | 16 | 1 | 683 |
### Per source (three-way rows)
| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9534 | 0.9458 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9905 | 0.9952 |
| Enron | 628 | 0 | 0 | 628 | 0.9841 | 0.9920 |
| Nazario | 222 | 0 | 222 | 0 | 0.9865 | 0.9932 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9866 | 0.9933 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9982 | 0.9991 |

### Top-25 phishing -> spam confusions (3 total, most confident first)

| # | doc_id | P(legit) | P(phish) | P(spam) | tokens (full) | source | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---:|---:|---|---|
| 1 | `aura:Nigerian_Fraud:001698` | 0.0949 | 0.0320 | 0.8730 | 2061 | Nigerian_Fraud | " Please add me to your mailing list. " │ "How to turn 3 into 10,000" This is not a get rich quick scheme, but with a little effort will make a nice little retu... |
| 2 | `aura:Nigerian_Fraud:000337` | 0.1620 | 0.3029 | 0.5352 | 330 | Nigerian_Fraud | Dddddddd R. Rrrrr Message from Vasil Nanchev. │ Hi, This is Vasil Nanchev. I just wanted to inform you about something which is really time sensitive. For the f... |
| 3 | `aura:Nigerian_Fraud:000824` | 0.1349 | 0.3997 | 0.4654 | 1089 | Nigerian_Fraud | Seeking joint-venture partners for ERW oil casing project │ Dear Sir or Madam: We are keenly interested in setting up a new factory to produce 120,000 ERW oil c... |
### Top-25 spam -> phishing confusions (1 total, most confident first)
| # | doc_id | P(legit) | P(phish) | P(spam) | tokens (full) | source | preview (subject │ body, 160 chars) |
|---:|---|---:|---:|---:|---:|---|---|
| 1 | `primary:077835` | 0.0080 | 0.8669 | 0.1251 | 1179 | Assassin | URGENT BUSINESS ASSISTANCE. │ --0-1118764524-1030968705=:44128 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit KEVIN GEORGE ABIDJAN... |

### Tokens over-represented in phishing read as spam vs phishing read correctly

| token | share of phish->spam docs | share of phish->phish docs | ratio |
|---|---:|---:|---:|
| `join` | 1.000 | 0.019 | 49.00 |
| `could` | 1.000 | 0.076 | 13.03 |
| `www` | 1.000 | 0.150 | 6.64 |
| `new` | 1.000 | 0.160 | 6.24 |
| `http` | 1.000 | 0.193 | 5.17 |
| `years` | 1.000 | 0.196 | 5.07 |
| `get` | 1.000 | 0.202 | 4.92 |
| `about` | 1.000 | 0.239 | 4.17 |
| `after` | 1.000 | 0.278 | 3.59 |
| `who` | 1.000 | 0.317 | 3.15 |
| `into` | 1.000 | 0.323 | 3.09 |
| `com` | 1.000 | 0.339 | 2.95 |
| `which` | 1.000 | 0.362 | 2.76 |
| `now` | 1.000 | 0.379 | 2.63 |
| `can` | 1.000 | 0.467 | 2.14 |
| `are` | 1.000 | 0.562 | 1.78 |
| `from` | 1.000 | 0.644 | 1.55 |
| `will` | 1.000 | 0.714 | 1.40 |
| `for` | 1.000 | 0.800 | 1.25 |
| `this` | 1.000 | 0.815 | 1.23 |

### Tokens over-represented in spam read as phishing vs spam read correctly

(too few rows in one of the groups)

### Predicted kind of the flagged rows (informational — includes the unsplit rows, which have no kind label)

| group | n | predicted Phishing | predicted Spam |
|---|---:|---:|---:|
| true positives flagged | 1,201 | 515 | 686 |
| false positives flagged | 13 | 1 | 12 |
| missed positives (kind the model would have given) | 20 | 3 | 17 |

## Patterns (simple lexical statistics, not a model explanation)

### Tokens over-represented in false negatives vs true positives (document frequency, smoothed ratio)

| token | share of FN docs | share of TP docs | ratio |
|---|---:|---:|---:|
| `invite` | 0.250 | 0.002 | 128.94 |
| `cut` | 0.150 | 0.002 | 82.05 |
| `practical` | 0.150 | 0.002 | 82.05 |
| `networking` | 0.200 | 0.002 | 75.36 |
| `audience` | 0.150 | 0.003 | 45.59 |
| `oct` | 0.150 | 0.003 | 45.59 |
| `innovative` | 0.150 | 0.003 | 45.59 |
| `event` | 0.200 | 0.005 | 40.58 |
| `angeles` | 0.150 | 0.004 | 37.30 |
| `anyway` | 0.150 | 0.004 | 37.30 |
| `feature` | 0.150 | 0.005 | 31.56 |
| `york` | 0.150 | 0.005 | 31.56 |
| `subscription` | 0.200 | 0.007 | 27.76 |
| `technologies` | 0.150 | 0.006 | 27.35 |
| `entertainment` | 0.150 | 0.007 | 24.13 |
| `staff` | 0.200 | 0.011 | 19.54 |
| `association` | 0.150 | 0.008 | 19.54 |
| `markets` | 0.150 | 0.009 | 17.84 |
| `hosting` | 0.150 | 0.009 | 17.84 |
| `mailman` | 0.150 | 0.009 | 17.84 |

### Tokens over-represented in false positives vs true negatives

| token | share of FP docs | share of TN docs | ratio |
|---|---:|---:|---:|
| `outlets` | 0.231 | 0.001 | 206.01 |
| `zzz-unspun` | 0.231 | 0.001 | 206.01 |
| `czzz-unspun` | 0.231 | 0.001 | 206.01 |
| `cook` | 0.231 | 0.001 | 206.01 |
| `full-color` | 0.385 | 0.002 | 179.85 |
| `muehlbauer` | 0.231 | 0.002 | 147.15 |
| `vanderkolk` | 0.231 | 0.002 | 147.15 |
| `eletra` | 0.231 | 0.002 | 147.15 |
| `loripatel` | 0.231 | 0.002 | 147.15 |
| `copyedited` | 0.231 | 0.002 | 147.15 |
| `jimduffy86` | 0.231 | 0.002 | 147.15 |
| `asbrand` | 0.231 | 0.002 | 147.15 |
| `patel` | 0.231 | 0.002 | 147.15 |
| `authoritatively` | 0.231 | 0.002 | 147.15 |
| `four-week` | 0.231 | 0.002 | 147.15 |
| `duffy` | 0.231 | 0.002 | 147.15 |
| `mediaunspun` | 0.231 | 0.002 | 147.15 |
| `savviest` | 0.231 | 0.002 | 147.15 |
| `dasbrand` | 0.231 | 0.002 | 147.15 |
| `sendashtml` | 0.231 | 0.002 | 147.15 |

### Composition of the error groups (share of rows)

| group | n | mailing-list / digest | HTML-heavy | very short (<20 content tokens) | truncated (>512) | lure phrase |
|---|---:|---:|---:|---:|---:|---:|
| FP | 13 | 46.2% | 0.0% | 0.0% | 61.5% | 0.0% |
| TN | 1,986 | 56.3% | 0.0% | 0.0% | 38.3% | 0.4% |
| FN | 20 | 30.0% | 0.0% | 10.0% | 35.0% | 0.0% |
| TP | 1,201 | 8.3% | 0.0% | 0.6% | 26.6% | 22.3% |

### Length confound (RUN_LOG entry 2)
| group | n | mean tokens (full) | mean tokens fed | median (full) | % truncated |
|---|---:|---:|---:|---:|---:|
| TP | 1,201 | 978.0 | 279.8 | 243 | 26.6 |
| TN | 1,986 | 732.6 | 358.4 | 392 | 38.3 |
| FP | 13 | 1814.3 | 377.2 | 875 | 61.5 |
| FN | 20 | 1807.8 | 284.1 | 291 | 35.0 |
| CORRECT | 3,187 | 825.1 | 328.8 | 339 | 33.9 |
| WRONG | 33 | 1810.4 | 320.8 | 368 | 45.5 |
| all legitimate | 1,999 | 739.7 | 358.6 | 393 | 38.5 |
| all unwanted | 1,221 | 991.6 | 279.9 | 243 | 26.7 |
| all label3=phish | 521 | 1349.4 | 355.1 | 412 | 39.9 |
| all label3=spam | 700 | 725.3 | 223.9 | 159 | 16.9 |
| length bucket (full tokens) | n | positive rate | recall | FPR | accuracy |
|---|---:|---:|---:|---:|---:|
| <=64 | 129 | 0.767 | 0.9495 | 0.0333 | 0.9535 |
| 65-128 | 427 | 0.590 | 0.9921 | 0.0114 | 0.9906 |
| 129-256 | 699 | 0.395 | 0.9928 | 0.0000 | 0.9971 |
| 257-512 | 870 | 0.308 | 0.9851 | 0.0033 | 0.9931 |
| >512 | 1,095 | 0.298 | 0.9785 | 0.0104 | 0.9863 |
## Summary
- The classifier catches **98.36% of unwanted mail (phishing or spam)** at threshold 0.5 (20 of 1,221 missed) while flagging 0.65% of legitimate mail (13 false alarms). Recall is the headline because a missed phish is the costly error for a small-business inbox.
- Holding false alarms to 1 in 100 legitimate emails keeps recall at 98.69% (threshold 0.434, flag when p >= threshold); at 1 in 1,000 it is 87.88%. The best-F1 threshold is 0.475 (F1 0.9877).
- Three-way: macro-F1 **0.9867** over 3,220 rows with a full label (recall legitimate 99.3%, phishing 98.7%, spam 97.6%). Given that an email is unwanted, the model tells phishing from spam 99.6% of the time (4 phishing read as spam, 1 spam read as phishing). The phishing/spam labels follow corpus provenance (Nazario + Nigerian fraud + synthetic lures = phishing; SpamAssassin + Enron-Spam spam folders = spam), so part of this confusion is corpus style, not lure content.
- On the 275-row credential-lure slice (37 documented phrases, 97.5% of its rows are labelled positive) recall is 100.00% with precision 100.00%: the highest-cost phish are caught at least as well as the average.
- Length confound: missed unwanted mail averages 1808 tokens vs 978 for caught; false positives average 1814 tokens vs 733 for correct legitimate mail (all legitimate 740, all unwanted 992).
  Legitimate mail is ~0.7x longer than unwanted mail on average; the error groups do not follow the length gap, so length is not the main shortcut.
  Phishing rows average 1349 tokens vs spam 725: a length gap between the two unwanted kinds is a second shortcut to watch in the phishing-vs-spam confusion.
- Recall by length is lowest in the <=64-token bucket (94.9%, 99 positives).
- Of the 13 false positives, 46.2% look like mailing-list or digest traffic, 0.0% are HTML-heavy and 0.0% are very short (TN reference: 56.3% / 0.0% / 0.0%).
- Tokens most over-represented in missed unwanted mail vs caught: `invite`, `cut`, `practical`, `networking`, `audience`, `oct`, `innovative`, `event`, `angeles`, `anyway`.
- Tokens most over-represented in false positives vs correct legitimate mail: `outlets`, `zzz-unspun`, `czzz-unspun`, `cook`, `full-color`, `muehlbauer`, `vanderkolk`, `eletra`, `loripatel`, `copyedited`.

	# Error analysis — test3 split (three-way holdout), run `ft-focal` (2026-10-09 10:46)

	All quoted emails are rows of public corpora (source Assassin, CEAS-08, Enron, Nazario, Nigerian_Fraud, TREC-07); nothing here is private mail. Threshold 0.5. Binary positive class = Unwanted (phishing or spam), P(unwanted) = P(phishing) + P(spam).

	## Counts (binary roll-up)

	- rows 3,220 (positives 1,221, 37.9%); label3 counts legit 1,999, phish 521, spam 700
	- recall (unwanted) 0.9836 — 20 missed unwanted emails (false negatives) out of 1,221
	- precision 0.9893 — 13 false alarms out of 1,214 flagged
	- F1 0.9864, accuracy 0.9898, specificity 0.9935
	- ROC-AUC 0.9993, PR-AUC 0.9989; recall at 1% FPR 0.9869, at 0.1% FPR 0.8788
	- credential-lure slice: 275 rows, 268 positive (97.5% label purity: slice precision cannot exceed what the phrase list selects), recall 1.0000, precision 1.0000, 0 missed

	## Top-25 false negatives (missed unwanted mail, most confident first)

	\| # \| doc_id \| P(unwanted) \| tokens (full) \| source \| label_set \| preview (subject │ body, 160 chars) \|
	\|---:\|---\|---:\|---:\|---\|---\|---\|
	\| 1 \| `primary:191966` \| 0.0224 \| 86 \| Enron \| spam \| re : saturday . here new and │ good day : i tried to call your 3 three time but your phone is not available . i think you did a mistake during filling the form ... \|
	\| 2 \| `primary:148887` \| 0.0467 \| 1097 \| Enron \| spam \| southern california demographic analysis workshop announcement ( january charset = windows - 1252 " > │ analyzing southern california communities : demographic ... \|
	\| 3 \| `aura:Nazario:000097` \| 0.1782 \| 39 \| Nazario \| phish \| Oil Payment Approval │ Find the attached document via Google secure to view closing statement. All parties have completed Documents. please sign these at your e... \|
	\| 4 \| `primary:010013` \| 0.2112 \| 135 \| Enron \| spam \| new birthday calendar │ hi i ' m setting up a birthday calendar and need your help . just click the link below and enter your birthday details . ( it ' s quick ... \|
	\| 5 \| `primary:007176` \| 0.2300 \| 281 \| Assassin \| spam \| TOMORROW -- Password: Soleil -- TV Casting │ Tomorrow will be huge. Password: Soleil has become the premiere Thursday night in New York - with a chill after-wor... \|
	\| 6 \| `aura:Nazario:000643` \| 0.2584 \| 368 \| Nazario \| phish \| Phishing Email Corpus │ Good Morning, Jose, My name is Michael Kotson, and I am a Cybersecurity researcher at MIT Lincoln Laboratory. I am developing a lecture ... \|
	\| 7 \| `primary:181246` \| 0.2858 \| 1051 \| Assassin \| spam \| [ILUG] (no subject) │ Suppliers of Computers, Printers, etc. & Consumables Central Data Supplies 24, Middleton Road, Banbury, Oxon, OX16 4QJ. Tel: 01295 271666 ... \|
	\| 8 \| `primary:095405` \| 0.2967 \| 40 \| Enron \| spam \| pocket micro scope id : sxo 683 │ re : ppql 308243023 not interested , please let us know : powerlinepromo @ yahoo . com \|
	\| 9 \| `primary:014300` \| 0.3203 \| 301 \| Enron \| spam \| cecil county networking lunch │ networking lunch chesapeake bay golf club in north east wednesday , november 17 th 11 : 15 am smallcompanies . com in conjunctio... \|
	\| 10 \| `primary:001939` \| 0.3348 \| 1153 \| Assassin \| spam \| Invite: Content Management Summit, Oct. 10th New York City │ ![CMS](http://www.cmsummit.com/images/right_02.gif) ![CMS](http://www.cmsummit.com/images/cms_title... \|
	\| 11 \| `primary:033220` \| 0.3678 \| 28444 \| Assassin \| spam \| Wir suchen Erotikdarsteller! │ This is a Multipart MIME message. Since your mail reader does not understand this format, some or all of this message may not be ... \|
	\| 12 \| `primary:025244` \| 0.3789 \| 8 \| Enron \| spam \| message subject │ goodbye \|
	\| 13 \| `primary:133074` \| 0.3791 \| 86 \| Enron \| spam \| his lips , his nose , │ good day : i tried to call your 3 three time but your phone is not available . i think you did a mistake during filling the form . anywa... \|
	\| 14 \| `aura:Nigerian_Fraud:003049` \| 0.3823 \| 516 \| Nigerian_Fraud \| phish \| Invitation from aditya monu │ This is a multi-part message in MIME format. --incrowd_6506e2e6636163656626f7892e636f26d05b3743 Content-Type: text/plain; charset=... \|
	\| 15 \| `aura:Nazario:000299` \| 0.3973 \| 46 \| Nazario \| phish \| RE: IT Administrator │ IT Administrator desk: has currently upgraded our mailbox service quote to 150GB space Faculty/Staff are hereby advice to click on WEBMAI... \|
	\| 16 \| `primary:046569` \| 0.4039 \| 1078 \| Enron \| spam \| invite : content management summit , oct . 10 th new york city │ dear friends , the online content business is doing fantastic . sound crazy ? perhaps . with ad... \|
	\| 17 \| `primary:135631` \| 0.4338 \| 18 \| Enron \| spam \| reply : shy mom wants to date you │ hello there ! sode \|
	\| 18 \| `primary:148857` \| 0.4750 \| 180 \| Assassin \| spam \| send REAL PAPER GREETING CARDS on-line! │ Cards In Advance Birthday Anniversary Get Well Soon Thinking of you Congratulations Sympathy Engagement Good Luck New ... \|
	\| 19 \| `primary:129677` \| 0.4929 \| 510 \| Assassin \| spam \| Proposal for cooperation │ Hello. I meant to contact http://taint.org webmaster, Am I correct, I hope? Actually, I am not quite sure... Did I or Natasha Fedorov... \|
	\| 20 \| `primary:075760` \| 0.4969 \| 719 \| Enron \| spam \| greece based professionals and organizations │ you areinvited to register on training consortiumjoin us by registering for free as a trainer , consultant , or s... \|

	## Top-25 false positives (legitimate flagged, most confident first)

	\| # \| doc_id \| P(unwanted) \| tokens (full) \| source \| label_set \| preview (subject │ body, 160 chars) \|
	\|---:\|---\|---:\|---:\|---\|---\|---\|
	\| 1 \| `primary:144048` \| 0.9853 \| 46 \| CEAS-08 \| legit \| Whiskas Scratching Post │ HTML Message - =?windows-1252?B?V2hpc2thcyBTY3JhdGNoaW5nIFBvc3QA?= \|
	\| 2 \| `primary:199386` \| 0.9471 \| 2206 \| Assassin \| legit \| Back to School with Buy4Now │ This is a multi-part message in MIME format. ------=_NextPart_000_22A2C_01C23F09.09760420 Content-Type: text/plain; charset="iso-8... \|
	\| 3 \| `primary:091801` \| 0.9436 \| 66 \| CEAS-08 \| legit \| Lots Of SPAM │ Hi List, i have posted my RAW email in http://pastebin.ca/918849 , i'm receiving 1000 to 4000 per day this king of mesages. SA also skipping this... \|
	\| 4 \| `primary:152763` \| 0.8427 \| 1228 \| CEAS-08 \| legit \| SAVE 30% on Best Recipe Cookbooks │ Untitled Document We don't want you to miss out on our e-mail offers! Please add our e-mail address, arfldpykmjrzc@americast... \|
	\| 5 \| `primary:165228` \| 0.8246 \| 4917 \| Assassin \| legit \| United We Fall? │ This is a multi-part message in MIME format... ------------=_1029244856-11739-4 Content-Type: text/plain; charset="iso-8859-1" Content-Disposi... \|
	\| 6 \| `primary:041713` \| 0.7668 \| 340 \| TREC-07 \| legit \| Electric Transmission Week complimentary trial │ 1105_ETW_Trial View this message as a Web page. FREE Trial Offer You're invited to receive a FREE trial subscri... \|
	\| 7 \| `primary:103927` \| 0.7592 \| 1587 \| CEAS-08 \| legit \| FREE Shipping on "America's Best Lost Recipes" Cookbook - Plus Free OXO(R) Tongs │ Untitled Document We don't want you to miss out on our e-mail offers! Please ... \|
	\| 8 \| `primary:055965` \| 0.7247 \| 5197 \| Assassin \| legit \| Brother, Can You Spare a Jet? │ This is a multi-part message in MIME format... ------------=_1029157891-23956-4 Content-Type: text/plain; charset="iso-8859-1" C... \|
	\| 9 \| `primary:170237` \| 0.6660 \| 88 \| CEAS-08 \| legit \| The jet set │ Well, it appears you have all taken an idea and run with it. My point was that "Knocked Up" appeared quite the good film, and the Williams one so ... \|
	\| 10 \| `primary:117398` \| 0.6510 \| 5959 \| Assassin \| legit \| AOL Keyword: Fraud Probe │ This is a multi-part message in MIME format... ------------=_1033046883-23957-4 Content-Type: text/plain; charset="iso-8859-1" Conten... \|
	\| 11 \| `primary:168120` \| 0.5796 \| 268 \| TREC-07 \| legit \| Please confirm your request to join Deficiv │ Hello deficient@speedy.uwaterloo.ca, We have received your request to join the Deficiv group hosted by Yahoo! Grou... \|
	\| 12 \| `primary:057514` \| 0.5173 \| 809 \| CEAS-08 \| legit \| WIN a $6,000 Cook Islands getaway │ HotelClub Newsletter Dear Tony, Subscribe to any free NEWS.com.au newsletter for your chance to WIN a $6,000 Cook Islands ge... \|
	\| 13 \| `primary:021911` \| 0.5051 \| 875 \| CEAS-08 \| legit \| It's here. Dell's first tablet PC. │ Dell would like to introduce our first tablet PC, the Latitude XT. Part of the trusted Latitude line, the XT gives you all ... \|

	## Three-way (legitimate / phishing / spam)

	Decision rule: unwanted iff P(phishing)+P(spam) >= threshold, then kind = argmax(P(phishing), P(spam)); the 3x3 matrix therefore collapses exactly onto the binary confusion matrix. argmax-of-3 reported alongside.

	Rows with a full label: 3,220; unwanted-but-unsplit rows (binary roll-up only): 0.

	\| true \ predicted \| Legitimate \| Phishing \| Spam \|
	\|---\|---:\|---:\|---:\|
	\| Legitimate \| 1,986 \| 1 \| 12 \|
	\| Phishing \| 4 \| 514 \| 3 \|
	\| Spam \| 16 \| 1 \| 683 \|

	- accuracy 0.9885, macro-F1 0.9867 (over 3 classes; over the 3 present: 0.9867); argmax-of-3 rule: macro-F1 0.9858, 6 rows differ between the rules

	\| class \| precision \| recall \| F1 \| support \|
	\|---\|---:\|---:\|---:\|---:\|
	\| Legitimate \| 0.9900 \| 0.9935 \| 0.9918 \| 1,999 \|
	\| Phishing \| 0.9961 \| 0.9866 \| 0.9913 \| 521 \|
	\| Spam \| 0.9785 \| 0.9757 \| 0.9771 \| 700 \|

	### Phishing vs spam on the 1,221 true unwanted rows

	Kind decision alone (given the row is unwanted, argmax(P(phishing), P(spam))): accuracy 0.9959; phishing read as spam 4, spam read as phishing 1.

	\| true \ predicted (roll-up rule) \| Legitimate (missed) \| Phishing \| Spam \|
	\|---\|---:\|---:\|---:\|
	\| Phishing \| 4 \| 514 \| 3 \|
	\| Spam \| 16 \| 1 \| 683 \|

	### Per source (three-way rows)

	\| source \| n \| Legitimate \| Phishing \| Spam \| accuracy \| macro-F1 (present classes) \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| Assassin \| 236 \| 164 \| 0 \| 72 \| 0.9534 \| 0.9458 \|
	\| CEAS-08 \| 736 \| 736 \| 0 \| 0 \| 0.9905 \| 0.9952 \|
	\| Enron \| 628 \| 0 \| 0 \| 628 \| 0.9841 \| 0.9920 \|
	\| Nazario \| 222 \| 0 \| 222 \| 0 \| 0.9865 \| 0.9932 \|
	\| Nigerian_Fraud \| 299 \| 0 \| 299 \| 0 \| 0.9866 \| 0.9933 \|
	\| TREC-07 \| 1,099 \| 1,099 \| 0 \| 0 \| 0.9982 \| 0.9991 \|

	### Top-25 phishing -> spam confusions (3 total, most confident first)

	\| # \| doc_id \| P(legit) \| P(phish) \| P(spam) \| tokens (full) \| source \| preview (subject │ body, 160 chars) \|
	\|---:\|---\|---:\|---:\|---:\|---:\|---\|---\|
	\| 1 \| `aura:Nigerian_Fraud:001698` \| 0.0949 \| 0.0320 \| 0.8730 \| 2061 \| Nigerian_Fraud \| " Please add me to your mailing list. " │ "How to turn 3 into 10,000" This is not a get rich quick scheme, but with a little effort will make a nice little retu... \|
	\| 2 \| `aura:Nigerian_Fraud:000337` \| 0.1620 \| 0.3029 \| 0.5352 \| 330 \| Nigerian_Fraud \| Dddddddd R. Rrrrr Message from Vasil Nanchev. │ Hi, This is Vasil Nanchev. I just wanted to inform you about something which is really time sensitive. For the f... \|
	\| 3 \| `aura:Nigerian_Fraud:000824` \| 0.1349 \| 0.3997 \| 0.4654 \| 1089 \| Nigerian_Fraud \| Seeking joint-venture partners for ERW oil casing project │ Dear Sir or Madam: We are keenly interested in setting up a new factory to produce 120,000 ERW oil c... \|

	### Top-25 spam -> phishing confusions (1 total, most confident first)

	\| # \| doc_id \| P(legit) \| P(phish) \| P(spam) \| tokens (full) \| source \| preview (subject │ body, 160 chars) \|
	\|---:\|---\|---:\|---:\|---:\|---:\|---\|---\|
	\| 1 \| `primary:077835` \| 0.0080 \| 0.8669 \| 0.1251 \| 1179 \| Assassin \| URGENT BUSINESS ASSISTANCE. │ --0-1118764524-1030968705=:44128 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit KEVIN GEORGE ABIDJAN... \|

	### Tokens over-represented in phishing read as spam vs phishing read correctly

	\| token \| share of phish->spam docs \| share of phish->phish docs \| ratio \|
	\|---\|---:\|---:\|---:\|
	\| `join` \| 1.000 \| 0.019 \| 49.00 \|
	\| `could` \| 1.000 \| 0.076 \| 13.03 \|
	\| `www` \| 1.000 \| 0.150 \| 6.64 \|
	\| `new` \| 1.000 \| 0.160 \| 6.24 \|
	\| `http` \| 1.000 \| 0.193 \| 5.17 \|
	\| `years` \| 1.000 \| 0.196 \| 5.07 \|
	\| `get` \| 1.000 \| 0.202 \| 4.92 \|
	\| `about` \| 1.000 \| 0.239 \| 4.17 \|
	\| `after` \| 1.000 \| 0.278 \| 3.59 \|
	\| `who` \| 1.000 \| 0.317 \| 3.15 \|
	\| `into` \| 1.000 \| 0.323 \| 3.09 \|
	\| `com` \| 1.000 \| 0.339 \| 2.95 \|
	\| `which` \| 1.000 \| 0.362 \| 2.76 \|
	\| `now` \| 1.000 \| 0.379 \| 2.63 \|
	\| `can` \| 1.000 \| 0.467 \| 2.14 \|
	\| `are` \| 1.000 \| 0.562 \| 1.78 \|
	\| `from` \| 1.000 \| 0.644 \| 1.55 \|
	\| `will` \| 1.000 \| 0.714 \| 1.40 \|
	\| `for` \| 1.000 \| 0.800 \| 1.25 \|
	\| `this` \| 1.000 \| 0.815 \| 1.23 \|

	### Tokens over-represented in spam read as phishing vs spam read correctly

	(too few rows in one of the groups)

	### Predicted kind of the flagged rows (informational — includes the unsplit rows, which have no kind label)

	\| group \| n \| predicted Phishing \| predicted Spam \|
	\|---\|---:\|---:\|---:\|
	\| true positives flagged \| 1,201 \| 515 \| 686 \|
	\| false positives flagged \| 13 \| 1 \| 12 \|
	\| missed positives (kind the model would have given) \| 20 \| 3 \| 17 \|

	## Patterns (simple lexical statistics, not a model explanation)

	### Tokens over-represented in false negatives vs true positives (document frequency, smoothed ratio)

	\| token \| share of FN docs \| share of TP docs \| ratio \|
	\|---\|---:\|---:\|---:\|
	\| `invite` \| 0.250 \| 0.002 \| 128.94 \|
	\| `cut` \| 0.150 \| 0.002 \| 82.05 \|
	\| `practical` \| 0.150 \| 0.002 \| 82.05 \|
	\| `networking` \| 0.200 \| 0.002 \| 75.36 \|
	\| `audience` \| 0.150 \| 0.003 \| 45.59 \|
	\| `oct` \| 0.150 \| 0.003 \| 45.59 \|
	\| `innovative` \| 0.150 \| 0.003 \| 45.59 \|
	\| `event` \| 0.200 \| 0.005 \| 40.58 \|
	\| `angeles` \| 0.150 \| 0.004 \| 37.30 \|
	\| `anyway` \| 0.150 \| 0.004 \| 37.30 \|
	\| `feature` \| 0.150 \| 0.005 \| 31.56 \|
	\| `york` \| 0.150 \| 0.005 \| 31.56 \|
	\| `subscription` \| 0.200 \| 0.007 \| 27.76 \|
	\| `technologies` \| 0.150 \| 0.006 \| 27.35 \|
	\| `entertainment` \| 0.150 \| 0.007 \| 24.13 \|
	\| `staff` \| 0.200 \| 0.011 \| 19.54 \|
	\| `association` \| 0.150 \| 0.008 \| 19.54 \|
	\| `markets` \| 0.150 \| 0.009 \| 17.84 \|
	\| `hosting` \| 0.150 \| 0.009 \| 17.84 \|
	\| `mailman` \| 0.150 \| 0.009 \| 17.84 \|

	### Tokens over-represented in false positives vs true negatives

	\| token \| share of FP docs \| share of TN docs \| ratio \|
	\|---\|---:\|---:\|---:\|
	\| `outlets` \| 0.231 \| 0.001 \| 206.01 \|
	\| `zzz-unspun` \| 0.231 \| 0.001 \| 206.01 \|
	\| `czzz-unspun` \| 0.231 \| 0.001 \| 206.01 \|
	\| `cook` \| 0.231 \| 0.001 \| 206.01 \|
	\| `full-color` \| 0.385 \| 0.002 \| 179.85 \|
	\| `muehlbauer` \| 0.231 \| 0.002 \| 147.15 \|
	\| `vanderkolk` \| 0.231 \| 0.002 \| 147.15 \|
	\| `eletra` \| 0.231 \| 0.002 \| 147.15 \|
	\| `loripatel` \| 0.231 \| 0.002 \| 147.15 \|
	\| `copyedited` \| 0.231 \| 0.002 \| 147.15 \|
	\| `jimduffy86` \| 0.231 \| 0.002 \| 147.15 \|
	\| `asbrand` \| 0.231 \| 0.002 \| 147.15 \|
	\| `patel` \| 0.231 \| 0.002 \| 147.15 \|
	\| `authoritatively` \| 0.231 \| 0.002 \| 147.15 \|
	\| `four-week` \| 0.231 \| 0.002 \| 147.15 \|
	\| `duffy` \| 0.231 \| 0.002 \| 147.15 \|
	\| `mediaunspun` \| 0.231 \| 0.002 \| 147.15 \|
	\| `savviest` \| 0.231 \| 0.002 \| 147.15 \|
	\| `dasbrand` \| 0.231 \| 0.002 \| 147.15 \|
	\| `sendashtml` \| 0.231 \| 0.002 \| 147.15 \|

	### Composition of the error groups (share of rows)

	\| group \| n \| mailing-list / digest \| HTML-heavy \| very short (<20 content tokens) \| truncated (>512) \| lure phrase \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| FP \| 13 \| 46.2% \| 0.0% \| 0.0% \| 61.5% \| 0.0% \|
	\| TN \| 1,986 \| 56.3% \| 0.0% \| 0.0% \| 38.3% \| 0.4% \|
	\| FN \| 20 \| 30.0% \| 0.0% \| 10.0% \| 35.0% \| 0.0% \|
	\| TP \| 1,201 \| 8.3% \| 0.0% \| 0.6% \| 26.6% \| 22.3% \|

	### Length confound (RUN_LOG entry 2)

	\| group \| n \| mean tokens (full) \| mean tokens fed \| median (full) \| % truncated \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| TP \| 1,201 \| 978.0 \| 279.8 \| 243 \| 26.6 \|
	\| TN \| 1,986 \| 732.6 \| 358.4 \| 392 \| 38.3 \|
	\| FP \| 13 \| 1814.3 \| 377.2 \| 875 \| 61.5 \|
	\| FN \| 20 \| 1807.8 \| 284.1 \| 291 \| 35.0 \|
	\| CORRECT \| 3,187 \| 825.1 \| 328.8 \| 339 \| 33.9 \|
	\| WRONG \| 33 \| 1810.4 \| 320.8 \| 368 \| 45.5 \|
	\| all legitimate \| 1,999 \| 739.7 \| 358.6 \| 393 \| 38.5 \|
	\| all unwanted \| 1,221 \| 991.6 \| 279.9 \| 243 \| 26.7 \|
	\| all label3=phish \| 521 \| 1349.4 \| 355.1 \| 412 \| 39.9 \|
	\| all label3=spam \| 700 \| 725.3 \| 223.9 \| 159 \| 16.9 \|

	\| length bucket (full tokens) \| n \| positive rate \| recall \| FPR \| accuracy \|
	\|---\|---:\|---:\|---:\|---:\|---:\|
	\| <=64 \| 129 \| 0.767 \| 0.9495 \| 0.0333 \| 0.9535 \|
	\| 65-128 \| 427 \| 0.590 \| 0.9921 \| 0.0114 \| 0.9906 \|
	\| 129-256 \| 699 \| 0.395 \| 0.9928 \| 0.0000 \| 0.9971 \|
	\| 257-512 \| 870 \| 0.308 \| 0.9851 \| 0.0033 \| 0.9931 \|
	\| >512 \| 1,095 \| 0.298 \| 0.9785 \| 0.0104 \| 0.9863 \|

	## Summary

	- The classifier catches 98.36% of unwanted mail (phishing or spam) at threshold 0.5 (20 of 1,221 missed) while flagging 0.65% of legitimate mail (13 false alarms). Recall is the headline because a missed phish is the costly error for a small-business inbox.
	- Holding false alarms to 1 in 100 legitimate emails keeps recall at 98.69% (threshold 0.434, flag when p >= threshold); at 1 in 1,000 it is 87.88%. The best-F1 threshold is 0.475 (F1 0.9877).
	- Three-way: macro-F1 0.9867 over 3,220 rows with a full label (recall legitimate 99.3%, phishing 98.7%, spam 97.6%). Given that an email is unwanted, the model tells phishing from spam 99.6% of the time (4 phishing read as spam, 1 spam read as phishing). The phishing/spam labels follow corpus provenance (Nazario + Nigerian fraud + synthetic lures = phishing; SpamAssassin + Enron-Spam spam folders = spam), so part of this confusion is corpus style, not lure content.
	- On the 275-row credential-lure slice (37 documented phrases, 97.5% of its rows are labelled positive) recall is 100.00% with precision 100.00%: the highest-cost phish are caught at least as well as the average.
	- Length confound: missed unwanted mail averages 1808 tokens vs 978 for caught; false positives average 1814 tokens vs 733 for correct legitimate mail (all legitimate 740, all unwanted 992).
	Legitimate mail is ~0.7x longer than unwanted mail on average; the error groups do not follow the length gap, so length is not the main shortcut.
	Phishing rows average 1349 tokens vs spam 725: a length gap between the two unwanted kinds is a second shortcut to watch in the phishing-vs-spam confusion.
	- Recall by length is lowest in the <=64-token bucket (94.9%, 99 positives).
	- Of the 13 false positives, 46.2% look like mailing-list or digest traffic, 0.0% are HTML-heavy and 0.0% are very short (TN reference: 56.3% / 0.0% / 0.0%).
	- Tokens most over-represented in missed unwanted mail vs caught: `invite`, `cut`, `practical`, `networking`, `audience`, `oct`, `innovative`, `event`, `angeles`, `anyway`.
	- Tokens most over-represented in false positives vs correct legitimate mail: `outlets`, `zzz-unspun`, `czzz-unspun`, `cook`, `full-color`, `muehlbauer`, `vanderkolk`, `eletra`, `loripatel`, `copyedited`.