# Data report — 2026-10-08T06:41:51+00:00

spam_policy `positive_class` (positive = phishing/spam, unwanted); quote stripping on; seed 42; holdout source `TREC-06`; elapsed 139.0s; peak RSS 6.58 GB.

## Sources
- labelled primary: `puyang2025/seven-phishing-email-datasets` @ `b7f3f8d99ebd4fd0af0a5afe027420e53f08d1a4` (203,017 rows, hub splits {'train': 162413, 'test': 40604} concatenated; the hub split leaks and is ignored)
- pretraining corpus: `corbt/enron-emails` @ `cfc06c758093d90993abce1a43668fb7357258a6` (517,401 rows)
- excluded (no body text / 92% subset): `it4lia/PhishingEmailCuratedDatasets_Cleaned`, `zefang-liu/phishing-email-dataset`

## Splits
| split | rows | positives | negatives | base rate |
|---|---|---|---|---|
| finetune_train | 157,113 | 75,675 | 81,438 | 0.4817 |
| finetune_val | 17,643 | 8,642 | 9,001 | 0.4898 |
| test_holdout | 16,302 | 3,894 | 12,408 | 0.2389 |
| pretrain (MLM) | 226,273 | - | - | 407,902,364 chars |

### finetune_train by source
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 3,705 | 1,388 | 5,093 | 27.3% |
| CEAS-08 | 15,547 | 18,631 | 34,178 | 54.5% |
| Enron | 12,741 | 11,482 | 24,223 | 47.4% |
| TREC-05 | 27,663 | 18,723 | 46,386 | 40.4% |
| TREC-07 | 21,782 | 25,451 | 47,233 | 53.9% |

### finetune_val by source
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 380 | 136 | 516 | 26.4% |
| CEAS-08 | 1,699 | 2,184 | 3,883 | 56.2% |
| Enron | 1,381 | 1,327 | 2,708 | 49.0% |
| TREC-05 | 2,997 | 2,052 | 5,049 | 40.6% |
| TREC-07 | 2,544 | 2,943 | 5,487 | 53.6% |

### test_holdout by source
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| TREC-06 | 12,408 | 3,894 | 16,302 | 23.9% |

## Normalisation and dials
- common.normalize_text: NFKC -> lowercase -> keep [a-z0-9] only (import it, never re-derive it; NFKC is an extension of the verify_data.md section 7 wording)
- dials: {'key_prefix_chars': 300, 'key_prefix_min_len': 80, 'body_key_min_alnum': 20, 'pretrain_min_alnum': 50, 'quote_strip_min_alnum': 200, 'shingle_cap': 2000}

## Drops and removals
- labelled rows with <20 alphanumerics (subject+body): 715 rows
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| CEAS-08 | 0 | 1 | 1 | 100.0% |
| Enron | 12 | 18 | 30 | 60.0% |
| TREC-05 | 51 | 526 | 577 | 91.2% |
| TREC-06 | 3 | 95 | 98 | 96.9% |
| TREC-07 | 0 | 9 | 9 | 100.0% |
- Ling excluded (academic list corpus, 2,859 rows by label {'0': 2401, '1': 458})
- pool label conflicts (identical normalised body labelled both 0 and 1; all members dropped): 18 rows in 8 groups
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Enron | 8 | 0 | 8 | 0.0% |
| TREC-05 | 0 | 10 | 10 | 100.0% |
- pool exact duplicates (k_full, keep first): 6,715 rows
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 1 | 184 | 185 | 99.5% |
| CEAS-08 | 58 | 1,020 | 1,078 | 94.6% |
| Enron | 1,641 | 693 | 2,334 | 29.7% |
| TREC-05 | 1,611 | 772 | 2,383 | 32.4% |
| TREC-07 | 31 | 704 | 735 | 95.8% |
- decontamination against the holdout: {'key_match_only': 4, 'shingle_link_only': 1516, 'both': 132, 'total': 1652}
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 1 | 10 | 11 | 90.9% |
| CEAS-08 | 8 | 6 | 14 | 42.9% |
| Enron | 8 | 456 | 464 | 98.3% |
| TREC-05 | 7 | 863 | 870 | 99.2% |
| TREC-07 | 1 | 292 | 293 | 99.7% |
- holdout internal exact duplicates kept (test set is never deduped): 116

## Near-duplicate clustering
- 50-char shingles, stride 25, first 1500 chars, link at >= 3 shared; 1 boilerplate shingles over cap 2,000 ignored (touching 2,930 rows)
- links at the removal stage (pool + holdout index): {'pool_pool': 3633009, 'pool_holdout': 5150, 'holdout_holdout': 15456}
- clusters 114,261 (mixed source x label: 1,781; largest 7,069); val fraction target 0.1 achieved 0.101
| stratum | rows | clusters | val target | val rows | val frac | clusters too large |
|---|---|---|---|---|---|---|
| Assassin|0 | 3,811 | 2,733 | 381 | 381 | 0.1 | 1 |
| Assassin|1 | 1,436 | 834 | 144 | 144 | 0.1003 | 0 |
| CEAS-08|0 | 16,993 | 11,149 | 1,699 | 1,699 | 0.1 | 1 |
| CEAS-08|1 | 21,586 | 12,202 | 2,159 | 2,159 | 0.1 | 1 |
| Enron|0 | 13,589 | 12,731 | 1,359 | 1,359 | 0.1 | 0 |
| Enron|1 | 11,393 | 8,589 | 1,139 | 1,139 | 0.1 | 0 |
| TREC-05|0 | 30,181 | 24,292 | 3,018 | 3,018 | 0.1 | 0 |
| TREC-05|1 | 21,933 | 14,361 | 2,193 | 2,229 | 0.1016 | 0 |
| TREC-07|0 | 25,438 | 11,740 | 2,544 | 2,544 | 0.1 | 1 |
| TREC-07|1 | 28,396 | 15,630 | 2,840 | 2,971 | 0.1046 | 0 |

## Pretraining corpus (corbt/enron-emails)
| stage | rows |
|---|---|
| rows in | 517,401 |
| exact dups (subject+body, raw) | 272,905 |
| < 50 alnum after stripping | 14,027 |
| exact dups after stripping | 107 |
| matches a holdout key (removed) | 0 |
| matches a fine-tune val key (removed) | 4,004 |
| >= 3 shingles shared with a holdout row (removed) | 85 |
| >= 3 shingles shared with a fine-tune val row (kept, counted) | 1,095 |
| overlaps a fine-tune train key (kept, allowed) | 35,172 |
| rows out | 226,273 |
- near-dup screen: 88 links to 19 holdout rows (by label {'0': 18, '1': 1}); 5,345 links to 883 val rows
- quote/forward stripping on: 21,832 bodies cut, 70,907 had a marker but kept the full body (< 200 alnum above it); marker hits {'forwarded': 33089, 'header_block': 5428, 'original': 54222}; retained chars 0.8948 of all bodies, 0.2548 of cut bodies
- 407,902,364 chars out, mean 1802.7 chars/doc

## Leakage assertions
| check | count | asserted zero | pass |
|---|---|---|---|
| holdout_vs_train_key_rows | 0 | True | True |
| holdout_vs_val_key_rows | 0 | True | True |
| holdout_vs_pretrain_key_rows | 0 | True | True |
| holdout_vs_train_shingle_links | 0 | True | True |
| holdout_vs_val_shingle_links | 0 | True | True |
| holdout_vs_pretrain_shingle_links | 0 | True | True |
| train_vs_val_k_full_rows | 0 | True | True |
| train_vs_val_any_key_rows | 0 | False | True |
| train_vs_val_shingle_links | 0 | False | True |
| val_vs_pretrain_key_rows | 0 | True | True |
| val_vs_pretrain_shingle_links | 5,336 | False | True |
| train_vs_pretrain_key_overlap | 102,100 | False | True |
| holdout_rows_in_train_or_val_by_source | 0 | True | True |
| test_holdout_rows_from_other_sources | 0 | True | True |
| pretrain_rows | 226,273 | False | True |

## Files
- `finetune_train`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet`
- `finetune_val`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet`
- `test_holdout`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet`
- `pretrain`: `/home/nathank/BITS/dev/Phishing Model/data/processed/pretrain.parquet`

## Three-way labels — build3 2026-10-08T20:21:36+00:00

label3 {'0': 'legit', '1': 'phish', '2': 'spam', '-1': 'unwanted'}; display names {'0': 'Legitimate', '1': 'Phishing', '2': 'Spam'}; binary roll-up "Unwanted (phishing or spam)" = P(phishing)+P(spam). Positive rows by provenance: {'Assassin': 2, 'Enron': 2, 'Ling': 2, 'TREC-05': -1, 'TREC-06': -1, 'TREC-07': -1, 'CEAS-08': -1, 'Nazario': 1, 'Nigerian_Fraud': 1}. unchanged: 1 for phish / spam / unwanted, 0 for legit. Elapsed 66.1s; peak RSS 3.23 GB; binary base from snapshot (`/home/nathank/BITS/dev/Phishing Model/data/processed/binary_snapshot`).

### New sources
- `kudzaiprichard/aura-phishing-email-corpus` / `Nazario.csv` @ `539567cee21d03c83bd2fb1b142c18a80904eb72`: 1,565 rows (1,565 used), label3 1 (phishing by provenance)
- `kudzaiprichard/aura-phishing-email-corpus` / `Nigerian_Fraud.csv` @ `539567cee21d03c83bd2fb1b142c18a80904eb72`: 3,332 rows (3,332 used), label3 1 (phishing by provenance)
- bigshp (`locuoco/the-biggest-spam-ham-phish-email-dataset-300000`): research verdict **DO_NOT_USE**, not ingested (`docs/research/bigshp_research.md`)

| source | loaded | dropped | kept | -> train | -> val | -> test_holdout3 |
|---|---|---|---|---|---|---|
| Nazario | 1,565 | 58 | 1,507 | 1,159 | 126 | 222 |
| Nigerian_Fraud | 3,332 | 828 | 2,504 | 1,991 | 214 | 299 |

- drops by reason: {'Nazario': {'mbox_internal_data': 2, 'within_new_k_full_dup': 56}, 'Nigerian_Fraud': {'shingle_link_test_holdout': 235, 'within_new_k_full_dup': 98, 'key_match_finetune_train': 54, 'near_dup_of_both_train_and_val': 424, 'key_match_test_holdout': 13, 'lt_20_alnum': 1, 'key_match_finetune_val': 3}}
- near-dup pins: {'clusters_total': 3239, 'clusters_pinned_train_by_train_or_pretrain_link': 238, 'clusters_pinned_val_by_val_link': 23, 'clusters_dropped_linked_to_both_train_and_val': 17, 'rows_dropped_linked_to_both_train_and_val': 424, 'clusters_with_a_holdout_linked_member': 235, 'rows_linked_to_train': 329, 'rows_linked_to_pretrain': 33, 'rows_linked_to_val': 35, 'rows_dropped_holdout_shingle_link': 235, 'largest_cluster': 52}
- bigshp label contradictions among exact matches: 0

### test_holdout3 rows taken from finetune_val
- legitimate from ['CEAS-08', 'TREC-07', 'Assassin'], spam from ['Assassin', 'Enron']; whole near-dup clusters only; excluded candidates by reason: {'Enron': {'cluster_has_an_ineligible_member': 261, 'shingle_link_to_a_new_row': 28, 'shingle_link_pretrain': 4}, 'Assassin': {'shingle_link_pretrain': 8, 'cluster_has_an_ineligible_member': 7}, 'CEAS-08': {'shingle_link_pretrain': 1}, 'TREC-07': {'cluster_has_an_ineligible_member': 1}}
- per class: {'Legitimate': {'target_rows': 2000, 'eligible_rows': 4621, 'fraction': 0.4328, 'moved_rows': 1999}, 'Spam': {'target_rows': 700, 'eligible_rows': 1155, 'fraction': 0.6061, 'moved_rows': 700}}
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 164 | 0 | 72 | 0 | 236 |
| CEAS-08 | 736 | 0 | 0 | 0 | 736 |
| Enron | 0 | 0 | 628 | 0 | 628 |
| TREC-07 | 1,099 | 0 | 0 | 0 | 1,099 |

### Splits by label3
#### finetune_train: 160,263 rows, by label3 {'-1': 62805, '0': 81438, '1': 3150, '2': 12870}, binary positives 78,825 (0.4918)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 3,705 | 0 | 1,388 | 0 | 5,093 |
| CEAS-08 | 15,547 | 0 | 0 | 18,631 | 34,178 |
| Enron | 12,741 | 0 | 11,482 | 0 | 24,223 |
| Nazario | 0 | 1,159 | 0 | 0 | 1,159 |
| Nigerian_Fraud | 0 | 1,991 | 0 | 0 | 1,991 |
| TREC-05 | 27,663 | 0 | 0 | 18,723 | 46,386 |
| TREC-07 | 21,782 | 0 | 0 | 25,451 | 47,233 |

#### finetune_val: 15,284 rows, by label3 {'-1': 7179, '0': 7002, '1': 340, '2': 763}, binary positives 8,282 (0.5419)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 216 | 0 | 64 | 0 | 280 |
| CEAS-08 | 963 | 0 | 0 | 2,184 | 3,147 |
| Enron | 1,381 | 0 | 699 | 0 | 2,080 |
| Nazario | 0 | 126 | 0 | 0 | 126 |
| Nigerian_Fraud | 0 | 214 | 0 | 0 | 214 |
| TREC-05 | 2,997 | 0 | 0 | 2,052 | 5,049 |
| TREC-07 | 1,445 | 0 | 0 | 2,943 | 4,388 |

#### test_holdout: 16,302 rows, by label3 {'-1': 3894, '0': 12408}, binary positives 3,894 (0.2389)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| TREC-06 | 12,408 | 0 | 0 | 3,894 | 16,302 |

#### test_holdout3: 3,220 rows, by label3 {'0': 1999, '1': 521, '2': 700}, binary positives 1,221 (0.3792)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 164 | 0 | 72 | 0 | 236 |
| CEAS-08 | 736 | 0 | 0 | 0 | 736 |
| Enron | 0 | 0 | 628 | 0 | 628 |
| Nazario | 0 | 222 | 0 | 0 | 222 |
| Nigerian_Fraud | 0 | 299 | 0 | 0 | 299 |
| TREC-07 | 1,099 | 0 | 0 | 0 | 1,099 |

### Leakage assertions (binary gate + three-way)
| check | count | asserted zero | pass |
|---|---|---|---|
| holdout_vs_train_key_rows | 0 | True | True |
| holdout_vs_val_key_rows | 0 | True | True |
| holdout_vs_pretrain_key_rows | 0 | True | True |
| holdout_vs_train_shingle_links | 0 | True | True |
| holdout_vs_val_shingle_links | 0 | True | True |
| holdout_vs_pretrain_shingle_links | 0 | True | True |
| train_vs_val_k_full_rows | 0 | True | True |
| train_vs_val_any_key_rows | 0 | False | True |
| train_vs_val_shingle_links | 0 | False | True |
| val_vs_pretrain_key_rows | 0 | True | True |
| val_vs_pretrain_shingle_links | 5,336 | False | True |
| train_vs_pretrain_key_overlap | 102,100 | False | True |
| holdout_rows_in_train_or_val_by_source | 0 | True | True |
| test_holdout_rows_from_other_sources | 0 | True | True |
| pretrain_rows | 226,273 | False | True |
| holdout3_vs_train_key_rows | 0 | True | True |
| holdout3_vs_val_key_rows | 0 | True | True |
| holdout3_vs_pretrain_key_rows | 0 | True | True |
| holdout3_vs_holdout_key_rows | 0 | True | True |
| holdout3_vs_train_shingle_links | 0 | True | True |
| holdout3_vs_val_shingle_links | 0 | True | True |
| holdout3_vs_pretrain_shingle_links | 0 | True | True |
| holdout3_vs_holdout_shingle_links | 0 | False | True |
| holdout3_rows_without_full_label3 | 0 | True | True |
| holdout3_rows_from_holdout_source | 0 | True | True |
| doc_id_shared_between_splits | 0 | True | True |
| label3_contract_violations | 0 | True | True |
| holdout3_rows | 3,220 | False | True |

### Files
- `finetune_train`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet`
- `finetune_val`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet`
- `test_holdout`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet`
- `test_holdout3`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout3.parquet`