File size: 13,624 Bytes
439e7b1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 | # Data report — 2026-10-08T06:41:51+00:00
spam_policy `positive_class` (positive = phishing/spam, unwanted); quote stripping on; seed 42; holdout source `TREC-06`; elapsed 139.0s; peak RSS 6.58 GB.
## Sources
- labelled primary: `puyang2025/seven-phishing-email-datasets` @ `b7f3f8d99ebd4fd0af0a5afe027420e53f08d1a4` (203,017 rows, hub splits {'train': 162413, 'test': 40604} concatenated; the hub split leaks and is ignored)
- pretraining corpus: `corbt/enron-emails` @ `cfc06c758093d90993abce1a43668fb7357258a6` (517,401 rows)
- excluded (no body text / 92% subset): `it4lia/PhishingEmailCuratedDatasets_Cleaned`, `zefang-liu/phishing-email-dataset`
## Splits
| split | rows | positives | negatives | base rate |
|---|---|---|---|---|
| finetune_train | 157,113 | 75,675 | 81,438 | 0.4817 |
| finetune_val | 17,643 | 8,642 | 9,001 | 0.4898 |
| test_holdout | 16,302 | 3,894 | 12,408 | 0.2389 |
| pretrain (MLM) | 226,273 | - | - | 407,902,364 chars |
### finetune_train by source
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 3,705 | 1,388 | 5,093 | 27.3% |
| CEAS-08 | 15,547 | 18,631 | 34,178 | 54.5% |
| Enron | 12,741 | 11,482 | 24,223 | 47.4% |
| TREC-05 | 27,663 | 18,723 | 46,386 | 40.4% |
| TREC-07 | 21,782 | 25,451 | 47,233 | 53.9% |
### finetune_val by source
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 380 | 136 | 516 | 26.4% |
| CEAS-08 | 1,699 | 2,184 | 3,883 | 56.2% |
| Enron | 1,381 | 1,327 | 2,708 | 49.0% |
| TREC-05 | 2,997 | 2,052 | 5,049 | 40.6% |
| TREC-07 | 2,544 | 2,943 | 5,487 | 53.6% |
### test_holdout by source
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| TREC-06 | 12,408 | 3,894 | 16,302 | 23.9% |
## Normalisation and dials
- common.normalize_text: NFKC -> lowercase -> keep [a-z0-9] only (import it, never re-derive it; NFKC is an extension of the verify_data.md section 7 wording)
- dials: {'key_prefix_chars': 300, 'key_prefix_min_len': 80, 'body_key_min_alnum': 20, 'pretrain_min_alnum': 50, 'quote_strip_min_alnum': 200, 'shingle_cap': 2000}
## Drops and removals
- labelled rows with <20 alphanumerics (subject+body): 715 rows
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| CEAS-08 | 0 | 1 | 1 | 100.0% |
| Enron | 12 | 18 | 30 | 60.0% |
| TREC-05 | 51 | 526 | 577 | 91.2% |
| TREC-06 | 3 | 95 | 98 | 96.9% |
| TREC-07 | 0 | 9 | 9 | 100.0% |
- Ling excluded (academic list corpus, 2,859 rows by label {'0': 2401, '1': 458})
- pool label conflicts (identical normalised body labelled both 0 and 1; all members dropped): 18 rows in 8 groups
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Enron | 8 | 0 | 8 | 0.0% |
| TREC-05 | 0 | 10 | 10 | 100.0% |
- pool exact duplicates (k_full, keep first): 6,715 rows
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 1 | 184 | 185 | 99.5% |
| CEAS-08 | 58 | 1,020 | 1,078 | 94.6% |
| Enron | 1,641 | 693 | 2,334 | 29.7% |
| TREC-05 | 1,611 | 772 | 2,383 | 32.4% |
| TREC-07 | 31 | 704 | 735 | 95.8% |
- decontamination against the holdout: {'key_match_only': 4, 'shingle_link_only': 1516, 'both': 132, 'total': 1652}
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 1 | 10 | 11 | 90.9% |
| CEAS-08 | 8 | 6 | 14 | 42.9% |
| Enron | 8 | 456 | 464 | 98.3% |
| TREC-05 | 7 | 863 | 870 | 99.2% |
| TREC-07 | 1 | 292 | 293 | 99.7% |
- holdout internal exact duplicates kept (test set is never deduped): 116
## Near-duplicate clustering
- 50-char shingles, stride 25, first 1500 chars, link at >= 3 shared; 1 boilerplate shingles over cap 2,000 ignored (touching 2,930 rows)
- links at the removal stage (pool + holdout index): {'pool_pool': 3633009, 'pool_holdout': 5150, 'holdout_holdout': 15456}
- clusters 114,261 (mixed source x label: 1,781; largest 7,069); val fraction target 0.1 achieved 0.101
| stratum | rows | clusters | val target | val rows | val frac | clusters too large |
|---|---|---|---|---|---|---|
| Assassin|0 | 3,811 | 2,733 | 381 | 381 | 0.1 | 1 |
| Assassin|1 | 1,436 | 834 | 144 | 144 | 0.1003 | 0 |
| CEAS-08|0 | 16,993 | 11,149 | 1,699 | 1,699 | 0.1 | 1 |
| CEAS-08|1 | 21,586 | 12,202 | 2,159 | 2,159 | 0.1 | 1 |
| Enron|0 | 13,589 | 12,731 | 1,359 | 1,359 | 0.1 | 0 |
| Enron|1 | 11,393 | 8,589 | 1,139 | 1,139 | 0.1 | 0 |
| TREC-05|0 | 30,181 | 24,292 | 3,018 | 3,018 | 0.1 | 0 |
| TREC-05|1 | 21,933 | 14,361 | 2,193 | 2,229 | 0.1016 | 0 |
| TREC-07|0 | 25,438 | 11,740 | 2,544 | 2,544 | 0.1 | 1 |
| TREC-07|1 | 28,396 | 15,630 | 2,840 | 2,971 | 0.1046 | 0 |
## Pretraining corpus (corbt/enron-emails)
| stage | rows |
|---|---|
| rows in | 517,401 |
| exact dups (subject+body, raw) | 272,905 |
| < 50 alnum after stripping | 14,027 |
| exact dups after stripping | 107 |
| matches a holdout key (removed) | 0 |
| matches a fine-tune val key (removed) | 4,004 |
| >= 3 shingles shared with a holdout row (removed) | 85 |
| >= 3 shingles shared with a fine-tune val row (kept, counted) | 1,095 |
| overlaps a fine-tune train key (kept, allowed) | 35,172 |
| rows out | 226,273 |
- near-dup screen: 88 links to 19 holdout rows (by label {'0': 18, '1': 1}); 5,345 links to 883 val rows
- quote/forward stripping on: 21,832 bodies cut, 70,907 had a marker but kept the full body (< 200 alnum above it); marker hits {'forwarded': 33089, 'header_block': 5428, 'original': 54222}; retained chars 0.8948 of all bodies, 0.2548 of cut bodies
- 407,902,364 chars out, mean 1802.7 chars/doc
## Leakage assertions
| check | count | asserted zero | pass |
|---|---|---|---|
| holdout_vs_train_key_rows | 0 | True | True |
| holdout_vs_val_key_rows | 0 | True | True |
| holdout_vs_pretrain_key_rows | 0 | True | True |
| holdout_vs_train_shingle_links | 0 | True | True |
| holdout_vs_val_shingle_links | 0 | True | True |
| holdout_vs_pretrain_shingle_links | 0 | True | True |
| train_vs_val_k_full_rows | 0 | True | True |
| train_vs_val_any_key_rows | 0 | False | True |
| train_vs_val_shingle_links | 0 | False | True |
| val_vs_pretrain_key_rows | 0 | True | True |
| val_vs_pretrain_shingle_links | 5,336 | False | True |
| train_vs_pretrain_key_overlap | 102,100 | False | True |
| holdout_rows_in_train_or_val_by_source | 0 | True | True |
| test_holdout_rows_from_other_sources | 0 | True | True |
| pretrain_rows | 226,273 | False | True |
## Files
- `finetune_train`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet`
- `finetune_val`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet`
- `test_holdout`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet`
- `pretrain`: `/home/nathank/BITS/dev/Phishing Model/data/processed/pretrain.parquet`
## Three-way labels — build3 2026-10-08T20:21:36+00:00
label3 {'0': 'legit', '1': 'phish', '2': 'spam', '-1': 'unwanted'}; display names {'0': 'Legitimate', '1': 'Phishing', '2': 'Spam'}; binary roll-up "Unwanted (phishing or spam)" = P(phishing)+P(spam). Positive rows by provenance: {'Assassin': 2, 'Enron': 2, 'Ling': 2, 'TREC-05': -1, 'TREC-06': -1, 'TREC-07': -1, 'CEAS-08': -1, 'Nazario': 1, 'Nigerian_Fraud': 1}. unchanged: 1 for phish / spam / unwanted, 0 for legit. Elapsed 66.1s; peak RSS 3.23 GB; binary base from snapshot (`/home/nathank/BITS/dev/Phishing Model/data/processed/binary_snapshot`).
### New sources
- `kudzaiprichard/aura-phishing-email-corpus` / `Nazario.csv` @ `539567cee21d03c83bd2fb1b142c18a80904eb72`: 1,565 rows (1,565 used), label3 1 (phishing by provenance)
- `kudzaiprichard/aura-phishing-email-corpus` / `Nigerian_Fraud.csv` @ `539567cee21d03c83bd2fb1b142c18a80904eb72`: 3,332 rows (3,332 used), label3 1 (phishing by provenance)
- bigshp (`locuoco/the-biggest-spam-ham-phish-email-dataset-300000`): research verdict **DO_NOT_USE**, not ingested (`docs/research/bigshp_research.md`)
| source | loaded | dropped | kept | -> train | -> val | -> test_holdout3 |
|---|---|---|---|---|---|---|
| Nazario | 1,565 | 58 | 1,507 | 1,159 | 126 | 222 |
| Nigerian_Fraud | 3,332 | 828 | 2,504 | 1,991 | 214 | 299 |
- drops by reason: {'Nazario': {'mbox_internal_data': 2, 'within_new_k_full_dup': 56}, 'Nigerian_Fraud': {'shingle_link_test_holdout': 235, 'within_new_k_full_dup': 98, 'key_match_finetune_train': 54, 'near_dup_of_both_train_and_val': 424, 'key_match_test_holdout': 13, 'lt_20_alnum': 1, 'key_match_finetune_val': 3}}
- near-dup pins: {'clusters_total': 3239, 'clusters_pinned_train_by_train_or_pretrain_link': 238, 'clusters_pinned_val_by_val_link': 23, 'clusters_dropped_linked_to_both_train_and_val': 17, 'rows_dropped_linked_to_both_train_and_val': 424, 'clusters_with_a_holdout_linked_member': 235, 'rows_linked_to_train': 329, 'rows_linked_to_pretrain': 33, 'rows_linked_to_val': 35, 'rows_dropped_holdout_shingle_link': 235, 'largest_cluster': 52}
- bigshp label contradictions among exact matches: 0
### test_holdout3 rows taken from finetune_val
- legitimate from ['CEAS-08', 'TREC-07', 'Assassin'], spam from ['Assassin', 'Enron']; whole near-dup clusters only; excluded candidates by reason: {'Enron': {'cluster_has_an_ineligible_member': 261, 'shingle_link_to_a_new_row': 28, 'shingle_link_pretrain': 4}, 'Assassin': {'shingle_link_pretrain': 8, 'cluster_has_an_ineligible_member': 7}, 'CEAS-08': {'shingle_link_pretrain': 1}, 'TREC-07': {'cluster_has_an_ineligible_member': 1}}
- per class: {'Legitimate': {'target_rows': 2000, 'eligible_rows': 4621, 'fraction': 0.4328, 'moved_rows': 1999}, 'Spam': {'target_rows': 700, 'eligible_rows': 1155, 'fraction': 0.6061, 'moved_rows': 700}}
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 164 | 0 | 72 | 0 | 236 |
| CEAS-08 | 736 | 0 | 0 | 0 | 736 |
| Enron | 0 | 0 | 628 | 0 | 628 |
| TREC-07 | 1,099 | 0 | 0 | 0 | 1,099 |
### Splits by label3
#### finetune_train: 160,263 rows, by label3 {'-1': 62805, '0': 81438, '1': 3150, '2': 12870}, binary positives 78,825 (0.4918)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 3,705 | 0 | 1,388 | 0 | 5,093 |
| CEAS-08 | 15,547 | 0 | 0 | 18,631 | 34,178 |
| Enron | 12,741 | 0 | 11,482 | 0 | 24,223 |
| Nazario | 0 | 1,159 | 0 | 0 | 1,159 |
| Nigerian_Fraud | 0 | 1,991 | 0 | 0 | 1,991 |
| TREC-05 | 27,663 | 0 | 0 | 18,723 | 46,386 |
| TREC-07 | 21,782 | 0 | 0 | 25,451 | 47,233 |
#### finetune_val: 15,284 rows, by label3 {'-1': 7179, '0': 7002, '1': 340, '2': 763}, binary positives 8,282 (0.5419)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 216 | 0 | 64 | 0 | 280 |
| CEAS-08 | 963 | 0 | 0 | 2,184 | 3,147 |
| Enron | 1,381 | 0 | 699 | 0 | 2,080 |
| Nazario | 0 | 126 | 0 | 0 | 126 |
| Nigerian_Fraud | 0 | 214 | 0 | 0 | 214 |
| TREC-05 | 2,997 | 0 | 0 | 2,052 | 5,049 |
| TREC-07 | 1,445 | 0 | 0 | 2,943 | 4,388 |
#### test_holdout: 16,302 rows, by label3 {'-1': 3894, '0': 12408}, binary positives 3,894 (0.2389)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| TREC-06 | 12,408 | 0 | 0 | 3,894 | 16,302 |
#### test_holdout3: 3,220 rows, by label3 {'0': 1999, '1': 521, '2': 700}, binary positives 1,221 (0.3792)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 164 | 0 | 72 | 0 | 236 |
| CEAS-08 | 736 | 0 | 0 | 0 | 736 |
| Enron | 0 | 0 | 628 | 0 | 628 |
| Nazario | 0 | 222 | 0 | 0 | 222 |
| Nigerian_Fraud | 0 | 299 | 0 | 0 | 299 |
| TREC-07 | 1,099 | 0 | 0 | 0 | 1,099 |
### Leakage assertions (binary gate + three-way)
| check | count | asserted zero | pass |
|---|---|---|---|
| holdout_vs_train_key_rows | 0 | True | True |
| holdout_vs_val_key_rows | 0 | True | True |
| holdout_vs_pretrain_key_rows | 0 | True | True |
| holdout_vs_train_shingle_links | 0 | True | True |
| holdout_vs_val_shingle_links | 0 | True | True |
| holdout_vs_pretrain_shingle_links | 0 | True | True |
| train_vs_val_k_full_rows | 0 | True | True |
| train_vs_val_any_key_rows | 0 | False | True |
| train_vs_val_shingle_links | 0 | False | True |
| val_vs_pretrain_key_rows | 0 | True | True |
| val_vs_pretrain_shingle_links | 5,336 | False | True |
| train_vs_pretrain_key_overlap | 102,100 | False | True |
| holdout_rows_in_train_or_val_by_source | 0 | True | True |
| test_holdout_rows_from_other_sources | 0 | True | True |
| pretrain_rows | 226,273 | False | True |
| holdout3_vs_train_key_rows | 0 | True | True |
| holdout3_vs_val_key_rows | 0 | True | True |
| holdout3_vs_pretrain_key_rows | 0 | True | True |
| holdout3_vs_holdout_key_rows | 0 | True | True |
| holdout3_vs_train_shingle_links | 0 | True | True |
| holdout3_vs_val_shingle_links | 0 | True | True |
| holdout3_vs_pretrain_shingle_links | 0 | True | True |
| holdout3_vs_holdout_shingle_links | 0 | False | True |
| holdout3_rows_without_full_label3 | 0 | True | True |
| holdout3_rows_from_holdout_source | 0 | True | True |
| doc_id_shared_between_splits | 0 | True | True |
| label3_contract_violations | 0 | True | True |
| holdout3_rows | 3,220 | False | True |
### Files
- `finetune_train`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet`
- `finetune_val`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet`
- `test_holdout`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet`
- `test_holdout3`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout3.parquet`
|