presentation/data/processed/data_report.md

Data report — 2026-10-08T06:41:51+00:00

spam_policy positive_class (positive = phishing/spam, unwanted); quote stripping on; seed 42; holdout source TREC-06; elapsed 139.0s; peak RSS 6.58 GB.

Sources

• labelled primary: puyang2025/seven-phishing-email-datasets @ b7f3f8d99ebd4fd0af0a5afe027420e53f08d1a4 (203,017 rows, hub splits {'train': 162413, 'test': 40604} concatenated; the hub split leaks and is ignored)
• pretraining corpus: corbt/enron-emails @ cfc06c758093d90993abce1a43668fb7357258a6 (517,401 rows)
• excluded (no body text / 92% subset): it4lia/PhishingEmailCuratedDatasets_Cleaned, zefang-liu/phishing-email-dataset

Splits

split	rows	positives	negatives	base rate
finetune_train	157,113	75,675	81,438	0.4817
finetune_val	17,643	8,642	9,001	0.4898
test_holdout	16,302	3,894	12,408	0.2389
pretrain (MLM)	226,273	-	-	407,902,364 chars

finetune_train by source

source	label 0	label 1	rows	positive rate
Assassin	3,705	1,388	5,093	27.3%
CEAS-08	15,547	18,631	34,178	54.5%
Enron	12,741	11,482	24,223	47.4%
TREC-05	27,663	18,723	46,386	40.4%
TREC-07	21,782	25,451	47,233	53.9%

finetune_val by source

source	label 0	label 1	rows	positive rate
Assassin	380	136	516	26.4%
CEAS-08	1,699	2,184	3,883	56.2%
Enron	1,381	1,327	2,708	49.0%
TREC-05	2,997	2,052	5,049	40.6%
TREC-07	2,544	2,943	5,487	53.6%

test_holdout by source

source	label 0	label 1	rows	positive rate
TREC-06	12,408	3,894	16,302	23.9%

Normalisation and dials

• common.normalize_text: NFKC -> lowercase -> keep [a-z0-9] only (import it, never re-derive it; NFKC is an extension of the verify_data.md section 7 wording)
• dials: {'key_prefix_chars': 300, 'key_prefix_min_len': 80, 'body_key_min_alnum': 20, 'pretrain_min_alnum': 50, 'quote_strip_min_alnum': 200, 'shingle_cap': 2000}

Drops and removals

• labelled rows with <20 alphanumerics (subject+body): 715 rows

  source	label 0	label 1	rows	positive rate
  CEAS-08	0	1	1	100.0%
  Enron	12	18	30	60.0%
  TREC-05	51	526	577	91.2%
  TREC-06	3	95	98	96.9%
  TREC-07	0	9	9	100.0%

• Ling excluded (academic list corpus, 2,859 rows by label {'0': 2401, '1': 458})
• pool label conflicts (identical normalised body labelled both 0 and 1; all members dropped): 18 rows in 8 groups

  source	label 0	label 1	rows	positive rate
  Enron	8	0	8	0.0%
  TREC-05	0	10	10	100.0%

• pool exact duplicates (k_full, keep first): 6,715 rows

  source	label 0	label 1	rows	positive rate
  Assassin	1	184	185	99.5%
  CEAS-08	58	1,020	1,078	94.6%
  Enron	1,641	693	2,334	29.7%
  TREC-05	1,611	772	2,383	32.4%
  TREC-07	31	704	735	95.8%

• decontamination against the holdout: {'key_match_only': 4, 'shingle_link_only': 1516, 'both': 132, 'total': 1652}

  source	label 0	label 1	rows	positive rate
  Assassin	1	10	11	90.9%
  CEAS-08	8	6	14	42.9%
  Enron	8	456	464	98.3%
  TREC-05	7	863	870	99.2%
  TREC-07	1	292	293	99.7%

• holdout internal exact duplicates kept (test set is never deduped): 116

Near-duplicate clustering

• 50-char shingles, stride 25, first 1500 chars, link at >= 3 shared; 1 boilerplate shingles over cap 2,000 ignored (touching 2,930 rows)
• links at the removal stage (pool + holdout index): {'pool_pool': 3633009, 'pool_holdout': 5150, 'holdout_holdout': 15456}
• clusters 114,261 (mixed source x label: 1,781; largest 7,069); val fraction target 0.1 achieved 0.101

  stratum	rows	clusters	val target	val rows	val frac	clusters too large
  Assassin	0	3,811	2,733	381	381	0.1
  Assassin	1	1,436	834	144	144	0.1003
  CEAS-08	0	16,993	11,149	1,699	1,699	0.1
  CEAS-08	1	21,586	12,202	2,159	2,159	0.1
  Enron	0	13,589	12,731	1,359	1,359	0.1
  Enron	1	11,393	8,589	1,139	1,139	0.1
  TREC-05	0	30,181	24,292	3,018	3,018	0.1
  TREC-05	1	21,933	14,361	2,193	2,229	0.1016
  TREC-07	0	25,438	11,740	2,544	2,544	0.1
  TREC-07	1	28,396	15,630	2,840	2,971	0.1046

Pretraining corpus (corbt/enron-emails)

stage	rows
rows in	517,401
exact dups (subject+body, raw)	272,905
< 50 alnum after stripping	14,027
exact dups after stripping	107
matches a holdout key (removed)	0
matches a fine-tune val key (removed)	4,004
>= 3 shingles shared with a holdout row (removed)	85
>= 3 shingles shared with a fine-tune val row (kept, counted)	1,095
overlaps a fine-tune train key (kept, allowed)	35,172
rows out	226,273

• near-dup screen: 88 links to 19 holdout rows (by label {'0': 18, '1': 1}); 5,345 links to 883 val rows
• quote/forward stripping on: 21,832 bodies cut, 70,907 had a marker but kept the full body (< 200 alnum above it); marker hits {'forwarded': 33089, 'header_block': 5428, 'original': 54222}; retained chars 0.8948 of all bodies, 0.2548 of cut bodies
• 407,902,364 chars out, mean 1802.7 chars/doc

Leakage assertions

check	count	asserted zero	pass
holdout_vs_train_key_rows	0	True	True
holdout_vs_val_key_rows	0	True	True
holdout_vs_pretrain_key_rows	0	True	True
holdout_vs_train_shingle_links	0	True	True
holdout_vs_val_shingle_links	0	True	True
holdout_vs_pretrain_shingle_links	0	True	True
train_vs_val_k_full_rows	0	True	True
train_vs_val_any_key_rows	0	False	True
train_vs_val_shingle_links	0	False	True
val_vs_pretrain_key_rows	0	True	True
val_vs_pretrain_shingle_links	5,336	False	True
train_vs_pretrain_key_overlap	102,100	False	True
holdout_rows_in_train_or_val_by_source	0	True	True
test_holdout_rows_from_other_sources	0	True	True
pretrain_rows	226,273	False	True

Files

• finetune_train: /home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet
• finetune_val: /home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet
• test_holdout: /home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet
• pretrain: /home/nathank/BITS/dev/Phishing Model/data/processed/pretrain.parquet

Three-way labels — build3 2026-10-08T20:21:36+00:00

label3 {'0': 'legit', '1': 'phish', '2': 'spam', '-1': 'unwanted'}; display names {'0': 'Legitimate', '1': 'Phishing', '2': 'Spam'}; binary roll-up "Unwanted (phishing or spam)" = P(phishing)+P(spam). Positive rows by provenance: {'Assassin': 2, 'Enron': 2, 'Ling': 2, 'TREC-05': -1, 'TREC-06': -1, 'TREC-07': -1, 'CEAS-08': -1, 'Nazario': 1, 'Nigerian_Fraud': 1}. unchanged: 1 for phish / spam / unwanted, 0 for legit. Elapsed 66.1s; peak RSS 3.23 GB; binary base from snapshot (/home/nathank/BITS/dev/Phishing Model/data/processed/binary_snapshot).

New sources

• kudzaiprichard/aura-phishing-email-corpus / Nazario.csv @ 539567cee21d03c83bd2fb1b142c18a80904eb72: 1,565 rows (1,565 used), label3 1 (phishing by provenance)
• kudzaiprichard/aura-phishing-email-corpus / Nigerian_Fraud.csv @ 539567cee21d03c83bd2fb1b142c18a80904eb72: 3,332 rows (3,332 used), label3 1 (phishing by provenance)
• bigshp (locuoco/the-biggest-spam-ham-phish-email-dataset-300000): research verdict DO_NOT_USE, not ingested (docs/research/bigshp_research.md)

source	loaded	dropped	kept	-> train	-> val	-> test_holdout3
Nazario	1,565	58	1,507	1,159	126	222
Nigerian_Fraud	3,332	828	2,504	1,991	214	299

• drops by reason: {'Nazario': {'mbox_internal_data': 2, 'within_new_k_full_dup': 56}, 'Nigerian_Fraud': {'shingle_link_test_holdout': 235, 'within_new_k_full_dup': 98, 'key_match_finetune_train': 54, 'near_dup_of_both_train_and_val': 424, 'key_match_test_holdout': 13, 'lt_20_alnum': 1, 'key_match_finetune_val': 3}}
• near-dup pins: {'clusters_total': 3239, 'clusters_pinned_train_by_train_or_pretrain_link': 238, 'clusters_pinned_val_by_val_link': 23, 'clusters_dropped_linked_to_both_train_and_val': 17, 'rows_dropped_linked_to_both_train_and_val': 424, 'clusters_with_a_holdout_linked_member': 235, 'rows_linked_to_train': 329, 'rows_linked_to_pretrain': 33, 'rows_linked_to_val': 35, 'rows_dropped_holdout_shingle_link': 235, 'largest_cluster': 52}
• bigshp label contradictions among exact matches: 0

test_holdout3 rows taken from finetune_val

• legitimate from ['CEAS-08', 'TREC-07', 'Assassin'], spam from ['Assassin', 'Enron']; whole near-dup clusters only; excluded candidates by reason: {'Enron': {'cluster_has_an_ineligible_member': 261, 'shingle_link_to_a_new_row': 28, 'shingle_link_pretrain': 4}, 'Assassin': {'shingle_link_pretrain': 8, 'cluster_has_an_ineligible_member': 7}, 'CEAS-08': {'shingle_link_pretrain': 1}, 'TREC-07': {'cluster_has_an_ineligible_member': 1}}
• per class: {'Legitimate': {'target_rows': 2000, 'eligible_rows': 4621, 'fraction': 0.4328, 'moved_rows': 1999}, 'Spam': {'target_rows': 700, 'eligible_rows': 1155, 'fraction': 0.6061, 'moved_rows': 700}}

  source	0 Legitimate	1 Phishing	2 Spam	-1 unwanted (unsplit)	rows
  Assassin	164	0	72	0	236
  CEAS-08	736	0	0	0	736
  Enron	0	0	628	0	628
  TREC-07	1,099	0	0	0	1,099

Splits by label3

finetune_train: 160,263 rows, by label3 {'-1': 62805, '0': 81438, '1': 3150, '2': 12870}, binary positives 78,825 (0.4918)

source	0 Legitimate	1 Phishing	2 Spam	-1 unwanted (unsplit)	rows
Assassin	3,705	0	1,388	0	5,093
CEAS-08	15,547	0	0	18,631	34,178
Enron	12,741	0	11,482	0	24,223
Nazario	0	1,159	0	0	1,159
Nigerian_Fraud	0	1,991	0	0	1,991
TREC-05	27,663	0	0	18,723	46,386
TREC-07	21,782	0	0	25,451	47,233

finetune_val: 15,284 rows, by label3 {'-1': 7179, '0': 7002, '1': 340, '2': 763}, binary positives 8,282 (0.5419)

source	0 Legitimate	1 Phishing	2 Spam	-1 unwanted (unsplit)	rows
Assassin	216	0	64	0	280
CEAS-08	963	0	0	2,184	3,147
Enron	1,381	0	699	0	2,080
Nazario	0	126	0	0	126
Nigerian_Fraud	0	214	0	0	214
TREC-05	2,997	0	0	2,052	5,049
TREC-07	1,445	0	0	2,943	4,388

test_holdout: 16,302 rows, by label3 {'-1': 3894, '0': 12408}, binary positives 3,894 (0.2389)

source	0 Legitimate	1 Phishing	2 Spam	-1 unwanted (unsplit)	rows
TREC-06	12,408	0	0	3,894	16,302

test_holdout3: 3,220 rows, by label3 {'0': 1999, '1': 521, '2': 700}, binary positives 1,221 (0.3792)

source	0 Legitimate	1 Phishing	2 Spam	-1 unwanted (unsplit)	rows
Assassin	164	0	72	0	236
CEAS-08	736	0	0	0	736
Enron	0	0	628	0	628
Nazario	0	222	0	0	222
Nigerian_Fraud	0	299	0	0	299
TREC-07	1,099	0	0	0	1,099

Leakage assertions (binary gate + three-way)

check	count	asserted zero	pass
holdout_vs_train_key_rows	0	True	True
holdout_vs_val_key_rows	0	True	True
holdout_vs_pretrain_key_rows	0	True	True
holdout_vs_train_shingle_links	0	True	True
holdout_vs_val_shingle_links	0	True	True
holdout_vs_pretrain_shingle_links	0	True	True
train_vs_val_k_full_rows	0	True	True
train_vs_val_any_key_rows	0	False	True
train_vs_val_shingle_links	0	False	True
val_vs_pretrain_key_rows	0	True	True
val_vs_pretrain_shingle_links	5,336	False	True
train_vs_pretrain_key_overlap	102,100	False	True
holdout_rows_in_train_or_val_by_source	0	True	True
test_holdout_rows_from_other_sources	0	True	True
pretrain_rows	226,273	False	True
holdout3_vs_train_key_rows	0	True	True
holdout3_vs_val_key_rows	0	True	True
holdout3_vs_pretrain_key_rows	0	True	True
holdout3_vs_holdout_key_rows	0	True	True
holdout3_vs_train_shingle_links	0	True	True
holdout3_vs_val_shingle_links	0	True	True
holdout3_vs_pretrain_shingle_links	0	True	True
holdout3_vs_holdout_shingle_links	0	False	True
holdout3_rows_without_full_label3	0	True	True
holdout3_rows_from_holdout_source	0	True	True
doc_id_shared_between_splits	0	True	True
label3_contract_violations	0	True	True
holdout3_rows	3,220	False	True

Files

• finetune_train: /home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet
• finetune_val: /home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet
• test_holdout: /home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet
• test_holdout3: /home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout3.parquet