Add presentation bundle (reports, curves, run log, code) for the Colab notebook

439e7b1 verified 15 days ago

13.6 kB

# Data report — 2026-10-08T06:41:51+00:00

spam_policy `positive_class` (positive = phishing/spam, unwanted); quote stripping on; seed 42; holdout source `TREC-06`; elapsed 139.0s; peak RSS 6.58 GB.

## Sources
- labelled primary: `puyang2025/seven-phishing-email-datasets` @ `b7f3f8d99ebd4fd0af0a5afe027420e53f08d1a4` (203,017 rows, hub splits {'train': 162413, 'test': 40604} concatenated; the hub split leaks and is ignored)
- pretraining corpus: `corbt/enron-emails` @ `cfc06c758093d90993abce1a43668fb7357258a6` (517,401 rows)
- excluded (no body text / 92% subset): `it4lia/PhishingEmailCuratedDatasets_Cleaned`, `zefang-liu/phishing-email-dataset`

## Splits
| split | rows | positives | negatives | base rate |
|---|---|---|---|---|
| finetune_train | 157,113 | 75,675 | 81,438 | 0.4817 |
| finetune_val | 17,643 | 8,642 | 9,001 | 0.4898 |
| test_holdout | 16,302 | 3,894 | 12,408 | 0.2389 |
| pretrain (MLM) | 226,273 | - | - | 407,902,364 chars |
### finetune_train by source
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 3,705 | 1,388 | 5,093 | 27.3% |
| CEAS-08 | 15,547 | 18,631 | 34,178 | 54.5% |
| Enron | 12,741 | 11,482 | 24,223 | 47.4% |
| TREC-05 | 27,663 | 18,723 | 46,386 | 40.4% |
| TREC-07 | 21,782 | 25,451 | 47,233 | 53.9% |

### finetune_val by source
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 380 | 136 | 516 | 26.4% |
| CEAS-08 | 1,699 | 2,184 | 3,883 | 56.2% |
| Enron | 1,381 | 1,327 | 2,708 | 49.0% |
| TREC-05 | 2,997 | 2,052 | 5,049 | 40.6% |
| TREC-07 | 2,544 | 2,943 | 5,487 | 53.6% |
### test_holdout by source
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| TREC-06 | 12,408 | 3,894 | 16,302 | 23.9% |

## Normalisation and dials
- common.normalize_text: NFKC -> lowercase -> keep [a-z0-9] only (import it, never re-derive it; NFKC is an extension of the verify_data.md section 7 wording)
- dials: {'key_prefix_chars': 300, 'key_prefix_min_len': 80, 'body_key_min_alnum': 20, 'pretrain_min_alnum': 50, 'quote_strip_min_alnum': 200, 'shingle_cap': 2000}

## Drops and removals
- labelled rows with <20 alphanumerics (subject+body): 715 rows
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| CEAS-08 | 0 | 1 | 1 | 100.0% |
| Enron | 12 | 18 | 30 | 60.0% |
| TREC-05 | 51 | 526 | 577 | 91.2% |
| TREC-06 | 3 | 95 | 98 | 96.9% |
| TREC-07 | 0 | 9 | 9 | 100.0% |
- Ling excluded (academic list corpus, 2,859 rows by label {'0': 2401, '1': 458})
- pool label conflicts (identical normalised body labelled both 0 and 1; all members dropped): 18 rows in 8 groups
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Enron | 8 | 0 | 8 | 0.0% |
| TREC-05 | 0 | 10 | 10 | 100.0% |
- pool exact duplicates (k_full, keep first): 6,715 rows
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 1 | 184 | 185 | 99.5% |
| CEAS-08 | 58 | 1,020 | 1,078 | 94.6% |
| Enron | 1,641 | 693 | 2,334 | 29.7% |
| TREC-05 | 1,611 | 772 | 2,383 | 32.4% |
| TREC-07 | 31 | 704 | 735 | 95.8% |
- decontamination against the holdout: {'key_match_only': 4, 'shingle_link_only': 1516, 'both': 132, 'total': 1652}
| source | label 0 | label 1 | rows | positive rate |
|---|---|---|---|---|
| Assassin | 1 | 10 | 11 | 90.9% |
| CEAS-08 | 8 | 6 | 14 | 42.9% |
| Enron | 8 | 456 | 464 | 98.3% |
| TREC-05 | 7 | 863 | 870 | 99.2% |
| TREC-07 | 1 | 292 | 293 | 99.7% |
- holdout internal exact duplicates kept (test set is never deduped): 116
## Near-duplicate clustering
- 50-char shingles, stride 25, first 1500 chars, link at >= 3 shared; 1 boilerplate shingles over cap 2,000 ignored (touching 2,930 rows)
- links at the removal stage (pool + holdout index): {'pool_pool': 3633009, 'pool_holdout': 5150, 'holdout_holdout': 15456}
- clusters 114,261 (mixed source x label: 1,781; largest 7,069); val fraction target 0.1 achieved 0.101
| stratum | rows | clusters | val target | val rows | val frac | clusters too large |
|---|---|---|---|---|---|---|
| Assassin|0 | 3,811 | 2,733 | 381 | 381 | 0.1 | 1 |
| Assassin|1 | 1,436 | 834 | 144 | 144 | 0.1003 | 0 |
| CEAS-08|0 | 16,993 | 11,149 | 1,699 | 1,699 | 0.1 | 1 |
| CEAS-08|1 | 21,586 | 12,202 | 2,159 | 2,159 | 0.1 | 1 |
| Enron|0 | 13,589 | 12,731 | 1,359 | 1,359 | 0.1 | 0 |
| Enron|1 | 11,393 | 8,589 | 1,139 | 1,139 | 0.1 | 0 |
| TREC-05|0 | 30,181 | 24,292 | 3,018 | 3,018 | 0.1 | 0 |
| TREC-05|1 | 21,933 | 14,361 | 2,193 | 2,229 | 0.1016 | 0 |
| TREC-07|0 | 25,438 | 11,740 | 2,544 | 2,544 | 0.1 | 1 |
| TREC-07|1 | 28,396 | 15,630 | 2,840 | 2,971 | 0.1046 | 0 |

## Pretraining corpus (corbt/enron-emails)
| stage | rows |
|---|---|
| rows in | 517,401 |
| exact dups (subject+body, raw) | 272,905 |
| < 50 alnum after stripping | 14,027 |
| exact dups after stripping | 107 |
| matches a holdout key (removed) | 0 |
| matches a fine-tune val key (removed) | 4,004 |
| >= 3 shingles shared with a holdout row (removed) | 85 |
| >= 3 shingles shared with a fine-tune val row (kept, counted) | 1,095 |
| overlaps a fine-tune train key (kept, allowed) | 35,172 |
| rows out | 226,273 |
- near-dup screen: 88 links to 19 holdout rows (by label {'0': 18, '1': 1}); 5,345 links to 883 val rows
- quote/forward stripping on: 21,832 bodies cut, 70,907 had a marker but kept the full body (< 200 alnum above it); marker hits {'forwarded': 33089, 'header_block': 5428, 'original': 54222}; retained chars 0.8948 of all bodies, 0.2548 of cut bodies
- 407,902,364 chars out, mean 1802.7 chars/doc
## Leakage assertions
| check | count | asserted zero | pass |
|---|---|---|---|
| holdout_vs_train_key_rows | 0 | True | True |
| holdout_vs_val_key_rows | 0 | True | True |
| holdout_vs_pretrain_key_rows | 0 | True | True |
| holdout_vs_train_shingle_links | 0 | True | True |
| holdout_vs_val_shingle_links | 0 | True | True |
| holdout_vs_pretrain_shingle_links | 0 | True | True |
| train_vs_val_k_full_rows | 0 | True | True |
| train_vs_val_any_key_rows | 0 | False | True |
| train_vs_val_shingle_links | 0 | False | True |
| val_vs_pretrain_key_rows | 0 | True | True |
| val_vs_pretrain_shingle_links | 5,336 | False | True |
| train_vs_pretrain_key_overlap | 102,100 | False | True |
| holdout_rows_in_train_or_val_by_source | 0 | True | True |
| test_holdout_rows_from_other_sources | 0 | True | True |
| pretrain_rows | 226,273 | False | True |

## Files
- `finetune_train`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet`
- `finetune_val`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet`
- `test_holdout`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet`
- `pretrain`: `/home/nathank/BITS/dev/Phishing Model/data/processed/pretrain.parquet`

## Three-way labels — build3 2026-10-08T20:21:36+00:00

label3 {'0': 'legit', '1': 'phish', '2': 'spam', '-1': 'unwanted'}; display names {'0': 'Legitimate', '1': 'Phishing', '2': 'Spam'}; binary roll-up "Unwanted (phishing or spam)" = P(phishing)+P(spam). Positive rows by provenance: {'Assassin': 2, 'Enron': 2, 'Ling': 2, 'TREC-05': -1, 'TREC-06': -1, 'TREC-07': -1, 'CEAS-08': -1, 'Nazario': 1, 'Nigerian_Fraud': 1}. unchanged: 1 for phish / spam / unwanted, 0 for legit. Elapsed 66.1s; peak RSS 3.23 GB; binary base from snapshot (`/home/nathank/BITS/dev/Phishing Model/data/processed/binary_snapshot`).

### New sources
- `kudzaiprichard/aura-phishing-email-corpus` / `Nazario.csv` @ `539567cee21d03c83bd2fb1b142c18a80904eb72`: 1,565 rows (1,565 used), label3 1 (phishing by provenance)
- `kudzaiprichard/aura-phishing-email-corpus` / `Nigerian_Fraud.csv` @ `539567cee21d03c83bd2fb1b142c18a80904eb72`: 3,332 rows (3,332 used), label3 1 (phishing by provenance)
- bigshp (`locuoco/the-biggest-spam-ham-phish-email-dataset-300000`): research verdict **DO_NOT_USE**, not ingested (`docs/research/bigshp_research.md`)

| source | loaded | dropped | kept | -> train | -> val | -> test_holdout3 |
|---|---|---|---|---|---|---|
| Nazario | 1,565 | 58 | 1,507 | 1,159 | 126 | 222 |
| Nigerian_Fraud | 3,332 | 828 | 2,504 | 1,991 | 214 | 299 |

- drops by reason: {'Nazario': {'mbox_internal_data': 2, 'within_new_k_full_dup': 56}, 'Nigerian_Fraud': {'shingle_link_test_holdout': 235, 'within_new_k_full_dup': 98, 'key_match_finetune_train': 54, 'near_dup_of_both_train_and_val': 424, 'key_match_test_holdout': 13, 'lt_20_alnum': 1, 'key_match_finetune_val': 3}}
- near-dup pins: {'clusters_total': 3239, 'clusters_pinned_train_by_train_or_pretrain_link': 238, 'clusters_pinned_val_by_val_link': 23, 'clusters_dropped_linked_to_both_train_and_val': 17, 'rows_dropped_linked_to_both_train_and_val': 424, 'clusters_with_a_holdout_linked_member': 235, 'rows_linked_to_train': 329, 'rows_linked_to_pretrain': 33, 'rows_linked_to_val': 35, 'rows_dropped_holdout_shingle_link': 235, 'largest_cluster': 52}
- bigshp label contradictions among exact matches: 0
### test_holdout3 rows taken from finetune_val
- legitimate from ['CEAS-08', 'TREC-07', 'Assassin'], spam from ['Assassin', 'Enron']; whole near-dup clusters only; excluded candidates by reason: {'Enron': {'cluster_has_an_ineligible_member': 261, 'shingle_link_to_a_new_row': 28, 'shingle_link_pretrain': 4}, 'Assassin': {'shingle_link_pretrain': 8, 'cluster_has_an_ineligible_member': 7}, 'CEAS-08': {'shingle_link_pretrain': 1}, 'TREC-07': {'cluster_has_an_ineligible_member': 1}}
- per class: {'Legitimate': {'target_rows': 2000, 'eligible_rows': 4621, 'fraction': 0.4328, 'moved_rows': 1999}, 'Spam': {'target_rows': 700, 'eligible_rows': 1155, 'fraction': 0.6061, 'moved_rows': 700}}
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 164 | 0 | 72 | 0 | 236 |
| CEAS-08 | 736 | 0 | 0 | 0 | 736 |
| Enron | 0 | 0 | 628 | 0 | 628 |
| TREC-07 | 1,099 | 0 | 0 | 0 | 1,099 |

### Splits by label3
#### finetune_train: 160,263 rows, by label3 {'-1': 62805, '0': 81438, '1': 3150, '2': 12870}, binary positives 78,825 (0.4918)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 3,705 | 0 | 1,388 | 0 | 5,093 |
| CEAS-08 | 15,547 | 0 | 0 | 18,631 | 34,178 |
| Enron | 12,741 | 0 | 11,482 | 0 | 24,223 |
| Nazario | 0 | 1,159 | 0 | 0 | 1,159 |
| Nigerian_Fraud | 0 | 1,991 | 0 | 0 | 1,991 |
| TREC-05 | 27,663 | 0 | 0 | 18,723 | 46,386 |
| TREC-07 | 21,782 | 0 | 0 | 25,451 | 47,233 |

#### finetune_val: 15,284 rows, by label3 {'-1': 7179, '0': 7002, '1': 340, '2': 763}, binary positives 8,282 (0.5419)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 216 | 0 | 64 | 0 | 280 |
| CEAS-08 | 963 | 0 | 0 | 2,184 | 3,147 |
| Enron | 1,381 | 0 | 699 | 0 | 2,080 |
| Nazario | 0 | 126 | 0 | 0 | 126 |
| Nigerian_Fraud | 0 | 214 | 0 | 0 | 214 |
| TREC-05 | 2,997 | 0 | 0 | 2,052 | 5,049 |
| TREC-07 | 1,445 | 0 | 0 | 2,943 | 4,388 |

#### test_holdout: 16,302 rows, by label3 {'-1': 3894, '0': 12408}, binary positives 3,894 (0.2389)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| TREC-06 | 12,408 | 0 | 0 | 3,894 | 16,302 |
#### test_holdout3: 3,220 rows, by label3 {'0': 1999, '1': 521, '2': 700}, binary positives 1,221 (0.3792)
| source | 0 Legitimate | 1 Phishing | 2 Spam | -1 unwanted (unsplit) | rows |
|---|---|---|---|---|---|
| Assassin | 164 | 0 | 72 | 0 | 236 |
| CEAS-08 | 736 | 0 | 0 | 0 | 736 |
| Enron | 0 | 0 | 628 | 0 | 628 |
| Nazario | 0 | 222 | 0 | 0 | 222 |
| Nigerian_Fraud | 0 | 299 | 0 | 0 | 299 |
| TREC-07 | 1,099 | 0 | 0 | 0 | 1,099 |
### Leakage assertions (binary gate + three-way)
| check | count | asserted zero | pass |
|---|---|---|---|
| holdout_vs_train_key_rows | 0 | True | True |
| holdout_vs_val_key_rows | 0 | True | True |
| holdout_vs_pretrain_key_rows | 0 | True | True |
| holdout_vs_train_shingle_links | 0 | True | True |
| holdout_vs_val_shingle_links | 0 | True | True |
| holdout_vs_pretrain_shingle_links | 0 | True | True |
| train_vs_val_k_full_rows | 0 | True | True |
| train_vs_val_any_key_rows | 0 | False | True |
| train_vs_val_shingle_links | 0 | False | True |
| val_vs_pretrain_key_rows | 0 | True | True |
| val_vs_pretrain_shingle_links | 5,336 | False | True |
| train_vs_pretrain_key_overlap | 102,100 | False | True |
| holdout_rows_in_train_or_val_by_source | 0 | True | True |
| test_holdout_rows_from_other_sources | 0 | True | True |
| pretrain_rows | 226,273 | False | True |
| holdout3_vs_train_key_rows | 0 | True | True |
| holdout3_vs_val_key_rows | 0 | True | True |
| holdout3_vs_pretrain_key_rows | 0 | True | True |
| holdout3_vs_holdout_key_rows | 0 | True | True |
| holdout3_vs_train_shingle_links | 0 | True | True |
| holdout3_vs_val_shingle_links | 0 | True | True |
| holdout3_vs_pretrain_shingle_links | 0 | True | True |
| holdout3_vs_holdout_shingle_links | 0 | False | True |
| holdout3_rows_without_full_label3 | 0 | True | True |
| holdout3_rows_from_holdout_source | 0 | True | True |
| doc_id_shared_between_splits | 0 | True | True |
| label3_contract_violations | 0 | True | True |
| holdout3_rows | 3,220 | False | True |
### Files
- `finetune_train`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet`
- `finetune_val`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet`
- `test_holdout`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet`
- `test_holdout3`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout3.parquet`

	# Data report — 2026-10-08T06:41:51+00:00

	spam_policy `positive_class` (positive = phishing/spam, unwanted); quote stripping on; seed 42; holdout source `TREC-06`; elapsed 139.0s; peak RSS 6.58 GB.

	## Sources
	- labelled primary: `puyang2025/seven-phishing-email-datasets` @ `b7f3f8d99ebd4fd0af0a5afe027420e53f08d1a4` (203,017 rows, hub splits {'train': 162413, 'test': 40604} concatenated; the hub split leaks and is ignored)
	- pretraining corpus: `corbt/enron-emails` @ `cfc06c758093d90993abce1a43668fb7357258a6` (517,401 rows)
	- excluded (no body text / 92% subset): `it4lia/PhishingEmailCuratedDatasets_Cleaned`, `zefang-liu/phishing-email-dataset`

	## Splits
	\| split \| rows \| positives \| negatives \| base rate \|
	\|---\|---\|---\|---\|---\|
	\| finetune_train \| 157,113 \| 75,675 \| 81,438 \| 0.4817 \|
	\| finetune_val \| 17,643 \| 8,642 \| 9,001 \| 0.4898 \|
	\| test_holdout \| 16,302 \| 3,894 \| 12,408 \| 0.2389 \|
	\| pretrain (MLM) \| 226,273 \| - \| - \| 407,902,364 chars \|

	### finetune_train by source
	\| source \| label 0 \| label 1 \| rows \| positive rate \|
	\|---\|---\|---\|---\|---\|
	\| Assassin \| 3,705 \| 1,388 \| 5,093 \| 27.3% \|
	\| CEAS-08 \| 15,547 \| 18,631 \| 34,178 \| 54.5% \|
	\| Enron \| 12,741 \| 11,482 \| 24,223 \| 47.4% \|
	\| TREC-05 \| 27,663 \| 18,723 \| 46,386 \| 40.4% \|
	\| TREC-07 \| 21,782 \| 25,451 \| 47,233 \| 53.9% \|

	### finetune_val by source
	\| source \| label 0 \| label 1 \| rows \| positive rate \|
	\|---\|---\|---\|---\|---\|
	\| Assassin \| 380 \| 136 \| 516 \| 26.4% \|
	\| CEAS-08 \| 1,699 \| 2,184 \| 3,883 \| 56.2% \|
	\| Enron \| 1,381 \| 1,327 \| 2,708 \| 49.0% \|
	\| TREC-05 \| 2,997 \| 2,052 \| 5,049 \| 40.6% \|
	\| TREC-07 \| 2,544 \| 2,943 \| 5,487 \| 53.6% \|

	### test_holdout by source
	\| source \| label 0 \| label 1 \| rows \| positive rate \|
	\|---\|---\|---\|---\|---\|
	\| TREC-06 \| 12,408 \| 3,894 \| 16,302 \| 23.9% \|

	## Normalisation and dials
	- common.normalize_text: NFKC -> lowercase -> keep [a-z0-9] only (import it, never re-derive it; NFKC is an extension of the verify_data.md section 7 wording)
	- dials: {'key_prefix_chars': 300, 'key_prefix_min_len': 80, 'body_key_min_alnum': 20, 'pretrain_min_alnum': 50, 'quote_strip_min_alnum': 200, 'shingle_cap': 2000}

	## Drops and removals
	- labelled rows with <20 alphanumerics (subject+body): 715 rows
	\| source \| label 0 \| label 1 \| rows \| positive rate \|
	\|---\|---\|---\|---\|---\|
	\| CEAS-08 \| 0 \| 1 \| 1 \| 100.0% \|
	\| Enron \| 12 \| 18 \| 30 \| 60.0% \|
	\| TREC-05 \| 51 \| 526 \| 577 \| 91.2% \|
	\| TREC-06 \| 3 \| 95 \| 98 \| 96.9% \|
	\| TREC-07 \| 0 \| 9 \| 9 \| 100.0% \|
	- Ling excluded (academic list corpus, 2,859 rows by label {'0': 2401, '1': 458})
	- pool label conflicts (identical normalised body labelled both 0 and 1; all members dropped): 18 rows in 8 groups
	\| source \| label 0 \| label 1 \| rows \| positive rate \|
	\|---\|---\|---\|---\|---\|
	\| Enron \| 8 \| 0 \| 8 \| 0.0% \|
	\| TREC-05 \| 0 \| 10 \| 10 \| 100.0% \|
	- pool exact duplicates (k_full, keep first): 6,715 rows
	\| source \| label 0 \| label 1 \| rows \| positive rate \|
	\|---\|---\|---\|---\|---\|
	\| Assassin \| 1 \| 184 \| 185 \| 99.5% \|
	\| CEAS-08 \| 58 \| 1,020 \| 1,078 \| 94.6% \|
	\| Enron \| 1,641 \| 693 \| 2,334 \| 29.7% \|
	\| TREC-05 \| 1,611 \| 772 \| 2,383 \| 32.4% \|
	\| TREC-07 \| 31 \| 704 \| 735 \| 95.8% \|
	- decontamination against the holdout: {'key_match_only': 4, 'shingle_link_only': 1516, 'both': 132, 'total': 1652}
	\| source \| label 0 \| label 1 \| rows \| positive rate \|
	\|---\|---\|---\|---\|---\|
	\| Assassin \| 1 \| 10 \| 11 \| 90.9% \|
	\| CEAS-08 \| 8 \| 6 \| 14 \| 42.9% \|
	\| Enron \| 8 \| 456 \| 464 \| 98.3% \|
	\| TREC-05 \| 7 \| 863 \| 870 \| 99.2% \|
	\| TREC-07 \| 1 \| 292 \| 293 \| 99.7% \|
	- holdout internal exact duplicates kept (test set is never deduped): 116

	## Near-duplicate clustering
	- 50-char shingles, stride 25, first 1500 chars, link at >= 3 shared; 1 boilerplate shingles over cap 2,000 ignored (touching 2,930 rows)
	- links at the removal stage (pool + holdout index): {'pool_pool': 3633009, 'pool_holdout': 5150, 'holdout_holdout': 15456}
	- clusters 114,261 (mixed source x label: 1,781; largest 7,069); val fraction target 0.1 achieved 0.101
	\| stratum \| rows \| clusters \| val target \| val rows \| val frac \| clusters too large \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| Assassin\|0 \| 3,811 \| 2,733 \| 381 \| 381 \| 0.1 \| 1 \|
	\| Assassin\|1 \| 1,436 \| 834 \| 144 \| 144 \| 0.1003 \| 0 \|
	\| CEAS-08\|0 \| 16,993 \| 11,149 \| 1,699 \| 1,699 \| 0.1 \| 1 \|
	\| CEAS-08\|1 \| 21,586 \| 12,202 \| 2,159 \| 2,159 \| 0.1 \| 1 \|
	\| Enron\|0 \| 13,589 \| 12,731 \| 1,359 \| 1,359 \| 0.1 \| 0 \|
	\| Enron\|1 \| 11,393 \| 8,589 \| 1,139 \| 1,139 \| 0.1 \| 0 \|
	\| TREC-05\|0 \| 30,181 \| 24,292 \| 3,018 \| 3,018 \| 0.1 \| 0 \|
	\| TREC-05\|1 \| 21,933 \| 14,361 \| 2,193 \| 2,229 \| 0.1016 \| 0 \|
	\| TREC-07\|0 \| 25,438 \| 11,740 \| 2,544 \| 2,544 \| 0.1 \| 1 \|
	\| TREC-07\|1 \| 28,396 \| 15,630 \| 2,840 \| 2,971 \| 0.1046 \| 0 \|

	## Pretraining corpus (corbt/enron-emails)
	\| stage \| rows \|
	\|---\|---\|
	\| rows in \| 517,401 \|
	\| exact dups (subject+body, raw) \| 272,905 \|
	\| < 50 alnum after stripping \| 14,027 \|
	\| exact dups after stripping \| 107 \|
	\| matches a holdout key (removed) \| 0 \|
	\| matches a fine-tune val key (removed) \| 4,004 \|
	\| >= 3 shingles shared with a holdout row (removed) \| 85 \|
	\| >= 3 shingles shared with a fine-tune val row (kept, counted) \| 1,095 \|
	\| overlaps a fine-tune train key (kept, allowed) \| 35,172 \|
	\| rows out \| 226,273 \|
	- near-dup screen: 88 links to 19 holdout rows (by label {'0': 18, '1': 1}); 5,345 links to 883 val rows
	- quote/forward stripping on: 21,832 bodies cut, 70,907 had a marker but kept the full body (< 200 alnum above it); marker hits {'forwarded': 33089, 'header_block': 5428, 'original': 54222}; retained chars 0.8948 of all bodies, 0.2548 of cut bodies
	- 407,902,364 chars out, mean 1802.7 chars/doc

	## Leakage assertions
	\| check \| count \| asserted zero \| pass \|
	\|---\|---\|---\|---\|
	\| holdout_vs_train_key_rows \| 0 \| True \| True \|
	\| holdout_vs_val_key_rows \| 0 \| True \| True \|
	\| holdout_vs_pretrain_key_rows \| 0 \| True \| True \|
	\| holdout_vs_train_shingle_links \| 0 \| True \| True \|
	\| holdout_vs_val_shingle_links \| 0 \| True \| True \|
	\| holdout_vs_pretrain_shingle_links \| 0 \| True \| True \|
	\| train_vs_val_k_full_rows \| 0 \| True \| True \|
	\| train_vs_val_any_key_rows \| 0 \| False \| True \|
	\| train_vs_val_shingle_links \| 0 \| False \| True \|
	\| val_vs_pretrain_key_rows \| 0 \| True \| True \|
	\| val_vs_pretrain_shingle_links \| 5,336 \| False \| True \|
	\| train_vs_pretrain_key_overlap \| 102,100 \| False \| True \|
	\| holdout_rows_in_train_or_val_by_source \| 0 \| True \| True \|
	\| test_holdout_rows_from_other_sources \| 0 \| True \| True \|
	\| pretrain_rows \| 226,273 \| False \| True \|

	## Files
	- `finetune_train`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet`
	- `finetune_val`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet`
	- `test_holdout`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet`
	- `pretrain`: `/home/nathank/BITS/dev/Phishing Model/data/processed/pretrain.parquet`

	## Three-way labels — build3 2026-10-08T20:21:36+00:00

	label3 {'0': 'legit', '1': 'phish', '2': 'spam', '-1': 'unwanted'}; display names {'0': 'Legitimate', '1': 'Phishing', '2': 'Spam'}; binary roll-up "Unwanted (phishing or spam)" = P(phishing)+P(spam). Positive rows by provenance: {'Assassin': 2, 'Enron': 2, 'Ling': 2, 'TREC-05': -1, 'TREC-06': -1, 'TREC-07': -1, 'CEAS-08': -1, 'Nazario': 1, 'Nigerian_Fraud': 1}. unchanged: 1 for phish / spam / unwanted, 0 for legit. Elapsed 66.1s; peak RSS 3.23 GB; binary base from snapshot (`/home/nathank/BITS/dev/Phishing Model/data/processed/binary_snapshot`).

	### New sources
	- `kudzaiprichard/aura-phishing-email-corpus` / `Nazario.csv` @ `539567cee21d03c83bd2fb1b142c18a80904eb72`: 1,565 rows (1,565 used), label3 1 (phishing by provenance)
	- `kudzaiprichard/aura-phishing-email-corpus` / `Nigerian_Fraud.csv` @ `539567cee21d03c83bd2fb1b142c18a80904eb72`: 3,332 rows (3,332 used), label3 1 (phishing by provenance)
	- bigshp (`locuoco/the-biggest-spam-ham-phish-email-dataset-300000`): research verdict DO_NOT_USE, not ingested (`docs/research/bigshp_research.md`)

	\| source \| loaded \| dropped \| kept \| -> train \| -> val \| -> test_holdout3 \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| Nazario \| 1,565 \| 58 \| 1,507 \| 1,159 \| 126 \| 222 \|
	\| Nigerian_Fraud \| 3,332 \| 828 \| 2,504 \| 1,991 \| 214 \| 299 \|

	- drops by reason: {'Nazario': {'mbox_internal_data': 2, 'within_new_k_full_dup': 56}, 'Nigerian_Fraud': {'shingle_link_test_holdout': 235, 'within_new_k_full_dup': 98, 'key_match_finetune_train': 54, 'near_dup_of_both_train_and_val': 424, 'key_match_test_holdout': 13, 'lt_20_alnum': 1, 'key_match_finetune_val': 3}}
	- near-dup pins: {'clusters_total': 3239, 'clusters_pinned_train_by_train_or_pretrain_link': 238, 'clusters_pinned_val_by_val_link': 23, 'clusters_dropped_linked_to_both_train_and_val': 17, 'rows_dropped_linked_to_both_train_and_val': 424, 'clusters_with_a_holdout_linked_member': 235, 'rows_linked_to_train': 329, 'rows_linked_to_pretrain': 33, 'rows_linked_to_val': 35, 'rows_dropped_holdout_shingle_link': 235, 'largest_cluster': 52}
	- bigshp label contradictions among exact matches: 0

	### test_holdout3 rows taken from finetune_val
	- legitimate from ['CEAS-08', 'TREC-07', 'Assassin'], spam from ['Assassin', 'Enron']; whole near-dup clusters only; excluded candidates by reason: {'Enron': {'cluster_has_an_ineligible_member': 261, 'shingle_link_to_a_new_row': 28, 'shingle_link_pretrain': 4}, 'Assassin': {'shingle_link_pretrain': 8, 'cluster_has_an_ineligible_member': 7}, 'CEAS-08': {'shingle_link_pretrain': 1}, 'TREC-07': {'cluster_has_an_ineligible_member': 1}}
	- per class: {'Legitimate': {'target_rows': 2000, 'eligible_rows': 4621, 'fraction': 0.4328, 'moved_rows': 1999}, 'Spam': {'target_rows': 700, 'eligible_rows': 1155, 'fraction': 0.6061, 'moved_rows': 700}}
	\| source \| 0 Legitimate \| 1 Phishing \| 2 Spam \| -1 unwanted (unsplit) \| rows \|
	\|---\|---\|---\|---\|---\|---\|
	\| Assassin \| 164 \| 0 \| 72 \| 0 \| 236 \|
	\| CEAS-08 \| 736 \| 0 \| 0 \| 0 \| 736 \|
	\| Enron \| 0 \| 0 \| 628 \| 0 \| 628 \|
	\| TREC-07 \| 1,099 \| 0 \| 0 \| 0 \| 1,099 \|

	### Splits by label3
	#### finetune_train: 160,263 rows, by label3 {'-1': 62805, '0': 81438, '1': 3150, '2': 12870}, binary positives 78,825 (0.4918)
	\| source \| 0 Legitimate \| 1 Phishing \| 2 Spam \| -1 unwanted (unsplit) \| rows \|
	\|---\|---\|---\|---\|---\|---\|
	\| Assassin \| 3,705 \| 0 \| 1,388 \| 0 \| 5,093 \|
	\| CEAS-08 \| 15,547 \| 0 \| 0 \| 18,631 \| 34,178 \|
	\| Enron \| 12,741 \| 0 \| 11,482 \| 0 \| 24,223 \|
	\| Nazario \| 0 \| 1,159 \| 0 \| 0 \| 1,159 \|
	\| Nigerian_Fraud \| 0 \| 1,991 \| 0 \| 0 \| 1,991 \|
	\| TREC-05 \| 27,663 \| 0 \| 0 \| 18,723 \| 46,386 \|
	\| TREC-07 \| 21,782 \| 0 \| 0 \| 25,451 \| 47,233 \|

	#### finetune_val: 15,284 rows, by label3 {'-1': 7179, '0': 7002, '1': 340, '2': 763}, binary positives 8,282 (0.5419)
	\| source \| 0 Legitimate \| 1 Phishing \| 2 Spam \| -1 unwanted (unsplit) \| rows \|
	\|---\|---\|---\|---\|---\|---\|
	\| Assassin \| 216 \| 0 \| 64 \| 0 \| 280 \|
	\| CEAS-08 \| 963 \| 0 \| 0 \| 2,184 \| 3,147 \|
	\| Enron \| 1,381 \| 0 \| 699 \| 0 \| 2,080 \|
	\| Nazario \| 0 \| 126 \| 0 \| 0 \| 126 \|
	\| Nigerian_Fraud \| 0 \| 214 \| 0 \| 0 \| 214 \|
	\| TREC-05 \| 2,997 \| 0 \| 0 \| 2,052 \| 5,049 \|
	\| TREC-07 \| 1,445 \| 0 \| 0 \| 2,943 \| 4,388 \|

	#### test_holdout: 16,302 rows, by label3 {'-1': 3894, '0': 12408}, binary positives 3,894 (0.2389)
	\| source \| 0 Legitimate \| 1 Phishing \| 2 Spam \| -1 unwanted (unsplit) \| rows \|
	\|---\|---\|---\|---\|---\|---\|
	\| TREC-06 \| 12,408 \| 0 \| 0 \| 3,894 \| 16,302 \|

	#### test_holdout3: 3,220 rows, by label3 {'0': 1999, '1': 521, '2': 700}, binary positives 1,221 (0.3792)
	\| source \| 0 Legitimate \| 1 Phishing \| 2 Spam \| -1 unwanted (unsplit) \| rows \|
	\|---\|---\|---\|---\|---\|---\|
	\| Assassin \| 164 \| 0 \| 72 \| 0 \| 236 \|
	\| CEAS-08 \| 736 \| 0 \| 0 \| 0 \| 736 \|
	\| Enron \| 0 \| 0 \| 628 \| 0 \| 628 \|
	\| Nazario \| 0 \| 222 \| 0 \| 0 \| 222 \|
	\| Nigerian_Fraud \| 0 \| 299 \| 0 \| 0 \| 299 \|
	\| TREC-07 \| 1,099 \| 0 \| 0 \| 0 \| 1,099 \|

	### Leakage assertions (binary gate + three-way)
	\| check \| count \| asserted zero \| pass \|
	\|---\|---\|---\|---\|
	\| holdout_vs_train_key_rows \| 0 \| True \| True \|
	\| holdout_vs_val_key_rows \| 0 \| True \| True \|
	\| holdout_vs_pretrain_key_rows \| 0 \| True \| True \|
	\| holdout_vs_train_shingle_links \| 0 \| True \| True \|
	\| holdout_vs_val_shingle_links \| 0 \| True \| True \|
	\| holdout_vs_pretrain_shingle_links \| 0 \| True \| True \|
	\| train_vs_val_k_full_rows \| 0 \| True \| True \|
	\| train_vs_val_any_key_rows \| 0 \| False \| True \|
	\| train_vs_val_shingle_links \| 0 \| False \| True \|
	\| val_vs_pretrain_key_rows \| 0 \| True \| True \|
	\| val_vs_pretrain_shingle_links \| 5,336 \| False \| True \|
	\| train_vs_pretrain_key_overlap \| 102,100 \| False \| True \|
	\| holdout_rows_in_train_or_val_by_source \| 0 \| True \| True \|
	\| test_holdout_rows_from_other_sources \| 0 \| True \| True \|
	\| pretrain_rows \| 226,273 \| False \| True \|
	\| holdout3_vs_train_key_rows \| 0 \| True \| True \|
	\| holdout3_vs_val_key_rows \| 0 \| True \| True \|
	\| holdout3_vs_pretrain_key_rows \| 0 \| True \| True \|
	\| holdout3_vs_holdout_key_rows \| 0 \| True \| True \|
	\| holdout3_vs_train_shingle_links \| 0 \| True \| True \|
	\| holdout3_vs_val_shingle_links \| 0 \| True \| True \|
	\| holdout3_vs_pretrain_shingle_links \| 0 \| True \| True \|
	\| holdout3_vs_holdout_shingle_links \| 0 \| False \| True \|
	\| holdout3_rows_without_full_label3 \| 0 \| True \| True \|
	\| holdout3_rows_from_holdout_source \| 0 \| True \| True \|
	\| doc_id_shared_between_splits \| 0 \| True \| True \|
	\| label3_contract_violations \| 0 \| True \| True \|
	\| holdout3_rows \| 3,220 \| False \| True \|

	### Files
	- `finetune_train`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_train.parquet`
	- `finetune_val`: `/home/nathank/BITS/dev/Phishing Model/data/processed/finetune_val.parquet`
	- `test_holdout`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout.parquet`
	- `test_holdout3`: `/home/nathank/BITS/dev/Phishing Model/data/processed/test_holdout3.parquet`