presentation/README.md

BITS Phishing Detector

Final project for the Skillspire AI & Machine Learning course (Nathan King, Build IT Systems LLC), presented late Oct / early Nov 2026. This README is the hand-off document for the instructor and for future BITS work. The running record of every decision, number and problem is RUN_LOG.md (entries 0-8 as of this writing); this file summarises it and explains how to reproduce it. Status as of 2026-10-08 14:00: data, tokenizer, smoke test and all src/, scripts/ and app/ code are done; MLM pretraining is running; the presentation notebook is in progress; fine-tuning, evaluation, export and deployment have not run yet.

---

1. What this is

Task	Classifier (not a regressor). Input: one email as subject + "\n\n" + body (plain text; sender, headers and attachments are never read). Output: probabilities for three classes, Legitimate / Phishing / Spam, plus the roll-up Unwanted (phishing or spam) = P(Phishing) + P(Spam).
Use	Defensive triage: flag unwanted mail for a human to review. Not a filter that deletes mail, not a tool for writing lures.
Model	One BERT-style pre-LayerNorm encoder written from scratch in plain PyTorch (src/model.py, no transformers, no pretrained weights): 24 layers, d_model 1024, 16 heads, FFN 4096, vocab 32,768 (own byte-level BPE), 512 tokens. 337,476,608 parameters with the tied MLM head, 337,443,842 as the classifier.
Training	Phase 1: masked-language-model pretraining on deduplicated Enron email. Phase 2: fine-tuning with a 3-way head. Rows whose corpus never separated spam from phishing (TREC-05/07, CEAS-08 positives) train on P(phishing)+P(spam) with a partial-label loss.
Core challenge	Public corpora label "unwanted" mail, not "phishing vs spam"; they are 1998-2008 heavy, overlap each other and the pretraining corpus, and legitimate mail is about twice as long as spam. The work is mostly in honest data: deduplication, leakage checks and a test set nothing upstream has seen.
Scope	One strong model. The experiments vary the instructor's dials (layers, epochs, loss, frozen layers, synthetic data, scratch vs pretrained init) on this one architecture; there is no architecture comparison.

2. Results

2.1 Headline metrics (deployed run ft-focal, evaluated 2026-10-09 10:46)

Model selection used the validation set only: ft-focal had the best validation F1 of the unwanted roll-up (0.9951) and the best validation three-way macro-F1 (0.9869) of the six runs in section 9. The two holdouts below were then evaluated once per run for reporting; ft-focal also leads on both of them.

TREC-06 holdout (16,302 emails, 23.9% unwanted; whole corpus unseen), threshold 0.5	value
Recall, unwanted (reported first: a missed phish is the costly error)	0.9741 (101 of 3,894 missed)
Precision, unwanted	0.8794 (520 false alarms on 12,408 legitimate; FPR 4.2%)
F1, unwanted	0.9243 (best achievable at any threshold: 0.9397 at p >= 0.68)
Accuracy	0.9619
ROC-AUC / PR-AUC	0.9946 / 0.9848
Recall at 1% FPR / at 0.1% FPR	0.8955 / 0.6497
Credential-lure slice (398 rows matching 37 lure phrases) recall / precision	0.9959 / 0.9758

TREC-06 is the hard, honest number: a different corpus (2006 TREC spam track) with its own labelling conventions. reports/error_analysis.md shows that many "false positives" are spam-looking mail that TREC-06 labels ham (quoted spam inside mailing-list threads, "remove me from your list" replies, sweepstakes mail) and that the largest block of misses are "ScanMail virus found" notices TREC-06 labels spam. Legitimate TREC-06 mail is also ~3x longer than its spam and 24% of it is truncated at 512 tokens.

test_holdout3 (3,220 emails: 1,999 legit / 521 phishing / 700 spam; tokenizer never saw them)	Legitimate	Phishing	Spam
Precision / Recall / F1	0.9900 / 0.9935 / 0.9918	0.9961 / 0.9866 / 0.9913	0.9785 / 0.9757 / 0.9771
Macro-F1 (3 classes) / accuracy	0.9867 / 0.9885
Binary roll-up (unwanted vs legitimate): F1 / recall / precision / ROC-AUC / PR-AUC	0.9864 / 0.9836 / 0.9893 / 0.9993 / 0.9989
Phishing vs spam, given the mail is unwanted (1,221 rows)	0.9959 (4 phishing called spam, 1 spam called phishing)

3x3 confusion matrix (rows true Legitimate / Phishing / Spam, columns predicted): [1986, 1, 12], [4, 514, 3], [16, 1, 683]. Three-way accuracy per source: Assassin 95.3% (n=236), CEAS-08 99.0% (n=736), Enron 98.4% (n=628), Nazario 98.6% (n=222), Nigerian_Fraud 98.7% (n=299), TREC-07 99.8% (n=1099).

Prediction time, one email, batch 1 (reports/latency.json, 100 TREC-06 samples, mean 280 tokens)	mean	p50	p95	vs 74 ms benchmark
GPU, bf16 (RTX 3090 Ti)	10.1 ms	8.9 ms	13.7 ms	0.14x (faster)
CPU, fp32, 16 threads	459 ms	445 ms	701 ms	6.2x (slower)

Plots next to these files: reports/confusion_matrix.png, confusion_matrix_test3.png, confusion_matrix3_test3.png (3x3), roc_curve*.png, pr_curve*.png, finetune_curves.png, pretrain_curves.png, eval_finetune_curves*.png, eval_pretrain_curves*.png; written error analyses error_analysis.md / error_analysis_test3.md. The fine-tune experiment table is finetune.py list (section 9). Every run's own evaluation lives in reports/eval_<run>/.

2.2 Measured facts that exist today

Fact	Value	Source
Parameters	337,476,608 (MLM, tied head) / 337,443,842 (classifier)	reports/smoke_test.json
Pretraining VRAM, micro-batch 16, no checkpointing	13.65 GiB reserved, 23,642 tok/s	reports/smoke_test.json
Pretraining VRAM, micro-batch 24, no checkpointing (used)	17.65 GiB reserved, 19.80 GiB total device incl. desktop, 25,227 tok/s	reports/smoke_test.json
Pretraining VRAM, micro-batch 24, gradient checkpointing	8.06 GiB reserved, 19,156 tok/s	reports/smoke_test.json
Measured bf16 matmul	63.9 TFLOPS (4096^3); the smoke test's 25k tok/s is ~63% MFU of the 80 TFLOPS peak	RUN_LOG 0, 3
Convergence check (fresh init, 30 steps)	MLM loss 10.58 -> 7.67 (ln V = 10.40)	RUN_LOG 3
Latency, one 512-token email, GPU bf16 (RTX 3090 Ti), untrained-weight smoke test	10.7 ms (trained checkpoint, real emails: section 2.1)	reports/smoke_test.json
Latency, one 512-token email, CPU fp32, 16 threads, untrained-weight smoke test	699 ms (319 ms at 128 tokens)	reports/smoke_test.json
Pretraining plan large-24L	12 epochs x 796 steps = 9,552 optimizer steps, micro-batch 24 x 11 = effective batch 264, peak LR 1.5e-4, warmup 573, cosine to 10%	checkpoints/pretrain/large-24L/run_summary.json
Pretraining result large-24L (complete 2026-10-09 03:31)	9,552 steps, 1.29B tokens, 15.05 h of training, 23,889 tok/s mean, peak 17.94 GiB reserved; train MLM loss 1.524, val MLM loss 1.986, masked-token accuracy 62.9% (val loss fell monotonically, no restart needed)	checkpoints/pretrain/large-24L/run_summary.json, metrics.csv

The smoke-test latencies were measured on untrained weights (latency does not depend on the weights); the numbers to quote are the trained-checkpoint measurements in section 2.1 (reports/latency.json). The instructor's 0.074 s/sample benchmark is met on GPU by a factor of ~7 and missed on CPU by a factor of ~6, which is why the Space runs on ZeroGPU.

2.3 Data splits

Binary build (RUN_LOG 1, data_prep.py build):

split	rows	positives	base rate	sources
finetune_train	157,113	75,675	48.2%	TREC-05 46,386 / TREC-07 47,233 / CEAS-08 34,178 / Enron 24,223 / Assassin 5,093
finetune_val	17,643	8,642	49.0%	same five, grouped by near-duplicate cluster
test_holdout (TREC-06)	16,302	3,894	23.9%	never seen by tokenizer, pretraining or fine-tuning
pretrain (MLM)	226,273 docs	-	-	corbt/enron-emails after dedup + quote stripping; 407.9M chars, 107.7M tokens

Three-way build (RUN_LOG 7, data_prep.py build3; label3 0 legit / 1 phishing / 2 spam / -1 unwanted-unsplit):

split	rows	legit	phishing	spam	unwanted (TREC/CEAS positives)
finetune_train	160,263	81,438	3,150	12,870	62,805
finetune_val	15,284	7,002	340	763	7,179
test_holdout (TREC-06, rows unchanged)	16,302	12,408	-	-	3,894
test_holdout3	3,220	1,999	521	700	-

Plus data/processed/synthetic.parquet: 2,000 template-generated rows (1,000 legitimate, 1,000 phishing lures, 0 spam), training-only, off by default. finetune_val shrank from 17,643 to 15,284 rows when test_holdout3 was carved out, so val numbers from binary-era runs are not strictly comparable.

---

3. Setup on Fedora 44

Machine used: Fedora 44 KDE, RTX 3090 Ti 24 GB (sm_86), NVIDIA driver 615.71.09 (CUDA 13.4), Ryzen 9 5950X, 40 GB RAM, system Python 3.14.7, uv 0.12.5 (~/.local/bin/uv), tmux.

cd "/home/nathank/BITS/dev/Phishing Model"
uv venv --python 3.14.7 .venv
uv pip install --python .venv/bin/python --torch-backend=cu130 -r requirements-train.txt
hf auth login                       # .venv/bin/hf; needed for the upload step only

Why these pins:

• --torch-backend=cu130, never auto. The plain PyPI torch 2.14.1 Linux wheel is the CUDA 13.0 build; driver 615 runs it by backward compatibility. uv 0.12.5's auto would pick cu132 on this driver. One resolution installs everything (180 packages, all binary wheels, venv ~6.1 GB).
• Python 3.14.7, not 3.15. torch.compile raises on 3.15, triton 3.8.0 requires <3.15 and spaces requires <3.15. Always pass --python 3.14.7.
• huggingface_hub==1.33.0 because tokenizers 0.23.2 requires huggingface-hub<2.0.
• fsspec==2026.7.0 because datasets 5.1.0 caps fsspec[http]<=2026.7.0 (2026.9.0 is newer and breaks the resolve). Do not uv pip install -U fsspec.

Verify the GPU (do not run while a training job holds the GPU unless you accept a small extra CUDA context):

nvidia-smi
.venv/bin/python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.get_device_name(0), torch.cuda.is_bf16_supported())"
# expected (2026-10-07): 2.14.1+cu130 13.0 NVIDIA GeForce RTX 3090 Ti True

---

4. Project layout

Phishing Model/
├── README.md                     this file
├── RUN_LOG.md                    dated log of every run, decision, number, bug and fix (the "challenges" source)
├── config.yaml                   every dial in one file (section 6)
├── requirements-train.txt        pinned local training stack (Python 3.14.7, torch 2.14.1 cu130)
├── .gitignore                    excludes .venv, data, checkpoints, logs, weights, tokens
├── src/
│   ├── common.py                 config loading, seeding, logging, text normalisation, dedup keys (shared by all)
│   ├── data_prep.py              download -> dedupe -> decontaminate -> split -> leakage asserts (build/check/build3/check3)
│   ├── tokenizer.py              byte-level BPE (NFKC, case kept): train on pretrain + finetune_train only; stats
│   ├── model.py                  from-scratch pre-LN encoder, MLM head, classification head (copied to app/)
│   ├── smoke_test.py             VRAM micro-batch finder, throughput, SDPA backend check, convergence, latency
│   ├── pretrain.py               resumable bf16 MLM pretraining (train/export/plot)
│   ├── make_synthetic.py         2,000 template-generated training rows (build/show)
│   ├── finetune.py               3-way classifier fine-tuning with the instructor's dials (train/plot/list)
│   ├── evaluate.py               the only reader of the holdouts: metrics, latency, plots, error analysis
│   └── export.py                 bf16 safetensors + config + model card -> model_export/<run>/ and app/
├── scripts/
│   ├── launch_pretrain.sh        start or resume pretraining in tmux session `phish-pretrain`
│   ├── run_experiments.sh        fine-tune queue: waits for large-24L/final, runs the six dial experiments,
│   │                             evaluates each into reports/eval_<run>/, exports ft-ce to model_export/ft-ce and app/
│   └── upload_to_hf.py           publish model repo + Space; dry run by default, `--yes` to write
├── app/                          the Hugging Face Space
│   ├── app.py                    Gradio UI: 3-class label, prediction time, token saliency, rule-based cues
│   ├── README.md                 Space front matter (sdk gradio 6.29.1, python 3.12.12) + description
│   ├── requirements.txt          Space pins (torch 2.13.0, spaces 0.51.3, ...)
│   ├── .gitattributes            LFS rules for weights and tokenizer
│   └── model.py, model.safetensors, config.json, tokenizer.json   written by export.py (not yet present)
├── FinalProject_PhishingDetector.ipynb   presentation notebook in the instructor's 9-step pattern (section 5.10);
│                                 written by the notebook step, in progress as of 2026-10-08 14:00
├── reference/
│   ├── FashionMNIST_Gradio_Full.ipynb   instructor's example notebook (the 9-step pattern)
│   └── INSTRUCTOR_REQUIREMENTS.md       grading rubric, verbatim
├── docs/research/
│   ├── research_dataset-schemas.md      schemas, provenance and licences of every candidate dataset
│   ├── verify_data.md                   independent verification of the data plan and holdout choice
│   ├── research_model-math.md           parameter / memory / FLOP / latency math for the encoder
│   ├── model_math.py                    pure-Python sizing calculator behind that document
│   ├── research_torch-python.md         torch 2.14.1 / CUDA 13.0 / Python 3.14 compatibility, SDPA backends
│   ├── research_other-libs.md           every other pin and its conflicts (fsspec, huggingface_hub, spaces)
│   ├── verify_pins.md                   independent verification of the pins
│   ├── research_gradio-spaces-hub.md    Gradio 6, ZeroGPU, Space front matter, Hub upload API
│   └── bigshp_research.md               why the 300k "spam/ham/phish" set was rejected
├── data/                         (git-ignored)
│   ├── raw/                      HF cache for corbt, Champa mirror CSVs (aura/), inspected-only 300k set (bigshp/)
│   └── processed/
│       ├── finetune_train.parquet, finetune_val.parquet    training / model-selection splits (with label3)
│       ├── test_holdout.parquet                            TREC-06 binary test set
│       ├── test_holdout3.parquet                           clean-provenance three-way test set
│       ├── pretrain.parquet                                MLM corpus (226,273 docs)
│       ├── pretrain_tokens.npy/.json, pretrain_val_tokens.npy   cached token stream used by pretrain.py
│       ├── synthetic.parquet, synthetic_report.json        template supplement + shortcut audit
│       ├── tokenizer.json, tokenizer_report.json           BPE vocabulary + per-split length stats
│       ├── splits_manifest.json, data_report.md            counts, removals, every leakage assertion
│       └── binary_snapshot/                                binary-era splits that build3 starts from
├── checkpoints/                  (git-ignored)
│   ├── pretrain/large-24L/       live run: step-NNNNNNN/, metrics.csv, run_summary.json, final/ when done
│   ├── pretrain/dev/             tiny self-test run (delete after Nathan's OK)
│   └── finetune/<run>/           best/, last/, metrics.csv, epochs.csv per fine-tune run (dev, dev-focal = self-tests)
├── reports/                      smoke_test.json; experiments.jsonl + finetune_curves.png + pretrain_curves.png are
│                                 dev-only until the real runs regenerate them (section 9); metrics*, latency*,
│                                 confusion/ROC/PR plots, error analyses, eval_<run>/ later
├── model_export/<run>/           export output (created by export.py)
└── logs/                         one timestamped log per script invocation + tmux stdout of the pretraining run

---

5. How to run each phase, in order

All commands run from the project root with .venv/bin/python (written PY below: PY=.venv/bin/python). Every script takes --config config.yaml before the subcommand and writes a log to logs/<script>-<timestamp>.log. Flags below were checked against each script's --help on 2026-10-08. While pretraining holds the GPU, prefix any other Python command with CUDA_VISIBLE_DEVICES="" and do not start smoke_test, finetune, evaluate on GPU, or a second training job.

5.1 Data (src/data_prep.py)

$PY src/data_prep.py inspect        # optional: schema + 2 rows per dataset, streaming
$PY src/data_prep.py build          # binary splits + pretrain corpus + manifest + data_report.md (~2.5 min, 6.9 GB RSS)
$PY src/data_prep.py check          # recompute every binary leakage assertion from the written files
$PY src/data_prep.py build3         # three-way labels, Champa phishing corpora, test_holdout3 (~1 min)
$PY src/data_prep.py check3         # recompute binary + three-way assertions

Flags: build --limit N / build3 --limit N are dev-only subsets (build3 --limit writes to data/processed/dev_threeway_limitN/); build --spam-policy {positive_class,map_to_zero,exclude} (default positive_class); build --no-strip-quotes; --shingle-cap N; build3 --bigshp {on,off} overrides the rejected 300k set (leave off); check3 --out-dir. Do not re-run build or tokenizer.py train while large-24L is running: a new pretrain.parquet or tokenizer.json rebuilds the token cache and the run then refuses to resume. build also rewrites the binary splits, so build3 must follow it.

5.2 Tokenizer (src/tokenizer.py)

$PY src/tokenizer.py train          # byte-level BPE, vocab 32,768, NFKC, case kept; ~30 s
$PY src/tokenizer.py stats          # length / truncation / [UNK] table per split -> tokenizer_report.json

train --vocab-size N, train --limit-docs N (dev). Trains only on pretrain.parquet + finetune_train.parquet; val and test files are refused by name and checked with an exact-hash tripwire.

5.3 Smoke test (src/smoke_test.py) — GPU must be free

$PY src/smoke_test.py               # -> reports/smoke_test.json + printed summary

Useful flags: --candidates 16,24,12,8, --ckpt-candidates, --steps 30, --skip-cpu, --n-docs 3000, --latency-samples 100, --cpu-threads 16. It probes micro-batches with the real gradient-accumulation pattern, measures tokens/s, checks which SDPA kernel runs, runs a 30-step convergence check and times single-email inference on GPU and CPU.

5.4 MLM pretraining (scripts/launch_pretrain.sh -> src/pretrain.py)

EPOCHS=12 MICRO_BATCH=24 scripts/launch_pretrain.sh     # how large-24L was launched (2026-10-08 12:28)
tmux attach -t phish-pretrain                           # watch; Ctrl-b d detaches; Ctrl-C inside = save + clean exit
RESUME=1 MICRO_BATCH=24 scripts/launch_pretrain.sh      # resume from the latest checkpoint

Environment variables: RUN_NAME (default large-24L), EPOCHS (script default 6; fixes the schedule length at first launch, ignored on resume), MICRO_BATCH (script default 16; always pass 24 for this run: 16 gives effective batch 256 instead of the run's 264 and the resume is refused), SESSION (default phish-pretrain), RESUME=1. The script refuses to start if the tmux session already exists; after the trainer exits the pane waits for Enter (or tmux kill-session -t phish-pretrain). Stdout is teed to logs/pretrain_<run>_<timestamp>.out.

Direct use: $PY src/pretrain.py train --run-name R [--resume [PATH|latest]] [--epochs N] [--max-steps N] [--max-hours H] [--micro-batch B] [--grad-checkpointing] [--limit-docs N] [--checkpoint-every N] [--log-every N] [--eval-every N] [--keep-checkpoints K] [--fresh]. After the run:

$PY src/pretrain.py plot --run-name large-24L      # -> reports/pretrain_curves.png
$PY src/pretrain.py export --run-name large-24L    # only if stopped early: builds final/ from the latest checkpoint

How it stops (every path writes a checkpoint first; checkpoints are atomic tmp-dir + rename):

Event	Behaviour	Exit code
Normal end	final/ written	0
Ctrl-C / SIGTERM	save at the next optimizer-step boundary	0
--max-hours reached	save	0
VRAM guard: this process's reserved peak > vram.hard_cap_gb (20 GiB)	save, stop before a real OOM	3
CUDA OOM (e.g. the desktop grabs VRAM) in forward/backward, optimizer step or eval	partial step discarded, data cursor rewound, save	3
5 consecutive non-finite losses (each such step is skipped)	status diverged, save	4
Config/plan mismatch on resume (model shape, pretrain: LR/betas/eps/decay/clip/masking/schedule, seed, effective batch, epochs, data)	refuses before training	2

Checkpoints every 500 steps and at every epoch end (keep 3, ~3.8 GB each) in checkpoints/pretrain/large-24L/step-NNNNNNN/; val MLM loss and accuracy on 64 fixed held-out windows every 500 steps in metrics.csv. If val MLM loss rises, stop the run (Ctrl-C) and use pretrain.py export.

5.5 Synthetic supplement (src/make_synthetic.py) — already built

$PY src/make_synthetic.py build [--n 2000] [--out PATH] [--seed S]   # -> data/processed/synthetic.parquet + _report.json
$PY src/make_synthetic.py show --k 6

5.6 Fine-tuning (src/finetune.py) — after pretraining, GPU free

Baseline (config defaults: 3 classes, cross-entropy over the partial-label NLL, 3 epochs, batch 32 = micro-batch 16 x 2, LR 3e-5, early stopping on val F1 of the unwanted roll-up, patience 2):

$PY src/finetune.py train --run-name ft-ce --init pretrained:checkpoints/pretrain/large-24L/final

ft-ce is the cross-entropy baseline. The queue ran all six runs of section 9 on 2026-10-09 (03:36-10:02); ft-focal won on the validation selection metric (val F1 of the unwanted roll-up 0.9951 vs 0.9929 for ft-ce, val three-way macro-F1 0.9869 vs 0.9782) and is therefore the run evaluated at top level, exported and deployed. Every example in sections 5.7-5.9 uses it.

The queue (scripts/run_experiments.sh) is how the experiments are meant to run: it waits for checkpoints/pretrain/large-24L/final/model.safetensors, renders pretrain.py plot, then trains the six runs of section 9 one after another with --eval-every 1250, plots each to reports/finetune_curves_<run>.png, evaluates each best/ on both holdouts into reports/eval_<run>/ (marker reports/eval_done_<run>), and finally evaluates + exports the main run (MAIN_RUN, default ft-ce) into the top-level reports/metrics.json / metrics_test3.json that section 2.1 quotes, model_export/<MAIN_RUN>/ and app/. The top-level evaluate + export were re-run by hand for ft-focal after the queue finished (section 9). A run whose best/ exists is skipped, so re-running the script continues after a crash.

tmux new-session -d -s phish-finetune 'scripts/run_experiments.sh 2>&1 | tee logs/experiments_$(date +%Y%m%d-%H%M%S).out'

Environment: PRETRAIN_RUN (default large-24L), EVAL_EVERY (default 1250), MAIN_RUN (default ft-ce).

The instructor's dials (all flags of train):

Dial	Flag	Values
Loss function	--loss	cross_entropy / weighted_cross_entropy (inverse-frequency) / focal (--focal-gamma 2.0)
Epochs	--epochs	e.g. 1-5; early stopping may end sooner
Number of layers	--n-layers N	keep the first N pretrained layers (e.g. 12 of 24)
Frozen layers	--freeze-layers K	freeze embeddings + first K layers
Synthetic data	--synthetic on|off	appends the 2,000-row supplement to train only
Init	--init scratch / --init pretrained:<dir>	scratch = ablation showing what pretraining buys
Others	--lr, --batch-size, --micro-batch, --num-labels 2|3, --seed, --eval-every, --log-every, --eval-batch-size
Dev only	--limit-rows N, --max-steps N, --cpu	not evidence

Always pass --init explicitly: without it the script takes the newest checkpoints/pretrain/*/final by modification time, and checkpoints/pretrain/dev/final (a tiny self-test model) exists.

$PY src/finetune.py train --run-name ft-focal --resume     # continue <run>/last after a crash or Ctrl-C
$PY src/finetune.py train --run-name ft-focal --overwrite  # discard an existing run dir of that name
$PY src/finetune.py list                                   # experiments.jsonl as a markdown table
$PY src/finetune.py plot --run-name ft-focal               # -> reports/finetune_curves.png (`--out PATH` to rename)

On resume, --init/--n-layers/--freeze-layers/--batch-size/--seed/--num-labels are refused; --lr is re-applied to the restored optimizer. Output: checkpoints/finetune/<run>/{best,last}/, metrics.csv, epochs.csv. The test files are refused by name and by source column.

5.7 Evaluation (src/evaluate.py) — the only script that opens the holdouts

$PY src/evaluate.py run --checkpoint checkpoints/finetune/ft-focal/best --split test,test3
$PY src/evaluate.py run --checkpoint checkpoints/finetune/ft-focal/best --split val   # model selection view, `_val` suffix

The queue runs this per experiment with --run-name <run> --out-dir reports/eval_<run>; only the main run is written to the top-level reports/ (section 2.1).

Writes reports/metrics.json, metrics_test3.json, latency.json (batch-1 timeit.default_timer around tokenize + forward + softmax, GPU bf16 and CPU fp32, next to the 74 ms benchmark), confusion_matrix*.png, confusion_matrix3_test3.png, roc_curve*.png, pr_curve*.png, error_analysis*.md (top false negatives / positives, phishing<->spam confusions, length-confound check, credential-lure slice), one line in reports/experiments.jsonl and one RUN_LOG.md entry. Flags: --threshold, --batch-size, --latency-samples, --cpu-threads, --max-len, --cpu, --out-dir, --data-dir, --run-name, --limit-rows (dev; no RUN_LOG entry), --no-run-log, --finetune-run-dir / --pretrain-run-dir (extra eval_*_curves.png). The binary metrics use P(unwanted) at 0.5; the three-way decision is roll-up first, then argmax(phishing, spam), so the 3x3 matrix collapses exactly onto the binary one (plain argmax-of-3 is reported alongside).

5.8 Export (src/export.py)

$PY src/export.py run --checkpoint checkpoints/finetune/ft-focal/best

Writes model_export/ft-focal/ (model.safetensors bf16 ~675 MB, config.json with label names, metrics and dials, tokenizer.json, README.md model card, export_manifest.json) and copies model.py (sha1-checked) plus the three weight files into app/. It refuses if reports/metrics*.json name a different checkpoint, so run 5.7 on the same best/ first (--allow-metrics-mismatch overrides and records it). Other flags: --out-dir, --app-dir, --dtype {bf16,fp32}, --metrics-json, --metrics3-json, --pretrain-config.

Local app check before uploading: CUDA_VISIBLE_DEVICES="" $PY app/app.py (CPU; drop the prefix once the GPU is free: app.py moves the model to CUDA at import time whenever a GPU is visible).

5.9 Upload (scripts/upload_to_hf.py) — Nathan only

$PY scripts/upload_to_hf.py --model-dir model_export/ft-focal --bundle-weights --check-auth   # dry run (default): prints the plan, no writes
$PY scripts/upload_to_hf.py --model-dir model_export/ft-focal --bundle-weights --yes          # live (what was run on 2026-10-12)

The dry run lists every file, size, repo, visibility, the hardware flavour and any blocking problem (a missing export, a missing app/model.py, or HF_TOKEN_FOR_SPACE unset when the weights are not bundled). Other flags: --bundle-weights (weights inside the Space, no secret needed; the mode used), --skip-secret, --hardware, --model-repo, --space-repo, --private/--no-private, --commit-message. --yes is only ever run with Nathan's explicit approval; the one live run happened on 2026-10-12 11:35 (RUN_LOG 24) after his approval of 2026-10-08. Lesson from that run: the Hub validates the Space card's YAML front matter at upload time and the dry run does not, so a colon inside an unquoted short_description failed the first attempt; the value is now quoted and the model-repo half of the first attempt was simply reused (LFS blobs dedupe by content hash).

5.10 The notebook

FinalProject_PhishingDetector.ipynb (project root; generated by scripts/build_notebook.py, last executed top to bottom on the CPU on 2026-10-12: 36 code cells, no errors, every artifact present) is the presentation walk-through. It mirrors the instructor's reference/FashionMNIST_Gradio_Full.ipynb step for step, with the nine headers quoted verbatim: "Step 1: Install Dependencies & Setup Device", "Step 2: Download Data & Create DataLoaders", "Step 3: Define the Neural Network", "Step 4: Train the Model", "Step 5: Test Inference Speed and Save the Model", "Step 6: Create the App Files (Optional)", "Step 7: Launch the Gradio Interface", "Step 8: Authenticate with Hugging Face", "Step 9: Upload the App to Hugging Face Spaces". It loads the artifacts above instead of training: the contract is that it runs top to bottom on CPU and prints "not available yet — produced by " for anything missing; with the 2026-10-09 artifacts in place it shows the real numbers, curves, confusion matrices and the ft-focal export.

CUDA_VISIBLE_DEVICES="" .venv/bin/jupyter lab
CUDA_VISIBLE_DEVICES="" .venv/bin/jupyter nbconvert --to notebook --execute --inplace FinalProject_PhishingDetector.ipynb

5.11 The Colab presentation notebook (FinalProject_PhishingDetector_Colab.ipynb)

Built by the same script with --colab (--out PATH picks the output file; the default mode and its output are unchanged). Design: the notebook is narrative, the Space is the compute. Nothing trains or infers in Colab; every prediction runs on the ZeroGPU Space, and all evidence is pulled from the Hub:

Need	Where it comes from in Colab
Metrics, curves, confusion matrices, experiments table, data report, run log, code	presentation/ in the model repo (85 files, 3.6 MB; relative paths mirror this project), fetched with snapshot_download into /content/phishing_bundle
Model definition and parameter count (Step 3)	presentation/src/model.py imported on the Colab CPU (torch is preinstalled there); falls back to the counts in config.json without torch
Inference speed (Step 5)	gradio_client.Client(SPACE, token=...).predict(subject, body, api_name="/predict") on six synthetic emails: in-app "Prediction Time (s)" measured on the Space's A10G (about 0.23 s cold, 0.01 s warm) plus the round-trip; latency.json adds the RTX 3090 Ti numbers
The app (Step 7)	gr.load(SPACE, src="spaces", token=...) embeds the Space UI; the private model repo's weights never leave the Hub. Gradio 6.29.1 has a bug rebuilding a Space whose Interface has examples= (Textbox.__init__() got an unexpected keyword argument 'class_id'), so the cell installs a two-line compatibility shim and retries, and falls back to a hand-built proxy gr.Interface over gradio_client if that also fails. When the Space is public the cell also shows the Space's own page in an IFrame of https://builditsystems-phishing-detector-demo.hf.space
Auth (Step 8)	Token resolution order: Colab secret HF_TOKEN -> os.environ["HF_TOKEN"] -> the hf auth login cache (outside Colab only when NOTEBOOK_ALLOW_LOGIN_CACHE=1, so a headless run never silently forwards the workstation's write token) -> none (enough once the repos are public). Only the token source is ever printed

How Nathan runs it: upload the .ipynb to Colab (File -> Upload notebook, or from Drive); while the repos are private, add a Colab Secret named HF_TOKEN with notebook access on, and make it a fine-grained read token scoped to the two repos, because gr.load forwards the token to the Space with every proxied call; run all cells. The first cell installs gradio==6.29.1 huggingface_hub==1.33.0 (only when it detects Colab). Once both repos are public no token is needed and the iframe appears automatically.

Verified on 2026-10-12: executed headlessly from an empty directory in a clean Python 3.12 venv (torch CPU, gradio 6.29.1, huggingface_hub 1.33.0) with no project file reachable: 36/36 code cells ran, zero errors, six live predictions from the Space, gr.load succeeded after the shim retry, the account name printed and no token-shaped string in any output; the numbers it renders were spot-checked against reports/metrics*.json (RUN_LOG 25). The Colab-only branches (%pip, userdata.get, inline launch()) are exercised by the IN_COLAB switch but were not run on a real Colab runtime; do one rehearsal there.

---

6. Config dials (config.yaml)

Section	What it controls
seed, paths	global seed 42; every relative path is resolved against the project root
model	encoder shape (24 / 1024 / 16 / 4096 / 32768 / 512), dropout, pre-LN, tied MLM head, gradient checkpointing
tokenizer	byte-level BPE, vocab 32,768, min frequency 2, specials [PAD] [UNK] [CLS] [SEP] [MASK], training corpora
data	sources, exclusions, holdout source (TREC-06), alnum minimums, dedup keys and shingle parameters, val fraction, synthetic, threeway (label3 by provenance, Champa mirror, the rejected bigshp set with its verdict, holdout3 carving)
vram	14-18 GiB target band, 20 GiB hard cap (GiB, compared with torch.cuda.max_memory_reserved()), allocator config
pretrain	bf16, packing, effective batch 256, micro-batch start 24 + candidates, 12 epochs, AdamW LR 1.5e-4 / betas (0.9, 0.98) / eps 1e-6 / decay 0.01 / clip 1.0, warmup, cosine to 10%, 15% masking 80/10/10, checkpoint/log/eval cadence
finetune	num_labels: 3, epochs, batch / micro-batch, LR 3e-5, warmup + linear decay, loss and class weights, focal gamma, early stopping metric + patience, frozen layers, padding multiple
eval	metric list, credential-lure slice, latency samples, threshold 0.5, test3 file, label contract (asserted against the code), top-k confusions, model-card metrics source
hf	owner, model and Space repo ids, private flag, Gradio 6.29.1, Python 3.12.12, zero-a10g, Space torch 2.13.0, safetensors bf16

Rule during a live pretraining run: do not edit model: or the pretrain: keys that shape the plan (lr, betas, eps, weight_decay, grad_clip, mlm_probability, mask_replace_probs, schedule, effective_batch_sequences, warmup_steps, warmup_fraction), seed, or anything that changes pretrain.parquet / tokenizer.json. Each is stored in the checkpoint and a resume refuses a mismatch. This is why num_labels lives under finetune:. finetune:, eval: and hf: can be edited at any time.

---

7. Data provenance and licences

Source	Used for	Licence / terms	Notes
puyang2025/seven-phishing-email-datasets @ b7f3f8d9 (203,017 rows)	labelled legit + unwanted: TREC-05/06/07, CEAS-08, Enron-Spam, SpamAssassin	Hub card "other"; upstream Champa, Rabbi & Zibran, "Seven Phishing Email Datasets", figshare 25432108, CC BY 4.0	Its own train/test split leaks (5.1% exact duplicates) and is ignored. Ling (2,859 academic list rows) excluded. Label 1 = "unwanted": these are spam-track corpora with no phishing/spam distinction.
corbt/enron-emails @ cfc06c75 (517,401 rows)	MLM pretraining only	CMU Enron corpus (FERC public release); no formal licence; CMU asks users to respect privacy	52.7% exact duplicates removed, quoted/forwarded history stripped -> 226,273 docs.
kudzaiprichard/aura-phishing-email-corpus @ 539567ce: Nazario.csv, Nigerian_Fraud.csv	phishing class (1,507 + 2,504 rows kept)	CC BY 4.0; byte-identical mirror of Champa, Rabbi & Zibran, "Phishing Email Curated Datasets", Zenodo, doi:10.5281/zenodo.8339691	Nazario phishing archive and advance-fee (419) fraud, 1995-2022. 235 Nigerian rows near-duplicating TREC-06 spam and 13 exact matches removed to keep the test set clean.
data/processed/synthetic.parquet (2,000 rows)	optional training supplement	generated by make_synthetic.py; fictional names, .example/.invalid domains, RFC 5737 IPs, 555-01xx phones	8 BEC / Microsoft 365 / invoice lure categories vs 8 ordinary business categories; shared filler pools and a token-purity audit against template shortcuts. Never in val or test, never presented as real mail.
Rejected: locuoco/the-biggest-spam-ham-phish-email-dataset-300000	nothing	MIT (Kaggle re-upload)	365,448 rows, 51.5% internal duplicates, a union of corpora we already hold; its "phish" label agrees with provenance only 60.5%; 36,020 rows are copies of our TREC-06 test set; phish rows went through a stopword-stripping pipeline the others did not (a label-proxy artifact). Not one row ingested. docs/research/bigshp_research.md.
Rejected: it4lia/PhishingEmailCuratedDatasets_Cleaned	nothing	CC BY 4.0	no body text (subjects/headers + 53 numeric features).
Rejected: zefang-liu/phishing-email-dataset	nothing	LGPL-3.0	lowercased re-packaging, 91.9% contained in the primary set, "enron" token shortcut.

Details and evidence: docs/research/research_dataset-schemas.md, docs/research/verify_data.md, data/processed/data_report.md. The exported model card states that phishing/spam labels follow corpus provenance.

---

8. Leakage policy

• TREC-06 is the binary test set and nothing upstream has seen it. It is excluded from tokenizer training, the MLM corpus and fine-tuning. It is Enron-clean (5 rows mention "enron", 0 enron.com addresses); TREC-05 ham and Enron-Spam ham are 86-89% inside the pretraining corpus, so neither could be the test set. Model input is subject + body only because TREC rewrote the spam headers.
• Decontamination. Exact keys on normalised text (full, 300-char prefix, subject+body prefix, subject+body full) plus near-duplicates (>= 3 shared 50-char shingles over the first 1,500 chars). 1,652 training rows that near-duplicate TREC-06 templates were removed; 85 pretraining docs near-duplicating holdout emails and 4,004 matching val keys were removed from the MLM corpus.
• Train/val split is grouped by near-duplicate cluster and stratified by source x label, so no template straddles train and val.
• test_holdout3 rules: legitimate rows only from CEAS-08, TREC-07, SpamAssassin (never Enron, which overlaps pretraining); spam from SpamAssassin and Enron-Spam spam folders; phishing from Nazario + Nigerian_Fraud. Rows come from finetune_val (never seen by the tokenizer) or the new sources (never seen by anything); whole clusters move or none; any cluster with a key or shingle link to the pretraining corpus or train is excluded.
• Asserted zeros (recomputed independently from the written files by check / check3; a non-zero count fails the build): holdout vs train / val / pretrain keys and shingle links; train vs val full-key rows; val vs pretrain keys; holdout source rows in train/val; holdout3 vs train / val / pretrain keys and shingle links; holdout3 vs TREC-06 keys; doc_id shared between splits; label3 contract violations. All are 0 (data/processed/data_report.md). Counted but allowed: 5,336 val-to-pretrain near-dup links (val is only used for model selection), 102,100 train-to-pretrain key overlaps (Enron ham).
• Code guards. The tokenizer refuses val/test files by name with an exact-hash tripwire (0 collisions); finetune.py refuses the holdout by file name and source column; pretrain.py never opens it; evaluate.py is the only reader.

---

9. Experimentation record

• reports/experiments.jsonl: finetune.py train appends one JSON line per run (dials: init, layers, frozen layers, epochs planned/run, loss, focal gamma, LR, batch, class weights, synthetic rows, seed, selection metric; parameter counts; steps, seconds, peak VRAM; best step; val metrics) and evaluate.py appends one line per invocation with the test metrics. finetune.py list renders it as the markdown table for the presentation. Generated from evidence, never typed up afterwards.
• Contents: the six real runs below (2026-10-09) plus one evaluate line per holdout evaluation. The dev self-test rows from 2026-10-08 were deleted before the queue ran (RUN_LOG 9). reports/finetune_curves.png is the ft-focal curve, reports/finetune_curves_<run>.png the others, reports/pretrain_curves.png the large-24L curve.
• Pretraining is one run (large-24L); its curve is metrics.csv -> pretrain.py plot.
• Fine-tune queue (scripts/run_experiments.sh, section 5.6; one architecture, one dial changed per run against the baseline). It ran unattended on 2026-10-09 from 03:36 to 10:02 (six runs, each evaluated on both holdouts). The baseline ft-ce was allowed 3 epochs; the five ablations 2 epochs each for run-time reasons; early stopping (patience 2 on the validation unwanted-F1, evaluated every 1,250 steps) ended five of the six before their epoch budget. All start from checkpoints/pretrain/large-24L/final except ft-scratch.

Run	Flags	Epochs	Question it answers
ft-ce	--loss cross_entropy	3	reference point: pretrained 24L, cross-entropy, synthetic off
ft-weighted	--loss weighted_cross_entropy	2	class imbalance (3,150 phishing vs 81,438 legit fully-labelled rows)
ft-focal	--loss focal (gamma 2.0 from config)	2	hard-example weighting
ft-synthetic	--loss cross_entropy --synthetic on	2	does the BEC/M365 supplement move anything (reported even if it does not)
ft-12L	--loss cross_entropy --n-layers 12	2	number of layers (half the stack, ~2x faster inference)
ft-scratch	--loss cross_entropy --init scratch	2	what MLM pretraining buys

9.1 Results (validation = selection; holdouts = reporting)

Selection view (validation split, 15,284 rows; the queue's early-stopping metric is the unwanted-F1 column):

Run	Loss / dial	Epochs run / planned	Stopped by	Best step	Train min	Peak GiB	Val F1 unwanted (selection)	Val macro-F1 3-way	Val P / R phishing
ft-ce	cross_entropy	1.25 / 3	early_stopping	5000	51	13.13	0.9929	0.9782	0.939 / 0.997
ft-weighted	weighted_cross_entropy	2.00 / 2	epochs_done	10000	84	13.13	0.9937	0.9775	0.963 / 0.997
ft-focal	focal (gamma 2.0)	2.00 / 2	early_stopping	8750	84	13.13	0.9951	0.9869	0.985 / 0.988
ft-synthetic	cross_entropy, synthetic on	1.23 / 2	early_stopping	5000	52	13.13	0.9927	0.9665	0.881 / 0.997
ft-12L	cross_entropy, 12 layers	1.75 / 2	early_stopping	6250	35	6.97	0.9925	0.9774	0.957 / 0.982
ft-scratch	cross_entropy, scratch init	1.50 / 2	early_stopping	5009	61	13.13	0.9875	0.9538	0.938 / 0.941

Held-out view (TREC-06: 16,302 rows, whole corpus unseen; test_holdout3: 3,220 rows, 1,999 / 521 / 700). Evaluated once per run after training; not used for selection:

Run	TREC-06 recall	precision	F1	FPR	ROC-AUC	PR-AUC	recall @ 1% FPR	test3 F1 unwanted	test3 macro-F1	F1 phishing	F1 spam
ft-ce	0.9404	0.8610	0.8990	4.8%	0.9875	0.9626	0.7745	0.9865	0.9822	0.9769	0.9779
ft-weighted	0.9818	0.8307	0.9000	6.3%	0.9943	0.9833	0.8767	0.9862	0.9852	0.9875	0.9767
ft-focal	0.9741	0.8794	0.9243	4.2%	0.9946	0.9848	0.8955	0.9864	0.9867	0.9913	0.9771
ft-synthetic	0.9679	0.8402	0.8995	5.8%	0.9908	0.9716	0.7871	0.9873	0.9814	0.9772	0.9749
ft-12L	0.9461	0.8120	0.8739	6.9%	0.9836	0.9513	0.6608	0.9823	0.9807	0.9807	0.9721
ft-scratch	0.9165	0.7752	0.8400	8.3%	0.9755	0.9456	0.7773	0.9767	0.9634	0.9574	0.9470

What the dials say: focal loss is the clear winner (fewer misses on TREC-06 at a lower false-alarm rate, and the best phishing-vs-spam separation); inverse-frequency class weights buy recall at the cost of precision; the 2,000-row synthetic supplement does not help (reported, as promised, even though it does not move anything); halving the stack to 12 layers costs ~5 F1 points on TREC-06 for ~2x faster inference; and scratch initialisation is 8 F1 points behind the same model with MLM pretraining, which is what pretraining buys. ft-focal was selected on validation alone, before the holdouts were read, and is the run evaluated at top level (RUN_LOG 22), exported to model_export/ft-focal/ + app/ and uploaded (section 10).

Not run, optional if time allows: --freeze-layers 12 (forgetting vs cost), an epochs sweep (--epochs 1 / 5), --num-labels 2 (binary baseline comparable with the binary-era plan).

---

10. Deployment

Item	Value
Model repo	BuildITSystems/phishing-detector-encoder (private): model.safetensors (bf16), config.json, tokenizer.json, README.md model card
Space	BuildITSystems/phishing-detector-demo (private), Gradio SDK 6.29.1, hardware zero-a10g (ZeroGPU)
Space stack	torch 2.13.0 on Python 3.12.12: ZeroGPU supports torch 2.8.0-2.13.0 and Python 3.12.12 / 3.10.13 only, and no torch.compile. Training uses 2.14.1 on 3.14; safetensors makes the weights portable. Gradio is pinned by sdk_version in app/README.md, not in requirements.txt.
Weights at startup	The three weight files are bundled into the Space (upload_to_hf.py --bundle-weights, the mode used), so no secret is needed and the Space works even while the model repo is private. app.py can alternatively download them from the repo named by the PHISH_MODEL_REPO variable with an HF_TOKEN read-token secret.
UI	Subject + body in; 3-class gr.Label, "Prediction Time (s)" (timeit.default_timer), gradient x input token highlights, rule-based cues shown for context only (they never feed the probability).

Before class: make both repos public (the instructor needs the model link and a Space that opens without an invite), then open the Space once so ZeroGPU is warm. create_repo(exist_ok=True) does not change the visibility of an existing repo, so do it in each repo's Settings page on huggingface.co, or:

.venv/bin/python -c "from huggingface_hub import HfApi; a=HfApi(); a.update_repo_settings('BuildITSystems/phishing-detector-encoder', private=False); a.update_repo_settings('BuildITSystems/phishing-detector-demo', repo_type='space', private=False)"

app/README.md (the Space card) already links the model repo.

---

11. Instructor requirement mapping

Requirement (reference/INSTRUCTOR_REQUIREMENTS.md)	Where it is satisfied
Classifier, not regressor	3-class softmax classifier + binary roll-up (section 1)
Dataset from torchvision.datasets	Deviation, agreed text variant: there is no torchvision email dataset. Public Hugging Face datasets loaded with datasets (section 7); the PyTorch data pipeline is kept (DataLoader below).
Problem statement: input, classes, challenge	Section 1
Algorithm and architecture, >= 2 layers	24-layer transformer encoder, src/model.py; justification in docs/research/research_model-math.md and RUN_LOG 0
Loss function	Partial-label NLL (= cross-entropy for fully-labelled rows) with --loss cross_entropy / weighted_cross_entropy / focal; MLM cross-entropy on masked tokens for pretraining
Optimizer	AdamW (fused, no decay on biases/LayerNorm), grad clip 1.0, bf16 autocast
Epochs, early stopping / LR scheduling	Fine-tune: early stopping on val F1 (patience 2), warmup + linear decay. Pretrain: warmup + cosine to 10%, val MLM loss every 500 steps
Justify choices	This README, RUN_LOG.md, docs/research/
DataLoader with batching, shuffling, preprocessing	finetune.make_loader: torch.utils.data.DataLoader with a seeded length-bucketed shuffled batch_sampler and a padding collate; tokenisation as the preprocessing transform (RUN_LOG 8: moved from a numpy sampler on 2026-10-08 13:55, losses verified identical, explicit generator to keep the dropout stream unchanged)
Evaluate on an unseen test set with standard metrics	evaluate.py on TREC-06 and test_holdout3: accuracy, precision, recall, F1, ROC-AUC, PR-AUC, confusion matrices (section 2.1)
Inference timer, seconds per single prediction	evaluate.py -> reports/latency.json (GPU and CPU, batch 1); smoke_test.py; the app's "Prediction Time (s)"
Gradio UI: input -> label + inference time	app/app.py (text input instead of an image upload)
HF Spaces with app.py, weights, requirements.txt	Section 10; upload_to_hf.py
Class: dataset, trained model, performance results	Sections 2, 7; reports/
Class: model on Hugging Face, link to instructor	https://huggingface.co/BuildITSystems/phishing-detector-encoder (make public first)
Class: live Gradio app on a Space, not gradio.live	https://huggingface.co/spaces/BuildITSystems/phishing-detector-demo
Class: Space has app.py, model.py, requirements.txt (pinned torch/gradio) + weights	app/; torch pinned in requirements.txt, gradio pinned by sdk_version
Class: show experimentation (layers, epochs, loss), record each run	Section 9, reports/experiments.jsonl, finetune.py list, RUN_LOG.md
Class: prediction time per sample vs 0.074 s, GPU and CPU	Section 2.1: 10.1 ms GPU bf16, 459 ms CPU fp32 per email (reports/latency.json)
Class: presentation covers data, algorithm, performance, challenges	RUN_LOG.md (OOM tuning, leakage, rejected datasets, reviewer-found bugs), sections 2, 7, 8, 12
Class: synthetic data allowed	make_synthetic.py, --synthetic on, reported with and without
Class: one strong model, not an architecture comparison	One architecture (24L encoder); the six runs of section 9 vary training dials only, and ft-focal is the one deployed
Notebook in the instructor's 9-step pattern	Section 5.10

---

12. Known limitations and ethics

• Defensive use only. It flags mail for a human; it is not a verdict, not a replacement for SPF/DKIM/DMARC, sender reputation, link or attachment scanning, and must not be used to tune lures against.
• Old corpora. Training mail is English, 1995-2022 with the bulk 1998-2008 (Enron, TREC, CEAS, SpamAssassin). Modern lures (MFA fatigue, QR codes, chat-app phishing) and non-English mail are out of distribution; the synthetic supplement only nudges this.
• Spam vs phishing follows provenance, not per-message review: phishing = Nazario + Nigerian fraud + synthetic lures; spam = SpamAssassin and Enron-Spam spam folders. Only 3,150 real phishing rows carry the distinction; 62,805 TREC/CEAS positives train as "unwanted" only. Trust the roll-up first.
• Length confound. Legitimate emails average about twice the tokens of unwanted ones in every split (TREC-06: 842 vs 283). Truncation at 512 tokens blunts it; evaluate.py checks it in the error analysis.
• Truncation. About 22% of emails exceed 512 tokens (~1,800 characters) and are judged on their opening.
• Test-set caveat. About 47% of TREC-06 ham is academic mailing-list traffic, which is not typical business mail.
• CPU latency. 699 ms per email on CPU vs the 74 ms benchmark; the benchmark is met on GPU only.
• Privacy. Enron mail is real correspondence released by FERC; nothing pasted into the Space is stored, but do not paste secrets or client content.

---

13. Troubleshooting

Symptom	Cause and fix
Pretraining exits 3 ("CUDA OOM" or "VRAM guard")	Another process took VRAM (browser, game, a second job) or the 20 GiB guard tripped. A checkpoint was saved. Check nvidia-smi, close the culprit, then RESUME=1 MICRO_BATCH=24 scripts/launch_pretrain.sh. The run's own peak is ~17.9 GiB reserved and the desktop adds ~2 GiB, so total device use sits near 20 GB. If it recurs, resume directly with $PY src/pretrain.py train --run-name large-24L --resume --micro-batch 24 --grad-checkpointing (8.06 GiB measured, ~25% slower). Accumulation = round(256 / micro-batch), so of the micro-batches that fit in 24 GB only 24 (x11) and 22 (x12) keep the effective batch at 264; 16 gives 256 and is refused on resume.
Resume refused, exit 2, "'effective_batch' differs"	launch_pretrain.sh defaults to MICRO_BATCH=16; pass MICRO_BATCH=24.
Resume refused, "'pretrain'/'schedule'/'data' differs"	config.yaml pretrain:/model:/seed was edited, or pretrain.parquet / tokenizer.json changed. Revert the edit; see section 6.
launch_pretrain.sh: "tmux session already exists"	The finished pane waits for Enter: tmux attach -t phish-pretrain and press Enter, or tmux kill-session -t phish-pretrain once the trainer has exited.
Fine-tune crashed or was stopped	$PY src/finetune.py train --run-name <run> --resume (restores optimizer, RNG and batch position).
Fine-tune picked the wrong pretrained model	Pass --init pretrained:checkpoints/pretrain/large-24L/final; the default is the newest */final by mtime and dev/final exists.
Install resolves a Python 3.15 interpreter	torch.compile, triton 3.8.0 and spaces all exclude 3.15. Recreate with uv venv --python 3.14.7 .venv.
Install fails on fsspec	datasets 5.1.0 caps fsspec<=2026.7.0; keep fsspec==2026.7.0 from requirements-train.txt and never upgrade it alone.
Install pulls cu132 wheels	--torch-backend=auto was used; reinstall with --torch-backend=cu130.
huggingface_hub 2.x conflict	tokenizers 0.23.2 needs <2.0; keep 1.33.0.
export.py refuses: metrics name another checkpoint	Run evaluate.py run --split test,test3 on the same best/ first.
Space build fails on torch	ZeroGPU allows torch <= 2.13.0 and Python 3.12.12; keep app/requirements.txt and the python_version front matter as they are.
Space starts but cannot load weights	HF_TOKEN secret missing or not a read token for the model repo, or PHISH_MODEL_REPO variable wrong; re-run the upload with HF_TOKEN_FOR_SPACE set, or use --bundle-weights.
Any Python command while pretraining runs	Prefix with CUDA_VISIBLE_DEVICES=""; never run smoke_test.py, a GPU fine-tune or GPU evaluation alongside it.