Refresh presentation bundle (RUN_LOG 25, README 5.11, builder --colab) and add the Colab notebook
07cdb81 verified | # BITS Phishing Detector | |
| Final project for the Skillspire AI & Machine Learning course (Nathan King, Build IT Systems LLC), presented | |
| late Oct / early Nov 2026. This README is the hand-off document for the instructor and for future BITS work. | |
| The running record of every decision, number and problem is `RUN_LOG.md` (entries 0-8 as of this | |
| writing); this file summarises it and explains how to reproduce it. Status as of 2026-10-08 14:00: data, | |
| tokenizer, smoke test and all `src/`, `scripts/` and `app/` code are done; MLM pretraining is running; the | |
| presentation notebook is in progress; fine-tuning, evaluation, export and deployment have not run yet. | |
| --- | |
| ## 1. What this is | |
| | | | | |
| |---|---| | |
| | Task | **Classifier** (not a regressor). Input: one email as `subject + "\n\n" + body` (plain text; sender, headers and attachments are never read). Output: probabilities for three classes, **Legitimate / Phishing / Spam**, plus the roll-up **Unwanted (phishing or spam)** = P(Phishing) + P(Spam). | | |
| | Use | Defensive triage: flag unwanted mail for a human to review. Not a filter that deletes mail, not a tool for writing lures. | | |
| | Model | One BERT-style pre-LayerNorm encoder written from scratch in plain PyTorch (`src/model.py`, no `transformers`, no pretrained weights): 24 layers, d_model 1024, 16 heads, FFN 4096, vocab 32,768 (own byte-level BPE), 512 tokens. **337,476,608 parameters** with the tied MLM head, **337,443,842** as the classifier. | | |
| | Training | Phase 1: masked-language-model pretraining on deduplicated Enron email. Phase 2: fine-tuning with a 3-way head. Rows whose corpus never separated spam from phishing (TREC-05/07, CEAS-08 positives) train on P(phishing)+P(spam) with a partial-label loss. | | |
| | Core challenge | Public corpora label "unwanted" mail, not "phishing vs spam"; they are 1998-2008 heavy, overlap each other and the pretraining corpus, and legitimate mail is about twice as long as spam. The work is mostly in honest data: deduplication, leakage checks and a test set nothing upstream has seen. | | |
| | Scope | **One strong model.** The experiments vary the instructor's dials (layers, epochs, loss, frozen layers, synthetic data, scratch vs pretrained init) on this one architecture; there is no architecture comparison. | | |
| ## 2. Results | |
| ### 2.1 Headline metrics (deployed run `ft-focal`, evaluated 2026-10-09 10:46) | |
| Model selection used the **validation** set only: `ft-focal` had the best validation F1 of the unwanted | |
| roll-up (0.9951) and the best validation three-way macro-F1 (0.9869) of the six runs in section 9. The two | |
| holdouts below were then evaluated once per run for reporting; `ft-focal` also leads on both of them. | |
| | TREC-06 holdout (16,302 emails, 23.9% unwanted; whole corpus unseen), threshold 0.5 | value | | |
| |---|---| | |
| | Recall, unwanted (reported first: a missed phish is the costly error) | **0.9741** (101 of 3,894 missed) | | |
| | Precision, unwanted | 0.8794 (520 false alarms on 12,408 legitimate; FPR 4.2%) | | |
| | F1, unwanted | **0.9243** (best achievable at any threshold: 0.9397 at p >= 0.68) | | |
| | Accuracy | 0.9619 | | |
| | ROC-AUC / PR-AUC | **0.9946** / 0.9848 | | |
| | Recall at 1% FPR / at 0.1% FPR | 0.8955 / 0.6497 | | |
| | Credential-lure slice (398 rows matching 37 lure phrases) recall / precision | 0.9959 / 0.9758 | | |
| TREC-06 is the hard, honest number: a different corpus (2006 TREC spam track) with its own labelling | |
| conventions. `reports/error_analysis.md` shows that many "false positives" are spam-looking mail that TREC-06 | |
| labels ham (quoted spam inside mailing-list threads, "remove me from your list" replies, sweepstakes mail) and | |
| that the largest block of misses are "ScanMail virus found" notices TREC-06 labels spam. Legitimate TREC-06 mail | |
| is also ~3x longer than its spam and 24% of it is truncated at 512 tokens. | |
| | test_holdout3 (3,220 emails: 1,999 legit / 521 phishing / 700 spam; tokenizer never saw them) | Legitimate | Phishing | Spam | | |
| |---|---|---|---| | |
| | Precision / Recall / F1 | 0.9900 / 0.9935 / 0.9918 | 0.9961 / 0.9866 / 0.9913 | 0.9785 / 0.9757 / 0.9771 | | |
| | Macro-F1 (3 classes) / accuracy | **0.9867** / 0.9885 | | | | |
| | Binary roll-up (unwanted vs legitimate): F1 / recall / precision / ROC-AUC / PR-AUC | 0.9864 / 0.9836 / 0.9893 / 0.9993 / 0.9989 | | | | |
| | Phishing vs spam, given the mail is unwanted (1,221 rows) | 0.9959 (4 phishing called spam, 1 spam called phishing) | | | | |
| 3x3 confusion matrix (rows true Legitimate / Phishing / Spam, columns predicted): `[1986, 1, 12]`, `[4, 514, 3]`, `[16, 1, 683]`. | |
| Three-way accuracy per source: Assassin 95.3% (n=236), CEAS-08 99.0% (n=736), Enron 98.4% (n=628), Nazario 98.6% (n=222), Nigerian_Fraud 98.7% (n=299), TREC-07 99.8% (n=1099). | |
| | Prediction time, one email, batch 1 (`reports/latency.json`, 100 TREC-06 samples, mean 280 tokens) | mean | p50 | p95 | vs 74 ms benchmark | | |
| |---|---|---|---|---| | |
| | GPU, bf16 (RTX 3090 Ti) | **10.1 ms** | 8.9 ms | 13.7 ms | 0.14x (faster) | | |
| | CPU, fp32, 16 threads | **459 ms** | 445 ms | 701 ms | 6.2x (slower) | | |
| Plots next to these files: `reports/confusion_matrix.png`, `confusion_matrix_test3.png`, | |
| `confusion_matrix3_test3.png` (3x3), `roc_curve*.png`, `pr_curve*.png`, `finetune_curves.png`, | |
| `pretrain_curves.png`, `eval_finetune_curves*.png`, `eval_pretrain_curves*.png`; written error analyses | |
| `error_analysis.md` / `error_analysis_test3.md`. The fine-tune experiment table is `finetune.py list` (section 9). | |
| Every run's own evaluation lives in `reports/eval_<run>/`. | |
| ### 2.2 Measured facts that exist today | |
| | Fact | Value | Source | | |
| |---|---|---| | |
| | Parameters | 337,476,608 (MLM, tied head) / 337,443,842 (classifier) | `reports/smoke_test.json` | | |
| | Pretraining VRAM, micro-batch 16, no checkpointing | 13.65 GiB reserved, 23,642 tok/s | `reports/smoke_test.json` | | |
| | Pretraining VRAM, micro-batch 24, no checkpointing (**used**) | 17.65 GiB reserved, 19.80 GiB total device incl. desktop, 25,227 tok/s | `reports/smoke_test.json` | | |
| | Pretraining VRAM, micro-batch 24, gradient checkpointing | 8.06 GiB reserved, 19,156 tok/s | `reports/smoke_test.json` | | |
| | Measured bf16 matmul | 63.9 TFLOPS (4096^3); the smoke test's 25k tok/s is ~63% MFU of the 80 TFLOPS peak | RUN_LOG 0, 3 | | |
| | Convergence check (fresh init, 30 steps) | MLM loss 10.58 -> 7.67 (ln V = 10.40) | RUN_LOG 3 | | |
| | Latency, one 512-token email, GPU bf16 (RTX 3090 Ti), untrained-weight smoke test | 10.7 ms (trained checkpoint, real emails: section 2.1) | `reports/smoke_test.json` | | |
| | Latency, one 512-token email, CPU fp32, 16 threads, untrained-weight smoke test | 699 ms (319 ms at 128 tokens) | `reports/smoke_test.json` | | |
| | Pretraining plan `large-24L` | 12 epochs x 796 steps = 9,552 optimizer steps, micro-batch 24 x 11 = effective batch 264, peak LR 1.5e-4, warmup 573, cosine to 10% | `checkpoints/pretrain/large-24L/run_summary.json` | | |
| | **Pretraining result `large-24L`** (complete 2026-10-09 03:31) | 9,552 steps, 1.29B tokens, 15.05 h of training, 23,889 tok/s mean, peak 17.94 GiB reserved; train MLM loss 1.524, **val MLM loss 1.986, masked-token accuracy 62.9%** (val loss fell monotonically, no restart needed) | `checkpoints/pretrain/large-24L/run_summary.json`, `metrics.csv` | | |
| The smoke-test latencies were measured on untrained weights (latency does not depend on the weights); the | |
| numbers to quote are the trained-checkpoint measurements in section 2.1 (`reports/latency.json`). The | |
| instructor's 0.074 s/sample benchmark is met on GPU by a factor of ~7 and missed on CPU by a factor of ~6, | |
| which is why the Space runs on ZeroGPU. | |
| ### 2.3 Data splits | |
| Binary build (RUN_LOG 1, `data_prep.py build`): | |
| | split | rows | positives | base rate | sources | | |
| |---|---|---|---|---| | |
| | finetune_train | 157,113 | 75,675 | 48.2% | TREC-05 46,386 / TREC-07 47,233 / CEAS-08 34,178 / Enron 24,223 / Assassin 5,093 | | |
| | finetune_val | 17,643 | 8,642 | 49.0% | same five, grouped by near-duplicate cluster | | |
| | test_holdout (TREC-06) | 16,302 | 3,894 | 23.9% | never seen by tokenizer, pretraining or fine-tuning | | |
| | pretrain (MLM) | 226,273 docs | - | - | corbt/enron-emails after dedup + quote stripping; 407.9M chars, 107.7M tokens | | |
| Three-way build (RUN_LOG 7, `data_prep.py build3`; label3 0 legit / 1 phishing / 2 spam / -1 unwanted-unsplit): | |
| | split | rows | legit | phishing | spam | unwanted (TREC/CEAS positives) | | |
| |---|---|---|---|---|---| | |
| | finetune_train | 160,263 | 81,438 | 3,150 | 12,870 | 62,805 | | |
| | finetune_val | 15,284 | 7,002 | 340 | 763 | 7,179 | | |
| | test_holdout (TREC-06, rows unchanged) | 16,302 | 12,408 | - | - | 3,894 | | |
| | **test_holdout3** | 3,220 | 1,999 | 521 | 700 | - | | |
| Plus `data/processed/synthetic.parquet`: 2,000 template-generated rows (1,000 legitimate, 1,000 phishing | |
| lures, 0 spam), training-only, off by default. finetune_val shrank from 17,643 to 15,284 rows when | |
| test_holdout3 was carved out, so val numbers from binary-era runs are not strictly comparable. | |
| --- | |
| ## 3. Setup on Fedora 44 | |
| Machine used: Fedora 44 KDE, RTX 3090 Ti 24 GB (sm_86), NVIDIA driver 615.71.09 (CUDA 13.4), Ryzen 9 5950X, | |
| 40 GB RAM, system Python 3.14.7, uv 0.12.5 (`~/.local/bin/uv`), tmux. | |
| ```bash | |
| cd "/home/nathank/BITS/dev/Phishing Model" | |
| uv venv --python 3.14.7 .venv | |
| uv pip install --python .venv/bin/python --torch-backend=cu130 -r requirements-train.txt | |
| hf auth login # .venv/bin/hf; needed for the upload step only | |
| ``` | |
| Why these pins: | |
| - **`--torch-backend=cu130`, never `auto`.** The plain PyPI torch 2.14.1 Linux wheel is the CUDA 13.0 build; | |
| driver 615 runs it by backward compatibility. uv 0.12.5's `auto` would pick cu132 on this driver. One | |
| resolution installs everything (180 packages, all binary wheels, venv ~6.1 GB). | |
| - **Python 3.14.7, not 3.15.** torch.compile raises on 3.15, triton 3.8.0 requires <3.15 and `spaces` requires | |
| <3.15. Always pass `--python 3.14.7`. | |
| - **`huggingface_hub==1.33.0`** because tokenizers 0.23.2 requires `huggingface-hub<2.0`. | |
| - **`fsspec==2026.7.0`** because datasets 5.1.0 caps `fsspec[http]<=2026.7.0` (2026.9.0 is newer and breaks the | |
| resolve). Do not `uv pip install -U fsspec`. | |
| Verify the GPU (do not run while a training job holds the GPU unless you accept a small extra CUDA context): | |
| ```bash | |
| nvidia-smi | |
| .venv/bin/python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.get_device_name(0), torch.cuda.is_bf16_supported())" | |
| # expected (2026-10-07): 2.14.1+cu130 13.0 NVIDIA GeForce RTX 3090 Ti True | |
| ``` | |
| --- | |
| ## 4. Project layout | |
| ``` | |
| Phishing Model/ | |
| ├── README.md this file | |
| ├── RUN_LOG.md dated log of every run, decision, number, bug and fix (the "challenges" source) | |
| ├── config.yaml every dial in one file (section 6) | |
| ├── requirements-train.txt pinned local training stack (Python 3.14.7, torch 2.14.1 cu130) | |
| ├── .gitignore excludes .venv, data, checkpoints, logs, weights, tokens | |
| ├── src/ | |
| │ ├── common.py config loading, seeding, logging, text normalisation, dedup keys (shared by all) | |
| │ ├── data_prep.py download -> dedupe -> decontaminate -> split -> leakage asserts (build/check/build3/check3) | |
| │ ├── tokenizer.py byte-level BPE (NFKC, case kept): train on pretrain + finetune_train only; stats | |
| │ ├── model.py from-scratch pre-LN encoder, MLM head, classification head (copied to app/) | |
| │ ├── smoke_test.py VRAM micro-batch finder, throughput, SDPA backend check, convergence, latency | |
| │ ├── pretrain.py resumable bf16 MLM pretraining (train/export/plot) | |
| │ ├── make_synthetic.py 2,000 template-generated training rows (build/show) | |
| │ ├── finetune.py 3-way classifier fine-tuning with the instructor's dials (train/plot/list) | |
| │ ├── evaluate.py the only reader of the holdouts: metrics, latency, plots, error analysis | |
| │ └── export.py bf16 safetensors + config + model card -> model_export/<run>/ and app/ | |
| ├── scripts/ | |
| │ ├── launch_pretrain.sh start or resume pretraining in tmux session `phish-pretrain` | |
| │ ├── run_experiments.sh fine-tune queue: waits for large-24L/final, runs the six dial experiments, | |
| │ │ evaluates each into reports/eval_<run>/, exports ft-ce to model_export/ft-ce and app/ | |
| │ └── upload_to_hf.py publish model repo + Space; dry run by default, `--yes` to write | |
| ├── app/ the Hugging Face Space | |
| │ ├── app.py Gradio UI: 3-class label, prediction time, token saliency, rule-based cues | |
| │ ├── README.md Space front matter (sdk gradio 6.29.1, python 3.12.12) + description | |
| │ ├── requirements.txt Space pins (torch 2.13.0, spaces 0.51.3, ...) | |
| │ ├── .gitattributes LFS rules for weights and tokenizer | |
| │ └── model.py, model.safetensors, config.json, tokenizer.json written by export.py (not yet present) | |
| ├── FinalProject_PhishingDetector.ipynb presentation notebook in the instructor's 9-step pattern (section 5.10); | |
| │ written by the notebook step, in progress as of 2026-10-08 14:00 | |
| ├── reference/ | |
| │ ├── FashionMNIST_Gradio_Full.ipynb instructor's example notebook (the 9-step pattern) | |
| │ └── INSTRUCTOR_REQUIREMENTS.md grading rubric, verbatim | |
| ├── docs/research/ | |
| │ ├── research_dataset-schemas.md schemas, provenance and licences of every candidate dataset | |
| │ ├── verify_data.md independent verification of the data plan and holdout choice | |
| │ ├── research_model-math.md parameter / memory / FLOP / latency math for the encoder | |
| │ ├── model_math.py pure-Python sizing calculator behind that document | |
| │ ├── research_torch-python.md torch 2.14.1 / CUDA 13.0 / Python 3.14 compatibility, SDPA backends | |
| │ ├── research_other-libs.md every other pin and its conflicts (fsspec, huggingface_hub, spaces) | |
| │ ├── verify_pins.md independent verification of the pins | |
| │ ├── research_gradio-spaces-hub.md Gradio 6, ZeroGPU, Space front matter, Hub upload API | |
| │ └── bigshp_research.md why the 300k "spam/ham/phish" set was rejected | |
| ├── data/ (git-ignored) | |
| │ ├── raw/ HF cache for corbt, Champa mirror CSVs (aura/), inspected-only 300k set (bigshp/) | |
| │ └── processed/ | |
| │ ├── finetune_train.parquet, finetune_val.parquet training / model-selection splits (with label3) | |
| │ ├── test_holdout.parquet TREC-06 binary test set | |
| │ ├── test_holdout3.parquet clean-provenance three-way test set | |
| │ ├── pretrain.parquet MLM corpus (226,273 docs) | |
| │ ├── pretrain_tokens.npy/.json, pretrain_val_tokens.npy cached token stream used by pretrain.py | |
| │ ├── synthetic.parquet, synthetic_report.json template supplement + shortcut audit | |
| │ ├── tokenizer.json, tokenizer_report.json BPE vocabulary + per-split length stats | |
| │ ├── splits_manifest.json, data_report.md counts, removals, every leakage assertion | |
| │ └── binary_snapshot/ binary-era splits that build3 starts from | |
| ├── checkpoints/ (git-ignored) | |
| │ ├── pretrain/large-24L/ live run: step-NNNNNNN/, metrics.csv, run_summary.json, final/ when done | |
| │ ├── pretrain/dev/ tiny self-test run (delete after Nathan's OK) | |
| │ └── finetune/<run>/ best/, last/, metrics.csv, epochs.csv per fine-tune run (dev, dev-focal = self-tests) | |
| ├── reports/ smoke_test.json; experiments.jsonl + finetune_curves.png + pretrain_curves.png are | |
| │ dev-only until the real runs regenerate them (section 9); metrics*, latency*, | |
| │ confusion/ROC/PR plots, error analyses, eval_<run>/ later | |
| ├── model_export/<run>/ export output (created by export.py) | |
| └── logs/ one timestamped log per script invocation + tmux stdout of the pretraining run | |
| ``` | |
| --- | |
| ## 5. How to run each phase, in order | |
| All commands run from the project root with `.venv/bin/python` (written `PY` below: | |
| `PY=.venv/bin/python`). Every script takes `--config config.yaml` before the subcommand and writes a log to | |
| `logs/<script>-.log`. Flags below were checked against each script's `--help` on 2026-10-08. | |
| **While pretraining holds the GPU, prefix any other Python command with `CUDA_VISIBLE_DEVICES=""` and do not | |
| start smoke_test, finetune, evaluate on GPU, or a second training job.** | |
| ### 5.1 Data (`src/data_prep.py`) | |
| ```bash | |
| $PY src/data_prep.py inspect # optional: schema + 2 rows per dataset, streaming | |
| $PY src/data_prep.py build # binary splits + pretrain corpus + manifest + data_report.md (~2.5 min, 6.9 GB RSS) | |
| $PY src/data_prep.py check # recompute every binary leakage assertion from the written files | |
| $PY src/data_prep.py build3 # three-way labels, Champa phishing corpora, test_holdout3 (~1 min) | |
| $PY src/data_prep.py check3 # recompute binary + three-way assertions | |
| ``` | |
| Flags: `build --limit N` / `build3 --limit N` are dev-only subsets (`build3 --limit` writes to | |
| `data/processed/dev_threeway_limitN/`); `build --spam-policy {positive_class,map_to_zero,exclude}` (default | |
| `positive_class`); `build --no-strip-quotes`; `--shingle-cap N`; `build3 --bigshp {on,off}` overrides the | |
| rejected 300k set (leave off); `check3 --out-dir`. | |
| **Do not re-run `build` or `tokenizer.py train` while `large-24L` is running**: a new `pretrain.parquet` or | |
| `tokenizer.json` rebuilds the token cache and the run then refuses to resume. `build` also rewrites the | |
| binary splits, so `build3` must follow it. | |
| ### 5.2 Tokenizer (`src/tokenizer.py`) | |
| ```bash | |
| $PY src/tokenizer.py train # byte-level BPE, vocab 32,768, NFKC, case kept; ~30 s | |
| $PY src/tokenizer.py stats # length / truncation / [UNK] table per split -> tokenizer_report.json | |
| ``` | |
| `train --vocab-size N`, `train --limit-docs N` (dev). Trains only on `pretrain.parquet` + `finetune_train.parquet`; | |
| val and test files are refused by name and checked with an exact-hash tripwire. | |
| ### 5.3 Smoke test (`src/smoke_test.py`) — GPU must be free | |
| ```bash | |
| $PY src/smoke_test.py # -> reports/smoke_test.json + printed summary | |
| ``` | |
| Useful flags: `--candidates 16,24,12,8`, `--ckpt-candidates`, `--steps 30`, `--skip-cpu`, `--n-docs 3000`, | |
| `--latency-samples 100`, `--cpu-threads 16`. It probes micro-batches with the real gradient-accumulation | |
| pattern, measures tokens/s, checks which SDPA kernel runs, runs a 30-step convergence check and times | |
| single-email inference on GPU and CPU. | |
| ### 5.4 MLM pretraining (`scripts/launch_pretrain.sh` -> `src/pretrain.py`) | |
| ```bash | |
| EPOCHS=12 MICRO_BATCH=24 scripts/launch_pretrain.sh # how large-24L was launched (2026-10-08 12:28) | |
| tmux attach -t phish-pretrain # watch; Ctrl-b d detaches; Ctrl-C inside = save + clean exit | |
| RESUME=1 MICRO_BATCH=24 scripts/launch_pretrain.sh # resume from the latest checkpoint | |
| ``` | |
| Environment variables: `RUN_NAME` (default `large-24L`), `EPOCHS` (script default 6; fixes the schedule | |
| length at first launch, ignored on resume), `MICRO_BATCH` (**script default 16; always pass 24 for this run**: | |
| 16 gives effective batch 256 instead of the run's 264 and the resume is refused), `SESSION` (default | |
| `phish-pretrain`), `RESUME=1`. The script refuses to start if the tmux session already exists; after the | |
| trainer exits the pane waits for Enter (or `tmux kill-session -t phish-pretrain`). Stdout is teed to | |
| `logs/pretrain_<run>_<timestamp>.out`. | |
| Direct use: `$PY src/pretrain.py train --run-name R [--resume [PATH|latest]] [--epochs N] [--max-steps N] | |
| [--max-hours H] [--micro-batch B] [--grad-checkpointing] [--limit-docs N] [--checkpoint-every N] | |
| [--log-every N] [--eval-every N] [--keep-checkpoints K] [--fresh]`. After the run: | |
| ```bash | |
| $PY src/pretrain.py plot --run-name large-24L # -> reports/pretrain_curves.png | |
| $PY src/pretrain.py export --run-name large-24L # only if stopped early: builds final/ from the latest checkpoint | |
| ``` | |
| How it stops (every path writes a checkpoint first; checkpoints are atomic tmp-dir + rename): | |
| | Event | Behaviour | Exit code | | |
| |---|---|---| | |
| | Normal end | `final/` written | 0 | | |
| | Ctrl-C / SIGTERM | save at the next optimizer-step boundary | 0 | | |
| | `--max-hours` reached | save | 0 | | |
| | **VRAM guard**: this process's reserved peak > `vram.hard_cap_gb` (20 GiB) | save, stop before a real OOM | 3 | | |
| | **CUDA OOM** (e.g. the desktop grabs VRAM) in forward/backward, optimizer step or eval | partial step discarded, data cursor rewound, save | 3 | | |
| | 5 consecutive non-finite losses (each such step is skipped) | status `diverged`, save | 4 | | |
| | Config/plan mismatch on resume (model shape, `pretrain:` LR/betas/eps/decay/clip/masking/schedule, seed, effective batch, epochs, data) | refuses before training | 2 | | |
| Checkpoints every 500 steps and at every epoch end (keep 3, ~3.8 GB each) in | |
| `checkpoints/pretrain/large-24L/step-NNNNNNN/`; val MLM loss and accuracy on 64 fixed held-out windows every | |
| 500 steps in `metrics.csv`. If val MLM loss rises, stop the run (Ctrl-C) and use `pretrain.py export`. | |
| ### 5.5 Synthetic supplement (`src/make_synthetic.py`) — already built | |
| ```bash | |
| $PY src/make_synthetic.py build [--n 2000] [--out PATH] [--seed S] # -> data/processed/synthetic.parquet + _report.json | |
| $PY src/make_synthetic.py show --k 6 | |
| ``` | |
| ### 5.6 Fine-tuning (`src/finetune.py`) — after pretraining, GPU free | |
| Baseline (config defaults: 3 classes, cross-entropy over the partial-label NLL, 3 epochs, batch 32 = micro-batch | |
| 16 x 2, LR 3e-5, early stopping on val F1 of the unwanted roll-up, patience 2): | |
| ```bash | |
| $PY src/finetune.py train --run-name ft-ce --init pretrained:checkpoints/pretrain/large-24L/final | |
| ``` | |
| `ft-ce` is the cross-entropy baseline. The queue ran all six runs of section 9 on 2026-10-09 (03:36-10:02); | |
| **`ft-focal` won on the validation selection metric** (val F1 of the unwanted roll-up 0.9951 vs 0.9929 for | |
| `ft-ce`, val three-way macro-F1 0.9869 vs 0.9782) and is therefore the run evaluated at top level, exported | |
| and deployed. Every example in sections 5.7-5.9 uses it. | |
| **The queue (`scripts/run_experiments.sh`)** is how the experiments are meant to run: it waits for | |
| `checkpoints/pretrain/large-24L/final/model.safetensors`, renders `pretrain.py plot`, then trains the six runs | |
| of section 9 one after another with `--eval-every 1250`, plots each to `reports/finetune_curves_<run>.png`, | |
| evaluates each `best/` on both holdouts into `reports/eval_<run>/` (marker `reports/eval_done_<run>`), and | |
| finally evaluates + exports the main run (`MAIN_RUN`, default `ft-ce`) into the **top-level** | |
| `reports/metrics.json` / `metrics_test3.json` that section 2.1 quotes, `model_export/<MAIN_RUN>/` and `app/`. | |
| The top-level evaluate + export were re-run by hand for `ft-focal` after the queue finished (section 9). | |
| A run whose `best/` exists is skipped, so re-running the script continues after a crash. | |
| ```bash | |
| tmux new-session -d -s phish-finetune 'scripts/run_experiments.sh 2>&1 | tee logs/experiments_$(date +%Y%m%d-%H%M%S).out' | |
| ``` | |
| Environment: `PRETRAIN_RUN` (default `large-24L`), `EVAL_EVERY` (default 1250), `MAIN_RUN` (default `ft-ce`). | |
| The instructor's dials (all flags of `train`): | |
| | Dial | Flag | Values | | |
| |---|---|---| | |
| | Loss function | `--loss` | `cross_entropy` / `weighted_cross_entropy` (inverse-frequency) / `focal` (`--focal-gamma 2.0`) | | |
| | Epochs | `--epochs` | e.g. 1-5; early stopping may end sooner | | |
| | Number of layers | `--n-layers N` | keep the first N pretrained layers (e.g. 12 of 24) | | |
| | Frozen layers | `--freeze-layers K` | freeze embeddings + first K layers | | |
| | Synthetic data | `--synthetic on\|off` | appends the 2,000-row supplement to train only | | |
| | Init | `--init scratch` / `--init pretrained:<dir>` | scratch = ablation showing what pretraining buys | | |
| | Others | `--lr`, `--batch-size`, `--micro-batch`, `--num-labels 2\|3`, `--seed`, `--eval-every`, `--log-every`, `--eval-batch-size` | | | |
| | Dev only | `--limit-rows N`, `--max-steps N`, `--cpu` | not evidence | | |
| Always pass `--init` explicitly: without it the script takes the newest `checkpoints/pretrain/*/final` by | |
| modification time, and `checkpoints/pretrain/dev/final` (a tiny self-test model) exists. | |
| ```bash | |
| $PY src/finetune.py train --run-name ft-focal --resume # continue /last after a crash or Ctrl-C | |
| $PY src/finetune.py train --run-name ft-focal --overwrite # discard an existing run dir of that name | |
| $PY src/finetune.py list # experiments.jsonl as a markdown table | |
| $PY src/finetune.py plot --run-name ft-focal # -> reports/finetune_curves.png (`--out PATH` to rename) | |
| ``` | |
| On resume, `--init/--n-layers/--freeze-layers/--batch-size/--seed/--num-labels` are refused; `--lr` is | |
| re-applied to the restored optimizer. Output: `checkpoints/finetune//{best,last}/`, `metrics.csv`, | |
| `epochs.csv`. The test files are refused by name and by `source` column. | |
| ### 5.7 Evaluation (`src/evaluate.py`) — the only script that opens the holdouts | |
| ```bash | |
| $PY src/evaluate.py run --checkpoint checkpoints/finetune/ft-focal/best --split test,test3 | |
| $PY src/evaluate.py run --checkpoint checkpoints/finetune/ft-focal/best --split val # model selection view, `_val` suffix | |
| ``` | |
| The queue runs this per experiment with `--run-name <run> --out-dir reports/eval_<run>`; only the main run is | |
| written to the top-level `reports/` (section 2.1). | |
| Writes `reports/metrics.json`, `metrics_test3.json`, `latency.json` (batch-1 `timeit.default_timer` around | |
| tokenize + forward + softmax, GPU bf16 and CPU fp32, next to the 74 ms benchmark), `confusion_matrix*.png`, | |
| `confusion_matrix3_test3.png`, `roc_curve*.png`, `pr_curve*.png`, `error_analysis*.md` (top false negatives / | |
| positives, phishing<->spam confusions, length-confound check, credential-lure slice), one line in | |
| `reports/experiments.jsonl` and one RUN_LOG.md entry. Flags: `--threshold`, `--batch-size`, | |
| `--latency-samples`, `--cpu-threads`, `--max-len`, `--cpu`, `--out-dir`, `--data-dir`, `--run-name`, | |
| `--limit-rows` (dev; no RUN_LOG entry), `--no-run-log`, `--finetune-run-dir` / `--pretrain-run-dir` (extra | |
| `eval_*_curves.png`). The binary metrics use P(unwanted) at 0.5; the three-way decision is roll-up first, | |
| then argmax(phishing, spam), so the 3x3 matrix collapses exactly onto the binary one (plain argmax-of-3 is | |
| reported alongside). | |
| ### 5.8 Export (`src/export.py`) | |
| ```bash | |
| $PY src/export.py run --checkpoint checkpoints/finetune/ft-focal/best | |
| ``` | |
| Writes `model_export/ft-focal/` (`model.safetensors` bf16 ~675 MB, `config.json` with label names, metrics and | |
| dials, `tokenizer.json`, `README.md` model card, `export_manifest.json`) and copies `model.py` (sha1-checked) | |
| plus the three weight files into `app/`. It refuses if `reports/metrics*.json` name a different checkpoint, so | |
| run 5.7 on the same `best/` first (`--allow-metrics-mismatch` overrides and records it). Other flags: | |
| `--out-dir`, `--app-dir`, `--dtype {bf16,fp32}`, `--metrics-json`, `--metrics3-json`, `--pretrain-config`. | |
| Local app check before uploading: `CUDA_VISIBLE_DEVICES="" $PY app/app.py` (CPU; drop the prefix once the GPU | |
| is free: `app.py` moves the model to CUDA at import time whenever a GPU is visible). | |
| ### 5.9 Upload (`scripts/upload_to_hf.py`) — Nathan only | |
| ```bash | |
| $PY scripts/upload_to_hf.py --model-dir model_export/ft-focal --bundle-weights --check-auth # dry run (default): prints the plan, no writes | |
| $PY scripts/upload_to_hf.py --model-dir model_export/ft-focal --bundle-weights --yes # live (what was run on 2026-10-12) | |
| ``` | |
| The dry run lists every file, size, repo, visibility, the hardware flavour and any blocking problem (a missing | |
| export, a missing `app/model.py`, or `HF_TOKEN_FOR_SPACE` unset when the weights are not bundled). Other flags: | |
| `--bundle-weights` (weights inside the Space, no secret needed; the mode used), `--skip-secret`, `--hardware`, | |
| `--model-repo`, `--space-repo`, `--private/--no-private`, `--commit-message`. `--yes` is only ever run with | |
| Nathan's explicit approval; the one live run happened on 2026-10-12 11:35 (RUN_LOG 24) after his approval of | |
| 2026-10-08. Lesson from that run: the Hub validates the Space card's YAML front matter at upload time and the | |
| dry run does not, so a colon inside an unquoted `short_description` failed the first attempt; the value is now | |
| quoted and the model-repo half of the first attempt was simply reused (LFS blobs dedupe by content hash). | |
| ### 5.10 The notebook | |
| `FinalProject_PhishingDetector.ipynb` (project root; generated by `scripts/build_notebook.py`, last executed top | |
| to bottom on the CPU on 2026-10-12: 36 code cells, no errors, every artifact present) is the presentation | |
| walk-through. It mirrors the instructor's | |
| `reference/FashionMNIST_Gradio_Full.ipynb` step for step, with the nine headers quoted verbatim: "Step 1: Install | |
| Dependencies & Setup Device", "Step 2: Download Data & Create DataLoaders", "Step 3: Define the Neural Network", | |
| "Step 4: Train the Model", "Step 5: Test Inference Speed and Save the Model", "Step 6: Create the App Files | |
| (Optional)", "Step 7: Launch the Gradio Interface", "Step 8: Authenticate with Hugging Face", "Step 9: Upload | |
| the App to Hugging Face Spaces". It **loads the artifacts above instead of training**: the contract is that it | |
| runs top to bottom on CPU and prints "not available yet — produced by <command>" for anything missing; with the | |
| 2026-10-09 artifacts in place it shows the real numbers, curves, confusion matrices and the `ft-focal` export. | |
| ```bash | |
| CUDA_VISIBLE_DEVICES="" .venv/bin/jupyter lab | |
| CUDA_VISIBLE_DEVICES="" .venv/bin/jupyter nbconvert --to notebook --execute --inplace FinalProject_PhishingDetector.ipynb | |
| ``` | |
| ### 5.11 The Colab presentation notebook (`FinalProject_PhishingDetector_Colab.ipynb`) | |
| Built by the same script with `--colab` (`--out PATH` picks the output file; the default mode and its output are | |
| unchanged). Design: **the notebook is narrative, the Space is the compute.** Nothing trains or infers in Colab; | |
| every prediction runs on the ZeroGPU Space, and all evidence is pulled from the Hub: | |
| | Need | Where it comes from in Colab | | |
| |---|---| | |
| | Metrics, curves, confusion matrices, experiments table, data report, run log, code | `presentation/` in the model repo (85 files, 3.6 MB; relative paths mirror this project), fetched with `snapshot_download` into `/content/phishing_bundle` | | |
| | Model definition and parameter count (Step 3) | `presentation/src/model.py` imported on the Colab CPU (torch is preinstalled there); falls back to the counts in `config.json` without torch | | |
| | Inference speed (Step 5) | `gradio_client.Client(SPACE, token=...).predict(subject, body, api_name="/predict")` on six synthetic emails: in-app "Prediction Time (s)" measured on the Space's A10G (about 0.23 s cold, 0.01 s warm) plus the round-trip; `latency.json` adds the RTX 3090 Ti numbers | | |
| | The app (Step 7) | `gr.load(SPACE, src="spaces", token=...)` embeds the Space UI; the private model repo's weights never leave the Hub. Gradio 6.29.1 has a bug rebuilding a Space whose Interface has `examples=` (`Textbox.__init__() got an unexpected keyword argument 'class_id'`), so the cell installs a two-line compatibility shim and retries, and falls back to a hand-built proxy `gr.Interface` over `gradio_client` if that also fails. When the Space is public the cell also shows the Space's own page in an `IFrame` of https://builditsystems-phishing-detector-demo.hf.space | | |
| | Auth (Step 8) | Token resolution order: Colab secret `HF_TOKEN` -> `os.environ["HF_TOKEN"]` -> the `hf auth login` cache (outside Colab only when `NOTEBOOK_ALLOW_LOGIN_CACHE=1`, so a headless run never silently forwards the workstation's write token) -> none (enough once the repos are public). Only the token *source* is ever printed | | |
| How Nathan runs it: upload the `.ipynb` to Colab (File -> Upload notebook, or from Drive); while the repos are | |
| private, add a Colab Secret named `HF_TOKEN` with notebook access on, and make it a **fine-grained read token | |
| scoped to the two repos**, because `gr.load` forwards the token to the Space with every proxied call; run all | |
| cells. The first cell installs `gradio==6.29.1 huggingface_hub==1.33.0` (only when it detects Colab). Once both | |
| repos are public no token is needed and the iframe appears automatically. | |
| Verified on 2026-10-12: executed headlessly from an empty directory in a clean Python 3.12 venv (torch CPU, | |
| gradio 6.29.1, huggingface_hub 1.33.0) with no project file reachable: 36/36 code cells ran, zero errors, six | |
| live predictions from the Space, `gr.load` succeeded after the shim retry, the account name printed and no | |
| token-shaped string in any output; the numbers it renders were spot-checked against `reports/metrics*.json` | |
| (RUN_LOG 25). The Colab-only branches (`%pip`, `userdata.get`, inline `launch()`) are exercised by the | |
| `IN_COLAB` switch but were not run on a real Colab runtime; do one rehearsal there. | |
| --- | |
| ## 6. Config dials (`config.yaml`) | |
| | Section | What it controls | | |
| |---|---| | |
| | `seed`, `paths` | global seed 42; every relative path is resolved against the project root | | |
| | `model` | encoder shape (24 / 1024 / 16 / 4096 / 32768 / 512), dropout, pre-LN, tied MLM head, gradient checkpointing | | |
| | `tokenizer` | byte-level BPE, vocab 32,768, min frequency 2, specials `[PAD] [UNK] [CLS] [SEP] [MASK]`, training corpora | | |
| | `data` | sources, exclusions, holdout source (TREC-06), alnum minimums, dedup keys and shingle parameters, val fraction, `synthetic`, `threeway` (label3 by provenance, Champa mirror, the rejected `bigshp` set with its verdict, holdout3 carving) | | |
| | `vram` | 14-18 GiB target band, 20 GiB hard cap (GiB, compared with `torch.cuda.max_memory_reserved()`), allocator config | | |
| | `pretrain` | bf16, packing, effective batch 256, micro-batch start 24 + candidates, 12 epochs, AdamW LR 1.5e-4 / betas (0.9, 0.98) / eps 1e-6 / decay 0.01 / clip 1.0, warmup, cosine to 10%, 15% masking 80/10/10, checkpoint/log/eval cadence | | |
| | `finetune` | `num_labels: 3`, epochs, batch / micro-batch, LR 3e-5, warmup + linear decay, loss and class weights, focal gamma, early stopping metric + patience, frozen layers, padding multiple | | |
| | `eval` | metric list, credential-lure slice, latency samples, threshold 0.5, test3 file, label contract (asserted against the code), top-k confusions, model-card metrics source | | |
| | `hf` | owner, model and Space repo ids, private flag, Gradio 6.29.1, Python 3.12.12, `zero-a10g`, Space torch 2.13.0, safetensors bf16 | | |
| **Rule during a live pretraining run:** do not edit `model:` or the `pretrain:` keys that shape the plan | |
| (`lr`, `betas`, `eps`, `weight_decay`, `grad_clip`, `mlm_probability`, `mask_replace_probs`, `schedule`, | |
| `effective_batch_sequences`, `warmup_steps`, `warmup_fraction`), `seed`, or anything that changes | |
| `pretrain.parquet` / `tokenizer.json`. Each is stored in the checkpoint and a resume refuses a mismatch. This is | |
| why `num_labels` lives under `finetune:`. `finetune:`, `eval:` and `hf:` can be edited at any time. | |
| --- | |
| ## 7. Data provenance and licences | |
| | Source | Used for | Licence / terms | Notes | | |
| |---|---|---|---| | |
| | `puyang2025/seven-phishing-email-datasets` @ b7f3f8d9 (203,017 rows) | labelled legit + unwanted: TREC-05/06/07, CEAS-08, Enron-Spam, SpamAssassin | Hub card "other"; upstream Champa, Rabbi & Zibran, "Seven Phishing Email Datasets", figshare 25432108, CC BY 4.0 | Its own train/test split leaks (5.1% exact duplicates) and is ignored. Ling (2,859 academic list rows) excluded. Label 1 = "unwanted": these are spam-track corpora with no phishing/spam distinction. | | |
| | `corbt/enron-emails` @ cfc06c75 (517,401 rows) | MLM pretraining only | CMU Enron corpus (FERC public release); no formal licence; CMU asks users to respect privacy | 52.7% exact duplicates removed, quoted/forwarded history stripped -> 226,273 docs. | | |
| | `kudzaiprichard/aura-phishing-email-corpus` @ 539567ce: `Nazario.csv`, `Nigerian_Fraud.csv` | phishing class (1,507 + 2,504 rows kept) | CC BY 4.0; byte-identical mirror of Champa, Rabbi & Zibran, "Phishing Email Curated Datasets", Zenodo, doi:10.5281/zenodo.8339691 | Nazario phishing archive and advance-fee (419) fraud, 1995-2022. 235 Nigerian rows near-duplicating TREC-06 spam and 13 exact matches removed to keep the test set clean. | | |
| | `data/processed/synthetic.parquet` (2,000 rows) | optional training supplement | generated by `make_synthetic.py`; fictional names, `.example`/`.invalid` domains, RFC 5737 IPs, 555-01xx phones | 8 BEC / Microsoft 365 / invoice lure categories vs 8 ordinary business categories; shared filler pools and a token-purity audit against template shortcuts. Never in val or test, never presented as real mail. | | |
| | Rejected: `locuoco/the-biggest-spam-ham-phish-email-dataset-300000` | nothing | MIT (Kaggle re-upload) | 365,448 rows, 51.5% internal duplicates, a union of corpora we already hold; its "phish" label agrees with provenance only 60.5%; **36,020 rows are copies of our TREC-06 test set**; phish rows went through a stopword-stripping pipeline the others did not (a label-proxy artifact). Not one row ingested. `docs/research/bigshp_research.md`. | | |
| | Rejected: `it4lia/PhishingEmailCuratedDatasets_Cleaned` | nothing | CC BY 4.0 | no body text (subjects/headers + 53 numeric features). | | |
| | Rejected: `zefang-liu/phishing-email-dataset` | nothing | LGPL-3.0 | lowercased re-packaging, 91.9% contained in the primary set, "enron" token shortcut. | | |
| Details and evidence: `docs/research/research_dataset-schemas.md`, `docs/research/verify_data.md`, | |
| `data/processed/data_report.md`. The exported model card states that phishing/spam labels follow corpus | |
| provenance. | |
| --- | |
| ## 8. Leakage policy | |
| - **TREC-06 is the binary test set and nothing upstream has seen it.** It is excluded from tokenizer training, | |
| the MLM corpus and fine-tuning. It is Enron-clean (5 rows mention "enron", 0 enron.com addresses); TREC-05 ham | |
| and Enron-Spam ham are 86-89% inside the pretraining corpus, so neither could be the test set. Model input is | |
| subject + body only because TREC rewrote the spam headers. | |
| - **Decontamination.** Exact keys on normalised text (full, 300-char prefix, subject+body prefix, subject+body | |
| full) plus near-duplicates (>= 3 shared 50-char shingles over the first 1,500 chars). 1,652 training rows that | |
| near-duplicate TREC-06 templates were removed; 85 pretraining docs near-duplicating holdout emails and 4,004 | |
| matching val keys were removed from the MLM corpus. | |
| - **Train/val split** is grouped by near-duplicate cluster and stratified by source x label, so no template | |
| straddles train and val. | |
| - **test_holdout3** rules: legitimate rows only from CEAS-08, TREC-07, SpamAssassin (never Enron, which overlaps | |
| pretraining); spam from SpamAssassin and Enron-Spam spam folders; phishing from Nazario + Nigerian_Fraud. | |
| Rows come from finetune_val (never seen by the tokenizer) or the new sources (never seen by anything); whole | |
| clusters move or none; any cluster with a key or shingle link to the pretraining corpus or train is excluded. | |
| - **Asserted zeros** (recomputed independently from the written files by `check` / `check3`; a non-zero count | |
| fails the build): holdout vs train / val / pretrain keys and shingle links; train vs val full-key rows; val vs | |
| pretrain keys; holdout source rows in train/val; holdout3 vs train / val / pretrain keys and shingle links; | |
| holdout3 vs TREC-06 keys; doc_id shared between splits; label3 contract violations. All are 0 | |
| (`data/processed/data_report.md`). Counted but allowed: 5,336 val-to-pretrain near-dup links (val is only | |
| used for model selection), 102,100 train-to-pretrain key overlaps (Enron ham). | |
| - **Code guards.** The tokenizer refuses val/test files by name with an exact-hash tripwire (0 collisions); | |
| `finetune.py` refuses the holdout by file name and `source` column; `pretrain.py` never opens it; | |
| `evaluate.py` is the only reader. | |
| --- | |
| ## 9. Experimentation record | |
| - **`reports/experiments.jsonl`**: `finetune.py train` appends one JSON line per run (dials: init, layers, | |
| frozen layers, epochs planned/run, loss, focal gamma, LR, batch, class weights, synthetic rows, seed, | |
| selection metric; parameter counts; steps, seconds, peak VRAM; best step; val metrics) and `evaluate.py` | |
| appends one line per invocation with the test metrics. `finetune.py list` renders it as the markdown table for | |
| the presentation. Generated from evidence, never typed up afterwards. | |
| - **Contents**: the six real runs below (2026-10-09) plus one evaluate line per holdout evaluation. The dev | |
| self-test rows from 2026-10-08 were deleted before the queue ran (RUN_LOG 9). `reports/finetune_curves.png` is | |
| the `ft-focal` curve, `reports/finetune_curves_.png` the others, `reports/pretrain_curves.png` the | |
| `large-24L` curve. | |
| - **Pretraining** is one run (`large-24L`); its curve is `metrics.csv` -> `pretrain.py plot`. | |
| - **Fine-tune queue** (`scripts/run_experiments.sh`, section 5.6; one architecture, one dial changed per run | |
| against the baseline). It ran unattended on 2026-10-09 from 03:36 to 10:02 (six runs, each evaluated on both | |
| holdouts). The baseline `ft-ce` was allowed 3 epochs; the five ablations 2 epochs each for run-time reasons; | |
| early stopping (patience 2 on the validation unwanted-F1, evaluated every 1,250 steps) ended five of the six | |
| before their epoch budget. All start from `checkpoints/pretrain/large-24L/final` except `ft-scratch`. | |
| | Run | Flags | Epochs | Question it answers | | |
| |---|---|---|---| | |
| | ft-ce | `--loss cross_entropy` | 3 | reference point: pretrained 24L, cross-entropy, synthetic off | | |
| | ft-weighted | `--loss weighted_cross_entropy` | 2 | class imbalance (3,150 phishing vs 81,438 legit fully-labelled rows) | | |
| | ft-focal | `--loss focal` (gamma 2.0 from config) | 2 | hard-example weighting | | |
| | ft-synthetic | `--loss cross_entropy --synthetic on` | 2 | does the BEC/M365 supplement move anything (reported even if it does not) | | |
| | ft-12L | `--loss cross_entropy --n-layers 12` | 2 | number of layers (half the stack, ~2x faster inference) | | |
| | ft-scratch | `--loss cross_entropy --init scratch` | 2 | what MLM pretraining buys | | |
| ### 9.1 Results (validation = selection; holdouts = reporting) | |
| Selection view (validation split, 15,284 rows; the queue's early-stopping metric is the unwanted-F1 column): | |
| | Run | Loss / dial | Epochs run / planned | Stopped by | Best step | Train min | Peak GiB | Val F1 unwanted (selection) | Val macro-F1 3-way | Val P / R phishing | | |
| |---|---|---|---|---|---|---|---|---|---| | |
| | ft-ce | cross_entropy | 1.25 / 3 | early_stopping | 5000 | 51 | 13.13 | 0.9929 | 0.9782 | 0.939 / 0.997 | | |
| | ft-weighted | weighted_cross_entropy | 2.00 / 2 | epochs_done | 10000 | 84 | 13.13 | 0.9937 | 0.9775 | 0.963 / 0.997 | | |
| | **ft-focal** | focal (gamma 2.0) | 2.00 / 2 | early_stopping | 8750 | 84 | 13.13 | **0.9951** | 0.9869 | 0.985 / 0.988 | | |
| | ft-synthetic | cross_entropy, synthetic on | 1.23 / 2 | early_stopping | 5000 | 52 | 13.13 | 0.9927 | 0.9665 | 0.881 / 0.997 | | |
| | ft-12L | cross_entropy, 12 layers | 1.75 / 2 | early_stopping | 6250 | 35 | 6.97 | 0.9925 | 0.9774 | 0.957 / 0.982 | | |
| | ft-scratch | cross_entropy, scratch init | 1.50 / 2 | early_stopping | 5009 | 61 | 13.13 | 0.9875 | 0.9538 | 0.938 / 0.941 | | |
| Held-out view (TREC-06: 16,302 rows, whole corpus unseen; test_holdout3: 3,220 rows, 1,999 / 521 / 700). Evaluated once per run after training; not used for selection: | |
| | Run | TREC-06 recall | precision | F1 | FPR | ROC-AUC | PR-AUC | recall @ 1% FPR | test3 F1 unwanted | test3 macro-F1 | F1 phishing | F1 spam | | |
| |---|---|---|---|---|---|---|---|---|---|---|---| | |
| | ft-ce | 0.9404 | 0.8610 | 0.8990 | 4.8% | 0.9875 | 0.9626 | 0.7745 | 0.9865 | 0.9822 | 0.9769 | 0.9779 | | |
| | ft-weighted | 0.9818 | 0.8307 | 0.9000 | 6.3% | 0.9943 | 0.9833 | 0.8767 | 0.9862 | 0.9852 | 0.9875 | 0.9767 | | |
| | **ft-focal** | 0.9741 | 0.8794 | **0.9243** | 4.2% | 0.9946 | 0.9848 | 0.8955 | 0.9864 | **0.9867** | 0.9913 | 0.9771 | | |
| | ft-synthetic | 0.9679 | 0.8402 | 0.8995 | 5.8% | 0.9908 | 0.9716 | 0.7871 | 0.9873 | 0.9814 | 0.9772 | 0.9749 | | |
| | ft-12L | 0.9461 | 0.8120 | 0.8739 | 6.9% | 0.9836 | 0.9513 | 0.6608 | 0.9823 | 0.9807 | 0.9807 | 0.9721 | | |
| | ft-scratch | 0.9165 | 0.7752 | 0.8400 | 8.3% | 0.9755 | 0.9456 | 0.7773 | 0.9767 | 0.9634 | 0.9574 | 0.9470 | | |
| What the dials say: focal loss is the clear winner (fewer misses on TREC-06 at a lower false-alarm rate, and | |
| the best phishing-vs-spam separation); inverse-frequency class weights buy recall at the cost of precision; | |
| the 2,000-row synthetic supplement does not help (reported, as promised, even though it does not move anything); | |
| halving the stack to 12 layers costs ~5 F1 points on TREC-06 for ~2x faster inference; and scratch | |
| initialisation is 8 F1 points behind the same model with MLM pretraining, which is what pretraining buys. | |
| `ft-focal` was selected on validation alone, before the holdouts were read, and is the run evaluated at top | |
| level (RUN_LOG 22), exported to `model_export/ft-focal/` + `app/` and uploaded (section 10). | |
| Not run, optional if time allows: `--freeze-layers 12` (forgetting vs cost), an epochs sweep (`--epochs 1` / | |
| `5`), `--num-labels 2` (binary baseline comparable with the binary-era plan). | |
| --- | |
| ## 10. Deployment | |
| | Item | Value | | |
| |---|---| | |
| | Model repo | `BuildITSystems/phishing-detector-encoder` (private): `model.safetensors` (bf16), `config.json`, `tokenizer.json`, `README.md` model card | | |
| | Space | `BuildITSystems/phishing-detector-demo` (private), Gradio SDK 6.29.1, hardware `zero-a10g` (ZeroGPU) | | |
| | Space stack | **torch 2.13.0 on Python 3.12.12**: ZeroGPU supports torch 2.8.0-2.13.0 and Python 3.12.12 / 3.10.13 only, and no torch.compile. Training uses 2.14.1 on 3.14; safetensors makes the weights portable. Gradio is pinned by `sdk_version` in `app/README.md`, not in `requirements.txt`. | | |
| | Weights at startup | The three weight files are **bundled into the Space** (`upload_to_hf.py --bundle-weights`, the mode used), so no secret is needed and the Space works even while the model repo is private. `app.py` can alternatively download them from the repo named by the `PHISH_MODEL_REPO` variable with an `HF_TOKEN` read-token secret. | | |
| | UI | Subject + body in; 3-class `gr.Label`, "Prediction Time (s)" (`timeit.default_timer`), gradient x input token highlights, rule-based cues shown for context only (they never feed the probability). | | |
| **Before class: make both repos public** (the instructor needs the model link and a Space that opens without | |
| an invite), then open the Space once so ZeroGPU is warm. `create_repo(exist_ok=True)` does not change the | |
| visibility of an existing repo, so do it in each repo's Settings page on huggingface.co, or: | |
| ```bash | |
| .venv/bin/python -c "from huggingface_hub import HfApi; a=HfApi(); a.update_repo_settings('BuildITSystems/phishing-detector-encoder', private=False); a.update_repo_settings('BuildITSystems/phishing-detector-demo', repo_type='space', private=False)" | |
| ``` | |
| `app/README.md` (the Space card) already links the model repo. | |
| --- | |
| ## 11. Instructor requirement mapping | |
| | Requirement (`reference/INSTRUCTOR_REQUIREMENTS.md`) | Where it is satisfied | | |
| |---|---| | |
| | Classifier, not regressor | 3-class softmax classifier + binary roll-up (section 1) | | |
| | Dataset from `torchvision.datasets` | **Deviation, agreed text variant**: there is no torchvision email dataset. Public Hugging Face datasets loaded with `datasets` (section 7); the PyTorch data pipeline is kept (DataLoader below). | | |
| | Problem statement: input, classes, challenge | Section 1 | | |
| | Algorithm and architecture, >= 2 layers | 24-layer transformer encoder, `src/model.py`; justification in `docs/research/research_model-math.md` and RUN_LOG 0 | | |
| | Loss function | Partial-label NLL (= cross-entropy for fully-labelled rows) with `--loss cross_entropy / weighted_cross_entropy / focal`; MLM cross-entropy on masked tokens for pretraining | | |
| | Optimizer | AdamW (fused, no decay on biases/LayerNorm), grad clip 1.0, bf16 autocast | | |
| | Epochs, early stopping / LR scheduling | Fine-tune: early stopping on val F1 (patience 2), warmup + linear decay. Pretrain: warmup + cosine to 10%, val MLM loss every 500 steps | | |
| | Justify choices | This README, RUN_LOG.md, `docs/research/` | | |
| | `DataLoader` with batching, shuffling, preprocessing | `finetune.make_loader`: `torch.utils.data.DataLoader` with a seeded length-bucketed shuffled `batch_sampler` and a padding collate; tokenisation as the preprocessing transform (RUN_LOG 8: moved from a numpy sampler on 2026-10-08 13:55, losses verified identical, explicit `generator` to keep the dropout stream unchanged) | | |
| | Evaluate on an unseen test set with standard metrics | `evaluate.py` on TREC-06 and test_holdout3: accuracy, precision, recall, F1, ROC-AUC, PR-AUC, confusion matrices (section 2.1) | | |
| | Inference timer, seconds per single prediction | `evaluate.py` -> `reports/latency.json` (GPU and CPU, batch 1); `smoke_test.py`; the app's "Prediction Time (s)" | | |
| | Gradio UI: input -> label + inference time | `app/app.py` (text input instead of an image upload) | | |
| | HF Spaces with `app.py`, weights, `requirements.txt` | Section 10; `upload_to_hf.py` | | |
| | Class: dataset, trained model, performance results | Sections 2, 7; `reports/` | | |
| | Class: model on Hugging Face, link to instructor | `https://huggingface.co/BuildITSystems/phishing-detector-encoder` (make public first) | | |
| | Class: live Gradio app on a Space, not gradio.live | `https://huggingface.co/spaces/BuildITSystems/phishing-detector-demo` | | |
| | Class: Space has app.py, model.py, requirements.txt (pinned torch/gradio) + weights | `app/`; torch pinned in requirements.txt, gradio pinned by `sdk_version` | | |
| | Class: show experimentation (layers, epochs, loss), record each run | Section 9, `reports/experiments.jsonl`, `finetune.py list`, RUN_LOG.md | | |
| | Class: prediction time per sample vs 0.074 s, GPU and CPU | Section 2.1: 10.1 ms GPU bf16, 459 ms CPU fp32 per email (`reports/latency.json`) | | |
| | Class: presentation covers data, algorithm, performance, challenges | RUN_LOG.md (OOM tuning, leakage, rejected datasets, reviewer-found bugs), sections 2, 7, 8, 12 | | |
| | Class: synthetic data allowed | `make_synthetic.py`, `--synthetic on`, reported with and without | | |
| | Class: one strong model, not an architecture comparison | One architecture (24L encoder); the six runs of section 9 vary training dials only, and `ft-focal` is the one deployed | | |
| | Notebook in the instructor's 9-step pattern | Section 5.10 | | |
| --- | |
| ## 12. Known limitations and ethics | |
| - **Defensive use only.** It flags mail for a human; it is not a verdict, not a replacement for SPF/DKIM/DMARC, | |
| sender reputation, link or attachment scanning, and must not be used to tune lures against. | |
| - **Old corpora.** Training mail is English, 1995-2022 with the bulk 1998-2008 (Enron, TREC, CEAS, | |
| SpamAssassin). Modern lures (MFA fatigue, QR codes, chat-app phishing) and non-English mail are out of | |
| distribution; the synthetic supplement only nudges this. | |
| - **Spam vs phishing follows provenance**, not per-message review: phishing = Nazario + Nigerian fraud + | |
| synthetic lures; spam = SpamAssassin and Enron-Spam spam folders. Only 3,150 real phishing rows carry the | |
| distinction; 62,805 TREC/CEAS positives train as "unwanted" only. Trust the roll-up first. | |
| - **Length confound.** Legitimate emails average about twice the tokens of unwanted ones in every split | |
| (TREC-06: 842 vs 283). Truncation at 512 tokens blunts it; `evaluate.py` checks it in the error analysis. | |
| - **Truncation.** About 22% of emails exceed 512 tokens (~1,800 characters) and are judged on their opening. | |
| - **Test-set caveat.** About 47% of TREC-06 ham is academic mailing-list traffic, which is not typical business mail. | |
| - **CPU latency.** 699 ms per email on CPU vs the 74 ms benchmark; the benchmark is met on GPU only. | |
| - **Privacy.** Enron mail is real correspondence released by FERC; nothing pasted into the Space is stored, | |
| but do not paste secrets or client content. | |
| --- | |
| ## 13. Troubleshooting | |
| | Symptom | Cause and fix | | |
| |---|---| | |
| | Pretraining exits 3 ("CUDA OOM" or "VRAM guard") | Another process took VRAM (browser, game, a second job) or the 20 GiB guard tripped. A checkpoint was saved. Check `nvidia-smi`, close the culprit, then `RESUME=1 MICRO_BATCH=24 scripts/launch_pretrain.sh`. The run's own peak is ~17.9 GiB reserved and the desktop adds ~2 GiB, so total device use sits near 20 GB. If it recurs, resume directly with `$PY src/pretrain.py train --run-name large-24L --resume --micro-batch 24 --grad-checkpointing` (8.06 GiB measured, ~25% slower). Accumulation = round(256 / micro-batch), so of the micro-batches that fit in 24 GB only 24 (x11) and 22 (x12) keep the effective batch at 264; 16 gives 256 and is refused on resume. | | |
| | Resume refused, exit 2, "'effective_batch' differs" | `launch_pretrain.sh` defaults to `MICRO_BATCH=16`; pass `MICRO_BATCH=24`. | | |
| | Resume refused, "'pretrain'/'schedule'/'data' differs" | `config.yaml` `pretrain:`/`model:`/`seed` was edited, or `pretrain.parquet` / `tokenizer.json` changed. Revert the edit; see section 6. | | |
| | `launch_pretrain.sh`: "tmux session already exists" | The finished pane waits for Enter: `tmux attach -t phish-pretrain` and press Enter, or `tmux kill-session -t phish-pretrain` once the trainer has exited. | | |
| | Fine-tune crashed or was stopped | `$PY src/finetune.py train --run-name <run> --resume` (restores optimizer, RNG and batch position). | | |
| | Fine-tune picked the wrong pretrained model | Pass `--init pretrained:checkpoints/pretrain/large-24L/final`; the default is the newest `*/final` by mtime and `dev/final` exists. | | |
| | Install resolves a Python 3.15 interpreter | torch.compile, triton 3.8.0 and `spaces` all exclude 3.15. Recreate with `uv venv --python 3.14.7 .venv`. | | |
| | Install fails on fsspec | datasets 5.1.0 caps `fsspec<=2026.7.0`; keep `fsspec==2026.7.0` from requirements-train.txt and never upgrade it alone. | | |
| | Install pulls cu132 wheels | `--torch-backend=auto` was used; reinstall with `--torch-backend=cu130`. | | |
| | huggingface_hub 2.x conflict | tokenizers 0.23.2 needs `<2.0`; keep 1.33.0. | | |
| | `export.py` refuses: metrics name another checkpoint | Run `evaluate.py run --split test,test3` on the same `best/` first. | | |
| | Space build fails on torch | ZeroGPU allows torch <= 2.13.0 and Python 3.12.12; keep `app/requirements.txt` and the `python_version` front matter as they are. | | |
| | Space starts but cannot load weights | `HF_TOKEN` secret missing or not a read token for the model repo, or `PHISH_MODEL_REPO` variable wrong; re-run the upload with `HF_TOKEN_FOR_SPACE` set, or use `--bundle-weights`. | | |
| | Any Python command while pretraining runs | Prefix with `CUDA_VISIBLE_DEVICES=""`; never run `smoke_test.py`, a GPU fine-tune or GPU evaluation alongside it. | | |