# BITS Phishing Detector

Final project for the Skillspire AI & Machine Learning course (Nathan King, Build IT Systems LLC), presented
late Oct / early Nov 2026. This README is the hand-off document for the instructor and for future BITS work.
The running record of every decision, number and problem is `RUN_LOG.md` (entries 0-8 as of this
writing); this file summarises it and explains how to reproduce it. Status as of 2026-10-08 14:00: data,
tokenizer, smoke test and all `src/`, `scripts/` and `app/` code are done; MLM pretraining is running; the
presentation notebook is in progress; fine-tuning, evaluation, export and deployment have not run yet.

---

## 1. What this is

| | |
|---|---|
| Task | **Classifier** (not a regressor). Input: one email as `subject + "\n\n" + body` (plain text; sender, headers and attachments are never read). Output: probabilities for three classes, **Legitimate / Phishing / Spam**, plus the roll-up **Unwanted (phishing or spam)** = P(Phishing) + P(Spam). |
| Use | Defensive triage: flag unwanted mail for a human to review. Not a filter that deletes mail, not a tool for writing lures. |
| Model | One BERT-style pre-LayerNorm encoder written from scratch in plain PyTorch (`src/model.py`, no `transformers`, no pretrained weights): 24 layers, d_model 1024, 16 heads, FFN 4096, vocab 32,768 (own byte-level BPE), 512 tokens. **337,476,608 parameters** with the tied MLM head, **337,443,842** as the classifier. |
| Training | Phase 1: masked-language-model pretraining on deduplicated Enron email. Phase 2: fine-tuning with a 3-way head. Rows whose corpus never separated spam from phishing (TREC-05/07, CEAS-08 positives) train on P(phishing)+P(spam) with a partial-label loss. |
| Core challenge | Public corpora label "unwanted" mail, not "phishing vs spam"; they are 1998-2008 heavy, overlap each other and the pretraining corpus, and legitimate mail is about twice as long as spam. The work is mostly in honest data: deduplication, leakage checks and a test set nothing upstream has seen. |
| Scope | **One strong model.** The experiments vary the instructor's dials (layers, epochs, loss, frozen layers, synthetic data, scratch vs pretrained init) on this one architecture; there is no architecture comparison. |

## 2. Results

### 2.1 Headline metrics (deployed run `ft-focal`, evaluated 2026-10-09 10:46)

Model selection used the **validation** set only: `ft-focal` had the best validation F1 of the unwanted
roll-up (0.9951) and the best validation three-way macro-F1 (0.9869) of the six runs in section 9. The two
holdouts below were then evaluated once per run for reporting; `ft-focal` also leads on both of them.

| TREC-06 holdout (16,302 emails, 23.9% unwanted; whole corpus unseen), threshold 0.5 | value |
|---|---|
| Recall, unwanted (reported first: a missed phish is the costly error) | **0.9741** (101 of 3,894 missed) |
| Precision, unwanted | 0.8794 (520 false alarms on 12,408 legitimate; FPR 4.2%) |
| F1, unwanted | **0.9243** (best achievable at any threshold: 0.9397 at p >= 0.68) |
| Accuracy | 0.9619 |
| ROC-AUC / PR-AUC | **0.9946** / 0.9848 |
| Recall at 1% FPR / at 0.1% FPR | 0.8955 / 0.6497 |
| Credential-lure slice (398 rows matching 37 lure phrases) recall / precision | 0.9959 / 0.9758 |

TREC-06 is the hard, honest number: a different corpus (2006 TREC spam track) with its own labelling
conventions. `reports/error_analysis.md` shows that many "false positives" are spam-looking mail that TREC-06
labels ham (quoted spam inside mailing-list threads, "remove me from your list" replies, sweepstakes mail) and
that the largest block of misses are "ScanMail virus found" notices TREC-06 labels spam. Legitimate TREC-06 mail
is also ~3x longer than its spam and 24% of it is truncated at 512 tokens.

| test_holdout3 (3,220 emails: 1,999 legit / 521 phishing / 700 spam; tokenizer never saw them) | Legitimate | Phishing | Spam |
|---|---|---|---|
| Precision / Recall / F1 | 0.9900 / 0.9935 / 0.9918 | 0.9961 / 0.9866 / 0.9913 | 0.9785 / 0.9757 / 0.9771 |
| Macro-F1 (3 classes) / accuracy | **0.9867** / 0.9885 | | |
| Binary roll-up (unwanted vs legitimate): F1 / recall / precision / ROC-AUC / PR-AUC | 0.9864 / 0.9836 / 0.9893 / 0.9993 / 0.9989 | | |
| Phishing vs spam, given the mail is unwanted (1,221 rows) | 0.9959 (4 phishing called spam, 1 spam called phishing) | | |

3x3 confusion matrix (rows true Legitimate / Phishing / Spam, columns predicted): `[1986, 1, 12]`, `[4, 514, 3]`, `[16, 1, 683]`.
Three-way accuracy per source: Assassin 95.3% (n=236), CEAS-08 99.0% (n=736), Enron 98.4% (n=628), Nazario 98.6% (n=222), Nigerian_Fraud 98.7% (n=299), TREC-07 99.8% (n=1099).

| Prediction time, one email, batch 1 (`reports/latency.json`, 100 TREC-06 samples, mean 280 tokens) | mean | p50 | p95 | vs 74 ms benchmark |
|---|---|---|---|---|
| GPU, bf16 (RTX 3090 Ti) | **10.1 ms** | 8.9 ms | 13.7 ms | 0.14x (faster) |
| CPU, fp32, 16 threads | **459 ms** | 445 ms | 701 ms | 6.2x (slower) |

Plots next to these files: `reports/confusion_matrix.png`, `confusion_matrix_test3.png`,
`confusion_matrix3_test3.png` (3x3), `roc_curve*.png`, `pr_curve*.png`, `finetune_curves.png`,
`pretrain_curves.png`, `eval_finetune_curves*.png`, `eval_pretrain_curves*.png`; written error analyses
`error_analysis.md` / `error_analysis_test3.md`. The fine-tune experiment table is `finetune.py list` (section 9).
Every run's own evaluation lives in `reports/eval_<run>/`.

### 2.2 Measured facts that exist today

| Fact | Value | Source |
|---|---|---|
| Parameters | 337,476,608 (MLM, tied head) / 337,443,842 (classifier) | `reports/smoke_test.json` |
| Pretraining VRAM, micro-batch 16, no checkpointing | 13.65 GiB reserved, 23,642 tok/s | `reports/smoke_test.json` |
| Pretraining VRAM, micro-batch 24, no checkpointing (**used**) | 17.65 GiB reserved, 19.80 GiB total device incl. desktop, 25,227 tok/s | `reports/smoke_test.json` |
| Pretraining VRAM, micro-batch 24, gradient checkpointing | 8.06 GiB reserved, 19,156 tok/s | `reports/smoke_test.json` |
| Measured bf16 matmul | 63.9 TFLOPS (4096^3); the smoke test's 25k tok/s is ~63% MFU of the 80 TFLOPS peak | RUN_LOG 0, 3 |
| Convergence check (fresh init, 30 steps) | MLM loss 10.58 -> 7.67 (ln V = 10.40) | RUN_LOG 3 |
| Latency, one 512-token email, GPU bf16 (RTX 3090 Ti), untrained-weight smoke test | 10.7 ms (trained checkpoint, real emails: section 2.1) | `reports/smoke_test.json` |
| Latency, one 512-token email, CPU fp32, 16 threads, untrained-weight smoke test | 699 ms (319 ms at 128 tokens) | `reports/smoke_test.json` |
| Pretraining plan `large-24L` | 12 epochs x 796 steps = 9,552 optimizer steps, micro-batch 24 x 11 = effective batch 264, peak LR 1.5e-4, warmup 573, cosine to 10% | `checkpoints/pretrain/large-24L/run_summary.json` |
| **Pretraining result `large-24L`** (complete 2026-10-09 03:31) | 9,552 steps, 1.29B tokens, 15.05 h of training, 23,889 tok/s mean, peak 17.94 GiB reserved; train MLM loss 1.524, **val MLM loss 1.986, masked-token accuracy 62.9%** (val loss fell monotonically, no restart needed) | `checkpoints/pretrain/large-24L/run_summary.json`, `metrics.csv` |

The smoke-test latencies were measured on untrained weights (latency does not depend on the weights); the
numbers to quote are the trained-checkpoint measurements in section 2.1 (`reports/latency.json`). The
instructor's 0.074 s/sample benchmark is met on GPU by a factor of ~7 and missed on CPU by a factor of ~6,
which is why the Space runs on ZeroGPU.

### 2.3 Data splits

Binary build (RUN_LOG 1, `data_prep.py build`):

| split | rows | positives | base rate | sources |
|---|---|---|---|---|
| finetune_train | 157,113 | 75,675 | 48.2% | TREC-05 46,386 / TREC-07 47,233 / CEAS-08 34,178 / Enron 24,223 / Assassin 5,093 |
| finetune_val | 17,643 | 8,642 | 49.0% | same five, grouped by near-duplicate cluster |
| test_holdout (TREC-06) | 16,302 | 3,894 | 23.9% | never seen by tokenizer, pretraining or fine-tuning |
| pretrain (MLM) | 226,273 docs | - | - | corbt/enron-emails after dedup + quote stripping; 407.9M chars, 107.7M tokens |

Three-way build (RUN_LOG 7, `data_prep.py build3`; label3 0 legit / 1 phishing / 2 spam / -1 unwanted-unsplit):

| split | rows | legit | phishing | spam | unwanted (TREC/CEAS positives) |
|---|---|---|---|---|---|
| finetune_train | 160,263 | 81,438 | 3,150 | 12,870 | 62,805 |
| finetune_val | 15,284 | 7,002 | 340 | 763 | 7,179 |
| test_holdout (TREC-06, rows unchanged) | 16,302 | 12,408 | - | - | 3,894 |
| **test_holdout3** | 3,220 | 1,999 | 521 | 700 | - |

Plus `data/processed/synthetic.parquet`: 2,000 template-generated rows (1,000 legitimate, 1,000 phishing
lures, 0 spam), training-only, off by default. finetune_val shrank from 17,643 to 15,284 rows when
test_holdout3 was carved out, so val numbers from binary-era runs are not strictly comparable.

---

## 3. Setup on Fedora 44

Machine used: Fedora 44 KDE, RTX 3090 Ti 24 GB (sm_86), NVIDIA driver 615.71.09 (CUDA 13.4), Ryzen 9 5950X,
40 GB RAM, system Python 3.14.7, uv 0.12.5 (`~/.local/bin/uv`), tmux.

```bash
cd "/home/nathank/BITS/dev/Phishing Model"
uv venv --python 3.14.7 .venv
uv pip install --python .venv/bin/python --torch-backend=cu130 -r requirements-train.txt
hf auth login                       # .venv/bin/hf; needed for the upload step only
```

Why these pins:

- **`--torch-backend=cu130`, never `auto`.** The plain PyPI torch 2.14.1 Linux wheel is the CUDA 13.0 build;
  driver 615 runs it by backward compatibility. uv 0.12.5's `auto` would pick cu132 on this driver. One
  resolution installs everything (180 packages, all binary wheels, venv ~6.1 GB).
- **Python 3.14.7, not 3.15.** torch.compile raises on 3.15, triton 3.8.0 requires <3.15 and `spaces` requires
  <3.15. Always pass `--python 3.14.7`.
- **`huggingface_hub==1.33.0`** because tokenizers 0.23.2 requires `huggingface-hub<2.0`.
- **`fsspec==2026.7.0`** because datasets 5.1.0 caps `fsspec[http]<=2026.7.0` (2026.9.0 is newer and breaks the
  resolve). Do not `uv pip install -U fsspec`.

Verify the GPU (do not run while a training job holds the GPU unless you accept a small extra CUDA context):

```bash
nvidia-smi
.venv/bin/python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.get_device_name(0), torch.cuda.is_bf16_supported())"
# expected (2026-10-07): 2.14.1+cu130 13.0 NVIDIA GeForce RTX 3090 Ti True
```

---

## 4. Project layout

```
Phishing Model/
├── README.md                     this file
├── RUN_LOG.md                    dated log of every run, decision, number, bug and fix (the "challenges" source)
├── config.yaml                   every dial in one file (section 6)
├── requirements-train.txt        pinned local training stack (Python 3.14.7, torch 2.14.1 cu130)
├── .gitignore                    excludes .venv, data, checkpoints, logs, weights, tokens
├── src/
│   ├── common.py                 config loading, seeding, logging, text normalisation, dedup keys (shared by all)
│   ├── data_prep.py              download -> dedupe -> decontaminate -> split -> leakage asserts (build/check/build3/check3)
│   ├── tokenizer.py              byte-level BPE (NFKC, case kept): train on pretrain + finetune_train only; stats
│   ├── model.py                  from-scratch pre-LN encoder, MLM head, classification head (copied to app/)
│   ├── smoke_test.py             VRAM micro-batch finder, throughput, SDPA backend check, convergence, latency
│   ├── pretrain.py               resumable bf16 MLM pretraining (train/export/plot)
│   ├── make_synthetic.py         2,000 template-generated training rows (build/show)
│   ├── finetune.py               3-way classifier fine-tuning with the instructor's dials (train/plot/list)
│   ├── evaluate.py               the only reader of the holdouts: metrics, latency, plots, error analysis
│   └── export.py                 bf16 safetensors + config + model card -> model_export/<run>/ and app/
├── scripts/
│   ├── launch_pretrain.sh        start or resume pretraining in tmux session `phish-pretrain`
│   ├── run_experiments.sh        fine-tune queue: waits for large-24L/final, runs the six dial experiments,
│   │                             evaluates each into reports/eval_<run>/, exports ft-ce to model_export/ft-ce and app/
│   └── upload_to_hf.py           publish model repo + Space; dry run by default, `--yes` to write
├── app/                          the Hugging Face Space
│   ├── app.py                    Gradio UI: 3-class label, prediction time, token saliency, rule-based cues
│   ├── README.md                 Space front matter (sdk gradio 6.29.1, python 3.12.12) + description
│   ├── requirements.txt          Space pins (torch 2.13.0, spaces 0.51.3, ...)
│   ├── .gitattributes            LFS rules for weights and tokenizer
│   └── model.py, model.safetensors, config.json, tokenizer.json   written by export.py (not yet present)
├── FinalProject_PhishingDetector.ipynb   presentation notebook in the instructor's 9-step pattern (section 5.10);
│                                 written by the notebook step, in progress as of 2026-10-08 14:00
├── reference/
│   ├── FashionMNIST_Gradio_Full.ipynb   instructor's example notebook (the 9-step pattern)
│   └── INSTRUCTOR_REQUIREMENTS.md       grading rubric, verbatim
├── docs/research/
│   ├── research_dataset-schemas.md      schemas, provenance and licences of every candidate dataset
│   ├── verify_data.md                   independent verification of the data plan and holdout choice
│   ├── research_model-math.md           parameter / memory / FLOP / latency math for the encoder
│   ├── model_math.py                    pure-Python sizing calculator behind that document
│   ├── research_torch-python.md         torch 2.14.1 / CUDA 13.0 / Python 3.14 compatibility, SDPA backends
│   ├── research_other-libs.md           every other pin and its conflicts (fsspec, huggingface_hub, spaces)
│   ├── verify_pins.md                   independent verification of the pins
│   ├── research_gradio-spaces-hub.md    Gradio 6, ZeroGPU, Space front matter, Hub upload API
│   └── bigshp_research.md               why the 300k "spam/ham/phish" set was rejected
├── data/                         (git-ignored)
│   ├── raw/                      HF cache for corbt, Champa mirror CSVs (aura/), inspected-only 300k set (bigshp/)
│   └── processed/
│       ├── finetune_train.parquet, finetune_val.parquet    training / model-selection splits (with label3)
│       ├── test_holdout.parquet                            TREC-06 binary test set
│       ├── test_holdout3.parquet                           clean-provenance three-way test set
│       ├── pretrain.parquet                                MLM corpus (226,273 docs)
│       ├── pretrain_tokens.npy/.json, pretrain_val_tokens.npy   cached token stream used by pretrain.py
│       ├── synthetic.parquet, synthetic_report.json        template supplement + shortcut audit
│       ├── tokenizer.json, tokenizer_report.json           BPE vocabulary + per-split length stats
│       ├── splits_manifest.json, data_report.md            counts, removals, every leakage assertion
│       └── binary_snapshot/                                binary-era splits that build3 starts from
├── checkpoints/                  (git-ignored)
│   ├── pretrain/large-24L/       live run: step-NNNNNNN/, metrics.csv, run_summary.json, final/ when done
│   ├── pretrain/dev/             tiny self-test run (delete after Nathan's OK)
│   └── finetune/<run>/           best/, last/, metrics.csv, epochs.csv per fine-tune run (dev, dev-focal = self-tests)
├── reports/                      smoke_test.json; experiments.jsonl + finetune_curves.png + pretrain_curves.png are
│                                 dev-only until the real runs regenerate them (section 9); metrics*, latency*,
│                                 confusion/ROC/PR plots, error analyses, eval_<run>/ later
├── model_export/<run>/           export output (created by export.py)
└── logs/                         one timestamped log per script invocation + tmux stdout of the pretraining run
```

---

## 5. How to run each phase, in order

All commands run from the project root with `.venv/bin/python` (written `PY` below:
`PY=.venv/bin/python`). Every script takes `--config config.yaml` before the subcommand and writes a log to
`logs/<script>-<timestamp>.log`. Flags below were checked against each script's `--help` on 2026-10-08.
**While pretraining holds the GPU, prefix any other Python command with `CUDA_VISIBLE_DEVICES=""` and do not
start smoke_test, finetune, evaluate on GPU, or a second training job.**

### 5.1 Data (`src/data_prep.py`)

```bash
$PY src/data_prep.py inspect        # optional: schema + 2 rows per dataset, streaming
$PY src/data_prep.py build          # binary splits + pretrain corpus + manifest + data_report.md (~2.5 min, 6.9 GB RSS)
$PY src/data_prep.py check          # recompute every binary leakage assertion from the written files
$PY src/data_prep.py build3         # three-way labels, Champa phishing corpora, test_holdout3 (~1 min)
$PY src/data_prep.py check3         # recompute binary + three-way assertions
```

Flags: `build --limit N` / `build3 --limit N` are dev-only subsets (`build3 --limit` writes to
`data/processed/dev_threeway_limitN/`); `build --spam-policy {positive_class,map_to_zero,exclude}` (default
`positive_class`); `build --no-strip-quotes`; `--shingle-cap N`; `build3 --bigshp {on,off}` overrides the
rejected 300k set (leave off); `check3 --out-dir`.
**Do not re-run `build` or `tokenizer.py train` while `large-24L` is running**: a new `pretrain.parquet` or
`tokenizer.json` rebuilds the token cache and the run then refuses to resume. `build` also rewrites the
binary splits, so `build3` must follow it.

### 5.2 Tokenizer (`src/tokenizer.py`)

```bash
$PY src/tokenizer.py train          # byte-level BPE, vocab 32,768, NFKC, case kept; ~30 s
$PY src/tokenizer.py stats          # length / truncation / [UNK] table per split -> tokenizer_report.json
```

`train --vocab-size N`, `train --limit-docs N` (dev). Trains only on `pretrain.parquet` + `finetune_train.parquet`;
val and test files are refused by name and checked with an exact-hash tripwire.

### 5.3 Smoke test (`src/smoke_test.py`) — GPU must be free

```bash
$PY src/smoke_test.py               # -> reports/smoke_test.json + printed summary
```

Useful flags: `--candidates 16,24,12,8`, `--ckpt-candidates`, `--steps 30`, `--skip-cpu`, `--n-docs 3000`,
`--latency-samples 100`, `--cpu-threads 16`. It probes micro-batches with the real gradient-accumulation
pattern, measures tokens/s, checks which SDPA kernel runs, runs a 30-step convergence check and times
single-email inference on GPU and CPU.

### 5.4 MLM pretraining (`scripts/launch_pretrain.sh` -> `src/pretrain.py`)

```bash
EPOCHS=12 MICRO_BATCH=24 scripts/launch_pretrain.sh     # how large-24L was launched (2026-10-08 12:28)
tmux attach -t phish-pretrain                           # watch; Ctrl-b d detaches; Ctrl-C inside = save + clean exit
RESUME=1 MICRO_BATCH=24 scripts/launch_pretrain.sh      # resume from the latest checkpoint
```

Environment variables: `RUN_NAME` (default `large-24L`), `EPOCHS` (script default 6; fixes the schedule
length at first launch, ignored on resume), `MICRO_BATCH` (**script default 16; always pass 24 for this run**:
16 gives effective batch 256 instead of the run's 264 and the resume is refused), `SESSION` (default
`phish-pretrain`), `RESUME=1`. The script refuses to start if the tmux session already exists; after the
trainer exits the pane waits for Enter (or `tmux kill-session -t phish-pretrain`). Stdout is teed to
`logs/pretrain_<run>_<timestamp>.out`.

Direct use: `$PY src/pretrain.py train --run-name R [--resume [PATH|latest]] [--epochs N] [--max-steps N]
[--max-hours H] [--micro-batch B] [--grad-checkpointing] [--limit-docs N] [--checkpoint-every N]
[--log-every N] [--eval-every N] [--keep-checkpoints K] [--fresh]`. After the run:

```bash
$PY src/pretrain.py plot --run-name large-24L      # -> reports/pretrain_curves.png
$PY src/pretrain.py export --run-name large-24L    # only if stopped early: builds final/ from the latest checkpoint
```

How it stops (every path writes a checkpoint first; checkpoints are atomic tmp-dir + rename):

| Event | Behaviour | Exit code |
|---|---|---|
| Normal end | `final/` written | 0 |
| Ctrl-C / SIGTERM | save at the next optimizer-step boundary | 0 |
| `--max-hours` reached | save | 0 |
| **VRAM guard**: this process's reserved peak > `vram.hard_cap_gb` (20 GiB) | save, stop before a real OOM | 3 |
| **CUDA OOM** (e.g. the desktop grabs VRAM) in forward/backward, optimizer step or eval | partial step discarded, data cursor rewound, save | 3 |
| 5 consecutive non-finite losses (each such step is skipped) | status `diverged`, save | 4 |
| Config/plan mismatch on resume (model shape, `pretrain:` LR/betas/eps/decay/clip/masking/schedule, seed, effective batch, epochs, data) | refuses before training | 2 |

Checkpoints every 500 steps and at every epoch end (keep 3, ~3.8 GB each) in
`checkpoints/pretrain/large-24L/step-NNNNNNN/`; val MLM loss and accuracy on 64 fixed held-out windows every
500 steps in `metrics.csv`. If val MLM loss rises, stop the run (Ctrl-C) and use `pretrain.py export`.

### 5.5 Synthetic supplement (`src/make_synthetic.py`) — already built

```bash
$PY src/make_synthetic.py build [--n 2000] [--out PATH] [--seed S]   # -> data/processed/synthetic.parquet + _report.json
$PY src/make_synthetic.py show --k 6
```

### 5.6 Fine-tuning (`src/finetune.py`) — after pretraining, GPU free

Baseline (config defaults: 3 classes, cross-entropy over the partial-label NLL, 3 epochs, batch 32 = micro-batch
16 x 2, LR 3e-5, early stopping on val F1 of the unwanted roll-up, patience 2):

```bash
$PY src/finetune.py train --run-name ft-ce --init pretrained:checkpoints/pretrain/large-24L/final
```

`ft-ce` is the cross-entropy baseline. The queue ran all six runs of section 9 on 2026-10-09 (03:36-10:02);
**`ft-focal` won on the validation selection metric** (val F1 of the unwanted roll-up 0.9951 vs 0.9929 for
`ft-ce`, val three-way macro-F1 0.9869 vs 0.9782) and is therefore the run evaluated at top level, exported
and deployed. Every example in sections 5.7-5.9 uses it.

**The queue (`scripts/run_experiments.sh`)** is how the experiments are meant to run: it waits for
`checkpoints/pretrain/large-24L/final/model.safetensors`, renders `pretrain.py plot`, then trains the six runs
of section 9 one after another with `--eval-every 1250`, plots each to `reports/finetune_curves_<run>.png`,
evaluates each `best/` on both holdouts into `reports/eval_<run>/` (marker `reports/eval_done_<run>`), and
finally evaluates + exports the main run (`MAIN_RUN`, default `ft-ce`) into the **top-level**
`reports/metrics.json` / `metrics_test3.json` that section 2.1 quotes, `model_export/<MAIN_RUN>/` and `app/`.
The top-level evaluate + export were re-run by hand for `ft-focal` after the queue finished (section 9).
A run whose `best/` exists is skipped, so re-running the script continues after a crash.

```bash
tmux new-session -d -s phish-finetune 'scripts/run_experiments.sh 2>&1 | tee logs/experiments_$(date +%Y%m%d-%H%M%S).out'
```

Environment: `PRETRAIN_RUN` (default `large-24L`), `EVAL_EVERY` (default 1250), `MAIN_RUN` (default `ft-ce`).

The instructor's dials (all flags of `train`):

| Dial | Flag | Values |
|---|---|---|
| Loss function | `--loss` | `cross_entropy` / `weighted_cross_entropy` (inverse-frequency) / `focal` (`--focal-gamma 2.0`) |
| Epochs | `--epochs` | e.g. 1-5; early stopping may end sooner |
| Number of layers | `--n-layers N` | keep the first N pretrained layers (e.g. 12 of 24) |
| Frozen layers | `--freeze-layers K` | freeze embeddings + first K layers |
| Synthetic data | `--synthetic on\|off` | appends the 2,000-row supplement to train only |
| Init | `--init scratch` / `--init pretrained:<dir>` | scratch = ablation showing what pretraining buys |
| Others | `--lr`, `--batch-size`, `--micro-batch`, `--num-labels 2\|3`, `--seed`, `--eval-every`, `--log-every`, `--eval-batch-size` | |
| Dev only | `--limit-rows N`, `--max-steps N`, `--cpu` | not evidence |

Always pass `--init` explicitly: without it the script takes the newest `checkpoints/pretrain/*/final` by
modification time, and `checkpoints/pretrain/dev/final` (a tiny self-test model) exists.

```bash
$PY src/finetune.py train --run-name ft-focal --resume     # continue <run>/last after a crash or Ctrl-C
$PY src/finetune.py train --run-name ft-focal --overwrite  # discard an existing run dir of that name
$PY src/finetune.py list                                   # experiments.jsonl as a markdown table
$PY src/finetune.py plot --run-name ft-focal               # -> reports/finetune_curves.png (`--out PATH` to rename)
```

On resume, `--init/--n-layers/--freeze-layers/--batch-size/--seed/--num-labels` are refused; `--lr` is
re-applied to the restored optimizer. Output: `checkpoints/finetune/<run>/{best,last}/`, `metrics.csv`,
`epochs.csv`. The test files are refused by name and by `source` column.

### 5.7 Evaluation (`src/evaluate.py`) — the only script that opens the holdouts

```bash
$PY src/evaluate.py run --checkpoint checkpoints/finetune/ft-focal/best --split test,test3
$PY src/evaluate.py run --checkpoint checkpoints/finetune/ft-focal/best --split val   # model selection view, `_val` suffix
```

The queue runs this per experiment with `--run-name <run> --out-dir reports/eval_<run>`; only the main run is
written to the top-level `reports/` (section 2.1).

Writes `reports/metrics.json`, `metrics_test3.json`, `latency.json` (batch-1 `timeit.default_timer` around
tokenize + forward + softmax, GPU bf16 and CPU fp32, next to the 74 ms benchmark), `confusion_matrix*.png`,
`confusion_matrix3_test3.png`, `roc_curve*.png`, `pr_curve*.png`, `error_analysis*.md` (top false negatives /
positives, phishing<->spam confusions, length-confound check, credential-lure slice), one line in
`reports/experiments.jsonl` and one RUN_LOG.md entry. Flags: `--threshold`, `--batch-size`,
`--latency-samples`, `--cpu-threads`, `--max-len`, `--cpu`, `--out-dir`, `--data-dir`, `--run-name`,
`--limit-rows` (dev; no RUN_LOG entry), `--no-run-log`, `--finetune-run-dir` / `--pretrain-run-dir` (extra
`eval_*_curves.png`). The binary metrics use P(unwanted) at 0.5; the three-way decision is roll-up first,
then argmax(phishing, spam), so the 3x3 matrix collapses exactly onto the binary one (plain argmax-of-3 is
reported alongside).

### 5.8 Export (`src/export.py`)

```bash
$PY src/export.py run --checkpoint checkpoints/finetune/ft-focal/best
```

Writes `model_export/ft-focal/` (`model.safetensors` bf16 ~675 MB, `config.json` with label names, metrics and
dials, `tokenizer.json`, `README.md` model card, `export_manifest.json`) and copies `model.py` (sha1-checked)
plus the three weight files into `app/`. It refuses if `reports/metrics*.json` name a different checkpoint, so
run 5.7 on the same `best/` first (`--allow-metrics-mismatch` overrides and records it). Other flags:
`--out-dir`, `--app-dir`, `--dtype {bf16,fp32}`, `--metrics-json`, `--metrics3-json`, `--pretrain-config`.

Local app check before uploading: `CUDA_VISIBLE_DEVICES="" $PY app/app.py` (CPU; drop the prefix once the GPU
is free: `app.py` moves the model to CUDA at import time whenever a GPU is visible).

### 5.9 Upload (`scripts/upload_to_hf.py`) — Nathan only

```bash
$PY scripts/upload_to_hf.py --model-dir model_export/ft-focal --bundle-weights --check-auth   # dry run (default): prints the plan, no writes
$PY scripts/upload_to_hf.py --model-dir model_export/ft-focal --bundle-weights --yes          # live (what was run on 2026-10-12)
```

The dry run lists every file, size, repo, visibility, the hardware flavour and any blocking problem (a missing
export, a missing `app/model.py`, or `HF_TOKEN_FOR_SPACE` unset when the weights are not bundled). Other flags:
`--bundle-weights` (weights inside the Space, no secret needed; the mode used), `--skip-secret`, `--hardware`,
`--model-repo`, `--space-repo`, `--private/--no-private`, `--commit-message`. `--yes` is only ever run with
Nathan's explicit approval; the one live run happened on 2026-10-12 11:35 (RUN_LOG 24) after his approval of
2026-10-08. Lesson from that run: the Hub validates the Space card's YAML front matter at upload time and the
dry run does not, so a colon inside an unquoted `short_description` failed the first attempt; the value is now
quoted and the model-repo half of the first attempt was simply reused (LFS blobs dedupe by content hash).

### 5.10 The notebook

`FinalProject_PhishingDetector.ipynb` (project root; generated by `scripts/build_notebook.py`, last executed top
to bottom on the CPU on 2026-10-12: 36 code cells, no errors, every artifact present) is the presentation
walk-through. It mirrors the instructor's
`reference/FashionMNIST_Gradio_Full.ipynb` step for step, with the nine headers quoted verbatim: "Step 1: Install
Dependencies & Setup Device", "Step 2: Download Data & Create DataLoaders", "Step 3: Define the Neural Network",
"Step 4: Train the Model", "Step 5: Test Inference Speed and Save the Model", "Step 6: Create the App Files
(Optional)", "Step 7: Launch the Gradio Interface", "Step 8: Authenticate with Hugging Face", "Step 9: Upload
the App to Hugging Face Spaces". It **loads the artifacts above instead of training**: the contract is that it
runs top to bottom on CPU and prints "not available yet — produced by <command>" for anything missing; with the
2026-10-09 artifacts in place it shows the real numbers, curves, confusion matrices and the `ft-focal` export.

```bash
CUDA_VISIBLE_DEVICES="" .venv/bin/jupyter lab
CUDA_VISIBLE_DEVICES="" .venv/bin/jupyter nbconvert --to notebook --execute --inplace FinalProject_PhishingDetector.ipynb
```

### 5.11 The Colab presentation notebook (`FinalProject_PhishingDetector_Colab.ipynb`)

Built by the same script with `--colab` (`--out PATH` picks the output file; the default mode and its output are
unchanged). Design: **the notebook is narrative, the Space is the compute.** Nothing trains or infers in Colab;
every prediction runs on the ZeroGPU Space, and all evidence is pulled from the Hub:

| Need | Where it comes from in Colab |
|---|---|
| Metrics, curves, confusion matrices, experiments table, data report, run log, code | `presentation/` in the model repo (85 files, 3.6 MB; relative paths mirror this project), fetched with `snapshot_download` into `/content/phishing_bundle` |
| Model definition and parameter count (Step 3) | `presentation/src/model.py` imported on the Colab CPU (torch is preinstalled there); falls back to the counts in `config.json` without torch |
| Inference speed (Step 5) | `gradio_client.Client(SPACE, token=...).predict(subject, body, api_name="/predict")` on six synthetic emails: in-app "Prediction Time (s)" measured on the Space's A10G (about 0.23 s cold, 0.01 s warm) plus the round-trip; `latency.json` adds the RTX 3090 Ti numbers |
| The app (Step 7) | `gr.load(SPACE, src="spaces", token=...)` embeds the Space UI; the private model repo's weights never leave the Hub. Gradio 6.29.1 has a bug rebuilding a Space whose Interface has `examples=` (`Textbox.__init__() got an unexpected keyword argument 'class_id'`), so the cell installs a two-line compatibility shim and retries, and falls back to a hand-built proxy `gr.Interface` over `gradio_client` if that also fails. When the Space is public the cell also shows the Space's own page in an `IFrame` of https://builditsystems-phishing-detector-demo.hf.space |
| Auth (Step 8) | Token resolution order: Colab secret `HF_TOKEN` -> `os.environ["HF_TOKEN"]` -> the `hf auth login` cache (outside Colab only when `NOTEBOOK_ALLOW_LOGIN_CACHE=1`, so a headless run never silently forwards the workstation's write token) -> none (enough once the repos are public). Only the token *source* is ever printed |

How Nathan runs it: upload the `.ipynb` to Colab (File -> Upload notebook, or from Drive); while the repos are
private, add a Colab Secret named `HF_TOKEN` with notebook access on, and make it a **fine-grained read token
scoped to the two repos**, because `gr.load` forwards the token to the Space with every proxied call; run all
cells. The first cell installs `gradio==6.29.1 huggingface_hub==1.33.0` (only when it detects Colab). Once both
repos are public no token is needed and the iframe appears automatically.

Verified on 2026-10-12: executed headlessly from an empty directory in a clean Python 3.12 venv (torch CPU,
gradio 6.29.1, huggingface_hub 1.33.0) with no project file reachable: 36/36 code cells ran, zero errors, six
live predictions from the Space, `gr.load` succeeded after the shim retry, the account name printed and no
token-shaped string in any output; the numbers it renders were spot-checked against `reports/metrics*.json`
(RUN_LOG 25). The Colab-only branches (`%pip`, `userdata.get`, inline `launch()`) are exercised by the
`IN_COLAB` switch but were not run on a real Colab runtime; do one rehearsal there.

---

## 6. Config dials (`config.yaml`)

| Section | What it controls |
|---|---|
| `seed`, `paths` | global seed 42; every relative path is resolved against the project root |
| `model` | encoder shape (24 / 1024 / 16 / 4096 / 32768 / 512), dropout, pre-LN, tied MLM head, gradient checkpointing |
| `tokenizer` | byte-level BPE, vocab 32,768, min frequency 2, specials `[PAD] [UNK] [CLS] [SEP] [MASK]`, training corpora |
| `data` | sources, exclusions, holdout source (TREC-06), alnum minimums, dedup keys and shingle parameters, val fraction, `synthetic`, `threeway` (label3 by provenance, Champa mirror, the rejected `bigshp` set with its verdict, holdout3 carving) |
| `vram` | 14-18 GiB target band, 20 GiB hard cap (GiB, compared with `torch.cuda.max_memory_reserved()`), allocator config |
| `pretrain` | bf16, packing, effective batch 256, micro-batch start 24 + candidates, 12 epochs, AdamW LR 1.5e-4 / betas (0.9, 0.98) / eps 1e-6 / decay 0.01 / clip 1.0, warmup, cosine to 10%, 15% masking 80/10/10, checkpoint/log/eval cadence |
| `finetune` | `num_labels: 3`, epochs, batch / micro-batch, LR 3e-5, warmup + linear decay, loss and class weights, focal gamma, early stopping metric + patience, frozen layers, padding multiple |
| `eval` | metric list, credential-lure slice, latency samples, threshold 0.5, test3 file, label contract (asserted against the code), top-k confusions, model-card metrics source |
| `hf` | owner, model and Space repo ids, private flag, Gradio 6.29.1, Python 3.12.12, `zero-a10g`, Space torch 2.13.0, safetensors bf16 |

**Rule during a live pretraining run:** do not edit `model:` or the `pretrain:` keys that shape the plan
(`lr`, `betas`, `eps`, `weight_decay`, `grad_clip`, `mlm_probability`, `mask_replace_probs`, `schedule`,
`effective_batch_sequences`, `warmup_steps`, `warmup_fraction`), `seed`, or anything that changes
`pretrain.parquet` / `tokenizer.json`. Each is stored in the checkpoint and a resume refuses a mismatch. This is
why `num_labels` lives under `finetune:`. `finetune:`, `eval:` and `hf:` can be edited at any time.

---

## 7. Data provenance and licences

| Source | Used for | Licence / terms | Notes |
|---|---|---|---|
| `puyang2025/seven-phishing-email-datasets` @ b7f3f8d9 (203,017 rows) | labelled legit + unwanted: TREC-05/06/07, CEAS-08, Enron-Spam, SpamAssassin | Hub card "other"; upstream Champa, Rabbi & Zibran, "Seven Phishing Email Datasets", figshare 25432108, CC BY 4.0 | Its own train/test split leaks (5.1% exact duplicates) and is ignored. Ling (2,859 academic list rows) excluded. Label 1 = "unwanted": these are spam-track corpora with no phishing/spam distinction. |
| `corbt/enron-emails` @ cfc06c75 (517,401 rows) | MLM pretraining only | CMU Enron corpus (FERC public release); no formal licence; CMU asks users to respect privacy | 52.7% exact duplicates removed, quoted/forwarded history stripped -> 226,273 docs. |
| `kudzaiprichard/aura-phishing-email-corpus` @ 539567ce: `Nazario.csv`, `Nigerian_Fraud.csv` | phishing class (1,507 + 2,504 rows kept) | CC BY 4.0; byte-identical mirror of Champa, Rabbi & Zibran, "Phishing Email Curated Datasets", Zenodo, doi:10.5281/zenodo.8339691 | Nazario phishing archive and advance-fee (419) fraud, 1995-2022. 235 Nigerian rows near-duplicating TREC-06 spam and 13 exact matches removed to keep the test set clean. |
| `data/processed/synthetic.parquet` (2,000 rows) | optional training supplement | generated by `make_synthetic.py`; fictional names, `.example`/`.invalid` domains, RFC 5737 IPs, 555-01xx phones | 8 BEC / Microsoft 365 / invoice lure categories vs 8 ordinary business categories; shared filler pools and a token-purity audit against template shortcuts. Never in val or test, never presented as real mail. |
| Rejected: `locuoco/the-biggest-spam-ham-phish-email-dataset-300000` | nothing | MIT (Kaggle re-upload) | 365,448 rows, 51.5% internal duplicates, a union of corpora we already hold; its "phish" label agrees with provenance only 60.5%; **36,020 rows are copies of our TREC-06 test set**; phish rows went through a stopword-stripping pipeline the others did not (a label-proxy artifact). Not one row ingested. `docs/research/bigshp_research.md`. |
| Rejected: `it4lia/PhishingEmailCuratedDatasets_Cleaned` | nothing | CC BY 4.0 | no body text (subjects/headers + 53 numeric features). |
| Rejected: `zefang-liu/phishing-email-dataset` | nothing | LGPL-3.0 | lowercased re-packaging, 91.9% contained in the primary set, "enron" token shortcut. |

Details and evidence: `docs/research/research_dataset-schemas.md`, `docs/research/verify_data.md`,
`data/processed/data_report.md`. The exported model card states that phishing/spam labels follow corpus
provenance.

---

## 8. Leakage policy

- **TREC-06 is the binary test set and nothing upstream has seen it.** It is excluded from tokenizer training,
  the MLM corpus and fine-tuning. It is Enron-clean (5 rows mention "enron", 0 enron.com addresses); TREC-05 ham
  and Enron-Spam ham are 86-89% inside the pretraining corpus, so neither could be the test set. Model input is
  subject + body only because TREC rewrote the spam headers.
- **Decontamination.** Exact keys on normalised text (full, 300-char prefix, subject+body prefix, subject+body
  full) plus near-duplicates (>= 3 shared 50-char shingles over the first 1,500 chars). 1,652 training rows that
  near-duplicate TREC-06 templates were removed; 85 pretraining docs near-duplicating holdout emails and 4,004
  matching val keys were removed from the MLM corpus.
- **Train/val split** is grouped by near-duplicate cluster and stratified by source x label, so no template
  straddles train and val.
- **test_holdout3** rules: legitimate rows only from CEAS-08, TREC-07, SpamAssassin (never Enron, which overlaps
  pretraining); spam from SpamAssassin and Enron-Spam spam folders; phishing from Nazario + Nigerian_Fraud.
  Rows come from finetune_val (never seen by the tokenizer) or the new sources (never seen by anything); whole
  clusters move or none; any cluster with a key or shingle link to the pretraining corpus or train is excluded.
- **Asserted zeros** (recomputed independently from the written files by `check` / `check3`; a non-zero count
  fails the build): holdout vs train / val / pretrain keys and shingle links; train vs val full-key rows; val vs
  pretrain keys; holdout source rows in train/val; holdout3 vs train / val / pretrain keys and shingle links;
  holdout3 vs TREC-06 keys; doc_id shared between splits; label3 contract violations. All are 0
  (`data/processed/data_report.md`). Counted but allowed: 5,336 val-to-pretrain near-dup links (val is only
  used for model selection), 102,100 train-to-pretrain key overlaps (Enron ham).
- **Code guards.** The tokenizer refuses val/test files by name with an exact-hash tripwire (0 collisions);
  `finetune.py` refuses the holdout by file name and `source` column; `pretrain.py` never opens it;
  `evaluate.py` is the only reader.

---

## 9. Experimentation record

- **`reports/experiments.jsonl`**: `finetune.py train` appends one JSON line per run (dials: init, layers,
  frozen layers, epochs planned/run, loss, focal gamma, LR, batch, class weights, synthetic rows, seed,
  selection metric; parameter counts; steps, seconds, peak VRAM; best step; val metrics) and `evaluate.py`
  appends one line per invocation with the test metrics. `finetune.py list` renders it as the markdown table for
  the presentation. Generated from evidence, never typed up afterwards.
- **Contents**: the six real runs below (2026-10-09) plus one evaluate line per holdout evaluation. The dev
  self-test rows from 2026-10-08 were deleted before the queue ran (RUN_LOG 9). `reports/finetune_curves.png` is
  the `ft-focal` curve, `reports/finetune_curves_<run>.png` the others, `reports/pretrain_curves.png` the
  `large-24L` curve.
- **Pretraining** is one run (`large-24L`); its curve is `metrics.csv` -> `pretrain.py plot`.
- **Fine-tune queue** (`scripts/run_experiments.sh`, section 5.6; one architecture, one dial changed per run
  against the baseline). It ran unattended on 2026-10-09 from 03:36 to 10:02 (six runs, each evaluated on both
  holdouts). The baseline `ft-ce` was allowed 3 epochs; the five ablations 2 epochs each for run-time reasons;
  early stopping (patience 2 on the validation unwanted-F1, evaluated every 1,250 steps) ended five of the six
  before their epoch budget. All start from `checkpoints/pretrain/large-24L/final` except `ft-scratch`.

| Run | Flags | Epochs | Question it answers |
|---|---|---|---|
| ft-ce | `--loss cross_entropy` | 3 | reference point: pretrained 24L, cross-entropy, synthetic off |
| ft-weighted | `--loss weighted_cross_entropy` | 2 | class imbalance (3,150 phishing vs 81,438 legit fully-labelled rows) |
| ft-focal | `--loss focal` (gamma 2.0 from config) | 2 | hard-example weighting |
| ft-synthetic | `--loss cross_entropy --synthetic on` | 2 | does the BEC/M365 supplement move anything (reported even if it does not) |
| ft-12L | `--loss cross_entropy --n-layers 12` | 2 | number of layers (half the stack, ~2x faster inference) |
| ft-scratch | `--loss cross_entropy --init scratch` | 2 | what MLM pretraining buys |

### 9.1 Results (validation = selection; holdouts = reporting)

Selection view (validation split, 15,284 rows; the queue's early-stopping metric is the unwanted-F1 column):

| Run | Loss / dial | Epochs run / planned | Stopped by | Best step | Train min | Peak GiB | Val F1 unwanted (selection) | Val macro-F1 3-way | Val P / R phishing |
|---|---|---|---|---|---|---|---|---|---|
| ft-ce | cross_entropy | 1.25 / 3 | early_stopping | 5000 | 51 | 13.13 | 0.9929 | 0.9782 | 0.939 / 0.997 |
| ft-weighted | weighted_cross_entropy | 2.00 / 2 | epochs_done | 10000 | 84 | 13.13 | 0.9937 | 0.9775 | 0.963 / 0.997 |
| **ft-focal** | focal (gamma 2.0) | 2.00 / 2 | early_stopping | 8750 | 84 | 13.13 | **0.9951** | 0.9869 | 0.985 / 0.988 |
| ft-synthetic | cross_entropy, synthetic on | 1.23 / 2 | early_stopping | 5000 | 52 | 13.13 | 0.9927 | 0.9665 | 0.881 / 0.997 |
| ft-12L | cross_entropy, 12 layers | 1.75 / 2 | early_stopping | 6250 | 35 | 6.97 | 0.9925 | 0.9774 | 0.957 / 0.982 |
| ft-scratch | cross_entropy, scratch init | 1.50 / 2 | early_stopping | 5009 | 61 | 13.13 | 0.9875 | 0.9538 | 0.938 / 0.941 |

Held-out view (TREC-06: 16,302 rows, whole corpus unseen; test_holdout3: 3,220 rows, 1,999 / 521 / 700). Evaluated once per run after training; not used for selection:

| Run | TREC-06 recall | precision | F1 | FPR | ROC-AUC | PR-AUC | recall @ 1% FPR | test3 F1 unwanted | test3 macro-F1 | F1 phishing | F1 spam |
|---|---|---|---|---|---|---|---|---|---|---|---|
| ft-ce | 0.9404 | 0.8610 | 0.8990 | 4.8% | 0.9875 | 0.9626 | 0.7745 | 0.9865 | 0.9822 | 0.9769 | 0.9779 |
| ft-weighted | 0.9818 | 0.8307 | 0.9000 | 6.3% | 0.9943 | 0.9833 | 0.8767 | 0.9862 | 0.9852 | 0.9875 | 0.9767 |
| **ft-focal** | 0.9741 | 0.8794 | **0.9243** | 4.2% | 0.9946 | 0.9848 | 0.8955 | 0.9864 | **0.9867** | 0.9913 | 0.9771 |
| ft-synthetic | 0.9679 | 0.8402 | 0.8995 | 5.8% | 0.9908 | 0.9716 | 0.7871 | 0.9873 | 0.9814 | 0.9772 | 0.9749 |
| ft-12L | 0.9461 | 0.8120 | 0.8739 | 6.9% | 0.9836 | 0.9513 | 0.6608 | 0.9823 | 0.9807 | 0.9807 | 0.9721 |
| ft-scratch | 0.9165 | 0.7752 | 0.8400 | 8.3% | 0.9755 | 0.9456 | 0.7773 | 0.9767 | 0.9634 | 0.9574 | 0.9470 |

What the dials say: focal loss is the clear winner (fewer misses on TREC-06 at a lower false-alarm rate, and
the best phishing-vs-spam separation); inverse-frequency class weights buy recall at the cost of precision;
the 2,000-row synthetic supplement does not help (reported, as promised, even though it does not move anything);
halving the stack to 12 layers costs ~5 F1 points on TREC-06 for ~2x faster inference; and scratch
initialisation is 8 F1 points behind the same model with MLM pretraining, which is what pretraining buys.
`ft-focal` was selected on validation alone, before the holdouts were read, and is the run evaluated at top
level (RUN_LOG 22), exported to `model_export/ft-focal/` + `app/` and uploaded (section 10).

Not run, optional if time allows: `--freeze-layers 12` (forgetting vs cost), an epochs sweep (`--epochs 1` /
`5`), `--num-labels 2` (binary baseline comparable with the binary-era plan).

---

## 10. Deployment

| Item | Value |
|---|---|
| Model repo | `BuildITSystems/phishing-detector-encoder` (private): `model.safetensors` (bf16), `config.json`, `tokenizer.json`, `README.md` model card |
| Space | `BuildITSystems/phishing-detector-demo` (private), Gradio SDK 6.29.1, hardware `zero-a10g` (ZeroGPU) |
| Space stack | **torch 2.13.0 on Python 3.12.12**: ZeroGPU supports torch 2.8.0-2.13.0 and Python 3.12.12 / 3.10.13 only, and no torch.compile. Training uses 2.14.1 on 3.14; safetensors makes the weights portable. Gradio is pinned by `sdk_version` in `app/README.md`, not in `requirements.txt`. |
| Weights at startup | The three weight files are **bundled into the Space** (`upload_to_hf.py --bundle-weights`, the mode used), so no secret is needed and the Space works even while the model repo is private. `app.py` can alternatively download them from the repo named by the `PHISH_MODEL_REPO` variable with an `HF_TOKEN` read-token secret. |
| UI | Subject + body in; 3-class `gr.Label`, "Prediction Time (s)" (`timeit.default_timer`), gradient x input token highlights, rule-based cues shown for context only (they never feed the probability). |

**Before class: make both repos public** (the instructor needs the model link and a Space that opens without
an invite), then open the Space once so ZeroGPU is warm. `create_repo(exist_ok=True)` does not change the
visibility of an existing repo, so do it in each repo's Settings page on huggingface.co, or:

```bash
.venv/bin/python -c "from huggingface_hub import HfApi; a=HfApi(); a.update_repo_settings('BuildITSystems/phishing-detector-encoder', private=False); a.update_repo_settings('BuildITSystems/phishing-detector-demo', repo_type='space', private=False)"
```

`app/README.md` (the Space card) already links the model repo.

---

## 11. Instructor requirement mapping

| Requirement (`reference/INSTRUCTOR_REQUIREMENTS.md`) | Where it is satisfied |
|---|---|
| Classifier, not regressor | 3-class softmax classifier + binary roll-up (section 1) |
| Dataset from `torchvision.datasets` | **Deviation, agreed text variant**: there is no torchvision email dataset. Public Hugging Face datasets loaded with `datasets` (section 7); the PyTorch data pipeline is kept (DataLoader below). |
| Problem statement: input, classes, challenge | Section 1 |
| Algorithm and architecture, >= 2 layers | 24-layer transformer encoder, `src/model.py`; justification in `docs/research/research_model-math.md` and RUN_LOG 0 |
| Loss function | Partial-label NLL (= cross-entropy for fully-labelled rows) with `--loss cross_entropy / weighted_cross_entropy / focal`; MLM cross-entropy on masked tokens for pretraining |
| Optimizer | AdamW (fused, no decay on biases/LayerNorm), grad clip 1.0, bf16 autocast |
| Epochs, early stopping / LR scheduling | Fine-tune: early stopping on val F1 (patience 2), warmup + linear decay. Pretrain: warmup + cosine to 10%, val MLM loss every 500 steps |
| Justify choices | This README, RUN_LOG.md, `docs/research/` |
| `DataLoader` with batching, shuffling, preprocessing | `finetune.make_loader`: `torch.utils.data.DataLoader` with a seeded length-bucketed shuffled `batch_sampler` and a padding collate; tokenisation as the preprocessing transform (RUN_LOG 8: moved from a numpy sampler on 2026-10-08 13:55, losses verified identical, explicit `generator` to keep the dropout stream unchanged) |
| Evaluate on an unseen test set with standard metrics | `evaluate.py` on TREC-06 and test_holdout3: accuracy, precision, recall, F1, ROC-AUC, PR-AUC, confusion matrices (section 2.1) |
| Inference timer, seconds per single prediction | `evaluate.py` -> `reports/latency.json` (GPU and CPU, batch 1); `smoke_test.py`; the app's "Prediction Time (s)" |
| Gradio UI: input -> label + inference time | `app/app.py` (text input instead of an image upload) |
| HF Spaces with `app.py`, weights, `requirements.txt` | Section 10; `upload_to_hf.py` |
| Class: dataset, trained model, performance results | Sections 2, 7; `reports/` |
| Class: model on Hugging Face, link to instructor | `https://huggingface.co/BuildITSystems/phishing-detector-encoder` (make public first) |
| Class: live Gradio app on a Space, not gradio.live | `https://huggingface.co/spaces/BuildITSystems/phishing-detector-demo` |
| Class: Space has app.py, model.py, requirements.txt (pinned torch/gradio) + weights | `app/`; torch pinned in requirements.txt, gradio pinned by `sdk_version` |
| Class: show experimentation (layers, epochs, loss), record each run | Section 9, `reports/experiments.jsonl`, `finetune.py list`, RUN_LOG.md |
| Class: prediction time per sample vs 0.074 s, GPU and CPU | Section 2.1: 10.1 ms GPU bf16, 459 ms CPU fp32 per email (`reports/latency.json`) |
| Class: presentation covers data, algorithm, performance, challenges | RUN_LOG.md (OOM tuning, leakage, rejected datasets, reviewer-found bugs), sections 2, 7, 8, 12 |
| Class: synthetic data allowed | `make_synthetic.py`, `--synthetic on`, reported with and without |
| Class: one strong model, not an architecture comparison | One architecture (24L encoder); the six runs of section 9 vary training dials only, and `ft-focal` is the one deployed |
| Notebook in the instructor's 9-step pattern | Section 5.10 |

---

## 12. Known limitations and ethics

- **Defensive use only.** It flags mail for a human; it is not a verdict, not a replacement for SPF/DKIM/DMARC,
  sender reputation, link or attachment scanning, and must not be used to tune lures against.
- **Old corpora.** Training mail is English, 1995-2022 with the bulk 1998-2008 (Enron, TREC, CEAS,
  SpamAssassin). Modern lures (MFA fatigue, QR codes, chat-app phishing) and non-English mail are out of
  distribution; the synthetic supplement only nudges this.
- **Spam vs phishing follows provenance**, not per-message review: phishing = Nazario + Nigerian fraud +
  synthetic lures; spam = SpamAssassin and Enron-Spam spam folders. Only 3,150 real phishing rows carry the
  distinction; 62,805 TREC/CEAS positives train as "unwanted" only. Trust the roll-up first.
- **Length confound.** Legitimate emails average about twice the tokens of unwanted ones in every split
  (TREC-06: 842 vs 283). Truncation at 512 tokens blunts it; `evaluate.py` checks it in the error analysis.
- **Truncation.** About 22% of emails exceed 512 tokens (~1,800 characters) and are judged on their opening.
- **Test-set caveat.** About 47% of TREC-06 ham is academic mailing-list traffic, which is not typical business mail.
- **CPU latency.** 699 ms per email on CPU vs the 74 ms benchmark; the benchmark is met on GPU only.
- **Privacy.** Enron mail is real correspondence released by FERC; nothing pasted into the Space is stored,
  but do not paste secrets or client content.

---

## 13. Troubleshooting

| Symptom | Cause and fix |
|---|---|
| Pretraining exits 3 ("CUDA OOM" or "VRAM guard") | Another process took VRAM (browser, game, a second job) or the 20 GiB guard tripped. A checkpoint was saved. Check `nvidia-smi`, close the culprit, then `RESUME=1 MICRO_BATCH=24 scripts/launch_pretrain.sh`. The run's own peak is ~17.9 GiB reserved and the desktop adds ~2 GiB, so total device use sits near 20 GB. If it recurs, resume directly with `$PY src/pretrain.py train --run-name large-24L --resume --micro-batch 24 --grad-checkpointing` (8.06 GiB measured, ~25% slower). Accumulation = round(256 / micro-batch), so of the micro-batches that fit in 24 GB only 24 (x11) and 22 (x12) keep the effective batch at 264; 16 gives 256 and is refused on resume. |
| Resume refused, exit 2, "'effective_batch' differs" | `launch_pretrain.sh` defaults to `MICRO_BATCH=16`; pass `MICRO_BATCH=24`. |
| Resume refused, "'pretrain'/'schedule'/'data' differs" | `config.yaml` `pretrain:`/`model:`/`seed` was edited, or `pretrain.parquet` / `tokenizer.json` changed. Revert the edit; see section 6. |
| `launch_pretrain.sh`: "tmux session already exists" | The finished pane waits for Enter: `tmux attach -t phish-pretrain` and press Enter, or `tmux kill-session -t phish-pretrain` once the trainer has exited. |
| Fine-tune crashed or was stopped | `$PY src/finetune.py train --run-name <run> --resume` (restores optimizer, RNG and batch position). |
| Fine-tune picked the wrong pretrained model | Pass `--init pretrained:checkpoints/pretrain/large-24L/final`; the default is the newest `*/final` by mtime and `dev/final` exists. |
| Install resolves a Python 3.15 interpreter | torch.compile, triton 3.8.0 and `spaces` all exclude 3.15. Recreate with `uv venv --python 3.14.7 .venv`. |
| Install fails on fsspec | datasets 5.1.0 caps `fsspec<=2026.7.0`; keep `fsspec==2026.7.0` from requirements-train.txt and never upgrade it alone. |
| Install pulls cu132 wheels | `--torch-backend=auto` was used; reinstall with `--torch-backend=cu130`. |
| huggingface_hub 2.x conflict | tokenizers 0.23.2 needs `<2.0`; keep 1.33.0. |
| `export.py` refuses: metrics name another checkpoint | Run `evaluate.py run --split test,test3` on the same `best/` first. |
| Space build fails on torch | ZeroGPU allows torch <= 2.13.0 and Python 3.12.12; keep `app/requirements.txt` and the `python_version` front matter as they are. |
| Space starts but cannot load weights | `HF_TOKEN` secret missing or not a read token for the model repo, or `PHISH_MODEL_REPO` variable wrong; re-run the upload with `HF_TOKEN_FOR_SPACE` set, or use `--bundle-weights`. |
| Any Python command while pretraining runs | Prefix with `CUDA_VISIBLE_DEVICES=""`; never run `smoke_test.py`, a GPU fine-tune or GPU evaluation alongside it. |