Refresh presentation bundle (RUN_LOG 25, README 5.11, builder --colab) and add the Colab notebook
07cdb81 verified 15 days ago
79.2 kB
# RUN_LOG — BITS Phishing Detector
Running record of what was run, with what settings, what happened, and what was fixed.
Newest entries at the bottom. This becomes the "challenges" section of the presentation.
---
## 0. 2026-10-07 — Plan confirmation, environment, pins
**Machine.** Fedora 44 KDE, RTX 3090 Ti 24 GB (sm_86), driver 615.71.09 (CUDA 13.4), Ryzen 9 5950X,
40 GB RAM, 741 GB free. System Python 3.14.7, uv 0.12.5. ~1.9 GB VRAM already used by the desktop.

**Decision: Python 3.14 stays.** Checked live: torch 2.14.1 and every other pinned package ship
cp314 (or abi3 / pure) Linux wheels; torch.compile is allowed on 3.14 (only 3.15 raises); triton 3.8.0
requires <3.15. No Python update needed. (uv also downloaded a managed CPython 3.12.14 as a side
effect of dry-resolving the Space requirements; harmless, kept for testing the Space stack locally.)

**Decision: torch 2.14.1, CUDA 13.0 build.** The plain PyPI Linux wheel *is* the cu130 build
(cuda-toolkit 13.0.3, cuDNN 9.24, NCCL 2.30.7). Driver 615 (CUDA 13.4) runs it by backward compatibility.
Install with `uv pip install --torch-backend=cu130`; never `--torch-backend=auto` (uv 0.12.5 would pick
cu132 on this driver).

**Constraint found: ZeroGPU pins the Space stack.** ZeroGPU supports torch 2.8.0 – 2.13.0 and Python
3.12.12 / 3.10.13 only, and does not support torch.compile. So the Space uses torch 2.13.0 on Python
3.12.12 while training uses 2.14.1 on 3.14. Weights are safetensors, so this is fine. Also: a private
model repo cannot be preloaded by the Space; weights download at startup with an HF_TOKEN secret.
**Constraint found: huggingface_hub must stay on 1.x.** tokenizers 0.23.2 requires huggingface-hub<2.0,
so huggingface_hub 1.33.0 (ships the `hf` CLI) everywhere.

**Data problems found (before downloading anything):**
- `it4lia/PhishingEmailCuratedDatasets_Cleaned` has **no body text** (subjects/headers + 53 numeric
  features). Unusable for a text model. Excluded.
- `zefang-liu/phishing-email-dataset` is a lowercased re-packaging already 91.9% contained in
  `puyang2025/seven-phishing-email-datasets` (one 17 MB row, 533 literal "empty" rows, and 20% of its
  "Safe" rows contain the token "enron" vs 0% of phishing: a shortcut). Excluded.
- The primary set's own train/test split leaks: 5.1% of test rows are exact duplicates of train rows,
  ~40% have a near-duplicate. We build our own split.
- Two sub-sources are Enron-derived: 89% of TREC-05 ham and 86% of Enron-Spam ham appear in the
  `corbt/enron-emails` pretraining corpus. Neither can be the test set.
- `corbt/enron-emails` is 52.7% exact duplicates (517,401 -> 244,496 distinct); ~117M tokens remain,
  about 230k packed 512-token sequences per epoch, so one epoch is ~1,144 optimizer steps at batch 256.
- No dataset separates "spam" from "phishing". Every label-1 row comes from a spam-track corpus.
  Mapping spam->0 would leave ~0 positives. **Proposed:** positive class = "phishing/spam (unwanted)",
  plus a credential-lure slice of the test set reported separately. PENDING Nathan's confirmation.

**Decision: hold out TREC-06 entirely as the test set** (16,259 rows after dropping 141 near-empty
rows; 12,394 legitimate / 3,865 positive = 23.8%). It is Enron-clean (5 rows mention "enron", 0
enron.com addresses). Caveats: ~47% of its ham is academic mailing-list traffic; its spam headers were
rewritten by TREC, so the model input is subject + body only. 1,766 training rows that near-duplicate
TREC-06 templates get removed from the training pool.

**Model math.** 24L / d1024 / 16 heads / FFN 4096 / vocab 32768 / seq 512 = 337,476,608 params
(pretraining) / 337,443,842 (classifier). AdamW state 5.0 GiB. Micro-batch 16 without checkpointing
is predicted at ~16.6 GiB total (7.4 GiB headroom) -> inside the 14–18 GB budget. Predicted
throughput 12k–18k tokens/s (30–45% MFU of 80 TFLOPS bf16) -> 2.3–3.5 h per 150M-token epoch.
Inference: ~14 ms per 512-token email on the 3090 Ti; ~0.3 s on the CPU in fp32 (the instructor's
0.074 s/sample benchmark is met on GPU, not on CPU at full length).

**Status.** Project scaffolded, venv created on 3.14.7, pinned install started (logs/install.log).
Next: verify GPU from torch, then data_prep -> tokenizer -> model -> smoke test -> pause for numbers.
**Step 2 result (22:45).** Pinned install succeeded (180 packages, all binary wheels; venv 6.1 GB).
From inside the venv: torch 2.14.1+cu130, CUDA 13.0 build, cuDNN 9.24; device NVIDIA GeForce RTX
3090 Ti, sm_86, 23.60 GiB, bf16 supported; SDPA Flash / memory-efficient / math backends all run
(Flash with no mask, memory-efficient with a bool padding mask); bf16 4096^3 matmul measured
63.9 TFLOPS (spec ~80); torch.compile works on Python 3.14 (triton built its driver stub with a
harmless _POSIX_C_SOURCE warning); free VRAM at idle with the desktop up: 21.5 GiB. All pinned
libraries import; config.yaml parses. `hf` CLI present, not logged in yet (Nathan to run `hf auth login`).
**Agent policy (per Nathan, 21:14).** Model and effort are set per agent from here on: mechanical
lookups -> Sonnet/low; judgment-heavy research -> Opus or Fable/medium-high; math, core code and
adversarial verification -> Fable/high. The step-1 research run inherited Fable for all 7 agents.
## 1. 2026-10-07 23:41 — Data build (data_prep.py build + check)

**Run.** `.venv/bin/python src/data_prep.py build` then `check`. 2 min 20 s wall, peak RSS 6.9 GB,
HF cache 1.5 GB (puyang2025 @ b7f3f8d9, corbt @ cfc06c75). Every asserted leakage check is 0 in
both the build-time recount and the independent `check` over the written files.

| split | rows | positives | base rate | sources |
|---|---|---|---|---|
| finetune_train | 157,113 | 75,675 | 48.2% | TREC-05 46,386 / TREC-07 47,233 / CEAS-08 34,178 / Enron 24,223 / Assassin 5,093 |
| finetune_val | 17,643 | 8,642 | 49.0% | same five, grouped by near-dup cluster so no template straddles train/val |
| test_holdout (TREC-06) | 16,302 | 3,894 | 23.9% | never touched by tokenizer, pretraining, or fine-tuning |
| pretrain (MLM) | 226,273 docs | - | - | corbt/enron-emails after dedup + quote stripping; 408M chars |
**What was removed and why.** 715 labelled rows with <20 alphanumerics (91% of them TREC-05/TREC-06
garbage spam rows). Ling excluded (2,859 academic rows). 18 rows in 8 groups where the identical body
carries both labels (Enron ham vs TREC-05 spam newsletters) — dropped as label noise. 6,715 exact
duplicates in the pool. **1,652 training rows removed as near-duplicates of TREC-06 templates**
(TREC-05 spam 863, Enron spam 456, TREC-07 spam 292): these are spam templates shared across the
TREC years; keeping them would let the model memorise test-set templates. corbt: 517,401 rows ->
272,905 exact duplicates removed (52.7%), 14,027 too short after quote stripping, 4,004 matching a
val email removed, 85 near-duplicates of holdout emails removed, 35,172 overlapping fine-tune TRAIN
emails kept (allowed: those are Enron ham). Quote stripping cut 21,832 bodies (kept 89% of all
characters; the >=200-alnum guard left 70,907 marked bodies intact).
**Deviation from the research estimate.** Holdout is 16,302 rows (research: 16,259) because the
<20-alnum filter is applied to subject+body rather than body alone: 43 TREC-06 rows have a real
subject and a near-empty body, which is a legitimate model input. Pretraining corpus is 408M chars
(research: ~466M) because quote stripping and the val screen remove more than the estimate assumed.
**Problem noted.** The val split has 5,336 near-duplicate links into the pretraining corpus
(883 val emails, Enron boilerplate). Counted, not removed: val is only used for model selection
and the holdout is the real gate. Exact val matches (4,004) were removed from pretraining.
## 2. 2026-10-07 23:43 — Tokenizer (tokenizer.py train + stats)
**Run.** Byte-level BPE, NFKC only (case preserved on purpose: ALL-CAPS is a spam signal), vocab 32,768,
min_frequency 2, specials [PAD]=0 [UNK]=1 [CLS]=2 [SEP]=3 [MASK]=4. Trained on pretrain.parquet
(226,273 docs, 408M chars) + finetune_train.parquet (157,113 docs, 278M chars) only; val and test are
refused by name and an exact-hash tripwire against the holdout found 0 collisions. Training took 28 s
(49 s wall incl. the tripwire), 3.6 GB RSS. Saved data/processed/tokenizer.json.
| split | rows | mean tok | median | p95 | chars/tok | % > 512 | mean tok legit / positive |
|---|---:|---:|---:|---:|---:|---:|---|
| pretrain (20k sample) | 20,000 | 476 | 198 | 1,560 | 3.76 | 20.5% | - |
| finetune_train | 157,113 | 487 | 227 | 1,540 | 3.63 | 24.1% | 621 / 343 |
| finetune_val | 17,643 | 454 | 200 | 1,421 | 3.58 | 22.3% | 594 / 307 |
| test_holdout | 16,302 | 709 | 244 | 1,254 | 3.43 | 21.9% | 842 / 283 |

**Observations.** [UNK] rate is 0 by construction (full 256-byte alphabet). Round trips are exact
except for NFKC folding (full-width "Ｍｉｃｒｏｓｏｆｔ" -> "Microsoft", a deliberate anti-evasion
property). About 22% of emails are longer than 512 tokens and get truncated; the pretraining corpus
is ~108M tokens at 3.76 chars/token. **Length is a confound:** legitimate emails average ~2x the
tokens of positives in every split (holdout 842 vs 283). Truncation at 512 blunts this, and the
error analysis must check whether the classifier leans on length. Outliers exist (one 886,635-token
email in train) and are harmless under truncation.

## 3. 2026-10-07 23:46 — Smoke test (smoke_test.py, real packed text, reports/smoke_test.json)

**Model.** 337,476,608 params (MLM, tied head) / 337,443,842 (classifier) — exactly the research
figure. 3,000 real pretraining docs -> 1,592,926 tokens -> 3,111 packed 512-token sequences; in-code
no-leak check: 0 of 3,000 docs match a holdout email.

**VRAM (bf16 autocast, fused AdamW, real gradient-accumulation pattern, desktop using 2.08 GiB):**

| micro-batch | checkpointing | peak reserved | total device used | tokens/s |
|---|---|---|---|---|
| 8 | off | 9.71 GiB | 11.85 GiB | 21,493 |
| 12 | off | 11.78 GiB | 13.92 GiB | 23,487 |
| **16** | off | **13.65 GiB** | 15.80 GiB | **23,642** |
| 24 | off | 17.65 GiB | 19.80 GiB | 25,227 |
| 24 | on | 8.06 GiB | 10.21 GiB | 19,156 |

The finder picked 24 (largest in the 14–18 GiB band) but total device use then sits at 19.8 GiB,
0.2 GiB under the 20 GiB "never exceed" line. Micro-batch 16 is 6% slower, gives an exact effective
batch of 256 (24 x 11 = 264), and leaves ~8 GiB of headroom for desktop spikes. **Recommendation:
micro-batch 16 x 16 accumulation for the long run.** Checkpointing is not needed for memory and
costs ~25% throughput; keep it off.

**Throughput / ETA.** 25,187 tok/s at B=24 (23,642 at B=16). Measured 0.2595 tokens/char on the
packed sample -> the 408M-char corpus is ~105.8M tokens/epoch = ~807 optimizer steps at 256 x 512.
About 1.2–1.3 h per epoch; 3 epochs ~3.75 h, 6 epochs ~7.5 h. Prediction was 2.3–3.5 h/epoch at
12k–18k tok/s: the real machine is ~1.5x faster than the 30–45% MFU assumption (25k tok/s ≈ 63% MFU
of the 80 TFLOPS peak).

**SDPA.** Flash runs on the mask-free packed path; memory-efficient runs with the bool padding mask;
Flash correctly refuses a mask (no silent math fallback).

**Convergence.** 30 steps at B=24, LR 1e-4, 10-step warmup, real text: loss 10.58 -> 7.67
(ln V = 10.40). The pipeline learns.

**Inference latency (batch 1, 512 tokens).** GPU bf16: 10.7 ms (0.14x the instructor's 74 ms).
CPU fp32, 16 threads: 699 ms at 512 tokens, 319 ms at 128 tokens (9.4x / 4.3x the benchmark).
The research's 336 ms / 79 ms CPU estimate was 2–4x too optimistic; CPU inference is per-op-overhead
bound, so truncation barely helps. **The 74 ms benchmark is met on GPU only -> the Space must be
ZeroGPU, not CPU-basic.**

**Challenges found by the reviewers before this run (fixed):** the first finder measured peak
VRAM with a single fwd/bwd/opt step and under-reported the real accumulation peak by one fp32
gradient copy (1.26 GiB); the first MLM-corpus decontamination used exact keys only and missed
near-duplicate (quoted/forwarded, 419-template) variants of holdout emails; floor-division
accumulation would have reported effective batch 256 while training at 240 for B=24;
safetensors refuses the tied state_dict unless the alias key is dropped (helper added);
the tied decoder leaks gradient into the [PAD] embedding row (post-accumulate hook added).
## 4. 2026-10-08 00:40 — Step-4 code build (pretrain / finetune / evaluate / export / app / upload / synthetic)
**What was built** (each file by one agent, then an independent adversarial reviewer who ran it, then
a fix pass): `src/pretrain.py` (resumable MLM trainer, 1,019 lines), `src/finetune.py` (classifier
trainer with the instructor's dials as flags; experiments.jsonl record per run), `src/evaluate.py`
(the only script allowed to open the TREC-06 holdout; metrics, latency, plots, error analysis),
`src/export.py` (bf16 safetensors + model card + byte-identical app/model.py), `app/app.py` +
`requirements.txt` + `README.md` (Gradio 6 / ZeroGPU app with saliency and rule-based signals),
`scripts/upload_to_hf.py` (dry-run by default, `--yes` required), `src/make_synthetic.py`
(2,000 template-generated rows, 8 positive + 8 legitimate business categories, source=synthetic).

**Verified by running.** pretrain: fresh run -> kill -> `--resume` reproduces the exact data order
(window indices checked against the seeded permutation), optimizer step counter and LR; SIGINT saves
a checkpoint and exits 0; SIGKILL between checkpoints is recovered on resume (orphan metrics rows
dropped); a forced OOM saves and exits 3; a 5 GiB test cap trips the VRAM guard; `--max-hours` stops
cleanly; resume refuses a changed plan or model shape. The full token cache was built as a side
effect: 107,498,500 train tokens + 1,021,888 val tokens (0.266 tok/char), 42.8 s. finetune: resume
reproduces batch order; `--n-layers 4` loads only the first four pretrained layers; focal(gamma=0)
== cross-entropy; the holdout parquet is never opened (inotify-verified). 14 agents, 451 tool calls.

**Bugs the reviewers caught and the fixers closed (for the challenges slide):** a `--lr` override on
`--resume` was silently discarded by the restored optimizer state but recorded in the experiment log
as if applied; resuming a finished fine-tune run trained one extra step at lr 0; changing frozen
layers on resume crashed with a raw torch error; metrics CSVs kept orphan rows after a hard crash;
evaluate/export and app each had 2–3 majors (details in docs/research/ and the agents' reports).

**Changes I made myself after reading pretrain.py:** (1) the OOM handler now covers the optimizer
step and the eval pass too, not just the forward/backward loop, so a desktop VRAM spike during the
500-step eval cannot lose 45 minutes of work; (2) a resume with nothing left to do no longer
overwrites run_summary.json; (3) `--resume` compares `--limit-docs` with the checkpoint before
tokenising anything; (4) **every packed window now starts with [CLS]** (511 stream tokens follow),
as RoBERTa prepends `<s>`: the reviewer found 0 occurrences of [CLS] in the 107M-token stream,
which would have left the classifier's pooled position untrained until fine-tuning.
**Known gaps going into the long run:** the real 337M fine-tune, evaluate and export have only run on
tiny dev models/subsets; `--synthetic on` path untested end to end; ZeroGPU runtime behaviour
untested until the first Space build. Dev checkpoints from the self-tests sit in
checkpoints/{pretrain,finetune}/dev* (~15 GB) pending Nathan's OK to delete.
## 5. 2026-10-08 12:28 — Pretraining run `large-24L` launched (Nathan's settings)
**Decisions (Nathan, 2026-10-08).** Micro-batch 24 (17.65 GiB reserved measured; total GPU use ~20 GB
with the desktop). 12 epochs: ~14 h at 25k tok/s, 9,552 optimizer steps, warmup 573 steps, cosine to
10% of 1.5e-4. Rationale: dynamic masking makes every pass different and data-constrained scaling
results show repeated passes keep helping with diminishing returns; val MLM loss (64 fixed windows
from the held-out last 1% of documents) is evaluated every 500 steps and the run is stopped if it
rises. Positive class is named **"Spam/Phishing"** in the app, metrics and model card (the sources do
not separate spam from phishing). Hugging Face login confirmed (BuildITSystems).
**Launch.** `EPOCHS=12 MICRO_BATCH=24 scripts/launch_pretrain.sh` -> tmux session `phish-pretrain`,
stdout in logs/pretrain_large-24L_20261008-122800.out. Trainer: 210,368 packed windows of
[CLS] + 511 tokens (107.7M tokens/epoch), 796 steps/epoch, micro-batch 24 x 11 accumulation =
**effective batch 264** (256 is not divisible by 24; the realised batch is used in every number),
checkpoints every 500 steps (keep 3, ~3.8 GB each) + final/. Token cache reused (0 s). GPU at
launch: 20.3 GB used, 98% util, 365 W.
## 6. 2026-10-08 ~13:00 — Scope change: three-way labels (legitimate / phishing / spam)
**Nathan's decision.** The classifier should output legitimate, phishing, or spam. Our current positives
cannot be split (TREC-05/07 and CEAS-08 labelled everything unwanted as "spam"), so phishing-labelled text
from separate provenance is being added: the Champa et al. raw corpus mirror on the Hub
(`kudzaiprichard/aura-phishing-email-corpus`, CC-BY-4.0, byte-identical to Zenodo 8339691) with
`Nazario.csv` (1,565 phishing, 1995–2022) and `Nigerian_Fraud.csv` (3,332 advance-fee fraud), plus our
synthetic BITS-style lures. Spam class from provenance: SpamAssassin, Enron-Spam and (if included) Ling
spam folders. The 250k-row `locuoco/the-biggest-spam-ham-phish-email-dataset-300000` (labels ham/phish/
spam, provenance undocumented) is inspected first and used only if its labels agree with known provenance
and it adds rows we do not have, after deduplication against every split and the TREC-06 holdout.

**Design.** One encoder, one 3-class head, partial-label loss: rows with clean provenance train with
3-class cross-entropy; the ~70k TREC/CEAS positives train on the probability mass of {phishing, spam}
("unwanted") so they still shape the legitimate-vs-unwanted decision. Binary roll-up P(unwanted) =
P(phishing)+P(spam) keeps TREC-06 as the clean binary test; a new `test_holdout3` slice of clean-
provenance rows (drawn from sources never seen by the tokenizer or the pretraining corpus) gives the
3x3 confusion matrix. `num_labels` lives under `finetune:` in config.yaml, not under `model:`, because
the live pretraining run refuses to resume if its model snapshot changes.

## 7. 2026-10-08 13:21 — Three-way data build (build3) and the 300k-set verdict

**300k set: rejected after inspection** (`docs/research/bigshp_research.md`). `locuoco/the-biggest-spam-ham-phish-
email-dataset-300000` is 365,448 rows, not deduplicated (51.5% within-file duplicates), and is a union of
Kaggle re-packagings of corpora we already hold: 97.6% of its ham, 92.5% of its phish and 90.2% of its
spam rows match our text. Its ham/spam labels agree with our provenance 99.9%/99.8% because they are our
labels copied back; its **phish label agrees only 60.5%**: 14,660 "phish" rows are SpamAssassin/Enron-Spam
spam-folder mail, 17,714 are TREC/CEAS spam-track rows, and only 4,784 are confirmed phishing provenance
(Nazario/Nigerian, which we take raw from the Champa mirror). **36,020 rows (9.9%) are copies of our frozen
TREC-06 test set.** Every "phish" row went through a stopword-stripping pipeline that ham/spam rows did
not, a preprocessing artifact that is a near-perfect label proxy and would not transfer to real inbound
mail. Decision: not one row ingested; `data.threeway.bigshp.enabled: false` with the verdict recorded.

**Champa mirror ingested** (`kudzaiprichard/aura-phishing-email-corpus`, CC-BY-4.0): Nazario 1,565 rows ->
1,507 kept; Nigerian_Fraud 3,332 -> 2,504 kept. Drops worth knowing: 235 Nigerian-fraud rows are near-
duplicates of TREC-06 spam (419 letters recur across corpora) and 13 match holdout keys exactly: all
removed so the TREC-06 test stays clean; 424 more were dropped because their clusters touched both train
and val.
**Three-way splits (label3: 0 legit / 1 phishing / 2 spam / -1 unwanted-unsplit):**
| split | rows | legit | phishing | spam | unwanted (TREC/CEAS positives) |
|---|---|---|---|---|---|
| finetune_train | 160,263 | 81,438 | 3,150 | 12,870 | 62,805 |
| finetune_val | 15,284 | 7,002 | 340 | 763 | 7,179 |
| test_holdout (TREC-06, unchanged rows) | 16,302 | 12,408 | - | - | 3,894 |
| **test_holdout3 (new, clean provenance)** | 3,220 | 1,999 | 521 | 700 | - |
test_holdout3 legit = CEAS-08 736 + TREC-07 1,099 + Assassin 164 (never Enron: no pretraining overlap);
spam = Enron-Spam 628 + Assassin 72; phishing = Nazario 222 + Nigerian 299. All of it came from
finetune_val (never seen by the tokenizer) or the new sources (never seen by anything). finetune_val
shrank from 17,643 to 15,284 rows, so new val F1 values are not strictly comparable with the binary runs.
Leakage: every asserted check is 0 in `check3` and the binary `check` still passes on the rewritten files.
Existing rows in train/test_holdout are byte-identical to the binary build (data/processed/binary_snapshot).
**Honest caveat.** Only 3,150 fully-labelled phishing training rows (plus 1,000 synthetic lures) carry the
phishing-vs-spam distinction; 62,805 positives are trained as "unwanted" only. Expect the three-way split
to be the weaker decision; the binary roll-up on TREC-06 stays the headline.

## 8. 2026-10-08 13:55 — Fine-tune batching moved into torch.utils.data.DataLoader

The instructor's written spec requires PyTorch's DataLoader for batching and shuffling; finetune.py had
its own numpy sampler. Every batch now goes through `torch.utils.data.DataLoader` (`make_loader`): a
row-index Dataset, the seeded length-bucketed batch order as `batch_sampler`, a collate function that
pads each micro-batch to its own longest row. Verified on the CPU fixture: losses and metrics identical
to the previous code at every step. One problem found on the way: a DataLoader draws its worker base
seed from the global torch RNG when iterated, which shifted the dropout stream; passing an explicit
`generator` removed the side effect (first run differed at step 1, 1.1093 vs 1.1125; identical after).

## 9. 2026-10-08 14:15 — Dev checkpoints removed; fine-tuning queue armed

Nathan approved both. Deleted the agents' self-test checkpoints (checkpoints/pretrain/dev,
checkpoints/finetune/dev, checkpoints/finetune/dev-focal; 15.3 GB) and their two dev lines in
reports/experiments.jsonl (backup in the session scratchpad). `scripts/run_experiments.sh` is running in
tmux session `phish-finetune`, waiting for checkpoints/pretrain/large-24L/final/model.safetensors; it then
runs ft-ce (3 epochs) and the five ablations (weighted CE, focal, synthetic on, 12 layers, scratch; 2 epochs
each), evaluates every best/ on TREC-06 + the three-way holdout in one call, and exports ft-ce to
model_export/ft-ce + app/. Pretraining at step 1,100 / 9,552, loss 5.96.
## 10. 2026-10-08 14:30 — Presentation notebook and README
`FinalProject_PhishingDetector.ipynb` (generated by `scripts/build_notebook.py`, 48 cells) mirrors the
instructor's nine "Step N" headers verbatim and in order, builds real `torch.utils.data.DataLoader`s from the
processed splits in Step 2, instantiates the 337M model and prints its parameter table in Step 3, plots the
live pretraining curve in Step 4 (shows "RUN IN PROGRESS" until final/ exists), times 100 single emails
instructor-style in Step 5, shows the app files, runs the app's predict on built-in synthetic examples,
checks the HF login and dry-runs the upload. It executes top to bottom on the CPU in ~70 s with zero errors
today and prints "not available yet — produced by <command>" for the 12 artifacts the training queue will
create; re-executing it after the queue fills everything in. Reviewer catches fixed before acceptance: one
header lacked the instructor's "(Optional)", a dev-checkpoint caption stated a wrong step count, a
Challenges bullet misdescribed the uv torch-backend issue, and nothing kept the notebook off the GPU when
opened from a plain shell (now forced to CPU while the pretraining run owns the GPU).
`README.md` documents setup, every phase's commands (verified with --help), config dials, data provenance
and licences, the leakage policy, the experiment queue, deployment, the instructor-requirement mapping and
the limitations. Reviewer catches fixed: it had documented a fictional experiment grid and a notebook path
that did not exist.
## 11. 2026-10-08 19:15 — Hugging Face upload approved (deferred until the export exists)
Nathan asked for the upload. Dry run at 19:14: the plan is a private model repo
`BuildITSystems/phishing-detector-encoder` (model.safetensors, config.json, tokenizer.json, README.md) and a
private ZeroGPU Space `BuildITSystems/phishing-detector-demo` (app.py, model.py, requirements.txt, README.md,
.gitattributes + bundled weights, so no token secret is needed). It refuses to run today because
model_export/ft-ce does not exist yet (the fine-tuning queue produces it tomorrow morning). The upload is
scheduled to run automatically once the export lands and the dry run reports no problems. Pretraining at
19:14: step 4,280 / 9,552 (epoch 6 of 12), train MLM loss 2.10, on schedule for ~02:00.

## 12. 2026-10-09 02:45 — Pretraining status check (scheduled)

Run `large-24L` at step 9,020 / 9,552 (epoch 12 of 12), ETA ~03:30: about an hour behind the 14 h
estimate because each 500-step validation and 3.8 GB checkpoint write pauses training. Throughput
steady at ~24.1k tokens/s, reserved VRAM 17.94 GiB, device total 20.2 GB with the desktop, GPU 86 C.
No watchdog action, no OOM, no warnings since launch. Validation MLM loss on the held-out 1% of
documents fell monotonically through all twelve passes (6.54 -> 6.38 -> 5.81 -> 5.25 -> 3.91 -> 3.16 ->
2.78 -> 2.58 -> 2.44 -> 2.36 -> 2.27 -> 2.20 -> 2.14 -> 2.11 -> 2.07 -> 2.04 -> 2.03 -> 2.01), masked-token
accuracy 14.7% -> 62.9%, so twelve passes over the same corpus did not overfit; the biggest drop came
between steps 2,000 and 3,000 (epochs 3-4), when the model stopped predicting frequent tokens and started
using context. Train loss 1.55 (per-step, with dropout). The fine-tuning queue is waiting for final/.

## 13. 2026-10-09 03:31 — MLM pretraining run `large-24L` (complete)

**Run.** `python src/pretrain.py train --run-name large-24L`; 337,476,608 parameters (EncoderForMLM, tied head). micro-batch 24 x accumulation 11 = 264 sequences x 512 tokens per step; bf16 autocast, fused AdamW [0.9, 0.98] eps 1e-06 wd 0.01 (no decay on biases/LayerNorm), clip 1.0; cosine schedule peak 1.5e-04 -> 10% over 9552 planned steps (12 epochs x 796 steps) with 573 warmup steps; dynamic 15% masking 80/10/10, loss on masked positions only; 210,368 packed 512-token windows (226,273 docs), 1,999 val windows.

**Result.** 9552 optimizer steps (12.00 epochs), 1291.1M tokens seen, total training time 15h 03m 05s (54,186 s) over 1 invocation(s), mean 23,889 tokens/s; peak VRAM reserved 17.94 GiB / allocated 17.49 GiB; final train MLM loss 1.5237, val MLM loss 1.9863 (masked-token accuracy 62.9%).

**Artifacts.** checkpoints/pretrain/large-24L/ (metrics.csv, run_summary.json, step-*/ and final/); log /home/nathank/BITS/dev/Phishing Model/logs/pretrain-20261008-122802.log.
## 14. 2026-10-09 04:07 — Hand-off to fine-tuning (scheduled check)
Pretraining `large-24L` finished at 03:31 after 15 h 03 m: 9,552 optimizer steps, 1.29 B tokens seen
(12 passes over 107.5 M), mean 23,889 tokens/s, peak VRAM reserved 17.94 GiB, final train MLM loss 1.52,
validation MLM loss 1.99 with 62.9% masked-token accuracy (from 10.4 / ~0% at random init). final/ holds
the fp32 encoder (1.35 GB) + tokenizer + config. The watchdog saw final/ and exited without ever acting.
The queue started `ft-ce` at 03:36: 3 epochs x 5,009 steps at batch 32 (micro-batch 16), 13.1 GiB reserved,
~75 examples/s, ETA ~05:30 for this run. First validation at step 3,750 (0.75 epoch), 15,284 rows:
unwanted-vs-legitimate F1 0.9855, ROC-AUC 0.9995, PR-AUC 0.9996;
three-way slice (n=8105) macro-F1 0.9775, accuracy 0.9921, per-class F1 [legit, phishing, spam] = [0.996 0.967 0.970];
phishing-vs-spam on true unwanted rows [pp ps sp ss] = [340 0 6 757]. Val F1 is not comparable with the earlier binary dev numbers (val shrank by the 2,699 rows moved to test_holdout3).
Queue estimate: ft-ce done ~05:30, then five 2-epoch ablations (~1 h each + evaluation) -> ~11:00-11:30.

## Fine-tune run `ft-ce` — 2026-10-09 04:28
**Dials.** init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 24 layers, 0 frozen; loss cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 3 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 337,444,867 (337,444,867 trainable).
**Result.** 6250 steps, stopped by early_stopping; training time 51.5 min (eval 4.8 min of it); peak reserved 13.13 GiB. Best val at step 5000 (epoch 0), Unwanted (phishing or spam) roll-up: F1 0.9929, precision 0.9938, recall 0.9920, accuracy 0.9923, ROC-AUC 0.9996, PR-AUC 0.9996; F1-optimal threshold 0.318. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9782, accuracy 0.9922; per-class F1 Legitimate 0.9958, Phishing 0.9672, Spam 0.9716; phishing-vs-spam on true unwanted (n=1,103): phish->phish 340, phish->spam 0, spam->phish 4, spam->spam 759. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-ce/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-ce/last.

## 15. 2026-10-09 04:28 — Evaluation on the test + test3 splits (evaluate.py run, `ft-ce`)

**Checkpoint.** `/home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-ce/best` (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.
### test split (TREC-06 binary holdout): `test_holdout.parquet`, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 285.1 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9404** (232 missed of 3,894) |
| precision | 0.8610 (591 false alarms) |
| F1 | 0.8990 |
| accuracy | 0.9495 |
| specificity | 0.9524 |
| ROC-AUC | 0.9875 |
| PR-AUC | 0.9626 |
| recall @ 1% FPR | 0.7745 |
| recall @ 0.1% FPR | 0.3529 |
| best-F1 threshold | 0.821 (F1 0.9090) |
| credential-lure slice | 398 rows (61.1% labelled positive), recall 0.9959, precision 0.8996 |

**Three-way:** no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,662 flagged true positives the model called 1,294 phishing and 2,368 spam.

### test3 split (three-way holdout): `test_holdout3.parquet`, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 240.9 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.
| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9853** (18 missed of 1,221) |
| precision | 0.9877 (15 false alarms) |
| F1 | 0.9865 |
| accuracy | 0.9898 |
| specificity | 0.9925 |
| ROC-AUC | 0.9992 |
| PR-AUC | 0.9985 |
| recall @ 1% FPR | 0.9869 |
| recall @ 0.1% FPR | 0.8690 |
| best-F1 threshold | 0.411 (F1 0.9869) |
| credential-lure slice | 275 rows (97.5% labelled positive), recall 0.9925, precision 1.0000 |
**Three-way** on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9863, **macro-F1 0.9822**; Legitimate P 0.991 / R 0.992 / F1 0.992 (n=1,999); Phishing P 0.981 / R 0.973 / F1 0.977 (n=521); Spam P 0.977 / R 0.979 / F1 0.978 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,984/4/11; Phishing 9/507/5; Spam 9/6/685. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9894 (7 phishing->spam, 6 spam->phishing).
| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9449 | 0.9464 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9905 | 0.9952 |
| Enron | 628 | 0 | 0 | 628 | 0.9889 | 0.9944 |
| Nazario | 222 | 0 | 222 | 0 | 0.9505 | 0.9746 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9900 | 0.9950 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9973 | 0.9986 |

**Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms.** GPU bf16: 9.95 ms avg / 8.72 ms p50 over 100 emails (0.13x). CPU fp32 (16 threads): 514.07 ms avg / 526.19 ms p50 over 100 emails (6.95x).
**Length confound (test).** FN mean 256 tokens vs TP 287; FP 763 vs TN 848 (see error_analysis.md for the per-bucket table).
**Artifacts.** `reports/eval_ft-ce`: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py `plot`).

## Fine-tune run `ft-weighted` — 2026-10-09 05:55
**Dials.** init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 24 layers, 0 frozen; loss weighted_cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 337,444,867 (337,444,867 trainable).
**Result.** 10018 steps, stopped by epochs_done; training time 83.6 min (eval 8.0 min of it); peak reserved 13.13 GiB. Best val at step 10000 (epoch 1), Unwanted (phishing or spam) roll-up: F1 0.9937, precision 0.9911, recall 0.9964, accuracy 0.9932, ROC-AUC 0.9998, PR-AUC 0.9998; F1-optimal threshold 0.899. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9775, accuracy 0.9903; per-class F1 Legitimate 0.9944, Phishing 0.9798, Spam 0.9583; phishing-vs-spam on true unwanted (n=1,103): phish->phish 339, phish->spam 1, spam->phish 2, spam->spam 761. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-weighted/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-weighted/last.
## 16. 2026-10-09 05:55 — Evaluation on the test + test3 splits (evaluate.py run, `ft-weighted`)
**Checkpoint.** `/home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-weighted/best` (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

### test split (TREC-06 binary holdout): `test_holdout.parquet`, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 283.4 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.
| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9818** (71 missed of 3,894) |
| precision | 0.8307 (779 false alarms) |
| F1 | 0.9000 |
| accuracy | 0.9479 |
| specificity | 0.9372 |
| ROC-AUC | 0.9943 |
| PR-AUC | 0.9833 |
| recall @ 1% FPR | 0.8767 |
| recall @ 0.1% FPR | 0.6335 |
| best-F1 threshold | 0.989 (F1 0.9383) |
| credential-lure slice | 398 rows (61.1% labelled positive), recall 0.9959, precision 0.9565 |
**Three-way:** no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,823 flagged true positives the model called 801 phishing and 3,022 spam.
### test3 split (three-way holdout): `test_holdout3.parquet`, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 238.3 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9934** (8 missed of 1,221) |
| precision | 0.9790 (26 false alarms) |
| F1 | 0.9862 |
| accuracy | 0.9894 |
| specificity | 0.9870 |
| ROC-AUC | 0.9993 |
| PR-AUC | 0.9988 |
| recall @ 1% FPR | 0.9910 |
| recall @ 0.1% FPR | 0.7060 |
| best-F1 threshold | 0.728 (F1 0.9878) |
| credential-lure slice | 275 rows (97.5% labelled positive), recall 1.0000, precision 1.0000 |

**Three-way** on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9876, **macro-F1 0.9852**; Legitimate P 0.996 / R 0.987 / F1 0.991 (n=1,999); Phishing P 0.988 / R 0.987 / F1 0.988 (n=521); Spam P 0.964 / R 0.990 / F1 0.977 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,973/5/21; Phishing 2/514/5; Spam 6/1/693. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9943 (6 phishing->spam, 1 spam->phishing).

| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9619 | 0.9603 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9823 | 0.9911 |
| Enron | 628 | 0 | 0 | 628 | 0.9936 | 0.9968 |
| Nazario | 222 | 0 | 222 | 0 | 0.9820 | 0.9909 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9900 | 0.9950 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9936 | 0.9968 |
**Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms.** GPU bf16: 10.12 ms avg / 8.95 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 411.57 ms avg / 397.57 ms p50 over 100 emails (5.56x).
**Length confound (test).** FN mean 310 tokens vs TP 284; FP 1779 vs TN 782 (see error_analysis.md for the per-bucket table).

**Artifacts.** `reports/eval_ft-weighted`: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py `plot`).

## Fine-tune run `ft-focal` — 2026-10-09 07:21
**Dials.** init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 24 layers, 0 frozen; loss focal (gamma 2.0); lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 337,444,867 (337,444,867 trainable).
**Result.** 10018 steps, stopped by early_stopping; training time 83.9 min (eval 8.0 min of it); peak reserved 13.13 GiB. Best val at step 8750 (epoch 1), Unwanted (phishing or spam) roll-up: F1 0.9951, precision 0.9954, recall 0.9948, accuracy 0.9947, ROC-AUC 0.9998, PR-AUC 0.9998; F1-optimal threshold 0.421. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9869, accuracy 0.9947; per-class F1 Legitimate 0.9971, Phishing 0.9868, Spam 0.9768; phishing-vs-spam on true unwanted (n=1,103): phish->phish 339, phish->spam 1, spam->phish 1, spam->spam 762. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-focal/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-focal/last.
## 17. 2026-10-09 07:22 — Evaluation on the test + test3 splits (evaluate.py run, `ft-focal`)
**Checkpoint.** `/home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-focal/best` (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

### test split (TREC-06 binary holdout): `test_holdout.parquet`, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 282.8 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.
| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9741** (101 missed of 3,894) |
| precision | 0.8794 (520 false alarms) |
| F1 | 0.9243 |
| accuracy | 0.9619 |
| specificity | 0.9581 |
| ROC-AUC | 0.9946 |
| PR-AUC | 0.9848 |
| recall @ 1% FPR | 0.8955 |
| recall @ 0.1% FPR | 0.6497 |
| best-F1 threshold | 0.682 (F1 0.9397) |
| credential-lure slice | 398 rows (61.1% labelled positive), recall 0.9959, precision 0.9758 |
**Three-way:** no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,793 flagged true positives the model called 974 phishing and 2,819 spam.
### test3 split (three-way holdout): `test_holdout3.parquet`, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 238.7 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9836** (20 missed of 1,221) |
| precision | 0.9893 (13 false alarms) |
| F1 | 0.9864 |
| accuracy | 0.9898 |
| specificity | 0.9935 |
| ROC-AUC | 0.9993 |
| PR-AUC | 0.9989 |
| recall @ 1% FPR | 0.9869 |
| recall @ 0.1% FPR | 0.8788 |
| best-F1 threshold | 0.475 (F1 0.9877) |
| credential-lure slice | 275 rows (97.5% labelled positive), recall 1.0000, precision 1.0000 |

**Three-way** on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9885, **macro-F1 0.9867**; Legitimate P 0.990 / R 0.993 / F1 0.992 (n=1,999); Phishing P 0.996 / R 0.987 / F1 0.991 (n=521); Spam P 0.979 / R 0.976 / F1 0.977 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,986/1/12; Phishing 4/514/3; Spam 16/1/683. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9959 (4 phishing->spam, 1 spam->phishing).

| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9534 | 0.9458 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9905 | 0.9952 |
| Enron | 628 | 0 | 0 | 628 | 0.9841 | 0.9920 |
| Nazario | 222 | 0 | 222 | 0 | 0.9865 | 0.9932 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9866 | 0.9933 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9982 | 0.9991 |
**Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms.** GPU bf16: 10.34 ms avg / 9.01 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 357.98 ms avg / 340.34 ms p50 over 100 emails (4.84x).
**Length confound (test).** FN mean 322 tokens vs TP 284; FP 1237 vs TN 827 (see error_analysis.md for the per-bucket table).

**Artifacts.** `reports/eval_ft-focal`: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py `plot`).

## Fine-tune run `ft-synthetic` — 2026-10-09 08:16
**Dials.** init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 24 layers, 0 frozen; loss cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic on (2000 rows); selection val_f1_unwanted; train rows 162,263, val rows 15,284; params 337,444,867 (337,444,867 trainable).
**Result.** 6250 steps, stopped by early_stopping; training time 52.0 min (eval 4.8 min of it); peak reserved 13.13 GiB. Best val at step 5000 (epoch 0), Unwanted (phishing or spam) roll-up: F1 0.9927, precision 0.9904, recall 0.9949, accuracy 0.9920, ROC-AUC 0.9996, PR-AUC 0.9997; F1-optimal threshold 0.609. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9665, accuracy 0.9891; per-class F1 Legitimate 0.9940, Phishing 0.9352, Spam 0.9704; phishing-vs-spam on true unwanted (n=1,103): phish->phish 340, phish->spam 0, spam->phish 6, spam->spam 757. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-synthetic/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-synthetic/last.

## 18. 2026-10-09 08:16 — Evaluation on the test + test3 splits (evaluate.py run, `ft-synthetic`)

**Checkpoint.** `/home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-synthetic/best` (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.
### test split (TREC-06 binary holdout): `test_holdout.parquet`, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 282.3 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9679** (125 missed of 3,894) |
| precision | 0.8402 (717 false alarms) |
| F1 | 0.8995 |
| accuracy | 0.9483 |
| specificity | 0.9422 |
| ROC-AUC | 0.9908 |
| PR-AUC | 0.9716 |
| recall @ 1% FPR | 0.7871 |
| recall @ 0.1% FPR | 0.4779 |
| best-F1 threshold | 0.962 (F1 0.9186) |
| credential-lure slice | 398 rows (61.1% labelled positive), recall 0.9959, precision 0.9528 |

**Three-way:** no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,769 flagged true positives the model called 1,486 phishing and 2,283 spam.

### test3 split (three-way holdout): `test_holdout3.parquet`, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 238.3 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.
| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9885** (14 missed of 1,221) |
| precision | 0.9861 (17 false alarms) |
| F1 | 0.9873 |
| accuracy | 0.9904 |
| specificity | 0.9915 |
| ROC-AUC | 0.9993 |
| PR-AUC | 0.9988 |
| recall @ 1% FPR | 0.9885 |
| recall @ 0.1% FPR | 0.8395 |
| best-F1 threshold | 0.508 (F1 0.9873) |
| credential-lure slice | 275 rows (97.5% labelled positive), recall 1.0000, precision 1.0000 |
**Three-way** on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9860, **macro-F1 0.9814**; Legitimate P 0.993 / R 0.991 / F1 0.992 (n=1,999); Phishing P 0.968 / R 0.987 / F1 0.977 (n=521); Spam P 0.980 / R 0.970 / F1 0.975 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,982/6/11; Phishing 4/514/3; Spam 10/11/679. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9869 (5 phishing->spam, 11 spam->phishing).
| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9280 | 0.9296 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9851 | 0.9925 |
| Enron | 628 | 0 | 0 | 628 | 0.9873 | 0.9936 |
| Nazario | 222 | 0 | 222 | 0 | 0.9820 | 0.9909 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9900 | 0.9950 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9982 | 0.9991 |

**Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms.** GPU bf16: 10.05 ms avg / 8.9 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 481.35 ms avg / 464.28 ms p50 over 100 emails (6.50x).
**Length confound (test).** FN mean 256 tokens vs TP 286; FP 686 vs TN 854 (see error_analysis.md for the per-bucket table).
**Artifacts.** `reports/eval_ft-synthetic`: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py `plot`).

## Fine-tune run `ft-12L` — 2026-10-09 08:54
**Dials.** init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 12 layers, 0 frozen; loss cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 186,290,179 (186,290,179 trainable).
**Result.** 8750 steps, stopped by early_stopping; training time 35.3 min (eval 3.1 min of it); peak reserved 6.97 GiB. Best val at step 6250 (epoch 1), Unwanted (phishing or spam) roll-up: F1 0.9925, precision 0.9936, recall 0.9914, accuracy 0.9919, ROC-AUC 0.9995, PR-AUC 0.9996; F1-optimal threshold 0.171. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9774, accuracy 0.9916; per-class F1 Legitimate 0.9954, Phishing 0.9695, Spam 0.9673; phishing-vs-spam on true unwanted (n=1,103): phish->phish 338, phish->spam 2, spam->phish 2, spam->spam 761. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-12L/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-12L/last.

## 19. 2026-10-09 08:54 — Evaluation on the test + test3 splits (evaluate.py run, `ft-12L`)

**Checkpoint.** `/home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-12L/best` (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.
### test split (TREC-06 binary holdout): `test_holdout.parquet`, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 591.7 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 1.7 GB.

| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9461** (210 missed of 3,894) |
| precision | 0.8120 (853 false alarms) |
| F1 | 0.8739 |
| accuracy | 0.9348 |
| specificity | 0.9313 |
| ROC-AUC | 0.9836 |
| PR-AUC | 0.9513 |
| recall @ 1% FPR | 0.6608 |
| recall @ 0.1% FPR | 0.4073 |
| best-F1 threshold | 0.955 (F1 0.8852) |
| credential-lure slice | 398 rows (61.1% labelled positive), recall 0.9959, precision 0.9030 |

**Three-way:** no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,684 flagged true positives the model called 802 phishing and 2,882 spam.

### test3 split (three-way holdout): `test_holdout3.parquet`, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 501.6 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 1.7 GB.
| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9787** (26 missed of 1,221) |
| precision | 0.9860 (17 false alarms) |
| F1 | 0.9823 |
| accuracy | 0.9866 |
| specificity | 0.9915 |
| ROC-AUC | 0.9991 |
| PR-AUC | 0.9983 |
| recall @ 1% FPR | 0.9795 |
| recall @ 0.1% FPR | 0.7748 |
| best-F1 threshold | 0.169 (F1 0.9849) |
| credential-lure slice | 275 rows (97.5% labelled positive), recall 0.9888, precision 1.0000 |
**Three-way** on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9842, **macro-F1 0.9807**; Legitimate P 0.987 / R 0.991 / F1 0.989 (n=1,999); Phishing P 0.988 / R 0.973 / F1 0.981 (n=521); Spam P 0.973 / R 0.971 / F1 0.972 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,982/5/12; Phishing 7/507/7; Spam 19/1/680. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9910 (10 phishing->spam, 1 spam->phishing).
| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9619 | 0.9556 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9864 | 0.9932 |
| Enron | 628 | 0 | 0 | 628 | 0.9793 | 0.9895 |
| Nazario | 222 | 0 | 222 | 0 | 0.9685 | 0.9840 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9766 | 0.9882 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9955 | 0.9977 |

**Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms.** GPU bf16: 5.05 ms avg / 4.42 ms p50 over 100 emails (0.07x). CPU fp32 (16 threads): 246.62 ms avg / 242.67 ms p50 over 100 emails (3.33x).
**Length confound (test).** FN mean 260 tokens vs TP 286; FP 1105 vs TN 825 (see error_analysis.md for the per-bucket table).
**Artifacts.** `reports/eval_ft-12L`: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py `plot`).

## Fine-tune run `ft-scratch` — 2026-10-09 09:57
**Dials.** init scratch; 3 classes; 24 layers, 0 frozen; loss cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 337,444,867 (337,444,867 trainable).
**Result.** 7500 steps, stopped by early_stopping; training time 60.9 min (eval 5.5 min of it); peak reserved 13.13 GiB. Best val at step 5009 (epoch 0), Unwanted (phishing or spam) roll-up: F1 0.9875, precision 0.9864, recall 0.9885, accuracy 0.9864, ROC-AUC 0.9985, PR-AUC 0.9986; F1-optimal threshold 0.601. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9538, accuracy 0.9832; per-class F1 Legitimate 0.9915, Phishing 0.9398, Spam 0.9300; phishing-vs-spam on true unwanted (n=1,103): phish->phish 322, phish->spam 18, spam->phish 1, spam->spam 762. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-scratch/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-scratch/last.

## 20. 2026-10-09 09:57 — Evaluation on the test + test3 splits (evaluate.py run, `ft-scratch`)

**Checkpoint.** `/home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-scratch/best` (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.
### test split (TREC-06 binary holdout): `test_holdout.parquet`, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 282.4 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9165** (325 missed of 3,894) |
| precision | 0.7752 (1,035 false alarms) |
| F1 | 0.8400 |
| accuracy | 0.9166 |
| specificity | 0.9166 |
| ROC-AUC | 0.9755 |
| PR-AUC | 0.9456 |
| recall @ 1% FPR | 0.7773 |
| recall @ 0.1% FPR | 0.4954 |
| best-F1 threshold | 0.941 (F1 0.8772) |
| credential-lure slice | 398 rows (61.1% labelled positive), recall 0.9959, precision 0.9308 |

**Three-way:** no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,569 flagged true positives the model called 332 phishing and 3,237 spam.

### test3 split (three-way holdout): `test_holdout3.parquet`, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 237.5 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.
| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9771** (28 missed of 1,221) |
| precision | 0.9763 (29 false alarms) |
| F1 | 0.9767 |
| accuracy | 0.9823 |
| specificity | 0.9855 |
| ROC-AUC | 0.9983 |
| PR-AUC | 0.9974 |
| recall @ 1% FPR | 0.9672 |
| recall @ 0.1% FPR | 0.8354 |
| best-F1 threshold | 0.366 (F1 0.9776) |
| credential-lure slice | 275 rows (97.5% labelled positive), recall 0.9925, precision 1.0000 |
**Three-way** on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9727, **macro-F1 0.9634**; Legitimate P 0.986 / R 0.985 / F1 0.986 (n=1,999); Phishing P 0.990 / R 0.927 / F1 0.957 (n=521); Spam P 0.925 / R 0.970 / F1 0.947 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,970/3/26; Phishing 9/483/29; Spam 19/2/679. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9713 (32 phishing->spam, 3 spam->phishing).
| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9407 | 0.9330 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9796 | 0.9897 |
| Enron | 628 | 0 | 0 | 628 | 0.9777 | 0.9887 |
| Nazario | 222 | 0 | 222 | 0 | 0.8964 | 0.9454 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9498 | 0.9743 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9936 | 0.9968 |

**Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms.** GPU bf16: 10.06 ms avg / 8.92 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 354.05 ms avg / 338.14 ms p50 over 100 emails (4.78x).
**Length confound (test).** FN mean 334 tokens vs TP 280; FP 728 vs TN 855 (see error_analysis.md for the per-bucket table).
**Artifacts.** `reports/eval_ft-scratch`: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py `plot`).

## 21. 2026-10-09 09:59 — Evaluation on the test + test3 splits (evaluate.py run, `ft-ce`)

**Checkpoint.** `/home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-ce/best` (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.
### test split (TREC-06 binary holdout): `test_holdout.parquet`, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 285.2 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9404** (232 missed of 3,894) |
| precision | 0.8610 (591 false alarms) |
| F1 | 0.8990 |
| accuracy | 0.9495 |
| specificity | 0.9524 |
| ROC-AUC | 0.9875 |
| PR-AUC | 0.9626 |
| recall @ 1% FPR | 0.7745 |
| recall @ 0.1% FPR | 0.3529 |
| best-F1 threshold | 0.821 (F1 0.9090) |
| credential-lure slice | 398 rows (61.1% labelled positive), recall 0.9959, precision 0.8996 |

**Three-way:** no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,662 flagged true positives the model called 1,294 phishing and 2,368 spam.

### test3 split (three-way holdout): `test_holdout3.parquet`, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 239.6 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.
| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9853** (18 missed of 1,221) |
| precision | 0.9877 (15 false alarms) |
| F1 | 0.9865 |
| accuracy | 0.9898 |
| specificity | 0.9925 |
| ROC-AUC | 0.9992 |
| PR-AUC | 0.9985 |
| recall @ 1% FPR | 0.9869 |
| recall @ 0.1% FPR | 0.8690 |
| best-F1 threshold | 0.411 (F1 0.9869) |
| credential-lure slice | 275 rows (97.5% labelled positive), recall 0.9925, precision 1.0000 |
**Three-way** on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9863, **macro-F1 0.9822**; Legitimate P 0.991 / R 0.992 / F1 0.992 (n=1,999); Phishing P 0.981 / R 0.973 / F1 0.977 (n=521); Spam P 0.977 / R 0.979 / F1 0.978 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,984/4/11; Phishing 9/507/5; Spam 9/6/685. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9894 (7 phishing->spam, 6 spam->phishing).
| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9449 | 0.9464 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9905 | 0.9952 |
| Enron | 628 | 0 | 0 | 628 | 0.9889 | 0.9944 |
| Nazario | 222 | 0 | 222 | 0 | 0.9505 | 0.9746 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9900 | 0.9950 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9973 | 0.9986 |

**Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms.** GPU bf16: 9.89 ms avg / 8.93 ms p50 over 100 emails (0.13x). CPU fp32 (16 threads): 513.57 ms avg / 485.83 ms p50 over 100 emails (6.94x).
**Length confound (test).** FN mean 256 tokens vs TP 287; FP 763 vs TN 848 (see error_analysis.md for the per-bucket table).
**Artifacts.** `/home/nathank/BITS/dev/Phishing Model/reports`: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py `plot`).

## 22. 2026-10-09 10:46 — Evaluation on the test + test3 splits (evaluate.py run, `ft-focal`)

**Checkpoint.** `/home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-focal/best` (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.
### test split (TREC-06 binary holdout): `test_holdout.parquet`, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 284.1 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9741** (101 missed of 3,894) |
| precision | 0.8794 (520 false alarms) |
| F1 | 0.9243 |
| accuracy | 0.9619 |
| specificity | 0.9581 |
| ROC-AUC | 0.9946 |
| PR-AUC | 0.9848 |
| recall @ 1% FPR | 0.8955 |
| recall @ 0.1% FPR | 0.6497 |
| best-F1 threshold | 0.682 (F1 0.9397) |
| credential-lure slice | 398 rows (61.1% labelled positive), recall 0.9959, precision 0.9758 |

**Three-way:** no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,793 flagged true positives the model called 974 phishing and 2,819 spam.

### test3 split (three-way holdout): `test_holdout3.parquet`, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 237.6 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.
| binary roll-up metric (P(unwanted) = P(phishing) + P(spam)) | value |
|---|---:|
| **recall (unwanted)** | **0.9836** (20 missed of 1,221) |
| precision | 0.9893 (13 false alarms) |
| F1 | 0.9864 |
| accuracy | 0.9898 |
| specificity | 0.9935 |
| ROC-AUC | 0.9993 |
| PR-AUC | 0.9989 |
| recall @ 1% FPR | 0.9869 |
| recall @ 0.1% FPR | 0.8788 |
| best-F1 threshold | 0.475 (F1 0.9877) |
| credential-lure slice | 275 rows (97.5% labelled positive), recall 1.0000, precision 1.0000 |
**Three-way** on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9885, **macro-F1 0.9867**; Legitimate P 0.990 / R 0.993 / F1 0.992 (n=1,999); Phishing P 0.996 / R 0.987 / F1 0.991 (n=521); Spam P 0.979 / R 0.976 / F1 0.977 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,986/1/12; Phishing 4/514/3; Spam 16/1/683. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9959 (4 phishing->spam, 1 spam->phishing).
| source | n | Legitimate | Phishing | Spam | accuracy | macro-F1 (present classes) |
|---|---:|---:|---:|---:|---:|---:|
| Assassin | 236 | 164 | 0 | 72 | 0.9534 | 0.9458 |
| CEAS-08 | 736 | 736 | 0 | 0 | 0.9905 | 0.9952 |
| Enron | 628 | 0 | 0 | 628 | 0.9841 | 0.9920 |
| Nazario | 222 | 0 | 222 | 0 | 0.9865 | 0.9932 |
| Nigerian_Fraud | 299 | 0 | 299 | 0 | 0.9866 | 0.9933 |
| TREC-07 | 1,099 | 1,099 | 0 | 0 | 0.9982 | 0.9991 |

**Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms.** GPU bf16: 10.08 ms avg / 8.9 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 458.57 ms avg / 445.01 ms p50 over 100 emails (6.20x).
**Length confound (test).** FN mean 322 tokens vs TP 284; FP 1237 vs TN 827 (see error_analysis.md for the per-bucket table).
**Artifacts.** `/home/nathank/BITS/dev/Phishing Model/reports`: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py `plot`).

## 23. 2026-10-12 11:30 — Fine-tune queue results, model selection (`ft-focal`), export

**Queue** (`scripts/run_experiments.sh`, tmux `phish-finetune`): finished 2026-10-09 10:02:22, every step exit 0
(`ft-ce` 03:36-04:31, `ft-weighted` -05:57, `ft-focal` -07:24, `ft-synthetic` -08:19, `ft-12L` -08:56,
`ft-scratch` -09:59, then top-level evaluate + export of the pre-designated main run `ft-ce`). Log
`logs/experiments_20261008-141528.out`. Per-run evaluations in `reports/eval_<run>/` (entries 15-20, 21).

**Selection rule and result.** The queue's main run was `ft-ce` by default, but the rule stated in README section 9
was "if another run wins on val F1 of the unwanted roll-up, it becomes the main run". `ft-focal` (focal loss,
gamma 2.0) won on the validation set on both selection metrics: val F1 unwanted 0.9951 vs 0.9929 (`ft-ce`), val
three-way macro-F1 0.9869 vs 0.9782; it also had the best val phishing precision/recall (0.985 / 0.988). The
decision was taken from the validation numbers; the holdouts were evaluated once per run for reporting and
confirm it (TREC-06 F1 0.9243 vs 0.8990, recall at 1% FPR 0.8955 vs 0.7745; test3 macro-F1 0.9867 vs 0.9822).
Top-level evaluate was re-run for `ft-focal` on 2026-10-09 10:46 (entry 22, `reports/metrics*.json`,
`latency.json`, plots, `error_analysis*.md`).

**Held-out results of the deployed run `ft-focal`:**

| split | numbers |
|---|---|
| TREC-06 (16,302 rows, 23.9% unwanted, whole corpus unseen) | recall 0.9741 (101 missed), precision 0.8794 (520 false alarms, FPR 4.2%), F1 0.9243, accuracy 0.9619, ROC-AUC 0.9946, PR-AUC 0.9848, recall at 1% / 0.1% FPR 0.8955 / 0.6497 |
| test_holdout3 (3,220 rows: 1,999 / 521 / 700) | macro-F1 0.9867, accuracy 0.9885; F1 legit / phishing / spam 0.9918 / 0.9913 / 0.9771; binary roll-up F1 0.9864, ROC-AUC 0.9993; phishing-vs-spam on true unwanted 0.9959 (4 phishing->spam, 1 spam->phishing); 3x3 [[1986, 1, 12], [4, 514, 3], [16, 1, 683]] |
| latency, batch 1, 100 TREC-06 emails (mean 280 tokens) | GPU bf16 10.08 ms mean / 8.90 p50 / 13.72 p95 (0.14x the 74 ms benchmark); CPU fp32 16 threads 458.6 ms mean / 445.0 p50 / 701.1 p95 (6.20x) |
**What the ablations say** (all six on the same holdouts, README 9.1): class-weighted CE trades precision for
recall (TREC-06 recall 0.982 but FPR 6.3%); focal wins on every metric; the 2,000-row synthetic supplement
changes nothing measurable (TREC-06 F1 0.8995 vs 0.8990; val macro-F1 lower, because its 1,000 phishing lures
are template-generated); 12 layers lose ~5 F1 points on TREC-06 (0.8739) for ~2x faster inference; scratch
init loses 8 points (0.8400) and 1.7 ROC-AUC points, which is what the 15 h of MLM pretraining buys.
**Challenge recorded: TREC-06 labelling conventions.** The TREC-06 numbers are the honest, cross-corpus ones and
sit well below the in-distribution validation numbers (F1 0.92 vs 0.995). `reports/error_analysis.md` shows the
two causes: (1) TREC-06 labels as *ham* a lot of mail that looks like spam to any text model (spam quoted inside
mailing-list discussions, "please remove me from your list" replies, sweepstakes mail, "57 Million Email
Addresses - ONLY $149!!" on a developer list), which accounts for the most confident false positives; (2) its
largest block of *spam* is "ScanMail Message: virus found" gateway notices, which every run misses. Legitimate
TREC-06 mail is also ~3x longer than its spam and 24% of it is truncated at 512 tokens; false positives are
truncated 31.5% of the time vs 23.6% for true negatives. Neither is a leakage problem, but both cap what any
text-only classifier can score on this corpus, and they are the right thing to show in the presentation when
explaining the gap between 0.995 validation and 0.92 TREC-06.

**Export.** 2026-10-12 11:28: `export.py run --checkpoint checkpoints/finetune/ft-focal/best --out-dir
model_export/ft-focal --app-dir app` -> `model_export/ft-focal/` (bf16 safetensors 674.9 MB, config.json with
labels + metrics + dials, tokenizer.json, model-card README.md, export_manifest.json) and the same three files
into `app/`; `app/model.py` is byte-identical to `src/model.py` (sha1 39b3773c, asserted). This replaced the
`ft-ce` weights that the queue had copied into `app/` on 10-09 10:02; `model_export/ft-ce/` is kept for
comparison. README (sections 2.1, 2.2, 5.6-5.8, 9, 10, 11), `app/README.md` (Space card: bundled weights,
training recipe, real model-card link) and `scripts/build_notebook.py` were updated to name `ft-focal` as the
deployed run; the notebook was rebuilt.

**Gap in the record.** The 2026-10-09 10:43 scheduled check-in completed the queue review and the `ft-focal`
top-level evaluation (entry 22) but stalled before export/upload; resumed on 2026-10-12 11:27 at Nathan's
request. `find -newermt` confirmed nothing in the project changed in between.

## 24. 2026-10-12 11:35 — Hugging Face upload (private model repo + private ZeroGPU Space)

Approved by Nathan on 2026-10-08 19:15 ("can you upload the hugging face?"); deferred until an export existed.
Account: `BuildITSystems` via the `hf auth login` cache (token never printed or written anywhere).

**Dry run** 11:30 (`scripts/upload_to_hf.py --model-dir model_export/ft-focal --bundle-weights`): no problems;
plan = 4 files / 645.9 MiB to the model repo, 8 files / 645.9 MiB to the Space (weights bundled, so no
`HF_TOKEN` secret), hardware `zero-a10g`, both private.

**Live run 1** 11:31 (`--yes`): `create_repo` + `upload_folder` for the model repo succeeded (commit
`80bd85c5`: README.md model card, config.json, model.safetensors 674.9 MB bf16, tokenizer.json). The Space was
created and its `PHISH_MODEL_REPO` variable set, but `upload_folder(app/)` was refused by the Hub's README
validation: `Invalid YAML in README.md: bad indentation of a mapping entry (11:45)` — the unquoted
`short_description: From-scratch 337M encoder: legitimate / phishing / spam` contains a colon. The dry run does
not validate the card's YAML, so this was invisible until the live call. Exit 1.

**Fix** 11:34: quoted the value in `app/README.md` (55 chars, under the 60-char limit); verified with
`yaml.safe_load` and `huggingface_hub.RepoCard`. Lesson added to README 5.9.

**Live run 2** 11:35 (`--yes` again, idempotent): model-repo files already present (LFS content dedupe; commit
unchanged), Space `upload_folder` succeeded (commit `98f90d7b`). Verified read-only afterwards:

| repo | private | files |
|---|---|---|
| https://huggingface.co/BuildITSystems/phishing-detector-encoder | yes | .gitattributes, README.md, config.json, model.safetensors 674.9 MB, tokenizer.json 2.3 MB |
| https://huggingface.co/spaces/BuildITSystems/phishing-detector-demo | yes | .gitattributes, README.md (card), app.py, model.py, requirements.txt, config.json, model.safetensors 674.9 MB, tokenizer.json 2.3 MB |

Space runtime at 11:36: stage `BUILDING`, requested hardware `zero-a10g`; variable
`PHISH_MODEL_REPO=BuildITSystems/phishing-detector-encoder`; no secrets. Deployed weights = `ft-focal`
(entry 23). Before class Nathan flips both repos to public (README section 10 one-liner) and opens the Space
once so ZeroGPU is warm.

**Build and live check (2026-10-12 11:37-11:39).** Space stages: BUILDING 11:35 -> APP_STARTING 11:36 -> RUNNING
on `zero-a10g` 11:37 (about 2 minutes end to end). Verified through the private Space's Gradio API
(`gradio_client`, endpoint `/predict(subject, body)`, token from the login cache): a sample credential-lure
email ("verify your account within 24 hours", lookalike PayPal link) returned verdict **Phishing** 95.6%,
Spam 3.5%, Legitimate 0.9%, roll-up Unwanted 99.1%, in-app "Prediction Time (s)" 0.253 (cold ZeroGPU first
call), round-trip 3.6 s, with the token highlights and the rule-based signals populated. The deployment is
working end to end.

## 25. 2026-10-12 11:58 — Presentation bundle on the Hub + Colab variant of the notebook

**Why.** Nathan will present from Google Colab, where none of the project files exist and the local GPU is
irrelevant. Design: the notebook is narrative only; every prediction runs on the ZeroGPU Space
(`gr.load(..., src="spaces")` to embed the Space UI, `gradio_client` for the timing loop), and the evidence the
notebook renders (metrics, curves, experiments table, data report, run log, code) is fetched from the model repo.

**Bundle upload (approved by Nathan 2026-10-12 11:50).** `presentation/` in the private model repo
`BuildITSystems/phishing-detector-encoder`, commit `439e7b1a`: 85 files, 3.6 MB, relative paths mirrored from
the project (`reports/` JSON/PNG/MD incl. `eval_<run>/`, `checkpoints/pretrain/large-24L/{run_summary.json,
metrics.csv,final/config.json}`, `checkpoints/finetune/<run>/{metrics.csv,epochs.csv}`,
`data/processed/{splits_manifest,tokenizer_report,pretrain_tokens}.json` + `data_report.md`,
`docs/research/research_model-math.md`, `src/*.py`, `app/{app.py,model.py,requirements.txt,README.md}`,
`config.yaml`, `requirements-train.txt`, `scripts/*`, `model_export/ft-focal/{config.json,README.md,
export_manifest.json}`, `RUN_LOG.md`, `README.md`). Excluded on purpose: every parquet/npy/safetensors/log.
Scanned before upload for tokens (`hf_...`), `HF_TOKEN=` values, Nathan's e-mail address, private keys and
cloud keys: clean. The quoted emails in `error_analysis*.md` are rows of public corpora (TREC-06 etc.), as the
files themselves state. The Hub warned that `presentation/README.md` has no YAML front matter (it is the
project README, not a model card): harmless.
**Colab notebook build (workflow wf_4200da59, 8 agents, 53 min, 12:00-12:53).** `scripts/build_notebook.py`
gained `--colab` and `--out PATH`; cells are tagged `local` / `colab` / shared so both notebooks come from one
source (default mode proven byte-identical in cell sources against the executed deliverable, which was never
rewritten: `cmp` identical before and after). The Colab variant (49 cells, 36 code) keeps the nine instructor
headers and the story; Steps 2-4 and the Results read the Hub bundle; Step 5 calls the Space through
`gradio_client`; Step 7 embeds the Space with `gr.load`; Step 8 logs in only when the token is not already
active (huggingface_hub 1.33 `login(token=...)` otherwise rewrites the token files unconditionally); Step 9
quotes entry 24. Two adversarial reviews found 6 items, all fixed: per-call error handling around the Space
calls (one ZeroGPU abort must not kill the cell), the `login()` behaviour above, `analytics_enabled=False` on
the client (the env var alone does not disable gradio_client telemetry), a note that `gr.load` forwards the token
to the Space (use a fine-grained read token as the Colab secret), the login-cache fallback made opt-in outside
Colab (`NOTEBOOK_ALLOW_LOGIN_CACHE=1`), and silencing the doomed first `gr.load` attempt.
**Challenge recorded: `gr.load` bug in gradio 6.29.1.** `gr.load("BuildITSystems/phishing-detector-demo",
src="spaces")` raises `TypeError: Textbox.__init__() got an unexpected keyword argument 'class_id'`:
`Blocks.from_config` rebuilds the Interface's examples `Dataset`, whose serialised child components are
`{"name", "class_id"}` dicts, and `gradio.components.base.get_component_instance` pops only `name`. Any Space
whose `gr.Interface` has `examples=` triggers it. Reproduced independently on the workstation. Fix in the
notebook: catch the error, install an idempotent shim that drops `class_id`, retry `gr.load`; keep a hand-built
proxy `gr.Interface` over `gradio_client` as the last fallback. Verified: `gr.load` succeeds after the shim.
**Headless Colab-mimic test (clean uv venv, CPython 3.12.14, torch 2.14.1+cpu, gradio 6.29.1, gradio_client
2.7.2, huggingface_hub 1.33.0, executed from an empty scratch directory, `HF_TOKEN` unset, token only from the
login cache via the opt-in flag).** Round 1 ran 36/36 cells with no errors but `gr.load` had fallen back to the
proxy (flagged as a failure by the harness); after the shim fix round 2 passed every check: 36/36 cells, 0
errors, 18 s wall, bundle 86 files / 3.4 MiB, six live Space predictions (in-app times 0.229 s cold then 0.009-
0.011 s; e.g. "Microsoft 365 password expires" -> Phishing 93.7%, "FINAL HOURS: 70% OFF" -> Spam 99.0%, "Agenda
for Thursday" -> Legitimate 98.6%), `gr.load` served and closed, whoami = BuildITSystems, zero token-shaped
strings in outputs or raw file. Final verification: TREC-06 F1 0.9243 / recall 0.9741 / ROC-AUC 0.9946 and the
binary confusion 11888/520/101/3793, three-way macro-F1 0.9867 and the 3x3 matrix all match `reports/`.
Presenter notes, non-blocking: the iframe cell prints "needs a public Space" until the Space is public; the
`gr.load` cell shows the shim trace line; some outputs echo workstation absolute paths that live inside the
mirrored JSON files.
**gradio 6.30.0 checked (released after the pin):** `gr.load` fails identically (`class_id`), tested in a
throwaway venv against the Space, so bumping the pin would not remove the shim. Keep 6.29.1 (= the Space's
`sdk_version`) in the Colab notebook until a release fixes `get_component_instance`.
**Bundle refresh 2026-10-12 13:05:** re-uploaded `presentation/RUN_LOG.md`, `presentation/README.md` and
`presentation/scripts/build_notebook.py` (all changed by this entry) and added
`presentation/FinalProject_PhishingDetector_Colab.ipynb` so the notebook can be downloaded from the model repo
and uploaded to Colab in one step. Same approval as the bundle; same scan; nothing else changed on the Hub.