presentation/RUN_LOG.md

RUN_LOG — BITS Phishing Detector

Running record of what was run, with what settings, what happened, and what was fixed. Newest entries at the bottom. This becomes the "challenges" section of the presentation.

---

0. 2026-10-07 — Plan confirmation, environment, pins

Machine. Fedora 44 KDE, RTX 3090 Ti 24 GB (sm_86), driver 615.71.09 (CUDA 13.4), Ryzen 9 5950X, 40 GB RAM, 741 GB free. System Python 3.14.7, uv 0.12.5. ~1.9 GB VRAM already used by the desktop.

Decision: Python 3.14 stays. Checked live: torch 2.14.1 and every other pinned package ship cp314 (or abi3 / pure) Linux wheels; torch.compile is allowed on 3.14 (only 3.15 raises); triton 3.8.0 requires <3.15. No Python update needed. (uv also downloaded a managed CPython 3.12.14 as a side effect of dry-resolving the Space requirements; harmless, kept for testing the Space stack locally.)

Decision: torch 2.14.1, CUDA 13.0 build. The plain PyPI Linux wheel is the cu130 build (cuda-toolkit 13.0.3, cuDNN 9.24, NCCL 2.30.7). Driver 615 (CUDA 13.4) runs it by backward compatibility. Install with uv pip install --torch-backend=cu130; never --torch-backend=auto (uv 0.12.5 would pick cu132 on this driver).

Constraint found: ZeroGPU pins the Space stack. ZeroGPU supports torch 2.8.0 – 2.13.0 and Python 3.12.12 / 3.10.13 only, and does not support torch.compile. So the Space uses torch 2.13.0 on Python 3.12.12 while training uses 2.14.1 on 3.14. Weights are safetensors, so this is fine. Also: a private model repo cannot be preloaded by the Space; weights download at startup with an HF_TOKEN secret.

Constraint found: huggingface_hub must stay on 1.x. tokenizers 0.23.2 requires huggingface-hub<2.0, so huggingface_hub 1.33.0 (ships the hf CLI) everywhere.

Data problems found (before downloading anything):

• it4lia/PhishingEmailCuratedDatasets_Cleaned has no body text (subjects/headers + 53 numeric features). Unusable for a text model. Excluded.
• zefang-liu/phishing-email-dataset is a lowercased re-packaging already 91.9% contained in puyang2025/seven-phishing-email-datasets (one 17 MB row, 533 literal "empty" rows, and 20% of its "Safe" rows contain the token "enron" vs 0% of phishing: a shortcut). Excluded.
• The primary set's own train/test split leaks: 5.1% of test rows are exact duplicates of train rows, ~40% have a near-duplicate. We build our own split.
• Two sub-sources are Enron-derived: 89% of TREC-05 ham and 86% of Enron-Spam ham appear in the corbt/enron-emails pretraining corpus. Neither can be the test set.
• corbt/enron-emails is 52.7% exact duplicates (517,401 -> 244,496 distinct); ~117M tokens remain, about 230k packed 512-token sequences per epoch, so one epoch is ~1,144 optimizer steps at batch 256.
• No dataset separates "spam" from "phishing". Every label-1 row comes from a spam-track corpus. Mapping spam->0 would leave ~0 positives. Proposed: positive class = "phishing/spam (unwanted)", plus a credential-lure slice of the test set reported separately. PENDING Nathan's confirmation.

Decision: hold out TREC-06 entirely as the test set (16,259 rows after dropping 141 near-empty rows; 12,394 legitimate / 3,865 positive = 23.8%). It is Enron-clean (5 rows mention "enron", 0 enron.com addresses). Caveats: ~47% of its ham is academic mailing-list traffic; its spam headers were rewritten by TREC, so the model input is subject + body only. 1,766 training rows that near-duplicate TREC-06 templates get removed from the training pool.

Model math. 24L / d1024 / 16 heads / FFN 4096 / vocab 32768 / seq 512 = 337,476,608 params (pretraining) / 337,443,842 (classifier). AdamW state 5.0 GiB. Micro-batch 16 without checkpointing is predicted at ~16.6 GiB total (7.4 GiB headroom) -> inside the 14–18 GB budget. Predicted throughput 12k–18k tokens/s (30–45% MFU of 80 TFLOPS bf16) -> 2.3–3.5 h per 150M-token epoch. Inference: ~14 ms per 512-token email on the 3090 Ti; ~0.3 s on the CPU in fp32 (the instructor's 0.074 s/sample benchmark is met on GPU, not on CPU at full length).

Status. Project scaffolded, venv created on 3.14.7, pinned install started (logs/install.log). Next: verify GPU from torch, then data_prep -> tokenizer -> model -> smoke test -> pause for numbers.

Step 2 result (22:45). Pinned install succeeded (180 packages, all binary wheels; venv 6.1 GB). From inside the venv: torch 2.14.1+cu130, CUDA 13.0 build, cuDNN 9.24; device NVIDIA GeForce RTX 3090 Ti, sm_86, 23.60 GiB, bf16 supported; SDPA Flash / memory-efficient / math backends all run (Flash with no mask, memory-efficient with a bool padding mask); bf16 4096^3 matmul measured 63.9 TFLOPS (spec ~80); torch.compile works on Python 3.14 (triton built its driver stub with a harmless _POSIX_C_SOURCE warning); free VRAM at idle with the desktop up: 21.5 GiB. All pinned libraries import; config.yaml parses. hf CLI present, not logged in yet (Nathan to run hf auth login).

Agent policy (per Nathan, 21:14). Model and effort are set per agent from here on: mechanical lookups -> Sonnet/low; judgment-heavy research -> Opus or Fable/medium-high; math, core code and adversarial verification -> Fable/high. The step-1 research run inherited Fable for all 7 agents.

1. 2026-10-07 23:41 — Data build (data_prep.py build + check)

Run. .venv/bin/python src/data_prep.py build then check. 2 min 20 s wall, peak RSS 6.9 GB, HF cache 1.5 GB (puyang2025 @ b7f3f8d9, corbt @ cfc06c75). Every asserted leakage check is 0 in both the build-time recount and the independent check over the written files.

split	rows	positives	base rate	sources
finetune_train	157,113	75,675	48.2%	TREC-05 46,386 / TREC-07 47,233 / CEAS-08 34,178 / Enron 24,223 / Assassin 5,093
finetune_val	17,643	8,642	49.0%	same five, grouped by near-dup cluster so no template straddles train/val
test_holdout (TREC-06)	16,302	3,894	23.9%	never touched by tokenizer, pretraining, or fine-tuning
pretrain (MLM)	226,273 docs	-	-	corbt/enron-emails after dedup + quote stripping; 408M chars

What was removed and why. 715 labelled rows with <20 alphanumerics (91% of them TREC-05/TREC-06 garbage spam rows). Ling excluded (2,859 academic rows). 18 rows in 8 groups where the identical body carries both labels (Enron ham vs TREC-05 spam newsletters) — dropped as label noise. 6,715 exact duplicates in the pool. **1,652 training rows removed as near-duplicates of TREC-06 templates** (TREC-05 spam 863, Enron spam 456, TREC-07 spam 292): these are spam templates shared across the TREC years; keeping them would let the model memorise test-set templates. corbt: 517,401 rows -> 272,905 exact duplicates removed (52.7%), 14,027 too short after quote stripping, 4,004 matching a val email removed, 85 near-duplicates of holdout emails removed, 35,172 overlapping fine-tune TRAIN emails kept (allowed: those are Enron ham). Quote stripping cut 21,832 bodies (kept 89% of all characters; the >=200-alnum guard left 70,907 marked bodies intact).

Deviation from the research estimate. Holdout is 16,302 rows (research: 16,259) because the <20-alnum filter is applied to subject+body rather than body alone: 43 TREC-06 rows have a real subject and a near-empty body, which is a legitimate model input. Pretraining corpus is 408M chars (research: ~466M) because quote stripping and the val screen remove more than the estimate assumed.

Problem noted. The val split has 5,336 near-duplicate links into the pretraining corpus (883 val emails, Enron boilerplate). Counted, not removed: val is only used for model selection and the holdout is the real gate. Exact val matches (4,004) were removed from pretraining.

2. 2026-10-07 23:43 — Tokenizer (tokenizer.py train + stats)

Run. Byte-level BPE, NFKC only (case preserved on purpose: ALL-CAPS is a spam signal), vocab 32,768, min_frequency 2, specials [PAD]=0 [UNK]=1 [CLS]=2 [SEP]=3 [MASK]=4. Trained on pretrain.parquet (226,273 docs, 408M chars) + finetune_train.parquet (157,113 docs, 278M chars) only; val and test are refused by name and an exact-hash tripwire against the holdout found 0 collisions. Training took 28 s (49 s wall incl. the tripwire), 3.6 GB RSS. Saved data/processed/tokenizer.json.

split	rows	mean tok	median	p95	chars/tok	% > 512	mean tok legit / positive
pretrain (20k sample)	20,000	476	198	1,560	3.76	20.5%	-
finetune_train	157,113	487	227	1,540	3.63	24.1%	621 / 343
finetune_val	17,643	454	200	1,421	3.58	22.3%	594 / 307
test_holdout	16,302	709	244	1,254	3.43	21.9%	842 / 283

Observations. [UNK] rate is 0 by construction (full 256-byte alphabet). Round trips are exact except for NFKC folding (full-width "Ｍｉｃｒｏｓｏｆｔ" -> "Microsoft", a deliberate anti-evasion property). About 22% of emails are longer than 512 tokens and get truncated; the pretraining corpus is ~108M tokens at 3.76 chars/token. Length is a confound: legitimate emails average ~2x the tokens of positives in every split (holdout 842 vs 283). Truncation at 512 blunts this, and the error analysis must check whether the classifier leans on length. Outliers exist (one 886,635-token email in train) and are harmless under truncation.

3. 2026-10-07 23:46 — Smoke test (smoke_test.py, real packed text, reports/smoke_test.json)

Model. 337,476,608 params (MLM, tied head) / 337,443,842 (classifier) — exactly the research figure. 3,000 real pretraining docs -> 1,592,926 tokens -> 3,111 packed 512-token sequences; in-code no-leak check: 0 of 3,000 docs match a holdout email.

VRAM (bf16 autocast, fused AdamW, real gradient-accumulation pattern, desktop using 2.08 GiB):

micro-batch	checkpointing	peak reserved	total device used	tokens/s
8	off	9.71 GiB	11.85 GiB	21,493
12	off	11.78 GiB	13.92 GiB	23,487
16	off	13.65 GiB	15.80 GiB	23,642
24	off	17.65 GiB	19.80 GiB	25,227
24	on	8.06 GiB	10.21 GiB	19,156

The finder picked 24 (largest in the 14–18 GiB band) but total device use then sits at 19.8 GiB, 0.2 GiB under the 20 GiB "never exceed" line. Micro-batch 16 is 6% slower, gives an exact effective batch of 256 (24 x 11 = 264), and leaves ~8 GiB of headroom for desktop spikes. Recommendation: micro-batch 16 x 16 accumulation for the long run. Checkpointing is not needed for memory and costs ~25% throughput; keep it off.

Throughput / ETA. 25,187 tok/s at B=24 (23,642 at B=16). Measured 0.2595 tokens/char on the packed sample -> the 408M-char corpus is ~105.8M tokens/epoch = ~807 optimizer steps at 256 x 512. About 1.2–1.3 h per epoch; 3 epochs ~3.75 h, 6 epochs ~7.5 h. Prediction was 2.3–3.5 h/epoch at 12k–18k tok/s: the real machine is ~1.5x faster than the 30–45% MFU assumption (25k tok/s ≈ 63% MFU of the 80 TFLOPS peak).

SDPA. Flash runs on the mask-free packed path; memory-efficient runs with the bool padding mask; Flash correctly refuses a mask (no silent math fallback).

Convergence. 30 steps at B=24, LR 1e-4, 10-step warmup, real text: loss 10.58 -> 7.67 (ln V = 10.40). The pipeline learns.

Inference latency (batch 1, 512 tokens). GPU bf16: 10.7 ms (0.14x the instructor's 74 ms). CPU fp32, 16 threads: 699 ms at 512 tokens, 319 ms at 128 tokens (9.4x / 4.3x the benchmark). The research's 336 ms / 79 ms CPU estimate was 2–4x too optimistic; CPU inference is per-op-overhead bound, so truncation barely helps. The 74 ms benchmark is met on GPU only -> the Space must be ZeroGPU, not CPU-basic.

Challenges found by the reviewers before this run (fixed): the first finder measured peak VRAM with a single fwd/bwd/opt step and under-reported the real accumulation peak by one fp32 gradient copy (1.26 GiB); the first MLM-corpus decontamination used exact keys only and missed near-duplicate (quoted/forwarded, 419-template) variants of holdout emails; floor-division accumulation would have reported effective batch 256 while training at 240 for B=24; safetensors refuses the tied state_dict unless the alias key is dropped (helper added); the tied decoder leaks gradient into the [PAD] embedding row (post-accumulate hook added).

4. 2026-10-08 00:40 — Step-4 code build (pretrain / finetune / evaluate / export / app / upload / synthetic)

What was built (each file by one agent, then an independent adversarial reviewer who ran it, then a fix pass): src/pretrain.py (resumable MLM trainer, 1,019 lines), src/finetune.py (classifier trainer with the instructor's dials as flags; experiments.jsonl record per run), src/evaluate.py (the only script allowed to open the TREC-06 holdout; metrics, latency, plots, error analysis), src/export.py (bf16 safetensors + model card + byte-identical app/model.py), app/app.py + requirements.txt + README.md (Gradio 6 / ZeroGPU app with saliency and rule-based signals), scripts/upload_to_hf.py (dry-run by default, --yes required), src/make_synthetic.py (2,000 template-generated rows, 8 positive + 8 legitimate business categories, source=synthetic).

Verified by running. pretrain: fresh run -> kill -> --resume reproduces the exact data order (window indices checked against the seeded permutation), optimizer step counter and LR; SIGINT saves a checkpoint and exits 0; SIGKILL between checkpoints is recovered on resume (orphan metrics rows dropped); a forced OOM saves and exits 3; a 5 GiB test cap trips the VRAM guard; --max-hours stops cleanly; resume refuses a changed plan or model shape. The full token cache was built as a side effect: 107,498,500 train tokens + 1,021,888 val tokens (0.266 tok/char), 42.8 s. finetune: resume reproduces batch order; --n-layers 4 loads only the first four pretrained layers; focal(gamma=0) == cross-entropy; the holdout parquet is never opened (inotify-verified). 14 agents, 451 tool calls.

Bugs the reviewers caught and the fixers closed (for the challenges slide): a --lr override on --resume was silently discarded by the restored optimizer state but recorded in the experiment log as if applied; resuming a finished fine-tune run trained one extra step at lr 0; changing frozen layers on resume crashed with a raw torch error; metrics CSVs kept orphan rows after a hard crash; evaluate/export and app each had 2–3 majors (details in docs/research/ and the agents' reports).

Changes I made myself after reading pretrain.py: (1) the OOM handler now covers the optimizer step and the eval pass too, not just the forward/backward loop, so a desktop VRAM spike during the 500-step eval cannot lose 45 minutes of work; (2) a resume with nothing left to do no longer overwrites run_summary.json; (3) --resume compares --limit-docs with the checkpoint before tokenising anything; (4) every packed window now starts with [CLS] (511 stream tokens follow), as RoBERTa prepends <s>: the reviewer found 0 occurrences of [CLS] in the 107M-token stream, which would have left the classifier's pooled position untrained until fine-tuning.

Known gaps going into the long run: the real 337M fine-tune, evaluate and export have only run on tiny dev models/subsets; --synthetic on path untested end to end; ZeroGPU runtime behaviour untested until the first Space build. Dev checkpoints from the self-tests sit in checkpoints/{pretrain,finetune}/dev* (~15 GB) pending Nathan's OK to delete.

5. 2026-10-08 12:28 — Pretraining run large-24L launched (Nathan's settings)

Decisions (Nathan, 2026-10-08). Micro-batch 24 (17.65 GiB reserved measured; total GPU use ~20 GB with the desktop). 12 epochs: ~14 h at 25k tok/s, 9,552 optimizer steps, warmup 573 steps, cosine to 10% of 1.5e-4. Rationale: dynamic masking makes every pass different and data-constrained scaling results show repeated passes keep helping with diminishing returns; val MLM loss (64 fixed windows from the held-out last 1% of documents) is evaluated every 500 steps and the run is stopped if it rises. Positive class is named "Spam/Phishing" in the app, metrics and model card (the sources do not separate spam from phishing). Hugging Face login confirmed (BuildITSystems).

Launch. EPOCHS=12 MICRO_BATCH=24 scripts/launch_pretrain.sh -> tmux session phish-pretrain, stdout in logs/pretrain_large-24L_20261008-122800.out. Trainer: 210,368 packed windows of [CLS] + 511 tokens (107.7M tokens/epoch), 796 steps/epoch, micro-batch 24 x 11 accumulation = effective batch 264 (256 is not divisible by 24; the realised batch is used in every number), checkpoints every 500 steps (keep 3, ~3.8 GB each) + final/. Token cache reused (0 s). GPU at launch: 20.3 GB used, 98% util, 365 W.

6. 2026-10-08 ~13:00 — Scope change: three-way labels (legitimate / phishing / spam)

Nathan's decision. The classifier should output legitimate, phishing, or spam. Our current positives cannot be split (TREC-05/07 and CEAS-08 labelled everything unwanted as "spam"), so phishing-labelled text from separate provenance is being added: the Champa et al. raw corpus mirror on the Hub (kudzaiprichard/aura-phishing-email-corpus, CC-BY-4.0, byte-identical to Zenodo 8339691) with Nazario.csv (1,565 phishing, 1995–2022) and Nigerian_Fraud.csv (3,332 advance-fee fraud), plus our synthetic BITS-style lures. Spam class from provenance: SpamAssassin, Enron-Spam and (if included) Ling spam folders. The 250k-row locuoco/the-biggest-spam-ham-phish-email-dataset-300000 (labels ham/phish/ spam, provenance undocumented) is inspected first and used only if its labels agree with known provenance and it adds rows we do not have, after deduplication against every split and the TREC-06 holdout.

Design. One encoder, one 3-class head, partial-label loss: rows with clean provenance train with 3-class cross-entropy; the ~70k TREC/CEAS positives train on the probability mass of {phishing, spam} ("unwanted") so they still shape the legitimate-vs-unwanted decision. Binary roll-up P(unwanted) = P(phishing)+P(spam) keeps TREC-06 as the clean binary test; a new test_holdout3 slice of clean- provenance rows (drawn from sources never seen by the tokenizer or the pretraining corpus) gives the 3x3 confusion matrix. num_labels lives under finetune: in config.yaml, not under model:, because the live pretraining run refuses to resume if its model snapshot changes.

7. 2026-10-08 13:21 — Three-way data build (build3) and the 300k-set verdict

300k set: rejected after inspection (docs/research/bigshp_research.md). locuoco/the-biggest-spam-ham-phish- email-dataset-300000 is 365,448 rows, not deduplicated (51.5% within-file duplicates), and is a union of Kaggle re-packagings of corpora we already hold: 97.6% of its ham, 92.5% of its phish and 90.2% of its spam rows match our text. Its ham/spam labels agree with our provenance 99.9%/99.8% because they are our labels copied back; its phish label agrees only 60.5%: 14,660 "phish" rows are SpamAssassin/Enron-Spam spam-folder mail, 17,714 are TREC/CEAS spam-track rows, and only 4,784 are confirmed phishing provenance (Nazario/Nigerian, which we take raw from the Champa mirror). 36,020 rows (9.9%) are copies of our frozen TREC-06 test set. Every "phish" row went through a stopword-stripping pipeline that ham/spam rows did not, a preprocessing artifact that is a near-perfect label proxy and would not transfer to real inbound mail. Decision: not one row ingested; data.threeway.bigshp.enabled: false with the verdict recorded.

Champa mirror ingested (kudzaiprichard/aura-phishing-email-corpus, CC-BY-4.0): Nazario 1,565 rows -> 1,507 kept; Nigerian_Fraud 3,332 -> 2,504 kept. Drops worth knowing: 235 Nigerian-fraud rows are near- duplicates of TREC-06 spam (419 letters recur across corpora) and 13 match holdout keys exactly: all removed so the TREC-06 test stays clean; 424 more were dropped because their clusters touched both train and val.

Three-way splits (label3: 0 legit / 1 phishing / 2 spam / -1 unwanted-unsplit):

split	rows	legit	phishing	spam	unwanted (TREC/CEAS positives)
finetune_train	160,263	81,438	3,150	12,870	62,805
finetune_val	15,284	7,002	340	763	7,179
test_holdout (TREC-06, unchanged rows)	16,302	12,408	-	-	3,894
test_holdout3 (new, clean provenance)	3,220	1,999	521	700	-

test_holdout3 legit = CEAS-08 736 + TREC-07 1,099 + Assassin 164 (never Enron: no pretraining overlap); spam = Enron-Spam 628 + Assassin 72; phishing = Nazario 222 + Nigerian 299. All of it came from finetune_val (never seen by the tokenizer) or the new sources (never seen by anything). finetune_val shrank from 17,643 to 15,284 rows, so new val F1 values are not strictly comparable with the binary runs. Leakage: every asserted check is 0 in check3 and the binary check still passes on the rewritten files. Existing rows in train/test_holdout are byte-identical to the binary build (data/processed/binary_snapshot).

Honest caveat. Only 3,150 fully-labelled phishing training rows (plus 1,000 synthetic lures) carry the phishing-vs-spam distinction; 62,805 positives are trained as "unwanted" only. Expect the three-way split to be the weaker decision; the binary roll-up on TREC-06 stays the headline.

8. 2026-10-08 13:55 — Fine-tune batching moved into torch.utils.data.DataLoader

The instructor's written spec requires PyTorch's DataLoader for batching and shuffling; finetune.py had its own numpy sampler. Every batch now goes through torch.utils.data.DataLoader (make_loader): a row-index Dataset, the seeded length-bucketed batch order as batch_sampler, a collate function that pads each micro-batch to its own longest row. Verified on the CPU fixture: losses and metrics identical to the previous code at every step. One problem found on the way: a DataLoader draws its worker base seed from the global torch RNG when iterated, which shifted the dropout stream; passing an explicit generator removed the side effect (first run differed at step 1, 1.1093 vs 1.1125; identical after).

9. 2026-10-08 14:15 — Dev checkpoints removed; fine-tuning queue armed

Nathan approved both. Deleted the agents' self-test checkpoints (checkpoints/pretrain/dev, checkpoints/finetune/dev, checkpoints/finetune/dev-focal; 15.3 GB) and their two dev lines in reports/experiments.jsonl (backup in the session scratchpad). scripts/run_experiments.sh is running in tmux session phish-finetune, waiting for checkpoints/pretrain/large-24L/final/model.safetensors; it then runs ft-ce (3 epochs) and the five ablations (weighted CE, focal, synthetic on, 12 layers, scratch; 2 epochs each), evaluates every best/ on TREC-06 + the three-way holdout in one call, and exports ft-ce to model_export/ft-ce + app/. Pretraining at step 1,100 / 9,552, loss 5.96.

10. 2026-10-08 14:30 — Presentation notebook and README

FinalProject_PhishingDetector.ipynb (generated by scripts/build_notebook.py, 48 cells) mirrors the instructor's nine "Step N" headers verbatim and in order, builds real torch.utils.data.DataLoaders from the processed splits in Step 2, instantiates the 337M model and prints its parameter table in Step 3, plots the live pretraining curve in Step 4 (shows "RUN IN PROGRESS" until final/ exists), times 100 single emails instructor-style in Step 5, shows the app files, runs the app's predict on built-in synthetic examples, checks the HF login and dry-runs the upload. It executes top to bottom on the CPU in ~70 s with zero errors today and prints "not available yet — produced by " for the 12 artifacts the training queue will create; re-executing it after the queue fills everything in. Reviewer catches fixed before acceptance: one header lacked the instructor's "(Optional)", a dev-checkpoint caption stated a wrong step count, a Challenges bullet misdescribed the uv torch-backend issue, and nothing kept the notebook off the GPU when opened from a plain shell (now forced to CPU while the pretraining run owns the GPU). README.md documents setup, every phase's commands (verified with --help), config dials, data provenance and licences, the leakage policy, the experiment queue, deployment, the instructor-requirement mapping and the limitations. Reviewer catches fixed: it had documented a fictional experiment grid and a notebook path that did not exist.

11. 2026-10-08 19:15 — Hugging Face upload approved (deferred until the export exists)

Nathan asked for the upload. Dry run at 19:14: the plan is a private model repo BuildITSystems/phishing-detector-encoder (model.safetensors, config.json, tokenizer.json, README.md) and a private ZeroGPU Space BuildITSystems/phishing-detector-demo (app.py, model.py, requirements.txt, README.md, .gitattributes + bundled weights, so no token secret is needed). It refuses to run today because model_export/ft-ce does not exist yet (the fine-tuning queue produces it tomorrow morning). The upload is scheduled to run automatically once the export lands and the dry run reports no problems. Pretraining at 19:14: step 4,280 / 9,552 (epoch 6 of 12), train MLM loss 2.10, on schedule for ~02:00.

12. 2026-10-09 02:45 — Pretraining status check (scheduled)

Run large-24L at step 9,020 / 9,552 (epoch 12 of 12), ETA ~03:30: about an hour behind the 14 h estimate because each 500-step validation and 3.8 GB checkpoint write pauses training. Throughput steady at ~24.1k tokens/s, reserved VRAM 17.94 GiB, device total 20.2 GB with the desktop, GPU 86 C. No watchdog action, no OOM, no warnings since launch. Validation MLM loss on the held-out 1% of documents fell monotonically through all twelve passes (6.54 -> 6.38 -> 5.81 -> 5.25 -> 3.91 -> 3.16 -> 2.78 -> 2.58 -> 2.44 -> 2.36 -> 2.27 -> 2.20 -> 2.14 -> 2.11 -> 2.07 -> 2.04 -> 2.03 -> 2.01), masked-token accuracy 14.7% -> 62.9%, so twelve passes over the same corpus did not overfit; the biggest drop came between steps 2,000 and 3,000 (epochs 3-4), when the model stopped predicting frequent tokens and started using context. Train loss 1.55 (per-step, with dropout). The fine-tuning queue is waiting for final/.

13. 2026-10-09 03:31 — MLM pretraining run large-24L (complete)

Run. python src/pretrain.py train --run-name large-24L; 337,476,608 parameters (EncoderForMLM, tied head). micro-batch 24 x accumulation 11 = 264 sequences x 512 tokens per step; bf16 autocast, fused AdamW [0.9, 0.98] eps 1e-06 wd 0.01 (no decay on biases/LayerNorm), clip 1.0; cosine schedule peak 1.5e-04 -> 10% over 9552 planned steps (12 epochs x 796 steps) with 573 warmup steps; dynamic 15% masking 80/10/10, loss on masked positions only; 210,368 packed 512-token windows (226,273 docs), 1,999 val windows.

Result. 9552 optimizer steps (12.00 epochs), 1291.1M tokens seen, total training time 15h 03m 05s (54,186 s) over 1 invocation(s), mean 23,889 tokens/s; peak VRAM reserved 17.94 GiB / allocated 17.49 GiB; final train MLM loss 1.5237, val MLM loss 1.9863 (masked-token accuracy 62.9%).

Artifacts. checkpoints/pretrain/large-24L/ (metrics.csv, run_summary.json, step-*/ and final/); log /home/nathank/BITS/dev/Phishing Model/logs/pretrain-20261008-122802.log.

14. 2026-10-09 04:07 — Hand-off to fine-tuning (scheduled check)

Pretraining large-24L finished at 03:31 after 15 h 03 m: 9,552 optimizer steps, 1.29 B tokens seen (12 passes over 107.5 M), mean 23,889 tokens/s, peak VRAM reserved 17.94 GiB, final train MLM loss 1.52, validation MLM loss 1.99 with 62.9% masked-token accuracy (from 10.4 / 0% at random init). final/ holds the fp32 encoder (1.35 GB) + tokenizer + config. The watchdog saw final/ and exited without ever acting. The queue started ft-ce at 03:36: 3 epochs x 5,009 steps at batch 32 (micro-batch 16), 13.1 GiB reserved, ~75 examples/s, ETA ~05:30 for this run. First validation at step 3,750 (0.75 epoch), 15,284 rows: unwanted-vs-legitimate F1 0.9855, ROC-AUC 0.9995, PR-AUC 0.9996; three-way slice (n=8105) macro-F1 0.9775, accuracy 0.9921, per-class F1 [legit, phishing, spam] = [0.996 0.967 0.970]; phishing-vs-spam on true unwanted rows [pp ps sp ss] = [340 0 6 757]. Val F1 is not comparable with the earlier binary dev numbers (val shrank by the 2,699 rows moved to test_holdout3). Queue estimate: ft-ce done ~05:30, then five 2-epoch ablations (1 h each + evaluation) -> ~11:00-11:30.

Fine-tune run ft-ce — 2026-10-09 04:28

Dials. init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 24 layers, 0 frozen; loss cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 3 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 337,444,867 (337,444,867 trainable). Result. 6250 steps, stopped by early_stopping; training time 51.5 min (eval 4.8 min of it); peak reserved 13.13 GiB. Best val at step 5000 (epoch 0), Unwanted (phishing or spam) roll-up: F1 0.9929, precision 0.9938, recall 0.9920, accuracy 0.9923, ROC-AUC 0.9996, PR-AUC 0.9996; F1-optimal threshold 0.318. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9782, accuracy 0.9922; per-class F1 Legitimate 0.9958, Phishing 0.9672, Spam 0.9716; phishing-vs-spam on true unwanted (n=1,103): phish->phish 340, phish->spam 0, spam->phish 4, spam->spam 759. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-ce/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-ce/last.

15. 2026-10-09 04:28 — Evaluation on the test + test3 splits (evaluate.py run, ft-ce)

Checkpoint. /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-ce/best (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

test split (TREC-06 binary holdout): test_holdout.parquet, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 285.1 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9404 (232 missed of 3,894)
precision	0.8610 (591 false alarms)
F1	0.8990
accuracy	0.9495
specificity	0.9524
ROC-AUC	0.9875
PR-AUC	0.9626
recall @ 1% FPR	0.7745
recall @ 0.1% FPR	0.3529
best-F1 threshold	0.821 (F1 0.9090)
credential-lure slice	398 rows (61.1% labelled positive), recall 0.9959, precision 0.8996

Three-way: no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,662 flagged true positives the model called 1,294 phishing and 2,368 spam.

test3 split (three-way holdout): test_holdout3.parquet, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 240.9 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9853 (18 missed of 1,221)
precision	0.9877 (15 false alarms)
F1	0.9865
accuracy	0.9898
specificity	0.9925
ROC-AUC	0.9992
PR-AUC	0.9985
recall @ 1% FPR	0.9869
recall @ 0.1% FPR	0.8690
best-F1 threshold	0.411 (F1 0.9869)
credential-lure slice	275 rows (97.5% labelled positive), recall 0.9925, precision 1.0000

Three-way on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9863, macro-F1 0.9822; Legitimate P 0.991 / R 0.992 / F1 0.992 (n=1,999); Phishing P 0.981 / R 0.973 / F1 0.977 (n=521); Spam P 0.977 / R 0.979 / F1 0.978 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,984/4/11; Phishing 9/507/5; Spam 9/6/685. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9894 (7 phishing->spam, 6 spam->phishing).

source	n	Legitimate	Phishing	Spam	accuracy	macro-F1 (present classes)
Assassin	236	164	0	72	0.9449	0.9464
CEAS-08	736	736	0	0	0.9905	0.9952
Enron	628	0	0	628	0.9889	0.9944
Nazario	222	0	222	0	0.9505	0.9746
Nigerian_Fraud	299	0	299	0	0.9900	0.9950
TREC-07	1,099	1,099	0	0	0.9973	0.9986

Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms. GPU bf16: 9.95 ms avg / 8.72 ms p50 over 100 emails (0.13x). CPU fp32 (16 threads): 514.07 ms avg / 526.19 ms p50 over 100 emails (6.95x).

Length confound (test). FN mean 256 tokens vs TP 287; FP 763 vs TN 848 (see error_analysis.md for the per-bucket table).

Artifacts. reports/eval_ft-ce: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py plot).

Fine-tune run ft-weighted — 2026-10-09 05:55

Dials. init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 24 layers, 0 frozen; loss weighted_cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 337,444,867 (337,444,867 trainable). Result. 10018 steps, stopped by epochs_done; training time 83.6 min (eval 8.0 min of it); peak reserved 13.13 GiB. Best val at step 10000 (epoch 1), Unwanted (phishing or spam) roll-up: F1 0.9937, precision 0.9911, recall 0.9964, accuracy 0.9932, ROC-AUC 0.9998, PR-AUC 0.9998; F1-optimal threshold 0.899. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9775, accuracy 0.9903; per-class F1 Legitimate 0.9944, Phishing 0.9798, Spam 0.9583; phishing-vs-spam on true unwanted (n=1,103): phish->phish 339, phish->spam 1, spam->phish 2, spam->spam 761. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-weighted/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-weighted/last.

16. 2026-10-09 05:55 — Evaluation on the test + test3 splits (evaluate.py run, ft-weighted)

Checkpoint. /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-weighted/best (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

test split (TREC-06 binary holdout): test_holdout.parquet, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 283.4 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9818 (71 missed of 3,894)
precision	0.8307 (779 false alarms)
F1	0.9000
accuracy	0.9479
specificity	0.9372
ROC-AUC	0.9943
PR-AUC	0.9833
recall @ 1% FPR	0.8767
recall @ 0.1% FPR	0.6335
best-F1 threshold	0.989 (F1 0.9383)
credential-lure slice	398 rows (61.1% labelled positive), recall 0.9959, precision 0.9565

Three-way: no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,823 flagged true positives the model called 801 phishing and 3,022 spam.

test3 split (three-way holdout): test_holdout3.parquet, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 238.3 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9934 (8 missed of 1,221)
precision	0.9790 (26 false alarms)
F1	0.9862
accuracy	0.9894
specificity	0.9870
ROC-AUC	0.9993
PR-AUC	0.9988
recall @ 1% FPR	0.9910
recall @ 0.1% FPR	0.7060
best-F1 threshold	0.728 (F1 0.9878)
credential-lure slice	275 rows (97.5% labelled positive), recall 1.0000, precision 1.0000

Three-way on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9876, macro-F1 0.9852; Legitimate P 0.996 / R 0.987 / F1 0.991 (n=1,999); Phishing P 0.988 / R 0.987 / F1 0.988 (n=521); Spam P 0.964 / R 0.990 / F1 0.977 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,973/5/21; Phishing 2/514/5; Spam 6/1/693. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9943 (6 phishing->spam, 1 spam->phishing).

source	n	Legitimate	Phishing	Spam	accuracy	macro-F1 (present classes)
Assassin	236	164	0	72	0.9619	0.9603
CEAS-08	736	736	0	0	0.9823	0.9911
Enron	628	0	0	628	0.9936	0.9968
Nazario	222	0	222	0	0.9820	0.9909
Nigerian_Fraud	299	0	299	0	0.9900	0.9950
TREC-07	1,099	1,099	0	0	0.9936	0.9968

Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms. GPU bf16: 10.12 ms avg / 8.95 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 411.57 ms avg / 397.57 ms p50 over 100 emails (5.56x).

Length confound (test). FN mean 310 tokens vs TP 284; FP 1779 vs TN 782 (see error_analysis.md for the per-bucket table).

Artifacts. reports/eval_ft-weighted: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py plot).

Fine-tune run ft-focal — 2026-10-09 07:21

Dials. init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 24 layers, 0 frozen; loss focal (gamma 2.0); lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 337,444,867 (337,444,867 trainable). Result. 10018 steps, stopped by early_stopping; training time 83.9 min (eval 8.0 min of it); peak reserved 13.13 GiB. Best val at step 8750 (epoch 1), Unwanted (phishing or spam) roll-up: F1 0.9951, precision 0.9954, recall 0.9948, accuracy 0.9947, ROC-AUC 0.9998, PR-AUC 0.9998; F1-optimal threshold 0.421. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9869, accuracy 0.9947; per-class F1 Legitimate 0.9971, Phishing 0.9868, Spam 0.9768; phishing-vs-spam on true unwanted (n=1,103): phish->phish 339, phish->spam 1, spam->phish 1, spam->spam 762. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-focal/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-focal/last.

17. 2026-10-09 07:22 — Evaluation on the test + test3 splits (evaluate.py run, ft-focal)

Checkpoint. /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-focal/best (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

test split (TREC-06 binary holdout): test_holdout.parquet, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 282.8 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9741 (101 missed of 3,894)
precision	0.8794 (520 false alarms)
F1	0.9243
accuracy	0.9619
specificity	0.9581
ROC-AUC	0.9946
PR-AUC	0.9848
recall @ 1% FPR	0.8955
recall @ 0.1% FPR	0.6497
best-F1 threshold	0.682 (F1 0.9397)
credential-lure slice	398 rows (61.1% labelled positive), recall 0.9959, precision 0.9758

Three-way: no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,793 flagged true positives the model called 974 phishing and 2,819 spam.

test3 split (three-way holdout): test_holdout3.parquet, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 238.7 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9836 (20 missed of 1,221)
precision	0.9893 (13 false alarms)
F1	0.9864
accuracy	0.9898
specificity	0.9935
ROC-AUC	0.9993
PR-AUC	0.9989
recall @ 1% FPR	0.9869
recall @ 0.1% FPR	0.8788
best-F1 threshold	0.475 (F1 0.9877)
credential-lure slice	275 rows (97.5% labelled positive), recall 1.0000, precision 1.0000

Three-way on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9885, macro-F1 0.9867; Legitimate P 0.990 / R 0.993 / F1 0.992 (n=1,999); Phishing P 0.996 / R 0.987 / F1 0.991 (n=521); Spam P 0.979 / R 0.976 / F1 0.977 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,986/1/12; Phishing 4/514/3; Spam 16/1/683. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9959 (4 phishing->spam, 1 spam->phishing).

source	n	Legitimate	Phishing	Spam	accuracy	macro-F1 (present classes)
Assassin	236	164	0	72	0.9534	0.9458
CEAS-08	736	736	0	0	0.9905	0.9952
Enron	628	0	0	628	0.9841	0.9920
Nazario	222	0	222	0	0.9865	0.9932
Nigerian_Fraud	299	0	299	0	0.9866	0.9933
TREC-07	1,099	1,099	0	0	0.9982	0.9991

Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms. GPU bf16: 10.34 ms avg / 9.01 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 357.98 ms avg / 340.34 ms p50 over 100 emails (4.84x).

Length confound (test). FN mean 322 tokens vs TP 284; FP 1237 vs TN 827 (see error_analysis.md for the per-bucket table).

Artifacts. reports/eval_ft-focal: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py plot).

Fine-tune run ft-synthetic — 2026-10-09 08:16

Dials. init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 24 layers, 0 frozen; loss cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic on (2000 rows); selection val_f1_unwanted; train rows 162,263, val rows 15,284; params 337,444,867 (337,444,867 trainable). Result. 6250 steps, stopped by early_stopping; training time 52.0 min (eval 4.8 min of it); peak reserved 13.13 GiB. Best val at step 5000 (epoch 0), Unwanted (phishing or spam) roll-up: F1 0.9927, precision 0.9904, recall 0.9949, accuracy 0.9920, ROC-AUC 0.9996, PR-AUC 0.9997; F1-optimal threshold 0.609. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9665, accuracy 0.9891; per-class F1 Legitimate 0.9940, Phishing 0.9352, Spam 0.9704; phishing-vs-spam on true unwanted (n=1,103): phish->phish 340, phish->spam 0, spam->phish 6, spam->spam 757. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-synthetic/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-synthetic/last.

18. 2026-10-09 08:16 — Evaluation on the test + test3 splits (evaluate.py run, ft-synthetic)

Checkpoint. /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-synthetic/best (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

test split (TREC-06 binary holdout): test_holdout.parquet, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 282.3 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9679 (125 missed of 3,894)
precision	0.8402 (717 false alarms)
F1	0.8995
accuracy	0.9483
specificity	0.9422
ROC-AUC	0.9908
PR-AUC	0.9716
recall @ 1% FPR	0.7871
recall @ 0.1% FPR	0.4779
best-F1 threshold	0.962 (F1 0.9186)
credential-lure slice	398 rows (61.1% labelled positive), recall 0.9959, precision 0.9528

Three-way: no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,769 flagged true positives the model called 1,486 phishing and 2,283 spam.

test3 split (three-way holdout): test_holdout3.parquet, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 238.3 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9885 (14 missed of 1,221)
precision	0.9861 (17 false alarms)
F1	0.9873
accuracy	0.9904
specificity	0.9915
ROC-AUC	0.9993
PR-AUC	0.9988
recall @ 1% FPR	0.9885
recall @ 0.1% FPR	0.8395
best-F1 threshold	0.508 (F1 0.9873)
credential-lure slice	275 rows (97.5% labelled positive), recall 1.0000, precision 1.0000

Three-way on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9860, macro-F1 0.9814; Legitimate P 0.993 / R 0.991 / F1 0.992 (n=1,999); Phishing P 0.968 / R 0.987 / F1 0.977 (n=521); Spam P 0.980 / R 0.970 / F1 0.975 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,982/6/11; Phishing 4/514/3; Spam 10/11/679. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9869 (5 phishing->spam, 11 spam->phishing).

source	n	Legitimate	Phishing	Spam	accuracy	macro-F1 (present classes)
Assassin	236	164	0	72	0.9280	0.9296
CEAS-08	736	736	0	0	0.9851	0.9925
Enron	628	0	0	628	0.9873	0.9936
Nazario	222	0	222	0	0.9820	0.9909
Nigerian_Fraud	299	0	299	0	0.9900	0.9950
TREC-07	1,099	1,099	0	0	0.9982	0.9991

Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms. GPU bf16: 10.05 ms avg / 8.9 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 481.35 ms avg / 464.28 ms p50 over 100 emails (6.50x).

Length confound (test). FN mean 256 tokens vs TP 286; FP 686 vs TN 854 (see error_analysis.md for the per-bucket table).

Artifacts. reports/eval_ft-synthetic: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py plot).

Fine-tune run ft-12L — 2026-10-09 08:54

Dials. init pretrained:/home/nathank/BITS/dev/Phishing Model/checkpoints/pretrain/large-24L/final; 3 classes; 12 layers, 0 frozen; loss cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 186,290,179 (186,290,179 trainable). Result. 8750 steps, stopped by early_stopping; training time 35.3 min (eval 3.1 min of it); peak reserved 6.97 GiB. Best val at step 6250 (epoch 1), Unwanted (phishing or spam) roll-up: F1 0.9925, precision 0.9936, recall 0.9914, accuracy 0.9919, ROC-AUC 0.9995, PR-AUC 0.9996; F1-optimal threshold 0.171. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9774, accuracy 0.9916; per-class F1 Legitimate 0.9954, Phishing 0.9695, Spam 0.9673; phishing-vs-spam on true unwanted (n=1,103): phish->phish 338, phish->spam 2, spam->phish 2, spam->spam 761. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-12L/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-12L/last.

19. 2026-10-09 08:54 — Evaluation on the test + test3 splits (evaluate.py run, ft-12L)

Checkpoint. /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-12L/best (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

test split (TREC-06 binary holdout): test_holdout.parquet, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 591.7 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 1.7 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9461 (210 missed of 3,894)
precision	0.8120 (853 false alarms)
F1	0.8739
accuracy	0.9348
specificity	0.9313
ROC-AUC	0.9836
PR-AUC	0.9513
recall @ 1% FPR	0.6608
recall @ 0.1% FPR	0.4073
best-F1 threshold	0.955 (F1 0.8852)
credential-lure slice	398 rows (61.1% labelled positive), recall 0.9959, precision 0.9030

Three-way: no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,684 flagged true positives the model called 802 phishing and 2,882 spam.

test3 split (three-way holdout): test_holdout3.parquet, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 501.6 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 1.7 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9787 (26 missed of 1,221)
precision	0.9860 (17 false alarms)
F1	0.9823
accuracy	0.9866
specificity	0.9915
ROC-AUC	0.9991
PR-AUC	0.9983
recall @ 1% FPR	0.9795
recall @ 0.1% FPR	0.7748
best-F1 threshold	0.169 (F1 0.9849)
credential-lure slice	275 rows (97.5% labelled positive), recall 0.9888, precision 1.0000

Three-way on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9842, macro-F1 0.9807; Legitimate P 0.987 / R 0.991 / F1 0.989 (n=1,999); Phishing P 0.988 / R 0.973 / F1 0.981 (n=521); Spam P 0.973 / R 0.971 / F1 0.972 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,982/5/12; Phishing 7/507/7; Spam 19/1/680. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9910 (10 phishing->spam, 1 spam->phishing).

source	n	Legitimate	Phishing	Spam	accuracy	macro-F1 (present classes)
Assassin	236	164	0	72	0.9619	0.9556
CEAS-08	736	736	0	0	0.9864	0.9932
Enron	628	0	0	628	0.9793	0.9895
Nazario	222	0	222	0	0.9685	0.9840
Nigerian_Fraud	299	0	299	0	0.9766	0.9882
TREC-07	1,099	1,099	0	0	0.9955	0.9977

Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms. GPU bf16: 5.05 ms avg / 4.42 ms p50 over 100 emails (0.07x). CPU fp32 (16 threads): 246.62 ms avg / 242.67 ms p50 over 100 emails (3.33x).

Length confound (test). FN mean 260 tokens vs TP 286; FP 1105 vs TN 825 (see error_analysis.md for the per-bucket table).

Artifacts. reports/eval_ft-12L: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py plot).

Fine-tune run ft-scratch — 2026-10-09 09:57

Dials. init scratch; 3 classes; 24 layers, 0 frozen; loss cross_entropy; lr 3e-05; batch 32 (micro 16); epochs 2 planned; synthetic off (0 rows); selection val_f1_unwanted; train rows 160,263, val rows 15,284; params 337,444,867 (337,444,867 trainable). Result. 7500 steps, stopped by early_stopping; training time 60.9 min (eval 5.5 min of it); peak reserved 13.13 GiB. Best val at step 5009 (epoch 0), Unwanted (phishing or spam) roll-up: F1 0.9875, precision 0.9864, recall 0.9885, accuracy 0.9864, ROC-AUC 0.9985, PR-AUC 0.9986; F1-optimal threshold 0.601. Three-way on 8,105 fully-labelled val rows: macro-F1 0.9538, accuracy 0.9832; per-class F1 Legitimate 0.9915, Phishing 0.9398, Spam 0.9300; phishing-vs-spam on true unwanted (n=1,103): phish->phish 322, phish->spam 18, spam->phish 1, spam->spam 762. Checkpoints: /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-scratch/best, /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-scratch/last.

20. 2026-10-09 09:57 — Evaluation on the test + test3 splits (evaluate.py run, ft-scratch)

Checkpoint. /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-scratch/best (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

test split (TREC-06 binary holdout): test_holdout.parquet, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 282.4 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9165 (325 missed of 3,894)
precision	0.7752 (1,035 false alarms)
F1	0.8400
accuracy	0.9166
specificity	0.9166
ROC-AUC	0.9755
PR-AUC	0.9456
recall @ 1% FPR	0.7773
recall @ 0.1% FPR	0.4954
best-F1 threshold	0.941 (F1 0.8772)
credential-lure slice	398 rows (61.1% labelled positive), recall 0.9959, precision 0.9308

Three-way: no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,569 flagged true positives the model called 332 phishing and 3,237 spam.

test3 split (three-way holdout): test_holdout3.parquet, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 237.5 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9771 (28 missed of 1,221)
precision	0.9763 (29 false alarms)
F1	0.9767
accuracy	0.9823
specificity	0.9855
ROC-AUC	0.9983
PR-AUC	0.9974
recall @ 1% FPR	0.9672
recall @ 0.1% FPR	0.8354
best-F1 threshold	0.366 (F1 0.9776)
credential-lure slice	275 rows (97.5% labelled positive), recall 0.9925, precision 1.0000

Three-way on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9727, macro-F1 0.9634; Legitimate P 0.986 / R 0.985 / F1 0.986 (n=1,999); Phishing P 0.990 / R 0.927 / F1 0.957 (n=521); Spam P 0.925 / R 0.970 / F1 0.947 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,970/3/26; Phishing 9/483/29; Spam 19/2/679. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9713 (32 phishing->spam, 3 spam->phishing).

source	n	Legitimate	Phishing	Spam	accuracy	macro-F1 (present classes)
Assassin	236	164	0	72	0.9407	0.9330
CEAS-08	736	736	0	0	0.9796	0.9897
Enron	628	0	0	628	0.9777	0.9887
Nazario	222	0	222	0	0.8964	0.9454
Nigerian_Fraud	299	0	299	0	0.9498	0.9743
TREC-07	1,099	1,099	0	0	0.9936	0.9968

Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms. GPU bf16: 10.06 ms avg / 8.92 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 354.05 ms avg / 338.14 ms p50 over 100 emails (4.78x).

Length confound (test). FN mean 334 tokens vs TP 280; FP 728 vs TN 855 (see error_analysis.md for the per-bucket table).

Artifacts. reports/eval_ft-scratch: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py plot).

21. 2026-10-09 09:59 — Evaluation on the test + test3 splits (evaluate.py run, ft-ce)

Checkpoint. /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-ce/best (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

test split (TREC-06 binary holdout): test_holdout.parquet, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 285.2 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9404 (232 missed of 3,894)
precision	0.8610 (591 false alarms)
F1	0.8990
accuracy	0.9495
specificity	0.9524
ROC-AUC	0.9875
PR-AUC	0.9626
recall @ 1% FPR	0.7745
recall @ 0.1% FPR	0.3529
best-F1 threshold	0.821 (F1 0.9090)
credential-lure slice	398 rows (61.1% labelled positive), recall 0.9959, precision 0.8996

Three-way: no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,662 flagged true positives the model called 1,294 phishing and 2,368 spam.

test3 split (three-way holdout): test_holdout3.parquet, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 239.6 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9853 (18 missed of 1,221)
precision	0.9877 (15 false alarms)
F1	0.9865
accuracy	0.9898
specificity	0.9925
ROC-AUC	0.9992
PR-AUC	0.9985
recall @ 1% FPR	0.9869
recall @ 0.1% FPR	0.8690
best-F1 threshold	0.411 (F1 0.9869)
credential-lure slice	275 rows (97.5% labelled positive), recall 0.9925, precision 1.0000

Three-way on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9863, macro-F1 0.9822; Legitimate P 0.991 / R 0.992 / F1 0.992 (n=1,999); Phishing P 0.981 / R 0.973 / F1 0.977 (n=521); Spam P 0.977 / R 0.979 / F1 0.978 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,984/4/11; Phishing 9/507/5; Spam 9/6/685. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9894 (7 phishing->spam, 6 spam->phishing).

source	n	Legitimate	Phishing	Spam	accuracy	macro-F1 (present classes)
Assassin	236	164	0	72	0.9449	0.9464
CEAS-08	736	736	0	0	0.9905	0.9952
Enron	628	0	0	628	0.9889	0.9944
Nazario	222	0	222	0	0.9505	0.9746
Nigerian_Fraud	299	0	299	0	0.9900	0.9950
TREC-07	1,099	1,099	0	0	0.9973	0.9986

Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms. GPU bf16: 9.89 ms avg / 8.93 ms p50 over 100 emails (0.13x). CPU fp32 (16 threads): 513.57 ms avg / 485.83 ms p50 over 100 emails (6.94x).

Length confound (test). FN mean 256 tokens vs TP 287; FP 763 vs TN 848 (see error_analysis.md for the per-bucket table).

Artifacts. /home/nathank/BITS/dev/Phishing Model/reports: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py plot).

22. 2026-10-09 10:46 — Evaluation on the test + test3 splits (evaluate.py run, ft-focal)

Checkpoint. /home/nathank/BITS/dev/Phishing Model/checkpoints/finetune/ft-focal/best (3-class head; three-way label contract of RUN_LOG entry 6). Binary roll-up metrics use the same definitions as the binary-era runs so they stay comparable.

test split (TREC-06 binary holdout): test_holdout.parquet, 16,302 rows (3,894 unwanted, 23.9%; label3 legit 12,408, unwanted 3,894). Inference 284.1 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9741 (101 missed of 3,894)
precision	0.8794 (520 false alarms)
F1	0.9243
accuracy	0.9619
specificity	0.9581
ROC-AUC	0.9946
PR-AUC	0.9848
recall @ 1% FPR	0.8955
recall @ 0.1% FPR	0.6497
best-F1 threshold	0.682 (F1 0.9397)
credential-lure slice	398 rows (61.1% labelled positive), recall 0.9959, precision 0.9758

Three-way: no phishing-vs-spam ground truth on this split (no rows labelled phishing or spam: every positive is unwanted-but-unsplit (no phishing-vs-spam ground truth)); informational: of the 3,793 flagged true positives the model called 974 phishing and 2,819 spam.

test3 split (three-way holdout): test_holdout3.parquet, 3,220 rows (1,221 unwanted, 37.9%; label3 legit 1,999, phish 521, spam 700). Inference 237.6 rows/s on cuda (bf16 autocast, batch 64, length-bucketed), peak VRAM 2.27 GB.

binary roll-up metric (P(unwanted) = P(phishing) + P(spam))	value
recall (unwanted)	0.9836 (20 missed of 1,221)
precision	0.9893 (13 false alarms)
F1	0.9864
accuracy	0.9898
specificity	0.9935
ROC-AUC	0.9993
PR-AUC	0.9989
recall @ 1% FPR	0.9869
recall @ 0.1% FPR	0.8788
best-F1 threshold	0.475 (F1 0.9877)
credential-lure slice	275 rows (97.5% labelled positive), recall 1.0000, precision 1.0000

Three-way on the 3,220 rows with label3 in {0,1,2} (0 unsplit rows enter the roll-up only; rule: rollup_then_argmax): accuracy 0.9885, macro-F1 0.9867; Legitimate P 0.990 / R 0.993 / F1 0.992 (n=1,999); Phishing P 0.996 / R 0.987 / F1 0.991 (n=521); Spam P 0.979 / R 0.976 / F1 0.977 (n=700). 3x3 (rows true, cols predicted Legitimate/Phishing/Spam): Legitimate 1,986/1/12; Phishing 4/514/3; Spam 16/1/683. Phishing-vs-spam on the 1,221 true unwanted rows: kind accuracy 0.9959 (4 phishing->spam, 1 spam->phishing).

source	n	Legitimate	Phishing	Spam	accuracy	macro-F1 (present classes)
Assassin	236	164	0	72	0.9534	0.9458
CEAS-08	736	736	0	0	0.9905	0.9952
Enron	628	0	0	628	0.9841	0.9920
Nazario	222	0	222	0	0.9865	0.9932
Nigerian_Fraud	299	0	299	0	0.9866	0.9933
TREC-07	1,099	1,099	0	0	0.9982	0.9991

Latency (batch 1, tokenize + forward + softmax, timeit.default_timer) vs the instructor's 74 ms. GPU bf16: 10.08 ms avg / 8.9 ms p50 over 100 emails (0.14x). CPU fp32 (16 threads): 458.57 ms avg / 445.01 ms p50 over 100 emails (6.20x).

Length confound (test). FN mean 322 tokens vs TP 284; FP 1237 vs TN 827 (see error_analysis.md for the per-bucket table).

Artifacts. /home/nathank/BITS/dev/Phishing Model/reports: metrics.json, latency.json, confusion_matrix.png, roc_curve.png, pr_curve.png, error_analysis.md; metrics_test3.json, latency_test3.json, confusion_matrix_test3.png, confusion_matrix3_test3.png, roc_curve_test3.png, pr_curve_test3.png, error_analysis_test3.md (training curves: reports/pretrain_curves.png and finetune_curves.png from pretrain.py / finetune.py plot).

23. 2026-10-12 11:30 — Fine-tune queue results, model selection (ft-focal), export

Queue (scripts/run_experiments.sh, tmux phish-finetune): finished 2026-10-09 10:02:22, every step exit 0 (ft-ce 03:36-04:31, ft-weighted -05:57, ft-focal -07:24, ft-synthetic -08:19, ft-12L -08:56, ft-scratch -09:59, then top-level evaluate + export of the pre-designated main run ft-ce). Log logs/experiments_20261008-141528.out. Per-run evaluations in reports/eval_<run>/ (entries 15-20, 21).

Selection rule and result. The queue's main run was ft-ce by default, but the rule stated in README section 9 was "if another run wins on val F1 of the unwanted roll-up, it becomes the main run". ft-focal (focal loss, gamma 2.0) won on the validation set on both selection metrics: val F1 unwanted 0.9951 vs 0.9929 (ft-ce), val three-way macro-F1 0.9869 vs 0.9782; it also had the best val phishing precision/recall (0.985 / 0.988). The decision was taken from the validation numbers; the holdouts were evaluated once per run for reporting and confirm it (TREC-06 F1 0.9243 vs 0.8990, recall at 1% FPR 0.8955 vs 0.7745; test3 macro-F1 0.9867 vs 0.9822). Top-level evaluate was re-run for ft-focal on 2026-10-09 10:46 (entry 22, reports/metrics*.json, latency.json, plots, error_analysis*.md).

Held-out results of the deployed run ft-focal:

split	numbers
TREC-06 (16,302 rows, 23.9% unwanted, whole corpus unseen)	recall 0.9741 (101 missed), precision 0.8794 (520 false alarms, FPR 4.2%), F1 0.9243, accuracy 0.9619, ROC-AUC 0.9946, PR-AUC 0.9848, recall at 1% / 0.1% FPR 0.8955 / 0.6497
test_holdout3 (3,220 rows: 1,999 / 521 / 700)	macro-F1 0.9867, accuracy 0.9885; F1 legit / phishing / spam 0.9918 / 0.9913 / 0.9771; binary roll-up F1 0.9864, ROC-AUC 0.9993; phishing-vs-spam on true unwanted 0.9959 (4 phishing->spam, 1 spam->phishing); 3x3 [[1986, 1, 12], [4, 514, 3], [16, 1, 683]]
latency, batch 1, 100 TREC-06 emails (mean 280 tokens)	GPU bf16 10.08 ms mean / 8.90 p50 / 13.72 p95 (0.14x the 74 ms benchmark); CPU fp32 16 threads 458.6 ms mean / 445.0 p50 / 701.1 p95 (6.20x)

What the ablations say (all six on the same holdouts, README 9.1): class-weighted CE trades precision for recall (TREC-06 recall 0.982 but FPR 6.3%); focal wins on every metric; the 2,000-row synthetic supplement changes nothing measurable (TREC-06 F1 0.8995 vs 0.8990; val macro-F1 lower, because its 1,000 phishing lures are template-generated); 12 layers lose ~5 F1 points on TREC-06 (0.8739) for ~2x faster inference; scratch init loses 8 points (0.8400) and 1.7 ROC-AUC points, which is what the 15 h of MLM pretraining buys.

Challenge recorded: TREC-06 labelling conventions. The TREC-06 numbers are the honest, cross-corpus ones and sit well below the in-distribution validation numbers (F1 0.92 vs 0.995). reports/error_analysis.md shows the two causes: (1) TREC-06 labels as ham a lot of mail that looks like spam to any text model (spam quoted inside mailing-list discussions, "please remove me from your list" replies, sweepstakes mail, "57 Million Email Addresses - ONLY $149!!" on a developer list), which accounts for the most confident false positives; (2) its largest block of spam is "ScanMail Message: virus found" gateway notices, which every run misses. Legitimate TREC-06 mail is also ~3x longer than its spam and 24% of it is truncated at 512 tokens; false positives are truncated 31.5% of the time vs 23.6% for true negatives. Neither is a leakage problem, but both cap what any text-only classifier can score on this corpus, and they are the right thing to show in the presentation when explaining the gap between 0.995 validation and 0.92 TREC-06.

Export. 2026-10-12 11:28: export.py run --checkpoint checkpoints/finetune/ft-focal/best --out-dir model_export/ft-focal --app-dir app -> model_export/ft-focal/ (bf16 safetensors 674.9 MB, config.json with labels + metrics + dials, tokenizer.json, model-card README.md, export_manifest.json) and the same three files into app/; app/model.py is byte-identical to src/model.py (sha1 39b3773c, asserted). This replaced the ft-ce weights that the queue had copied into app/ on 10-09 10:02; model_export/ft-ce/ is kept for comparison. README (sections 2.1, 2.2, 5.6-5.8, 9, 10, 11), app/README.md (Space card: bundled weights, training recipe, real model-card link) and scripts/build_notebook.py were updated to name ft-focal as the deployed run; the notebook was rebuilt.

Gap in the record. The 2026-10-09 10:43 scheduled check-in completed the queue review and the ft-focal top-level evaluation (entry 22) but stalled before export/upload; resumed on 2026-10-12 11:27 at Nathan's request. find -newermt confirmed nothing in the project changed in between.

24. 2026-10-12 11:35 — Hugging Face upload (private model repo + private ZeroGPU Space)

Approved by Nathan on 2026-10-08 19:15 ("can you upload the hugging face?"); deferred until an export existed. Account: BuildITSystems via the hf auth login cache (token never printed or written anywhere).

Dry run 11:30 (scripts/upload_to_hf.py --model-dir model_export/ft-focal --bundle-weights): no problems; plan = 4 files / 645.9 MiB to the model repo, 8 files / 645.9 MiB to the Space (weights bundled, so no HF_TOKEN secret), hardware zero-a10g, both private.

Live run 1 11:31 (--yes): create_repo + upload_folder for the model repo succeeded (commit 80bd85c5: README.md model card, config.json, model.safetensors 674.9 MB bf16, tokenizer.json). The Space was created and its PHISH_MODEL_REPO variable set, but upload_folder(app/) was refused by the Hub's README validation: Invalid YAML in README.md: bad indentation of a mapping entry (11:45) — the unquoted short_description: From-scratch 337M encoder: legitimate / phishing / spam contains a colon. The dry run does not validate the card's YAML, so this was invisible until the live call. Exit 1.

Fix 11:34: quoted the value in app/README.md (55 chars, under the 60-char limit); verified with yaml.safe_load and huggingface_hub.RepoCard. Lesson added to README 5.9.

Live run 2 11:35 (--yes again, idempotent): model-repo files already present (LFS content dedupe; commit unchanged), Space upload_folder succeeded (commit 98f90d7b). Verified read-only afterwards:

repo	private	files
https://huggingface.co/BuildITSystems/phishing-detector-encoder	yes	.gitattributes, README.md, config.json, model.safetensors 674.9 MB, tokenizer.json 2.3 MB
https://huggingface.co/spaces/BuildITSystems/phishing-detector-demo	yes	.gitattributes, README.md (card), app.py, model.py, requirements.txt, config.json, model.safetensors 674.9 MB, tokenizer.json 2.3 MB

Space runtime at 11:36: stage BUILDING, requested hardware zero-a10g; variable PHISH_MODEL_REPO=BuildITSystems/phishing-detector-encoder; no secrets. Deployed weights = ft-focal (entry 23). Before class Nathan flips both repos to public (README section 10 one-liner) and opens the Space once so ZeroGPU is warm.

Build and live check (2026-10-12 11:37-11:39). Space stages: BUILDING 11:35 -> APP_STARTING 11:36 -> RUNNING on zero-a10g 11:37 (about 2 minutes end to end). Verified through the private Space's Gradio API (gradio_client, endpoint /predict(subject, body), token from the login cache): a sample credential-lure email ("verify your account within 24 hours", lookalike PayPal link) returned verdict Phishing 95.6%, Spam 3.5%, Legitimate 0.9%, roll-up Unwanted 99.1%, in-app "Prediction Time (s)" 0.253 (cold ZeroGPU first call), round-trip 3.6 s, with the token highlights and the rule-based signals populated. The deployment is working end to end.

25. 2026-10-12 11:58 — Presentation bundle on the Hub + Colab variant of the notebook

Why. Nathan will present from Google Colab, where none of the project files exist and the local GPU is irrelevant. Design: the notebook is narrative only; every prediction runs on the ZeroGPU Space (gr.load(..., src="spaces") to embed the Space UI, gradio_client for the timing loop), and the evidence the notebook renders (metrics, curves, experiments table, data report, run log, code) is fetched from the model repo.

Bundle upload (approved by Nathan 2026-10-12 11:50). presentation/ in the private model repo BuildITSystems/phishing-detector-encoder, commit 439e7b1a: 85 files, 3.6 MB, relative paths mirrored from the project (reports/ JSON/PNG/MD incl. eval_<run>/, checkpoints/pretrain/large-24L/{run_summary.json, metrics.csv,final/config.json}, checkpoints/finetune/<run>/{metrics.csv,epochs.csv}, data/processed/{splits_manifest,tokenizer_report,pretrain_tokens}.json + data_report.md, docs/research/research_model-math.md, src/*.py, app/{app.py,model.py,requirements.txt,README.md}, config.yaml, requirements-train.txt, scripts/*, model_export/ft-focal/{config.json,README.md, export_manifest.json}, RUN_LOG.md, README.md). Excluded on purpose: every parquet/npy/safetensors/log. Scanned before upload for tokens (hf_...), HF_TOKEN= values, Nathan's e-mail address, private keys and cloud keys: clean. The quoted emails in error_analysis*.md are rows of public corpora (TREC-06 etc.), as the files themselves state. The Hub warned that presentation/README.md has no YAML front matter (it is the project README, not a model card): harmless.

Colab notebook build (workflow wf_4200da59, 8 agents, 53 min, 12:00-12:53). scripts/build_notebook.py gained --colab and --out PATH; cells are tagged local / colab / shared so both notebooks come from one source (default mode proven byte-identical in cell sources against the executed deliverable, which was never rewritten: cmp identical before and after). The Colab variant (49 cells, 36 code) keeps the nine instructor headers and the story; Steps 2-4 and the Results read the Hub bundle; Step 5 calls the Space through gradio_client; Step 7 embeds the Space with gr.load; Step 8 logs in only when the token is not already active (huggingface_hub 1.33 login(token=...) otherwise rewrites the token files unconditionally); Step 9 quotes entry 24. Two adversarial reviews found 6 items, all fixed: per-call error handling around the Space calls (one ZeroGPU abort must not kill the cell), the login() behaviour above, analytics_enabled=False on the client (the env var alone does not disable gradio_client telemetry), a note that gr.load forwards the token to the Space (use a fine-grained read token as the Colab secret), the login-cache fallback made opt-in outside Colab (NOTEBOOK_ALLOW_LOGIN_CACHE=1), and silencing the doomed first gr.load attempt.

Challenge recorded: gr.load bug in gradio 6.29.1. gr.load("BuildITSystems/phishing-detector-demo", src="spaces") raises TypeError: Textbox.__init__() got an unexpected keyword argument 'class_id': Blocks.from_config rebuilds the Interface's examples Dataset, whose serialised child components are {"name", "class_id"} dicts, and gradio.components.base.get_component_instance pops only name. Any Space whose gr.Interface has examples= triggers it. Reproduced independently on the workstation. Fix in the notebook: catch the error, install an idempotent shim that drops class_id, retry gr.load; keep a hand-built proxy gr.Interface over gradio_client as the last fallback. Verified: gr.load succeeds after the shim.

Headless Colab-mimic test (clean uv venv, CPython 3.12.14, torch 2.14.1+cpu, gradio 6.29.1, gradio_client 2.7.2, huggingface_hub 1.33.0, executed from an empty scratch directory, HF_TOKEN unset, token only from the login cache via the opt-in flag). Round 1 ran 36/36 cells with no errors but gr.load had fallen back to the proxy (flagged as a failure by the harness); after the shim fix round 2 passed every check: 36/36 cells, 0 errors, 18 s wall, bundle 86 files / 3.4 MiB, six live Space predictions (in-app times 0.229 s cold then 0.009- 0.011 s; e.g. "Microsoft 365 password expires" -> Phishing 93.7%, "FINAL HOURS: 70% OFF" -> Spam 99.0%, "Agenda for Thursday" -> Legitimate 98.6%), gr.load served and closed, whoami = BuildITSystems, zero token-shaped strings in outputs or raw file. Final verification: TREC-06 F1 0.9243 / recall 0.9741 / ROC-AUC 0.9946 and the binary confusion 11888/520/101/3793, three-way macro-F1 0.9867 and the 3x3 matrix all match reports/. Presenter notes, non-blocking: the iframe cell prints "needs a public Space" until the Space is public; the gr.load cell shows the shim trace line; some outputs echo workstation absolute paths that live inside the mirrored JSON files.

gradio 6.30.0 checked (released after the pin): gr.load fails identically (class_id), tested in a throwaway venv against the Space, so bumping the pin would not remove the shim. Keep 6.29.1 (= the Space's sdk_version) in the Colab notebook until a release fixes get_component_instance.

Bundle refresh 2026-10-12 13:05: re-uploaded presentation/RUN_LOG.md, presentation/README.md and presentation/scripts/build_notebook.py (all changed by this entry) and added presentation/FinalProject_PhishingDetector_Colab.ipynb so the notebook can be downloaded from the model repo and uploaded to Colab in one step. Same approval as the bundle; same scan; nothing else changed on the Hub.