{ "cells": [ { "cell_type": "markdown", "id": "cell-00", "metadata": {}, "source": [ "# BITS Phishing Detector: a from-scratch 337M-parameter email classifier\n", "\n", "**Nathan King, Build IT Systems LLC (BITS)**\n", "Skillspire AI & Machine Learning course, final project. Presented in class, late October / early November 2026.\n", "\n", "**Abstract.** Small-business inboxes are triaged by people, not security teams, so this project builds a\n", "*defensive* phishing / spam triage model that a person can paste an email into. It is a text **classifier**\n", "(three classes: **legitimate**, **phishing**, **spam**) and nothing in it is pretrained or borrowed: the model is a\n", "24-layer BERT-style encoder written in plain PyTorch (`src/model.py`, 337M parameters), the tokenizer is a\n", "byte-level BPE trained on our own corpus, and the weights come from two training phases on one RTX 3090 Ti:\n", "masked-language-model pretraining on deduplicated Enron email, then fine-tuning on public labelled corpora\n", "with a partial-label loss (the TREC / CEAS positives never separated spam from phishing, so they train as\n", "\"unwanted\"). The clean TREC-06 corpus is held out as the binary test set and a separate clean-provenance\n", "three-way holdout gives the 3x3 confusion matrix. The classifier is exported to safetensors and deployed as a\n", "Gradio app on a Hugging Face Space (ZeroGPU). This notebook follows the nine steps of the course reference\n", "notebook and loads the real artifacts of the project; anything that is still being produced while the GPU\n", "trains is reported as \"not available yet\" together with the command that produces it." ] }, { "cell_type": "markdown", "id": "cell-01", "metadata": {}, "source": [ "**Colab edition.** This copy of the notebook runs with none of the project files present: the narrative, tables and\n", "curves are read from the `presentation/` bundle inside the model repo on the Hugging Face Hub (reports, metrics, curves,\n", "configs and source files, mirrored at the project's relative paths: no weights, no datasets), and every computation that\n", "needs the 337M-parameter model is delegated to the deployed Space on ZeroGPU. Nothing trains here and no weights are\n", "downloaded. While the two repos are private, add a Colab secret named `HF_TOKEN` (notebook access switched on). Make it a\n", "**fine-grained read-only token scoped to the two repos** (`BuildITSystems/phishing-detector-encoder` and\n", "`BuildITSystems/phishing-detector-demo`): `gr.load` and `gradio_client` forward the token to the Space with every proxied\n", "call, so it should never be a write token. Once the repos are public no token is needed." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-02", "metadata": {}, "outputs": [], "source": [ "# The Hub identifiers the notebook depends on (change them here if the repos are renamed or forked).\n", "MODEL_REPO = \"BuildITSystems/phishing-detector-encoder\" # model repo: bf16 weights + the presentation/ bundle\n", "SPACE = \"BuildITSystems/phishing-detector-demo\" # ZeroGPU Space running app/app.py (weights bundled)\n", "SPACE_HOST = \"https://builditsystems-phishing-detector-demo.hf.space\" # the Space's embed host (iframe; needs a public Space)\n", "PRESENTATION_PREFIX = \"presentation\" # folder in MODEL_REPO mirroring the project's relative paths" ] }, { "cell_type": "markdown", "id": "cell-03", "metadata": {}, "source": [ "**Step 1: Install Dependencies & Setup Device**\n", "\n", "Colab preinstalls torch, pandas and matplotlib; the two pins installed here are `gradio` 6.29.1 (the Space's own SDK\n", "version, so `gr.load` speaks the same protocol) and `huggingface_hub` 1.33.0 (the version the project's upload script was\n", "written against). The token is resolved in this order and only its *source* is printed, never its value: Colab secret\n", "`HF_TOKEN` -> environment variable `HF_TOKEN` -> the `hf auth login` cache -> none (public repos still work). Then the\n", "`presentation/` bundle (a few MB) is downloaded from the model repo and `ROOT` points at it, so every later cell reads the\n", "same relative paths as the local notebook; a file that is not in the bundle is reported by `artifact()` as \"not available\n", "yet\" with the command that produces it, exactly as on the workstation. \"Device\" here means the remote one: the Space's\n", "runtime is reported next to whatever accelerator this Colab runtime happens to have, which the model never uses." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-04", "metadata": {}, "outputs": [], "source": [ "import os, sys, json, time, re, tempfile\n", "from pathlib import Path\n", "from timeit import default_timer as timer\n", "\n", "IN_COLAB = \"google.colab\" in sys.modules\n", "if IN_COLAB:\n", " %pip install -q gradio==6.29.1 huggingface_hub==1.33.0\n", "else:\n", " print(\"not running in Colab: no pip install, the already-installed gradio / huggingface_hub are used\")\n", "os.environ.setdefault(\"HF_HUB_DISABLE_PROGRESS_BARS\", \"1\") # quiet downloads (must precede the huggingface_hub import)\n", "os.environ.setdefault(\"GRADIO_ANALYTICS_ENABLED\", \"False\") # gradio / gradio_client telemetry off\n", "os.environ.setdefault(\"HF_HUB_DISABLE_TELEMETRY\", \"1\") # huggingface_hub telemetry off\n", "import warnings\n", "warnings.filterwarnings(\"ignore\", message=\"IProgress not found\") # tqdm's ipywidgets nag in kernels without ipywidgets (Colab has it)\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import yaml\n", "import gradio\n", "import huggingface_hub\n", "from huggingface_hub import HfApi, snapshot_download\n", "from IPython.display import display, Image, Markdown\n", "\n", "T_NOTEBOOK_START = timer()\n", "pd.set_option(\"display.width\", 160); pd.set_option(\"display.max_columns\", 40)\n", "print(f\"python {sys.version.split()[0]} gradio {gradio.__version__} huggingface_hub {huggingface_hub.__version__} \"\n", " f\"pandas {pd.__version__} running in Colab: {IN_COLAB}\")\n", "\n", "# Token: Colab secret HF_TOKEN -> environment variable HF_TOKEN -> `hf auth login` cache -> none. Only the SOURCE is printed.\n", "# The token is forwarded to the Space by gr.load / gradio_client, so it should be a read-only token scoped to the two repos.\n", "# Outside Colab the login cache usually holds the workstation's WRITE token, so that fallback is opt-in there:\n", "# set NOTEBOOK_ALLOW_LOGIN_CACHE=1 (or export HF_TOKEN=) before running the headless test.\n", "def resolve_token():\n", " if IN_COLAB:\n", " try:\n", " from google.colab import userdata\n", " if userdata.get(\"HF_TOKEN\"):\n", " return userdata.get(\"HF_TOKEN\"), \"Colab secret HF_TOKEN\"\n", " except Exception: # the secret is not defined, or notebook access is off in the Secrets panel\n", " pass\n", " if os.environ.get(\"HF_TOKEN\"):\n", " return os.environ[\"HF_TOKEN\"], \"environment variable HF_TOKEN\"\n", " if (IN_COLAB or os.environ.get(\"NOTEBOOK_ALLOW_LOGIN_CACHE\")) and huggingface_hub.get_token():\n", " return huggingface_hub.get_token(), \"huggingface_hub login cache (hf auth login)\"\n", " return None, \"none (anonymous: works only once the repos are public)\"\n", "\n", "TOKEN, TOKEN_SOURCE = resolve_token()\n", "print(\"Hugging Face token source:\", TOKEN_SOURCE)\n", "api = HfApi(token=TOKEN)" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-05", "metadata": {}, "outputs": [], "source": [ "# The presentation/ bundle: reports, curves, metrics, configs and source files mirrored at the project's relative paths (no\n", "# weights, no datasets). ROOT points at it, so every later cell reads the same paths as the local notebook.\n", "BUNDLE_DIR = Path(\"/content/phishing_bundle\") if IN_COLAB else Path(tempfile.gettempdir()) / \"phishing_bundle\"\n", "try:\n", " snap = Path(snapshot_download(MODEL_REPO, repo_type=\"model\", allow_patterns=[f\"{PRESENTATION_PREFIX}/*\", \"config.json\"],\n", " token=TOKEN, local_dir=BUNDLE_DIR))\n", "except Exception as e:\n", " raise RuntimeError(f\"cannot download the bundle from {MODEL_REPO} ({type(e).__name__}): while the repo is private this notebook \"\n", " \"needs a read token for the BuildITSystems account (Colab: Secrets panel -> HF_TOKEN, notebook access on)\") from e\n", "ROOT = snap / PRESENTATION_PREFIX\n", "EXPORT_CONFIG = json.loads((snap / \"config.json\").read_text()) # the deployed model's config.json: labels, dials, test metrics\n", "bundle_files = sorted(p for p in ROOT.rglob(\"*\") if p.is_file())\n", "print(f\"bundle: {len(bundle_files)} files, {sum(p.stat().st_size for p in bundle_files) / 2**20:.1f} MiB under {ROOT}\")\n", "\n", "PRETRAIN_RUN = \"large-24L\" # RUN_NAME in scripts/launch_pretrain.sh\n", "CFG = yaml.safe_load((ROOT / \"config.yaml\").read_text()) # the project config; its paths stay relative (nothing resolves them here)\n", "\n", "MISSING: list[tuple[str, str]] = []\n", "def artifact(rel: str, produced_by: str):\n", " \"\"\"Path to a project artifact in the bundle, or None (with a one-line notice) if it is not there.\"\"\"\n", " p = ROOT / rel\n", " if p.exists():\n", " return p\n", " print(f\"not available yet — {rel} (produced by: {produced_by})\")\n", " MISSING.append((rel, produced_by))\n", " return None\n", "\n", "def show(df, title=None, fmt=None):\n", " if title:\n", " display(Markdown(f\"**{title}**\"))\n", " display(df.style.format(fmt) if fmt else df)" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-06", "metadata": {}, "outputs": [], "source": [ "# \"Device\": the model never runs in this notebook. Compute lives on the Space; the accelerator Colab gave this runtime is informational.\n", "try:\n", " rt = api.get_space_runtime(SPACE)\n", " SPACE_HW = rt.hardware\n", " print(f\"Space {SPACE}: stage {rt.stage}, hardware {rt.hardware} <- the device every prediction in this notebook runs on\")\n", "except Exception as e:\n", " SPACE_HW = CFG[\"hf\"][\"space_hardware\"]\n", " print(f\"Space runtime not readable ({type(e).__name__}): private Space without a token, or the Space does not exist; \"\n", " f\"config.yaml says hardware {SPACE_HW}\")\n", "try:\n", " import torch\n", " TORCH_OK = True\n", " local = f\"torch {torch.__version__}, cuda available: {torch.cuda.is_available()}\" + (f\" ({torch.cuda.get_device_name(0)})\" if torch.cuda.is_available() else \"\")\n", "except Exception as e: # torch is preinstalled on Colab; elsewhere the notebook falls back to the counts recorded in config.json\n", " TORCH_OK = False\n", " local = f\"torch not importable ({type(e).__name__})\"\n", "print(f\"this runtime: {local} (informational: not used for the model)\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-07", "metadata": {}, "outputs": [], "source": [ "# The GPU facts recorded by the smoke test on 2026-10-07 (the GPU is not visible to this kernel).\n", "smoke = artifact(\"reports/smoke_test.json\", \"src/smoke_test.py\")\n", "SMOKE = json.loads(smoke.read_text()) if smoke else {} # reused by Step 5 and Results (no retyped numbers)\n", "if smoke:\n", " s = SMOKE[\"device\"]\n", " print(\"Training GPU (from reports/smoke_test.json):\")\n", " for k in (\"device\", \"sm\", \"vram_gib\", \"bf16\", \"torch\", \"cuda_build\", \"cudnn\", \"python\"):\n", " print(f\"{k:>10}: {s[k]}\")\n", "\n", "print(\"\\nPinned training environment (requirements-train.txt):\")\n", "print((ROOT / \"requirements-train.txt\").read_text())" ] }, { "cell_type": "markdown", "id": "cell-08", "metadata": {}, "source": [ "**Step 2: Download Data & Create DataLoaders**\n", "\n", "The course reference downloads FashionMNIST from `torchvision.datasets`; there is no phishing corpus in torchvision,\n", "so our data comes from the Hugging Face Hub and is rebuilt locally by `src/data_prep.py` (download, normalise,\n", "dedupe, decontaminate, split, assert zero leakage). The Hub datasets that were inspected:\n", "\n", "| dataset | role | verdict |\n", "|---|---|---|\n", "| `puyang2025/seven-phishing-email-datasets` | labelled emails (TREC-05/06/07, CEAS-08, Enron-Spam, SpamAssassin, Ling) | **used**; its own train/test split leaks (5.1% exact / ~40% near duplicates), so we split ourselves |\n", "| `corbt/enron-emails` | unlabelled Enron mail for MLM pretraining | **used** after removing 52.7% exact duplicates and stripping quoted tails |\n", "| `it4lia/PhishingEmailCuratedDatasets_Cleaned` | labelled | **excluded**: no body text, only subjects + 53 numeric features |\n", "| `zefang-liu/phishing-email-dataset` | labelled | **excluded**: 91.9% exact subset of the primary set, lowercased, and 20% of its \"safe\" rows contain the token *enron* (a shortcut) |\n", "| `kudzaiprichard/aura-phishing-email-corpus` (Nazario + Nigerian fraud) | phishing-only corpora for the three-way labels | **used** (added in RUN_LOG entry 7) |\n", "| `locuoco/...-300000` | ready-made 3-class set | **rejected**: 9.9% of its rows are copies of our TREC-06 test set and its \"phish\" label is a preprocessing artifact |\n", "\n", "**Holdouts.** TREC-06 (16,302 rows, 23.9% positive) is the binary test set: it is the one sub-corpus that is not\n", "Enron-derived (89% of TREC-05 ham and 86% of Enron-Spam ham appear in the pretraining corpus), and it was never seen\n", "by the tokenizer, the pretraining or the fine-tuning. A second, clean-provenance **three-way holdout** (3,220 rows:\n", "legitimate / phishing / spam) gives the 3x3 confusion matrix. Train and validation are split by near-duplicate\n", "cluster so no spam template straddles the two." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-09", "metadata": {}, "outputs": [], "source": [ "manifest = json.loads((ROOT / \"data/processed/splits_manifest.json\").read_text())\n", "ds = manifest[\"datasets\"]\n", "rows = [\n", " (\"labelled primary\", ds[\"labelled_primary\"][\"repo\"], ds[\"labelled_primary\"][\"rows_loaded\"],\n", " f\"hub splits {ds['labelled_primary']['hub_splits_concatenated']} concatenated (the hub split leaks)\"),\n", " (\"pretraining corpus\", ds[\"pretrain_corpus\"][\"repo\"], ds[\"pretrain_corpus\"][\"rows_total\"],\n", " f\"-> {manifest['pretrain']['rows_out']:,} docs after dedup/quote stripping\"),\n", "] + [(\"excluded\", r, \"-\", reason) for r, reason in zip(ds[\"excluded\"],\n", " (\"no body text (subjects + 53 numeric features only)\", \"91.9% exact subset of the primary set, lowercased, 'enron' token shortcut\"))]\n", "rows.append((\"phishing corpora (3-way)\", CFG[\"data\"][\"threeway\"][\"champa\"][\"repo\"],\n", " sum(manifest[\"threeway\"][\"new_rows\"][\"loaded_by_source\"].values()), \"Nazario + Nigerian_Fraud, CC-BY-4.0\"))\n", "show(pd.DataFrame(rows, columns=[\"role\", \"Hub dataset\", \"rows loaded\", \"note\"]).astype({\"rows loaded\": \"string\"}), \"Hub datasets\")\n", "print(f\"data build: {manifest['timestamp']} seed {manifest['seed']} elapsed {manifest['elapsed_seconds']} s peak RSS {manifest['peak_rss_gb']} GB\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-10", "metadata": {}, "outputs": [], "source": [ "# Dedup / leakage procedure: every row is normalised (NFKC -> lowercase -> [a-z0-9]) and keyed by sha1 of the full\n", "# text, a 300-char prefix and subject+body; near-duplicates share >= 3 of their 50-char shingles (stride 25).\n", "# These are the checks data_prep asserts to be ZERO (recomputed independently from the written files by `check3`).\n", "leak = manifest[\"threeway\"][\"leakage\"]\n", "tbl = pd.DataFrame([{\"check\": k, \"count\": v[\"count\"], \"asserted zero\": v[\"asserted\"], \"pass\": v[\"pass\"], \"detail\": v.get(\"detail\", \"\")}\n", " for k, v in leak.items()])\n", "show(tbl[tbl[\"asserted zero\"]].reset_index(drop=True), \"Leakage checks asserted to be zero (binary gate + three-way)\")\n", "show(tbl[~tbl[\"asserted zero\"]].reset_index(drop=True), \"Informational counters (not asserted)\")\n", "D = manifest[\"drops\"]\n", "drops = pd.DataFrame([\n", " (\"labelled rows with < 20 alphanumerics (garbage / empty)\", D[\"labelled_min_alnum\"][\"rows\"]),\n", " (\"Ling corpus excluded (academic mailing list)\", sum(D[\"ling_excluded_by_label\"].values())),\n", " (\"identical body carrying both labels (label noise)\", D[\"pool_label_conflicts\"][\"rows_dropped\"]),\n", " (\"exact duplicates in the labelled pool\", sum(sum(v.values()) for v in D[\"pool_exact_dups_by_source_label\"].values())),\n", " (\"training rows that are exact/near duplicates of TREC-06 holdout rows\", D[\"decontamination_vs_holdout\"][\"by_reason\"][\"total\"]),\n", " (\"pretraining rows: exact duplicates\", manifest[\"pretrain\"][\"exact_dups_raw\"]),\n", " (\"pretraining rows matching a validation email\", manifest[\"pretrain\"][\"val_key_removed\"]),\n", " (\"pretraining rows near-duplicating a holdout email\", manifest[\"pretrain\"][\"holdout_near_dup_removed\"]),\n", "], columns=[\"removed\", \"rows\"]).set_index(\"removed\")\n", "show(drops, \"Rows removed by the dedup / decontamination procedure\", fmt=\"{:,}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-11", "metadata": {}, "outputs": [], "source": [ "# Rows per split by class, from the counts data_prep.py wrote into splits_manifest.json (the parquet splits themselves stay on\n", "# the workstation: the bundle carries reports, configs and source files, never data or weights).\n", "LABEL3_NAME = {0: \"Legitimate\", 1: \"Phishing\", 2: \"Spam\", -1: \"unwanted (unsplit)\"}\n", "SPLITS = [\"finetune_train\", \"finetune_val\", \"test_holdout\", \"test_holdout3\"]\n", "tw = manifest[\"threeway\"]\n", "lab = pd.DataFrame([{\"split\": split, \"source\": src, \"class\": LABEL3_NAME[int(l3)], \"rows\": n}\n", " for split in SPLITS for src, by_l3 in tw[\"counts\"][split].items() for l3, n in by_l3.items()])\n", "by_split = lab.pivot_table(index=\"split\", columns=\"class\", values=\"rows\", aggfunc=\"sum\", fill_value=0).reindex(SPLITS)\n", "by_split[\"rows\"] = by_split.sum(axis=1)\n", "by_split[\"binary positive rate\"] = pd.Series({s: tw[\"totals\"][s][\"binary_base_rate\"] for s in SPLITS})\n", "show(by_split, \"Rows per split by label3 (EDA, from splits_manifest.json)\")\n", "show(lab.pivot_table(index=[\"split\", \"source\"], columns=\"class\", values=\"rows\", aggfunc=\"sum\", fill_value=0), \"Rows per split x source x class\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-12", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(9, 3.8))\n", "cols = [\"Legitimate\", \"Phishing\", \"Spam\", \"unwanted (unsplit)\"]\n", "colors = [\"#2a78d6\", \"#eb6834\", \"#eda100\", \"#9aa0a6\"]\n", "bottom = np.zeros(len(SPLITS))\n", "for c, col in zip(cols, colors):\n", " vals = by_split[c].reindex(SPLITS).fillna(0).to_numpy() if c in by_split else np.zeros(len(SPLITS))\n", " ax.bar(SPLITS, vals, bottom=bottom, label=c, color=col)\n", " bottom += vals\n", "ax.set_ylabel(\"rows\"); ax.set_title(\"Labelled splits by class\", loc=\"left\"); ax.legend(frameon=False)\n", "for s in (\"top\", \"right\"): ax.spines[s].set_visible(False)\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-13", "metadata": {}, "outputs": [], "source": [ "# Tokenizer: byte-level BPE, vocab 32,768, NFKC only (case preserved: ALL-CAPS is a spam signal), trained on the\n", "# pretraining corpus + finetune_train ONLY (val and the holdouts are refused by name; 0 holdout hash collisions).\n", "# tokenizer.json travels with the weights (model repo root and the Space); the report written next to it carries the\n", "# statistics and the worked examples, so nothing is tokenised on this runtime.\n", "tok_report = json.loads((ROOT / \"data/processed/tokenizer_report.json\").read_text())\n", "tr = tok_report[\"train\"]\n", "print(f\"vocab {tr['final_vocab_size']:,} specials {tr['special_token_ids']} normalizer {tr['normalizer']} \"\n", " f\"pre-tokenizer {tr['pre_tokenizer']}\\ntrained on {list(tr['training_files'])} in {tr['train_seconds']} s; \"\n", " f\"excluded {tr['excluded_files']}; holdout exact collisions {tr['holdout_exact_collisions']}\")\n", "show(pd.DataFrame(tok_report[\"stats\"][\"splits\"]).set_index(\"split\")\n", " [[\"rows\", \"mean_tokens\", \"median_tokens\", \"p95_tokens\", \"chars_per_token\", \"pct_truncated\", \"mean_tokens_by_label\"]],\n", " \"Token-length statistics per split (tokenizer.py stats; max_seq_len 512)\")\n", "for ex in tok_report[\"stats\"][\"examples\"][:3]:\n", " toks = ex[\"tokens\"]\n", " print(f\"\\n[{ex['name']}] {ex['text'][:90]!r}\\n {ex['n_tokens_with_specials']} tokens: {toks[:28]}{' ...' if len(toks) > 28 else ''}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-14", "metadata": {}, "outputs": [], "source": [ "# Token length per split from the tokenizer report. The local notebook tokenises a seeded 5,000-row sample of finetune_train\n", "# live and draws a histogram per class; that sample needs the parquet data, so here the per-split statistics computed by\n", "# tokenizer.py on the workstation stand in for it.\n", "st = pd.DataFrame(tok_report[\"stats\"][\"splits\"]).set_index(\"split\")\n", "fig, ax = plt.subplots(figsize=(9, 3.8))\n", "x = np.arange(len(st)); w = 0.27\n", "for off, col, color in ((-w, \"median_tokens\", \"#2a78d6\"), (0, \"mean_tokens\", \"#eb6834\"), (w, \"p95_tokens\", \"#eda100\")):\n", " ax.bar(x + off, st[col], width=w, color=color, label=col.replace(\"_\", \" \"))\n", "ax.axhline(CFG[\"model\"][\"max_seq_len\"], color=\"black\", linestyle=\"--\", linewidth=1, label=\"max_seq_len 512 (truncation)\")\n", "ax.set_xticks(x, st.index); ax.set_yscale(\"log\"); ax.set_ylabel(\"tokens per email (log)\")\n", "ax.set_title(\"Token length per split (tokenizer.py stats)\", loc=\"left\"); ax.legend(frameon=False, fontsize=8)\n", "for s in (\"top\", \"right\"): ax.spines[s].set_visible(False)\n", "plt.tight_layout(); plt.show()\n", "print(\"rows longer than 512 tokens (truncated): \" + \", \".join(f\"{s} {v:.1f}%\" for s, v in st[\"pct_truncated\"].items())\n", " + \". Length is a known confound: legitimate mail is ~2x longer on average; the error analysis checks whether the model leans on it.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-15", "metadata": {}, "outputs": [], "source": [ "# The DataLoaders are built by src/finetune.py on the workstation (they need the parquet splits and torch). The bundle carries\n", "# the source, so the three definitions that make the torch.utils.data.DataLoader are shown from the file text: a seeded,\n", "# length-bucketed batch sampler, a row-index Dataset and the padding collate.\n", "ft_src = (ROOT / \"src/finetune.py\").read_text()\n", "def source_of(name, text):\n", " \"\"\"Top-level `def`/`class` of a source file read as text (no import needed).\"\"\"\n", " m = re.search(rf\"^(?:def|class) {name}\\b.*?(?=^\\S|\\Z)\", text, re.S | re.M)\n", " return m.group(0).rstrip() if m else f\"# {name}: not found\"\n", "for fn in (\"epoch_batches\", \"RowIndexDataset\", \"make_loader\"):\n", " print(source_of(fn, ft_src), \"\\n\")\n", "ft_cfg = CFG[\"finetune\"]\n", "MULTIPLE = int(ft_cfg[\"dynamic_padding_multiple\"]); MAX_LEN = int(CFG[\"model\"][\"max_seq_len\"])\n", "print(f\"dials: batch_size {ft_cfg['batch_size']}, eval_batch_size {ft_cfg['eval_batch_size']}, max_seq_len {MAX_LEN}, pad to a multiple of {MULTIPLE}; \"\n", " \"padded positions carry [PAD]=0 and mask False. 'unwanted' rows (TREC/CEAS positives) allow BOTH phishing and spam: \"\n", " \"the loss is -log(P(phishing)+P(spam)), the partial-label NLL.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-16", "metadata": {}, "outputs": [], "source": [ "# Pretraining data: documents are tokenised once, concatenated with [SEP] after each, and cut into packed windows of\n", "# [CLS] + 511 stream tokens (no padding, no attention mask -> the Flash SDPA kernel). Dynamic 15% masking per batch.\n", "pt = json.loads((ROOT / \"data/processed/pretrain_tokens.json\").read_text())\n", "print(f\"documents {pt['n_docs']:,} ({pt['n_docs_train']:,} train / {pt['n_docs_val']:,} val: {pt['val_rule']})\")\n", "print(f\"tokens {pt['n_tokens_train']:,} train + {pt['n_tokens_val']:,} val ({pt['tokens_per_char']} tokens/char over {pt['chars']:,} chars; \"\n", " f\"encoded in {pt['encode_seconds']} s, stored as {pt['dtype']})\")\n", "print(f\"packing {pt['packing']}\")\n", "summ = artifact(f\"checkpoints/pretrain/{PRETRAIN_RUN}/run_summary.json\", f\"src/pretrain.py train --run-name {PRETRAIN_RUN}\")\n", "if summ:\n", " d = json.loads(summ.read_text())[\"config\"]\n", " print(f\"windows {d['data']['n_windows']:,} x 512 per epoch -> {d['steps_per_epoch']} optimizer steps at effective batch \"\n", " f\"{d['effective_batch']} (micro-batch {d['micro_batch']} x {d['grad_accumulation']} accumulation); {d['data']['n_val_windows']:,} val windows\")" ] }, { "cell_type": "markdown", "id": "cell-17", "metadata": {}, "source": [ "**Step 3: Define the Neural Network**\n", "\n", "The reference notebook defines a small CNN; ours is `src/model.py`, a from-scratch **pre-LayerNorm BERT-style\n", "encoder** in plain PyTorch (no `transformers`, no pretrained weights): token + learned position embeddings, 24\n", "encoder layers (`x = x + Dropout(OutProj(SDPA(QKV(LN1(x)))))`, `x = x + Dropout(FFN_down(GELU(FFN_up(LN2(x)))))`),\n", "a final LayerNorm, and two heads. Pre-LN is chosen because a 24-layer stack trained from scratch tolerates the\n", "learning rate and warmup far better than post-LN. Attention runs through `F.scaled_dot_product_attention` so the fused\n", "Flash / memory-efficient kernels never materialise the 512x512 score matrix. The **MLM head** ties its decoder to the\n", "token embedding (saves 33.5M parameters); the **classification head** reads the `[CLS]` position and outputs three\n", "logits (legitimate / phishing / spam); rows whose corpus never separated spam from phishing train on the probability\n", "mass of {phishing, spam}. The course requires at least 2 layers; this model has 24." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-18", "metadata": {}, "outputs": [], "source": [ "# src/model.py from the bundle. With torch present (Colab preinstalls it) the module is imported and inspected exactly as the\n", "# local notebook does; without torch the same definitions are shown from the file text.\n", "sys.path.insert(0, str(ROOT / \"src\"))\n", "model_src = (ROOT / \"src/model.py\").read_text()\n", "if TORCH_OK:\n", " import inspect\n", " from model import EncoderConfig, EncoderLayer, EncoderForMLM, EncoderForClassification, count_parameters\n", " src_cfg = inspect.getsource(EncoderConfig).splitlines()\n", " layer_init, layer_fwd = inspect.getsource(EncoderLayer.__init__), inspect.getsource(EncoderLayer.forward)\n", "else:\n", " src_cfg = source_of(\"EncoderConfig\", model_src).splitlines()\n", " parts = re.split(r\"(?=^ def )\", source_of(\"EncoderLayer\", model_src), flags=re.M)\n", " layer_init = next(p for p in parts if p.startswith(\" def __init__\"))\n", " layer_fwd = next(p for p in parts if p.startswith(\" def forward\"))\n", "print(\"class EncoderConfig: # dataclass: the dials of the encoder\")\n", "print(\"\\n\".join(l for l in src_cfg if re.match(r\"^ \\w+: [\\w| ]+ = \", l)))\n", "print(\"\\nclass EncoderLayer(nn.Module): # one pre-LN block; 24 of these are stacked\")\n", "print(layer_init)\n", "print(layer_fwd)" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-19", "metadata": {}, "outputs": [], "source": [ "# Parameter counts: with torch, both wrappers are instantiated on this runtime's CPU (fp32, ~1.3 GB each) and counted, as in\n", "# the local notebook; without torch the counts recorded in the deployed model's config.json are shown.\n", "model_cfg_d = {**CFG[\"model\"], \"num_labels\": CFG[\"finetune\"][\"num_labels\"]}\n", "if TORCH_OK:\n", " model_cfg = EncoderConfig.from_dict(model_cfg_d)\n", " print(\"config:\", model_cfg.to_dict())\n", " mlm_model = EncoderForMLM(model_cfg)\n", " cls_model = EncoderForClassification.from_pretrained_encoder(mlm_model) # encoder weights copied, fresh 3-class head\n", " counts = pd.DataFrame({\"EncoderForMLM (tied head)\": count_parameters(mlm_model),\n", " \"EncoderForClassification (3 classes)\": count_parameters(cls_model)})\n", " assert mlm_model.head.decoder.weight is mlm_model.encoder.embeddings.tok.weight, \"MLM head must be tied\"\n", " del mlm_model, cls_model\n", " how = \"instantiated on this runtime's CPU and counted\"\n", "else:\n", " from types import SimpleNamespace\n", " model_cfg = SimpleNamespace(**model_cfg_d)\n", " print(\"config:\", model_cfg_d)\n", " counts = pd.DataFrame({\"EncoderForClassification (3 classes)\": EXPORT_CONFIG[\"finetune_checkpoint\"][\"params\"]})\n", " how = \"recorded in the deployed model's config.json; torch is not importable here\"\n", "show(counts, f\"Parameter counts ({how})\", fmt=\"{:,}\")\n", "print(f\"the classifier has {model_cfg.n_layers} encoder layers (course minimum: 2); head: CLS -> dropout -> Linear -> tanh -> dropout -> Linear({model_cfg.num_labels})\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-20", "metadata": {}, "outputs": [], "source": [ "# Why 24 layers / d=1024 / vocab 32,768: the parameter-count table from docs/research/research_model-math.md section 1\n", "# (configs A-F; the brief asked for ~330M parameters). Config A is the BERT-large shape and lands +2.3% over the target.\n", "math_md = (ROOT / \"docs/research/research_model-math.md\").read_text().splitlines()\n", "start = next(i for i, l in enumerate(math_md) if l.startswith(\"## 1.\"))\n", "end = next(i for i, l in enumerate(math_md) if l.startswith(\"## 2.\"))\n", "table = [l for l in math_md[start:end] if l.startswith(\"|\")]\n", "header = [h.strip() for h in table[0].strip(\"|\").split(\"|\")]\n", "body = [[c.strip().replace(\"**\", \"\") for c in l.strip(\"|\").split(\"|\")] for l in table[2:]]\n", "show(pd.DataFrame(body, columns=header).set_index(\"config\"), \"Candidate configurations (research_model-math.md section 1)\")\n", "print(f\"Chosen: A = {model_cfg.n_layers}L d{model_cfg.d_model} h{model_cfg.n_heads} f{model_cfg.d_ff} V{model_cfg.vocab_size} -> \"\n", " f\"{counts.loc['total'].iloc[-1]:,} parameters in the classifier, counted above.\")" ] }, { "cell_type": "markdown", "id": "cell-21", "metadata": {}, "source": [ "**Step 4: Train the Model**\n", "\n", "Training happens in two phases, both far too long for a notebook cell, so they run as scripts inside `tmux` on the\n", "RTX 3090 Ti and this notebook reads their metrics. **Phase 1, MLM pretraining** (`src/pretrain.py`): AdamW, peak LR\n", "1.5e-4, 6% warmup, cosine decay to 10%, effective batch 264 sequences x 512 tokens, bf16 autocast, 12 epochs over 107.7M\n", "tokens (~9,552 optimizer steps). **Phase 2, fine-tuning** (`src/finetune.py`): the instructor's dials are CLI flags\n", "(`--n-layers`, `--epochs`, `--loss cross_entropy|weighted_cross_entropy|focal`, `--lr`, `--freeze-layers`, `--synthetic`),\n", "AdamW with linear warmup/decay, **early stopping** on the validation F1 of the unwanted roll-up (patience 2), and every\n", "run appends one line to `reports/experiments.jsonl` so the experimentation table is generated from evidence. The six\n", "fine-tuning experiments (loss function x3, synthetic data, 12 layers, scratch init) are queued in\n", "`scripts/run_experiments.sh` (tmux `phish-finetune`), which waits for `final/` and then trains, evaluates and exports\n", "each run unattended (RUN_LOG entry 9)." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-22", "metadata": {}, "outputs": [], "source": [ "print(\"Phase 1 launch (scripts/launch_pretrain.sh; Nathan's settings on 2026-10-08):\")\n", "print(\" EPOCHS=12 MICRO_BATCH=24 scripts/launch_pretrain.sh # -> tmux session 'phish-pretrain'\")\n", "print(\" tmux attach -t phish-pretrain # Ctrl-b d detaches; Ctrl-C saves a checkpoint and exits\")\n", "print(\" RESUME=1 scripts/launch_pretrain.sh # continue from the latest checkpoint\")\n", "print(\"Phase 2 (after final/ exists), the dials are flags:\")\n", "print(\" .venv/bin/python src/finetune.py train --run-name ft-ce --loss cross_entropy --epochs 3 --lr 3e-5\")\n", "print(\" .venv/bin/python src/finetune.py train --run-name ft-focal --loss focal --focal-gamma 2.0\")\n", "print(\" .venv/bin/python src/finetune.py train --run-name ft-12L --n-layers 12 # fewer layers\")\n", "print(\" .venv/bin/python src/finetune.py list # experiments table\\n\")\n", "queue = ROOT / \"scripts/run_experiments.sh\"\n", "if queue.exists():\n", " lines = queue.read_text().splitlines()\n", " a = next(i for i, l in enumerate(lines) if l.startswith(\"EXPERIMENTS=(\"))\n", " b = next(i for i in range(a, len(lines)) if lines[i].strip() == \")\")\n", " print(\"Queued experiments (scripts/run_experiments.sh, tmux 'phish-finetune'; name | flags, all --init pretrained unless scratch):\")\n", " print(\"\\n\".join(lines[a:b + 1]), \"\\n\")\n", "print((ROOT / \"scripts/launch_pretrain.sh\").read_text())" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-23", "metadata": {}, "outputs": [], "source": [ "# Phase 1 live metrics: checkpoints/pretrain//metrics.csv grows while the run trains.\n", "mpath = artifact(f\"checkpoints/pretrain/{PRETRAIN_RUN}/metrics.csv\", f\"src/pretrain.py train --run-name {PRETRAIN_RUN}\")\n", "final_cfg = artifact(f\"checkpoints/pretrain/{PRETRAIN_RUN}/final/config.json\",\n", " f\"written automatically when the pretraining run completes (or: src/pretrain.py export --run-name {PRETRAIN_RUN})\")\n", "if mpath:\n", " # the trainer appends while we read: drop a partially written last row instead of failing on NaN\n", " pm = pd.read_csv(mpath).dropna(subset=[\"step\", \"loss\", \"loss_avg\", \"lr\", \"tokens_per_s\", \"train_seconds_total\"])\n", " last = pm.iloc[-1]\n", " plan = json.loads((ROOT / f\"checkpoints/pretrain/{PRETRAIN_RUN}/run_summary.json\").read_text())[\"config\"]\n", " status = \"COMPLETE\" if final_cfg else \"RUN IN PROGRESS\"\n", " print(f\"[{status}] step {int(last.step):,} / {plan['total_steps']:,} epoch {last.epoch_frac:.2f} / {plan['epochs']} \"\n", " f\"loss {last.loss:.4f} (avg {last.loss_avg:.4f}) lr {last.lr:.2e} {int(last.tokens_per_s):,} tokens/s \"\n", " f\"peak reserved {last.max_reserved_gb} GiB\")\n", " val = pm.dropna(subset=[\"val_mlm_loss\"])\n", " if len(val):\n", " v = val.iloc[-1]\n", " print(f\"last val MLM loss {v.val_mlm_loss:.4f} val masked-token accuracy {v.val_mlm_acc:.3f} at step {int(v.step)} (ln V = {np.log(CFG['model']['vocab_size']):.2f})\")\n", " print(f\"Total training time: {last.train_seconds_total:.2f} seconds{' so far' if not final_cfg else ''} \"\n", " f\"({last.train_seconds_total / 3600:.2f} h; projected {last.train_seconds_total / last.step * plan['total_steps'] / 3600:.1f} h for the full plan)\")\n", "\n", " fig, ax = plt.subplots(figsize=(9.5, 4))\n", " ax.plot(pm.step, pm.loss, color=\"#9ec3ee\", linewidth=0.9, label=\"train MLM loss (20-step)\")\n", " ax.plot(pm.step, pm.loss_avg, color=\"#2a78d6\", linewidth=1.8, label=\"train MLM loss (running avg)\")\n", " if len(val):\n", " ax.plot(val.step, val.val_mlm_loss, \"o-\", color=\"#eb6834\", label=\"val MLM loss (every 500 steps)\")\n", " ax.axhline(np.log(CFG[\"model\"][\"vocab_size\"]), color=\"grey\", linestyle=\":\", label=\"ln(vocab) = uniform guess\")\n", " ax.set_xlabel(\"optimizer step\"); ax.set_ylabel(\"MLM loss\"); ax.set_title(f\"Pretraining {PRETRAIN_RUN}: {status.lower()}\", loc=\"left\")\n", " ax2 = ax.twinx(); ax2.plot(pm.step, pm.lr, color=\"#1baf7a\", linewidth=1, label=\"learning rate\"); ax2.set_ylabel(\"lr\", color=\"#1baf7a\")\n", " h1, l1 = ax.get_legend_handles_labels(); h2, l2 = ax2.get_legend_handles_labels(); ax.legend(h1 + h2, l1 + l2, frameon=False, fontsize=8)\n", " for s in (\"top\",): ax.spines[s].set_visible(False); ax2.spines[s].set_visible(False)\n", " plt.tight_layout(); plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-24", "metadata": {}, "outputs": [], "source": [ "# reports/pretrain_curves.png is written by `pretrain.py plot --run-name `; show it only if it belongs to this run.\n", "png = artifact(\"reports/pretrain_curves.png\", f\"src/pretrain.py plot --run-name {PRETRAIN_RUN}\")\n", "if png and summ:\n", " import re as _re\n", " S = json.loads(summ.read_text())\n", " # run_summary.json records the finish time and the log path (pretrain-YYYYMMDD-HHMMSS.log); the launch time comes from the latter.\n", " started = S.get(\"started\")\n", " if not started and S.get(\"log\"):\n", " m = _re.search(r\"(\\d{4})(\\d{2})(\\d{2})-(\\d{2})(\\d{2})(\\d{2})\", Path(S[\"log\"]).name)\n", " if m:\n", " started = \"{}-{}-{}T{}:{}:{}\".format(*m.groups())\n", " started = started or S.get(\"finished\") or \"\"\n", " if time.strftime(\"%Y-%m-%dT%H:%M:%S\", time.localtime(png.stat().st_mtime)) >= started:\n", " display(Image(filename=str(png), width=900))\n", " else:\n", " print(f\"reports/pretrain_curves.png is from an earlier DEV self-test run (older than the {PRETRAIN_RUN} launch at {started}); \"\n", " f\"regenerate with: src/pretrain.py plot --run-name {PRETRAIN_RUN}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-25", "metadata": {}, "outputs": [], "source": [ "# Phase 2 experiments table (reports/experiments.jsonl, one JSON line per run). Dev self-test runs (tiny subsets, a\n", "# handful of steps, scratch init) are marked: they prove the code path, they are not results.\n", "def is_dev(rec):\n", " # A dev self-test trains on a row subset (--limit-rows) or is named dev*; a real run capped with --max-steps\n", " # (a legitimate time budget) is NOT a dev run.\n", " d = rec.get(\"dials\", {})\n", " return bool(rec.get(\"dev_run\") or d.get(\"limit_rows\") or rec[\"run_name\"].startswith(\"dev\"))\n", "\n", "exp_path = artifact(\"reports/experiments.jsonl\", \"src/finetune.py train ... (one line per run) and src/evaluate.py run\")\n", "EXPERIMENTS = [json.loads(l) for l in exp_path.read_text().splitlines() if l.strip()] if exp_path else []\n", "if exp_path and not EXPERIMENTS:\n", " print(\"reports/experiments.jsonl exists but is empty: the dev self-test lines were removed (RUN_LOG entry 9); \"\n", " \"the queued runs append to it (produced by: scripts/run_experiments.sh -> src/finetune.py train).\")\n", "if EXPERIMENTS:\n", " rows = []\n", " for r in EXPERIMENTS:\n", " if r.get(\"stage\") == \"evaluate\":\n", " continue\n", " d, v = r.get(\"dials\", {}), r.get(\"val\") or {}\n", " rows.append({\"run\": r[\"run_name\"], \"dev run?\": \"DEV\" if is_dev(r) else \"\", \"init\": r.get(\"init\"), \"classes\": d.get(\"num_labels\", 2),\n", " \"layers\": d.get(\"n_layers\"), \"frozen\": d.get(\"frozen_layers\"), \"epochs run/planned\": f\"{d.get('epochs_run')}/{d.get('epochs_planned')}\",\n", " \"loss\": d.get(\"loss\"), \"lr\": d.get(\"lr\"), \"batch\": d.get(\"batch_size\"), \"synthetic\": d.get(\"synthetic\"),\n", " \"train rows\": r.get(\"n_train\"), \"train min\": round(r.get(\"train_seconds\", 0) / 60, 1), \"peak GiB\": r.get(\"peak_vram_gb\"),\n", " \"val acc\": v.get(\"accuracy\"), \"val P\": v.get(\"precision\"), \"val R\": v.get(\"recall\"), \"val F1\": v.get(\"f1\"),\n", " \"val ROC-AUC\": v.get(\"roc_auc\"), \"val macro-F1 3w\": v.get(\"macro_f1_3way\"), \"stopped by\": r.get(\"stopped_by\")})\n", " EXP_DF = pd.DataFrame(rows).round(4)\n", " if len(EXP_DF):\n", " show(EXP_DF.set_index(\"run\"), \"Fine-tuning experiments (finetune.py list)\")\n", " real = [r for r in EXPERIMENTS if r.get(\"stage\") != \"evaluate\" and not is_dev(r)]\n", " if not real:\n", " print(\"Only DEV self-test runs exist so far: the real fine-tuning runs start once checkpoints/pretrain/large-24L/final/ exists \"\n", " \"(produced by: scripts/run_experiments.sh -> src/finetune.py train --run-name ...).\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-26", "metadata": {}, "outputs": [], "source": [ "# Fine-tuning curves of the best real run (highest val F1 of the unwanted roll-up); until one exists, the newest dev run is\n", "# shown and labelled as such. Early stopping: evaluation every 500 steps, stop after `patience` evaluations without a\n", "# better val F1; LR: linear warmup over 6% of the steps, then linear decay to 0 (BERT fine-tuning schedule).\n", "real = [r for r in EXPERIMENTS if r.get(\"stage\") != \"evaluate\" and not is_dev(r) and (r.get(\"val\") or {}).get(\"f1\") is not None]\n", "dev = [r for r in EXPERIMENTS if r.get(\"stage\") != \"evaluate\" and is_dev(r)]\n", "best = max(real, key=lambda r: r[\"val\"][\"f1\"]) if real else None\n", "tag = \"\"\n", "if best is None:\n", " artifact(\"checkpoints/finetune//metrics.csv\", \"src/finetune.py train --run-name (after pretraining completes)\")\n", " if dev:\n", " best, tag = dev[-1], \" (DEV self-test run, not a result)\"\n", "BEST_FT_RUN = best[\"run_name\"] if best else None\n", "if best:\n", " run_dir = ROOT / \"checkpoints/finetune\" / best[\"run_name\"]\n", " m = pd.read_csv(run_dir / \"metrics.csv\") if (run_dir / \"metrics.csv\").exists() else None\n", " e = pd.read_csv(run_dir / \"epochs.csv\") if (run_dir / \"epochs.csv\").exists() else None\n", " print(f\"run {best['run_name']}{tag}: init {best['init']}, loss {best['dials']['loss']}, lr {best['dials']['lr']}, \"\n", " f\"{best['steps_run']} steps, best step {best['best_step']}, val F1 {best['val']['f1']:.4f}, stopped by {best['stopped_by']}\")\n", " if m is not None and len(m):\n", " fig, axes = plt.subplots(1, 2, figsize=(11, 3.6))\n", " axes[0].plot(m.step, m.loss, color=\"#2a78d6\", label=\"train loss\"); axes[0].set_xlabel(\"step\"); axes[0].set_title(f\"{best['run_name']}{tag}: train loss\", loc=\"left\", fontsize=10)\n", " if e is not None and len(e):\n", " axes[1].plot(e.step, e.f1, \"o-\", color=\"#eb6834\", label=\"val F1 (unwanted roll-up)\")\n", " axes[1].plot(e.step, e.roc_auc, \"s--\", color=\"#1baf7a\", label=\"val ROC-AUC\")\n", " best_rows = e[e.is_best == 1]\n", " axes[1].scatter(best_rows.step, best_rows.f1, s=90, facecolors=\"none\", edgecolors=\"black\", label=\"best (checkpoint kept)\")\n", " axes[1].set_xlabel(\"step\"); axes[1].set_title(\"validation metrics at each evaluation\", loc=\"left\", fontsize=10); axes[1].legend(frameon=False, fontsize=8)\n", " for ax in axes:\n", " for s in (\"top\", \"right\"): ax.spines[s].set_visible(False)\n", " plt.tight_layout(); plt.show()\n", " # The queue writes reports/finetune_curves_.png per run and reports/finetune_curves.png for the main run; a PNG\n", " # older than the run's own experiments.jsonl line is a leftover from an earlier run and is not shown.\n", " curves = artifact(f\"reports/finetune_curves_{best['run_name']}.png\", f\"src/finetune.py plot --run-name {best['run_name']} --out reports/finetune_curves_{best['run_name']}.png\")\n", " curves = curves or (ROOT / \"reports/finetune_curves.png\" if (ROOT / \"reports/finetune_curves.png\").exists() else None)\n", " if curves and not tag:\n", " png_time = time.strftime(\"%Y-%m-%dT%H:%M:%S\", time.localtime(curves.stat().st_mtime))\n", " if png_time >= best.get(\"timestamp\", \"\"):\n", " display(Image(filename=str(curves), width=900))\n", " else:\n", " print(f\"{curves.relative_to(ROOT)} predates run {best['run_name']} ({png_time} < {best.get('timestamp')}): stale, not shown; \"\n", " f\"regenerate with src/finetune.py plot --run-name {best['run_name']}\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-27", "metadata": {}, "outputs": [], "source": [ "# Loss-function comparison (cross_entropy vs weighted_cross_entropy vs focal) over the REAL runs, same dials otherwise.\n", "real = [r for r in EXPERIMENTS if r.get(\"stage\") != \"evaluate\" and not is_dev(r) and r.get(\"val\")]\n", "if real:\n", " cmp = pd.DataFrame([{\"run\": r[\"run_name\"], \"loss\": r[\"dials\"][\"loss\"], \"focal_gamma\": r[\"dials\"].get(\"focal_gamma\"),\n", " \"class_weights\": r[\"dials\"].get(\"class_weights\"), \"val F1\": r[\"val\"][\"f1\"], \"val P\": r[\"val\"][\"precision\"],\n", " \"val R\": r[\"val\"][\"recall\"], \"val ROC-AUC\": r[\"val\"][\"roc_auc\"], \"val PR-AUC\": r[\"val\"][\"pr_auc\"],\n", " \"val macro-F1 3w\": r[\"val\"].get(\"macro_f1_3way\")} for r in real]).sort_values(\"val F1\", ascending=False)\n", " show(cmp.set_index(\"run\"), \"Loss-function comparison (real runs)\")\n", "else:\n", " artifact(\"reports/experiments.jsonl (real runs with --loss cross_entropy / weighted_cross_entropy / focal)\",\n", " \"scripts/run_experiments.sh -> src/finetune.py train --run-name ft-ce|ft-weighted|ft-focal --loss \")\n", " dev_losses = sorted({(r[\"dials\"].get(\"loss\"), r.get(\"steps_run\"), r.get(\"n_train\")) for r in EXPERIMENTS if r.get(\"stage\") != \"evaluate\" and is_dev(r)})\n", " if dev_losses:\n", " print(\"The DEV lines above (\" + \"; \".join(f\"{l}: {s} steps on {n:,} rows\" for l, s, n in dev_losses) + \") only prove the loss paths run.\")\n", " print(\"The three loss paths share one code path (src/finetune.py); focal(gamma=0) was verified to equal cross_entropy exactly (RUN_LOG entry 4).\")" ] }, { "cell_type": "markdown", "id": "cell-28", "metadata": {}, "source": [ "**Step 5: Test Inference Speed and Save the Model**\n", "\n", "The reference times `predict_and_time` on 100 images; the local notebook does the same on 100 TREC-06 emails on the\n", "workstation CPU. Here the model is not present, so the timing is taken where the model actually serves: each of the app's\n", "own synthetic example emails (read out of `app/app.py` in the bundle) is sent to the Space's `/predict` endpoint through\n", "`gradio_client`, and the Space answers with the in-app **Prediction Time (s)** (tokenize + forward + softmax on the ZeroGPU,\n", "the number the Gradio UI shows) next to the full round-trip from this runtime. The recorded workstation numbers\n", "(`reports/latency.json`: RTX 3090 Ti bf16 and CPU fp32 on 100 holdout emails) and the instructor's 0.074 s benchmark are\n", "shown for comparison. \"Saving the model\" happened on the workstation (`src/export.py` -> bf16 safetensors + config +\n", "tokenizer + model card, then `scripts/upload_to_hf.py`); this notebook lists the model repo's files with their sizes instead\n", "of writing a copy." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-29", "metadata": {}, "outputs": [], "source": [ "import ast\n", "\n", "def app_examples(path):\n", " \"\"\"The EXAMPLES list of app/app.py, read from the file TEXT: importing app.py would import torch and spaces at module level.\"\"\"\n", " for node in ast.parse(path.read_text()).body:\n", " targets = node.targets if isinstance(node, ast.Assign) else [node.target] if isinstance(node, ast.AnnAssign) else []\n", " if any(isinstance(t, ast.Name) and t.id == \"EXAMPLES\" for t in targets):\n", " return [[str(x[0]), str(x[1])] for x in ast.literal_eval(node.value)]\n", " raise ValueError(\"EXAMPLES not found in app.py\")\n", "\n", "FALLBACK_EMAILS = [ # three hard-coded synthetic emails (lure, spam, legitimate) if the extraction fails\n", " [\"Action Required: verify your account within 24 hours\",\n", " \"Dear customer,\\n\\nUnusual sign-in activity was detected. Verify your account within 24 hours at http://secure-login-verify.example \"\n", " \"or it will be suspended.\\n\\nSecurity Team\"],\n", " [\"FINAL HOURS: 70% OFF designer watches + FREE shipping!!!\",\n", " \"CONGRATULATIONS! You were SELECTED for our VIP SALE. 70% OFF everything, FREE shipping TODAY ONLY. Shop now: http://bit.ly/vip-deal\"],\n", " [\"Agenda for Thursday's project sync\",\n", " \"Hi all,\\n\\nAttaching the agenda for Thursday's 10:00 sync: firewall migration status, Q4 hardware budget, onboarding checklist. \"\n", " \"Reply by Wednesday noon with additions.\\n\\nThanks,\\nNathan\"],\n", "]\n", "app_py = artifact(\"app/app.py\", \"the Space source (app/ on the workstation, uploaded by scripts/upload_to_hf.py)\")\n", "try:\n", " SAMPLES = app_examples(app_py) if app_py else FALLBACK_EMAILS\n", " print(f\"{len(SAMPLES)} synthetic example emails extracted from app/app.py (EXAMPLES)\" if app_py else \"using the 3 built-in synthetic emails\")\n", "except Exception as e:\n", " SAMPLES = FALLBACK_EMAILS\n", " print(f\"could not extract EXAMPLES from app.py ({type(e).__name__}): using the 3 built-in synthetic emails\")\n", "for subject, body in SAMPLES:\n", " print(f\" - {subject[:72]!r} ({len(body)} chars)\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-30", "metadata": {}, "outputs": [], "source": [ "# Inference speed where the model runs: the Space's /predict endpoint, one call per email, through gradio_client.\n", "from gradio_client import Client\n", "\n", "SPACE_STATS = []\n", "client = None\n", "try:\n", " t0 = timer()\n", " client = Client(SPACE, token=TOKEN, verbose=False, analytics_enabled=False)\n", " print(f\"connected to {SPACE} in {timer() - t0:.1f} s (endpoint /predict: subject, body -> label, Prediction Time (s), highlights, signals)\")\n", "except Exception as e:\n", " print(f\"cannot connect to the Space ({type(e).__name__}: {str(e)[:160]}): private Space without a token, or the Space is not running\")\n", "if client is not None:\n", " for subject, body in SAMPLES:\n", " t0 = timer()\n", " try:\n", " label, secs, highlights, signals = client.predict(subject, body, api_name=\"/predict\")\n", " except Exception as e: # ZeroGPU quota / \"GPU task aborted\" / timeout: skip this email, keep the rest\n", " print(f\"\\n=== {subject}\\n Space call failed ({type(e).__name__}: {str(e)[:160]}); skipped\")\n", " continue\n", " round_trip = timer() - t0\n", " conf = {c[\"label\"]: float(c[\"confidence\"]) for c in (label.get(\"confidences\") or [])} if isinstance(label, dict) else {}\n", " verdict = label.get(\"label\") if isinstance(label, dict) else str(label)\n", " first_line = (str(signals).strip().splitlines() or [\"\"])[0]\n", " SPACE_STATS.append({\"subject\": subject[:52], \"verdict\": verdict, **{f\"P({k})\": v for k, v in conf.items()},\n", " \"Prediction Time (s), on the Space\": float(secs), \"round-trip (s)\": round_trip})\n", " print(f\"\\n=== {subject}\\n verdict {verdict}: \" + \", \".join(f\"{k} {100 * v:.1f}%\" for k, v in conf.items())\n", " + f\"\\n Prediction Time (s): {float(secs):.3f} on the Space round-trip from here: {round_trip:.2f} s {first_line}\")\n", "if SPACE_STATS:\n", " SP = pd.DataFrame(SPACE_STATS).set_index(\"subject\")\n", " show(SP, f\"{len(SP)} synthetic emails through the Space's /predict endpoint\", fmt={c: \"{:.3f}\" for c in SP.columns if c != \"verdict\"})\n", " t_app, t_rt = SP[\"Prediction Time (s), on the Space\"], SP[\"round-trip (s)\"]\n", " print(f\"Average in-app prediction time: {t_app.mean():.4f} s on the Space's GPU ({SPACE_HW}; the first call, {t_app.iloc[0]:.3f} s, \"\n", " f\"includes the ZeroGPU warm-up); average round-trip {t_rt.mean():.2f} s (queue + network + the app's saliency pass). \"\n", " f\"Instructor benchmark: 0.074 s.\")\n", "elif client is not None:\n", " print(\"no successful Space call: every request was rejected (ZeroGPU quota / Space restarting); the recorded workstation latencies follow\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-31", "metadata": {}, "outputs": [], "source": [ "# The recorded workstation latencies (reports/latency.json: evaluate.py on 100 TREC-06 holdout emails) next to the Space numbers.\n", "lat = artifact(\"reports/latency.json\", \"src/evaluate.py run --checkpoint checkpoints/finetune//best --split test,test3\")\n", "BENCH_S = 0.074\n", "rows = []\n", "if lat:\n", " L = json.loads(lat.read_text()); BENCH_S = L.get(\"instructor_benchmark_ms\", 74.0) / 1000\n", " for k, what in ((\"gpu_bf16\", \"RTX 3090 Ti bf16\"), (\"cpu_fp32\", \"CPU fp32, 16 threads\")):\n", " if L.get(k):\n", " rows.append({\"source\": f\"workstation {what} (evaluate.py, {L[k]['n']} holdout emails)\", \"avg s/sample\": L[k][\"avg_ms\"] / 1000,\n", " \"p50 s\": L[k].get(\"p50_ms\", np.nan) / 1000, \"n\": L[k][\"n\"]})\n", "if SPACE_STATS:\n", " rows.append({\"source\": f\"Space {SPACE_HW}, in-app Prediction Time (this notebook, synthetic emails)\", \"avg s/sample\": float(t_app.mean()),\n", " \"p50 s\": float(t_app.median()), \"n\": len(t_app)})\n", " rows.append({\"source\": \"Space, full round-trip from this runtime\", \"avg s/sample\": float(t_rt.mean()), \"p50 s\": float(t_rt.median()), \"n\": len(t_rt)})\n", "rows.append({\"source\": \"instructor benchmark\", \"avg s/sample\": BENCH_S, \"p50 s\": np.nan, \"n\": None})\n", "show(pd.DataFrame(rows).set_index(\"source\"), \"Single-email prediction time (tokenize + forward + softmax)\", fmt={\"avg s/sample\": \"{:.4f}\", \"p50 s\": \"{:.4f}\"})\n", "if lat:\n", " print(f\"workstation GPU {L['gpu_bf16']['avg_ms']:.1f} ms = {L['gpu_bf16']['ratio_to_benchmark']:.2f}x the {BENCH_S * 1000:.0f} ms benchmark; \"\n", " f\"CPU {L['cpu_fp32']['avg_ms']:.0f} ms = {L['cpu_fp32']['ratio_to_benchmark']:.1f}x: hence a ZeroGPU Space rather than a CPU one.\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-32", "metadata": {}, "outputs": [], "source": [ "# \"Save the model\": the export already lives on the Hub. The model repo's own files (what the Space and anyone with access load).\n", "info = api.repo_info(MODEL_REPO, repo_type=\"model\", files_metadata=True)\n", "own = [s for s in info.siblings if not s.rfilename.startswith(PRESENTATION_PREFIX + \"/\")]\n", "bundle = [s for s in info.siblings if s.rfilename.startswith(PRESENTATION_PREFIX + \"/\")]\n", "show(pd.DataFrame([{\"file\": s.rfilename, \"MiB\": (s.size or 0) / 2**20, \"LFS\": s.lfs is not None} for s in own]).set_index(\"file\"),\n", " f\"https://huggingface.co/{MODEL_REPO} (private: {info.private}, last modified {str(info.last_modified)[:16]})\", fmt={\"MiB\": \"{:,.1f}\"})\n", "print(f\"+ {len(bundle)} files / {sum((s.size or 0) for s in bundle) / 2**20:.1f} MiB under {PRESENTATION_PREFIX}/ (this notebook's inputs)\")\n", "fc = EXPORT_CONFIG[\"finetune_checkpoint\"]\n", "print(f\"model.safetensors: {EXPORT_CONFIG['export_dtype']} export of checkpoints/finetune/{Path(EXPORT_CONFIG['source_checkpoint']).parent.name}/best \"\n", " f\"({fc['params']['total']:,} parameters, {EXPORT_CONFIG['num_labels']} classes {fc['label_names']}), exported {EXPORT_CONFIG['exported']}; \"\n", " f\"config.json carries the test metrics: TREC-06 F1 {EXPORT_CONFIG['metrics']['f1']:.4f}, recall {EXPORT_CONFIG['metrics']['recall']:.4f}, \"\n", " f\"ROC-AUC {EXPORT_CONFIG['metrics']['roc_auc']:.4f}; three-way macro-F1 {EXPORT_CONFIG['metrics_threeway']['macro_f1']:.4f}.\")" ] }, { "cell_type": "markdown", "id": "cell-33", "metadata": {}, "source": [ "**Step 6: Create the App Files (Optional)**\n", "\n", "Optional in the reference, not here: the Space is one of the project's deliverables. The files the reference writes\n", "(`app.py`, `model.py`, `requirements.txt`) are the ones `scripts/upload_to_hf.py` pushed to the Space, together with the bf16\n", "weights, `config.json` and `tokenizer.json` (bundled, so the Space needs no download and no secret at startup). The Space's\n", "file list comes from the Hub; the first lines of `app.py`, the pinned `requirements.txt` and the Space card's front matter are\n", "read from the bundle's copy of `app/`." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-34", "metadata": {}, "outputs": [], "source": [ "try:\n", " space_files = api.list_repo_files(SPACE, repo_type=\"space\")\n", " print(f\"Space https://huggingface.co/spaces/{SPACE} files ({len(space_files)}): \" + \", \".join(space_files))\n", "except Exception as e:\n", " print(f\"Space file list not readable ({type(e).__name__}): private Space without a token\")\n", "print(\"\\n--- app/app.py (first 40 lines) ---\")\n", "if app_py:\n", " print(\"\\n\".join(app_py.read_text().splitlines()[:40]))\n", "req = artifact(\"app/requirements.txt\", \"app/ on the workstation (pinned for the ZeroGPU runtime)\")\n", "if req:\n", " print(\"\\n--- app/requirements.txt ---\"); print(req.read_text())\n", "card = artifact(\"app/README.md\", \"app/ on the workstation (the Space card)\")\n", "if card:\n", " print(\"--- app/README.md front matter (Space configuration) ---\"); print(card.read_text().split(\"---\")[1].strip())" ] }, { "cell_type": "markdown", "id": "cell-35", "metadata": {}, "source": [ "**Step 7: Launch the Gradio Interface**\n", "\n", "The reference calls `demo.launch(share=True)`. Here the interface *is* the deployed Space: `gr.load(SPACE, src=\"spaces\")`\n", "builds a Gradio app whose every call is proxied to the Space (the model never leaves ZeroGPU), and `launch()` renders it\n", "inline in Colab (Colab shows Gradio through a temporary public link). Gradio 6.29.1 has a bug in `gr.load` for Spaces whose\n", "`gr.Interface` carries an examples table: `Blocks.from_config` rebuilds the `Dataset` component from `{name, class_id}`\n", "dicts and hands `class_id` on to the child `Textbox.__init__`, which rejects it. The cell first calls `gr.load` as is; if\n", "it fails with exactly that `TypeError`, a four-line compatibility shim (drop `class_id` before the child component is\n", "instantiated, the same thing newer Gradio releases do) is installed and `gr.load` is retried, so the interface shown is the\n", "Space's own. Only if `gr.load` still fails does the cell fall back to a hand-built `gr.Interface` around the Step 5\n", "`gradio_client` call. Outside Colab the cell starts and immediately stops a headless server as a smoke test. When the Space\n", "is public, the native page is also embedded in an iframe; while it is private the iframe would only show a login page, so it\n", "is skipped with a note." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-36", "metadata": {}, "outputs": [], "source": [ "import io, contextlib, warnings\n", "import gradio as gr\n", "\n", "def shim_gr_load_dataset_bug():\n", " \"\"\"gradio 6.29.1: gr.load -> Blocks.from_config rebuilds a Space's examples `Dataset` from {name, class_id} dicts and\n", " get_component_instance passes `class_id` on to the child component's __init__, which rejects it. Drop that key.\"\"\"\n", " import gradio.components.dataset as ds\n", " if getattr(ds.get_component_instance, \"_class_id_shim\", False):\n", " return\n", " original = ds.get_component_instance\n", " def patched(comp, *args, **kwargs):\n", " if isinstance(comp, dict) and \"class_id\" in comp:\n", " comp = {k: v for k, v in comp.items() if k != \"class_id\"}\n", " return original(comp, *args, **kwargs)\n", " patched._class_id_shim = True\n", " ds.get_component_instance = patched\n", "\n", "def load_space():\n", " with warnings.catch_warnings(), contextlib.redirect_stdout(io.StringIO()): # hide gradio's chatter + theme UserWarning\n", " warnings.simplefilter(\"ignore\")\n", " return gr.load(SPACE, src=\"spaces\", token=TOKEN)\n", "\n", "demo, demo_kind = None, None\n", "try:\n", " try:\n", " demo = load_space()\n", " demo_kind = \"gr.load(src='spaces'): the Space's own interface, every call proxied\"\n", " except TypeError as e:\n", " if \"class_id\" not in str(e):\n", " raise\n", " print(f\"gr.load hit the gradio {gr.__version__} Dataset/class_id bug ({str(e)[:90]}); installing the compat shim and retrying\")\n", " shim_gr_load_dataset_bug()\n", " demo = load_space()\n", " demo_kind = \"gr.load(src='spaces') with the class_id compat shim: the Space's own interface, every call proxied\"\n", "except Exception as e:\n", " print(f\"gr.load({SPACE!r}) failed: {type(e).__name__}: {str(e)[:160]} -> hand-built proxy interface\")\n", "if demo is None and client is not None:\n", " # Fallback: the same interface built by hand around the gradio_client connection of Step 5 (labels mirror app/app.py).\n", " def proxy_predict(subject, body):\n", " label, secs, highlights, signals = client.predict(subject, body, api_name=\"/predict\")\n", " confidences = {c[\"label\"]: c[\"confidence\"] for c in label[\"confidences\"]} if isinstance(label, dict) and \"confidences\" in label else label\n", " spans = [(h[\"token\"], h.get(\"class_or_confidence\")) if isinstance(h, dict) else tuple(h) for h in highlights]\n", " return confidences, secs, spans, signals\n", " demo = gr.Interface(\n", " fn=proxy_predict,\n", " inputs=[gr.Textbox(label=\"Subject\", lines=1), gr.Textbox(label=\"Email body\", lines=14)],\n", " outputs=[gr.Label(label=\"Prediction\"), gr.Number(label=\"Prediction Time (s)\"),\n", " gr.HighlightedText(label=\"Tokens the model weighed most\", color_map={\"high\": \"#d62728\", \"medium\": \"#ff9896\"},\n", " combine_adjacent=True, show_label=True),\n", " gr.Markdown(label=\"Signals\")],\n", " examples=SAMPLES, cache_examples=False, title=\"BITS Phishing Email Detector (proxied to the Space)\",\n", " description=f\"Every prediction below runs on {SPACE} ({SPACE_HW}); this runtime only forwards the request.\")\n", " demo_kind = \"gr.Interface around gradio_client (gr.load fallback), every call proxied\"\n", "if demo is None:\n", " print(\"no interface: the Space is unreachable from this runtime (private without a token, or not running)\")\n", "else:\n", " print(\"interface:\", demo_kind)\n", "if demo is not None:\n", " if IN_COLAB:\n", " demo.launch() # inline in Colab; every prediction is proxied to the Space\n", " else:\n", " demo.launch(prevent_thread_lock=True, inline=False, share=False, quiet=True)\n", " print(f\"headless smoke test: the proxied interface was served at {demo.local_url} and is closed again\")\n", " demo.close()" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-37", "metadata": {}, "outputs": [], "source": [ "from IPython.display import IFrame\n", "\n", "try:\n", " space_public = api.space_info(SPACE).private is False\n", "except Exception:\n", " space_public = False\n", "if space_public:\n", " display(IFrame(SPACE_HOST, \"100%\", 900))\n", "else:\n", " print(f\"The iframe embed needs a public Space; {SPACE} is private right now. Flip it to public in the Hub settings before class \"\n", " f\"and the native page appears here: {SPACE_HOST}\")" ] }, { "cell_type": "markdown", "id": "cell-38", "metadata": {}, "source": [ "**Step 8: Authenticate with Hugging Face**\n", "\n", "The reference pastes a write token into `notebook_login()`. Here the token came from the Colab secret (or the environment /\n", "login cache) in Step 1 and every Hub call in this notebook passes it explicitly, so `login()` is only needed when the token is\n", "not yet the one `huggingface_hub` resolves on its own (in huggingface_hub 1.33.0 `login(token=...)` always validates with\n", "`whoami` and re-saves the token to the runtime's HF cache; `skip_if_logged_in` only governs the interactive flows). When the\n", "token already comes from the login cache or the Colab secret, nothing is written. The only thing printed is the account name\n", "the Hub answers with: the token value never appears in the notebook, its outputs or the repository." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-39", "metadata": {}, "outputs": [], "source": [ "from huggingface_hub import login\n", "\n", "if TOKEN and TOKEN != huggingface_hub.get_token():\n", " login(token=TOKEN, add_to_git_credential=False) # validates with whoami and saves the token in this runtime's HF cache (Colab: ephemeral disk)\n", " print(\"token saved for this runtime's huggingface_hub\")\n", "elif TOKEN:\n", " print(\"token already active for huggingface_hub (login cache / Colab secret): nothing written\")\n", "else:\n", " print(\"no token found (anonymous): the private repos stay unreadable until they are public\")\n", "try:\n", " print(\"Logged in to Hugging Face as:\", HfApi(token=TOKEN).whoami()[\"name\"])\n", "except Exception as e:\n", " print(f\"whoami failed ({type(e).__name__}): no token / offline\")\n", "print(\"The token is never stored in this notebook or in the repository.\")" ] }, { "cell_type": "markdown", "id": "cell-40", "metadata": {}, "source": [ "**Step 9: Upload the App to Hugging Face Spaces**\n", "\n", "The reference uploads a folder with `HfApi().upload_folder`. Ours is `scripts/upload_to_hf.py` (in the bundle under\n", "`scripts/`): it creates the private model repo and the private ZeroGPU Space named in `config.yaml`, uploads\n", "`model_export/ft-focal/` and `app/` (with the weights bundled into the Space, so no secret is needed) and prints both URLs.\n", "It is a dry run unless `--yes` is given, and the live run was done once from the workstation terminal (RUN_LOG entry 24),\n", "never from a notebook. This cell shows the command that was run, the two URLs, the Space's current runtime and the log entry." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-41", "metadata": {}, "outputs": [], "source": [ "hf = CFG[\"hf\"]\n", "print(\"Live upload (run once from the workstation terminal, 2026-10-12, RUN_LOG entry 24):\")\n", "print(\" .venv/bin/python scripts/upload_to_hf.py --yes --model-dir model_export/ft-focal --bundle-weights\\n\")\n", "print(f\"model repo : https://huggingface.co/{MODEL_REPO}\")\n", "print(f\"Space : https://huggingface.co/spaces/{SPACE} (sdk {hf['space_sdk']} {hf['space_sdk_version']}, hardware {hf['space_hardware']}, \"\n", " f\"python {hf['space_python_version']}, torch {hf['space_torch']})\")\n", "try:\n", " rt = api.get_space_runtime(SPACE)\n", " print(f\"Space runtime now: stage {rt.stage}, hardware {rt.hardware}\")\n", "except Exception as e:\n", " print(f\"Space runtime not readable ({type(e).__name__})\")\n", "log = artifact(\"RUN_LOG.md\", \"kept by hand on the workstation and mirrored into the bundle\")\n", "if log:\n", " lines = log.read_text().splitlines()\n", " start = next((i for i, l in enumerate(lines) if l.startswith(\"## 24.\")), None)\n", " if start is None:\n", " print(\"RUN_LOG entry 24 not found\")\n", " else:\n", " para = []\n", " for l in lines[start + 1:]: # the first paragraph = the first non-empty block after the header\n", " if l.strip():\n", " para.append(l)\n", " elif para:\n", " break\n", " display(Markdown(f\"> **{lines[start].lstrip('# ')}**\\n>\\n> \" + \"\\n> \".join(para)))" ] }, { "cell_type": "markdown", "id": "cell-42", "metadata": {}, "source": [ "## Results\n", "\n", "Performance on the **unseen** test sets, written by `src/evaluate.py` (the only script allowed to open the holdouts).\n", "The headline is the binary roll-up on TREC-06: P(unwanted) = P(phishing) + P(spam) at threshold 0.5, with recall on the\n", "unwanted class first (a missed phish is the costly error). The three-way holdout gives the 3x3 confusion matrix; the\n", "phishing-vs-spam boundary follows corpus provenance and is expected to be softer than legitimate-vs-unwanted." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-43", "metadata": {}, "outputs": [], "source": [ "def metric_tables(path, title):\n", " M = json.loads(path.read_text())\n", " at, tf = M.get(\"at_threshold\", {}), M.get(\"threshold_free\", {})\n", " print(f\"{title}: checkpoint {M.get('checkpoint')}, {M['split_sizes']['rows']:,} rows, positive rate {M['split_sizes']['positive_rate']}, threshold {M.get('threshold')}\")\n", " binary = {\"recall (unwanted)\": at.get(\"recall\"), \"precision\": at.get(\"precision\"), \"F1\": at.get(\"f1\"), \"accuracy\": at.get(\"accuracy\"),\n", " \"specificity\": at.get(\"specificity\"), \"ROC-AUC\": tf.get(\"roc_auc\"), \"PR-AUC\": tf.get(\"pr_auc\"),\n", " \"recall @ 1% FPR\": (tf.get(\"recall_at_fpr_1pct\") or {}).get(\"recall\"), \"recall @ 0.1% FPR\": (tf.get(\"recall_at_fpr_0_1pct\") or {}).get(\"recall\")}\n", " show(pd.DataFrame(binary, index=[\"value\"]).T, f\"{title}: binary roll-up\", fmt=\"{:.4f}\")\n", " cm = at.get(\"confusion_matrix\")\n", " if cm:\n", " show(pd.DataFrame([[cm[\"tn\"], cm[\"fp\"]], [cm[\"fn\"], cm[\"tp\"]]], index=[\"true legitimate\", \"true unwanted\"],\n", " columns=[\"pred legitimate\", \"pred unwanted\"]), \"Binary confusion matrix\")\n", " lure = (M.get(\"credential_lure_slice\") or {}).get(\"metrics\")\n", " if lure:\n", " show(pd.DataFrame({k: lure.get(k) for k in (\"n\", \"n_positive\", \"recall\", \"precision\", \"f1\", \"accuracy\")}, index=[\"credential-lure slice\"]).T, \"Credential-lure slice\")\n", " tw = M.get(\"threeway\")\n", " if tw and tw.get(\"confusion_matrix\"):\n", " labs = tw[\"confusion_matrix\"][\"labels\"]\n", " show(pd.DataFrame(tw[\"confusion_matrix\"][\"matrix\"], index=[f\"true {l}\" for l in labs], columns=[f\"pred {l}\" for l in labs]), \"3x3 confusion matrix\")\n", " show(pd.DataFrame(tw[\"per_class\"]).T, f\"Per-class metrics (macro-F1 {tw.get('macro_f1'):.4f})\" if tw.get(\"macro_f1\") is not None else \"Per-class metrics\")\n", " return M\n", "\n", "RESULTS = {}\n", "for rel, title in ((\"reports/metrics.json\", \"TREC-06 binary holdout\"), (\"reports/metrics_test3.json\", \"Three-way holdout\")):\n", " p = artifact(rel, \"src/evaluate.py run --checkpoint checkpoints/finetune/ft-focal/best --split test,test3 (main run ft-focal, selected on validation F1)\")\n", " if p:\n", " RESULTS[rel] = metric_tables(p, title)" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-44", "metadata": {}, "outputs": [], "source": [ "# Ablations on the SAME holdouts: run_experiments.sh evaluates every queued run into reports/eval_/ (one directory per\n", "# run, so no report overwrites another); the main run (ft-focal: best validation F1 of the unwanted roll-up and best validation\n", "# three-way macro-F1 among the six, chosen before looking at the test numbers) is also written to the top-level reports/ shown above.\n", "abl = []\n", "for d in sorted((ROOT / \"reports\").glob(\"eval_*\")):\n", " mj, m3 = d / \"metrics.json\", d / \"metrics_test3.json\"\n", " if mj.exists():\n", " M = json.loads(mj.read_text()); at, tf = M.get(\"at_threshold\", {}), M.get(\"threshold_free\", {})\n", " row = {\"run\": d.name[len(\"eval_\"):], \"TREC-06 recall\": at.get(\"recall\"), \"precision\": at.get(\"precision\"), \"F1\": at.get(\"f1\"),\n", " \"ROC-AUC\": tf.get(\"roc_auc\"), \"PR-AUC\": tf.get(\"pr_auc\"), \"recall @ 1% FPR\": (tf.get(\"recall_at_fpr_1pct\") or {}).get(\"recall\")}\n", " if m3.exists():\n", " tw = json.loads(m3.read_text()).get(\"threeway\") or {}\n", " row[\"3-way macro-F1\"] = tw.get(\"macro_f1\"); row[\"3-way accuracy\"] = tw.get(\"accuracy\")\n", " abl.append(row)\n", "if abl:\n", " show(pd.DataFrame(abl).set_index(\"run\").sort_values(\"F1\", ascending=False), \"Ablations on the unseen holdouts (reports/eval_/)\", fmt=\"{:.4f}\")\n", "else:\n", " artifact(\"reports/eval_/metrics.json\", \"scripts/run_experiments.sh -> src/evaluate.py run --out-dir reports/eval_ (one per queued run)\")" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-45", "metadata": {}, "outputs": [], "source": [ "for rel in (\"reports/confusion_matrix.png\", \"reports/roc_curve.png\", \"reports/pr_curve.png\", \"reports/confusion_matrix3_test3.png\",\n", " \"reports/confusion_matrix_test3.png\"):\n", " p = artifact(rel, \"src/evaluate.py run --split test,test3\")\n", " if p and p.stat().st_size < 2_000_000:\n", " display(Image(filename=str(p), width=520))\n", "for rel in (\"reports/error_analysis.md\", \"reports/error_analysis_test3.md\"):\n", " p = artifact(rel, \"src/evaluate.py run --split test,test3\")\n", " if p:\n", " display(Markdown(\"\\n\".join(p.read_text().splitlines()[:40]) + \"\\n\\n*(truncated; full file: \" + rel + \")*\"))" ] }, { "cell_type": "code", "execution_count": null, "id": "cell-46", "metadata": {}, "outputs": [], "source": [ "# Latency as recorded by evaluate.py (GPU bf16 and CPU fp32 on the same 100 holdout emails) next to the benchmark.\n", "if lat:\n", " L = json.loads(lat.read_text())\n", " show(pd.DataFrame({k: v for k, v in L.items() if isinstance(v, dict)}).T, f\"reports/latency.json (benchmark {L.get('instructor_benchmark_ms')} ms)\")\n", "else:\n", " SL = SMOKE.get(\"latency\", {})\n", " g, c = SL.get(\"gpu_bf16_512_no_mask\", {}).get(\"avg_ms\"), SL.get(\"cpu_fp32_512\", {}).get(\"avg_ms\")\n", " print(\"latency.json not written yet (produced by: scripts/run_experiments.sh -> src/evaluate.py run); the measured numbers so far are in the \"\n", " f\"Step 5 table (smoke test GPU {g} ms bf16, CPU {c} ms fp32 at 512 tokens).\")" ] }, { "cell_type": "markdown", "id": "cell-47", "metadata": {}, "source": [ "## Challenges and what we learned\n", "\n", "Distilled from `RUN_LOG.md` (entries 0-9), which records every decision, number and problem.\n", "\n", "- **Python 3.14 wheels.** The workstation runs Python 3.14.7; every pin (torch 2.14.1 cu130, tokenizers, safetensors, pyarrow, ...) had to be checked live for a cp314 wheel before the venv was built. The plain PyPI torch wheel *is* the CUDA 13.0 build, and driver 615 (CUDA 13.4) runs any 13.x build by backward compatibility; `uv --torch-backend=auto` would nevertheless have resolved an unverified cu132 build instead of the verified cu130 wheel (uv 0.12.5 iterates cu132 before cu130 for drivers >= 580), so the install pins `--torch-backend=cu130`.\n", "- **ZeroGPU pins the Space.** ZeroGPU supports only torch 2.8-2.13 on Python 3.12.12, no `torch.compile`, and cannot preload a private model repo, so the Space uses torch 2.13.0 and downloads the weights at startup with an `HF_TOKEN` secret. Training (torch 2.14.1 / 3.14) and serving (2.13.0 / 3.12) are different stacks; safetensors makes that safe.\n", "- **Datasets without body text.** `it4lia/...Cleaned` turned out to hold only subjects and 53 numeric features; `zefang-liu/...` was a lossy 92% copy of the primary set with an \"enron\" shortcut. Both were excluded before downloading anything.\n", "- **Hub-split leakage.** The primary set's own train/test split has 5.1% exact and ~40% near-duplicate test rows in train; we built our own split and asserted zero leakage with independent recomputation.\n", "- **Enron contamination.** 89% of TREC-05 ham and 86% of Enron-Spam ham are in the pretraining corpus, so neither could be the test set; TREC-06 (5 rows mention Enron) became the holdout, and 1,652 training rows that near-duplicate TREC-06 spam templates were removed.\n", "- **Near-duplicate decontamination.** Exact-key dedup missed quoted/forwarded 419-letter variants of holdout emails in the MLM corpus; the reviewer caught it, and 50-char shingle matching (>= 3 shared) now gates every split. 52.7% of `corbt/enron-emails` was exact duplicates.\n", "- **VRAM finder bug.** The first batch finder measured peak memory with one forward/backward and under-reported the real gradient-accumulation peak by one fp32 gradient copy (1.26 GiB); fixed before the long run. Checkpointing turned out unnecessary (B=16 fits with 8 GiB headroom) and costs ~25% throughput.\n", "- **Effective batch 264, not 256.** Micro-batch 24 does not divide 256; floor division would have reported 256 while training at 240. The trainer now uses ceil(256/24) = 11 accumulation steps and records the realised 264 everywhere, including the LR schedule.\n", "- **CPU latency reality.** The research estimate of ~0.34 s per 512-token email on the 5950X was 2x optimistic: measured 0.70 s (0.32 s at 128 tokens) because CPU inference is per-op-overhead bound. The 0.074 s benchmark is met on GPU (10.7 ms), hence ZeroGPU rather than a free CPU Space.\n", "- **The 300k three-class set was rejected.** Inspection showed it is a union of Kaggle re-packagings of corpora we already hold, 9.9% of it copies of our frozen TREC-06 test set, with a \"phish\" label that agrees with real provenance only 60% of the time and a stopword-stripping artifact that would be a near-perfect label proxy. Not one row was ingested.\n", "- **Three-way label scarcity.** No public corpus separates spam from phishing at scale: only 3,150 clean phishing training rows (Nazario, Nigerian fraud) plus 1,000 synthetic lures carry the distinction, while 62,805 TREC/CEAS positives are trained as \"unwanted\" with a partial-label loss. The binary roll-up on TREC-06 is the headline; the three-way split is the weaker decision and is reported as such.\n", "- **Reviewer-caught trainer bugs** (fixed before any real run): an `--lr` override silently discarded on resume but logged as applied; a resumed finished run training one extra step at lr 0; orphan metric rows after a hard crash; 0 occurrences of `[CLS]` in the packed MLM stream (every window now starts with `[CLS]`, so the classifier's pooled position is pretrained).\n", "- **DataLoader migration with an RNG side effect.** The course requires `torch.utils.data.DataLoader` for batching and shuffling; `finetune.py` had its own numpy sampler. Moving every batch through `DataLoader` (row-index Dataset + seeded length-bucketed `batch_sampler` + padding collate, Step 2) was verified loss-identical at every step on the CPU fixture, except that iterating a DataLoader draws its worker base seed from the global torch RNG and shifted the dropout stream (step-1 loss 1.1093 vs 1.1125). Passing an explicit `generator` removed the side effect.\n", "- **Self-test artifacts are not results.** The agents' dev checkpoints (scratch init, a handful of steps on 2,000 rows) and their `experiments.jsonl` lines were deleted before the real queue was armed, so nothing in `reports/` or `checkpoints/` can be mistaken for a result; this notebook labels anything named `dev*` and shows \"not available yet\" until the queue writes the real artifacts." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-48", "metadata": {}, "outputs": [], "source": [ "elapsed = timer() - T_NOTEBOOK_START\n", "print(f\"Notebook executed top-to-bottom in {elapsed:.0f} s (in Colab: {IN_COLAB}); model compute on the Space {SPACE}, \"\n", " f\"artifacts from {MODEL_REPO}/{PRESENTATION_PREFIX}.\")\n", "if MISSING:\n", " print(f\"\\n{len(MISSING)} artifact(s) not in the bundle:\")\n", " for rel, by in dict(MISSING).items():\n", " print(f\" - {rel}\\n produced by: {by}\")\n", "else:\n", " print(\"Every artifact of the bundle was present.\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }