File size: 6,358 Bytes
46cc63a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 | {
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Notebook 13 — Hyper-Optimization Sprints (Break 0.80 F1)\n",
"\n",
"Foundation: **Frozen Golden Baseline** (`unitary/toxic-bert`, 6-label sigmoid `toxic` score).\n",
"\n",
"**Objective:** Test F1 weighted **> 0.80** with train–test gap **< 5%** (briefing rule).\n",
"\n",
"| Exp | Method | 5-Fold CV |\n",
"|-----|--------|----------|\n",
"| **1** | Multi-pivot aug (DE/FR/ES) + head-only train | ✅ |\n",
"| **2** | Advanced TTA (Original + DE + FR weighted) | ✅ |\n",
"| **3** | CLS hidden states + style meta → LR C=0.01 | ✅ |\n",
"| **4** | Ultra-fine threshold (0.05–0.30, step 0.001) on best of 1–3 | ✅ |\n",
"\n",
"Artifacts: `models/notebook_13/` · Reports: `reports/notebook_13/sprint_results.json`\n",
"\n",
"```bash\n",
"uv run python -m src.experiments.notebook_13_sprints\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Setup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import sys\n",
"from pathlib import Path\n",
"\n",
"import pandas as pd\n",
"\n",
"PROJECT_ROOT = Path.cwd().resolve()\n",
"if not (PROJECT_ROOT / \"configs\").exists() and (PROJECT_ROOT.parent / \"configs\").exists():\n",
" PROJECT_ROOT = PROJECT_ROOT.parent\n",
"if str(PROJECT_ROOT) not in sys.path:\n",
" sys.path.insert(0, str(PROJECT_ROOT))\n",
"\n",
"ARTIFACT_DIR = PROJECT_ROOT / \"models\" / \"notebook_13\"\n",
"REPORT_DIR = PROJECT_ROOT / \"reports\" / \"notebook_13\"\n",
"RESULTS_PATH = REPORT_DIR / \"sprint_results.json\"\n",
"print(ARTIFACT_DIR)\n",
"print(RESULTS_PATH)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Run all sprints (long-running — translation + CV)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from src.experiments.notebook_13_sprints import main\n",
"\n",
"main()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Load results (if already executed)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if not RESULTS_PATH.exists():\n",
" raise FileNotFoundError(f\"Run sprints first: uv run python -m src.experiments.notebook_13_sprints\")\n",
"\n",
"results = json.loads(RESULTS_PATH.read_text())\n",
"comparison = pd.DataFrame(results[\"comparison_table\"])\n",
"comparison"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Per-fold gap monitor"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"rows = []\n",
"for key in (\"golden_baseline_cv\", \"exp1\", \"exp2\", \"exp3\", \"exp4\"):\n",
" block = results.get(key, {})\n",
" for f in block.get(\"folds\", []):\n",
" rows.append({\n",
" \"experiment\": key,\n",
" \"fold\": f[\"fold\"],\n",
" \"f1_test\": f[\"f1_test\"],\n",
" \"gap_pp\": f[\"train_test_gap_pp\"],\n",
" \"gap_ok\": f[\"gap_ok\"],\n",
" \"status\": \"PASS\" if f[\"gap_ok\"] else \"FAIL_GAP\",\n",
" })\n",
"pd.DataFrame(rows).pivot_table(\n",
" index=\"experiment\", values=[\"f1_test\", \"gap_pp\"], aggfunc=[\"mean\", \"max\"]\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Comparison markdown report"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Markdown, display\n",
"\n",
"md = REPORT_DIR / \"comparison_table.md\"\n",
"if md.exists():\n",
" display(Markdown(md.read_text()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"**Sprint results:** `reports/notebook_13/sprint_results.json`\n",
"\n",
"| Sprint | Mean F1 (test) | Max gap (pp) | All folds gap OK | Mean F1 ≥ 0.80 |\n",
"|--------|----------------|--------------|------------------|----------------|\n",
"| Golden Baseline (CV) | 0.7748 | 8.09 | ❌ | ❌ |\n",
"| Exp1 Multi-Pivot + Head | 0.7493 | 12.42 | ❌ | ❌ |\n",
"| Exp2 Advanced TTA | 0.7592 | 6.53 | ❌ | ❌ |\n",
"| Exp3 Meta Stacking | **0.7894** | 9.77 | ❌ | ❌ |\n",
"| Exp4 Ultra-Fine Thresh | 0.7704 | 9.42 | ❌ | ❌ |\n",
"\n",
"**Which sprint reached 0.80?** No sprint passed **both** constraints on all 5 folds. Best single folds: **Exp3 fold 0** (F1=0.8147, gap=3.39 pp) and **Exp4 fold 4** (F1=0.8083, gap=0.18 pp, threshold≈0.299).\n",
"\n",
"**Final train–test gap:** Best average gap discipline: Golden Baseline / Exp2 TTA (~3.3–3.6 pp mean). Exp3 has highest mean F1 but **FAIL_GAP** (6.94 pp mean).\n",
"\n",
"**Production recommendation:** **Frozen Golden Baseline** for briefing compliance (~0.77–0.79 CV F1, minimal overfit). Exp3+Exp4 threshold tuning is promising on individual folds but not stable across CV.\n",
"\n",
"Artifacts: `models/notebook_13/` (augment cache, head-only checkpoints)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.12.0"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
|