{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Notebook 13 — Hyper-Optimization Sprints (Break 0.80 F1)\n",
        "\n",
        "Foundation: **Frozen Golden Baseline** (`unitary/toxic-bert`, 6-label sigmoid `toxic` score).\n",
        "\n",
        "**Objective:** Test F1 weighted **> 0.80** with train–test gap **< 5%** (briefing rule).\n",
        "\n",
        "| Exp | Method | 5-Fold CV |\n",
        "|-----|--------|----------|\n",
        "| **1** | Multi-pivot aug (DE/FR/ES) + head-only train | ✅ |\n",
        "| **2** | Advanced TTA (Original + DE + FR weighted) | ✅ |\n",
        "| **3** | CLS hidden states + style meta → LR C=0.01 | ✅ |\n",
        "| **4** | Ultra-fine threshold (0.05–0.30, step 0.001) on best of 1–3 | ✅ |\n",
        "\n",
        "Artifacts: `models/notebook_13/` · Reports: `reports/notebook_13/sprint_results.json`\n",
        "\n",
        "```bash\n",
        "uv run python -m src.experiments.notebook_13_sprints\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 0. Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import json\n",
        "import sys\n",
        "from pathlib import Path\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "PROJECT_ROOT = Path.cwd().resolve()\n",
        "if not (PROJECT_ROOT / \"configs\").exists() and (PROJECT_ROOT.parent / \"configs\").exists():\n",
        "    PROJECT_ROOT = PROJECT_ROOT.parent\n",
        "if str(PROJECT_ROOT) not in sys.path:\n",
        "    sys.path.insert(0, str(PROJECT_ROOT))\n",
        "\n",
        "ARTIFACT_DIR = PROJECT_ROOT / \"models\" / \"notebook_13\"\n",
        "REPORT_DIR = PROJECT_ROOT / \"reports\" / \"notebook_13\"\n",
        "RESULTS_PATH = REPORT_DIR / \"sprint_results.json\"\n",
        "print(ARTIFACT_DIR)\n",
        "print(RESULTS_PATH)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 1. Run all sprints (long-running — translation + CV)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from src.experiments.notebook_13_sprints import main\n",
        "\n",
        "main()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2. Load results (if already executed)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "if not RESULTS_PATH.exists():\n",
        "    raise FileNotFoundError(f\"Run sprints first: uv run python -m src.experiments.notebook_13_sprints\")\n",
        "\n",
        "results = json.loads(RESULTS_PATH.read_text())\n",
        "comparison = pd.DataFrame(results[\"comparison_table\"])\n",
        "comparison"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3. Per-fold gap monitor"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "rows = []\n",
        "for key in (\"golden_baseline_cv\", \"exp1\", \"exp2\", \"exp3\", \"exp4\"):\n",
        "    block = results.get(key, {})\n",
        "    for f in block.get(\"folds\", []):\n",
        "        rows.append({\n",
        "            \"experiment\": key,\n",
        "            \"fold\": f[\"fold\"],\n",
        "            \"f1_test\": f[\"f1_test\"],\n",
        "            \"gap_pp\": f[\"train_test_gap_pp\"],\n",
        "            \"gap_ok\": f[\"gap_ok\"],\n",
        "            \"status\": \"PASS\" if f[\"gap_ok\"] else \"FAIL_GAP\",\n",
        "        })\n",
        "pd.DataFrame(rows).pivot_table(\n",
        "    index=\"experiment\", values=[\"f1_test\", \"gap_pp\"], aggfunc=[\"mean\", \"max\"]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 4. Comparison markdown report"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from IPython.display import Markdown, display\n",
        "\n",
        "md = REPORT_DIR / \"comparison_table.md\"\n",
        "if md.exists():\n",
        "    display(Markdown(md.read_text()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Conclusion\n",
        "\n",
        "**Sprint results:** `reports/notebook_13/sprint_results.json`\n",
        "\n",
        "| Sprint | Mean F1 (test) | Max gap (pp) | All folds gap OK | Mean F1 ≥ 0.80 |\n",
        "|--------|----------------|--------------|------------------|----------------|\n",
        "| Golden Baseline (CV) | 0.7748 | 8.09 | ❌ | ❌ |\n",
        "| Exp1 Multi-Pivot + Head | 0.7493 | 12.42 | ❌ | ❌ |\n",
        "| Exp2 Advanced TTA | 0.7592 | 6.53 | ❌ | ❌ |\n",
        "| Exp3 Meta Stacking | **0.7894** | 9.77 | ❌ | ❌ |\n",
        "| Exp4 Ultra-Fine Thresh | 0.7704 | 9.42 | ❌ | ❌ |\n",
        "\n",
        "**Which sprint reached 0.80?** No sprint passed **both** constraints on all 5 folds. Best single folds: **Exp3 fold 0** (F1=0.8147, gap=3.39 pp) and **Exp4 fold 4** (F1=0.8083, gap=0.18 pp, threshold≈0.299).\n",
        "\n",
        "**Final train–test gap:** Best average gap discipline: Golden Baseline / Exp2 TTA (~3.3–3.6 pp mean). Exp3 has highest mean F1 but **FAIL_GAP** (6.94 pp mean).\n",
        "\n",
        "**Production recommendation:** **Frozen Golden Baseline** for briefing compliance (~0.77–0.79 CV F1, minimal overfit). Exp3+Exp4 threshold tuning is promising on individual folds but not stable across CV.\n",
        "\n",
        "Artifacts: `models/notebook_13/` (augment cache, head-only checkpoints)."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.12.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}