{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stable Production — LR-TFIDF + 5-Fold CV\n",
    "\n",
    "Production settings from `configs/stable_training.yaml`:\n",
    "- **TF-IDF:** `max_features=800`, bigrams, `sublinear_tf`\n",
    "- **LR:** `C=0.05` with grid search until train–test gap < 5 pp\n",
    "- **Augmentation:** toxic-only back-translation (EN→ES→EN) + cosine dedup\n",
    "- **Evaluation:** stratified 5-fold CV on the train+val pool\n",
    "\n",
    "Run the full pipeline from repo root:\n",
    "```bash\n",
    "uv sync --extra hf --extra train\n",
    "uv run python -m src.pipeline.run_stable_pipeline\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded: stable_run_20260524_190417.json (run_id=20260524_190417)\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "import yaml\n",
    "\n",
    "PROJECT_ROOT = Path.cwd().resolve()\n",
    "if not (PROJECT_ROOT / \"configs\").exists() and (PROJECT_ROOT.parent / \"configs\").exists():\n",
    "    PROJECT_ROOT = PROJECT_ROOT.parent\n",
    "\n",
    "cfg = yaml.safe_load(open(PROJECT_ROOT / \"configs\" / \"stable_training.yaml\"))\n",
    "reports_dir = PROJECT_ROOT / \"reports\" / \"stable\"\n",
    "runs = sorted(reports_dir.glob(\"stable_run_*.json\"))\n",
    "assert runs, \"No stable_run_*.json — run the pipeline first\"\n",
    "latest = runs[-1]\n",
    "metrics = json.loads(latest.read_text())\n",
    "run_id = metrics[\"run_id\"]\n",
    "print(f\"Loaded: {latest.name} (run_id={run_id})\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Augmentation summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "enabled                          True\n",
       "strategy             back_translation\n",
       "train_size_before                 677\n",
       "train_size_after                  877\n",
       "added_samples                     200\n",
       "dtype: object"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "aug = metrics.get(\"augmentation\", {})\n",
    "pd.Series(aug)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. LR gap search (holdout test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>metric</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>F1 weighted (test)</td>\n",
       "      <td>0.6546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>F1 weighted (train, orig)</td>\n",
       "      <td>0.7721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Train–test gap (pp)</td>\n",
       "      <td>11.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ROC-AUC (test)</td>\n",
       "      <td>0.7312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Chosen C</td>\n",
       "      <td>0.005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>max_features</td>\n",
       "      <td>800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Gap OK (&lt;5pp)</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      metric   value\n",
       "0         F1 weighted (test)  0.6546\n",
       "1  F1 weighted (train, orig)  0.7721\n",
       "2        Train–test gap (pp)   11.74\n",
       "3             ROC-AUC (test)  0.7312\n",
       "4                   Chosen C   0.005\n",
       "5               max_features     800\n",
       "6              Gap OK (<5pp)   False"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr = metrics[\"logistic_regression\"]\n",
    "gap_search = metrics.get(\"lr_gap_search\", {})\n",
    "\n",
    "rows = [\n",
    "    {\"metric\": \"F1 weighted (test)\", \"value\": lr[\"f1_weighted\"]},\n",
    "    {\"metric\": \"F1 weighted (train, orig)\", \"value\": lr[\"f1_train\"]},\n",
    "    {\"metric\": \"Train–test gap (pp)\", \"value\": lr[\"train_test_gap_pp\"]},\n",
    "    {\"metric\": \"ROC-AUC (test)\", \"value\": lr[\"roc_auc\"]},\n",
    "    {\"metric\": \"Chosen C\", \"value\": lr.get(\"C\", gap_search.get(\"C\"))},\n",
    "    {\"metric\": \"max_features\", \"value\": lr.get(\"max_features\", gap_search.get(\"max_features\"))},\n",
    "    {\"metric\": \"Gap OK (<5pp)\", \"value\": lr.get(\"gap_ok\", gap_search.get(\"gap_ok\"))},\n",
    "]\n",
    "display(pd.DataFrame(rows))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Stratified 5-fold CV (LR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1: 0.6636 ± 0.0223  |  fold gap max: 14.66 pp  |  stable: False\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fold</th>\n",
       "      <th>f1_weighted</th>\n",
       "      <th>train_val_gap_pp</th>\n",
       "      <th>roc_auc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.680685</td>\n",
       "      <td>11.05</td>\n",
       "      <td>0.7239</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.650439</td>\n",
       "      <td>12.59</td>\n",
       "      <td>0.7341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0.697292</td>\n",
       "      <td>7.12</td>\n",
       "      <td>0.7277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0.653930</td>\n",
       "      <td>13.11</td>\n",
       "      <td>0.6728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0.635539</td>\n",
       "      <td>14.66</td>\n",
       "      <td>0.6856</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fold  f1_weighted  train_val_gap_pp  roc_auc\n",
       "0     0     0.680685             11.05   0.7239\n",
       "1     1     0.650439             12.59   0.7341\n",
       "2     2     0.697292              7.12   0.7277\n",
       "3     3     0.653930             13.11   0.6728\n",
       "4     4     0.635539             14.66   0.6856"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv = metrics[\"cv_logistic_regression\"]\n",
    "print(\n",
    "    f\"F1: {cv['f1_mean']} ± {cv['f1_std']}  |  \"\n",
    "    f\"fold gap max: {cv['gap_max']*100:.2f} pp  |  \"\n",
    "    f\"stable: {cv['stable_across_folds']}\"\n",
    ")\n",
    "fold_df = pd.DataFrame(cv[\"folds\"])\n",
    "fold_df[[\"fold\", \"f1_weighted\", \"train_val_gap_pp\", \"roc_auc\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This notebook summarizes **LR-TFIDF** from the latest stable production run.\n",
    "Check `reports/stable/integrated_report_{run_id}.md` for the combined LR + DistilBERT + ensemble report.\n",
    "The 5-fold CV **F1 std** measures stability across data segments; the **train–val gap** per fold tracks overfitting within each split.\n",
    "Target rubric: |train − test| < 5 pp and test F1 > 0.80 — tune `logistic_regression.gap_search.param_grid` if gaps remain high."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}