{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Phase 5: Expert Aggressive — Toxic-BERT + Hybrid\n", "\n", "**Expert Adaptation** strategy to break the F1 plateau:\n", "\n", "| Change | Setting |\n", "|--------|--------|\n", "| Base model | `unitary/toxic-bert` (head-only fine-tune) |\n", "| LR bottleneck | TF-IDF `max_features=250` |\n", "| Threshold | Val-set search maximizing **F1-toxic** |\n", "| Hybrid weights | **0.7** Toxic-BERT + **0.3** LR |\n", "| Augmentation | EN→**DE**→EN back-translation (higher diversity) |\n", "\n", "Run from repo root (long-running — augmentation + fine-tune):\n", "\n", "```bash\n", "uv sync --extra hf --extra train\n", "uv run python -m src.pipeline.run_expert_pipeline\n", "```\n", "\n", "Or execute the pipeline cell below inside this notebook." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 0. Setup" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Config: expert_training.yaml\n", "Pivot lang: de\n", "Model: unitary/toxic-bert (head_only)\n" ] } ], "source": [ "import json\n", "import sys\n", "from pathlib import Path\n", "\n", "import pandas as pd\n", "import yaml\n", "\n", "PROJECT_ROOT = Path.cwd().resolve()\n", "if not (PROJECT_ROOT / \"configs\").exists() and (PROJECT_ROOT.parent / \"configs\").exists():\n", " PROJECT_ROOT = PROJECT_ROOT.parent\n", "\n", "if str(PROJECT_ROOT) not in sys.path:\n", " sys.path.insert(0, str(PROJECT_ROOT))\n", "\n", "cfg_path = PROJECT_ROOT / \"configs\" / \"expert_training.yaml\"\n", "cfg = yaml.safe_load(open(cfg_path))\n", "reports_dir = PROJECT_ROOT / \"reports\" / \"expert\"\n", "print(f\"Config: {cfg_path.name}\")\n", "print(f\"Pivot lang: {cfg['augmentation']['pivot_lang']}\")\n", "print(f\"Model: {cfg['transformer']['model_id']} ({cfg['transformer']['freeze_mode']})\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Run Phase 5 pipeline" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:39:47 | INFO | src.pipeline.run_expert_pipeline | ============================================================\n", "2026-05-24 19:39:47 | INFO | src.pipeline.run_expert_pipeline | EXPERT PIPELINE (Phase 5) — run=20260524_193947\n", "2026-05-24 19:39:47 | INFO | src.pipeline.run_expert_pipeline | ============================================================\n", "2026-05-24 19:39:47 | INFO | src.data.loader | Cargando dataset: /Users/miraekang/proyectos/ai-nlp/data/raw/youtoxic_english_1000.csv\n", "2026-05-24 19:39:47 | INFO | src.data.loader | Shape: (1000, 15)\n", "2026-05-24 19:39:47 | INFO | src.data.loader | Columnas validadas ✅\n", "2026-05-24 19:39:47 | WARNING | src.data.loader | 3 duplicados eliminados\n", "2026-05-24 19:39:47 | INFO | src.data.loader | Toxicos: 459 (46.0%)\n", "2026-05-24 19:39:47 | INFO | src.pipeline.run_expert_pipeline | Augmentation EN→DE→EN (toxic only)\n", "2026-05-24 19:39:47 | INFO | src.features.augmentation | Back-translation: 312 toxic samples\n", "2026-05-24 19:42:02 | INFO | src.features.augmentation | Back-translation produced 295 samples\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.\n", "Loading weights: 100%|██████████| 103/103 [00:00<00:00, 9092.34it/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:42:08 | INFO | src.features.augmentation | Dedup: kept 209/295 (dropped 86 with cosine > 0.95)\n", "2026-05-24 19:42:08 | INFO | src.features.augmentation | Train size after augmentation: 886 (+209)\n", "2026-05-24 19:42:08 | INFO | src.pipeline.run_expert_pipeline | LR-TFIDF (max_features=250) + gap search\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | Training stable LR — C=0.05\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | LR gap search — C=0.05 max_features=250 min_df=3 train_f1=0.7703 test_f1=0.6563 gap=0.1139\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | Training stable LR — C=0.03\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | LR gap search — C=0.03 max_features=250 min_df=5 train_f1=0.7572 test_f1=0.6563 gap=0.1008\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | Training stable LR — C=0.02\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | LR gap search — C=0.02 max_features=250 min_df=5 train_f1=0.7572 test_f1=0.6563 gap=0.1008\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | Training stable LR — C=0.01\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | LR gap search — C=0.01 max_features=250 min_df=8 train_f1=0.7570 test_f1=0.6563 gap=0.1006\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | Training stable LR — C=0.005\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | LR gap search — C=0.005 max_features=250 min_df=10 train_f1=0.7511 test_f1=0.6509 gap=0.1003\n", "2026-05-24 19:42:08 | WARNING | src.models.hybrid_ensemble | LR gap still 0.1003 after grid search; using best gap C=0.005\n", "2026-05-24 19:42:08 | INFO | src.models.hybrid_ensemble | Stable LR saved: /Users/miraekang/proyectos/ai-nlp/models/expert_lr_tfidf.joblib\n", "2026-05-24 19:42:08 | INFO | src.pipeline.run_expert_pipeline | Toxic-BERT — head-only fine-tune + val threshold tuning\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Map: 100%|██████████| 886/886 [00:00<00:00, 22618.23 examples/s]\n", "Map: 100%|██████████| 120/120 [00:00<00:00, 17820.93 examples/s]\n", "Map: 100%|██████████| 200/200 [00:00<00:00, 23323.72 examples/s]\n", "[transformers] You passed `num_labels=2` which is incompatible to the `id2label` map of length `6`.\n", "Loading weights: 100%|██████████| 201/201 [00:00<00:00, 8948.39it/s]\n", "[transformers] \u001b[1mBertForSequenceClassification LOAD REPORT\u001b[0m from: unitary/toxic-bert\n", "Key | Status | \n", "------------------+----------+---------------------------------------------------------------------------------------\n", "classifier.weight | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([6, 768]) vs model:torch.Size([2, 768])\n", "classifier.bias | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([6]) vs model:torch.Size([2]) \n", "\n", "Notes:\n", "- MISMATCH:\tckpt weights were loaded, but they did not match the original empty weight shapes.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:42:09 | INFO | src.models.transformer_trainer | Head-only freeze — trainable 592,130/109,483,778 (0.54%)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "[transformers] warmup_ratio is deprecated and will be removed in v5.2. Use `warmup_steps` instead.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:42:10 | INFO | src.models.transformer_trainer | Training unitary/toxic-bert (head_only freeze)...\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "data": { "text/html": [ "\n", "
\n", " \n", " \n", " [ 444/1110 01:19 < 01:59, 5.56 it/s, Epoch 4/10]\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
EpochTraining LossValidation LossF1 ToxicF1 WeightedPrecisionRecallRoc Auc
10.5547570.5595790.6909090.7166670.6909090.6909090.814545
20.5072700.5609500.6909090.7166670.6909090.6909090.810350
30.5582990.5584040.6730770.7147440.7142860.6363640.812028
40.5036840.5644210.6851850.7161900.6981130.6727270.812308

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:42:31 | INFO | src.models.transformer_trainer | Gap monitor — train_f1=0.7925 val_f1=0.6909 gap=0.1016\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 3.64it/s]\n", "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n", "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:42:49 | INFO | src.models.transformer_trainer | Gap monitor — train_f1=0.7949 val_f1=0.6909 gap=0.1040\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 3.87it/s]\n", "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n", "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:43:09 | INFO | src.models.transformer_trainer | Gap monitor — train_f1=0.7952 val_f1=0.6731 gap=0.1221\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.57it/s]\n", "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n", "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:43:29 | INFO | src.models.transformer_trainer | Gap monitor — train_f1=0.7965 val_f1=0.6852 gap=0.1113\n", "2026-05-24 19:43:29 | INFO | src.models.transformer_trainer | Early stop: no f1_toxic improvement for 3 epochs\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 3.03it/s]\n", "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:43:31 | INFO | src.models.transformer_trainer | Val threshold tuning — best_t=0.33 val_f1_toxic=0.7313\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Writing model shards: 100%|██████████| 1/1 [00:00<00:00, 2.89it/s]\n", "Map: 100%|██████████| 886/886 [00:00<00:00, 27925.88 examples/s]\n", "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:752: UserWarning: 'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.\n", " super().__init__(loader)\n" ] }, { "data": { "text/html": [], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/miraekang/proyectos/ai-nlp/.venv/lib/python3.12/site-packages/sklearn/metrics/_ranking.py:442: UndefinedMetricWarning: Only one class is present in y_true. ROC AUC score is not defined in that case.\n", " warnings.warn(\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "2026-05-24 19:43:51 | INFO | src.pipeline.run_expert_pipeline | Expert report: /Users/miraekang/proyectos/ai-nlp/reports/expert/integrated_report_20260524_193947.md\n", "2026-05-24 19:43:51 | INFO | src.pipeline.run_expert_pipeline | ============================================================\n", "2026-05-24 19:43:51 | INFO | src.pipeline.run_expert_pipeline | Toxic-BERT-expert: F1-toxic=0.7489 ⚠️ | toxic gap=0.0418 ✅ | threshold=0.33\n", "2026-05-24 19:43:51 | INFO | src.pipeline.run_expert_pipeline | LR-TFIDF-expert: F1-toxic=0.6301 ⚠️ | toxic gap=0.0008 ✅ | threshold=0.05\n", "2026-05-24 19:43:51 | INFO | src.pipeline.run_expert_pipeline | Hybrid-ToxicBERT+LR: F1-toxic=0.7489 ⚠️ | toxic gap=0.0428 ✅ | threshold=0.38\n", "2026-05-24 19:43:51 | INFO | src.pipeline.run_expert_pipeline | ============================================================\n", "Completed run_id=20260524_193947\n" ] } ], "source": [ "from src.pipeline.run_expert_pipeline import run_expert_pipeline\n", "\n", "metrics = run_expert_pipeline(config_path=cfg_path)\n", "run_id = metrics[\"run_id\"]\n", "print(f\"Completed run_id={run_id}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Holdout test — F1-toxic and gap" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
modelf1_toxic_testf1_toxic_traintoxic_gap_ppgap_ok_<5ppf1_target_>0.75thresholdroc_auc
0Toxic-BERT0.74890.79074.18TrueFalse0.330.8768
1LR-TFIDF-2500.63010.63090.08TrueFalse0.050.7056
2Hybrid 0.7/0.30.74890.79174.28TrueFalse0.380.8773
\n", "
" ], "text/plain": [ " model f1_toxic_test f1_toxic_train toxic_gap_pp gap_ok_<5pp \\\n", "0 Toxic-BERT 0.7489 0.7907 4.18 True \n", "1 LR-TFIDF-250 0.6301 0.6309 0.08 True \n", "2 Hybrid 0.7/0.3 0.7489 0.7917 4.28 True \n", "\n", " f1_target_>0.75 threshold roc_auc \n", "0 False 0.33 0.8768 \n", "1 False 0.05 0.7056 \n", "2 False 0.38 0.8773 " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def _row(key, label):\n", " m = metrics.get(key, {})\n", " if not m:\n", " return None\n", " return {\n", " \"model\": label,\n", " \"f1_toxic_test\": m.get(\"f1_toxic\"),\n", " \"f1_toxic_train\": m.get(\"f1_toxic_train\"),\n", " \"toxic_gap_pp\": m.get(\"train_test_gap_toxic_pp\"),\n", " \"gap_ok_<5pp\": m.get(\"gap_toxic_ok\", False),\n", " \"f1_target_>0.75\": (m.get(\"f1_toxic\") or 0) > 0.75,\n", " \"threshold\": m.get(\"threshold\"),\n", " \"roc_auc\": m.get(\"roc_auc\"),\n", " }\n", "\n", "summary = pd.DataFrame(\n", " [\n", " r\n", " for r in [\n", " _row(\"transformer\", \"Toxic-BERT\"),\n", " _row(\"logistic_regression\", \"LR-TFIDF-250\"),\n", " _row(\"ensemble\", \"Hybrid 0.7/0.3\"),\n", " ]\n", " if r\n", " ]\n", ")\n", "summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Integrated report" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/markdown": [ "# Phase 5 Expert Adaptation — 20260524_193947\n", "\n", "## Targets\n", "- Test **F1-toxic** > 0.75\n", "- |Train F1-toxic − Test F1-toxic| < 5 pp (0.05)\n", "\n", "## Holdout test (tuned thresholds on validation)\n", "\n", "| Model | F1-toxic (test) | F1-toxic (train) | Toxic gap (pp) | Threshold | Gap OK |\n", "|-------|-------------------|--------------------|----------------|-----------|--------|\n", "| Toxic-BERT | 0.7489 | 0.7907 | 4.18 | 0.33 | ✅ |\n", "| LR-TFIDF (250 feat) | 0.6301 | 0.6309 | 0.08 | 0.05 | ✅ |\n", "| Hybrid 0.7/0.3 | 0.7489 | 0.7917 | 4.28 | 0.38 | ✅ |\n", "\n", "## Augmentation\n", "- Pivot language: de\n", "- Train size: 677 → 886 (+209)\n", "\n", "## Verdict\n", "**Toxic-BERT** toxic gap < 5 pp ✅; **Hybrid** toxic gap < 5 pp ✅\n", "\n", "- JSON: `reports/expert/expert_run_20260524_193947.json`\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from IPython.display import Markdown, display\n", "\n", "md_path = reports_dir / f\"integrated_report_{run_id}.md\"\n", "if md_path.exists():\n", " display(Markdown(md_path.read_text()))\n", "else:\n", " print(\"Report not found\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Conclusion\n", "\n", "Phase 5 applies **Toxic-BERT** with a frozen backbone, a **250-feature** LR safety net, **validation threshold tuning** on F1-toxic, and a **0.7/0.3** hybrid.\n", "Augmentation uses a **German** pivot for more diverse toxic paraphrases.\n", "\n", "Success criteria:\n", "- **F1-toxic (test) > 0.75**\n", "- **|F1-toxic train − F1-toxic test| < 5 pp** (`gap_toxic_ok`)\n", "\n", "Artifacts: `models/expert_toxic_bert/`, `models/expert_lr_tfidf.joblib`, `reports/expert/expert_run_{run_id}.json`." ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.7" } }, "nbformat": 4, "nbformat_minor": 5 }