{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "228b68f5",
   "metadata": {},
   "source": [
    "# 05 \u2014 Model Comparison & Best Model Selection\n",
    "\n",
    "Compare all 4 trained models using a weighted composite score:\n",
    "\n",
    "| Metric | Weight |\n",
    "|--------|--------|\n",
    "| Macro F1 | 50% |\n",
    "| Top-1 Accuracy | 20% |\n",
    "| Inference Latency | 15% |\n",
    "| Model Size | 10% |\n",
    "| Calibration | 5% |\n",
    "\n",
    "### Expected Output\n",
    "- Radar chart comparing all models\n",
    "- Grouped bar chart of key metrics\n",
    "- Comparison table with rankings\n",
    "- Best model recommendation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7c96a7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os, json\n",
    "from pathlib import Path\n",
    "\n",
    "PROJECT_ROOT = Path(os.getcwd()).resolve()\n",
    "if 'notebooks' in str(PROJECT_ROOT):\n",
    "    PROJECT_ROOT = PROJECT_ROOT.parent.parent\n",
    "sys.path.insert(0, str(PROJECT_ROOT))\n",
    "print(f'Project root: {PROJECT_ROOT}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17733cd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from ml.src.evaluation.compare_models import (\n",
    "    compute_weighted_scores,\n",
    "    plot_comparison_radar,\n",
    "    plot_comparison_bar,\n",
    "    generate_comparison_report,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c29af067",
   "metadata": {},
   "source": [
    "## 1. Load All Model Reports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cea1417d",
   "metadata": {},
   "outputs": [],
   "source": [
    "reports_dir = PROJECT_ROOT / 'ml' / 'artifacts' / 'reports'\n",
    "model_names = ['mlp', 'cnn', 'resnet', 'vit']\n",
    "\n",
    "model_reports = []\n",
    "for name in model_names:\n",
    "    report_path = reports_dir / f'{name}_report.json'\n",
    "    if report_path.exists():\n",
    "        with open(report_path) as f:\n",
    "            report = json.load(f)\n",
    "        model_reports.append(report)\n",
    "        print(f'\u2713 Loaded {name} report')\n",
    "    else:\n",
    "        print(f'\u2717 Missing {name} report \u2014 run notebook {model_names.index(name)+1:02d} first')\n",
    "\n",
    "print(f'\\nLoaded {len(model_reports)} / {len(model_names)} model reports')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "570fe372",
   "metadata": {},
   "source": [
    "## 2. Summary Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18f372b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = []\n",
    "for r in model_reports:\n",
    "    metrics = r.get('metrics', {})\n",
    "    latency = r.get('latency', {})\n",
    "    rows.append({\n",
    "        'Model': r['model_name'],\n",
    "        'Accuracy': f\"{metrics.get('accuracy', 0):.4f}\",\n",
    "        'Macro F1': f\"{metrics.get('macro_f1', 0):.4f}\",\n",
    "        'Macro Precision': f\"{metrics.get('macro_precision', 0):.4f}\",\n",
    "        'Macro Recall': f\"{metrics.get('macro_recall', 0):.4f}\",\n",
    "        'Latency (ms)': f\"{latency.get('avg_ms', 0):.1f}\",\n",
    "        'Size (MB)': f\"{r.get('model_size_mb', 0):.1f}\",\n",
    "        'Parameters': f\"{r.get('num_parameters', 0):,}\",\n",
    "    })\n",
    "\n",
    "df_summary = pd.DataFrame(rows)\n",
    "df_summary.style.set_caption('Model Comparison Summary')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1823a6bb",
   "metadata": {},
   "source": [
    "## 3. Weighted Scoring & Ranking"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e4addc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "figures_dir = PROJECT_ROOT / 'ml' / 'artifacts' / 'figures' / 'comparison'\n",
    "figures_dir.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "comparison_report = generate_comparison_report(\n",
    "    model_reports,\n",
    "    output_dir=figures_dir,\n",
    ")\n",
    "\n",
    "print('\\n=== Rankings ===')\n",
    "for r in comparison_report['rankings']:\n",
    "    print(f\"  #{r['rank']} {r['model_name']:>8s}  \"\n",
    "          f\"composite={r['composite_score']:.4f}  \"\n",
    "          f\"F1={r['raw_metrics']['macro_f1']:.4f}  \"\n",
    "          f\"Acc={r['raw_metrics']['accuracy']:.4f}  \"\n",
    "          f\"Lat={r['raw_metrics']['latency_ms']:.1f}ms  \"\n",
    "          f\"Size={r['raw_metrics']['model_size_mb']:.1f}MB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45637652",
   "metadata": {},
   "source": [
    "## 4. Radar Chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "672dd90d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image as IPImage, display\n",
    "radar_path = figures_dir / 'comparison_radar.png'\n",
    "if radar_path.exists():\n",
    "    display(IPImage(filename=str(radar_path), width=600))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb4d3040",
   "metadata": {},
   "source": [
    "## 5. Bar Chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0101ced1",
   "metadata": {},
   "outputs": [],
   "source": [
    "bar_path = figures_dir / 'comparison_bar.png'\n",
    "if bar_path.exists():\n",
    "    display(IPImage(filename=str(bar_path), width=800))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fd19a50",
   "metadata": {},
   "source": [
    "## 6. Per-Model Confusion Matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c62b87c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, len(model_reports), figsize=(6*len(model_reports), 5))\n",
    "if len(model_reports) == 1:\n",
    "    axes = [axes]\n",
    "\n",
    "for idx, report in enumerate(model_reports):\n",
    "    cm_path = PROJECT_ROOT / 'ml' / 'artifacts' / 'figures' / report['model_name'] / 'confusion_matrix.png'\n",
    "    if cm_path.exists():\n",
    "        img = plt.imread(str(cm_path))\n",
    "        axes[idx].imshow(img)\n",
    "        axes[idx].set_title(report['model_name'], fontweight='bold')\n",
    "    axes[idx].axis('off')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6adb115",
   "metadata": {},
   "source": [
    "## 7. Best Model Recommendation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca574420",
   "metadata": {},
   "outputs": [],
   "source": [
    "best = comparison_report['rankings'][0]\n",
    "print('=' * 60)\n",
    "print(f'\ud83c\udfc6 RECOMMENDED MODEL: {best[\"model_name\"].upper()}')\n",
    "print('=' * 60)\n",
    "print(f'  Composite Score: {best[\"composite_score\"]:.4f}')\n",
    "print(f'  Macro F1:        {best[\"raw_metrics\"][\"macro_f1\"]:.4f}')\n",
    "print(f'  Accuracy:        {best[\"raw_metrics\"][\"accuracy\"]:.4f}')\n",
    "print(f'  Latency:         {best[\"raw_metrics\"][\"latency_ms\"]:.1f} ms')\n",
    "print(f'  Model Size:      {best[\"raw_metrics\"][\"model_size_mb\"]:.1f} MB')\n",
    "print('\\nCheckpoint path:')\n",
    "print(f'  ml/artifacts/checkpoints/{best[\"model_name\"]}_best.pth')\n",
    "print('\\nTo deploy this model, update the backend environment variable:')\n",
    "print(f'  MODEL_PATH=ml/artifacts/checkpoints/{best[\"model_name\"]}_best.pth')\n",
    "print(f'  MODEL_NAME={best[\"model_name\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c83b31c",
   "metadata": {},
   "source": [
    "## 8. Export for Deployment\n",
    "\n",
    "Copy the best checkpoint to the `models/` directory for the backend to load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c1084ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "best_name = best['model_name']\n",
    "src = PROJECT_ROOT / 'ml' / 'artifacts' / 'checkpoints' / f'{best_name}_best.pth'\n",
    "dst = PROJECT_ROOT / 'models' / f'{best_name}_best.pth'\n",
    "\n",
    "if src.exists():\n",
    "    shutil.copy2(src, dst)\n",
    "    print(f'\u2713 Copied {src.name} \u2192 {dst}')\n",
    "\n",
    "    # Also save class names for the backend\n",
    "    import torch\n",
    "    ckpt = torch.load(src, map_location='cpu', weights_only=False)\n",
    "    if 'class_names' in ckpt:\n",
    "        classes_path = PROJECT_ROOT / 'models' / 'classes.txt'\n",
    "        with open(classes_path, 'w') as f:\n",
    "            f.write(','.join(ckpt['class_names']))\n",
    "        print(f'\u2713 Updated {classes_path}')\n",
    "else:\n",
    "    print(f'\u2717 Checkpoint not found: {src}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebda2f85",
   "metadata": {},
   "source": [
    "---\n",
    "**\u2705 Model comparison complete!**\n",
    "\n",
    "### Next Steps\n",
    "1. Start the backend: `PYTHONPATH=. uvicorn backend.app.main:app --reload`\n",
    "2. Start the frontend: `cd frontend && npm run dev`\n",
    "3. Upload images and verify predictions"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Cattle Classifier (Python 3.12)",
   "language": "python",
   "name": "cattle-classifier"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}