scripts/RandomForest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-00",
   "metadata": {},
   "source": [
    "# Random Forest (with hyperparameter tuning)\n",
    "\n",
    "A **random forest** takes bagging one step further: it builds many trees on\n",
    "bootstrap samples *and* lets each split consider only a random subset of features.\n",
    "This decorrelates the trees and usually beats plain bagging.\n",
    "\n",
    "Rather than guess good settings, we let **cross-validated grid search** find the\n",
    "hyperparameters that give the best **AUC**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-01",
   "metadata": {},
   "source": [
    "## 1. Imports\n",
    "\n",
    "`GridSearchCV` runs the hyperparameter search; `RandomForestClassifier` is the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-02",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-06-15T15:08:27.848511Z",
     "iopub.status.busy": "2026-06-15T15:08:27.848339Z",
     "iopub.status.idle": "2026-06-15T15:08:30.272401Z",
     "shell.execute_reply": "2026-06-15T15:08:30.272002Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.inspection import permutation_importance\n",
    "from sklearn import metrics\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-03",
   "metadata": {},
   "source": [
    "## 2. Load and prepare the data\n",
    "\n",
    "Same preparation as the other notebooks: drop the sensitive attributes and recode\n",
    "`Credit_risk` to `0 = good`, `1 = bad`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-04",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-06-15T15:08:30.274618Z",
     "iopub.status.busy": "2026-06-15T15:08:30.274269Z",
     "iopub.status.idle": "2026-06-15T15:08:30.281775Z",
     "shell.execute_reply": "2026-06-15T15:08:30.281304Z"
    }
   },
   "outputs": [],
   "source": [
    "# load and prepare data\n",
    "\n",
    "data = pd.read_csv('../german_credit_from_r.csv')\n",
    "data.drop(['Foreign_worker', 'Gender'], axis=1, inplace=True)\n",
    "data['Credit_risk'] = data['Credit_risk'].map({'GOOD': 0, 'BAD': 1})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-05",
   "metadata": {},
   "source": [
    "## 3. Train/test split\n",
    "\n",
    "The same 20% hold-out with `random_state=42`, so results are comparable to the\n",
    "CART and Bagging notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-06",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-06-15T15:08:30.283544Z",
     "iopub.status.busy": "2026-06-15T15:08:30.283432Z",
     "iopub.status.idle": "2026-06-15T15:08:30.287177Z",
     "shell.execute_reply": "2026-06-15T15:08:30.286862Z"
    }
   },
   "outputs": [],
   "source": [
    "# train/test split\n",
    "\n",
    "X = data.drop('Credit_risk', axis=1)\n",
    "y = data['Credit_risk']\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-07",
   "metadata": {},
   "source": [
    "## 4. Preprocessing\n",
    "\n",
    "The same impute + one-hot `ColumnTransformer` as before. We keep it separate from\n",
    "the model so the grid search can vary the model while reusing the preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-08",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-06-15T15:08:30.289135Z",
     "iopub.status.busy": "2026-06-15T15:08:30.288993Z",
     "iopub.status.idle": "2026-06-15T15:08:30.292077Z",
     "shell.execute_reply": "2026-06-15T15:08:30.291699Z"
    }
   },
   "outputs": [],
   "source": [
    "# define preprocessing pipeline\n",
    "\n",
    "numeric_features = [\"Duration\", \"Credit_amount\", \"Installment_rate\", \"Resident_since\", \"Age\", \"Existing_credits\", \"People_maintenance_for\"]\n",
    "numeric_transformer = SimpleImputer(strategy=\"mean\")\n",
    "\n",
    "categorical_features = [\"Account_status\", \"Credit_history\", \"Purpose\", \"Savings_bonds\", \"Present_employment_since\", \"Other_debtors_guarantors\", \"Property\", \"Other_installment_plans\", \"Housing\", \"Job\", \"Telephone\"]\n",
    "categorical_transformer = OneHotEncoder(handle_unknown=\"ignore\")\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        (\"num\", numeric_transformer, numeric_features),\n",
    "        (\"cat\", categorical_transformer, categorical_features),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-09",
   "metadata": {},
   "source": [
    "## 5. Tuning the forest\n",
    "\n",
    "A random forest has several settings that control how complex it grows — how many\n",
    "trees, how deep each one is, and how many features a split may consider. Rather\n",
    "than guess, we let `GridSearchCV` try every combination and measure each with\n",
    "**5-fold cross-validation on the training data**, scoring on **AUC**. The best\n",
    "combination is then refit automatically.\n",
    "\n",
    "💡 **Try it:** widen the grid (e.g. add more `max_depth` values). Does a larger\n",
    "search find a meaningfully better model, or does the AUC plateau?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-10",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-06-15T15:08:30.293934Z",
     "iopub.status.busy": "2026-06-15T15:08:30.293822Z",
     "iopub.status.idle": "2026-06-15T15:08:40.573385Z",
     "shell.execute_reply": "2026-06-15T15:08:40.572805Z"
    }
   },
   "outputs": [],
   "source": [
    "# tune the random forest with cross-validation, scoring on AUC\n",
    "\n",
    "pipe = Pipeline([\n",
    "    (\"preprocessor\", preprocessor),\n",
    "    (\"classifier\", RandomForestClassifier(criterion='entropy', random_state=42, n_jobs=-1)),\n",
    "])\n",
    "\n",
    "param_grid = {\n",
    "    \"classifier__n_estimators\": [300],\n",
    "    \"classifier__max_depth\": [None, 6, 12],\n",
    "    \"classifier__min_samples_leaf\": [1, 5, 20],\n",
    "    \"classifier__max_features\": [\"sqrt\", 0.5],\n",
    "}\n",
    "\n",
    "search = GridSearchCV(pipe, param_grid, scoring=\"roc_auc\", cv=5, n_jobs=-1)\n",
    "search.fit(X_train, y_train)\n",
    "\n",
    "print(\"Best parameters:\")\n",
    "for k, v in search.best_params_.items():\n",
    "    print(f\"  {k.replace('classifier__', '')}: {v}\")\n",
    "print(\"Best cross-validated AUC:\", np.round(search.best_score_, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-11",
   "metadata": {},
   "source": [
    "## 6. Evaluate the tuned model\n",
    "\n",
    "We score the best model on the untouched test set with **AUC**. Compare it to the\n",
    "CART and Bagging notebooks — the random forest should rank applicants better than\n",
    "a single tree or a simple bagged ensemble."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cell-12",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-06-15T15:08:40.584765Z",
     "iopub.status.busy": "2026-06-15T15:08:40.584518Z",
     "iopub.status.idle": "2026-06-15T15:08:40.611795Z",
     "shell.execute_reply": "2026-06-15T15:08:40.611358Z"
    }
   },
   "outputs": [],
   "source": [
    "# evaluate the tuned model on the held-out test set\n",
    "\n",
    "model = search.best_estimator_\n",
    "proba = model.predict_proba(X_test)[:, 1]\n",
    "print('AUC:', np.round(metrics.roc_auc_score(y_test, proba), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf05b42c",
   "metadata": {},
   "source": [
    "## 7. Which features drive the predictions?\n",
    "\n",
    "A random forest is accurate but hard to read — unlike the single CART tree, we\n",
    "can't just look at it. **Permutation importance** answers *\"how much does each\n",
    "feature matter?\"* empirically: we randomly **shuffle one feature's values** in the\n",
    "test set (breaking its link to the target) and measure how much the **AUC drops**.\n",
    "A large drop means the model leaned heavily on that feature; little or no drop\n",
    "means it barely used it.\n",
    "\n",
    "We repeat the shuffle `n_repeats=10` times per feature and average the results, so\n",
    "they're stable — the black error bars show the variation across repeats.\n",
    "`random_state=42` keeps it reproducible.\n",
    "\n",
    "💡 **Try it:** the most important features here should line up with the splits\n",
    "near the top of the CART tree. Do they match your intuition about what makes a\n",
    "loan risky?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7ce1813",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-06-15T15:08:40.613635Z",
     "iopub.status.busy": "2026-06-15T15:08:40.613448Z",
     "iopub.status.idle": "2026-06-15T15:08:42.208731Z",
     "shell.execute_reply": "2026-06-15T15:08:42.207005Z"
    }
   },
   "outputs": [],
   "source": [
    "# permutation importance: shuffle each feature and measure the drop in test AUC\n",
    "\n",
    "result = permutation_importance(\n",
    "    model, X_test, y_test,\n",
    "    scoring=\"roc_auc\", n_repeats=10, random_state=42, n_jobs=-1,\n",
    ")\n",
    "\n",
    "importances = (\n",
    "    pd.DataFrame(\n",
    "        {\"mean\": result.importances_mean, \"std\": result.importances_std},\n",
    "        index=X_test.columns,\n",
    "    )\n",
    "    .sort_values(\"mean\")\n",
    ")\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8, 6))\n",
    "ax.barh(importances.index, importances[\"mean\"], xerr=importances[\"std\"],\n",
    "        color=\"steelblue\", edgecolor=\"black\")\n",
    "ax.axvline(0, color=\"grey\", linewidth=0.8)\n",
    "ax.set_xlabel(\"Drop in test AUC when the feature is shuffled\")\n",
    "ax.set_title(\"Permutation feature importance (Random Forest)\")\n",
    "fig.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58d56edf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "germancredit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.23"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}