{ "cells": [ { "cell_type": "markdown", "id": "57755c91", "metadata": {}, "source": [ "# Bagging (Bootstrap Aggregating)\n", "\n", "A single decision tree is readable but **unstable** — small changes in the data\n", "can produce a very different tree. **Bagging** reduces that instability: we train\n", "many trees, each on a different random sample of the data, then **average** their\n", "predictions. The result is usually more accurate and more stable than any single tree.\n", "\n", "Here we build a bagged ensemble of 10 trees by hand so you can see exactly how it works." ] }, { "cell_type": "markdown", "id": "e8108e35", "metadata": {}, "source": [ "## 1. Imports\n", "\n", "Note `resample` — that's the tool we use to draw the random **bootstrap samples**\n", "that make each tree different." ] }, { "cell_type": "code", "execution_count": 1, "id": "cell-02", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:32:08.941147Z", "iopub.status.busy": "2026-06-08T10:32:08.941049Z", "iopub.status.idle": "2026-06-08T10:32:10.481231Z", "shell.execute_reply": "2026-06-08T10:32:10.480886Z" } }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "from sklearn import tree\n", "from sklearn.metrics import roc_auc_score\n", "from sklearn.utils import resample\n", "from sklearn.preprocessing import OneHotEncoder\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.compose import ColumnTransformer\n" ] }, { "cell_type": "markdown", "id": "f28c510c", "metadata": {}, "source": [ "## 2. Load and prepare the data\n", "\n", "Same preparation as the CART notebook: drop the sensitive attributes\n", "(`Foreign_worker`, `Gender`) and recode `Credit_risk` to `0 = good`, `1 = bad`." ] }, { "cell_type": "code", "execution_count": 2, "id": "cell-04", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:32:10.483403Z", "iopub.status.busy": "2026-06-08T10:32:10.483235Z", "iopub.status.idle": "2026-06-08T10:32:10.490089Z", "shell.execute_reply": "2026-06-08T10:32:10.489775Z" } }, "outputs": [], "source": [ "# load and prepare data\n", "\n", "data = pd.read_csv('../german_credit_from_r.csv')\n", "data.drop(['Foreign_worker', 'Gender'], axis=1, inplace=True)\n", "data['Credit_risk'] = data['Credit_risk'].map({'GOOD': 0, 'BAD': 1})" ] }, { "cell_type": "markdown", "id": "d87e9981", "metadata": {}, "source": [ "## 3. Train/test split\n", "\n", "Hold out 20% as a test set, with `random_state=42` for reproducibility — the same\n", "split as the other notebooks, so the results are comparable." ] }, { "cell_type": "code", "execution_count": 3, "id": "cell-06", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:32:10.491672Z", "iopub.status.busy": "2026-06-08T10:32:10.491586Z", "iopub.status.idle": "2026-06-08T10:32:10.495497Z", "shell.execute_reply": "2026-06-08T10:32:10.495182Z" } }, "outputs": [], "source": [ "# train/test split\n", "\n", "X = data.drop('Credit_risk', axis=1)\n", "y = data['Credit_risk']\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)" ] }, { "cell_type": "markdown", "id": "b3b80d6d", "metadata": {}, "source": [ "## 4. Preprocessing\n", "\n", "Identical pipeline to the CART notebook: impute numeric columns and one-hot encode\n", "categorical ones. We define a single tree here (`max_depth=3`); the bagging loop\n", "below will refit it many times on different samples." ] }, { "cell_type": "code", "execution_count": 4, "id": "cell-08", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:32:10.497257Z", "iopub.status.busy": "2026-06-08T10:32:10.497141Z", "iopub.status.idle": "2026-06-08T10:32:10.500031Z", "shell.execute_reply": "2026-06-08T10:32:10.499764Z" } }, "outputs": [], "source": [ "# define preprocessing pipeline\n", "\n", "numeric_features = [\"Duration\", \"Credit_amount\", \"Installment_rate\", \"Resident_since\", \"Age\", \"Existing_credits\", \"People_maintenance_for\"]\n", "numeric_features_selected = numeric_features\n", "numeric_transformer = Pipeline(\n", " steps=[(\"imputer\", SimpleImputer(strategy=\"mean\"))]\n", ")\n", "\n", "categorical_features = [\"Account_status\", \"Credit_history\", \"Purpose\", \"Savings_bonds\", \"Present_employment_since\", \"Other_debtors_guarantors\", \"Property\", \"Other_installment_plans\", \"Housing\", \"Job\", \"Telephone\"]\n", "categorical_features_selected = categorical_features\n", "categorical_transformer = Pipeline(\n", " steps=[\n", " (\"encoder\", OneHotEncoder(handle_unknown=\"ignore\"))\n", " ]\n", ")\n", "preprocessor = ColumnTransformer(\n", " transformers=[\n", " (\"num\", numeric_transformer, numeric_features_selected),\n", " (\"cat\", categorical_transformer, categorical_features_selected),\n", " ]\n", ")\n", "\n", "pipe = Pipeline([\n", " (\"preprocessor\", preprocessor),\n", " ('classifier', tree.DecisionTreeClassifier(criterion='entropy', max_depth=3, random_state=42))\n", "])\n" ] }, { "cell_type": "markdown", "id": "fdeb01b0", "metadata": {}, "source": [ "## 5. The bagging loop\n", "\n", "This is the heart of the method. We repeat 10 times:\n", "\n", "1. **Bootstrap sample** — draw a random sample of the training data *with\n", " replacement* (same size as the original, so some rows repeat and others are\n", " left out). `random_state=i` makes each of the 10 samples different but reproducible.\n", "2. **Fit** a fresh tree on that sample.\n", "3. **Predict** the probability of \"bad\" for every test applicant and store it.\n", "\n", "Each tree sees a slightly different world, so each makes slightly different\n", "mistakes — exactly what we want.\n", "\n", "💡 **Try it:** raise the number of iterations from 10 to 50. The per-tree AUCs\n", "vary quite a bit — that variability is the instability bagging smooths out." ] }, { "cell_type": "code", "execution_count": null, "id": "cell-10", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:32:10.501476Z", "iopub.status.busy": "2026-06-08T10:32:10.501378Z", "iopub.status.idle": "2026-06-08T10:32:10.585572Z", "shell.execute_reply": "2026-06-08T10:32:10.585252Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "AUC for iteration 1: 0.7142084385142444\n", "AUC for iteration 2: 0.6811515807188364\n", "AUC for iteration 3: 0.737768962615699\n", "AUC for iteration 4: 0.7345834835917779\n", "AUC for iteration 5: 0.7146291621589133\n", "AUC for iteration 6: 0.754297391513403\n", "AUC for iteration 7: 0.7832672196177426\n", "AUC for iteration 8: 0.6941940137035703\n", "AUC for iteration 9: 0.7649957927635535\n", "AUC for iteration 10: 0.7204591898064672\n" ] } ], "source": [ "# Initialize an empty array to collect predictions from bagging\n", "preds = np.zeros((len(y_test), 10))\n", "\n", "# Loop for 10 iterations\n", "for i in range(10):\n", " \n", " # Create a bootstrap sample of the training data\n", " X_train_sampled, y_train_sampled = resample(X_train, y_train, replace=True, n_samples=len(X_train), random_state=i)\n", " \n", " # Fit the model on the sampled data\n", " pipe.fit(X_train_sampled, y_train_sampled)\n", "\n", " # Make predictions on the test set (probabilities for AUC)\n", " predictions_test = pipe.predict_proba(X_test)[:, 1]\n", "\n", " # Calculate AUC score\n", " auc_score = roc_auc_score(y_test, predictions_test)\n", " print(f\"AUC for iteration {i+1}: {auc_score}\")\n", " \n", " # Collect predictions from this model iteration\n", " preds[:, i] = predictions_test\n", "\n", "# The preds array now contains predictions from 10 different trees." ] }, { "cell_type": "markdown", "id": "bac0582d", "metadata": {}, "source": [ "## 6. The predictions matrix\n", "\n", "Each **row** is a test applicant; each **column** is one tree's predicted\n", "probability of \"bad\". Bagging happens when we collapse these 10 columns into one." ] }, { "cell_type": "code", "execution_count": 6, "id": "cell-12", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:32:10.587246Z", "iopub.status.busy": "2026-06-08T10:32:10.587131Z", "iopub.status.idle": "2026-06-08T10:32:10.602020Z", "shell.execute_reply": "2026-06-08T10:32:10.601713Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
0123456789
00.3201060.5177870.2734080.3741940.3118640.2547530.6758240.3633220.4214660.488806
10.3201060.4173230.2734080.3741940.3118640.2547530.3409090.6636360.4214660.257485
20.3201060.5177870.5206190.3741940.3118640.6256410.3409090.3633220.4214660.488806
30.3201060.5177870.2734080.3741940.5888890.2547530.3571430.3633220.4214660.488806
40.1830990.1969700.0000000.0114940.0000000.0079370.2686570.2077920.2307690.250000
.................................
1950.0436680.1807230.0396040.2000000.0379150.1310340.0590720.0568720.1369050.034653
1960.8421050.8823530.5206190.3741940.3118640.6256410.3409090.6636360.4214660.888889
1970.8421050.5177870.5206190.3741940.3118640.6256410.6758240.6636360.4214660.488806
1980.3201060.5177870.5206190.3741940.5888890.6256410.6758240.3633220.4214660.488806
1990.1830990.1969700.3478260.0114940.4237290.1310340.2686570.2077920.2307690.250000
\n", "

200 rows × 10 columns

\n", "
" ], "text/plain": [ " 0 1 2 3 4 5 6 \\\n", "0 0.320106 0.517787 0.273408 0.374194 0.311864 0.254753 0.675824 \n", "1 0.320106 0.417323 0.273408 0.374194 0.311864 0.254753 0.340909 \n", "2 0.320106 0.517787 0.520619 0.374194 0.311864 0.625641 0.340909 \n", "3 0.320106 0.517787 0.273408 0.374194 0.588889 0.254753 0.357143 \n", "4 0.183099 0.196970 0.000000 0.011494 0.000000 0.007937 0.268657 \n", ".. ... ... ... ... ... ... ... \n", "195 0.043668 0.180723 0.039604 0.200000 0.037915 0.131034 0.059072 \n", "196 0.842105 0.882353 0.520619 0.374194 0.311864 0.625641 0.340909 \n", "197 0.842105 0.517787 0.520619 0.374194 0.311864 0.625641 0.675824 \n", "198 0.320106 0.517787 0.520619 0.374194 0.588889 0.625641 0.675824 \n", "199 0.183099 0.196970 0.347826 0.011494 0.423729 0.131034 0.268657 \n", "\n", " 7 8 9 \n", "0 0.363322 0.421466 0.488806 \n", "1 0.663636 0.421466 0.257485 \n", "2 0.363322 0.421466 0.488806 \n", "3 0.363322 0.421466 0.488806 \n", "4 0.207792 0.230769 0.250000 \n", ".. ... ... ... \n", "195 0.056872 0.136905 0.034653 \n", "196 0.663636 0.421466 0.888889 \n", "197 0.663636 0.421466 0.488806 \n", "198 0.363322 0.421466 0.488806 \n", "199 0.207792 0.230769 0.250000 \n", "\n", "[200 rows x 10 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.DataFrame(preds)" ] }, { "cell_type": "markdown", "id": "3dd075d9", "metadata": {}, "source": [ "## 7. Aggregate and evaluate\n", "\n", "We **average** the 10 probability columns into a single ensemble prediction and\n", "score its AUC. Compare it to the individual per-tree AUCs above — the average is\n", "**better than a typical single tree**, because averaging cancels their independent errors.\n", "\n", "💡 **Try it:** is the ensemble AUC higher than the best single tree, or just higher\n", "than the average one? What does that tell you about when bagging helps most?" ] }, { "cell_type": "code", "execution_count": 7, "id": "cell-14", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:32:10.604154Z", "iopub.status.busy": "2026-06-08T10:32:10.604013Z", "iopub.status.idle": "2026-06-08T10:32:10.606951Z", "shell.execute_reply": "2026-06-08T10:32:10.606683Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Final AUC using averaged predictions: 0.8001562687823056\n" ] } ], "source": [ "# Calculate the mean of predictions across rows\n", "preds_mean = np.mean(preds, axis=1)\n", "\n", "# Calculate AUC using the averaged predictions\n", "final_auc = roc_auc_score(y_test, preds_mean)\n", "print(f\"Final AUC using averaged predictions: {final_auc}\")" ] } ], "metadata": { "kernelspec": { "display_name": "germancredit", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.15" } }, "nbformat": 4, "nbformat_minor": 5 }