{ "cells": [ { "cell_type": "markdown", "id": "cell-00", "metadata": {}, "source": [ "# Classification with a Decision Tree (CART)\n", "\n", "We train a single **decision tree** to predict whether a loan applicant is a\n", "*good* or *bad* credit risk, using the German Credit dataset (1,000 past\n", "applicants). Decision trees are a great first model because the result is easy to\n", "read: it's just a sequence of yes/no questions.\n", "\n", "We follow the standard supervised-learning workflow: **load → split → preprocess → fit → evaluate.**" ] }, { "cell_type": "markdown", "id": "cell-01", "metadata": {}, "source": [ "## 1. Imports\n", "\n", "[scikit-learn](https://scikit-learn.org/) provides the model and the\n", "preprocessing/evaluation tools; pandas and numpy handle the data." ] }, { "cell_type": "code", "execution_count": 1, "id": "cell-02", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:52:08.449606Z", "iopub.status.busy": "2026-06-08T10:52:08.448949Z", "iopub.status.idle": "2026-06-08T10:52:11.701293Z", "shell.execute_reply": "2026-06-08T10:52:11.700566Z" } }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import OneHotEncoder\n", "from sklearn.impute import SimpleImputer\n", "from sklearn.compose import ColumnTransformer\n", "from sklearn import tree\n", "from sklearn import metrics" ] }, { "cell_type": "markdown", "id": "cell-03", "metadata": {}, "source": [ "## 2. Load and prepare the data\n", "\n", "Each row is one applicant. We make two preparation choices:\n", "\n", "- **Drop `Foreign_worker` and `Gender`.** These are sensitive attributes — using\n", " them to judge creditworthiness would be discriminatory, so we exclude them.\n", "- **Recode the target** `Credit_risk` to numbers: `0 = good`, `1 = bad`. Models\n", " need numeric labels." ] }, { "cell_type": "code", "execution_count": 2, "id": "cell-04", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:52:11.706673Z", "iopub.status.busy": "2026-06-08T10:52:11.705757Z", "iopub.status.idle": "2026-06-08T10:52:11.721517Z", "shell.execute_reply": "2026-06-08T10:52:11.720617Z" } }, "outputs": [], "source": [ "# load and prepare data\n", "\n", "data = pd.read_csv('../german_credit_from_r.csv')\n", "data.drop(['Foreign_worker', 'Gender'], axis=1, inplace=True)\n", "data['Credit_risk'] = data['Credit_risk'].map({'GOOD': 0, 'BAD': 1})" ] }, { "cell_type": "markdown", "id": "cell-05", "metadata": {}, "source": [ "## 3. Train/test split\n", "\n", "We hold out 20% of the data as a **test set** the model never sees during\n", "training, to estimate how it will do on *new* applicants. `random_state=42` fixes\n", "the split so the notebook is reproducible." ] }, { "cell_type": "code", "execution_count": 3, "id": "cell-06", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:52:11.726745Z", "iopub.status.busy": "2026-06-08T10:52:11.726004Z", "iopub.status.idle": "2026-06-08T10:52:11.732974Z", "shell.execute_reply": "2026-06-08T10:52:11.731919Z" } }, "outputs": [], "source": [ "# train/test split\n", "\n", "X = data.drop('Credit_risk', axis=1)\n", "y = data['Credit_risk']\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)" ] }, { "cell_type": "markdown", "id": "cell-07", "metadata": {}, "source": [ "## 4. Preprocessing\n", "\n", "Different column types need different handling, bundled in a `ColumnTransformer`:\n", "\n", "- **Numeric** features (age, duration, amount, …): fill missing values with the\n", " column mean. Trees split on thresholds, so we don't need to scale them.\n", "- **Categorical** features (account status, purpose, …): **one-hot encode** into\n", " 0/1 columns. `handle_unknown=\"ignore\"` is safe if a category shows up in the\n", " test set that wasn't in training.\n", "\n", "Preprocessing and the classifier are chained in one `Pipeline`, so identical\n", "steps apply to train and test data." ] }, { "cell_type": "code", "execution_count": 4, "id": "cell-08", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:52:11.737193Z", "iopub.status.busy": "2026-06-08T10:52:11.736879Z", "iopub.status.idle": "2026-06-08T10:52:11.740943Z", "shell.execute_reply": "2026-06-08T10:52:11.740106Z" } }, "outputs": [], "source": [ "# define preprocessing pipeline\n", "\n", "numeric_features = [\"Duration\", \"Credit_amount\", \"Installment_rate\", \"Resident_since\", \"Age\", \"Existing_credits\", \"People_maintenance_for\"]\n", "numeric_transformer = Pipeline(steps=[(\"imputer\", SimpleImputer(strategy=\"mean\"))])\n", "\n", "categorical_features = [\"Account_status\", \"Credit_history\", \"Purpose\", \"Savings_bonds\", \"Present_employment_since\", \"Other_debtors_guarantors\", \"Property\", \"Other_installment_plans\", \"Housing\", \"Job\", \"Telephone\"]\n", "categorical_transformer = Pipeline(steps=[(\"encoder\", OneHotEncoder(handle_unknown=\"ignore\"))])\n", "\n", "preprocessor = ColumnTransformer(\n", " transformers=[\n", " (\"num\", numeric_transformer, numeric_features),\n", " (\"cat\", categorical_transformer, categorical_features),\n", " ]\n", ")\n", "\n", "pipe = Pipeline([\n", " (\"preprocessor\", preprocessor),\n", " (\"classifier\", tree.DecisionTreeClassifier(criterion='entropy', max_depth=3, random_state=42)),\n", "])" ] }, { "cell_type": "markdown", "id": "cell-09", "metadata": {}, "source": [ "## 5. Fit the model\n", "\n", "We grow the tree using **entropy** (information gain) to pick splits, capped at\n", "`max_depth=3` so it stays small and readable.\n", "\n", "💡 **Try it:** change `max_depth` (e.g. 2, 5, 10). A deeper tree fits the training\n", "data more closely — but does it actually do better on the *test* set, or does it\n", "start to **overfit**?" ] }, { "cell_type": "code", "execution_count": 5, "id": "cell-10", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:52:11.745797Z", "iopub.status.busy": "2026-06-08T10:52:11.745012Z", "iopub.status.idle": "2026-06-08T10:52:11.803167Z", "shell.execute_reply": "2026-06-08T10:52:11.801875Z" } }, "outputs": [ { "data": { "text/html": [ "
Pipeline(steps=[('preprocessor',\n",
       "                 ColumnTransformer(transformers=[('num',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer())]),\n",
       "                                                  ['Duration', 'Credit_amount',\n",
       "                                                   'Installment_rate',\n",
       "                                                   'Resident_since', 'Age',\n",
       "                                                   'Existing_credits',\n",
       "                                                   'People_maintenance_for']),\n",
       "                                                 ('cat',\n",
       "                                                  Pipeline(steps=[('encoder',\n",
       "                                                                   OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                  ['Account_status',\n",
       "                                                   'Credit_history', 'Purpose',\n",
       "                                                   'Savings_bonds',\n",
       "                                                   'Present_employment_since',\n",
       "                                                   'Other_debtors_guarantors',\n",
       "                                                   'Property',\n",
       "                                                   'Other_installment_plans',\n",
       "                                                   'Housing', 'Job',\n",
       "                                                   'Telephone'])])),\n",
       "                ('classifier',\n",
       "                 DecisionTreeClassifier(criterion='entropy', max_depth=3,\n",
       "                                        random_state=42))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "Pipeline(steps=[('preprocessor',\n", " ColumnTransformer(transformers=[('num',\n", " Pipeline(steps=[('imputer',\n", " SimpleImputer())]),\n", " ['Duration', 'Credit_amount',\n", " 'Installment_rate',\n", " 'Resident_since', 'Age',\n", " 'Existing_credits',\n", " 'People_maintenance_for']),\n", " ('cat',\n", " Pipeline(steps=[('encoder',\n", " OneHotEncoder(handle_unknown='ignore'))]),\n", " ['Account_status',\n", " 'Credit_history', 'Purpose',\n", " 'Savings_bonds',\n", " 'Present_employment_since',\n", " 'Other_debtors_guarantors',\n", " 'Property',\n", " 'Other_installment_plans',\n", " 'Housing', 'Job',\n", " 'Telephone'])])),\n", " ('classifier',\n", " DecisionTreeClassifier(criterion='entropy', max_depth=3,\n", " random_state=42))])" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# fit the model on the training data\n", "\n", "pipe.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "id": "cell-11", "metadata": {}, "source": [ "## 6. Evaluate\n", "\n", "We score the model on the held-out test set using **AUC** (Area Under the ROC\n", "Curve). AUC measures how well the model *ranks* applicants by risk — **1.0** is a\n", "perfect ranking, **0.5** is no better than random. Because it looks at the whole\n", "ranking, it doesn't depend on picking a particular probability cut-off.\n", "\n", "💡 **Try it:** note this AUC — you'll compare it against the Bagging and Random\n", "Forest notebooks, which build on this single tree." ] }, { "cell_type": "code", "execution_count": 6, "id": "cell-12", "metadata": { "execution": { "iopub.execute_input": "2026-06-08T10:52:11.807364Z", "iopub.status.busy": "2026-06-08T10:52:11.806756Z", "iopub.status.idle": "2026-06-08T10:52:11.821134Z", "shell.execute_reply": "2026-06-08T10:52:11.820129Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "AUC: 0.769\n" ] } ], "source": [ "# evaluate on the test set\n", "\n", "preds = pd.DataFrame(pipe.predict_proba(X_test))\n", "preds.columns = ['prob_0', 'prob_1']\n", "fpr, tpr, thresholds = metrics.roc_curve(y_test, preds[\"prob_1\"], pos_label=1)\n", "\n", "print('AUC: ', np.round(metrics.auc(fpr, tpr), 3))" ] } ], "metadata": { "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.15" } }, "nbformat": 4, "nbformat_minor": 5 }