{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "LPPvT9ymC3IT" }, "source": [ "# Problem Statement" ] }, { "cell_type": "markdown", "metadata": { "id": "qGihLYlPigGF" }, "source": [ "## Business Context" ] }, { "cell_type": "markdown", "metadata": { "id": "pcYVQzSbllJN" }, "source": [ "In modern manufacturing, predictive maintenance plays a critical role in ensuring equipment availability and minimizing unplanned downtime. Unexpected machine failures can cause significant delays and financial losses. A manufacturing company is working to enhance the efficiency and reliability of its predictive maintenance system by leveraging operational and sensor data collected from machine tools. This data includes variables such as air temperature, process temperature, rotational speed, torque, tool wear, and failure indicators.\n", "\n", "While the company has developed a machine learning model to predict failures, the current process for data handling, model training, and deployment is manual and time-consuming—requiring engineers to rerun notebooks each time new data becomes available. This introduces inefficiencies and delays in responding to potential failures.\n", "\n", "To address this, there is a clear need for an MLOps pipeline that can seamlessly handle data registration, preprocessing, model training with tuning, and deployment. Such a pipeline would enable real-time integration of new data, reduce manual effort, and support timely, data-driven maintenance decisions—ultimately improving production reliability and reducing operational costs." ] }, { "cell_type": "markdown", "metadata": { "id": "LN0KWTL4i2B-" }, "source": [ "## Objective" ] }, { "cell_type": "markdown", "metadata": { "id": "XupSlU6QlmDQ" }, "source": [ "As an MLOps engineer, you have to design and implement a comprehensive CI/CD pipeline integrated with MLflow for continuous machine learning experimentation tracking. This pipeline will automate critical tasks including data registration, preprocessing, model training with hyperparameter tuning, and deployment to production. The objective is to streamline the entire machine learning workflow, allowing for automated execution of each component whenever new data is ingested, thereby minimizing manual intervention, enhancing operational efficiency, and ensuring robust tracking of experiments and model artifacts through MLflow. This will facilitate better collaboration among team members, faster iterations, and more reliable deployment of machine learning models into production environments." ] }, { "cell_type": "markdown", "metadata": { "id": "cuC5FGgnEmSF" }, "source": [ "## Pre-requisites" ] }, { "cell_type": "markdown", "metadata": { "id": "XEuuy1fIEmSF" }, "source": [ "This project deploys everything on **GitHub Actions** (the ML pipeline) and **Streamlit Community Cloud** (the live app). No Hugging Face and no Codespaces are used.\n", "\n", "**1. Create a Personal Access Token (PAT)**\n", "- GitHub → **Settings** → **Developer settings** → **Personal access tokens** → **Tokens (classic)**\n", "- **Generate new token (classic)** and enable these scopes:\n", " - `repo` (push code + let the pipeline commit the trained model back)\n", " - `workflow` (push the Actions workflow file)\n", "- Copy the token.\n", "\n", "**2. Store the token in Colab secrets**\n", "- In Colab, click the **key icon** (left sidebar) → **Add new secret**\n", "- Name: **GH_TOKEN** (must match `COLAB_SECRET_NAME` in the config cell below)\n", "- Value: paste your token, and toggle **Notebook access** on.\n", "\n", "**3. Create a Streamlit Community Cloud account**\n", "- Go to **https://share.streamlit.io** and sign in with the **same GitHub account** so it can access your repo. You'll deploy the app from there at the very end (see the last section).\n", "\n", "**4. Get an ngrok authtoken (for local MLflow experimentation)**\n", "- Go to **https://dashboard.ngrok.com/authtokens**, sign up/sign in, and copy your authtoken.\n", "- You'll paste this into the MLflow/ngrok cell later, in the **Experimentation and Tracking (Development Environment)** section." ] }, { "cell_type": "markdown", "source": [ "##Installing and Importing Necessary Libraries" ], "metadata": { "id": "hoTSxB0eV3jf" } }, { "cell_type": "code", "source": [ "!pip install -q PyGithub==2.9.1\n", "!pip install mlflow==3.0.1 pyngrok==7.2.12 -q" ], "metadata": { "id": "sGegY4MmVi_A" }, "execution_count": 112, "outputs": [] }, { "cell_type": "markdown", "source": [ "**Note:**\n", "\n", "- After running the above cell, kindly restart the notebook kernel (for Jupyter Notebook) or runtime (for Google Colab) and run all cells sequentially from the next cell.\n", "\n", "- On executing the above line of code, you might see a warning regarding package dependencies. This error message can be ignored as the above code ensures that all necessary libraries and their dependencies are maintained to successfully execute the code in this notebook." ], "metadata": { "id": "ai3Z1xkFV80E" } }, { "cell_type": "code", "source": [ "import os\n", "from github import Github, GithubException\n", "\n", "from pyngrok import ngrok\n", "import subprocess\n", "import mlflow\n", "\n", "import pandas as pd\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n", "from sklearn.compose import make_column_transformer\n", "from sklearn.pipeline import make_pipeline\n", "import xgboost as xgb\n", "from sklearn.model_selection import GridSearchCV\n", "from sklearn.metrics import classification_report\n", "import joblib" ], "metadata": { "id": "U54adAhfVsWz" }, "execution_count": 2, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "NoWqvBRDEmSG" }, "source": [ "## Configuration" ] }, { "cell_type": "code", "metadata": { "id": "pOQH9MPAEmSG" }, "execution_count": null, "outputs": [], "source": [ "# Edit these three values, then run every cell top to bottom.\n", "GITHUB_USERNAME = \"Swetha1929\" # Your GitHub username\n", "REPO_NAME = \"https://github.com/Swetha1929/predictive-maintenance-mlops\" # Repository name\n", "COLAB_SECRET_NAME = \"GITHUB_TOKEN\" # Name of the secret in Colab\n", "\n", "REPO = f\"{GITHUB_USERNAME}/{REPO_NAME}\"\n", "BRANCH = \"main\"" ] }, { "cell_type": "markdown", "source": [ "### Secrets in Colab" ], "metadata": { "id": "Xv4xyPs-YzOh" } }, { "cell_type": "markdown", "source": [ "**Secrets** in Google Colab provide a secure way to store sensitive information such as API keys, access tokens, and passwords. Instead of hardcoding these values in your notebook, you can save them as secrets and access them securely whenever needed." ], "metadata": { "id": "ZTn9QfuPFtMP" } }, { "cell_type": "markdown", "source": [ "1. In the left pane of Google Colab, click the fifth icon, which looks like a key." ], "metadata": { "id": "UcXbMr9-Y2b4" } }, { "cell_type": "markdown", "source": [ "" ], "metadata": { "id": "O4nzux__Yuns" } }, { "cell_type": "markdown", "source": [ "2. Then, click **+ Add new secret**. In the **Name** field, enter a name to identify the GitHub Personal Access Token, for example, **Gittoken**. In the **Value** field, paste the token. Once done, close the **Secrets** window. We will access this token programmatically later." ], "metadata": { "id": "_GbLKduqY9-c" } }, { "cell_type": "markdown", "source": [ "" ], "metadata": { "id": "RkJ6WU2sZJ-S" } }, { "cell_type": "code", "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "alv74F-cEmSH", "outputId": "fd4567ca-47d4-4cd4-90cf-cee3c6e7b1bf" }, "execution_count": 9, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Token loaded.\n" ] } ], "source": [ "COLAB_SECRET_NAME = \"GITHUB_TOKEN\"\n", "\n", "import os\n", "from google.colab import userdata\n", "\n", "os.environ[\"GH_TOKEN\"] = userdata.get(COLAB_SECRET_NAME)\n", "print(\"Token loaded.\")" ] }, { "cell_type": "markdown", "metadata": { "id": "0LbSu_p2jYfe" }, "source": [ "# Model Building" ] }, { "cell_type": "markdown", "metadata": { "id": "9DtS3gNDjBbR" }, "source": [ "## Data Registration" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "id": "9R55s4cdrqYW" }, "outputs": [], "source": [ "os.makedirs(\"week_3_mls/data\", exist_ok=True)" ] }, { "cell_type": "markdown", "metadata": { "id": "WxXiD9ZXxodF" }, "source": [ "Once the **data** folder created after executing the above cell, please upload the **machine-failure-prediction.csv** in to the folder" ] }, { "cell_type": "code", "execution_count": 114, "metadata": { "id": "oAyK_beOrqYW" }, "outputs": [], "source": [ "# Create a folder for storing the model building files\n", "os.makedirs(\"/content/week_3_mls/model_building\", exist_ok=True)" ] }, { "cell_type": "markdown", "metadata": { "id": "OzqvKufQrqYW" }, "source": [ "1. **Imports Libraries**: Brings in `pandas` to read and validate the dataset.\n", "2. **Loads the Raw Dataset**: Reads the CSV that was uploaded into `week_3_mls/data/`.\n", "3. **Validates Columns**: Checks that all the expected columns are present before treating the dataset as \"registered\".\n", "4. **Reports a Summary**: Prints the row/column counts and the class balance of the target column.\n", "\n", "This keeps \"registration\" lightweight and file-based: the CSV lives in the GitHub repo itself, so there's no external dataset store to manage." ] }, { "cell_type": "code", "execution_count": 149, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "BDU0DQGErqYW", "outputId": "6be8224d-d088-479f-e5d1-d615cd94dcba" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Dataset registered successfully.\n", "Rows: 10000\n", "Columns: 14\n", "Columns: ['UDI', 'Product ID', 'Type', 'Air temperature [K]', 'Process temperature [K]', 'Rotational speed [rpm]', 'Torque [Nm]', 'Tool wear [min]', 'Machine failure', 'TWF', 'HDF', 'PWF', 'OSF', 'RNF']\n", "\n", "Target distribution:\n", "Machine failure\n", "0 9661\n", "1 339\n", "Name: count, dtype: int64\n", "\n", "Dataset saved at: /content/week_3_mls/data/machine-failure-prediction.csv\n" ] } ], "source": [ "from pathlib import Path\n", "import pandas as pd\n", "\n", "# -----------------------------\n", "# Paths\n", "# -----------------------------\n", "# BASE_DIR = Path(__file__).resolve().parent\n", "# In Colab, __file__ is not defined. We use an explicit path based on the\n", "# directory where this script (data_register.py) would be written.\n", "BASE_DIR = Path(\"/content/week_3_mls/model_building\")\n", "DATA_PATH = BASE_DIR.parent / \"data\" / \"machine-failure-prediction.csv\"\n", "\n", "# -----------------------------\n", "# Load dataset\n", "# -----------------------------\n", "if not DATA_PATH.exists():\n", " raise FileNotFoundError(f\"Dataset not found at: {DATA_PATH}\")\n", "\n", "df = pd.read_csv(DATA_PATH)\n", "df.columns = df.columns.str.strip()\n", "\n", "# -----------------------------\n", "# Validate columns\n", "# -----------------------------\n", "expected_columns = [\n", " \"UDI\", \"Product ID\", \"Type\", \"Air temperature [K]\",\n", " \"Process temperature [K]\", \"Rotational speed [rpm]\",\n", " \"Torque [Nm]\", \"Tool wear [min]\", \"Machine failure\",\n", " \"TWF\", \"HDF\", \"PWF\", \"OSF\", \"RNF\"\n", "]\n", "\n", "missing = [col for col in expected_columns if col not in df.columns]\n", "if missing:\n", " raise ValueError(f\"Dataset is missing expected columns: {missing}\")\n", "\n", "# -----------------------------\n", "# Basic info\n", "# -----------------------------\n", "print(\"Dataset registered successfully.\")\n", "print(f\"Rows: {df.shape[0]}\")\n", "print(f\"Columns: {df.shape[1]}\")\n", "print(\"Columns:\", df.columns.tolist())\n", "print(\"\\nTarget distribution:\")\n", "print(df[\"Machine failure\"].value_counts())\n", "\n", "# -----------------------------\n", "# Save cleaned dataset back\n", "# -----------------------------\n", "DATA_PATH.parent.mkdir(parents=True, exist_ok=True)\n", "df.to_csv(DATA_PATH, index=False)\n", "\n", "print(f\"\\nDataset saved at: {DATA_PATH}\")" ] }, { "cell_type": "code", "source": [ "df.columns" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "1Gu5l832tlnw", "outputId": "72f9c3e4-74a5-444c-d75d-15c02f4684aa" }, "execution_count": 116, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "Index(['UDI', 'Product ID', 'Type', 'Air temperature [K]',\n", " 'Process temperature [K]', 'Rotational speed [rpm]', 'Torque [Nm]',\n", " 'Tool wear [min]', 'Machine failure', 'TWF', 'HDF', 'PWF', 'OSF',\n", " 'RNF'],\n", " dtype='object')" ] }, "metadata": {}, "execution_count": 116 } ] }, { "cell_type": "code", "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('/content/week_3_mls/data/machine-failure-prediction.csv')\n", "df.columns = df.columns.str.strip() # Strip whitespace from column names\n", "print(\"DataFrame Columns after stripping whitespace:\", df.columns.tolist())\n", "\n", "# Validate that the expected columns are present before registering it\n", "expected_columns = [\n", " \"UDI\", \"Type\", \"Air temperature [K]\", \"Process temperature [K]\",\n", " \"Rotational speed [rpm]\", \"Torque [Nm]\", \"Tool wear [min]\", \"Machine failure\"\n", "]\n", "missing = [c for c in expected_columns if c not in df.columns]\n", "if missing:\n", " raise ValueError(f\"Dataset is missing expected columns: {missing}\")\n", "\n", "print(\"Dataset registered successfully.\")\n", "print(f\"Rows: {df.shape[0]}, Columns: {df.shape[1]}\")\n", "print(\"Columns:\", list(df.columns))\n", "print(\"Failure distribution:\")\n", "print(df[\"Machine failure\"].value_counts())" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "yQabtyHnstre", "outputId": "a49ff5e4-159a-4152-f371-b151c9b62606" }, "execution_count": 117, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "DataFrame Columns after stripping whitespace: ['UDI', 'Product ID', 'Type', 'Air temperature [K]', 'Process temperature [K]', 'Rotational speed [rpm]', 'Torque [Nm]', 'Tool wear [min]', 'Machine failure', 'TWF', 'HDF', 'PWF', 'OSF', 'RNF']\n", "Dataset registered successfully.\n", "Rows: 10000, Columns: 14\n", "Columns: ['UDI', 'Product ID', 'Type', 'Air temperature [K]', 'Process temperature [K]', 'Rotational speed [rpm]', 'Torque [Nm]', 'Tool wear [min]', 'Machine failure', 'TWF', 'HDF', 'PWF', 'OSF', 'RNF']\n", "Failure distribution:\n", "Machine failure\n", "0 9661\n", "1 339\n", "Name: count, dtype: int64\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "hh2TjRG5WJ4Z" }, "source": [ "## Data Preparation" ] }, { "cell_type": "markdown", "metadata": { "id": "kHWdLAZCrqYW" }, "source": [ "1. **Imports Necessary Libraries**: Brings in `pandas` and `train_test_split`.\n", "2. **Dataset Loading**: Reads the registered CSV from `week_3_mls/data/`.\n", "3. **Data Preparation**: Drops the `UDI` identifier column and splits features/target into train and test sets (stratified on `Failure` to preserve the class imbalance).\n", "4. **Saving Prepared Data**: Writes `Xtrain.csv`, `Xtest.csv`, `ytrain.csv`, and `ytest.csv` to disk, which the GitHub Actions workflow passes to the next job as an artifact.\n", "\n", "**Note:** `Type` is kept as raw H/L/M strings here. It's one-hot-encoded later, inside the model pipeline, not label-encoded, so training and the deployed app stay consistent." ] }, { "cell_type": "code", "execution_count": 120, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "6mwk_9rsrqYW", "outputId": "cfa47120-8574-4bc2-8215-2b34613013bc" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Data preparation completed successfully.\n", "Saved files: Xtrain.csv, Xtest.csv, ytrain.csv, ytest.csv\n" ] } ], "source": [ "from pathlib import Path\n", "import pandas as pd\n", "from sklearn.model_selection import train_test_split\n", "\n", "# Replace Path(__file__).resolve().parent with the absolute path\n", "# where this script would conceptually reside if it were a file.\n", "# In the context of writing to 'week_3_mls/model_building/prep.py',\n", "# its base directory is '/content/week_3_mls/model_building'.\n", "BASE_DIR = Path(\"/content/week_3_mls/model_building\")\n", "DATA_PATH = BASE_DIR.parent / \"data\" / \"machine-failure-prediction.csv\"\n", "\n", "if not DATA_PATH.exists():\n", " raise FileNotFoundError(f\"Dataset not found at: {DATA_PATH}\")\n", "\n", "df = pd.read_csv(DATA_PATH)\n", "df.columns = df.columns.str.strip()\n", "\n", "TARGET_COL = \"Machine failure\"\n", "if TARGET_COL not in df.columns:\n", " raise ValueError(\n", " f\"Target column '{TARGET_COL}' not found. Available columns: {df.columns.tolist()}\"\n", " )\n", "\n", "X = df.drop(columns=[TARGET_COL])\n", "y = df[TARGET_COL]\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.2, random_state=42, stratify=y\n", ")\n", "\n", "X_train.to_csv(\"Xtrain.csv\", index=False)\n", "X_test.to_csv(\"Xtest.csv\", index=False)\n", "y_train.to_csv(\"ytrain.csv\", index=False)\n", "y_test.to_csv(\"ytest.csv\", index=False)\n", "\n", "print(\"Data preparation completed successfully.\")\n", "print(\"Saved files: Xtrain.csv, Xtest.csv, ytrain.csv, ytest.csv\")" ] }, { "cell_type": "code", "source": [ "df" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 424 }, "id": "c53UXntcsGqD", "outputId": "84c89c6a-a357-4392-ac41-91e04f88b3bf" }, "execution_count": 121, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " UDI Product ID Type Air temperature [K] Process temperature [K] \\\n", "0 1 M14860 M 298.1 308.6 \n", "1 2 L47181 L 298.2 308.7 \n", "2 3 L47182 L 298.1 308.5 \n", "3 4 L47183 L 298.2 308.6 \n", "4 5 L47184 L 298.2 308.7 \n", "... ... ... ... ... ... \n", "9995 9996 M24855 M 298.8 308.4 \n", "9996 9997 H39410 H 298.9 308.4 \n", "9997 9998 M24857 M 299.0 308.6 \n", "9998 9999 H39412 H 299.0 308.7 \n", "9999 10000 M24859 M 299.0 308.7 \n", "\n", " Rotational speed [rpm] Torque [Nm] Tool wear [min] Machine failure \\\n", "0 1551 42.8 0 0 \n", "1 1408 46.3 3 0 \n", "2 1498 49.4 5 0 \n", "3 1433 39.5 7 0 \n", "4 1408 40.0 9 0 \n", "... ... ... ... ... \n", "9995 1604 29.5 14 0 \n", "9996 1632 31.8 17 0 \n", "9997 1645 33.4 22 0 \n", "9998 1408 48.5 25 0 \n", "9999 1500 40.2 30 0 \n", "\n", " TWF HDF PWF OSF RNF \n", "0 0 0 0 0 0 \n", "1 0 0 0 0 0 \n", "2 0 0 0 0 0 \n", "3 0 0 0 0 0 \n", "4 0 0 0 0 0 \n", "... ... ... ... ... ... \n", "9995 0 0 0 0 0 \n", "9996 0 0 0 0 0 \n", "9997 0 0 0 0 0 \n", "9998 0 0 0 0 0 \n", "9999 0 0 0 0 0 \n", "\n", "[10000 rows x 14 columns]" ], "text/html": [ "\n", "
| \n", " | UDI | \n", "Product ID | \n", "Type | \n", "Air temperature [K] | \n", "Process temperature [K] | \n", "Rotational speed [rpm] | \n", "Torque [Nm] | \n", "Tool wear [min] | \n", "Machine failure | \n", "TWF | \n", "HDF | \n", "PWF | \n", "OSF | \n", "RNF | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "1 | \n", "M14860 | \n", "M | \n", "298.1 | \n", "308.6 | \n", "1551 | \n", "42.8 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 1 | \n", "2 | \n", "L47181 | \n", "L | \n", "298.2 | \n", "308.7 | \n", "1408 | \n", "46.3 | \n", "3 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 2 | \n", "3 | \n", "L47182 | \n", "L | \n", "298.1 | \n", "308.5 | \n", "1498 | \n", "49.4 | \n", "5 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 3 | \n", "4 | \n", "L47183 | \n", "L | \n", "298.2 | \n", "308.6 | \n", "1433 | \n", "39.5 | \n", "7 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 4 | \n", "5 | \n", "L47184 | \n", "L | \n", "298.2 | \n", "308.7 | \n", "1408 | \n", "40.0 | \n", "9 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| ... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
| 9995 | \n", "9996 | \n", "M24855 | \n", "M | \n", "298.8 | \n", "308.4 | \n", "1604 | \n", "29.5 | \n", "14 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 9996 | \n", "9997 | \n", "H39410 | \n", "H | \n", "298.9 | \n", "308.4 | \n", "1632 | \n", "31.8 | \n", "17 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 9997 | \n", "9998 | \n", "M24857 | \n", "M | \n", "299.0 | \n", "308.6 | \n", "1645 | \n", "33.4 | \n", "22 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 9998 | \n", "9999 | \n", "H39412 | \n", "H | \n", "299.0 | \n", "308.7 | \n", "1408 | \n", "48.5 | \n", "25 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
| 9999 | \n", "10000 | \n", "M24859 | \n", "M | \n", "299.0 | \n", "308.7 | \n", "1500 | \n", "40.2 | \n", "30 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "0 | \n", "
10000 rows × 14 columns
\n", "