{ "cells": [ { "cell_type": "markdown", "id": "b453c150", "metadata": {}, "source": [ "# 03 - Feature Engineering\n", "\n", "Build the modeling dataset from `jira_issues_cleaned.csv`. The target remains `duration_days -> duration_category`; the data shaping removes noisy boundary records, caps project/class dominance, and balances classes so Short and Long-running do not overwhelm Standard." ] }, { "cell_type": "markdown", "id": "63174149", "metadata": {}, "source": [ "## 03-01 Import feature engineering tools\n", "\n", "Load the lightweight utilities used in this notebook. pathlib handles project-relative paths, and pandas handles feature creation, filtering, and export." ] }, { "cell_type": "code", "execution_count": 1, "id": "8ca9b2a2", "metadata": { "execution": { "iopub.execute_input": "2026-07-07T04:32:06.739611Z", "iopub.status.busy": "2026-07-07T04:32:06.739172Z", "iopub.status.idle": "2026-07-07T04:32:07.260518Z", "shell.execute_reply": "2026-07-07T04:32:07.259640Z" } }, "outputs": [], "source": [ "from pathlib import Path\n", "import pandas as pd" ] }, { "cell_type": "markdown", "id": "a5fae254", "metadata": {}, "source": [ "## 03-02 Load cleaned source data\n", "\n", "Read the cleaned dataset from the previous notebook, copy it for modeling work, and convert timestamp fields back into datetime values." ] }, { "cell_type": "code", "execution_count": 2, "id": "59616763", "metadata": { "execution": { "iopub.execute_input": "2026-07-07T04:32:07.263709Z", "iopub.status.busy": "2026-07-07T04:32:07.263404Z", "iopub.status.idle": "2026-07-07T04:32:22.562481Z", "shell.execute_reply": "2026-07-07T04:32:22.561450Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Cleaned source rows: 937,203\n", "Cleaned source columns: 17\n" ] } ], "source": [ "PROJECT_ROOT = Path.cwd().parent if Path.cwd().name == \"notebooks\" else Path.cwd()\n", "INPUT_PATH = PROJECT_ROOT / \"data\" / \"processed\" / \"jira_issues_cleaned.csv\"\n", "OUTPUT_DIR = PROJECT_ROOT / \"data\" / \"processed\"\n", "\n", "jira_df = pd.read_csv(INPUT_PATH)\n", "task_df = jira_df.copy()\n", "\n", "for column in [\"created\", \"resolutiondate\"]:\n", " task_df[column] = pd.to_datetime(task_df[column], errors=\"coerce\")\n", "\n", "print(f\"Cleaned source rows: {task_df.shape[0]:,}\")\n", "print(f\"Cleaned source columns: {task_df.shape[1]:,}\")" ] }, { "cell_type": "markdown", "id": "95078842", "metadata": {}, "source": [ "## 03-03 Create and filter duration values\n", "\n", "Calculate duration_days from the created and resolution timestamps, then keep durations within the usable modeling range." ] }, { "cell_type": "code", "execution_count": 3, "id": "8d171b36", "metadata": { "execution": { "iopub.execute_input": "2026-07-07T04:32:22.565691Z", "iopub.status.busy": "2026-07-07T04:32:22.565393Z", "iopub.status.idle": "2026-07-07T04:32:22.807284Z", "shell.execute_reply": "2026-07-07T04:32:22.806127Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Rows before duration filtering: 937,203" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Rows after duration filtering: 578,204\n" ] }, { "data": { "text/plain": [ "count 578204.000000\n", "mean 15.878555\n", "std 21.139684\n", "min 0.083333\n", "25% 1.239546\n", "50% 6.050874\n", "75% 21.893226\n", "90% 49.805259\n", "95% 66.695716\n", "99% 84.510098\n", "max 89.999618\n", "Name: duration_days, dtype: float64" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "task_df[\"duration_days\"] = (\n", " task_df[\"resolutiondate\"] - task_df[\"created\"]\n", ").dt.total_seconds() / (60 * 60 * 24)\n", "\n", "rows_before = len(task_df)\n", "task_df = task_df[\n", " task_df[\"duration_days\"].notna()\n", " & (task_df[\"duration_days\"] >= (2 / 24))\n", " & (task_df[\"duration_days\"] <= 90)\n", "].copy()\n", "\n", "print(f\"Rows before duration filtering: {rows_before:,}\")\n", "print(f\"Rows after duration filtering: {len(task_df):,}\")\n", "task_df[\"duration_days\"].describe(percentiles=[0.25, 0.5, 0.75, 0.9, 0.95, 0.99])" ] }, { "cell_type": "markdown", "id": "fc1b3957", "metadata": {}, "source": [ "## 03-04 Assign duration categories\n", "\n", "Convert numeric durations into the three target classes used by the classifier and inspect the initial class balance." ] }, { "cell_type": "code", "execution_count": 4, "id": "1699f4f2", "metadata": { "execution": { "iopub.execute_input": "2026-07-07T04:32:22.809991Z", "iopub.status.busy": "2026-07-07T04:32:22.809743Z", "iopub.status.idle": "2026-07-07T04:32:23.018051Z", "shell.execute_reply": "2026-07-07T04:32:23.016980Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countpercent
duration_category
Short21327436.89
Standard17983131.10
Long-running18509932.01
\n", "
" ], "text/plain": [ " count percent\n", "duration_category \n", "Short 213274 36.89\n", "Standard 179831 31.10\n", "Long-running 185099 32.01" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def duration_category(days):\n", " if days <= 3:\n", " return \"Short\"\n", " if days <= 15:\n", " return \"Standard\"\n", " return \"Long-running\"\n", "\n", "\n", "duration_order = [\"Short\", \"Standard\", \"Long-running\"]\n", "task_df[\"duration_category\"] = task_df[\"duration_days\"].apply(duration_category)\n", "\n", "class_summary = pd.DataFrame({\n", " \"count\": task_df[\"duration_category\"].value_counts().reindex(duration_order),\n", " \"percent\": task_df[\"duration_category\"]\n", " .value_counts(normalize=True)\n", " .reindex(duration_order)\n", " .mul(100)\n", " .round(2),\n", "})\n", "\n", "class_summary" ] }, { "cell_type": "markdown", "id": "8cc99676", "metadata": {}, "source": [ "## 03-05 Keep full duration class ranges\n", "\n", "Keep valid examples across the full class ranges so the final balanced dataset has enough rows for model training." ] }, { "cell_type": "code", "execution_count": 5, "id": "7521e172", "metadata": { "execution": { "iopub.execute_input": "2026-07-07T04:32:23.020445Z", "iopub.status.busy": "2026-07-07T04:32:23.020179Z", "iopub.status.idle": "2026-07-07T04:32:23.324884Z", "shell.execute_reply": "2026-07-07T04:32:23.323530Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Rows removed outside full duration windows: 0\n", "Rows after duration-window check: 578,204\n" ] } ], "source": [ "# Keep the full valid range for each duration class.\n", "# The earlier duration filter already removed invalid and extreme examples.\n", "duration_window_mask = (\n", " (task_df[\"duration_category\"].eq(\"Short\") & (task_df[\"duration_days\"] <= 3))\n", " | (\n", " task_df[\"duration_category\"].eq(\"Standard\")\n", " & task_df[\"duration_days\"].between(3, 15, inclusive=\"both\")\n", " )\n", " | (\n", " task_df[\"duration_category\"].eq(\"Long-running\")\n", " & (task_df[\"duration_days\"] >= 15)\n", " )\n", ")\n", "\n", "rows_before = len(task_df)\n", "task_df = task_df.loc[duration_window_mask].copy()\n", "\n", "print(f\"Rows removed outside full duration windows: {rows_before - len(task_df):,}\")\n", "print(f\"Rows after duration-window check: {len(task_df):,}\")" ] }, { "cell_type": "markdown", "id": "648f5056", "metadata": {}, "source": [ "## 03-06 Keep consistent project and issue groups\n", "\n", "Retain project and issue-type combinations with enough history and a clear duration-class signal, reducing mixed groups that add label noise." ] }, { "cell_type": "code", "execution_count": 6, "id": "bd2f9871", "metadata": { "execution": { "iopub.execute_input": "2026-07-07T04:32:23.327940Z", "iopub.status.busy": "2026-07-07T04:32:23.327631Z", "iopub.status.idle": "2026-07-07T04:32:23.703605Z", "shell.execute_reply": "2026-07-07T04:32:23.702431Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Rows removed from low-signal project/issue groups: 319,482\n", "Rows after consistency filtering: 258,722\n" ] }, { "data": { "text/plain": [ "duration_category\n", "Short 139682\n", "Standard 37408\n", "Long-running 81632\n", "Name: count, dtype: int64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Keep project/issue-type combinations where duration class has historical signal.\n", "# This removes mixed groups that make Standard especially noisy, while retaining all classes.\n", "group_columns = [\"project_key\", \"issuetype_name\"]\n", "minimum_group_size = 25\n", "minimum_category_share = 0.35\n", "\n", "group_counts = (\n", " task_df\n", " .groupby(group_columns + [\"duration_category\"], observed=True)\n", " .size()\n", " .rename(\"category_count\")\n", " .reset_index()\n", ")\n", "group_totals = (\n", " group_counts\n", " .groupby(group_columns, observed=True)[\"category_count\"]\n", " .sum()\n", " .rename(\"group_count\")\n", " .reset_index()\n", ")\n", "group_counts = group_counts.merge(group_totals, on=group_columns)\n", "group_counts[\"category_share\"] = group_counts[\"category_count\"] / group_counts[\"group_count\"]\n", "\n", "consistent_groups = group_counts.loc[\n", " (group_counts[\"group_count\"] >= minimum_group_size)\n", " & (group_counts[\"category_share\"] >= minimum_category_share),\n", " group_columns + [\"duration_category\"],\n", "]\n", "\n", "rows_before = len(task_df)\n", "task_df = task_df.merge(\n", " consistent_groups,\n", " on=group_columns + [\"duration_category\"],\n", " how=\"inner\",\n", ")\n", "\n", "print(f\"Rows removed from low-signal project/issue groups: {rows_before - len(task_df):,}\")\n", "print(f\"Rows after consistency filtering: {len(task_df):,}\")\n", "task_df[\"duration_category\"].value_counts().reindex(duration_order)" ] }, { "cell_type": "markdown", "id": "455284e3", "metadata": {}, "source": [ "## 03-07 Cap project-class dominance\n", "\n", "Limit the number of rows from each project and duration class while keeping enough examples to exceed 100,000 final rows." ] }, { "cell_type": "code", "execution_count": 7, "id": "47fbdbb2", "metadata": { "execution": { "iopub.execute_input": "2026-07-07T04:32:23.707591Z", "iopub.status.busy": "2026-07-07T04:32:23.707287Z", "iopub.status.idle": "2026-07-07T04:32:24.590071Z", "shell.execute_reply": "2026-07-07T04:32:24.588859Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Rows after project/class cap: 218,629\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\Omar\\AppData\\Local\\Temp\\ipykernel_13728\\3248045568.py:7: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.\n", " .apply(\n" ] }, { "data": { "text/plain": [ "duration_category\n", "Short 107867\n", "Standard 33857\n", "Long-running 76905\n", "Name: count, dtype: int64" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Prevent a few large projects from dominating the classifier.\n", "max_rows_per_project_class = 2_500\n", "\n", "task_df = (\n", " task_df\n", " .groupby([\"project_key\", \"duration_category\"], group_keys=False, observed=True)\n", " .apply(\n", " lambda group: group.sample(\n", " n=min(len(group), max_rows_per_project_class),\n", " random_state=42,\n", " )\n", " )\n", " .reset_index(drop=True)\n", ")\n", "\n", "\n", "print(f\"Rows after project/class cap: {len(task_df):,}\")\n", "task_df[\"duration_category\"].value_counts().reindex(duration_order)" ] }, { "cell_type": "markdown", "id": "bcd1e0b1", "metadata": {}, "source": [ "## 03-08 Balance target classes\n", "\n", "Downsample each duration category to the smallest available class size, keeping class balance without duplicating rows." ] }, { "cell_type": "code", "execution_count": 8, "id": "9c350a81", "metadata": { "execution": { "iopub.execute_input": "2026-07-07T04:32:24.593258Z", "iopub.status.busy": "2026-07-07T04:32:24.592946Z", "iopub.status.idle": "2026-07-07T04:32:24.815802Z", "shell.execute_reply": "2026-07-07T04:32:24.814403Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Target rows per class: 33,857" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
countpercent
duration_category
Short3385733.33
Standard3385733.33
Long-running3385733.33
\n", "
" ], "text/plain": [ " count percent\n", "duration_category \n", "Short 33857 33.33\n", "Standard 33857 33.33\n", "Long-running 33857 33.33" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Balance classes without duplicating rows. Standard is often the hardest class, so the\n", "# final class size is anchored to the smallest available class after cleanup.\n", "class_counts = task_df[\"duration_category\"].value_counts()\n", "target_class_size = int(class_counts.min())\n", "\n", "balanced_parts = []\n", "for category in duration_order:\n", " category_df = task_df.loc[task_df[\"duration_category\"].eq(category)]\n", " balanced_parts.append(\n", " category_df.sample(n=target_class_size, random_state=42)\n", " )\n", "\n", "task_df = (\n", " pd.concat(balanced_parts, ignore_index=True)\n", " .sample(frac=1, random_state=42)\n", " .reset_index(drop=True)\n", ")\n", "\n", "balanced_summary = pd.DataFrame({\n", " \"count\": task_df[\"duration_category\"].value_counts().reindex(duration_order),\n", " \"percent\": task_df[\"duration_category\"]\n", " .value_counts(normalize=True)\n", " .reindex(duration_order)\n", " .mul(100)\n", " .round(2),\n", "})\n", "\n", "print(f\"Target rows per class: {target_class_size:,}\")\n", "balanced_summary" ] }, { "cell_type": "markdown", "id": "65c93b7f", "metadata": {}, "source": [ "## 03-09 Save final modeling dataset\n", "\n", "Add simple calendar features, select the final model columns, and save the full and sample datasets for training and app use." ] }, { "cell_type": "code", "execution_count": 9, "id": "18baf073", "metadata": { "execution": { "iopub.execute_input": "2026-07-07T04:32:24.818308Z", "iopub.status.busy": "2026-07-07T04:32:24.818071Z", "iopub.status.idle": "2026-07-07T04:32:26.846015Z", "shell.execute_reply": "2026-07-07T04:32:26.844997Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved final cleaned CSV file to: C:\\Users\\Omar\\Desktop\\task-ml\\Task-Duration-Classifier\\data\\processed\\final_cleaned.csv\n", "Saved sample CSV file to: C:\\Users\\Omar\\Desktop\\task-ml\\Task-Duration-Classifier\\data\\processed\\final_cleaned_sample.csv\n", "Final modeling rows: 101,571\n", "Final modeling columns: 18\n" ] } ], "source": [ "task_df[\"created_year\"] = task_df[\"created\"].dt.year\n", "task_df[\"created_month\"] = task_df[\"created\"].dt.month\n", "\n", "final_cleaned_columns = [\n", " \"summary\",\n", " \"description\",\n", " \"priority_name\",\n", " \"issuetype_name\",\n", " \"project_key\",\n", " \"project_category_name\",\n", " \"summary_char_count\",\n", " \"summary_word_count\",\n", " \"description_char_count\",\n", " \"description_word_count\",\n", " \"has_description\",\n", " \"labels_count\",\n", " \"has_assignee\",\n", " \"votes_votes\",\n", " \"watches_watch_count\",\n", " \"created_year\",\n", " \"created_month\",\n", " \"duration_category\",\n", "]\n", "\n", "final_cleaned_df = task_df[final_cleaned_columns].copy()\n", "\n", "final_cleaned_path = OUTPUT_DIR / \"final_cleaned.csv\"\n", "sample_path = OUTPUT_DIR / \"final_cleaned_sample.csv\"\n", "\n", "final_cleaned_df.to_csv(final_cleaned_path, index=False)\n", "final_cleaned_df.sample(n=min(100, len(final_cleaned_df)), random_state=42).to_csv(sample_path, index=False)\n", "\n", "print(f\"Saved final cleaned CSV file to: {final_cleaned_path}\")\n", "print(f\"Saved sample CSV file to: {sample_path}\")\n", "print(f\"Final modeling rows: {final_cleaned_df.shape[0]:,}\")\n", "print(f\"Final modeling columns: {final_cleaned_df.shape[1]:,}\")" ] } ], "metadata": { "kernelspec": { "display_name": "base", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" } }, "nbformat": 4, "nbformat_minor": 5 }