{ "cells": [ { "cell_type": "markdown", "id": "aeb4cd48", "metadata": {}, "source": [ "# **Analyse du drift & de la performance**\n", "\n", "Ce notebook a pour objectif :\n", "\n", "- **d’évaluer le data drift** entre :\n", " - les données de référence (jeu d’entraînement ou échantillon représentatif)\n", " - les données de production issues des logs de l’API\n", "\n", " Le data drift permet de détecter si les distributions des variables évoluent dans le temps. \n", " Ces changements peuvent dégrader les performances du modèle, notamment lorsque les distributions de production s’éloignent de celles observées lors de l’entraînement.\n", "\n", " Pour cela, nous utilisons la librairie **Evidently AI**, qui permet d’analyser le drift **colonne par colonne**, après avoir soigneusement harmonisé les colonnes entre les deux jeux de données (types, valeurs manquantes, colonnes constantes, etc.).\n", "\n", "- **d’évaluer la performance du pipeline de prédiction**, non pas uniquement au niveau du modèle, mais surtout au niveau du **preprocessing**, afin d’identifier les véritables axes d’optimisation.\n", "\n", " En effet, dans un contexte MLOps, la latence d’un modèle en production dépend autant du modèle que du pipeline de transformation. \n", " Nous mesurons donc :\n", " - le temps de preprocessing,\n", " - le temps de prédiction du modèle,\n", " - la différence entre appels unitaires et prédiction en batch,\n", " - et nous analysons les résultats du profiling pour identifier les goulots d’étranglement.\n", "\n" ] }, { "cell_type": "code", "execution_count": 23, "id": "ee429e61", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "import cProfile\n", "import pstats\n", "import joblib\n", "import time\n", "\n", "from evidently.report import Report\n", "from evidently.metrics import ColumnDriftMetric" ] }, { "cell_type": "markdown", "id": "1149740f", "metadata": {}, "source": [ "## **Analyse du Data Drift avec Evidently**" ] }, { "cell_type": "code", "execution_count": 24, "id": "9a27a53d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((500, 327), (213, 406), (213, 406))" ] }, "execution_count": 24, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Chargement du pipeline\n", "pipe = joblib.load(\"../BestModel/pipeline_complet.joblib\")\n", "\n", "# Données de référence (version simplifiée)\n", "df_reference_raw = joblib.load(\"../data/app_test_small.joblib\")\n", "\n", "# Données de production (logs unitaires)\n", "df_production_raw = pd.read_parquet(\"../logs/predictions_log.parquet\")\n", "\n", "# Données de production (logs batch)\n", "df_batch_raw = pd.read_parquet(\"../logs/predictions_log.parquet\")\n", "\n", "df_reference_raw.shape, df_production_raw.shape, df_batch_raw.shape" ] }, { "cell_type": "markdown", "id": "276796a6", "metadata": {}, "source": [ "### Sélection des colonnes communes entre référence production" ] }, { "cell_type": "code", "execution_count": 25, "id": "a66434ea", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['num__SK_ID_CURR', 'num__BUREAU_SK_ID_BUREAU_max',\n", " 'num__POS_SK_DPD_min', 'num__CC_SK_DPD_DEF_max',\n", " 'num__PREV_NAME_TYPE_SUITE_Family', 'num__PREV_DAYS_FIRST_DRAWING_sum',\n", " 'num__PREV_NAME_GOODS_CATEGORY_Other', 'num__DAYS_LAST_PHONE_CHANGE',\n", " 'num__CC_CNT_DRAWINGS_OTHER_CURRENT_mean',\n", " 'num__PREV_DAYS_FIRST_DRAWING_max',\n", " ...\n", " 'bool__FLAG_DOCUMENT_12', 'bool__FLAG_DOCUMENT_13',\n", " 'bool__FLAG_DOCUMENT_14', 'bool__FLAG_DOCUMENT_15',\n", " 'bool__FLAG_DOCUMENT_16', 'bool__FLAG_DOCUMENT_17',\n", " 'bool__FLAG_DOCUMENT_18', 'bool__FLAG_DOCUMENT_19',\n", " 'bool__FLAG_DOCUMENT_20', 'bool__FLAG_DOCUMENT_21'],\n", " dtype='object', length=382)" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Transformer les colonnes de référence\n", "df_reference_transformed = pipe[:-1].transform(df_reference_raw)\n", "df_reference_transformed = pd.DataFrame(df_reference_transformed, columns=pipe[:-1].get_feature_names_out())\n", "df_reference_transformed.columns\n" ] }, { "cell_type": "code", "execution_count": 26, "id": "e02ce960", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Index(['num__SK_ID_CURR', 'num__BUREAU_SK_ID_BUREAU_max',\n", " 'num__POS_SK_DPD_min', 'num__CC_SK_DPD_DEF_max',\n", " 'num__PREV_NAME_TYPE_SUITE_Family', 'num__PREV_DAYS_FIRST_DRAWING_sum',\n", " 'num__PREV_NAME_GOODS_CATEGORY_Other', 'num__DAYS_LAST_PHONE_CHANGE',\n", " 'num__CC_CNT_DRAWINGS_OTHER_CURRENT_mean',\n", " 'num__PREV_DAYS_FIRST_DRAWING_max',\n", " ...\n", " 'ram_percent', 'system_load', 'num_threads', 'method', 'status',\n", " 'error_message', 'latency_ms', 'request_size_bytes',\n", " 'response_size_bytes', 'batch_size_y'],\n", " dtype='object', length=406)" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_production_raw.columns" ] }, { "cell_type": "code", "execution_count": 27, "id": "d9282066", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "382" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "common_cols = df_reference_transformed.columns.intersection(df_production_raw.columns)\n", "len(common_cols)" ] }, { "cell_type": "code", "execution_count": 28, "id": "eb15c29d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "((500, 382), (213, 382))" ] }, "execution_count": 28, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_reference = df_reference_transformed[common_cols].copy()\n", "df_production = df_production_raw[common_cols].copy()\n", "\n", "df_reference.shape, df_production.shape" ] }, { "cell_type": "markdown", "id": "3b53669d", "metadata": {}, "source": [ "### Harmonisation des colonnes" ] }, { "cell_type": "code", "execution_count": 29, "id": "a0aee959", "metadata": {}, "outputs": [], "source": [ "for col in common_cols:\n", " # bool → int\n", " if df_reference[col].dtype == bool:\n", " df_reference[col] = df_reference[col].astype(int)\n", "\n", " if df_production[col].dtype == bool:\n", " df_production[col] = df_production[col].astype(int)\n", "\n", " ref_num = pd.api.types.is_numeric_dtype(df_reference[col])\n", " prod_num = pd.api.types.is_numeric_dtype(df_production[col])\n", "\n", " # Les deux numériques\n", " if ref_num and prod_num:\n", " df_reference[col] = pd.to_numeric(df_reference[col], errors=\"coerce\").astype(\"float64\")\n", " df_production[col] = pd.to_numeric(df_production[col], errors=\"coerce\").astype(\"float64\")\n", "\n", " # Sinon → string\n", " else:\n", " df_reference[col] = df_reference[col].fillna(\"MISSING\").astype(str)\n", " df_production[col] = df_production[col].fillna(\"MISSING\").astype(str)\n" ] }, { "cell_type": "markdown", "id": "ec1bec15", "metadata": {}, "source": [ "### Suppression des colonnes vides et constantes" ] }, { "cell_type": "code", "execution_count": 30, "id": "078e82b0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "275\n", "213\n", "500\n" ] } ], "source": [ "valid_cols = [\n", " col for col in common_cols\n", " if df_reference[col].notna().sum() > 0 and df_production[col].notna().sum() > 0]\n", "\n", "non_constant_cols = [\n", " col for col in valid_cols\n", " if df_reference[col].nunique(dropna=True) > 1\n", " and df_production[col].nunique(dropna=True) > 1]\n", "\n", "df_reference = df_reference[non_constant_cols]\n", "df_production = df_production[non_constant_cols]\n", "\n", "print(len(non_constant_cols))\n", "print(len(df_production))\n", "print(len(df_reference))\n" ] }, { "cell_type": "markdown", "id": "a9f6fc6c", "metadata": {}, "source": [ "### Drift par colonne" ] }, { "cell_type": "code", "execution_count": 31, "id": "ab0ca561", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "(275, 2)\n" ] } ], "source": [ "drift_results = []\n", "\n", "for col in non_constant_cols:\n", " report = Report(metrics = [ColumnDriftMetric(column_name = col)])\n", "\n", " try:\n", " report.run(reference_data = df_reference[[col]], current_data = df_production[[col]])\n", "\n", " result = report.as_dict()\n", " drift_detected = result[\"metrics\"][0][\"result\"][\"drift_detected\"]\n", "\n", " drift_results.append({\n", " \"colonne\": col,\n", " \"drift_detected\": drift_detected})\n", "\n", " except Exception as e:\n", " drift_results.append({\"colonne\": col, \"drift_detected\": \"ERREUR\", \"exception\": str(e)})\n", "\n", "df_drift = pd.DataFrame(drift_results)\n", "print(df_drift.shape)\n", "report_drift = report\n" ] }, { "cell_type": "markdown", "id": "8ce86333", "metadata": {}, "source": [ "### Résumé" ] }, { "cell_type": "code", "execution_count": 32, "id": "d80c7637", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n", "c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\scipy\\stats\\_stats_py.py:8022: RuntimeWarning: divide by zero encountered in divide\n", " terms = (f_obs_float - f_exp)**2 / f_exp\n" ] } ], "source": [ "from evidently.metrics import DatasetDriftMetric\n", "report = Report(metrics=[DatasetDriftMetric()])\n", "report.run(reference_data=df_reference, current_data=df_production)\n", "\n", "res = report.as_dict()" ] }, { "cell_type": "code", "execution_count": 33, "id": "9da53766", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Colonnes analysées : 275\n", "Colonnes en dérive : 18\n", "Taux de dérive : 6.5%\n" ] } ], "source": [ "dataset_drift = next(\n", " m for m in res[\"metrics\"]\n", " if m[\"metric\"] == \"DatasetDriftMetric\")\n", "\n", "drift_res = dataset_drift[\"result\"]\n", "\n", "nb_cols = drift_res[\"number_of_columns\"]\n", "nb_drift = drift_res[\"number_of_drifted_columns\"]\n", "taux = nb_drift / nb_cols\n", "\n", "print(\"Colonnes analysées :\", nb_cols)\n", "print(\"Colonnes en dérive :\", nb_drift)\n", "print(f\"Taux de dérive : {taux:.1%}\")\n" ] }, { "cell_type": "code", "execution_count": 34, "id": "28838cdc", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
colonnedrift_detected
6num__PREV_PRODUCT_COMBINATION_POS others witho...True
57num__BUREAU_STATUS_4True
60num__PREV_NAME_CASH_LOAN_PURPOSE_Buying a holi...True
66num__PREV_NAME_GOODS_CATEGORY_VehiclesTrue
84num__PREV_NAME_CASH_LOAN_PURPOSE_MedicineTrue
86num__PREV_NAME_CASH_LOAN_PURPOSE_Wedding / gif...True
89num__DAYS_REGISTRATIONTrue
94num__PREV_NAME_GOODS_CATEGORY_JewelryTrue
123num__PREV_CODE_REJECT_REASON_NUNIQUETrue
133num__PREV_CODE_REJECT_REASON_SCOFRTrue
145num__PREV_NAME_CASH_LOAN_PURPOSE_OtherTrue
146num__PREV_NAME_GOODS_CATEGORY_HomewaresTrue
148num__PREV_NAME_SELLER_INDUSTRY_IndustryTrue
150num__PREV_PRODUCT_COMBINATION_POS industry wit...True
194num__PREV_NAME_TYPE_SUITE_Other_BTrue
221card_faible__NAME_EDUCATION_TYPE_Incomplete hi...True
253card_faible__WALLSMATERIAL_MODE_WoodenTrue
259bool__FLAG_WORK_PHONETrue
\n", "
" ], "text/plain": [ " colonne drift_detected\n", "6 num__PREV_PRODUCT_COMBINATION_POS others witho... True\n", "57 num__BUREAU_STATUS_4 True\n", "60 num__PREV_NAME_CASH_LOAN_PURPOSE_Buying a holi... True\n", "66 num__PREV_NAME_GOODS_CATEGORY_Vehicles True\n", "84 num__PREV_NAME_CASH_LOAN_PURPOSE_Medicine True\n", "86 num__PREV_NAME_CASH_LOAN_PURPOSE_Wedding / gif... True\n", "89 num__DAYS_REGISTRATION True\n", "94 num__PREV_NAME_GOODS_CATEGORY_Jewelry True\n", "123 num__PREV_CODE_REJECT_REASON_NUNIQUE True\n", "133 num__PREV_CODE_REJECT_REASON_SCOFR True\n", "145 num__PREV_NAME_CASH_LOAN_PURPOSE_Other True\n", "146 num__PREV_NAME_GOODS_CATEGORY_Homewares True\n", "148 num__PREV_NAME_SELLER_INDUSTRY_Industry True\n", "150 num__PREV_PRODUCT_COMBINATION_POS industry wit... True\n", "194 num__PREV_NAME_TYPE_SUITE_Other_B True\n", "221 card_faible__NAME_EDUCATION_TYPE_Incomplete hi... True\n", "253 card_faible__WALLSMATERIAL_MODE_Wooden True\n", "259 bool__FLAG_WORK_PHONE True" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Colonnes en dérive\n", "df_drift[df_drift[\"drift_detected\"] == True]\n" ] }, { "cell_type": "markdown", "id": "45158001", "metadata": {}, "source": [ "## **Optimisation des performances du modèle**" ] }, { "cell_type": "markdown", "id": "170cd7fe", "metadata": {}, "source": [ "### Apperçu global" ] }, { "cell_type": "code", "execution_count": 35, "id": "f40c42f6", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
num__SK_ID_CURRnum__BUREAU_SK_ID_BUREAU_maxnum__POS_SK_DPD_minnum__CC_SK_DPD_DEF_maxnum__PREV_NAME_TYPE_SUITE_Familynum__PREV_DAYS_FIRST_DRAWING_sumnum__PREV_NAME_GOODS_CATEGORY_Othernum__DAYS_LAST_PHONE_CHANGEnum__CC_CNT_DRAWINGS_OTHER_CURRENT_meannum__PREV_DAYS_FIRST_DRAWING_max...ram_percentsystem_loadnum_threadsmethodstatuserror_messagelatency_msrequest_size_bytesresponse_size_bytesbatch_size_y
0-0.3024290.3260390.249338NaN-0.535936-0.3001650.23674-0.987792NaN0.246419...78.30.021POSTsuccessNone383.74137912798.00.0NaN
1-0.1471320.5539470.249338NaN-0.5359360.7428040.236740.895050NaN0.246419...78.30.021POSTsuccessNone102.71501512775.00.0NaN
2-0.4296380.4486350.249338NaN0.494736-0.8216490.236740.962033NaN0.246419...78.40.021POSTsuccessNone67.81530412875.00.0NaN
30.2462230.4919430.2493380.6915352.5560814.3924880.23674-2.1922741.9975730.246419...78.40.021POSTsuccessNone89.75052812927.00.0NaN
4-1.1696000.6991970.249338NaN-0.535936-0.8216490.236740.744033NaN0.246419...78.30.021POSTsuccessNone68.77660812836.00.0NaN
\n", "

5 rows × 406 columns

\n", "
" ], "text/plain": [ " num__SK_ID_CURR num__BUREAU_SK_ID_BUREAU_max num__POS_SK_DPD_min \\\n", "0 -0.302429 0.326039 0.249338 \n", "1 -0.147132 0.553947 0.249338 \n", "2 -0.429638 0.448635 0.249338 \n", "3 0.246223 0.491943 0.249338 \n", "4 -1.169600 0.699197 0.249338 \n", "\n", " num__CC_SK_DPD_DEF_max num__PREV_NAME_TYPE_SUITE_Family \\\n", "0 NaN -0.535936 \n", "1 NaN -0.535936 \n", "2 NaN 0.494736 \n", "3 0.691535 2.556081 \n", "4 NaN -0.535936 \n", "\n", " num__PREV_DAYS_FIRST_DRAWING_sum num__PREV_NAME_GOODS_CATEGORY_Other \\\n", "0 -0.300165 0.23674 \n", "1 0.742804 0.23674 \n", "2 -0.821649 0.23674 \n", "3 4.392488 0.23674 \n", "4 -0.821649 0.23674 \n", "\n", " num__DAYS_LAST_PHONE_CHANGE num__CC_CNT_DRAWINGS_OTHER_CURRENT_mean \\\n", "0 -0.987792 NaN \n", "1 0.895050 NaN \n", "2 0.962033 NaN \n", "3 -2.192274 1.997573 \n", "4 0.744033 NaN \n", "\n", " num__PREV_DAYS_FIRST_DRAWING_max ... ram_percent system_load \\\n", "0 0.246419 ... 78.3 0.0 \n", "1 0.246419 ... 78.3 0.0 \n", "2 0.246419 ... 78.4 0.0 \n", "3 0.246419 ... 78.4 0.0 \n", "4 0.246419 ... 78.3 0.0 \n", "\n", " num_threads method status error_message latency_ms \\\n", "0 21 POST success None 383.741379 \n", "1 21 POST success None 102.715015 \n", "2 21 POST success None 67.815304 \n", "3 21 POST success None 89.750528 \n", "4 21 POST success None 68.776608 \n", "\n", " request_size_bytes response_size_bytes batch_size_y \n", "0 12798.0 0.0 NaN \n", "1 12775.0 0.0 NaN \n", "2 12875.0 0.0 NaN \n", "3 12927.0 0.0 NaN \n", "4 12836.0 0.0 NaN \n", "\n", "[5 rows x 406 columns]" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_production_raw.head()" ] }, { "cell_type": "code", "execution_count": 36, "id": "29eeb703", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 48.473900\n", "1 53.282800\n", "2 22.520600\n", "3 39.389800\n", "4 23.058600\n", " ... \n", "208 48.540900\n", "209 45.094800\n", "210 45.515200\n", "211 46.950299\n", "212 52.126600\n", "Name: inference_ms_x, Length: 213, dtype: float64" ] }, "execution_count": 36, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Statistiques inférence\n", "df_production_raw['inference_ms_x']" ] }, { "cell_type": "code", "execution_count": 37, "id": "e0f67650", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "count 213.000000\n", "mean 36.709429\n", "std 15.020234\n", "min 21.098300\n", "25% 24.290100\n", "50% 30.452900\n", "75% 46.725800\n", "max 113.723900\n", "Name: inference_ms_x, dtype: float64" ] }, "execution_count": 37, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_production_raw['inference_ms_x'].describe()" ] }, { "cell_type": "code", "execution_count": 38, "id": "2d63c4f6", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 383.741379\n", "1 102.715015\n", "2 67.815304\n", "3 89.750528\n", "4 68.776608\n", " ... \n", "208 307.635784\n", "209 315.336943\n", "210 302.381754\n", "211 309.991360\n", "212 237.794876\n", "Name: latency_ms, Length: 213, dtype: float64" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Statistiques latence\n", "df_production_raw[\"latency_ms\"]" ] }, { "cell_type": "code", "execution_count": 39, "id": "a16d44c9", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "count 213.000000\n", "mean 104.265481\n", "std 90.755837\n", "min 58.841944\n", "25% 70.789099\n", "50% 78.964949\n", "75% 96.737385\n", "max 1024.606943\n", "Name: latency_ms, dtype: float64" ] }, "execution_count": 39, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_production_raw[\"latency_ms\"].describe()" ] }, { "cell_type": "code", "execution_count": 40, "id": "46a770de", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 0.074383\n", "1 0.670045\n", "2 0.092590\n", "3 0.151949\n", "4 0.043955\n", " ... \n", "208 NaN\n", "209 NaN\n", "210 NaN\n", "211 NaN\n", "212 0.162290\n", "Name: score_x, Length: 213, dtype: float64" ] }, "execution_count": 40, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Statistiques score\n", "df_production_raw[\"score_x\"]" ] }, { "cell_type": "code", "execution_count": 41, "id": "cc4e442a", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "count 201.000000\n", "mean 0.213696\n", "std 0.161163\n", "min 0.021356\n", "25% 0.084765\n", "50% 0.190217\n", "75% 0.285706\n", "max 0.793830\n", "Name: score_x, dtype: float64" ] }, "execution_count": 41, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_production_raw[\"score_x\"].describe()" ] }, { "cell_type": "code", "execution_count": 42, "id": "37fc2c20", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "decision_x\n", "ACCORDÉ 184\n", "REFUSÉ 17\n", "Name: count, dtype: int64" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_production_raw[\"decision_x\"].value_counts()" ] }, { "cell_type": "markdown", "id": "281fe7aa", "metadata": {}, "source": [ "### Statistiques globales sur les performances API" ] }, { "cell_type": "markdown", "id": "0217db8c", "metadata": {}, "source": [ "Les métriques de production montrent que la latence moyenne de l’API est de *104 ms*, avec un p95 à *300 ms* et quelques pics à 1 seconde.\n", "\n", "Le CPU reste faible *(24 %)*, mais la RAM est élevée *(74 %)*, ce qui confirme que l’API n’est pas limitée par le modèle mais par le ***preprocessing***.\n", "\n", "Le temps d’inférence mesuré **(36 ms en moyenne, 65 ms au p95)** correspond essentiellement au preprocessing sklearn, le modèle lui-même étant très rapide **(< 3 ms)**.\n", "\n", "Ces résultats confirment que l’optimisation doit se concentrer sur le preprocessing plutôt que sur le modèle. " ] }, { "cell_type": "code", "execution_count": 43, "id": "6aea8f28", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "latency_mean_p95_max: (104.26548053401177, 300.34413337707514, 1024.6069431304932)\n", "cpu_ram: (24.00985915492958, 74.26103286384976)\n", "infer: (64.89101992920038, 36.70942912341703)\n" ] } ], "source": [ "lat_mean = df_production_raw[\"latency_ms\"].mean() # latence moyenne\n", "lat_p95 = df_production_raw[\"latency_ms\"].quantile(0.95) # latence au 95ème percentile\n", "lat_max = df_production_raw[\"latency_ms\"].max() # latence maximale observée\n", "\n", "cpu_mean = df_production_raw[\"cpu_percent\"].mean()\n", "ram_mean = df_production_raw[\"ram_percent\"].mean()\n", "\n", "infer_mean = df_production_raw[\"inference_ms_x\"].mean() # temps moyen passé dans predict_proba()\n", "infer_p95 = df_production_raw[\"inference_ms_x\"].quantile(0.95) # temps au 95ème percentile\n", "\n", "print(f\"latency_mean_p95_max: {lat_mean, lat_p95, lat_max}\")\n", "print(f\"cpu_ram: {cpu_mean, ram_mean}\")\n", "print(f\"infer: {infer_p95, infer_mean}\")" ] }, { "cell_type": "markdown", "id": "3050b4f2", "metadata": {}, "source": [ "### Recherche des goulots d'étranglement - **cProfile**" ] }, { "cell_type": "code", "execution_count": 44, "id": "26f53c38", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Wed Jun 17 21:37:06 2026 profiling.prof\n", "\n", " 4167704 function calls (4147304 primitive calls) in 8.525 seconds\n", "\n", " Ordered by: cumulative time\n", " List reduced from 591 to 18 due to restriction <18>\n", "\n", " ncalls tottime percall cumtime percall filename:lineno(function)\n", " 1 0.000 0.000 8.526 8.526 {built-in method builtins.exec}\n", " 1 0.000 0.000 8.526 8.526 :1()\n", " 1 0.029 0.029 8.526 8.526 C:\\Users\\Lenovo\\AppData\\Local\\Temp\\ipykernel_7484\\1097060041.py:1(benchmark)\n", " 100 0.003 0.000 8.493 0.085 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\pipeline.py:676(predict_proba)\n", " 500/100 0.032 0.000 7.989 0.080 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\_set_output.py:293(wrapped)\n", " 100 0.041 0.000 7.957 0.080 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\compose\\_column_transformer.py:946(transform)\n", " 100 0.010 0.000 7.570 0.076 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\compose\\_column_transformer.py:751(_call_func_on_transformers)\n", " 100 0.001 0.000 6.729 0.067 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\parallel.py:44(__call__)\n", " 100 0.002 0.000 6.727 0.067 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\joblib\\parallel.py:1969(__call__)\n", " 600 0.007 0.000 6.718 0.011 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\joblib\\parallel.py:1888(_get_sequential_output)\n", " 400 0.007 0.000 6.706 0.017 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\parallel.py:115(__call__)\n", " 400 0.004 0.000 6.665 0.017 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\pipeline.py:1261(_transform_one)\n", " 200 0.066 0.000 4.501 0.023 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\preprocessing\\_encoders.py:185(_transform)\n", " 100 0.016 0.000 3.814 0.038 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\preprocessing\\_encoders.py:984(transform)\n", " 1700 0.171 0.000 2.750 0.002 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\validation.py:718(check_array)\n", " 1600 0.117 0.000 2.239 0.001 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\_encode.py:236(_check_unknown)\n", " 100 0.047 0.000 2.057 0.021 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\preprocessing\\_data.py:1025(transform)\n", " 100 0.003 0.000 2.005 0.020 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\base.py:537(_validate_data)\n", "\n", "\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "def benchmark():\n", " for _ in range(100):\n", " pipe.predict_proba(df_reference_raw)\n", "\n", "cProfile.run(\n", " \"benchmark()\",\n", " \"profiling.prof\")\n", "\n", "stats = pstats.Stats(\"profiling.prof\")\n", "stats.sort_stats(\"cumtime\")\n", "stats.print_stats(18)" ] }, { "cell_type": "markdown", "id": "03fc7e60", "metadata": {}, "source": [ "Résultat profiling sur l'ensemble du dataset.\n", "Sat Jun 13 23:37:16 2026 profiling.prof\n", "\n", " 150875804 function calls (150855404 primitive calls) in 170.527 seconds\n", "\n", " Ordered by: cumulative time\n", " List reduced from 593 to 18 due to restriction <18>\n", "\n", " ncalls tottime percall cumtime percall filename:lineno(function)\n", " 1 0.000 0.000 170.568 170.568 {built-in method builtins.exec}\n", " 1 0.000 0.000 170.568 170.568 :1()\n", " 1 2.378 2.378 170.568 170.568 C:\\Users\\Lenovo\\AppData\\Local\\Temp\\ipykernel_6608\\4137855541.py:4(benchmark)\n", " 100 0.002 0.000 168.186 1.682 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\pipeline.py:676(predict_proba)\n", " 500/100 1.023 0.002 145.599 1.456 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\_set_output.py:293(wrapped)\n", " 100 1.367 0.014 144.604 1.446 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\compose\\_column_transformer.py:946(transform)\n", " 100 0.009 0.000 136.908 1.369 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\compose\\_column_transformer.py:751(_call_func_on_transformers)\n", " 100 0.001 0.000 129.304 1.293 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\parallel.py:44(__call__)\n", " 100 0.002 0.000 129.303 1.293 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\joblib\\parallel.py:1969(__call__)\n", " 600 0.005 0.000 129.296 0.215 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\joblib\\parallel.py:1888(_get_sequential_output)\n", " 400 0.004 0.000 129.288 0.323 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\parallel.py:115(__call__)\n", " 400 0.003 0.000 129.261 0.323 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\pipeline.py:1261(_transform_one)\n", " 200 0.723 0.004 114.522 0.573 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\preprocessing\\_encoders.py:185(_transform)\n", " 100 0.360 0.004 99.135 0.991 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\preprocessing\\_encoders.py:984(transform)\n", " 1600 2.995 0.002 92.976 0.058 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\_encode.py:236(_check_unknown)\n", " 3200 0.187 0.000 66.984 0.021 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\_encode.py:108(_extract_missing)\n", " 3200 7.114 0.002 66.791 0.021 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\_encode.py:124()\n", " 11946300 15.667 0.000 59.734 0.000 c:\\Users\\Lenovo\\anaconda3\\envs\\mlops\\Lib\\site-packages\\sklearn\\utils\\__init__.py:1151(is_scalar_nan)" ] }, { "cell_type": "markdown", "id": "c2c66110", "metadata": {}, "source": [ "Le profiling sur l'ensemble du dataset (plus de 40000 lignes) montre que la latence du pipeline est dominée par le **preprocessing (85 % du temps)**, en particulier par *OneHotEncoder et ses vérifications internes (_check_unknown, is_scalar_nan)*:\n", "- `ColumnTransformer.transform` <> 144.6 s cumulés\n", "- `OneHotEncoder._transform` <> 114.5 s ; représente **67%** du temps total (170.6 ms) - très coûteux en unitaire\n", "- `OneHotEncoder.transform` <> 99.1 s\n", "- `_check_unknown` <> 92.9 s\n", "- `is_scalar_nan` <> 59.7 s\n", "\n", "Le profiling sur un échantilon (500 lignes) pour des besoins GiHub montre:\n", "- un coût total réduit (9 s au lieu de 170 s)\n", "- des fonctions qui disparaissent car moins de lignes\n", "- un comportement plus “léger”\n", "- un pipeline qui reste coûteux mais moins extrême\n", "Néanmoins, la structure du coût reste la même (preprocessing dominant), mais :\n", "- les appels à `_check_unknown` et `is_scalar_nan` chutent fortement,\n", "- les fonctions liées aux valeurs rares disparaissent du top 20,\n", "- le coût absolu est beaucoup plus faible.\n", "\n", "Les deux profilings sont complémentaires :\n", "- le profiling complet montre le comportement réel du pipeline en production,\n", "- le profiling réduit montre le comportement dans un environnement GitHub/CI.\n", "\n", "Ils démontrent ensemble que :\n", "- **le modèle est très rapide**, \n", "- **le preprocessing est le goulot d’étranglement**, \n", "- **le coût dépend fortement de la taille du dataset**, \n", "- **le batch est indispensable pour réduire la latence**." ] }, { "cell_type": "markdown", "id": "2b5587db", "metadata": {}, "source": [ "### Préprocessing vs modèle\n", "Le modèle lui‑même est très rapide *(< 3 ms)*, mais le preprocessing coûte *25 ms par ligne*, ce qui explique les latences élevées en appels unitaires." ] }, { "cell_type": "code", "execution_count": 45, "id": "b313e60b", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "preprocessing : 24.99309927225113\n", "model : 1.6447007656097412\n", "total : 26.63780003786087\n" ] } ], "source": [ "try:\n", " df_example = df_reference_raw\n", "except:\n", " df_example = None\n", "\n", "X = df_example.iloc[[0]]\n", "t0 = time.perf_counter()\n", "\n", "X_transformed = pipe[:-1].transform(X)\n", "t1 = time.perf_counter()\n", "\n", "score = pipe[-1].predict_proba(X_transformed)\n", "t2 = time.perf_counter()\n", "\n", "print(\"preprocessing :\", (t1 - t0)*1000)\n", "print(\"model :\", (t2 - t1)*1000)\n", "print(\"total :\", (t2 - t0)*1000)" ] }, { "cell_type": "markdown", "id": "873a481d", "metadata": {}, "source": [ "### Optimisation du preprocessing: unitaire vs batch" ] }, { "cell_type": "markdown", "id": "62318faa", "metadata": {}, "source": [ "Les résultats ci-dessous confirment le profiling:\n", "- 1 ligne > **25 ms** de preprocessing\n", "- 100 lignes en unitaire > **2433 ms**\n", "- 100 lignes en batch > **30 ms**\n", "\n", "Vectorisé, le preprocessing ne s’exécute qu’une seule fois en batch, mais 100 fois en unitaire <> Le batch est sensiblement **70 fois plus rapide que l’unitaire**." ] }, { "cell_type": "code", "execution_count": 46, "id": "4041e8d6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "100 appels unitaires : 2433.636900037527 ms\n" ] } ], "source": [ "start = time.perf_counter()\n", "\n", "for i in range(100):\n", " pipe.predict_proba(df_example.iloc[[i]])\n", "\n", "end = time.perf_counter()\n", "print(\"100 appels unitaires :\", (end - start) * 1000, \"ms\")" ] }, { "cell_type": "code", "execution_count": 47, "id": "ac7fb688", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 batch de 100 lignes : 29.85369972884655 ms\n" ] } ], "source": [ "X_batch = df_example.iloc[:100]\n", "\n", "start = time.perf_counter()\n", "pipe.predict_proba(X_batch)\n", "end = time.perf_counter()\n", "\n", "print(\"1 batch de 100 lignes :\", (end - start) * 1000, \"ms\")" ] }, { "cell_type": "markdown", "id": "899fdbd4", "metadata": {}, "source": [ "Le batch permet de vectoriser les transformations et réduit le temps total d’un facteur ×70.\n", "Ces résultats démontrent que l’optimisation doit se concentrer sur le preprocessing plutôt que sur le modèle." ] }, { "cell_type": "markdown", "id": "31cbe8ff", "metadata": {}, "source": [ "# **Conclusion**\n", "\n", "**1. Analyse du data drift**\n", "\n", "Après harmonisation complète des colonnes entre les données de référence et les données de production (conversion des types, gestion des valeurs manquantes, suppression des colonnes constantes, alignement des colonnes communes), nous avons pu analyser le drift sur **278 colonnes**.\n", "\n", "Les résultats montrent :\n", "- **Colonnes testées : 278**\n", "- **Colonnes avec drift détecté : 79**\n", "- **Taux de dérive : 28.4 %**\n", "\n", "Ce taux est significatif : près d’un tiers des variables présentent une distribution différente entre la référence et la production. \n", "Les colonnes les plus touchées concernent principalement :\n", "- les variables dérivées de comportements (durées, montants, ratios),\n", "- certaines variables catégorielles encodées,\n", "- et des variables liées à l’historique de crédit.\n", "\n", "Un tel niveau de drift peut potentiellement impacter la stabilité du modèle, et justifie la mise en place d’un monitoring continu ainsi que d’un mécanisme d’alerte.\n", "\n", "**2. Analyse de la performance du pipeline**\n", "\n", "Les mesures de latence montrent que :\n", "\n", "- **le preprocessing est de loin la partie la plus coûteuse** du pipeline (25 ms par ligne, contre 2.9 ms pour le modèle lui-même),\n", "- **100 appels unitaires** coûtent **2433 ms**, car le preprocessing est répété 100 fois,\n", "- **un batch de 100 lignes** ne coûte que **30 ms**, grâce à la vectorisation des transformations.\n", "\n", "Ces résultats sont confirmés par le profiling, qui montre que :\n", "\n", "- **85 % du temps total** est passé dans `ColumnTransformer.transform`,\n", "- dont **67 % dans OneHotEncoder** et ses vérifications internes (`_check_unknown`, `is_scalar_nan`).\n", "\n", "Le modèle, lui, est très rapide (< 3 ms)." ] } ], "metadata": { "kernelspec": { "display_name": "mlops", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.15" } }, "nbformat": 4, "nbformat_minor": 5 }