docs/03_feature_engineering.md

# Feature Engineering Results

## 1. Implemented Strategy
We implemented a comprehensive feature engineering pipeline with customer-level aggregation:

### Data Cleaning & Type Conversion
-   **Numeric Parsing:** Cleaned `Age`, `Annual_Income`, `Outstanding_Debt`, `Num_of_Delayed_Payment`, `Num_of_Loan`, `Amount_invested_monthly`, `Monthly_Balance`, `Changed_Credit_Limit` (removed underscores, commas, and handled invalid values).

### Feature Extraction
-   **Credit History Age:** Converted from "X Years Y Months" format to total months.
-   **Loan Features:**
    -   `Loan_Count_Calculated`: Count of different loan types.
    -   `Loan_<Type>`: One-Hot encoded top 8 loan types (Auto, Credit-Builder, Personal, Home Equity, Mortgage, Student, Debt Consolidation, Payday).
-   **Financial Ratios:**
    -   `Debt_to_Income_Ratio`: Outstanding Debt / Annual Income (financial risk metric).
    -   `Debt_Per_Loan`: Outstanding Debt / Loan Count.
    -   `Installment_to_Income`: Monthly EMI / Monthly Salary (debt service capacity).
    -   `Delayed_Per_Loan`: Num of Delayed Payments / Loan Count (payment reliability).
-   **Interaction Features:**
    -   `DTI_x_LoanCount`: Debt-to-Income × Loan Count (combined risk).
    -   `Log_Annual_Income`: Log-transformed income (handles skewness).

### Imputation & Aggregation
-   **Grouped Imputation:** Median salary imputation grouped by Occupation (more accurate than global median).
-   **Customer-Level Aggregation:** Reduced 150,000 monthly rows to 25,000 unique customers:
    -   **Stable fields** (Age, loan flags): First value.
    -   **Monthly-changing fields** (Income, Balance, EMI): Mean.
    -   **Count fields** (Delayed payments, inquiries): Sum.
    -   **Categorical fields** (Payment Behaviour, Credit Mix): Mode.

### Encoding & Scaling
-   **Ordinal Encoding:** Credit_Mix (Bad=0, Standard=1, Good=2).
-   **One-Hot Encoding:** Occupation, Payment_Behaviour, Month.
-   **Label Encoding:** Target (Credit_Score).
-   **No Global Scaling:** Features remain unscaled to preserve tree model performance (trees are invariant to feature scaling).

## 2. Model Performance Comparison

| Model | Dataset | Accuracy | Notes |
| :--- | :--- | :--- | :--- |
| **Baseline** (Simple Logistic Regression) | 5-Fold CV | **0.7211** | Strong linear baseline |
| **Logistic Regression** (with feature engineering + scaling) | Validation Split | **0.6544** | Linear model struggles with complex features |
| **Random Forest** (hyperparameter tuned) | Validation Split | **0.7340** | ✅ **Exceeds baseline by 1.3%** |

### Random Forest Class-Wise Performance

| Class | Precision | Recall | F1-Score | Support |
| :--- | :--- | :--- | :--- | :--- |
| Poor (0) | 0.59 | 0.84 | 0.69 | 501 |
| Standard (1) | 0.73 | 0.81 | 0.77 | 832 |
| Good (2) | 0.86 | 0.63 | 0.73 | 1,167 |
| **Weighted Avg** | **0.76** | **0.73** | **0.73** | **2,500** |

## 3. Key Insights

### Why Logistic Regression Performance Dropped
1. **Non-linear Feature Interactions:** Engineered features (DTI × LoanCount, Debt_Per_Loan) capture non-linear relationships that linear models cannot leverage.
2. **Dimensionality Curse:** One-Hot encoding of multiple categorical features (Occupation, Payment_Behaviour) increased feature space without linear model regularization.
3. **Information Loss:** Dropping `Annual_Income` in favor of `Log_Annual_Income` may have removed linear signal if the true relationship isn't purely logarithmic.

### Why Random Forest Excels
1. **Non-linear Decision Boundaries:** Trees naturally capture feature interactions without explicit engineering.
2. **High Recall on Poor Scores:** 84% recall on class 0 (Poor) is critical for risk management—catches risky customers.
3. **Balanced Performance:** Weighted F1-score of 0.73 shows good generalization across all credit score classes.
4. **Robustness:** Hyperparameters (max_depth=10, balanced_class_weight) prevent overfitting while leveraging complex features.

## 4. Recommendations for Next Phase (04_model_optimization.ipynb)

✅ **Keep the engineered features** — They provide valuable signal for non-linear models.
✅ **Continue with tree-based models** — Random Forest, XGBoost will unlock feature complexity better than linear models.
✅ **Perform feature importance analysis** — Identify which engineered features drive predictions.
✅ **Cross-validate with stratified k-fold** — Ensure 73.4% accuracy is stable across data splits.
✅ **Compare with XGBoost** — Gradient boosting may outperform bagging approaches.
✅ **Class-wise optimization** — Focus on improving recall for "Poor" customers (high-risk detection).

## 5. Data Quality Improvements Made
-   ✅ Handled missing values in `Monthly_Inhand_Salary`, `Type_of_Loan`, `Credit_History_Age`.
-   ✅ Cleaned numeric columns with special characters (underscores, commas).
-   ✅ Removed outliers in `Num_of_Delayed_Payment` (clipped at 99th percentile).
-   ✅ Aggregated to customer level to prevent temporal leakage and reduce noise.
-   ✅ Verified no NaN values remain before model training.