src/Machine_Learning/README.md

# RiskEnsembleClassifier
### Risk-Aware Policy Clause Importance Ranking — Durgesh Yadav: Risk Modeling & ML Lead

---

## What This Does

Takes a policy document's clauses and automatically predicts the **risk severity** of each clause (Low / Medium / High / Critical), then ranks them by risk so the most dangerous clauses appear first.

---

## Folder Structure

```
src/
└── Machine_Learning/
    ├── RiskEnsembleClassifier.py   ← main training + evaluation script
    ├── test_model.py               ← testing script (csv / interactive / error analysis)
    ├── RiskEnsembleClassifier.pkl  ← saved model bundle (gitignored if large)
    ├── ranked_clauses.csv          ← output: all clauses ranked by risk score
    └── requirements.txt
```

---

## How to Run

```bash
# Install dependencies
pip install -r requirements.txt

# Train and evaluate the model
python src/Machine_Learning/RiskEnsembleClassifier.py

# Or with custom data path
python src/Machine_Learning/RiskEnsembleClassifier.py --data "path/to/dataset.csv"
```

---

## Models Trained

Three baseline models were trained and compared before arriving at the final ensemble.

### Model 1 — Logistic Regression (Baseline)

Simple linear classifier used as the starting reference point.

| Class    | Precision | Recall | F1-Score | Support |
|----------|-----------|--------|----------|---------|
| Critical | 0.7823    | 0.6950 | 0.7361   | 200     |
| High     | 0.6912    | 0.7150 | 0.7029   | 200     |
| Low      | 0.7234    | 0.8100 | 0.7644   | 200     |
| Medium   | 0.6978    | 0.6700 | 0.6836   | 200     |
| **Overall Accuracy** | | | **0.7225** | **800** |

**Critical Clause Recall: 0.6950**

---

### Model 2 — Random Forest

Ensemble of decision trees, better than logistic regression but still limited by bag-of-words features.

| Class    | Precision | Recall | F1-Score | Support |
|----------|-----------|--------|----------|---------|
| Critical | 0.8534    | 0.8050 | 0.8285   | 200     |
| High     | 0.7823    | 0.8000 | 0.7910   | 200     |
| Low      | 0.7945    | 0.8750 | 0.8329   | 200     |
| Medium   | 0.7812    | 0.7450 | 0.7627   | 200     |
| **Overall Accuracy** | | | **0.8063** | **800** |

**Critical Clause Recall: 0.8050**

---

### Model 3 — XGBoost (Tuned)

Gradient boosting with hyperparameter tuning via RandomizedSearchCV.

| Class    | Precision | Recall | F1-Score | Support |
|----------|-----------|--------|----------|---------|
| Critical | 1.0000    | 0.9050 | 0.9501   | 200     |
| High     | 0.8955    | 0.9000 | 0.8978   | 200     |
| Low      | 0.8609    | 0.9900 | 0.9209   | 200     |
| Medium   | 0.8936    | 0.8400 | 0.8660   | 200     |
| **Overall Accuracy** | | | **0.9088** | **800** |

**Critical Clause Recall: 0.9050**

---

### ✅ Final Model — RiskEnsembleClassifier (XGBoost + LightGBM + ExtraTrees)

Best performing model. Combines 3 models using soft voting (averages class probabilities).

| Class    | Precision | Recall | F1-Score | Support |
|----------|-----------|--------|----------|---------|
| Critical | 1.0000    | 0.9000 | 0.9474   | 200     |
| High     | 0.9069    | 0.9250 | 0.9158   | 200     |
| Low      | 0.8795    | 0.9850 | 0.9292   | 200     |
| Medium   | 0.8854    | 0.8500 | 0.8673   | 200     |
| **Overall Accuracy** | | | **0.9150** | **800** |

**Critical Clause Recall: 0.9000**

---

## Model Comparison Summary

| Model | Accuracy | Critical Recall | Critical F1 | Notes |
|-------|----------|-----------------|-------------|-------|
| Logistic Regression | 72.25% | 0.695 | 0.736 | Baseline |
| Random Forest | 80.63% | 0.805 | 0.829 | Better than baseline |
| XGBoost (Tuned) | 90.88% | 0.905 | 0.950 | Big jump with tuning |
| **RiskEnsembleClassifier** | **91.50%** | **0.900** | **0.947** | **Final model** |

> **Why XGBoost alone scored higher Critical Recall (0.905) vs Ensemble (0.900)?**
> The ensemble slightly smooths out extreme predictions — it trades a tiny bit of critical recall for better overall balance across all 4 classes. The ensemble wins on overall accuracy (91.5% vs 90.88%).

---

## Confusion Matrix (Final Model)

```
              Predicted
              Critical  High  Low  Medium
Actual  Critical   180    11    0       9
        High         0   185    5      10
        Low          0     0  197       3
        Medium       0     8   22     170
```

Key observations:
- Critical → only 9 misclassified as Medium, 11 as High. Zero misclassified as Low ✓
- Low → nearly perfect (197/200 correct)
- Medium → 22 confused with Low (hardest class to distinguish)

---

## Feature Engineering

| Feature Type | Method | Count |
|---|---|---|
| Word n-grams | TF-IDF (1,2)-grams, sublinear_tf | 2000 |
| Char n-grams | TF-IDF (3,5)-grams, char_wb | 1000 |
| Numeric features | modal_score, consequence_score, conditional_score, has_negation, obligation_count, penalty_flag, word_count | 7 |
| **Total features** | | **3007** |

### Top 15 Important Features (XGBoost sub-model)

| Feature | Importance |
|---|---|
| ment (char) | 0.0215 |
| rmati (char) | 0.0143 |
| penalty_flag | 0.0123 |
| men (char) | 0.0112 |
| eri (char) | 0.0112 |
| acti (char) | 0.0101 |
| section agreement | 0.0085 |
| shall conduct | 0.0075 |
| consequence_score | 0.0069 |
| has_negation | 0.0065 |

---

## Difficulties Faced & How Overcome

### 1. Data Leakage in TF-IDF
**Problem:** Initially `tfidf.fit_transform()` was called on the full dataset before the train/test split. This meant the model had seen test-set vocabulary during training, giving artificially inflated metrics.

**Solution:** Restructured the pipeline — `fit_transform` only on training data, `transform` on test data. Accuracy dropped slightly but became trustworthy.

---

### 2. Memory Issue with Dense Matrices
**Problem:** Calling `.toarray()` on TF-IDF output converted sparse matrices to dense numpy arrays. With 3000+ features × 4000 rows this consumed ~100MB+ RAM and slowed everything down.

**Solution:** Kept all matrices in sparse format (`scipy.sparse.csr_matrix`) throughout and used `sp.hstack()` for combining. Memory usage dropped by ~10x.

---

### 3. GridSearchCV Taking Too Long
**Problem:** Original `GridSearchCV` with 4×3×3×3 = 108 combinations × 3 folds = 324 full XGBoost training runs. On this dataset it would have taken hours.

**Solution:** Switched to `RandomizedSearchCV` with `n_iter=20` — samples 20 random combinations instead of all 108. Runs in ~10 minutes with near-equivalent results.

---

### 4. Wrong Scoring Metric in Hyperparameter Search
**Problem:** GridSearchCV was using `scoring='accuracy'` but the most important metric for this project is **Critical clause recall** — a model that ignores the Critical class can still score high accuracy.

**Solution:** Created a custom scorer combining `f1_macro` + `critical_recall` (50/50 weight). This ensures hyperparameter search optimises for what actually matters.

---

### 5. Class Imbalance Handling
**Problem:** `sample_weight` was passed to `RandomizedSearchCV` causing a `UserWarning` about statistically incorrect results since the custom scorer doesn't support sample weights.

**Solution:** Moved class balancing inside each model using `class_weight="balanced"` parameter (LightGBM, ExtraTrees) instead of external sample weights. Warning eliminated.

---

### 6. IndentationError Crashing Script
**Problem:** A mid-file `import` statement had 4 spaces of accidental indentation, causing an `IndentationError` before the GridSearch even ran.

**Solution:** Moved all imports to the top of the file following PEP8 conventions.

---

## Saved Model

The trained model is exported as a pickle bundle containing everything needed for inference:

```python
import pickle, scipy.sparse as sp

with open("RiskEnsembleClassifier.pkl", "rb") as f:
    bundle = pickle.load(f)

# Predict on new clause
text  = ["employee must disclose all conflicts of interest immediately"]
word  = bundle["tfidf_word"].transform(text)
char  = bundle["tfidf_char"].transform(text)
num   = sp.csr_matrix(bundle["scaler"].transform([[0]*7]))
X_new = sp.hstack([word, char, num], format="csr")

pred  = bundle["model"].predict(X_new)
label = bundle["label_encoder"].inverse_transform(pred)
print(label)  # ['Critical']
```

---

## Tech Stack

```
Python 3.13
scikit-learn    — TF-IDF, LabelEncoder, VotingClassifier, ExtraTrees, metrics
xgboost         — XGBClassifier
lightgbm        — LGBMClassifier
scipy           — sparse matrix operations
pandas          — data handling
numpy           — numerical ops
pickle          — model serialisation
```

---

## Output Files

| File | Description |
|---|---|
| `RiskEnsembleClassifier.pkl` | Saved model bundle (model + transformers + encoder) |
| `ranked_clauses.csv` | All 4000 clauses ranked by composite risk score |