doc/RISK_DISCOVERY_COMPARISON.md

# Risk Discovery Methods - Comparison Implementation

## Overview
Implemented 3 additional risk discovery methods beyond the original K-Means clustering to enable comprehensive comparison of different approaches.

---

## 🎯 Implemented Methods (4 Total)

### 1. ✅ K-Means Clustering (Original)
**File**: `risk_discovery.py`  
**Status**: Already existed

**Characteristics**:
- Fast and scalable
- Creates spherical clusters
- Clear, non-overlapping boundaries
- Requires predefined number of clusters

**Best For**:
- Large datasets (10K+ clauses)
- When you need consistent, fast results
- Clear risk categorization needed

---

### 2. ✅ LDA Topic Modeling (NEW)
**File**: `risk_discovery_alternatives.py`  
**Class**: `TopicModelingRiskDiscovery`

**Characteristics**:
- Probabilistic approach
- Clauses can belong to multiple topics
- Discovers interpretable topic-word distributions
- Slower than K-Means

**Advantages**:
- ✅ Natural handling of overlapping risks
- ✅ Provides probability distribution per clause
- ✅ Topic words are highly interpretable
- ✅ Works well with legal text (multi-thematic)

**Disadvantages**:
- ❌ Requires tuning (alpha, beta parameters)
- ❌ Slower training (20 iterations)
- ❌ Less clear boundaries than K-Means

**Parameters**:
- `n_topics`: Number of topics to discover (default: 7)
- `doc_topic_prior`: Alpha - document-topic density (default: 0.1)
- `topic_word_prior`: Beta - topic-word density (default: 0.01)

**Quality Metrics**:
- Perplexity (lower is better)
- Log-likelihood (higher is better)
- Topic diversity (entropy)

**Example Output**:
```python
{
  'topic_id': 0,
  'topic_name': 'Topic_LIABILITY',
  'top_words': ['liability', 'damages', 'indemnify', 'loss', 'liable'],
  'word_weights': [0.045, 0.038, 0.031, ...],
  'clause_count': 2847,
  'proportion': 0.284
}
```

---

### 3. ✅ Hierarchical Clustering (NEW)
**File**: `risk_discovery_alternatives.py`  
**Class**: `HierarchicalRiskDiscovery`

**Characteristics**:
- Builds hierarchy of clusters (dendrogram)
- Can be cut at different levels
- Agglomerative (bottom-up) approach
- Deterministic results

**Advantages**:
- ✅ Discovers nested risk structure
- ✅ No need to specify clusters upfront
- ✅ Deterministic (same results every run)
- ✅ Can explore different granularities

**Disadvantages**:
- ❌ Slower for large datasets (O(n²) complexity)
- ❌ Memory intensive
- ❌ Cannot scale to 100K+ clauses easily

**Parameters**:
- `n_clusters`: Number of clusters to form (default: 7)
- `linkage`: Linkage criterion - 'ward', 'average', 'complete', 'single'

**Quality Metrics**:
- Silhouette score (-1 to 1, higher is better)
- Average cluster size
- Cluster balance

**Example Output**:
```python
{
  'cluster_id': 2,
  'cluster_name': 'RISK_INDEMNITY',
  'top_terms': ['indemnify', 'hold', 'harmless', 'defend', 'claims'],
  'term_scores': [0.234, 0.198, 0.176, ...],
  'clause_count': 1543,
  'proportion': 0.154
}
```

---

### 4. ✅ DBSCAN (Density-Based) (NEW)
**File**: `risk_discovery_alternatives.py`  
**Class**: `DensityBasedRiskDiscovery`

**Characteristics**:
- Density-based clustering
- Discovers clusters of arbitrary shapes
- Identifies outliers/noise automatically
- No need to specify number of clusters

**Advantages**:
- ✅ Finds clusters of any shape
- ✅ Identifies outliers (rare risks)
- ✅ Robust to noise
- ✅ No cluster number required

**Disadvantages**:
- ❌ Sensitive to eps parameter
- ❌ Struggles with varying density
- ❌ May produce many small clusters
- ❌ Requires careful tuning

**Parameters**:
- `eps`: Maximum distance between samples (default: 0.5)
- `min_samples`: Minimum samples in neighborhood (default: 5)
- `auto_tune`: Automatically tune eps (default: True)

**Quality Metrics**:
- Number of clusters found
- Outlier ratio (% of noise points)
- Average cluster size

**Special Feature**: **Outlier Detection**
- Identifies clauses that don't fit any pattern
- Useful for finding rare/unique risks
- Can flag unusual contract clauses

**Example Output**:
```python
{
  'cluster_id': 3,
  'cluster_name': 'RISK_PAYMENT_C3',
  'top_terms': ['payment', 'fee', 'invoice', 'dollar', 'cost'],
  'clause_count': 892,
  'is_core_cluster': True
}

# Plus outliers
{
  'n_outliers': 234,
  'outlier_ratio': 0.023,
  'outlier_clauses': ['Unusual clause text...']
}
```

---

## 📊 Comparison Features

### Automated Comparison Script
**File**: `compare_risk_discovery.py`

**What It Does**:
1. Loads CUAD dataset (or generates sample data)
2. Runs all 4 methods on the same clauses
3. Measures execution time for each
4. Compares quality metrics
5. Analyzes pattern diversity
6. Generates comprehensive report

**Usage**:
```bash
python compare_risk_discovery.py
```

**Output Files**:
- `risk_discovery_comparison_report.txt` - Human-readable report
- `risk_discovery_comparison_results.json` - Detailed results with metrics

**Report Includes**:
- Summary table with timing and quality
- Detailed analysis per method
- Pattern diversity metrics
- Top discovered patterns
- Recommendations for each method

---

## 🔬 Comparison Metrics

### Performance Metrics
1. **Execution Time**: Time to discover patterns
2. **Clauses/Second**: Processing throughput
3. **Scalability**: How method handles large datasets

### Quality Metrics
1. **Silhouette Score** (Hierarchical):
   - Range: -1 to 1
   - Higher is better
   - Measures cluster cohesion

2. **Perplexity** (LDA):
   - Lower is better
   - Measures model fit to data

3. **Outlier Ratio** (DBSCAN):
   - % of clauses marked as outliers
   - Indicates rare pattern detection

### Diversity Metrics
1. **Pattern Balance**: How evenly clauses distributed
2. **Pattern Size Variance**: Consistency of cluster sizes
3. **Topic Diversity** (LDA): Entropy of word distributions

---

## 🎯 Method Selection Guide

### Choose **K-Means** when:
- ✅ You have large datasets (10K+ clauses)
- ✅ You need fast, consistent results
- ✅ You want clear, non-overlapping categories
- ✅ You can specify number of risks upfront

### Choose **LDA** when:
- ✅ Clauses may have multiple risk types
- ✅ You need probability distributions
- ✅ Interpretability of topics is crucial
- ✅ You can afford slower processing

### Choose **Hierarchical** when:
- ✅ You want to explore risk hierarchies
- ✅ Dataset is moderate size (<10K clauses)
- ✅ You need deterministic results
- ✅ You want to analyze at multiple granularities

### Choose **DBSCAN** when:
- ✅ You need to identify outliers/rare risks
- ✅ Cluster shapes are not spherical
- ✅ You don't know number of clusters
- ✅ You want automatic noise handling

---

## 📈 Performance Comparison (Estimated)

Based on 5,000 clauses:

| Method        | Time    | Scalability | Quality | Interpretability |
|---------------|---------|-------------|---------|------------------|
| K-Means       | 5-10s   | Excellent   | Good    | Good             |
| LDA           | 30-60s  | Good        | Good    | Excellent        |
| Hierarchical  | 15-30s  | Moderate    | Good    | Good             |
| DBSCAN        | 10-20s  | Good        | Variable| Good             |

---

## 🔧 Integration with Training Pipeline

### Option 1: Use Single Method
```python
# In trainer.py
from risk_discovery import UnsupervisedRiskDiscovery  # Original

# Or choose alternative
from risk_discovery_alternatives import TopicModelingRiskDiscovery

risk_discovery = TopicModelingRiskDiscovery(n_topics=7)
```

### Option 2: Compare Before Training
```bash
# Run comparison first
python compare_risk_discovery.py

# Review results
cat risk_discovery_comparison_report.txt

# Choose best method for your data
python train.py  # Will use chosen method
```

### Option 3: Ensemble Approach (Advanced)
```python
# Use multiple methods and combine
from risk_discovery import UnsupervisedRiskDiscovery
from risk_discovery_alternatives import TopicModelingRiskDiscovery

kmeans = UnsupervisedRiskDiscovery(n_clusters=7)
lda = TopicModelingRiskDiscovery(n_topics=7)

kmeans_labels = kmeans.discover_risk_patterns(clauses)
lda_labels = lda.discover_risk_patterns(clauses)

# Combine using voting or averaging
ensemble_labels = combine_predictions(kmeans_labels, lda_labels)
```

---

## 📝 Code Statistics

### New Implementation
- **New File**: `risk_discovery_alternatives.py` (570 lines)
- **New File**: `compare_risk_discovery.py` (450 lines)
- **Total New Code**: ~1,020 lines
- **New Classes**: 3 (TopicModelingRiskDiscovery, HierarchicalRiskDiscovery, DensityBasedRiskDiscovery)
- **New Methods**: 20+

### Dependencies Required
All methods use existing dependencies:
- `sklearn.decomposition.LatentDirichletAllocation` (LDA)
- `sklearn.cluster.AgglomerativeClustering` (Hierarchical)
- `sklearn.cluster.DBSCAN` (Density-based)
- Already in `requirements.txt`

---

## 🧪 Testing the Comparison

### Quick Test (Sample Data)
```bash
# Uses synthetic sample clauses
python compare_risk_discovery.py
```

### Full Test (CUAD Dataset)
```bash
# Requires CUAD dataset at dataset/CUAD_v1/CUAD_v1.json
python compare_risk_discovery.py
```

### Expected Output
```
🔬 RISK DISCOVERY METHOD COMPARISON

Loading CUAD dataset...
✅ Loaded 5000 clauses for comparison

METHOD 1: K-Means Clustering
  Execution time: 8.34 seconds
  ✅ 7 clusters found

METHOD 2: LDA Topic Modeling
  Execution time: 45.67 seconds
  Perplexity: 1234.56
  ✅ 7 topics found

METHOD 3: Hierarchical Clustering
  Execution time: 23.12 seconds
  Silhouette Score: 0.234
  ✅ 7 clusters found

METHOD 4: DBSCAN
  Execution time: 12.45 seconds
  ✅ 9 clusters found, 123 outliers

📊 COMPARISON SUMMARY
[Detailed comparison table...]

💾 Report saved to: risk_discovery_comparison_report.txt
✅ Detailed results saved to: risk_discovery_comparison_results.json
```

---

## 🎉 Summary

### What Was Added
1. ✅ **3 new risk discovery methods** (LDA, Hierarchical, DBSCAN)
2. ✅ **Automated comparison framework**
3. ✅ **Comprehensive evaluation metrics**
4. ✅ **Method selection guidelines**
5. ✅ **Integration ready for training pipeline**

### What You Can Do Now
1. **Compare Methods**: Run `compare_risk_discovery.py`
2. **Choose Best Method**: Based on your data characteristics
3. **Train with Alternative**: Swap in any method for training
4. **Ensemble Learning**: Combine multiple methods (advanced)

### Next Steps
1. Run comparison on your CUAD dataset
2. Review the generated report
3. Choose the method that best fits your needs:
   - **Fast & Scalable** → K-Means
   - **Overlapping Risks** → LDA
   - **Risk Hierarchy** → Hierarchical
   - **Outlier Detection** → DBSCAN
4. Update `trainer.py` to use chosen method
5. Train Legal-BERT with optimized risk discovery

---

**Status**: ✅ **COMPLETE AND READY FOR TESTING**  
**Files**: `risk_discovery_alternatives.py`, `compare_risk_discovery.py`  
**Lines Added**: ~1,020 lines of production code