doc/RISK_DISCOVERY_COMPARISON.md

Risk Discovery Methods - Comparison Implementation

Overview

Implemented 3 additional risk discovery methods beyond the original K-Means clustering to enable comprehensive comparison of different approaches.

---

🎯 Implemented Methods (4 Total)

1. ✅ K-Means Clustering (Original)

File: risk_discovery.py
Status: Already existed

Characteristics:

• Fast and scalable
• Creates spherical clusters
• Clear, non-overlapping boundaries
• Requires predefined number of clusters

Best For:

• Large datasets (10K+ clauses)
• When you need consistent, fast results
• Clear risk categorization needed

---

2. ✅ LDA Topic Modeling (NEW)

File: risk_discovery_alternatives.py
Class: TopicModelingRiskDiscovery

Characteristics:

• Probabilistic approach
• Clauses can belong to multiple topics
• Discovers interpretable topic-word distributions
• Slower than K-Means

Advantages:

• ✅ Natural handling of overlapping risks
• ✅ Provides probability distribution per clause
• ✅ Topic words are highly interpretable
• ✅ Works well with legal text (multi-thematic)

Disadvantages:

• ❌ Requires tuning (alpha, beta parameters)
• ❌ Slower training (20 iterations)
• ❌ Less clear boundaries than K-Means

Parameters:

• n_topics: Number of topics to discover (default: 7)
• doc_topic_prior: Alpha - document-topic density (default: 0.1)
• topic_word_prior: Beta - topic-word density (default: 0.01)

Quality Metrics:

• Perplexity (lower is better)
• Log-likelihood (higher is better)
• Topic diversity (entropy)

Example Output:

{
  'topic_id': 0,
  'topic_name': 'Topic_LIABILITY',
  'top_words': ['liability', 'damages', 'indemnify', 'loss', 'liable'],
  'word_weights': [0.045, 0.038, 0.031, ...],
  'clause_count': 2847,
  'proportion': 0.284
}

---

3. ✅ Hierarchical Clustering (NEW)

File: risk_discovery_alternatives.py
Class: HierarchicalRiskDiscovery

Characteristics:

• Builds hierarchy of clusters (dendrogram)
• Can be cut at different levels
• Agglomerative (bottom-up) approach
• Deterministic results

Advantages:

• ✅ Discovers nested risk structure
• ✅ No need to specify clusters upfront
• ✅ Deterministic (same results every run)
• ✅ Can explore different granularities

Disadvantages:

• ❌ Slower for large datasets (O(n²) complexity)
• ❌ Memory intensive
• ❌ Cannot scale to 100K+ clauses easily

Parameters:

• n_clusters: Number of clusters to form (default: 7)
• linkage: Linkage criterion - 'ward', 'average', 'complete', 'single'

Quality Metrics:

• Silhouette score (-1 to 1, higher is better)
• Average cluster size
• Cluster balance

Example Output:

{
  'cluster_id': 2,
  'cluster_name': 'RISK_INDEMNITY',
  'top_terms': ['indemnify', 'hold', 'harmless', 'defend', 'claims'],
  'term_scores': [0.234, 0.198, 0.176, ...],
  'clause_count': 1543,
  'proportion': 0.154
}

---

4. ✅ DBSCAN (Density-Based) (NEW)

File: risk_discovery_alternatives.py
Class: DensityBasedRiskDiscovery

Characteristics:

• Density-based clustering
• Discovers clusters of arbitrary shapes
• Identifies outliers/noise automatically
• No need to specify number of clusters

Advantages:

• ✅ Finds clusters of any shape
• ✅ Identifies outliers (rare risks)
• ✅ Robust to noise
• ✅ No cluster number required

Disadvantages:

• ❌ Sensitive to eps parameter
• ❌ Struggles with varying density
• ❌ May produce many small clusters
• ❌ Requires careful tuning

Parameters:

• eps: Maximum distance between samples (default: 0.5)
• min_samples: Minimum samples in neighborhood (default: 5)
• auto_tune: Automatically tune eps (default: True)

Quality Metrics:

• Number of clusters found
• Outlier ratio (% of noise points)
• Average cluster size

Special Feature: Outlier Detection

• Identifies clauses that don't fit any pattern
• Useful for finding rare/unique risks
• Can flag unusual contract clauses

Example Output:

{
  'cluster_id': 3,
  'cluster_name': 'RISK_PAYMENT_C3',
  'top_terms': ['payment', 'fee', 'invoice', 'dollar', 'cost'],
  'clause_count': 892,
  'is_core_cluster': True
}

# Plus outliers
{
  'n_outliers': 234,
  'outlier_ratio': 0.023,
  'outlier_clauses': ['Unusual clause text...']
}

---

📊 Comparison Features

Automated Comparison Script

File: compare_risk_discovery.py

What It Does:

1. Loads CUAD dataset (or generates sample data)
2. Runs all 4 methods on the same clauses
3. Measures execution time for each
4. Compares quality metrics
5. Analyzes pattern diversity
6. Generates comprehensive report

Usage:

python compare_risk_discovery.py

Output Files:

• risk_discovery_comparison_report.txt - Human-readable report
• risk_discovery_comparison_results.json - Detailed results with metrics

Report Includes:

• Summary table with timing and quality
• Detailed analysis per method
• Pattern diversity metrics
• Top discovered patterns
• Recommendations for each method

---

🔬 Comparison Metrics

Performance Metrics

1. Execution Time: Time to discover patterns
2. Clauses/Second: Processing throughput
3. Scalability: How method handles large datasets

Quality Metrics

1. Silhouette Score (Hierarchical):

   • Range: -1 to 1
   • Higher is better
   • Measures cluster cohesion

2. Perplexity (LDA):

   • Lower is better
   • Measures model fit to data

3. Outlier Ratio (DBSCAN):

   • % of clauses marked as outliers
   • Indicates rare pattern detection

Diversity Metrics

1. Pattern Balance: How evenly clauses distributed
2. Pattern Size Variance: Consistency of cluster sizes
3. Topic Diversity (LDA): Entropy of word distributions

---

🎯 Method Selection Guide

Choose K-Means when:

• ✅ You have large datasets (10K+ clauses)
• ✅ You need fast, consistent results
• ✅ You want clear, non-overlapping categories
• ✅ You can specify number of risks upfront

Choose LDA when:

• ✅ Clauses may have multiple risk types
• ✅ You need probability distributions
• ✅ Interpretability of topics is crucial
• ✅ You can afford slower processing

Choose Hierarchical when:

• ✅ You want to explore risk hierarchies
• ✅ Dataset is moderate size (<10K clauses)
• ✅ You need deterministic results
• ✅ You want to analyze at multiple granularities

Choose DBSCAN when:

• ✅ You need to identify outliers/rare risks
• ✅ Cluster shapes are not spherical
• ✅ You don't know number of clusters
• ✅ You want automatic noise handling

---

📈 Performance Comparison (Estimated)

Based on 5,000 clauses:

Method	Time	Scalability	Quality	Interpretability
K-Means	5-10s	Excellent	Good	Good
LDA	30-60s	Good	Good	Excellent
Hierarchical	15-30s	Moderate	Good	Good
DBSCAN	10-20s	Good	Variable	Good

---

🔧 Integration with Training Pipeline

Option 1: Use Single Method

# In trainer.py
from risk_discovery import UnsupervisedRiskDiscovery  # Original

# Or choose alternative
from risk_discovery_alternatives import TopicModelingRiskDiscovery

risk_discovery = TopicModelingRiskDiscovery(n_topics=7)

Option 2: Compare Before Training

# Run comparison first
python compare_risk_discovery.py

# Review results
cat risk_discovery_comparison_report.txt

# Choose best method for your data
python train.py  # Will use chosen method

Option 3: Ensemble Approach (Advanced)

# Use multiple methods and combine
from risk_discovery import UnsupervisedRiskDiscovery
from risk_discovery_alternatives import TopicModelingRiskDiscovery

kmeans = UnsupervisedRiskDiscovery(n_clusters=7)
lda = TopicModelingRiskDiscovery(n_topics=7)

kmeans_labels = kmeans.discover_risk_patterns(clauses)
lda_labels = lda.discover_risk_patterns(clauses)

# Combine using voting or averaging
ensemble_labels = combine_predictions(kmeans_labels, lda_labels)

---

📝 Code Statistics

New Implementation

• New File: risk_discovery_alternatives.py (570 lines)
• New File: compare_risk_discovery.py (450 lines)
• Total New Code: ~1,020 lines
• New Classes: 3 (TopicModelingRiskDiscovery, HierarchicalRiskDiscovery, DensityBasedRiskDiscovery)
• New Methods: 20+

Dependencies Required

All methods use existing dependencies:

• sklearn.decomposition.LatentDirichletAllocation (LDA)
• sklearn.cluster.AgglomerativeClustering (Hierarchical)
• sklearn.cluster.DBSCAN (Density-based)
• Already in requirements.txt

---

🧪 Testing the Comparison

Quick Test (Sample Data)

# Uses synthetic sample clauses
python compare_risk_discovery.py

Full Test (CUAD Dataset)

# Requires CUAD dataset at dataset/CUAD_v1/CUAD_v1.json
python compare_risk_discovery.py

Expected Output

🔬 RISK DISCOVERY METHOD COMPARISON

Loading CUAD dataset...
✅ Loaded 5000 clauses for comparison

METHOD 1: K-Means Clustering
  Execution time: 8.34 seconds
  ✅ 7 clusters found

METHOD 2: LDA Topic Modeling
  Execution time: 45.67 seconds
  Perplexity: 1234.56
  ✅ 7 topics found

METHOD 3: Hierarchical Clustering
  Execution time: 23.12 seconds
  Silhouette Score: 0.234
  ✅ 7 clusters found

METHOD 4: DBSCAN
  Execution time: 12.45 seconds
  ✅ 9 clusters found, 123 outliers

📊 COMPARISON SUMMARY
[Detailed comparison table...]

💾 Report saved to: risk_discovery_comparison_report.txt
✅ Detailed results saved to: risk_discovery_comparison_results.json

---

🎉 Summary

What Was Added

1. ✅ 3 new risk discovery methods (LDA, Hierarchical, DBSCAN)
2. ✅ Automated comparison framework
3. ✅ Comprehensive evaluation metrics
4. ✅ Method selection guidelines
5. ✅ Integration ready for training pipeline

What You Can Do Now

1. Compare Methods: Run compare_risk_discovery.py
2. Choose Best Method: Based on your data characteristics
3. Train with Alternative: Swap in any method for training
4. Ensemble Learning: Combine multiple methods (advanced)

Next Steps

1. Run comparison on your CUAD dataset
2. Review the generated report
3. Choose the method that best fits your needs:
   • Fast & Scalable → K-Means
   • Overlapping Risks → LDA
   • Risk Hierarchy → Hierarchical
   • Outlier Detection → DBSCAN
4. Update trainer.py to use chosen method
5. Train Legal-BERT with optimized risk discovery

---

Status: ✅ COMPLETE AND READY FOR TESTING
Files: risk_discovery_alternatives.py, compare_risk_discovery.py
Lines Added: ~1,020 lines of production code