Initial commit: Python Dependency Compatibility Board with ML and LLM features

.gitignore

@@ -0,0 +1,45 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+*.egg-info/
+dist/
+build/
+
+# Virtual environments
+venv/
+env/
+ENV/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Training scripts and data (not needed for deployment)
+train_conflict_model.py
+generate_embeddings.py
+Synthetic data.py
+validation_tools.py
+scripts/
+synthetic_requirements_txt/
+synthetic_requirements_dataset.json
+
+# Problem3 folder (separate project)
+problem3/
+
+# Temporary files
+*.tmp
+*.log
+
+# Model files (optional - include if you want to deploy with models)
+# Uncomment the line below if you DON'T want to include trained models
+# models/
+

ML_MODELS_README.md

@@ -0,0 +1,168 @@
+# ML Models Integration Guide
+
+This document explains how to train and use the ML models for conflict prediction and package similarity.
+
+## Overview
+
+The project includes two ML models:
+
+1. **Conflict Prediction Model**: A Random Forest classifier that predicts whether a set of dependencies will have conflicts
+2. **Package Embeddings**: Pre-computed semantic embeddings for common Python packages for similarity matching
+
+## Training the Models
+
+### Step 1: Install Training Dependencies
+
+```bash
+pip install scikit-learn sentence-transformers numpy
+```
+
+### Step 2: Train Conflict Prediction Model
+
+```bash
+cd "code to upload"
+python train_conflict_model.py
+```
+
+This will:
+- Load the synthetic dataset (`synthetic_requirements_dataset.json`)
+- Extract features from requirements
+- Train a Random Forest classifier
+- Save the model to `models/conflict_predictor.pkl`
+- Display accuracy and feature importance
+
+**Expected Output:**
+- Model size: ~2-5 MB
+- Test accuracy: ~85-95% (depending on dataset)
+
+### Step 3: Generate Package Embeddings
+
+```bash
+python generate_embeddings.py
+```
+
+This will:
+- Load a sentence transformer model
+- Generate embeddings for common Python packages
+- Save embeddings to `models/package_embeddings.json`
+- Save model info to `models/embedding_info.json`
+
+**Expected Output:**
+- Embeddings file: ~5-10 MB
+- Embedding dimension: 384
+- Number of packages: ~100+
+
+## Model Files Structure
+
+After training, you should have:
+
+```
+code to upload/
+├── models/
+│   ├── conflict_predictor.pkl      # Classification model
+│   ├── package_embeddings.json     # Pre-computed embeddings
+│   └── embedding_info.json         # Model metadata
+```
+
+## Integration in Main App
+
+The models are automatically loaded when available:
+
+1. **Conflict Prediction**: Runs before detailed analysis to provide early warnings
+2. **Package Similarity**: Enhances spell-checking with semantic matching
+
+### Features
+
+- **Graceful Fallback**: If models aren't available, the app works with rule-based methods
+- **Lazy Loading**: Models load only when needed
+- **Error Handling**: ML failures don't break the app
+
+## Usage in Code
+
+### Conflict Prediction
+
+```python
+from ml_models import ConflictPredictor
+
+predictor = ConflictPredictor()
+has_conflict, confidence = predictor.predict(requirements_text)
+
+if has_conflict:
+    print(f"Conflict predicted with {confidence:.1%} confidence")
+```
+
+### Package Similarity
+
+```python
+from ml_models import PackageEmbeddings
+
+embeddings = PackageEmbeddings()
+similar = embeddings.find_similar("numpyy", top_k=3)
+# Returns: [('numpy', 0.95), ('scipy', 0.72), ...]
+
+best_match = embeddings.get_best_match("pandaz")
+# Returns: 'pandas'
+```
+
+## Hugging Face Spaces Deployment
+
+### Option 1: Include Models in Repo
+
+1. Train models locally
+2. Commit model files to the repo
+3. Models load automatically on Spaces
+
+**Pros**: Simple, no external dependencies  
+**Cons**: Larger repo size (~10-15 MB)
+
+### Option 2: Upload to Hugging Face Hub
+
+1. Train models locally
+2. Upload to Hugging Face Hub:
+   ```python
+   from huggingface_hub import upload_file
+   upload_file("models/conflict_predictor.pkl", repo_id="your-username/conflict-predictor")
+   ```
+3. Load from Hub in app:
+   ```python
+   from huggingface_hub import hf_hub_download
+   model_path = hf_hub_download(repo_id="your-username/conflict-predictor", filename="conflict_predictor.pkl")
+   ```
+
+**Pros**: Smaller repo, version control for models  
+**Cons**: Requires internet connection at startup
+
+## Performance
+
+- **Conflict Prediction**: <10ms per prediction
+- **Embedding Lookup**: <1ms (pre-computed) or ~50ms (on-the-fly)
+- **Model Loading**: ~1-2 seconds at startup
+
+## Troubleshooting
+
+### Models Not Loading
+
+- Check that `models/` directory exists
+- Verify model files are present
+- Check file permissions
+
+### Low Prediction Accuracy
+
+- Retrain with more data
+- Adjust feature engineering
+- Try different model parameters
+
+### Embeddings Not Working
+
+- Ensure `sentence-transformers` is installed
+- Check internet connection (for first-time model download)
+- Verify embeddings file format
+
+## Future Improvements
+
+- [ ] Train on larger, real-world dataset
+- [ ] Add version-specific embeddings
+- [ ] Implement online learning
+- [ ] Add confidence intervals
+- [ ] Support for custom model paths
+

README.md

@@ -0,0 +1,140 @@
+---
+title: Python Dependency Compatibility Board
+emoji: 🐍
+colorFrom: blue
+colorTo: purple
+sdk: gradio
+sdk_version: 4.0.0
+app_file: app.py
+pinned: false
+license: mit
+---
+
+# 🐍 Python Dependency Compatibility Board
+
+A powerful tool to analyze and resolve Python package dependencies. Check for version conflicts, compatibility issues, and generate clean `requirements.txt` files.
+
+## ✨ Features
+
+- **Multiple Input Methods**: Library list, requirements.txt paste, or file upload
+- **Conflict Detection**: Automatically detects version conflicts and compatibility issues
+- **🤖 AI-Powered Explanations**: Uses LLM to generate intelligent, natural language explanations for conflicts (with fallback to rule-based)
+- **Dependency Resolution**: Uses pip's resolver to find compatible versions
+- **Environment Aware**: Configure Python version, device (CPU/GPU), and OS
+- **Analysis Modes**: Quick (top-level) or Deep (with transitive dependencies)
+- **Resolution Strategies**: Latest compatible, stable/pinned, keep existing, or minimal changes
+- **Spell Checking**: Auto-corrects common spelling mistakes in package names
+- **Validation Utilities**: Benchmark against the bundled synthetic dataset and generate perturbed requirements for stress testing
+
+## 🚀 How to Use
+
+### Input Your Dependencies
+
+You can provide dependencies in three ways:
+
+1. **Library List**: Enter package names one per line
+   ```
+   pandas
+   torch
+   langchain
+   fastapi
+   ```
+
+2. **Requirements Text**: Paste your existing requirements.txt
+   ```
+   pandas==2.0.3
+   torch>=2.0.0
+   langchain==0.1.0
+   ```
+
+3. **File Upload**: Upload a requirements.txt file directly
+
+### Configure Environment
+
+- **Python Version**: Select your target Python version (3.8-3.12)
+- **Device**: CPU only, NVIDIA GPU (CUDA), Apple Silicon (MPS), or Custom
+- **Operating System**: Any, Linux, Windows, or macOS
+
+### Analysis & Resolution
+
+1. Choose **Analysis Mode**:
+   - **Quick**: Fast analysis of top-level dependencies
+   - **Deep**: Complete dependency tree with transitive dependencies
+
+2. Select **Resolution Strategy**:
+   - **latest_compatible**: Resolve to latest compatible versions
+   - **stable/pinned**: Prefer stable, pinned versions
+   - **keep_existing_pins**: Preserve your existing version pins
+   - **minimal_changes**: Make minimal changes to resolve conflicts
+
+3. Click **"Analyze & Resolve Dependencies"**
+
+4. Review the results and download your resolved `requirements.txt`
+
+## 🔍 What It Detects
+
+The tool automatically detects:
+
+- **Duplicate Packages**: Same package specified multiple times with conflicting versions
+- **PyTorch Compatibility**: Ensures pytorch-lightning>=2.0 works with torch>=2.0
+- **FastAPI/Pydantic**: Checks version compatibility (e.g., fastapi 0.78.x requires pydantic v1)
+- **TensorFlow/Keras**: Validates TensorFlow/Keras version pairs
+- **Version Conflicts**: Identifies incompatible version specifications
+
+## 🤖 AI Explanations
+
+When enabled, the tool uses LLM reasoning to provide:
+- **Clear Explanations**: Natural language descriptions of what the conflict is
+- **Why It Happens**: Technical reasons behind the conflict
+- **How to Fix**: Actionable solutions with specific version recommendations
+
+The LLM explanations use Hugging Face Inference API (free tier) and automatically fall back to rule-based explanations if the API is unavailable.
+
+## 📋 Example
+
+**Input:**
+```
+torch==1.8.0
+pytorch-lightning==2.2.0
+pandas==2.0.3
+```
+
+**Output:**
+```
+⚠️ Compatibility Issues Found:
+- pytorch-lightning>=2.0 requires torch>=2.0, but torch<2.0 is specified
+
+Resolved requirements.txt:
+torch==2.1.0
+pytorch-lightning==2.2.0
+pandas==2.0.3
+...
+```
+
+## 🛠️ Technical Details
+
+- Built with [Gradio](https://gradio.app/)
+- Uses `packaging` library for version parsing
+- Leverages pip's dependency resolver
+- Supports PEP 508 requirement specifications
+
+## 📝 Notes
+
+- Full dependency resolution requires pip >= 22.2
+- Deep mode may take longer for large dependency sets
+- The tool works best with packages available on PyPI
+- Platform-specific dependencies (e.g., CUDA) are detected but resolution may vary
+- Run `python validation_tools.py` to benchmark the built-in compatibility checks against synthetic cases.
+- Use `python scripts/perturb_requirements.py --help` to generate noisy/invalid requirements for robustness testing.
+
+## 🤝 Contributing
+
+Feel free to test the tool and report any issues! This tool is designed to help developers manage Python dependencies more effectively.
+
+## 📄 License
+
+MIT License - feel free to use and modify as needed.
+
+---
+
+**Made with ❤️ for the Python community**

app.py

@@ -0,0 +1,1030 @@
+"""
+Python Dependency Compatibility Board
+A tool to parse, analyze, and resolve Python package dependencies.
+"""
+
+import re
+import json
+import tempfile
+import subprocess
+from pathlib import Path
+from typing import List, Dict, Tuple, Optional, Set
+from difflib import get_close_matches
+import requests
+from packaging.requirements import Requirement
+from packaging.specifiers import SpecifierSet
+from packaging.version import Version
+
+# Import ML models (with graceful fallback)
+try:
+    from ml_models import ConflictPredictor, PackageEmbeddings
+    ML_AVAILABLE = True
+except ImportError:
+    ML_AVAILABLE = False
+    print("Warning: ML models not available. Some features will be disabled.")
+
+
+class DependencyParser:
+    """Parse requirements.txt and library lists into structured dependencies."""
+    
+    @staticmethod
+    def parse_requirements_text(text: str) -> List[Dict]:
+        """Parse requirements.txt content into structured format."""
+        dependencies = []
+        seen_packages = {}
+        
+        for line in text.strip().split('\n'):
+            line = line.strip()
+            if not line or line.startswith('#'):
+                continue
+            
+            # Remove comments
+            if '#' in line:
+                line = line[:line.index('#')].strip()
+            
+            try:
+                req = Requirement(line)
+                package_name = req.name.lower()
+                
+                # Handle duplicate packages
+                if package_name in seen_packages:
+                    # Merge or warn about duplicates
+                    existing = seen_packages[package_name]
+                    if existing['specifier'] != str(req.specifier):
+                        dependencies.append({
+                            'package': package_name,
+                            'specifier': str(req.specifier) if req.specifier else '',
+                            'extras': list(req.extras) if req.extras else [],
+                            'marker': str(req.marker) if req.marker else '',
+                            'original': line,
+                            'conflict': f"Duplicate: {existing['original']} vs {line}"
+                        })
+                    continue
+                
+                dep = {
+                    'package': package_name,
+                    'specifier': str(req.specifier) if req.specifier else '',
+                    'extras': list(req.extras) if req.extras else [],
+                    'marker': str(req.marker) if req.marker else '',
+                    'original': line,
+                    'conflict': None
+                }
+                dependencies.append(dep)
+                seen_packages[package_name] = dep
+            except Exception as e:
+                # Handle malformed lines
+                dependencies.append({
+                    'package': line.split('==')[0].split('>=')[0].split('<=')[0].split('[')[0].strip(),
+                    'specifier': '',
+                    'extras': [],
+                    'marker': '',
+                    'original': line,
+                    'conflict': f"Parse error: {str(e)}"
+                })
+        
+        return dependencies
+    
+    @staticmethod
+    def parse_library_list(text: str) -> List[Dict]:
+        """Parse a simple list of library names."""
+        dependencies = []
+        for line in text.strip().split('\n'):
+            line = line.strip()
+            if not line or line.startswith('#'):
+                continue
+            
+            # Extract package name (remove version specifiers if present)
+            package_name = re.split(r'[<>=!]', line)[0].strip()
+            package_name = re.split(r'\[', package_name)[0].strip()
+            
+            if package_name:
+                dependencies.append({
+                    'package': package_name.lower(),
+                    'specifier': '',
+                    'extras': [],
+                    'marker': '',
+                    'original': package_name,
+                    'conflict': None
+                })
+        
+        return dependencies
+
+
+class DependencyResolver:
+    """Resolve dependencies and check compatibility."""
+    
+    def __init__(self, python_version: str = "3.10", platform: str = "any", device: str = "cpu"):
+        self.python_version = python_version
+        self.platform = platform
+        self.device = device
+    
+    def build_dependency_graph(self, dependencies: List[Dict], deep_mode: bool = False) -> Dict:
+        """Build dependency graph (simplified - in production would query PyPI)."""
+        graph = {
+            'nodes': {},
+            'edges': [],
+            'conflicts': []
+        }
+        
+        for dep in dependencies:
+            package = dep['package']
+            graph['nodes'][package] = {
+                'specifier': dep['specifier'],
+                'extras': dep['extras'],
+                'marker': dep['marker'],
+                'conflict': dep.get('conflict')
+            }
+            
+            if dep.get('conflict'):
+                graph['conflicts'].append({
+                    'package': package,
+                    'reason': dep['conflict']
+                })
+        
+        # In deep mode, would fetch transitive dependencies from PyPI
+        # For now, we'll use a simplified approach
+        
+        return graph
+    
+    def check_compatibility(self, graph: Dict) -> Tuple[bool, List[str]]:
+        """Check version compatibility across the graph."""
+        issues = []
+        
+        # Check for duplicate package conflicts
+        for conflict in graph['conflicts']:
+            issues.append(f"Conflict in {conflict['package']}: {conflict['reason']}")
+        
+        # Check known compatibility issues
+        nodes = graph['nodes']
+        
+        # PyTorch Lightning + PyTorch compatibility
+        if 'pytorch-lightning' in nodes and 'torch' in nodes:
+            pl_spec = nodes['pytorch-lightning']['specifier']
+            torch_spec = nodes['torch']['specifier']
+            
+            # Simplified check - in production would parse versions properly
+            if '==2.' in pl_spec or '>=2.' in pl_spec:
+                if '==1.' in torch_spec or ('<2.' in torch_spec and '==1.' in torch_spec):
+                    issues.append("pytorch-lightning>=2.0 requires torch>=2.0, but torch<2.0 is specified")
+        
+        # FastAPI + Pydantic compatibility
+        if 'fastapi' in nodes and 'pydantic' in nodes:
+            fastapi_spec = nodes['fastapi']['specifier']
+            pydantic_spec = nodes['pydantic']['specifier']
+            
+            if '==0.78' in fastapi_spec or '==0.7' in fastapi_spec:
+                if '==2.' in pydantic_spec or '>=2.' in pydantic_spec:
+                    issues.append("fastapi==0.78.x requires pydantic v1, but pydantic v2 is specified")
+        
+        # TensorFlow + Keras compatibility
+        if 'tensorflow' in nodes and 'keras' in nodes:
+            tf_spec = nodes['tensorflow']['specifier']
+            keras_spec = nodes['keras']['specifier']
+            
+            if '==1.' in tf_spec:
+                if '==3.' in keras_spec or '>=3.' in keras_spec:
+                    issues.append("keras>=3.0 requires TensorFlow 2.x, but TensorFlow 1.x is specified")
+        
+        return len(issues) == 0, issues
+    
+    def resolve_dependencies(
+        self, 
+        dependencies: List[Dict], 
+        strategy: str = "latest_compatible"
+    ) -> Tuple[str, List[str]]:
+        """Resolve dependencies using specified strategy."""
+        # Remove duplicates and conflicts
+        seen_packages = {}
+        clean_dependencies = []
+        
+        for dep in dependencies:
+            if dep.get('conflict'):
+                continue
+            
+            package = dep['package']
+            if package in seen_packages:
+                # Keep the one with more specific version if available
+                existing = seen_packages[package]
+                if dep['specifier'] and not existing['specifier']:
+                    clean_dependencies.remove(existing)
+                    clean_dependencies.append(dep)
+                    seen_packages[package] = dep
+                continue
+            
+            clean_dependencies.append(dep)
+            seen_packages[package] = dep
+        
+        # Create a temporary requirements file
+        with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
+            req_lines = []
+            for dep in clean_dependencies:
+                req_lines.append(dep['original'])
+            f.write('\n'.join(req_lines))
+            temp_req_file = f.name
+        
+        warnings = []
+        
+        try:
+            # Try using pip's resolver with --dry-run and --report (pip 22.2+)
+            result = subprocess.run(
+                ['pip', 'install', '--dry-run', '--report', '-', '-r', temp_req_file],
+                capture_output=True,
+                text=True,
+                timeout=60
+            )
+            
+            if result.returncode == 0 and result.stdout.strip():
+                # Parse the JSON report
+                try:
+                    report = json.loads(result.stdout)
+                    resolved = []
+                    for package in report.get('install', []):
+                        name = package.get('metadata', {}).get('name', '')
+                        version = package.get('metadata', {}).get('version', '')
+                        if name and version:
+                            resolved.append(f"{name}=={version}")
+                    
+                    if resolved:
+                        return '\n'.join(sorted(resolved)), warnings
+                except json.JSONDecodeError:
+                    warnings.append("Could not parse pip resolution report. Using original requirements.")
+                except Exception as e:
+                    warnings.append(f"Error parsing resolution: {str(e)}")
+            
+            # Fallback: try pip-compile if available
+            try:
+                result = subprocess.run(
+                    ['pip-compile', '--dry-run', '--output-file', '-', temp_req_file],
+                    capture_output=True,
+                    text=True,
+                    timeout=60
+                )
+                if result.returncode == 0:
+                    return result.stdout.strip(), warnings
+            except FileNotFoundError:
+                pass
+            except Exception:
+                pass
+            
+            # Final fallback: return cleaned original requirements
+            resolved_lines = []
+            for dep in clean_dependencies:
+                line = dep['original']
+                # Apply strategy-based modifications
+                if strategy == "stable/pinned" and not dep['specifier']:
+                    # In a real implementation, would query PyPI for latest stable
+                    line = f"{dep['package']}  # Version not specified"
+                elif strategy == "keep_existing_pins":
+                    # Keep as-is
+                    pass
+                resolved_lines.append(line)
+            
+            if not warnings:
+                warnings.append("Using original requirements. For full resolution, ensure pip>=22.2 is installed.")
+            
+            return '\n'.join(resolved_lines), warnings
+        
+        except subprocess.TimeoutExpired:
+            warnings.append("Resolution timed out. Showing original requirements.")
+            return '\n'.join([d['original'] for d in clean_dependencies]), warnings
+        except Exception as e:
+            warnings.append(f"Resolution error: {str(e)}")
+            return '\n'.join([d['original'] for d in clean_dependencies]), warnings
+        finally:
+            Path(temp_req_file).unlink(missing_ok=True)
+
+
+class CatalogValidator:
+    """Validate package names against a simple ground-truth catalog."""
+
+    def __init__(self, catalog_path: Path = Path("data/package_name_catalog.json"), use_ml: bool = True):
+        self.catalog_path = catalog_path
+        self.valid_packages: Set[str] = set()
+        self.invalid_packages: Set[str] = set()
+        self.use_ml = use_ml and ML_AVAILABLE
+        self.embeddings = None
+        
+        self._load_catalog()
+        
+        # Load embeddings if available
+        if self.use_ml:
+            try:
+                self.embeddings = PackageEmbeddings()
+            except Exception as e:
+                print(f"Warning: Could not load embeddings: {e}")
+                self.use_ml = False
+
+    def _load_catalog(self) -> None:
+        if not self.catalog_path.exists():
+            return
+        try:
+            data = json.loads(self.catalog_path.read_text())
+            self.valid_packages = {p.lower() for p in data.get("valid_packages", [])}
+            self.invalid_packages = {p.lower() for p in data.get("invalid_packages", [])}
+        except Exception as exc:
+            # Keep going even if catalog is malformed
+            print(f"Warning: could not read catalog {self.catalog_path}: {exc}")
+
+    def suggest_correction(self, package_name: str, cutoff: float = 0.6) -> Optional[str]:
+        """Suggest a corrected package name using fuzzy matching and embeddings."""
+        if not self.valid_packages:
+            return None
+        
+        package_lower = package_name.lower()
+        
+        # If it's already valid, no correction needed
+        if package_lower in self.valid_packages:
+            return None
+        
+        # Try ML-based embedding similarity first (more accurate)
+        if self.use_ml and self.embeddings:
+            try:
+                best_match = self.embeddings.get_best_match(package_name, threshold=0.7)
+                if best_match and best_match in self.valid_packages:
+                    return best_match
+            except Exception:
+                pass
+        
+        # Fallback to fuzzy matching
+        matches = get_close_matches(
+            package_lower,
+            list(self.valid_packages),
+            n=1,
+            cutoff=cutoff
+        )
+        
+        if matches:
+            return matches[0]
+        return None
+    
+    def check_and_correct_packages(self, dependencies: List[Dict], auto_correct: bool = True) -> Tuple[List[Dict], List[str]]:
+        """Check packages and optionally correct spelling mistakes.
+        
+        Returns:
+            Tuple of (corrected_dependencies, warnings)
+        """
+        corrected_deps = []
+        warnings: List[str] = []
+        seen: Set[str] = set()
+        max_warnings = 15
+
+        for dep in dependencies:
+            package = dep["package"]
+            package_lower = package.lower()
+            
+            if package_lower in seen:
+                corrected_deps.append(dep)
+                continue
+            seen.add(package_lower)
+
+            # Check if it's explicitly invalid
+            if self.invalid_packages and package_lower in self.invalid_packages:
+                warnings.append(f"Package '{package}' is flagged as invalid in the catalog.")
+                if len(warnings) >= max_warnings:
+                    corrected_deps.append(dep)
+                    continue
+                
+                # Try to suggest a correction
+                suggestion = self.suggest_correction(package)
+                if suggestion:
+                    if auto_correct:
+                        corrected_dep = dep.copy()
+                        corrected_dep['package'] = suggestion
+                        corrected_dep['original'] = corrected_dep['original'].replace(package, suggestion, 1)
+                        corrected_deps.append(corrected_dep)
+                        warnings.append(f"  → Auto-corrected to '{suggestion}'")
+                    else:
+                        warnings.append(f"  → Did you mean '{suggestion}'?")
+                else:
+                    corrected_deps.append(dep)
+                continue
+
+            # Check if it's not in valid catalog and suggest correction
+            if self.valid_packages and package_lower not in self.valid_packages:
+                suggestion = self.suggest_correction(package)
+                if suggestion:
+                    if auto_correct:
+                        corrected_dep = dep.copy()
+                        corrected_dep['package'] = suggestion
+                        corrected_dep['original'] = corrected_dep['original'].replace(package, suggestion, 1)
+                        corrected_deps.append(corrected_dep)
+                        warnings.append(f"Package '{package}' not found. Auto-corrected to '{suggestion}'")
+                    else:
+                        warnings.append(f"Package '{package}' not found. Did you mean '{suggestion}'?")
+                    if len(warnings) >= max_warnings:
+                        break
+                else:
+                    warnings.append(
+                        f"Package '{package}' is not in the curated valid catalog. Check for typos or private packages."
+                    )
+                    corrected_deps.append(dep)
+                    if len(warnings) >= max_warnings:
+                        break
+            else:
+                # Package is valid, keep as-is
+                corrected_deps.append(dep)
+
+        if len(warnings) >= max_warnings:
+            warnings.append("Additional potential catalog issues omitted for brevity.")
+
+        return corrected_deps, warnings
+    
+    def check_packages(self, dependencies: List[Dict]) -> List[str]:
+        """Return warnings for packages that look suspicious or explicitly invalid."""
+        _, warnings = self.check_and_correct_packages(dependencies, auto_correct=False)
+        return warnings
+
+
+class ExplanationEngine:
+    """Generate intelligent explanations for dependency conflicts using LLM."""
+    
+    def __init__(self, use_llm: bool = True):
+        """
+        Initialize explanation engine.
+        
+        Args:
+            use_llm: If True, uses Hugging Face Inference API (free tier)
+                    If False, uses rule-based explanations only
+        """
+        self.use_llm = use_llm
+        # Using Hugging Face Inference API (free tier)
+        self.api_url = "https://api-inference.huggingface.co/models/gpt2"
+        self.headers = {"Content-Type": "application/json"}
+    
+    def generate_explanation(self, conflict: Dict, dependencies: List[Dict]) -> Dict:
+        """
+        Generate a detailed explanation for a conflict.
+        
+        Args:
+            conflict: Conflict dictionary with type, packages, message, etc.
+            dependencies: Full list of dependencies for context
+            
+        Returns:
+            Dictionary with explanation, why_it_happens, how_to_fix
+        """
+        # Build context about the conflict
+        conflict_type = conflict.get('type', 'unknown')
+        packages = conflict.get('packages', [conflict.get('package', 'unknown')])
+        message = conflict.get('message', '')
+        details = conflict.get('details', {})
+        
+        # Create prompt for LLM
+        prompt = self._create_prompt(conflict, dependencies)
+        
+        # Get LLM explanation
+        explanation_text = self._call_llm(prompt) if self.use_llm else self._fallback_explanation(prompt)
+        
+        # Parse and structure the explanation
+        return {
+            'summary': message,
+            'explanation': explanation_text,
+            'why_it_happens': self._extract_why(explanation_text, conflict),
+            'how_to_fix': self._extract_fix(explanation_text, conflict),
+            'packages_involved': packages,
+            'severity': conflict.get('severity', 'medium')
+        }
+    
+    def _create_prompt(self, conflict: Dict, dependencies: List[Dict]) -> str:
+        """Create a prompt for the LLM."""
+        conflict_type = conflict.get('type', 'unknown')
+        packages = conflict.get('packages', [conflict.get('package', 'unknown')])
+        message = conflict.get('message', '')
+        details = conflict.get('details', {})
+        
+        # Get relevant dependency info
+        relevant_deps = [d for d in dependencies if d['package'] in packages]
+        
+        prompt = f"""You are a Python dependency expert. Explain this dependency conflict clearly:
+
+Conflict: {message}
+Type: {conflict_type}
+Packages involved: {', '.join(packages)}
+
+Dependency details:
+"""
+        for dep in relevant_deps:
+            prompt += f"- {dep['package']}: {dep['specifier'] or 'no version specified'}\n"
+        
+        if details:
+            prompt += f"\nVersion constraints: {json.dumps(details)}\n"
+        
+        prompt += """
+Provide a clear, concise explanation that:
+1. Explains what the conflict is in simple terms
+2. Explains why this conflict happens (technical reason)
+3. Suggests how to fix it (specific version recommendations)
+
+Keep it under 150 words and use plain language.
+"""
+        return prompt
+    
+    def _call_llm(self, prompt: str) -> str:
+        """
+        Call LLM API to generate explanation.
+        Falls back to rule-based explanation if API fails.
+        """
+        try:
+            # Try Hugging Face Inference API (free tier)
+            payload = {
+                "inputs": prompt,
+                "parameters": {
+                    "max_new_tokens": 200,
+                    "temperature": 0.7,
+                    "return_full_text": False
+                }
+            }
+            
+            response = requests.post(
+                self.api_url,
+                headers=self.headers,
+                json=payload,
+                timeout=10
+            )
+            
+            if response.status_code == 200:
+                result = response.json()
+                if isinstance(result, list) and len(result) > 0:
+                    generated_text = result[0].get('generated_text', '')
+                    if generated_text:
+                        return generated_text.strip()
+            
+            # If API fails, fall back to rule-based
+            return self._fallback_explanation(prompt)
+        
+        except Exception as e:
+            # Fall back to rule-based explanation
+            return self._fallback_explanation(prompt)
+    
+    def _fallback_explanation(self, prompt: str) -> str:
+        """Generate rule-based explanation when LLM is unavailable."""
+        # Extract key info from prompt
+        if "pytorch-lightning" in prompt.lower() and "torch" in prompt.lower():
+            return """PyTorch Lightning 2.0+ requires PyTorch 2.0 or higher because it uses new PyTorch APIs and features that don't exist in version 1.x. The conflict happens because you're trying to use a newer version of PyTorch Lightning with an older version of PyTorch. To fix this, either upgrade PyTorch to 2.0+ or downgrade PyTorch Lightning to 1.x."""
+        
+        elif "fastapi" in prompt.lower() and "pydantic" in prompt.lower():
+            return """FastAPI 0.78.x was built for Pydantic v1, which has a different API than Pydantic v2. The conflict occurs because Pydantic v2 introduced breaking changes that FastAPI 0.78 doesn't support. To fix this, either upgrade FastAPI to 0.99+ (which supports Pydantic v2) or downgrade Pydantic to v1.x."""
+        
+        elif "tensorflow" in prompt.lower() and "keras" in prompt.lower():
+            return """Keras 3.0+ requires TensorFlow 2.x because it was redesigned to work with TensorFlow 2's eager execution and new features. TensorFlow 1.x uses a different execution model that Keras 3.0 doesn't support. To fix this, upgrade TensorFlow to 2.x or downgrade Keras to 2.x."""
+        
+        elif "duplicate" in prompt.lower():
+            return """You have the same package specified multiple times with different versions. This creates ambiguity about which version should be installed. To fix this, remove duplicate entries and keep only one version specification per package."""
+        
+        else:
+            return """This dependency conflict occurs due to incompatible version requirements between packages. Review the version constraints and ensure all packages are compatible with each other. Consider updating to compatible versions or using a dependency resolver."""
+    
+    def _extract_why(self, explanation: str, conflict: Dict) -> str:
+        """Extract the 'why it happens' part from explanation."""
+        # Simple extraction - look for sentences explaining the reason
+        sentences = explanation.split('.')
+        why_sentences = [s.strip() for s in sentences if any(word in s.lower() for word in ['because', 'due to', 'requires', 'needs', 'since'])]
+        return '. '.join(why_sentences[:2]) + '.' if why_sentences else "Version constraints are incompatible."
+    
+    def _extract_fix(self, explanation: str, conflict: Dict) -> str:
+        """Extract the 'how to fix' part from explanation."""
+        # Simple extraction - look for fix suggestions
+        sentences = explanation.split('.')
+        fix_sentences = [s.strip() for s in sentences if any(word in s.lower() for word in ['upgrade', 'downgrade', 'fix', 'change', 'update', 'remove'])]
+        return '. '.join(fix_sentences[:2]) + '.' if fix_sentences else "Adjust version constraints to compatible versions."
+
+
+def process_dependencies(
+    library_list: str,
+    requirements_text: str,
+    uploaded_file,
+    python_version: str,
+    device: str,
+    os_type: str,
+    mode: str,
+    resolution_strategy: str,
+    use_llm_explanations: bool = True,
+    use_ml_prediction: bool = True,
+    use_ml_spellcheck: bool = True,
+    show_ml_details: bool = False
+) -> Tuple[str, str, str]:
+    """Main processing function for Gradio interface."""
+    
+    # Collect dependencies from all sources
+    all_dependencies = []
+    
+    # Parse library list
+    if library_list:
+        parser = DependencyParser()
+        deps = parser.parse_library_list(library_list)
+        all_dependencies.extend(deps)
+    
+    # Parse requirements text
+    if requirements_text:
+        parser = DependencyParser()
+        deps = parser.parse_requirements_text(requirements_text)
+        all_dependencies.extend(deps)
+    
+    # Parse uploaded file
+    if uploaded_file:
+        try:
+            # Handle both string paths and file objects (Gradio 6.x compatibility)
+            if isinstance(uploaded_file, str):
+                file_path = uploaded_file
+            else:
+                # If it's a file object, get the path
+                file_path = uploaded_file.name if hasattr(uploaded_file, 'name') else str(uploaded_file)
+            
+            with open(file_path, 'r') as f:
+                content = f.read()
+            parser = DependencyParser()
+            deps = parser.parse_requirements_text(content)
+            all_dependencies.extend(deps)
+        except Exception as e:
+            return f"Error reading file: {str(e)}", "", ""
+    
+    if not all_dependencies:
+        return "Please provide at least one input: library list, requirements text, or uploaded file.", "", ""
+    
+    catalog_validator = CatalogValidator(use_ml=use_ml_spellcheck and ML_AVAILABLE)
+    # Auto-correct spelling mistakes in package names
+    all_dependencies, catalog_warnings = catalog_validator.check_and_correct_packages(all_dependencies, auto_correct=True)
+
+    # ML-based conflict prediction (pre-analysis)
+    ml_conflict_prediction = None
+    ml_confidence = 0.0
+    ml_details = ""
+    if use_ml_prediction and ML_AVAILABLE:
+        try:
+            predictor = ConflictPredictor()
+            requirements_text_for_ml = '\n'.join([d['original'] for d in all_dependencies])
+            has_conflict, confidence = predictor.predict(requirements_text_for_ml)
+            ml_conflict_prediction = has_conflict
+            ml_confidence = confidence
+            
+            # Build ML details output
+            ml_details = f"""
+### ML Model Details
+
+**Conflict Prediction Model:**
+- Prediction: {"Conflict Detected" if has_conflict else "No Conflict"}
+- Confidence: {confidence:.2%}
+- Model Type: Random Forest Classifier
+- Features Analyzed: Package presence, version specificity, conflict patterns
+
+"""
+            if show_ml_details:
+                # Get feature importance or additional details
+                ml_details += f"""
+**Raw Prediction:**
+- Has Conflict: {has_conflict}
+- Confidence Score: {confidence:.4f}
+- Probability Distribution: Conflict={confidence:.2%}, No Conflict={1-confidence:.2%}
+
+"""
+            
+            if has_conflict and confidence > 0.7:
+                catalog_warnings.append(
+                    f"ML Prediction: High probability ({confidence:.1%}) of conflicts detected"
+                )
+        except Exception as e:
+            print(f"ML prediction error: {e}")
+            ml_details = f"ML Prediction Error: {str(e)}"
+    elif use_ml_prediction and not ML_AVAILABLE:
+        ml_details = "ML models not available. Train models using `train_conflict_model.py` to enable this feature."
+
+    # Build dependency graph
+    resolver = DependencyResolver(python_version=python_version, platform=os_type, device=device)
+    deep_mode = (mode == "Deep (with transitive dependencies)")
+    graph = resolver.build_dependency_graph(all_dependencies, deep_mode=deep_mode)
+    
+    # Check compatibility
+    is_compatible, issues = resolver.check_compatibility(graph)
+    
+    # Convert string issues to structured format for LLM explanations
+    structured_issues = []
+    for issue in issues:
+        if isinstance(issue, str):
+            # Parse the issue string to extract package names and type
+            issue_dict = {
+                'type': 'version_incompatibility',
+                'message': issue,
+                'severity': 'high',
+                'details': {}
+            }
+            
+            # Extract package names from known patterns
+            packages = []
+            issue_lower = issue.lower()
+            
+            # Check for specific known conflicts
+            if 'pytorch-lightning' in issue_lower and 'torch' in issue_lower:
+                packages = ['pytorch-lightning', 'torch']
+                issue_dict['type'] = 'version_incompatibility'
+                # Extract version details
+                for dep in all_dependencies:
+                    if dep['package'] in packages:
+                        issue_dict['details'][dep['package']] = dep.get('specifier', '')
+            elif 'fastapi' in issue_lower and 'pydantic' in issue_lower:
+                packages = ['fastapi', 'pydantic']
+                issue_dict['type'] = 'version_incompatibility'
+                for dep in all_dependencies:
+                    if dep['package'] in packages:
+                        issue_dict['details'][dep['package']] = dep.get('specifier', '')
+            elif 'tensorflow' in issue_lower and 'keras' in issue_lower:
+                packages = ['tensorflow', 'keras']
+                issue_dict['type'] = 'version_incompatibility'
+                for dep in all_dependencies:
+                    if dep['package'] in packages:
+                        issue_dict['details'][dep['package']] = dep.get('specifier', '')
+            elif 'conflict in' in issue_lower:
+                # Duplicate package conflict
+                pkg = issue.split('Conflict in')[1].split(':')[0].strip()
+                packages = [pkg]
+                issue_dict['type'] = 'duplicate'
+                issue_dict['package'] = pkg
+            else:
+                # Generic: try to find packages mentioned in the issue
+                for dep in all_dependencies:
+                    if dep['package'] in issue_lower:
+                        packages.append(dep['package'])
+            
+            if packages:
+                issue_dict['packages'] = packages
+            else:
+                issue_dict['package'] = 'unknown'
+                issue_dict['packages'] = []
+            
+            structured_issues.append(issue_dict)
+        else:
+            structured_issues.append(issue)
+    
+    # Generate LLM explanations if enabled
+    explanations = []
+    if use_llm_explanations and structured_issues:
+        explanation_engine = ExplanationEngine(use_llm=use_llm_explanations)
+        for issue in structured_issues:
+            try:
+                explanation = explanation_engine.generate_explanation(issue, all_dependencies)
+                explanations.append(explanation)
+            except Exception as e:
+                # If explanation generation fails, just use the issue message
+                explanations.append({
+                    'summary': issue.get('message', str(issue)),
+                    'explanation': issue.get('message', str(issue)),
+                    'why_it_happens': 'Unable to generate explanation.',
+                    'how_to_fix': 'Review version constraints.',
+                    'packages_involved': issue.get('packages', []),
+                    'severity': issue.get('severity', 'medium')
+                })
+    
+    # Resolve dependencies
+    resolved_text, resolver_warnings = resolver.resolve_dependencies(all_dependencies, resolution_strategy)
+    warnings = catalog_warnings + resolver_warnings
+    
+    # Build output message
+    output_parts = []
+    output_parts.append("## Dependency Analysis Results\n\n")
+    
+    # Show ML prediction if available
+    if ML_AVAILABLE and ml_conflict_prediction is not None:
+        if ml_conflict_prediction:
+            output_parts.append(f"### ML Prediction: Potential Conflicts Detected (Confidence: {ml_confidence:.1%})\n\n")
+        else:
+            output_parts.append(f"### ML Prediction: Low Conflict Risk (Confidence: {ml_confidence:.1%})\n\n")
+    
+    if issues:
+        output_parts.append("### Compatibility Issues Found:\n")
+        if explanations:
+            # Show detailed LLM explanations
+            for i, (issue, explanation) in enumerate(zip(issues, explanations), 1):
+                output_parts.append(f"#### Issue #{i}: {explanation['summary']}\n\n")
+                output_parts.append(f"**Explanation:**\n{explanation['explanation']}\n\n")
+                output_parts.append(f"**Why this happens:**\n{explanation['why_it_happens']}\n\n")
+                output_parts.append(f"**How to fix:**\n{explanation['how_to_fix']}\n\n")
+                output_parts.append("---\n\n")
+        else:
+            # Fallback to simple list
+            for issue in issues:
+                output_parts.append(f"- {issue}\n")
+        output_parts.append("\n")
+    
+    # Separate corrections from other warnings
+    corrections = [w for w in warnings if "Auto-corrected" in w or "→" in w]
+    other_warnings = [w for w in warnings if w not in corrections]
+    
+    if corrections:
+        output_parts.append("### Spelling Corrections:\n")
+        for correction in corrections:
+            output_parts.append(f"- {correction}\n")
+        output_parts.append("\n")
+    
+    if other_warnings:
+        output_parts.append("### Warnings:\n")
+        for warning in other_warnings:
+            output_parts.append(f"- {warning}\n")
+        output_parts.append("\n")
+    
+    if is_compatible and not issues:
+        output_parts.append("### No compatibility issues detected!\n\n")
+    
+    output_parts.append(f"### Resolved Requirements ({len(all_dependencies)} packages):\n")
+    output_parts.append("```\n")
+    output_parts.append(resolved_text)
+    output_parts.append("\n```\n")
+    
+    # Add ML details if requested
+    if show_ml_details and ml_details:
+        output_parts.append(ml_details)
+    
+    return ''.join(output_parts), resolved_text, ml_details
+
+
+# Gradio Interface
+def create_interface():
+    """Create and return the Gradio interface."""
+    import gradio as gr
+    
+    with gr.Blocks(title="Python Dependency Compatibility Board") as app:
+        gr.Markdown("""
+        # Python Dependency Compatibility Board
+        
+        Analyze and resolve Python package dependencies with **AI-powered explanations** and **ML-based conflict prediction**.
+        
+        ## Key Features
+        
+        | Feature | Status | Description |
+        |---------|--------|-------------|
+        | **LLM Reasoning** | Active | AI-powered natural language explanations for conflicts |
+        | **ML Conflict Prediction** | {"Available" if ML_AVAILABLE else "Not Loaded"} | Machine learning model predicts conflicts before analysis |
+        | **Embedding-Based Spell Check** | {"Available" if ML_AVAILABLE else "Not Loaded"} | Semantic similarity matching for package names |
+        | **Auto-Correction** | Active | Automatically fixes spelling mistakes in package names |
+        | **Dependency Resolution** | Active | Resolves conflicts using pip's resolver |
+        
+        """)
+        
+        with gr.Row():
+            with gr.Column(scale=1):
+                gr.Markdown("### Input Methods")
+                
+                library_input = gr.Textbox(
+                    label="Library Names (one per line)",
+                    placeholder="pandas\ntorch\nlangchain\nfastapi",
+                    lines=5,
+                    info="Enter package names, one per line"
+                )
+                
+                requirements_input = gr.Textbox(
+                    label="Requirements.txt Content",
+                    placeholder="pandas==2.0.3\ntorch>=2.0.0\nlangchain==0.1.0",
+                    lines=10,
+                    info="Paste your requirements.txt content here"
+                )
+                
+                file_upload = gr.File(
+                    label="Upload requirements.txt",
+                    file_types=[".txt"]
+                )
+            
+            with gr.Column(scale=1):
+                gr.Markdown("### Environment Settings")
+                
+                python_version = gr.Dropdown(
+                    choices=["3.8", "3.9", "3.10", "3.11", "3.12"],
+                    value="3.10",
+                    label="Python Version",
+                    info="Target Python version"
+                )
+                
+                device = gr.Dropdown(
+                    choices=["CPU only", "NVIDIA GPU (CUDA)", "Apple Silicon (MPS)", "Custom / other"],
+                    value="CPU only",
+                    label="Device",
+                    info="Target device/platform"
+                )
+                
+                os_type = gr.Dropdown(
+                    choices=["Any / generic", "Linux (x86_64)", "Windows (x86_64)", "MacOS (Intel)", "MacOS (Apple Silicon)"],
+                    value="Any / generic",
+                    label="Operating System",
+                    info="Target operating system"
+                )
+                
+                mode = gr.Radio(
+                    choices=["Quick (top-level only)", "Deep (with transitive dependencies)"],
+                    value="Quick (top-level only)",
+                    label="Analysis Mode",
+                    info="Quick mode is faster, Deep mode includes all dependencies"
+                )
+                
+                resolution_strategy = gr.Dropdown(
+                    choices=["latest_compatible", "stable/pinned", "keep_existing_pins", "minimal_changes"],
+                    value="latest_compatible",
+                    label="Resolution Strategy",
+                    info="How to resolve version conflicts"
+                )
+                
+                gr.Markdown("---")
+                gr.Markdown("### AI & ML Features")
+                
+                use_llm = gr.Checkbox(
+                    label="**LLM Reasoning** - AI Explanations",
+                    value=True,
+                    info="Generate intelligent, natural language explanations for conflicts using LLM"
+                )
+                
+                use_ml_prediction = gr.Checkbox(
+                    label="**ML Conflict Prediction**",
+                    value=True,
+                    info=f"{'Model available - Predicts conflicts before detailed analysis' if ML_AVAILABLE else 'Model not loaded - Train models to enable'}"
+                )
+                
+                use_ml_spellcheck = gr.Checkbox(
+                    label="**ML Spell Check** (Embedding-based)",
+                    value=True,
+                    info=f"{'Model available - Uses semantic similarity for better corrections' if ML_AVAILABLE else 'Model not loaded - Train models to enable'}"
+                )
+                
+                show_ml_details = gr.Checkbox(
+                    label="Show ML Model Details",
+                    value=False,
+                    info="Display raw ML predictions and confidence scores"
+                )
+                
+                process_btn = gr.Button("Analyze & Resolve Dependencies", variant="primary", size="lg")
+        
+        with gr.Row():
+            output_display = gr.Markdown(
+                label="Analysis Results",
+                value="Results will appear here after processing..."
+            )
+        
+        with gr.Row():
+            with gr.Column(scale=2):
+                resolved_output = gr.Textbox(
+                    label="Resolved requirements.txt",
+                    lines=15,
+                    info="Copy this content to use as your requirements.txt file"
+                )
+                
+                download_btn = gr.File(
+                    label="Download requirements.txt",
+                    value=None,
+                    visible=True
+                )
+            
+            with gr.Column(scale=1):
+                ml_output = gr.Markdown(
+                    label="ML Model Output",
+                    value="ML predictions will appear here when enabled...",
+                    visible=True
+                )
+        
+        def process_and_download(*args):
+            # Extract all arguments
+            result_text, resolved_text, ml_details = process_dependencies(*args)
+            
+            # Create a temporary file for download
+            temp_file = None
+            if resolved_text and resolved_text.strip():
+                try:
+                    with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
+                        f.write(resolved_text)
+                        temp_file = f.name
+                except Exception as e:
+                    print(f"Error creating download file: {e}")
+            
+            # Format ML output
+            ml_output_text = ml_details if ml_details else "ML features disabled or models not available."
+            
+            return result_text, resolved_text, temp_file if temp_file else None, ml_output_text
+        
+        process_btn.click(
+            fn=process_and_download,
+            inputs=[library_input, requirements_input, file_upload, python_version, device, os_type, mode, resolution_strategy, use_llm, use_ml_prediction, use_ml_spellcheck, show_ml_details],
+            outputs=[output_display, resolved_output, download_btn, ml_output]
+        )
+        
+        gr.Markdown("""
+        ---
+        ### How to Use
+        
+        1. **Input your dependencies** using any of the three methods (or combine them)
+        2. **Configure your environment** (Python version, device, OS)
+        3. **Choose analysis mode**: Quick for fast results, Deep for complete dependency tree
+        4. **Select resolution strategy**: How to handle version conflicts
+        5. **Click "Analyze & Resolve Dependencies"**
+        6. **Review the results** and download the resolved requirements.txt
+        
+        ### Features
+        
+        - Parse multiple input formats
+        - Detect version conflicts
+        - Check compatibility across dependency graph
+        - Resolve dependencies using pip
+        - Generate clean, pip-compatible requirements.txt
+        - Environment-aware (Python version, platform, device)
+        """)
+    
+    return app
+
+
+if __name__ == "__main__":
+    app = create_interface()
+    # For Hugging Face Spaces, use default launch settings
+    # For local development, you can customize
+    app.launch()

data/ground_truth/gt_1 copy.txt

@@ -0,0 +1,6 @@
+torch
+torchvision
+torchvision.transforms as transforms
+torch.utils.data import DataLoader
+numpy as np
+scipy import stats

data/ground_truth/gt_1.txt

@@ -0,0 +1,6 @@
+torch
+torchvision
+torchvision.transforms as transforms
+torch.utils.data import DataLoader
+numpy
+scipy

data/package_name_catalog.json

@@ -0,0 +1,47 @@
+{
+  "valid_packages": [
+    "numpy",
+    "pandas",
+    "scipy",
+    "scikit-learn",
+    "pydantic",
+    "fastapi",
+    "torch",
+    "pytorch-lightning",
+    "tensorflow",
+    "keras",
+    "pillow",
+    "requests",
+    "httpx",
+    "langchain",
+    "openai",
+    "chromadb",
+    "uvicorn",
+    "starlette",
+    "sqlalchemy",
+    "alembic",
+    "redis"
+  ],
+  "invalid_packages": [
+    "numpyy",
+    "pandaz",
+    "scipy-pro",
+    "fastapi-pro",
+    "torchx",
+    "pytorch-brightning",
+    "tensorflower",
+    "kerras",
+    "pillow2",
+    "requests3",
+    "httxx",
+    "langchainz",
+    "opena1",
+    "chromad",
+    "uvicornx",
+    "starlite",
+    "sqalachemy",
+    "alembico",
+    "redis-plus",
+    "fakerlib"
+  ]
+}

ml_models.py

@@ -0,0 +1,217 @@
+"""
+ML Model Loader and Utilities
+Handles loading and using the conflict prediction model and package embeddings.
+"""
+
+import json
+import pickle
+from pathlib import Path
+from typing import Dict, List, Tuple, Optional
+import numpy as np
+from packaging.requirements import Requirement
+
+
+class ConflictPredictor:
+    """Load and use the conflict prediction model."""
+    
+    def __init__(self, model_path: Optional[Path] = None):
+        """Initialize the conflict predictor."""
+        if model_path is None:
+            model_path = Path(__file__).parent / "models" / "conflict_predictor.pkl"
+        
+        self.model = None
+        self.model_path = model_path
+        
+        if model_path.exists():
+            try:
+                with open(model_path, 'rb') as f:
+                    self.model = pickle.load(f)
+                print(f"✅ Loaded conflict prediction model from {model_path}")
+            except Exception as e:
+                print(f"⚠️ Could not load conflict prediction model: {e}")
+        else:
+            print(f"⚠️ Conflict prediction model not found at {model_path}")
+    
+    def extract_features(self, requirements_text: str) -> np.ndarray:
+        """Extract features from requirements text (same as training)."""
+        features = []
+        
+        packages = {}
+        lines = requirements_text.strip().split('\n')
+        num_packages = 0
+        has_pins = 0
+        version_specificity = []
+        
+        for line in lines:
+            line = line.strip()
+            if not line or line.startswith('#'):
+                continue
+            
+            try:
+                req = Requirement(line)
+                pkg_name = req.name.lower()
+                specifier = str(req.specifier) if req.specifier else ''
+                
+                if pkg_name in packages:
+                    features.append(1)  # has_duplicate flag
+                else:
+                    packages[pkg_name] = specifier
+                    num_packages += 1
+                    
+                    if specifier:
+                        has_pins += 1
+                        if '==' in specifier:
+                            version_specificity.append(3)
+                        elif '>=' in specifier or '<=' in specifier:
+                            version_specificity.append(2)
+                        else:
+                            version_specificity.append(1)
+                    else:
+                        version_specificity.append(0)
+            except:
+                pass
+        
+        feature_vec = []
+        feature_vec.append(min(num_packages / 20.0, 1.0))
+        feature_vec.append(has_pins / max(num_packages, 1))
+        feature_vec.append(np.mean(version_specificity) / 3.0 if version_specificity else 0)
+        feature_vec.append(1 if len(packages) < num_packages else 0)
+        
+        common_packages = [
+            'torch', 'pytorch-lightning', 'tensorflow', 'keras', 'fastapi', 'pydantic',
+            'numpy', 'pandas', 'scipy', 'scikit-learn', 'matplotlib', 'seaborn',
+            'requests', 'httpx', 'sqlalchemy', 'alembic', 'uvicorn', 'starlette',
+            'langchain', 'openai', 'chromadb', 'redis', 'celery', 'gunicorn',
+            'pillow', 'opencv-python', 'beautifulsoup4', 'scrapy', 'plotly', 'jax'
+        ]
+        
+        for pkg in common_packages:
+            feature_vec.append(1 if pkg in packages else 0)
+        
+        has_torch = 'torch' in packages
+        has_pl = 'pytorch-lightning' in packages
+        has_tf = 'tensorflow' in packages
+        has_keras = 'keras' in packages
+        has_fastapi = 'fastapi' in packages
+        has_pydantic = 'pydantic' in packages
+        
+        feature_vec.append(1 if (has_torch and has_pl) else 0)
+        feature_vec.append(1 if (has_tf and has_keras) else 0)
+        feature_vec.append(1 if (has_fastapi and has_pydantic) else 0)
+        
+        return np.array(feature_vec)
+    
+    def predict(self, requirements_text: str) -> Tuple[bool, float]:
+        """
+        Predict if requirements have conflicts.
+        
+        Returns:
+            (has_conflict, confidence_score)
+        """
+        if self.model is None:
+            return False, 0.0
+        
+        try:
+            features = self.extract_features(requirements_text)
+            features = features.reshape(1, -1)
+            
+            prediction = self.model.predict(features)[0]
+            probability = self.model.predict_proba(features)[0]
+            
+            has_conflict = bool(prediction)
+            confidence = float(probability[1] if has_conflict else probability[0])
+            
+            return has_conflict, confidence
+        except Exception as e:
+            print(f"Error in conflict prediction: {e}")
+            return False, 0.0
+
+
+class PackageEmbeddings:
+    """Load and use package embeddings for similarity matching."""
+    
+    def __init__(self, embeddings_path: Optional[Path] = None):
+        """Initialize package embeddings."""
+        if embeddings_path is None:
+            embeddings_path = Path(__file__).parent / "models" / "package_embeddings.json"
+        
+        self.embeddings = {}
+        self.embeddings_path = embeddings_path
+        self.model = None
+        
+        if embeddings_path.exists():
+            try:
+                with open(embeddings_path, 'r') as f:
+                    self.embeddings = json.load(f)
+                print(f"✅ Loaded {len(self.embeddings)} package embeddings from {embeddings_path}")
+            except Exception as e:
+                print(f"⚠️ Could not load embeddings: {e}")
+        else:
+            print(f"⚠️ Embeddings not found at {embeddings_path}")
+    
+    def _load_model(self):
+        """Lazy load the sentence transformer model."""
+        if self.model is None:
+            try:
+                from sentence_transformers import SentenceTransformer
+                self.model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
+            except ImportError:
+                print("⚠️ sentence-transformers not available, embedding similarity disabled")
+                return None
+        return self.model
+    
+    def get_embedding(self, package_name: str) -> Optional[np.ndarray]:
+        """Get embedding for a package (from cache or compute on-the-fly)."""
+        package_lower = package_name.lower()
+        
+        # Check cache first
+        if package_lower in self.embeddings:
+            return np.array(self.embeddings[package_lower])
+        
+        # Compute on-the-fly if model available
+        model = self._load_model()
+        if model is not None:
+            embedding = model.encode([package_name])[0]
+            # Cache it
+            self.embeddings[package_lower] = embedding.tolist()
+            return embedding
+        
+        return None
+    
+    def find_similar(self, package_name: str, top_k: int = 5, threshold: float = 0.6) -> List[Tuple[str, float]]:
+        """
+        Find similar packages using cosine similarity.
+        
+        Returns:
+            List of (package_name, similarity_score) tuples
+        """
+        query_emb = self.get_embedding(package_name)
+        if query_emb is None:
+            return []
+        
+        similarities = []
+        
+        for pkg, emb in self.embeddings.items():
+            if pkg == package_name.lower():
+                continue
+            
+            emb_array = np.array(emb)
+            # Cosine similarity
+            similarity = np.dot(query_emb, emb_array) / (
+                np.linalg.norm(query_emb) * np.linalg.norm(emb_array)
+            )
+            
+            if similarity >= threshold:
+                similarities.append((pkg, float(similarity)))
+        
+        # Sort by similarity and return top_k
+        similarities.sort(key=lambda x: x[1], reverse=True)
+        return similarities[:top_k]
+    
+    def get_best_match(self, package_name: str, threshold: float = 0.7) -> Optional[str]:
+        """Get the best matching package name."""
+        similar = self.find_similar(package_name, top_k=1, threshold=threshold)
+        if similar:
+            return similar[0][0]
+        return None
+

requirements.txt

@@ -0,0 +1,8 @@
+gradio>=4.44.1
+packaging>=23.0
+pip>=23.0
+requests>=2.31.0
+scikit-learn>=1.3.0
+sentence-transformers>=2.2.0
+numpy>=1.24.0
+