classifier/reason/README.md

# Healthcare Reason Classification System

This module implements a specialized classifier for healthcare visit reasons using real clinic data to classify patient queries into specific healthcare reason categories.

## Overview

The reason classifier addresses the challenge of routing medical healthcare queries to appropriate specialized departments. It classifies medical queries into specific reason categories based on actual healthcare visit data.

## Architecture

### Classification Categories

| Category | Description | Examples |
|----------|-------------|----------|
| `ROUTINE_CARE` | Routine healthcare, maintenance visits, general care | "I need routine foot care", "Regular nail care appointment" |
| `PAIN_CONDITIONS` | Various pain-related conditions and discomfort | "I have heel pain when I walk", "My ankle is sore" |
| `INJURIES` | Sprains, wounds, trauma-related conditions | "I sprained my ankle playing sports", "I have a wound that won't heal" |
| `SKIN_CONDITIONS` | Skin-related issues and conditions | "My toenail is ingrown and infected", "I have calluses on my feet" |
| `STRUCTURAL_ISSUES` | Structural problems and related conditions | "I have flat feet", "I need evaluation for plantar fasciitis" |
| `PROCEDURES` | Injections, surgical consultations, post-operative care | "I need a cortisone injection", "Post-surgical follow-up" |

### Technical Implementation

- **Base Model**: `sentence-transformers/embeddinggemma-300m-medical`
- **Architecture**: SetFit with frozen embeddings + trainable classification head
- **Training**: Real healthcare data from clinic appointment records
- **Integration**: Works as part of the complete healthcare routing system

## Quick Start

### 1. Train the Classifier

```bash
# Train with real healthcare data
python classifier/reason/train_reason.py

# The training script will:
# - Load real healthcare data from data/reason_for_visit_data.xlsx
# - Map reasons to categories using keyword matching
# - Train the classifier with frozen embeddings
# - Save the trained model to classifier/reason_checkpoints/
```

### 2. Use the CLI

```bash
# Classify a single reason query
python cli/reason_classifier_cli_new.py "I have heel pain when I walk"

# Interactive mode
python cli/reason_classifier_cli_new.py --interactive

# Batch processing
python cli/reason_classifier_cli_new.py --batch queries.txt --output results.json

# Use complete healthcare routing system
python cli/healthcare_classifier_cli.py "I need routine foot care"
```

### 3. Programmatic Usage

```python
from classifier.reason import ReasonClassifier, predict_single_reason

# Using the main classifier class
classifier = ReasonClassifier()
predictions = classifier.predict(["I have heel pain when I walk"])
print(predictions[0]['category'])  # Output: PAIN_CONDITIONS

# Using convenience function
result = predict_single_reason("I need routine foot care")
print(result['category'])  # Output: ROUTINE_CARE
print(result['confidence'])  # Confidence score
print(result['probabilities'])  # All category probabilities
```

## System Integration

### Complete Healthcare Routing Workflow

```
User Query
    ↓
Medical vs Insurance Classification
    ↓
┌─────────────────┬─────────────────┐
│   Insurance     │     Medical     │
│   Queries       │     Queries     │
│       ↓         │        ↓        │
│  Insurance      │   Reason        │
│  Department     │ Classification  │
│                 │        ↓        │
│                 │  • ROUTINE_CARE │
│                 │  • PAIN_CONDITIONS │
│                 │  • INJURIES     │
│                 │  • SKIN_CONDITIONS │
│                 │  • STRUCTURAL_ISSUES │
│                 │  • PROCEDURES   │
└─────────────────┴─────────────────┘
```

### Integration with Healthcare System

The reason classifier integrates as part of the complete healthcare routing system:

1. **Primary Classification**: Medical vs Insurance queries
2. **Reason Classification**: Medical queries → Specific reason categories
3. **Department Routing**: Route to appropriate specialized departments

## Training Data Strategy

### Real Healthcare Data

The system uses actual healthcare clinic data:

```python
# Data source: data/reason_for_visit_data.xlsx
# Contains real patient visit reasons and appointment types
# Examples from actual data:
# - "Heel pain"
# - "Routine foot care"
# - "Ingrown toenail"
# - "Ankle sprain"
# - "Plantar fasciitis"
```

### Category Mapping Strategy

The system uses keyword-based mapping to categorize real healthcare reasons:

```python
def map_reason_to_category(reason: str) -> int:
    reason_lower = reason.lower()
    
    # ROUTINE_CARE (routine care, maintenance visits)
    if any(word in reason_lower for word in ['routine', 'nail care', 'calluses']):
        return 0
    
    # PAIN_CONDITIONS (various pain-related conditions)
    elif any(word in reason_lower for word in ['pain', 'ache', 'sore']):
        return 1
    
    # ... other categories
```

## Performance Metrics

### Expected Performance
- **Accuracy**: Based on real healthcare data patterns
- **Categories**: 6 specialized healthcare reason categories
- **Confidence**: Variable based on training data quality

### Evaluation Framework

```bash
# Train and evaluate the model
python classifier/reason/train_reason.py

# Test the trained model
python classifier/reason/infer_reason.py

# Results include:
# - Training metrics
# - Category distribution
# - Example predictions with confidence scores
```

## File Structure

```
classifier/reason/
├── __init__.py              # Package initialization and exports
├── README.md               # This documentation
├── reason_classifier.py    # Main ReasonClassifier class
├── infer_reason.py        # Inference functions and utilities
└── train_reason.py        # Training script and functions
```

## API Reference

### ReasonClassifier

```python
class ReasonClassifier:
    def __init__(self, data_file: str = "data/reason_for_visit_data.xlsx")
    def predict(self, queries: List[str]) -> List[Dict]
    def train(self, train_data: pd.DataFrame = None, eval_data: Optional[pd.DataFrame] = None)
    def save_model(self, path: str)
    def load_model(self, path: str)
    def create_real_dataset(self) -> pd.DataFrame
    def analyze_real_data(self)
```

### Inference Functions

```python
def predict_single_reason(query: str) -> dict
def predict_reason_query(text: list[str], embedding_model, classifier_head) -> dict
def get_reason_models() -> tuple
def test_reason_classifier()
```

### Training Functions

```python
def get_reason_model(num_classes: int)
def get_reason_dataset() -> pd.DataFrame
def map_reason_to_category(reason: str) -> int
def preprocess_reason_data(df: pd.DataFrame) -> pd.DataFrame
```

## Data Requirements

### Healthcare Data Format

The system expects healthcare data in Excel format with these columns:

```
Required columns:
- "Reason For Visit": The primary reason for the healthcare visit
- "Appointment Type": Type of appointment (optional, used for context)

Example data:
| Reason For Visit | Appointment Type |
|------------------|------------------|
| Heel pain        | Follow-up        |
| Routine foot care| Maintenance      |
| Ingrown toenail  | New Patient      |
```

## Deployment Considerations

### Production Readiness

1. **Model Persistence**: Trained models saved with timestamps in `classifier/reason_checkpoints/`
2. **Error Handling**: Graceful fallbacks for prediction failures
3. **Real Data Integration**: Uses actual healthcare clinic data
4. **Device Support**: CPU/GPU/MPS compatibility

### Scalability

- **Batch Processing**: Efficient handling of multiple queries
- **Integration**: Works with existing healthcare routing system
- **Checkpoints**: Automatic model saving with timestamps

## Future Enhancements

### Data Improvements

1. **Expanded Dataset**: Include more healthcare specialties
2. **Active Learning**: Improve model with real-world feedback
3. **Multi-language Support**: Support for non-English healthcare queries

### Advanced Features

1. **Confidence Calibration**: Improve confidence score reliability
2. **Hierarchical Classification**: Sub-categories within reason types
3. **Context Awareness**: Consider patient history and appointment context

## Troubleshooting

### Common Issues

1. **Data Loading Errors**: Ensure `data/reason_for_visit_data.xlsx` exists
2. **Low Confidence**: May indicate need for more training data or model retraining
3. **Import Errors**: Ensure all dependencies are installed and paths are correct

### Debug Mode

```python
# Test the classifier with sample queries
from classifier.reason.infer_reason import test_reason_classifier
test_reason_classifier()

# Check model predictions with probabilities
from classifier.reason import predict_single_reason
result = predict_single_reason("ambiguous query")
print(result['probabilities'])
```

### Model Training Issues

```bash
# Check if healthcare data is available
ls -la data/reason_for_visit_data.xlsx

# Verify model training
python classifier/reason/train_reason.py

# Test inference after training
python classifier/reason/infer_reason.py
```

## Contributing

### Adding New Categories

1. Update `REASON_CATEGORIES` in `reason_classifier.py`, `infer_reason.py`, and `train_reason.py`
2. Update category mapping logic in `map_reason_to_category()`
3. Retrain the model with new categories
4. Update documentation and examples

### Improving Training Data

1. Add more real healthcare examples to the dataset
2. Improve keyword mapping for better categorization
3. Implement more sophisticated NLP techniques for category assignment

## License

This module is part of the health-query-classifier project and follows the same licensing terms.