GUIDES.md

# Medical Predictor Chatbot - Developer Guides

## 📚 All Documentation Files

### For Project Overview & Planning
- **PROJECT_ROADMAP.md** ← Start here for complete timeline
- **ANALYSIS.md** ← Current status & what's missing
- **GUIDES.md** ← This file (detailed guides for each phase)

---

## 🎯 Phase-by-Phase Implementation Guide

---

## PHASE 1: Environment & Infrastructure ✅ (85% Complete)

### What You've Done
- ✅ Created basic project structure
- ✅ Set up Gradio + Groq dependencies
- ✅ Created basic Groq API integration
- ✅ Implemented state management

### What You Need to Do (15% Remaining)

#### 1.1 Create `.env` File
```bash
cd /media/zunayed/HDD_code/chatbot\ with\ llm/medical-predictor-chatbot
echo "GROQ_API_KEY=gsk_your_api_key_here" > .env
```

Get your API key:
1. Visit https://console.groq.com/keys
2. Sign up (free)
3. Generate API key
4. Paste into `.env`

#### 1.2 Create `.gitignore`
```bash
cat > .gitignore << 'EOF'
.env
*.pkl
__pycache__/
venv/
.DS_Store
*.pyc
.pytest_cache/
.idea/
*.egg-info/
dist/
build/
app_state.json
EOF
```

#### 1.3 Create Folder Structure
```bash
mkdir -p app/services app/utils models
touch app/__init__.py app/services/__init__.py app/utils/__init__.py
```

#### 1.4 Verify Installation
```bash
pip install -r requirements.txt
python test_extractor.py  # Should print JSON with features
```

**✅ Phase 1 Complete When:**
- `.env` exists with valid GROQ_API_KEY
- `.gitignore` created
- Folder structure matches
- `test_extractor.py` runs without errors

---

## PHASE 2: Configuration & Schemas 🔲 (0% Complete)

### What to Create

#### 2.1 Create `app/config.py`
```python
import os
from pathlib import Path
from dotenv import load_dotenv

# Load environment
load_dotenv()

# Paths
BASE_DIR = Path(__file__).parent.parent
MODEL_DIR = BASE_DIR / "models"
MODEL_PATH = MODEL_DIR / "GradientBoosting_model.pkl"

# Model configuration
GROQ_API_KEY = os.getenv("GROQ_API_KEY")
GROQ_MODEL = "llama-3.1-8b-instant"
GROQ_TEMPERATURE = 0

# Medical features (all 16)
DEFAULT_MODEL_FEATURES = [
    "LengthOfStay", "Smoking", "Family History", "HbA1c", "Glucose",
    "Age", "Diet Score", "Alcohol", "Physical Activity", "Blood Pressure",
    "BMI", "Cholesterol", "Sleep Hours", "Stress Level",
    "Triglycerides", "Oxygen Saturation"
]

# Feature validation ranges (min, max, expected type)
FEATURE_RANGES = {
    "Age": (0, 150, float),
    "Glucose": (70, 400, float),
    "HbA1c": (3, 15, float),
    "BMI": (10, 60, float),
    "Cholesterol": (100, 400, float),
    "Triglycerides": (20, 500, float),
    "Blood Pressure": (60, 200, float),  # Systolic only simplified
    "Physical Activity": (0, 24, float),  # hours per week
    "Sleep Hours": (0, 24, float),
    "Stress Level": (1, 10, float),
    "Diet Score": (1, 10, float),
    "Smoking": (0, 1, int),  # 0=No, 1=Yes
    "Alcohol": (0, 1, int),   # 0=No, 1=Yes
    "Family History": (0, 1, int),  # 0=No, 1=Yes
    "LengthOfStay": (0, 365, int),  # days
    "Oxygen Saturation": (80, 100, float),  # percentage
}

# API configuration
GROQ_TIMEOUT = 30
MAX_RETRIES = 3

# App configuration
DEBUG_MODE = True
CONVERSATION_MAX_TURNS = 20
```

#### 2.2 Create `app/schemas.py`
```python
from typing import Optional
from pydantic import BaseModel, Field, validator

class MedicalFeatures(BaseModel):
    """All 16 required medical features"""
    
    Age: Optional[float] = None
    Glucose: Optional[float] = None
    HbA1c: Optional[float] = None
    BMI: Optional[float] = None
    Cholesterol: Optional[float] = None
    Triglycerides: Optional[float] = None
    BloodPressure: Optional[float] = None
    PhysicalActivity: Optional[float] = None
    SleepHours: Optional[float] = None
    StressLevel: Optional[float] = None
    DietScore: Optional[float] = None
    Smoking: Optional[int] = None
    Alcohol: Optional[int] = None
    FamilyHistory: Optional[int] = None
    LengthOfStay: Optional[int] = None
    OxygenSaturation: Optional[float] = None
    
    @validator("Age")
    def validate_age(cls, v):
        if v is not None and not (0 <= v <= 150):
            raise ValueError("Age must be between 0 and 150")
        return v
    
    # Add similar validators for other fields...
    
    class Config:
        use_enum_values = True
        arbitrary_types_allowed = True


class PredictionRequest(BaseModel):
    """Request for prediction"""
    features: MedicalFeatures


class PredictionResponse(BaseModel):
    """Prediction response"""
    prediction: int  # 0 or 1 (or disease class)
    probability: float  # 0.0 to 1.0
    risk_level: str  # "Low", "Medium", "High"
    explanation: str


class ExtractionResponse(BaseModel):
    """LLM extraction response"""
    extracted_features: MedicalFeatures
    confidence: float
```

**✅ Phase 2 Complete When:**
- Both files exist and have no import errors
- Run: `python -c "from app.config import *; from app.schemas import *"`
- No errors appear

**Time Estimate**: 15-20 minutes

---

## PHASE 3: ML Model Preparation 🔲 (0% Complete)

### Option A: Create a Synthetic Model (for testing)

Create `create_model.py`:
```python
import joblib
import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from pathlib import Path

# Create synthetic data
np.random.seed(42)
X = np.random.randn(100, 16)  # 16 features
y = np.random.randint(0, 2, 100)  # Binary classification

# Train model
model = GradientBoostingClassifier(n_estimators=50, random_state=42)
model.fit(X, y)

# Save
Path("models").mkdir(exist_ok=True)
joblib.dump(model, "models/GradientBoosting_model.pkl")
print("✅ Model saved to models/GradientBoosting_model.pkl")
print(f"Model expects {model.n_features_in_} features")
```

Run it:
```bash
python create_model.py
```

### Option B: Use Pre-trained Model
If you have an existing model file:
```bash
cp /path/to/your/GradientBoosting_model.pkl models/
```

**✅ Phase 3 Complete When:**
- `models/GradientBoosting_model.pkl` exists
- Test with: `python -c "import joblib; m = joblib.load('models/GradientBoosting_model.pkl'); print(f'Model loaded! Features: {m.n_features_in_}')"`

**Time Estimate**: 20-40 minutes

---

## PHASE 4: Service Layer Implementation 🔲 (50% Complete)

### 4.1 Implement `app/services/feature_builder.py`

```python
from typing import List, Dict, Any
from app.config import FEATURE_RANGES, DEFAULT_MODEL_FEATURES
from app.schemas import MedicalFeatures

def validate_feature(name: str, value: Any) -> Any:
    """Validate a single feature"""
    if value is None:
        return None
    
    if name not in FEATURE_RANGES:
        return None
    
    min_val, max_val, expected_type = FEATURE_RANGES[name]
    
    # Convert to expected type
    try:
        converted = expected_type(value)
    except (ValueError, TypeError):
        return None
    
    # Check range
    if not (min_val <= converted <= max_val):
        return None
    
    return converted


def prepare_features_dict(state: Dict[str, Any]) -> MedicalFeatures:
    """Convert state dict to MedicalFeatures model"""
    validated = {}
    
    for feature in DEFAULT_MODEL_FEATURES:
        value = state.get(feature)
        validated[feature] = validate_feature(feature, value)
    
    return MedicalFeatures(**validated)


def prepare_feature_vector(state: Dict[str, Any]) -> List[float]:
    """
    Convert state dict to feature vector for ML model.
    Returns array in correct feature order.
    Handles missing values with sensible defaults.
    """
    
    # Define feature order (MUST match training data order)
    feature_order = DEFAULT_MODEL_FEATURES
    
    vector = []
    for feature_name in feature_order:
        value = state.get(feature_name)
        
        if value is not None:
            vector.append(float(value))
        else:
            # Use feature mean or 0 for missing values
            vector.append(0.0)  # Simple approach - can be improved
    
    return vector


def is_ready_for_prediction(state: Dict[str, Any]) -> bool:
    """Check if enough features collected for prediction"""
    missing = [k for k, v in state.items() if v is None]
    # At least 14 out of 16 features
    return len(missing) <= 2
```

### 4.2 Implement `app/services/predictor.py`

```python
import joblib
import logging
from pathlib import Path
from app.config import MODEL_PATH
from app.schemas import PredictionResponse

logger = logging.getLogger(__name__)


class Predictor:
    """Load and use ML model for predictions"""
    
    def __init__(self, model_path: str = None):
        if model_path is None:
            model_path = MODEL_PATH
        
        self.model_path = Path(model_path)
        self.model = None
        self.load_model()
    
    def load_model(self):
        """Load model from disk"""
        if not self.model_path.exists():
            raise FileNotFoundError(f"Model not found at {self.model_path}")
        
        try:
            self.model = joblib.load(self.model_path)
            logger.info(f"✅ Model loaded from {self.model_path}")
        except Exception as e:
            logger.error(f"❌ Failed to load model: {e}")
            raise
    
    def predict(self, feature_vector: list) -> PredictionResponse:
        """
        Make prediction on feature vector
        
        Args:
            feature_vector: List of 16 floats in correct order
        
        Returns:
            PredictionResponse with prediction, probability, risk level
        """
        if self.model is None:
            raise RuntimeError("Model not loaded")
        
        try:
            # Make prediction
            prediction = self.model.predict([feature_vector])[0]
            
            # Get probability
            proba = self.model.predict_proba([feature_vector])[0]
            probability = float(max(proba))
            
            # Determine risk level
            if probability >= 0.8:
                risk_level = "High"
            elif probability >= 0.5:
                risk_level = "Medium"
            else:
                risk_level = "Low"
            
            return PredictionResponse(
                prediction=int(prediction),
                probability=probability,
                risk_level=risk_level,
                explanation=f"Model predicts class {prediction} with {probability*100:.1f}% confidence"
            )
        
        except Exception as e:
            logger.error(f"Prediction failed: {e}")
            raise


# Global predictor instance
predictor = None


def get_predictor():
    """Get or create predictor instance"""
    global predictor
    if predictor is None:
        predictor = Predictor()
    return predictor
```

**✅ Phase 4 Complete When:**
- Both services import without errors
- Test with: `python -c "from app.services.feature_builder import *; from app.services.predictor import *"`

**Time Estimate**: 30-40 minutes

---

## PHASE 5: Main Application Integration 🔲 (20% Complete)

### Update `app/main.py`

Replace the current `main.py` with:

```python
import gradio as gr
import logging

from app.services.llm_extractor import extract_features_from_text
from app.services.feature_builder import prepare_feature_vector, is_ready_for_prediction
from app.services.predictor import get_predictor
from app.memory import initialize_state, update_state, get_missing_features
from app.utils.helpers import generate_question

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Global state
state = initialize_state()
conversation_history = []


def chat_fn(message, history):
    """Main chat function for Gradio"""
    global state, conversation_history
    
    try:
        # Step 1: Extract features from user input
        logger.info(f"User: {message}")
        extracted = extract_features_from_text(message)
        logger.info(f"Extracted: {extracted}")
        
        # Step 2: Update memory with extracted features
        state = update_state(state, extracted)
        
        # Step 3: Check for missing features
        missing = get_missing_features(state)
        
        # Step 4: If all features collected, make prediction
        if not missing or is_ready_for_prediction(state):
            logger.info("All features collected! Making prediction...")
            
            # Prepare features for model
            feature_vector = prepare_feature_vector(state)
            
            # Get prediction
            predictor = get_predictor()
            result = predictor.predict(feature_vector)
            
            response = f"""
✅ All information collected!

**Prediction Results:**
- **Prediction**: Class {result.prediction}
- **Confidence**: {result.probability*100:.1f}%
- **Risk Level**: {result.risk_level}
- **Details**: {result.explanation}

---
*Note: This is a demonstration. Always consult with a healthcare professional for medical advice.*
            """
            
            return response
        
        else:
            # Ask for next missing feature
            next_question = generate_question(missing[0])
            remaining = len(missing)
            
            response = f"Got it! {next_question}\n\n_(Missing {remaining} more features)_"
            return response
    
    except Exception as e:
        logger.error(f"Error in chat: {e}")
        return f"❌ Error: {str(e)}. Please try again."


# Create Gradio interface
demo = gr.ChatInterface(
    fn=chat_fn,
    title="🏥 Medical Predictor Chatbot",
    description="Chat about your medical information. I'll ask follow-up questions and predict your health risk.",
    examples=[
        "I'm 45 years old and my glucose is 150",
        "I smoke and my stress level is 8",
        "My BMI is 28 and I exercise 5 hours per week"
    ]
)

if __name__ == "__main__":
    demo.launch()
```

**✅ Phase 5 Complete When:**
- `python app/main.py` runs without errors
- Gradio UI launches at http://localhost:7860
- Chat works (asks questions and eventually predicts)

**Time Estimate**: 25-35 minutes

---

## PHASE 6: Error Handling & Validation 🔲 (0% Complete)

### Add Error Handling to `app/services/llm_extractor.py`

```python
def extract_features_from_text(user_input: str) -> dict:
    """
    Uses Groq LLM to extract structured medical features from text.
    Returns dictionary with all required keys.
    """
    
    if not user_input or not user_input.strip():
        return {feature: None for feature in DEFAULT_MODEL_FEATURES}
    
    prompt = f"""
You are a medical information extraction system.

Extract the following features from the user input.

Return STRICT JSON only. No explanation.

Features:
{DEFAULT_MODEL_FEATURES}

Rules:
- If value is missing, use null
- Convert values to numbers where possible
- Smoking, Alcohol, Family History → 0 or 1
- Blood Pressure → numeric (e.g., 120)
- Output must be valid JSON

User Input:
\"\"\"{user_input}\"\"\"
"""
    
    try:
        response = client.chat.completions.create(
            model="llama-3.1-8b-instant",
            messages=[
                {"role": "system", "content": "You are a strict JSON generator."},
                {"role": "user", "content": prompt}
            ],
            temperature=0,
            timeout=30
        )
        
        content = response.choices[0].message.content
        
        try:
            data = json.loads(content)
        except json.JSONDecodeError:
            # Try extracting JSON from markdown code blocks
            if "```" in content:
                content = content.split("```")[1]
                if content.startswith("json"):
                    content = content[4:]
                data = json.loads(content)
            else:
                logger.warning("Could not parse LLM response as JSON")
                data = {feature: None for feature in DEFAULT_MODEL_FEATURES}
        
        return data
    
    except Exception as e:
        logger.error(f"LLM extraction error: {e}")
        return {feature: None for feature in DEFAULT_MODEL_FEATURES}
```

**✅ Phase 6 Complete When:**
- No unhandled exceptions when running app
- Graceful error messages shown to user

**Time Estimate**: 15-20 minutes

---

## PHASE 7: Hugging Face Spaces Deployment 🔲 (0% Complete)

### 7.1 Create HF Space

1. Go to https://huggingface.co/spaces/new
2. Fill in:
   - **Space name**: medical-predictor-chatbot
   - **License**: MIT
   - **SDK**: Gradio
   - **Visibility**: Public

### 7.2 Push Code to HF

```bash
# Navigate to your project
cd /media/zunayed/HDD_code/chatbot\ with\ llm/medical-predictor-chatbot

# Add HF remote (replace YOUR_USERNAME)
git remote add hf https://huggingface.co/spaces/YOUR_USERNAME/medical-predictor-chatbot

# Push to HF
git push hf main
```

### 7.3 Add Secrets in HF Space

1. Go to your Space page
2. Click "Settings" → "Repository Secrets"
3. Add secret:
   - Key: `GROQ_API_KEY`
   - Value: Your Groq API key from `console.groq.com/keys`

### 7.4 Update README.md

```markdown
---
title: Medical Predictor Chatbot
emoji: 🏥
colorFrom: blue
colorTo: green
sdk: gradio
app_file: app/main.py
pinned: false
---

# Medical Predictor Chatbot

A conversational AI chatbot that collects medical information through natural conversation and predicts health risks using machine learning.

## Features
- 💬 Conversational interface using Groq Llama 3 LLM
- 🏥 Extracts 16 medical features from free-form text
- 🤖 Makes predictions using scikit-learn GradientBoostingClassifier
- 📊 Shows confidence levels and risk assessment

## How It Works
1. Chat naturally about your health
2. The AI extracts medical information from your responses
3. Asks follow-up questions for missing information
4. Makes a health risk prediction once all data is collected

## Example Usage
- "I'm 45 years old and my glucose is 150"
- "I smoke and my stress level is 8"
- "My BMI is 28 and I exercise 5 hours per week"

## Disclaimer
⚠️ This is a demonstration tool. Always consult with healthcare professionals for medical advice.
```

**✅ Phase 7 Complete When:**
- Code deployed to HF Spaces
- App loads and works
- GROQ_API_KEY secret is set

**Time Estimate**: 15-30 minutes

---

## PHASE 8: Testing & Finalization 🔲 (0% Complete)

### Test Scenarios

#### Test 1: Feature Extraction
```bash
python test_extractor.py
# Check if JSON is returned with features
```

#### Test 2: Gradio UI (Local)
```bash
python app/main.py
# Open http://localhost:7860
# Test conversation flow
```

#### Test 3: Full Prediction Flow
Chat sequence:
1. "I'm 45 years old"
2. "My glucose is 150"
3. "I smoke"
4. (Continue answering questions for other features)
5. (Expect: Prediction with risk level)

#### Test 4: Error Handling
- Send empty message → Should handle gracefully
- Send gibberish → Should extract what it can
- Invalid values → Should validate and reject

**✅ Phase 8 Complete When:**
- All tests pass
- No errors in logs
- App works on HF Spaces
- README is complete

**Time Estimate**: 30-60 minutes

---

## 🎉 Success Criteria Checklist

- [ ] `.env` file created with GROQ_API_KEY
- [ ] `app/config.py` completed
- [ ] `app/schemas.py` completed
- [ ] `models/GradientBoosting_model.pkl` exists
- [ ] `app/services/feature_builder.py` completed
- [ ] `app/services/predictor.py` completed
- [ ] `app/main.py` updated with prediction
- [ ] Error handling added
- [ ] Logging configured
- [ ] Local testing successful
- [ ] HF Space created
- [ ] Code pushed to HF
- [ ] GROQ_API_KEY secret added to HF
- [ ] HF deployment successful
- [ ] README updated with documentation

---

## 🚀 Start Here!

**Next step**: Follow **PHASE 2** in this guide to create `config.py` and `schemas.py`

When ready, type: **"I'm ready for Phase 2"**