docs/troubleshooting/optimization-troubleshooting.md

# Search Optimization Troubleshooting Guide

## Overview

This guide helps diagnose and resolve issues with Atlas's search optimization features, including smart search decisions, caching problems, and performance issues.

## Common Issues

### Search Decision Problems

#### Issue: Too Many Unnecessary Searches

**Symptoms**:
- High search API costs
- Slow response times for follow-up questions
- `search_decision.should_search` is true for obvious follow-ups

**Diagnosis**:
```bash
# Check recent search decisions
curl http://localhost:7860/analytics/stats | jq '.search_usage_percentage'

# View specific decision details in responses
curl -X POST http://localhost:7860/chat -d '{
  "prompt": "Tell me more about that",
  "history": [{"role": "user", "content": "What is AI?"}]
}' | jq '.search_decision'
```

**Solutions**:

1. **Use Conservative Mode**:
```json
{
  "prompt": "Follow-up question",
  "search_decision_mode": "conservative"
}
```

2. **Check Conversation History Format**:
```javascript
// Correct format
history: [
  {"role": "user", "content": "What is AI?"},
  {"role": "assistant", "content": "AI is artificial intelligence..."}
]

// Alternative format  
history: [
  {"user": "What is AI?", "assistant": "AI is artificial intelligence..."}
]

// Incorrect - will cause unnecessary searches
history: [
  {"message": "What is AI?", "response": "AI is..."}  // Wrong keys
]
```

3. **Verify NLP Dependencies**:
```bash
# Check if spaCy model is loaded
python -c "import spacy; nlp = spacy.load('en_core_web_sm'); print('spaCy OK')"

# Check RAKE installation
python -c "from rake_nltk import Rake; print('RAKE OK')"
```

#### Issue: Missing Important Information

**Symptoms**:
- Outdated responses for current events
- `search_decision.should_search` is false for time-sensitive queries
- Users complaining about stale information

**Diagnosis**:
```bash
# Check search decision patterns
curl -X POST http://localhost:7860/chat -d '{
  "prompt": "What are today'\''s tech news?",
  "search_decision_mode": "balanced"
}' | jq '.search_decision'
```

**Solutions**:

1. **Use Aggressive Mode for Current Events**:
```json
{
  "prompt": "Latest developments in AI",
  "search_decision_mode": "aggressive"
}
```

2. **Force Search for Critical Updates**:
```json
{
  "prompt": "Current stock price of AAPL", 
  "force_search": true
}
```

3. **Add Recency Keywords**:
```json
{
  "prompt": "What are the latest news about climate change today?"
}
// Keywords like "latest", "today", "current" trigger searches
```

#### Issue: Inconsistent Search Decisions

**Symptoms**:
- Similar questions get different search decisions
- `search_decision.confidence` is very low (< 0.5)
- Decision method frequently falls back to "fallback_rule"

**Diagnosis**:
```bash
# Test decision consistency
for i in {1..5}; do
  curl -X POST http://localhost:7860/chat -d '{
    "prompt": "Tell me more about machine learning"
  }' | jq '.search_decision.should_search'
done
```

**Solutions**:

1. **Check AI Model Availability**:
```python
# Verify Gemini API key
import os
print("GOOGLE_API_KEY:", "✓" if os.getenv("GOOGLE_API_KEY") else "✗")
```

2. **Monitor AI Decision Cache**:
```bash
# Clear AI decision cache if stale
curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=all
```

3. **Review Conversation History Quality**:
```javascript
// Ensure meaningful history entries
const validHistory = history.filter(entry => 
  entry.role && entry.content && entry.content.length > 10
);
```

### Cache-Related Issues

#### Issue: Poor Cache Hit Rates

**Symptoms**:
- `cache_info.cache_hit` is frequently false
- High response times despite caching
- Cache hit rate < 30% in analytics

**Diagnosis**:
```bash
# Check cache performance
curl http://localhost:7860/analytics/cache | jq '{
  hit_rate: .cache_statistics.hit_rate_percentage,
  cache_size: .cache_statistics.cache_size,
  effectiveness: .cache_effectiveness
}'
```

**Solutions**:

1. **Check ChromaDB Configuration**:
```bash
# Verify ChromaDB dependencies
python -c "import chromadb; print('ChromaDB OK')"
python -c "from sentence_transformers import SentenceTransformer; print('SentenceTransformers OK')"
```

2. **Adjust Similarity Threshold**:
```python
# In cache configuration (environment variables)
CACHE_SIMILARITY_THRESHOLD=0.6  # Lower = more hits, less precision
CACHE_SIMILARITY_THRESHOLD=0.8  # Higher = fewer hits, more precision
```

3. **Monitor Query Patterns**:
```bash
# View popular queries
curl http://localhost:7860/analytics/cache | jq '.popular_queries'
```

#### Issue: Cache Storage Problems

**Symptoms**:
- `cache_info.stored_in_cache` is false
- Cache size not growing
- Persistent storage not working across restarts

**Diagnosis**:
```bash
# Check cache directory permissions
ls -la cache_db/
ls -la cache_results/

# Check disk space
df -h .
```

**Solutions**:

1. **Fix Directory Permissions**:
```bash
mkdir -p cache_db cache_results
chmod 755 cache_db cache_results
```

2. **Check Environment Variables**:
```bash
# Verify cache configuration
echo $CHROMADB_PATH
echo $CACHE_RESULTS_PATH
echo $CACHE_EMBEDDING_MODEL
```

3. **Clear Corrupted Cache**:
```bash
# Stop server, clear cache, restart
rm -rf cache_db cache_results
curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=all
```

#### Issue: Memory Usage Issues

**Symptoms**:  
- High memory consumption
- `cache_statistics.memory_usage_mb` increasing rapidly
- Server running out of memory

**Diagnosis**:
```bash
# Monitor cache memory usage
curl http://localhost:7860/analytics/cache | jq '.cache_statistics.memory_usage_mb'

# Check system memory
free -h
ps aux | grep python
```

**Solutions**:

1. **Adjust Cache Size Limits**:
```python
# In cache configuration
max_cache_size = 500  # Reduce from default 1000
```

2. **Implement Regular Cleanup**:
```bash
# Schedule cache cleanup
curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=expired
```

3. **Monitor Cache Efficiency**:
```bash
# Check entries per MB ratio
curl http://localhost:7860/analytics/cache | jq '.memory_efficiency'
```

### Performance Issues

#### Issue: Slow Search Decisions

**Symptoms**:
- Response times > 5 seconds for simple questions
- `decision_method` frequently shows "hybrid" or AI usage
- High CPU usage during decision making

**Diagnosis**:
```bash
# Time search decision performance
time curl -X POST http://localhost:7860/chat -d '{
  "prompt": "Simple question"
}' > /dev/null
```

**Solutions**:

1. **Optimize for Rule-Based Decisions**:
```json
{
  "search_decision_mode": "balanced"  // Uses more rules, less AI
}
```

2. **Check NLP Model Performance**:
```python
import time
import spacy

nlp = spacy.load("en_core_web_sm")
start = time.time()
doc = nlp("test sentence")
print(f"spaCy processing time: {time.time() - start:.3f}s")
```

3. **Monitor AI API Latency**:
```python
# Check Gemini API response times
import time
import google.generativeai as genai

start = time.time()  
response = model.generate_content("test")
print(f"Gemini latency: {time.time() - start:.3f}s")
```

#### Issue: High Memory Usage

**Symptoms**:
- Gradual memory increase over time
- Server crashes with out-of-memory errors
- Slow performance after extended usage

**Diagnosis**:
```bash
# Monitor memory usage patterns
ps aux | grep -E 'python|atlas' | head -5

# Check for memory leaks
curl http://localhost:7860/analytics/cache | jq '.cache_statistics'
```

**Solutions**:

1. **Implement Cache Limits**:
```python
# Set maximum cache entries
MAX_CACHE_ENTRIES = 1000
MAX_MEMORY_MB = 100
```

2. **Regular Cache Cleanup**:
```bash
# Automated cleanup script
#!/bin/bash
while true; do
  sleep 3600  # Every hour
  curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=expired
done
```

3. **Monitor Resource Usage**:
```bash
# Add monitoring script
watch 'curl -s http://localhost:7860/analytics/cache | jq ".cache_statistics.memory_usage_mb"'
```

## Diagnostic Tools

### Decision Analysis Script

```python
#!/usr/bin/env python3
"""Analyze search decision patterns"""

import requests
import json

def analyze_decisions(prompts):
    results = []
    for prompt in prompts:
        response = requests.post('http://localhost:7860/chat',
            json={'prompt': prompt}
        )
        data = response.json()
        results.append({
            'prompt': prompt,
            'should_search': data['search_decision']['should_search'],
            'reason': data['search_decision']['reason'],
            'confidence': data['search_decision']['confidence'],
            'method': data['search_decision'].get('decision_method')
        })
    return results

# Test cases
test_prompts = [
    "What is AI?",
    "Tell me more about that",
    "What's the latest news?",
    "Can you elaborate?",
    "How does machine learning work?"
]

results = analyze_decisions(test_prompts)
for result in results:
    print(f"'{result['prompt']}' -> {result['should_search']} ({result['confidence']:.2f}) - {result['reason']}")
```

### Cache Performance Monitor

```bash
#!/bin/bash
# Monitor cache performance over time

while true; do
  timestamp=$(date '+%Y-%m-%d %H:%M:%S')
  stats=$(curl -s http://localhost:7860/analytics/cache | jq '.cache_statistics')
  hit_rate=$(echo $stats | jq '.hit_rate_percentage')
  cache_size=$(echo $stats | jq '.cache_size')
  memory_mb=$(echo $stats | jq '.memory_usage_mb')
  
  echo "$timestamp - Hit Rate: ${hit_rate}%, Size: $cache_size, Memory: ${memory_mb}MB"
  sleep 60
done
```

### Health Check Script

```python
#!/usr/bin/env python3
"""Comprehensive health check for search optimization"""

import requests
import json
import sys

def health_check():
    issues = []
    
    # Check basic connectivity
    try:
        response = requests.get('http://localhost:7860/')
        if response.status_code != 200:
            issues.append("Server not responding correctly")
    except:
        issues.append("Cannot connect to server")
        return issues
    
    # Check search decision functionality
    try:
        response = requests.post('http://localhost:7860/chat', 
            json={'prompt': 'Test question'}
        )
        data = response.json()
        if 'search_decision' not in data:
            issues.append("Search decision not in response")
    except Exception as e:
        issues.append(f"Search decision error: {e}")
    
    # Check cache functionality  
    try:
        response = requests.get('http://localhost:7860/analytics/cache')
        if response.status_code != 200:
            issues.append("Cache analytics not working")
        else:
            cache_data = response.json()
            hit_rate = cache_data['cache_statistics']['hit_rate_percentage']
            if hit_rate < 10:
                issues.append(f"Very low cache hit rate: {hit_rate}%")
    except Exception as e:
        issues.append(f"Cache check error: {e}")
    
    # Check NLP dependencies
    try:
        import spacy
        nlp = spacy.load('en_core_web_sm')
    except Exception as e:
        issues.append(f"spaCy model error: {e}")
    
    try:
        from rake_nltk import Rake
    except Exception as e:
        issues.append(f"RAKE import error: {e}")
    
    return issues

if __name__ == "__main__":
    issues = health_check()
    if issues:
        print("❌ Issues found:")
        for issue in issues:
            print(f"  - {issue}")
        sys.exit(1)
    else:
        print("✅ All health checks passed")
        sys.exit(0)
```

## Configuration Troubleshooting

### Environment Variables

**Required Variables**:
```bash
# Essential for optimization
GOOGLE_API_KEY=your_key_here

# Cache configuration (optional)
CHROMADB_PATH=cache_db
CACHE_RESULTS_PATH=cache_results  
CACHE_EMBEDDING_MODEL=all-MiniLM-L6-v2
```

**Validation Script**:
```bash
#!/bin/bash
echo "Checking environment variables..."

if [ -z "$GOOGLE_API_KEY" ]; then
  echo "❌ GOOGLE_API_KEY not set"
else
  echo "✅ GOOGLE_API_KEY configured"
fi

if [ -d "$CHROMADB_PATH" ]; then
  echo "✅ ChromaDB path exists: $CHROMADB_PATH"
else
  echo "⚠️  ChromaDB path not found: $CHROMADB_PATH"
fi
```

### Dependency Issues

**Check All Dependencies**:
```python
#!/usr/bin/env python3
"""Check all optimization dependencies"""

dependencies = [
    ('spacy', 'spaCy NLP processing'),
    ('rake_nltk', 'RAKE keyword extraction'),
    ('chromadb', 'ChromaDB vector database'),
    ('sentence_transformers', 'Sentence embeddings'),
    ('google.generativeai', 'Google Gemini API')
]

for module, description in dependencies:
    try:
        __import__(module)
        print(f"✅ {description}: OK")
    except ImportError as e:
        print(f"❌ {description}: {e}")
```

## Performance Optimization

### Recommended Settings

**For High Traffic (Cost Optimization)**:
```json
{
  "search_decision_mode": "conservative",
  "cache_similarity_threshold": 0.6,
  "max_cache_size": 2000
}
```

**For Accuracy (Fresh Information)**:
```json
{
  "search_decision_mode": "aggressive", 
  "cache_similarity_threshold": 0.8,
  "force_search_for_news": true
}
```

**For Balanced Performance**:
```json
{
  "search_decision_mode": "balanced",
  "cache_similarity_threshold": 0.7,
  "ai_decision_timeout": 5.0
}
```

### Monitoring Metrics

**Key Metrics to Track**:
- Search reduction percentage (target: 40-60%)
- Cache hit rate (target: >50%)
- Response time improvement (target: 20-30% faster)
- Decision confidence (target: >0.7 average)
- False positive rate (searches when not needed: <5%)
- False negative rate (no search when needed: <5%)

**Monitoring Setup**:
```bash
# Create monitoring dashboard
curl http://localhost:7860/analytics/dashboard

# Set up alerting thresholds
if [ $(curl -s http://localhost:7860/analytics/cache | jq '.cache_statistics.hit_rate_percentage') < 30 ]; then
  echo "Alert: Low cache hit rate"
fi
```

## Getting Help

### Debug Information Collection

When reporting issues, include:

1. **System Information**:
```bash
python --version
pip list | grep -E 'spacy|chromadb|sentence|google'
df -h
free -h
```

2. **Configuration**:
```bash
env | grep -E 'GOOGLE|CACHE|CHROMADB'
ls -la cache_db/ cache_results/
```

3. **Recent Logs**:
```bash
# Server logs
tail -100 /var/log/atlas.log

# Decision patterns
curl http://localhost:7860/analytics/stats | jq '{
  search_usage: .search_usage_percentage,
  avg_response_time: .average_response_time_ms
}'
```

4. **Sample Requests**:
```bash
# Include problematic requests and responses
curl -X POST http://localhost:7860/chat -d '{
  "prompt": "Your problem prompt here"
}' | jq .
```

### Support Resources

- **Documentation**: [Search Optimization Guide](../features/search-optimization.md)
- **Developer Guide**: [Search Optimizer Developer Guide](../developer/search-optimizer-guide.md)
- **API Reference**: [Integration Guide](../api/integration-guide.md)
- **Performance**: [Setup Guide](../setup/SETUP.md)

For complex issues, create a detailed issue report with the debug information above and specific reproduction steps.