docs/troubleshooting/optimization-troubleshooting.md

Search Optimization Troubleshooting Guide

Overview

This guide helps diagnose and resolve issues with Atlas's search optimization features, including smart search decisions, caching problems, and performance issues.

Common Issues

Search Decision Problems

Issue: Too Many Unnecessary Searches

Symptoms:

• High search API costs
• Slow response times for follow-up questions
• search_decision.should_search is true for obvious follow-ups

Diagnosis:

# Check recent search decisions
curl http://localhost:7860/analytics/stats | jq '.search_usage_percentage'

# View specific decision details in responses
curl -X POST http://localhost:7860/chat -d '{
  "prompt": "Tell me more about that",
  "history": [{"role": "user", "content": "What is AI?"}]
}' | jq '.search_decision'

Solutions:

1. Use Conservative Mode:

{
  "prompt": "Follow-up question",
  "search_decision_mode": "conservative"
}

2. Check Conversation History Format:

// Correct format
history: [
  {"role": "user", "content": "What is AI?"},
  {"role": "assistant", "content": "AI is artificial intelligence..."}
]

// Alternative format  
history: [
  {"user": "What is AI?", "assistant": "AI is artificial intelligence..."}
]

// Incorrect - will cause unnecessary searches
history: [
  {"message": "What is AI?", "response": "AI is..."}  // Wrong keys
]

3. Verify NLP Dependencies:

# Check if spaCy model is loaded
python -c "import spacy; nlp = spacy.load('en_core_web_sm'); print('spaCy OK')"

# Check RAKE installation
python -c "from rake_nltk import Rake; print('RAKE OK')"

Issue: Missing Important Information

Symptoms:

• Outdated responses for current events
• search_decision.should_search is false for time-sensitive queries
• Users complaining about stale information

Diagnosis:

# Check search decision patterns
curl -X POST http://localhost:7860/chat -d '{
  "prompt": "What are today'\''s tech news?",
  "search_decision_mode": "balanced"
}' | jq '.search_decision'

Solutions:

1. Use Aggressive Mode for Current Events:

{
  "prompt": "Latest developments in AI",
  "search_decision_mode": "aggressive"
}

2. Force Search for Critical Updates:

{
  "prompt": "Current stock price of AAPL", 
  "force_search": true
}

3. Add Recency Keywords:

{
  "prompt": "What are the latest news about climate change today?"
}
// Keywords like "latest", "today", "current" trigger searches

Issue: Inconsistent Search Decisions

Symptoms:

• Similar questions get different search decisions
• search_decision.confidence is very low (< 0.5)
• Decision method frequently falls back to "fallback_rule"

Diagnosis:

# Test decision consistency
for i in {1..5}; do
  curl -X POST http://localhost:7860/chat -d '{
    "prompt": "Tell me more about machine learning"
  }' | jq '.search_decision.should_search'
done

Solutions:

1. Check AI Model Availability:

# Verify Gemini API key
import os
print("GOOGLE_API_KEY:", "✓" if os.getenv("GOOGLE_API_KEY") else "✗")

2. Monitor AI Decision Cache:

# Clear AI decision cache if stale
curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=all

3. Review Conversation History Quality:

// Ensure meaningful history entries
const validHistory = history.filter(entry => 
  entry.role && entry.content && entry.content.length > 10
);

Cache-Related Issues

Issue: Poor Cache Hit Rates

Symptoms:

• cache_info.cache_hit is frequently false
• High response times despite caching
• Cache hit rate < 30% in analytics

Diagnosis:

# Check cache performance
curl http://localhost:7860/analytics/cache | jq '{
  hit_rate: .cache_statistics.hit_rate_percentage,
  cache_size: .cache_statistics.cache_size,
  effectiveness: .cache_effectiveness
}'

Solutions:

1. Check ChromaDB Configuration:

# Verify ChromaDB dependencies
python -c "import chromadb; print('ChromaDB OK')"
python -c "from sentence_transformers import SentenceTransformer; print('SentenceTransformers OK')"

2. Adjust Similarity Threshold:

# In cache configuration (environment variables)
CACHE_SIMILARITY_THRESHOLD=0.6  # Lower = more hits, less precision
CACHE_SIMILARITY_THRESHOLD=0.8  # Higher = fewer hits, more precision

3. Monitor Query Patterns:

# View popular queries
curl http://localhost:7860/analytics/cache | jq '.popular_queries'

Issue: Cache Storage Problems

Symptoms:

• cache_info.stored_in_cache is false
• Cache size not growing
• Persistent storage not working across restarts

Diagnosis:

# Check cache directory permissions
ls -la cache_db/
ls -la cache_results/

# Check disk space
df -h .

Solutions:

1. Fix Directory Permissions:

mkdir -p cache_db cache_results
chmod 755 cache_db cache_results

2. Check Environment Variables:

# Verify cache configuration
echo $CHROMADB_PATH
echo $CACHE_RESULTS_PATH
echo $CACHE_EMBEDDING_MODEL

3. Clear Corrupted Cache:

# Stop server, clear cache, restart
rm -rf cache_db cache_results
curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=all

Issue: Memory Usage Issues

Symptoms:

• High memory consumption
• cache_statistics.memory_usage_mb increasing rapidly
• Server running out of memory

Diagnosis:

# Monitor cache memory usage
curl http://localhost:7860/analytics/cache | jq '.cache_statistics.memory_usage_mb'

# Check system memory
free -h
ps aux | grep python

Solutions:

1. Adjust Cache Size Limits:

# In cache configuration
max_cache_size = 500  # Reduce from default 1000

2. Implement Regular Cleanup:

# Schedule cache cleanup
curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=expired

3. Monitor Cache Efficiency:

# Check entries per MB ratio
curl http://localhost:7860/analytics/cache | jq '.memory_efficiency'

Performance Issues

Issue: Slow Search Decisions

Symptoms:

• Response times > 5 seconds for simple questions
• decision_method frequently shows "hybrid" or AI usage
• High CPU usage during decision making

Diagnosis:

# Time search decision performance
time curl -X POST http://localhost:7860/chat -d '{
  "prompt": "Simple question"
}' > /dev/null

Solutions:

1. Optimize for Rule-Based Decisions:

{
  "search_decision_mode": "balanced"  // Uses more rules, less AI
}

2. Check NLP Model Performance:

import time
import spacy

nlp = spacy.load("en_core_web_sm")
start = time.time()
doc = nlp("test sentence")
print(f"spaCy processing time: {time.time() - start:.3f}s")

3. Monitor AI API Latency:

# Check Gemini API response times
import time
import google.generativeai as genai

start = time.time()  
response = model.generate_content("test")
print(f"Gemini latency: {time.time() - start:.3f}s")

Issue: High Memory Usage

Symptoms:

• Gradual memory increase over time
• Server crashes with out-of-memory errors
• Slow performance after extended usage

Diagnosis:

# Monitor memory usage patterns
ps aux | grep -E 'python|atlas' | head -5

# Check for memory leaks
curl http://localhost:7860/analytics/cache | jq '.cache_statistics'

Solutions:

1. Implement Cache Limits:

# Set maximum cache entries
MAX_CACHE_ENTRIES = 1000
MAX_MEMORY_MB = 100

2. Regular Cache Cleanup:

# Automated cleanup script
#!/bin/bash
while true; do
  sleep 3600  # Every hour
  curl -X POST http://localhost:7860/analytics/cache/clear?cache_type=expired
done

3. Monitor Resource Usage:

# Add monitoring script
watch 'curl -s http://localhost:7860/analytics/cache | jq ".cache_statistics.memory_usage_mb"'

Diagnostic Tools

Decision Analysis Script

#!/usr/bin/env python3
"""Analyze search decision patterns"""

import requests
import json

def analyze_decisions(prompts):
    results = []
    for prompt in prompts:
        response = requests.post('http://localhost:7860/chat',
            json={'prompt': prompt}
        )
        data = response.json()
        results.append({
            'prompt': prompt,
            'should_search': data['search_decision']['should_search'],
            'reason': data['search_decision']['reason'],
            'confidence': data['search_decision']['confidence'],
            'method': data['search_decision'].get('decision_method')
        })
    return results

# Test cases
test_prompts = [
    "What is AI?",
    "Tell me more about that",
    "What's the latest news?",
    "Can you elaborate?",
    "How does machine learning work?"
]

results = analyze_decisions(test_prompts)
for result in results:
    print(f"'{result['prompt']}' -> {result['should_search']} ({result['confidence']:.2f}) - {result['reason']}")

Cache Performance Monitor

#!/bin/bash
# Monitor cache performance over time

while true; do
  timestamp=$(date '+%Y-%m-%d %H:%M:%S')
  stats=$(curl -s http://localhost:7860/analytics/cache | jq '.cache_statistics')
  hit_rate=$(echo $stats | jq '.hit_rate_percentage')
  cache_size=$(echo $stats | jq '.cache_size')
  memory_mb=$(echo $stats | jq '.memory_usage_mb')
  
  echo "$timestamp - Hit Rate: ${hit_rate}%, Size: $cache_size, Memory: ${memory_mb}MB"
  sleep 60
done

Health Check Script

#!/usr/bin/env python3
"""Comprehensive health check for search optimization"""

import requests
import json
import sys

def health_check():
    issues = []
    
    # Check basic connectivity
    try:
        response = requests.get('http://localhost:7860/')
        if response.status_code != 200:
            issues.append("Server not responding correctly")
    except:
        issues.append("Cannot connect to server")
        return issues
    
    # Check search decision functionality
    try:
        response = requests.post('http://localhost:7860/chat', 
            json={'prompt': 'Test question'}
        )
        data = response.json()
        if 'search_decision' not in data:
            issues.append("Search decision not in response")
    except Exception as e:
        issues.append(f"Search decision error: {e}")
    
    # Check cache functionality  
    try:
        response = requests.get('http://localhost:7860/analytics/cache')
        if response.status_code != 200:
            issues.append("Cache analytics not working")
        else:
            cache_data = response.json()
            hit_rate = cache_data['cache_statistics']['hit_rate_percentage']
            if hit_rate < 10:
                issues.append(f"Very low cache hit rate: {hit_rate}%")
    except Exception as e:
        issues.append(f"Cache check error: {e}")
    
    # Check NLP dependencies
    try:
        import spacy
        nlp = spacy.load('en_core_web_sm')
    except Exception as e:
        issues.append(f"spaCy model error: {e}")
    
    try:
        from rake_nltk import Rake
    except Exception as e:
        issues.append(f"RAKE import error: {e}")
    
    return issues

if __name__ == "__main__":
    issues = health_check()
    if issues:
        print("❌ Issues found:")
        for issue in issues:
            print(f"  - {issue}")
        sys.exit(1)
    else:
        print("✅ All health checks passed")
        sys.exit(0)

Configuration Troubleshooting

Environment Variables

Required Variables:

# Essential for optimization
GOOGLE_API_KEY=your_key_here

# Cache configuration (optional)
CHROMADB_PATH=cache_db
CACHE_RESULTS_PATH=cache_results  
CACHE_EMBEDDING_MODEL=all-MiniLM-L6-v2

Validation Script:

#!/bin/bash
echo "Checking environment variables..."

if [ -z "$GOOGLE_API_KEY" ]; then
  echo "❌ GOOGLE_API_KEY not set"
else
  echo "✅ GOOGLE_API_KEY configured"
fi

if [ -d "$CHROMADB_PATH" ]; then
  echo "✅ ChromaDB path exists: $CHROMADB_PATH"
else
  echo "⚠️  ChromaDB path not found: $CHROMADB_PATH"
fi

Dependency Issues

Check All Dependencies:

#!/usr/bin/env python3
"""Check all optimization dependencies"""

dependencies = [
    ('spacy', 'spaCy NLP processing'),
    ('rake_nltk', 'RAKE keyword extraction'),
    ('chromadb', 'ChromaDB vector database'),
    ('sentence_transformers', 'Sentence embeddings'),
    ('google.generativeai', 'Google Gemini API')
]

for module, description in dependencies:
    try:
        __import__(module)
        print(f"✅ {description}: OK")
    except ImportError as e:
        print(f"❌ {description}: {e}")

Performance Optimization

Recommended Settings

For High Traffic (Cost Optimization):

{
  "search_decision_mode": "conservative",
  "cache_similarity_threshold": 0.6,
  "max_cache_size": 2000
}

For Accuracy (Fresh Information):

{
  "search_decision_mode": "aggressive", 
  "cache_similarity_threshold": 0.8,
  "force_search_for_news": true
}

For Balanced Performance:

{
  "search_decision_mode": "balanced",
  "cache_similarity_threshold": 0.7,
  "ai_decision_timeout": 5.0
}

Monitoring Metrics

Key Metrics to Track:

• Search reduction percentage (target: 40-60%)
• Cache hit rate (target: >50%)
• Response time improvement (target: 20-30% faster)
• Decision confidence (target: >0.7 average)
• False positive rate (searches when not needed: <5%)
• False negative rate (no search when needed: <5%)

Monitoring Setup:

# Create monitoring dashboard
curl http://localhost:7860/analytics/dashboard

# Set up alerting thresholds
if [ $(curl -s http://localhost:7860/analytics/cache | jq '.cache_statistics.hit_rate_percentage') < 30 ]; then
  echo "Alert: Low cache hit rate"
fi

Getting Help

Debug Information Collection

When reporting issues, include:

1. System Information:

python --version
pip list | grep -E 'spacy|chromadb|sentence|google'
df -h
free -h

2. Configuration:

env | grep -E 'GOOGLE|CACHE|CHROMADB'
ls -la cache_db/ cache_results/

3. Recent Logs:

# Server logs
tail -100 /var/log/atlas.log

# Decision patterns
curl http://localhost:7860/analytics/stats | jq '{
  search_usage: .search_usage_percentage,
  avg_response_time: .average_response_time_ms
}'

4. Sample Requests:

# Include problematic requests and responses
curl -X POST http://localhost:7860/chat -d '{
  "prompt": "Your problem prompt here"
}' | jq .

Support Resources

• Documentation: Search Optimization Guide
• Developer Guide: Search Optimizer Developer Guide
• API Reference: Integration Guide
• Performance: Setup Guide

For complex issues, create a detailed issue report with the debug information above and specific reproduction steps.