Design/llm_fallback_system_design.md

LLM Fallback System Design Document

Date: January 2025
Author: BeatDebate Team
Status: Design Phase
Review Status: Pending

---

1. Problem Statement

Objective: Implement a graceful fallback mechanism for music queries that fall outside BeatDebate's predefined intent categories, ensuring users always receive music recommendations even when queries exceed the specialized 4-agent system's scope.

Current State: BeatDebate's 4-agent system excels at handling specific intent categories (By Artist, Artist Similarity, Discovery, Genre/Mood, Contextual, Hybrid, Follow-ups). However, when users submit queries outside these categories or when the backend fails to return recommendations, users may receive error messages or no response.

Value Proposition:

• 100% Query Coverage: Every user query gets a music recommendation response
• Graceful Degradation: Maintain service quality even for edge cases
• Transparent Experience: Users understand when fallback is used
• System Robustness: Handle backend failures and unknown intents seamlessly
• User Retention: Prevent abandonment due to failed queries

---

2. Goals & Non-Goals

✅ In Scope

• Fallback Trigger Detection: Identify when to use LLM fallback (unknown intent or failed recommendations)
• Gemini Flash 2.0 Integration: Leverage existing LLM infrastructure for fallback recommendations
• Response Format Consistency: Maintain same UI format with clear fallback disclaimer
• Error Handling: Graceful handling of both unknown intents and system failures
• User Transparency: Clear indication when fallback system is active
• Chat History Integration: Include conversation context in fallback requests
• Track Info Display: Format fallback responses for Last.fm/Spotify/YouTube links

❌ Out of Scope (v1)

• Complex Fallback Logic: Advanced multi-agent fallback (single LLM call only)
• Fallback Analytics: Detailed tracking of fallback usage patterns
• Fallback Caching: Caching of fallback responses
• Custom Fallback Models: Using different LLMs for different query types
• Fallback Training: Fine-tuning models for music recommendations

---

3. Architecture Overview

User Query
    ↓
┌─────────────────────────┐
│   Chat Interface        │
│   (Frontend)            │
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│   Backend API           │
│   /recommendations      │
└────────────┬────────────┘
             │
         Success? ─────────── No ────┐
             │                       │
            Yes                      │
             │                       ▼
             ▼              ┌─────────────────────────┐
┌─────────────────────────┐ │   LLM Fallback          │
│   4-Agent System        │ │   (Gemini Flash 2.0)    │  
│   Response              │ │                         │
└─────────────────────────┘ └────────────┬────────────┘
             │                           │
             │                           ▼
             │              ┌─────────────────────────┐
             │              │   Fallback Response     │
             │              │   Formatter             │
             │              └────────────┬────────────┘
             │                           │
             ▼                           ▼
┌─────────────────────────────────────────────────────┐
│           Chat Interface Response                   │
│        (Same Format + Disclaimer)                   │
└─────────────────────────────────────────────────────┘

---

4. Technical Design

4.1 Fallback Trigger Detection

4.1.1 Backend Response Analysis

class FallbackTrigger(Enum):
    UNKNOWN_INTENT = "unknown_intent"          # Backend returns unknown intent
    NO_RECOMMENDATIONS = "no_recommendations"   # Backend returns empty results
    API_ERROR = "api_error"                    # Backend returns error status
    TIMEOUT = "timeout"                        # Backend request timeout

4.1.2 Trigger Detection Logic

async def _should_use_fallback(self, response: Optional[Dict]) -> Tuple[bool, FallbackTrigger]:
    """
    Determine if fallback should be used based on backend response.
    
    Returns:
        Tuple of (should_fallback, trigger_reason)
    """
    if response is None:
        return True, FallbackTrigger.API_ERROR
    
    if response.get("intent") == "unknown" or response.get("intent") == "unsupported":
        return True, FallbackTrigger.UNKNOWN_INTENT
    
    recommendations = response.get("recommendations", [])
    if not recommendations or len(recommendations) == 0:
        return True, FallbackTrigger.NO_RECOMMENDATIONS
    
    return False, None

4.2 LLM Fallback Service

4.2.1 Fallback Request Structure

@dataclass
class FallbackRequest:
    query: str
    session_id: str
    chat_context: Optional[Dict] = None
    trigger_reason: FallbackTrigger = None
    max_recommendations: int = 10

4.2.2 Gemini Integration

class LLMFallbackService:
    """Service for handling LLM-based music recommendations when 4-agent system fails."""
    
    def __init__(self, gemini_client):
        self.gemini_client = gemini_client
        self.logger = logging.getLogger(__name__)
    
    async def get_fallback_recommendations(
        self, 
        request: FallbackRequest
    ) -> Dict[str, Any]:
        """
        Get music recommendations from Gemini Flash 2.0 as fallback.
        
        Args:
            request: Fallback request with query and context
            
        Returns:
            Formatted response matching regular recommendation format
        """
        prompt = self._build_fallback_prompt(request)
        
        try:
            response = await self.gemini_client.generate_content_async(prompt)
            parsed_response = self._parse_gemini_response(response)
            
            return {
                "recommendations": parsed_response["tracks"],
                "explanation": parsed_response["explanation"],
                "fallback_used": True,
                "fallback_reason": request.trigger_reason.value,
                "confidence": 0.7  # Default fallback confidence
            }
            
        except Exception as e:
            self.logger.error(f"Fallback service failed: {e}")
            return self._create_emergency_response(request.query)

4.2.3 Fallback Prompt Engineering

def _build_fallback_prompt(self, request: FallbackRequest) -> str:
    """Build optimized prompt for music recommendations."""
    
    context_info = ""
    if request.chat_context:
        previous_queries = request.chat_context.get("previous_queries", [])
        if previous_queries:
            context_info = f"\nConversation context: {', '.join(previous_queries[-2:])}"
    
    prompt = f"""
You are a music recommendation assistant. The user asked: "{request.query}"{context_info}

Provide exactly {request.max_recommendations} music track recommendations in this JSON format:
{{
    "tracks": [
        {{
            "title": "Track Name",
            "artist": "Artist Name", 
            "confidence": 0.85,
            "reason": "Brief explanation why this fits the request"
        }}
    ],
    "explanation": "Overall explanation of the recommendation approach"
}}

Guidelines:
- Focus on diverse, high-quality music recommendations
- Include mix of popular and lesser-known tracks when appropriate
- Ensure artist and title are accurate and searchable
- Provide confidence scores between 0.6-0.9
- Keep reasons concise but meaningful
- Consider the conversation context if provided

Respond ONLY with valid JSON.
"""
    return prompt

4.3 Chat Interface Integration

4.3.1 Modified Process Flow

async def process_message(
    self, 
    message: str, 
    history: List[Tuple[str, str]]
) -> Tuple[str, List[Tuple[str, str]], str]:
    """Enhanced process_message with fallback support."""
    
    if not message.strip():
        return "", history, ""
    
    logger.info(f"Processing message: {message}")
    
    try:
        # Primary: Get recommendations from 4-agent system
        recommendations_response = await self._get_recommendations(message)
        
        # Check if fallback is needed
        should_fallback, trigger_reason = await self._should_use_fallback(
            recommendations_response
        )
        
        if should_fallback:
            logger.info(f"Using LLM fallback due to: {trigger_reason.value}")
            recommendations_response = await self._get_fallback_recommendations(
                message, trigger_reason
            )
        
        if recommendations_response:
            # Format response with fallback indicator
            formatted_response = self.response_formatter.format_recommendations(
                recommendations_response
            )
            
            # Add to history and create player HTML
            history.append((message, formatted_response))
            self._update_conversation_history(message, formatted_response, recommendations_response)
            
            lastfm_player_html = self._create_lastfm_player_html(
                recommendations_response.get("recommendations", [])
            )
            
            return "", history, lastfm_player_html
        else:
            # Emergency fallback
            error_response = self._create_emergency_response(message)
            history.append((message, error_response))
            return "", history, ""
            
    except Exception as e:
        logger.error(f"Error processing message: {e}")
        error_response = f"An error occurred: {str(e)}"
        history.append((message, error_response))
        return "", history, ""

4.3.2 Fallback Response Formatting

async def _get_fallback_recommendations(
    self, 
    query: str, 
    trigger_reason: FallbackTrigger
) -> Dict[str, Any]:
    """Get fallback recommendations from LLM service."""
    
    fallback_request = FallbackRequest(
        query=query,
        session_id=self.session_id,
        chat_context=self._get_chat_context(),
        trigger_reason=trigger_reason,
        max_recommendations=10
    )
    
    fallback_response = await self.fallback_service.get_fallback_recommendations(
        fallback_request
    )
    
    # Add fallback disclaimer to response
    if fallback_response and fallback_response.get("fallback_used"):
        # Insert disclaimer at the beginning of explanation
        original_explanation = fallback_response.get("explanation", "")
        fallback_explanation = (
            f"**⚠️ DEFAULTING TO REGULAR LLM** - This query is outside our "
            f"specialized 4-agent system's scope.\n\n{original_explanation}"
        )
        fallback_response["explanation"] = fallback_explanation
    
    return fallback_response

4.4 Response Formatter Updates

4.4.1 Fallback Indicator Styling

class ResponseFormatter:
    def format_recommendations(self, response: Dict[str, Any]) -> str:
        """Enhanced formatter with fallback indication."""
        
        if response.get("fallback_used"):
            disclaimer = self._create_fallback_disclaimer(
                response.get("fallback_reason", "unknown")
            )
            formatted_response = f"{disclaimer}\n\n{self._format_tracks(response)}"
        else:
            formatted_response = self._format_tracks(response)
        
        return formatted_response
    
    def _create_fallback_disclaimer(self, reason: str) -> str:
        """Create styled fallback disclaimer."""
        return (
            "🔄 **DEFAULTING TO REGULAR LLM**\n"
            "*This query is outside our specialized 4-agent system's scope. "
            "Using general AI assistance for recommendations.*\n"
            "---"
        )

---

5. Implementation Plan

5.1 Phase 1: Core Fallback Infrastructure (Week 1)

1. LLMFallbackService Implementation

   • Create src/services/llm_fallback_service.py
   • Implement Gemini Flash 2.0 integration
   • Add fallback prompt engineering
   • Create fallback response parsing

2. Trigger Detection Logic

   • Enhance chat_interface.py with fallback detection
   • Add FallbackTrigger enum and detection methods
   • Update _get_recommendations method

3. Basic Testing

   • Unit tests for LLMFallbackService
   • Unit tests for trigger detection
   • Mock Gemini responses for testing

5.2 Phase 2: UI Integration (Week 1)

1. Response Formatter Updates

   • Add fallback disclaimer formatting
   • Ensure consistent track display format
   • Maintain Last.fm/Spotify/YouTube links

2. Chat Interface Updates

   • Integrate fallback service into message processing
   • Update conversation history handling
   • Add fallback logging and monitoring

3. Integration Testing

   • End-to-end testing with mock failures
   • Test various fallback scenarios
   • Verify UI consistency

5.3 Phase 3: Testing & Refinement (Week 1-2)

1. Comprehensive Testing

   • Test unknown intent queries
   • Test backend failure scenarios
   • Test conversation context preservation
   • Performance testing

2. Prompt Optimization

   • Refine Gemini prompts for better music recommendations
   • Test edge cases and unusual queries
   • Optimize response quality

3. Documentation & Monitoring

   • Update API documentation
   • Add monitoring for fallback usage
   • Create troubleshooting guides

---

6. Risk Analysis & Mitigation

6.1 Technical Risks

Risk: Gemini API failures causing total system failure
Mitigation:

• Implement emergency response for double-failures
• Add retry logic with exponential backoff
• Create static fallback recommendations for critical failures

Risk: Fallback response quality significantly lower than 4-agent system
Mitigation:

• Extensive prompt engineering and testing
• Clear user expectations through disclaimer
• Monitor user feedback and iterate

Risk: Response time degradation due to additional LLM call
Mitigation:

• Implement async processing
• Set reasonable timeout limits
• Consider caching common fallback responses

6.2 User Experience Risks

Risk: Users might prefer fallback over main system
Mitigation:

• Clear messaging about specialized system capabilities
• Ensure 4-agent system provides superior recommendations for in-scope queries
• Monitor usage patterns

Risk: Fallback disclaimer might reduce trust
Mitigation:

• Frame as "additional coverage" rather than "failure"
• Emphasize transparency as a feature
• Provide clear indication of when main system is active

---

7. Success Metrics

7.1 Functional Metrics

• Query Coverage: 100% of queries receive some form of recommendation
• Fallback Accuracy: Fallback triggers correctly identify edge cases
• Response Consistency: Fallback responses maintain UI format standards
• Error Reduction: Significant decrease in failed query responses

7.2 Quality Metrics

• Response Time: Fallback responses within 5-10 seconds
• Recommendation Quality: User feedback on fallback recommendations
• Context Preservation: Chat context maintained across fallback usage
• System Robustness: Graceful handling of all failure scenarios

---

8. Future Enhancements

8.1 Advanced Fallback Logic

• Multi-step fallback with different LLMs
• Specialized fallback agents for different query types
• Hybrid fallback combining multiple data sources

8.2 Fallback Analytics

• Detailed tracking of fallback usage patterns
• A/B testing of different fallback approaches
• User preference learning for fallback scenarios

8.3 Proactive Fallback

• Confidence-based fallback triggering
• Parallel execution of main system and fallback
• Smart routing based on query analysis

---

9. Conclusion

The LLM Fallback System design provides a robust safety net for BeatDebate, ensuring every user query receives a meaningful response while maintaining transparency about system capabilities. This approach balances user experience with technical robustness, creating a more reliable and trustworthy music recommendation system.

The phased implementation approach allows for iterative refinement while minimizing risk to the existing 4-agent system. The clear architectural separation ensures the fallback mechanism can be enhanced independently without affecting core functionality.

---

Document Version: 1.0
Created: January 2025
Status: Design Phase
Next Phase: Implementation Planning