# LLM Fallback System Design Document **Date**: January 2025 **Author**: BeatDebate Team **Status**: Design Phase **Review Status**: Pending --- ## 1. Problem Statement **Objective**: Implement a graceful fallback mechanism for music queries that fall outside BeatDebate's predefined intent categories, ensuring users always receive music recommendations even when queries exceed the specialized 4-agent system's scope. **Current State**: BeatDebate's 4-agent system excels at handling specific intent categories (By Artist, Artist Similarity, Discovery, Genre/Mood, Contextual, Hybrid, Follow-ups). However, when users submit queries outside these categories or when the backend fails to return recommendations, users may receive error messages or no response. **Value Proposition**: - **100% Query Coverage**: Every user query gets a music recommendation response - **Graceful Degradation**: Maintain service quality even for edge cases - **Transparent Experience**: Users understand when fallback is used - **System Robustness**: Handle backend failures and unknown intents seamlessly - **User Retention**: Prevent abandonment due to failed queries --- ## 2. Goals & Non-Goals ### ✅ In Scope - **Fallback Trigger Detection**: Identify when to use LLM fallback (unknown intent or failed recommendations) - **Gemini Flash 2.0 Integration**: Leverage existing LLM infrastructure for fallback recommendations - **Response Format Consistency**: Maintain same UI format with clear fallback disclaimer - **Error Handling**: Graceful handling of both unknown intents and system failures - **User Transparency**: Clear indication when fallback system is active - **Chat History Integration**: Include conversation context in fallback requests - **Track Info Display**: Format fallback responses for Last.fm/Spotify/YouTube links ### ❌ Out of Scope (v1) - **Complex Fallback Logic**: Advanced multi-agent fallback (single LLM call only) - **Fallback Analytics**: Detailed tracking of fallback usage patterns - **Fallback Caching**: Caching of fallback responses - **Custom Fallback Models**: Using different LLMs for different query types - **Fallback Training**: Fine-tuning models for music recommendations --- ## 3. Architecture Overview ``` User Query ↓ ┌─────────────────────────┐ │ Chat Interface │ │ (Frontend) │ └────────────┬────────────┘ │ ▼ ┌─────────────────────────┐ │ Backend API │ │ /recommendations │ └────────────┬────────────┘ │ Success? ─────────── No ────┐ │ │ Yes │ │ ▼ ▼ ┌─────────────────────────┐ ┌─────────────────────────┐ │ LLM Fallback │ │ 4-Agent System │ │ (Gemini Flash 2.0) │ │ Response │ │ │ └─────────────────────────┘ └────────────┬────────────┘ │ │ │ ▼ │ ┌─────────────────────────┐ │ │ Fallback Response │ │ │ Formatter │ │ └────────────┬────────────┘ │ │ ▼ ▼ ┌─────────────────────────────────────────────────────┐ │ Chat Interface Response │ │ (Same Format + Disclaimer) │ └─────────────────────────────────────────────────────┘ ``` --- ## 4. Technical Design ### 4.1 Fallback Trigger Detection #### 4.1.1 Backend Response Analysis ```python class FallbackTrigger(Enum): UNKNOWN_INTENT = "unknown_intent" # Backend returns unknown intent NO_RECOMMENDATIONS = "no_recommendations" # Backend returns empty results API_ERROR = "api_error" # Backend returns error status TIMEOUT = "timeout" # Backend request timeout ``` #### 4.1.2 Trigger Detection Logic ```python async def _should_use_fallback(self, response: Optional[Dict]) -> Tuple[bool, FallbackTrigger]: """ Determine if fallback should be used based on backend response. Returns: Tuple of (should_fallback, trigger_reason) """ if response is None: return True, FallbackTrigger.API_ERROR if response.get("intent") == "unknown" or response.get("intent") == "unsupported": return True, FallbackTrigger.UNKNOWN_INTENT recommendations = response.get("recommendations", []) if not recommendations or len(recommendations) == 0: return True, FallbackTrigger.NO_RECOMMENDATIONS return False, None ``` ### 4.2 LLM Fallback Service #### 4.2.1 Fallback Request Structure ```python @dataclass class FallbackRequest: query: str session_id: str chat_context: Optional[Dict] = None trigger_reason: FallbackTrigger = None max_recommendations: int = 10 ``` #### 4.2.2 Gemini Integration ```python class LLMFallbackService: """Service for handling LLM-based music recommendations when 4-agent system fails.""" def __init__(self, gemini_client): self.gemini_client = gemini_client self.logger = logging.getLogger(__name__) async def get_fallback_recommendations( self, request: FallbackRequest ) -> Dict[str, Any]: """ Get music recommendations from Gemini Flash 2.0 as fallback. Args: request: Fallback request with query and context Returns: Formatted response matching regular recommendation format """ prompt = self._build_fallback_prompt(request) try: response = await self.gemini_client.generate_content_async(prompt) parsed_response = self._parse_gemini_response(response) return { "recommendations": parsed_response["tracks"], "explanation": parsed_response["explanation"], "fallback_used": True, "fallback_reason": request.trigger_reason.value, "confidence": 0.7 # Default fallback confidence } except Exception as e: self.logger.error(f"Fallback service failed: {e}") return self._create_emergency_response(request.query) ``` #### 4.2.3 Fallback Prompt Engineering ```python def _build_fallback_prompt(self, request: FallbackRequest) -> str: """Build optimized prompt for music recommendations.""" context_info = "" if request.chat_context: previous_queries = request.chat_context.get("previous_queries", []) if previous_queries: context_info = f"\nConversation context: {', '.join(previous_queries[-2:])}" prompt = f""" You are a music recommendation assistant. The user asked: "{request.query}"{context_info} Provide exactly {request.max_recommendations} music track recommendations in this JSON format: {{ "tracks": [ {{ "title": "Track Name", "artist": "Artist Name", "confidence": 0.85, "reason": "Brief explanation why this fits the request" }} ], "explanation": "Overall explanation of the recommendation approach" }} Guidelines: - Focus on diverse, high-quality music recommendations - Include mix of popular and lesser-known tracks when appropriate - Ensure artist and title are accurate and searchable - Provide confidence scores between 0.6-0.9 - Keep reasons concise but meaningful - Consider the conversation context if provided Respond ONLY with valid JSON. """ return prompt ``` ### 4.3 Chat Interface Integration #### 4.3.1 Modified Process Flow ```python async def process_message( self, message: str, history: List[Tuple[str, str]] ) -> Tuple[str, List[Tuple[str, str]], str]: """Enhanced process_message with fallback support.""" if not message.strip(): return "", history, "" logger.info(f"Processing message: {message}") try: # Primary: Get recommendations from 4-agent system recommendations_response = await self._get_recommendations(message) # Check if fallback is needed should_fallback, trigger_reason = await self._should_use_fallback( recommendations_response ) if should_fallback: logger.info(f"Using LLM fallback due to: {trigger_reason.value}") recommendations_response = await self._get_fallback_recommendations( message, trigger_reason ) if recommendations_response: # Format response with fallback indicator formatted_response = self.response_formatter.format_recommendations( recommendations_response ) # Add to history and create player HTML history.append((message, formatted_response)) self._update_conversation_history(message, formatted_response, recommendations_response) lastfm_player_html = self._create_lastfm_player_html( recommendations_response.get("recommendations", []) ) return "", history, lastfm_player_html else: # Emergency fallback error_response = self._create_emergency_response(message) history.append((message, error_response)) return "", history, "" except Exception as e: logger.error(f"Error processing message: {e}") error_response = f"An error occurred: {str(e)}" history.append((message, error_response)) return "", history, "" ``` #### 4.3.2 Fallback Response Formatting ```python async def _get_fallback_recommendations( self, query: str, trigger_reason: FallbackTrigger ) -> Dict[str, Any]: """Get fallback recommendations from LLM service.""" fallback_request = FallbackRequest( query=query, session_id=self.session_id, chat_context=self._get_chat_context(), trigger_reason=trigger_reason, max_recommendations=10 ) fallback_response = await self.fallback_service.get_fallback_recommendations( fallback_request ) # Add fallback disclaimer to response if fallback_response and fallback_response.get("fallback_used"): # Insert disclaimer at the beginning of explanation original_explanation = fallback_response.get("explanation", "") fallback_explanation = ( f"**⚠️ DEFAULTING TO REGULAR LLM** - This query is outside our " f"specialized 4-agent system's scope.\n\n{original_explanation}" ) fallback_response["explanation"] = fallback_explanation return fallback_response ``` ### 4.4 Response Formatter Updates #### 4.4.1 Fallback Indicator Styling ```python class ResponseFormatter: def format_recommendations(self, response: Dict[str, Any]) -> str: """Enhanced formatter with fallback indication.""" if response.get("fallback_used"): disclaimer = self._create_fallback_disclaimer( response.get("fallback_reason", "unknown") ) formatted_response = f"{disclaimer}\n\n{self._format_tracks(response)}" else: formatted_response = self._format_tracks(response) return formatted_response def _create_fallback_disclaimer(self, reason: str) -> str: """Create styled fallback disclaimer.""" return ( "🔄 **DEFAULTING TO REGULAR LLM**\n" "*This query is outside our specialized 4-agent system's scope. " "Using general AI assistance for recommendations.*\n" "---" ) ``` --- ## 5. Implementation Plan ### 5.1 Phase 1: Core Fallback Infrastructure (Week 1) 1. **LLMFallbackService Implementation** - Create `src/services/llm_fallback_service.py` - Implement Gemini Flash 2.0 integration - Add fallback prompt engineering - Create fallback response parsing 2. **Trigger Detection Logic** - Enhance `chat_interface.py` with fallback detection - Add `FallbackTrigger` enum and detection methods - Update `_get_recommendations` method 3. **Basic Testing** - Unit tests for `LLMFallbackService` - Unit tests for trigger detection - Mock Gemini responses for testing ### 5.2 Phase 2: UI Integration (Week 1) 1. **Response Formatter Updates** - Add fallback disclaimer formatting - Ensure consistent track display format - Maintain Last.fm/Spotify/YouTube links 2. **Chat Interface Updates** - Integrate fallback service into message processing - Update conversation history handling - Add fallback logging and monitoring 3. **Integration Testing** - End-to-end testing with mock failures - Test various fallback scenarios - Verify UI consistency ### 5.3 Phase 3: Testing & Refinement (Week 1-2) 1. **Comprehensive Testing** - Test unknown intent queries - Test backend failure scenarios - Test conversation context preservation - Performance testing 2. **Prompt Optimization** - Refine Gemini prompts for better music recommendations - Test edge cases and unusual queries - Optimize response quality 3. **Documentation & Monitoring** - Update API documentation - Add monitoring for fallback usage - Create troubleshooting guides --- ## 6. Risk Analysis & Mitigation ### 6.1 Technical Risks **Risk**: Gemini API failures causing total system failure **Mitigation**: - Implement emergency response for double-failures - Add retry logic with exponential backoff - Create static fallback recommendations for critical failures **Risk**: Fallback response quality significantly lower than 4-agent system **Mitigation**: - Extensive prompt engineering and testing - Clear user expectations through disclaimer - Monitor user feedback and iterate **Risk**: Response time degradation due to additional LLM call **Mitigation**: - Implement async processing - Set reasonable timeout limits - Consider caching common fallback responses ### 6.2 User Experience Risks **Risk**: Users might prefer fallback over main system **Mitigation**: - Clear messaging about specialized system capabilities - Ensure 4-agent system provides superior recommendations for in-scope queries - Monitor usage patterns **Risk**: Fallback disclaimer might reduce trust **Mitigation**: - Frame as "additional coverage" rather than "failure" - Emphasize transparency as a feature - Provide clear indication of when main system is active --- ## 7. Success Metrics ### 7.1 Functional Metrics - **Query Coverage**: 100% of queries receive some form of recommendation - **Fallback Accuracy**: Fallback triggers correctly identify edge cases - **Response Consistency**: Fallback responses maintain UI format standards - **Error Reduction**: Significant decrease in failed query responses ### 7.2 Quality Metrics - **Response Time**: Fallback responses within 5-10 seconds - **Recommendation Quality**: User feedback on fallback recommendations - **Context Preservation**: Chat context maintained across fallback usage - **System Robustness**: Graceful handling of all failure scenarios --- ## 8. Future Enhancements ### 8.1 Advanced Fallback Logic - Multi-step fallback with different LLMs - Specialized fallback agents for different query types - Hybrid fallback combining multiple data sources ### 8.2 Fallback Analytics - Detailed tracking of fallback usage patterns - A/B testing of different fallback approaches - User preference learning for fallback scenarios ### 8.3 Proactive Fallback - Confidence-based fallback triggering - Parallel execution of main system and fallback - Smart routing based on query analysis --- ## 9. Conclusion The LLM Fallback System design provides a robust safety net for BeatDebate, ensuring every user query receives a meaningful response while maintaining transparency about system capabilities. This approach balances user experience with technical robustness, creating a more reliable and trustworthy music recommendation system. The phased implementation approach allows for iterative refinement while minimizing risk to the existing 4-agent system. The clear architectural separation ensures the fallback mechanism can be enhanced independently without affecting core functionality. --- **Document Version**: 1.0 **Created**: January 2025 **Status**: Design Phase **Next Phase**: Implementation Planning