feat: enhance recommendation system with improved routing, latency optimizations, and onboarding features

chore: update requirements and documentation for intent classifier and RAG evaluation