chore: remove obsolete files and update project structure

feat: integrate A/B testing framework and enhance RAG diversity in recommendation system

chore: update requirements and documentation for intent classifier and RAG evaluation