tests/README.md

Test Suite - Auto Tagging RAG System

Overview

This directory contains comprehensive pytest test cases for the Auto Tagging RAG System, including tests for the MCP server, accuracy, user experience, robustness, and non-technical user scenarios.

Test Files

test_mcp_server.py

MCP Server Tests - Tests for Model Context Protocol server functionality

• Tool listing (list_tools)
• Document search tool (search_documents) with all pipelines
• Evaluation tool (evaluate_retrieval)
• Error handling and edge cases
• Tag operators (OR/AND/NOT)
• Default parameter handling

test_accuracy.py

Accuracy Tests - Tests for tag generation, retrieval, and evaluation metrics

• Tag generation accuracy (YAKE, KeyBERT, spaCy, Janome)
• Retrieval accuracy across all pipelines
• Evaluation metrics (Precision@k, nDCG@k, MRR)
• Metric range validation

test_ux.py

User Experience Tests - Tests for user workflows and interface

• Document upload workflow
• Manual tag input
• Search workflows
• Evaluation workflows
• Session persistence
• Tag visualization
• Document count display

test_robustness.py

Robustness Tests - Tests for error handling and edge cases

• Empty query handling
• Invalid k values
• Missing tags
• Invalid operators
• Empty documents
• Special characters
• Large k values
• Data integrity
• Performance tests

test_user_scenarios.py

Non-Technical User Scenarios - Tests for users without technical knowledge

• First-time user document upload
• Simple search queries
• Evaluation with sample queries
• Custom tag input
• Session persistence
• Real-world workflows

test_japanese_support.py

Japanese Language Support - Tests for Japanese language processing

• Japanese tag generation
• Language detection
• Japanese document processing
• Japanese search queries
• Mixed language handling

conftest.py

Pytest Fixtures - Shared fixtures for all tests

• Temporary persistence directories
• RAGManager instances
• Evaluator instances
• Session managers
• Sample documents and queries
• MCP server instances
• Populated RAG managers

Running Tests

Install Dependencies

pip install -r requirements.txt

Note: pytest>=7.0.0 and pytest-asyncio>=0.21.0 are included in requirements.txt.

Run All Tests

pytest tests/ -v

Run Specific Test File

# MCP Server tests
pytest tests/test_mcp_server.py -v

# Accuracy tests
pytest tests/test_accuracy.py -v

# UX tests
pytest tests/test_ux.py -v

# Robustness tests
pytest tests/test_robustness.py -v

# User scenario tests
pytest tests/test_user_scenarios.py -v

# Japanese support tests
pytest tests/test_japanese_support.py -v

Run Specific Test Class

pytest tests/test_mcp_server.py::TestMCPServer -v

Run Specific Test Case

pytest tests/test_mcp_server.py::TestMCPServer::test_search_documents_base_rag -v

Run with Coverage

pip install pytest-cov
pytest tests/ --cov=core --cov=app --cov-report=html

Run Asynchronous Tests

MCP server tests use pytest.mark.asyncio. Ensure pytest-asyncio is installed:

pip install pytest-asyncio
pytest tests/test_mcp_server.py -v

Test Structure

Test Categories

1. MCP Server Tests (15+ tests)

   • Tool listing and discovery
   • All pipeline types (Base, Tag Filter, Hybrid, Hybrid Rerank)
   • Tag operators
   • Evaluation tool
   • Error handling

2. Accuracy Tests (10+ tests)

   • Tag generation for English and Japanese
   • Retrieval accuracy
   • Evaluation metrics

3. User Experience Tests (5+ tests)

   • Workflow tests
   • Interface tests
   • User interactions

4. Robustness Tests (10+ tests)

   • Error handling
   • Edge cases
   • Data integrity
   • Performance

5. User Scenario Tests (8+ tests)

   • Non-technical user workflows
   • Real-world scenarios

6. Japanese Support Tests (5+ tests)

   • Japanese language processing
   • Mixed language handling

Total: 50+ test cases

Test Fixtures

Available Fixtures

• temp_persist_dir: Temporary directory for ChromaDB (auto-cleanup)
• rag_manager: RAGManager instance with temporary persistence
• evaluator: RAGEvaluator instance
• session_manager: SessionManager instance
• session_rag_manager: SessionAwareRAGManager instance
• sample_documents: Sample document data (emergency, medical, surgery)
• sample_queries: Sample evaluation queries
• mcp_server: MCP server instance for testing
• populated_rag_manager: RAGManager with sample documents pre-indexed

Writing New Tests

Example Test Case

def test_my_feature(populated_rag_manager):
    """Test my new feature"""
    query = "test query"
    result = populated_rag_manager.base_rag.retrieve(query, k=3)
    
    assert result is not None
    assert len(result.sources) > 0

Using Fixtures

def test_with_fixture(rag_manager, sample_documents):
    """Test using fixtures"""
    doc_data = sample_documents["emergency"]
    # Use doc_data for testing

Async Tests (MCP Server)

@pytest.mark.asyncio
async def test_mcp_tool(mcp_server):
    """Test MCP tool"""
    result = await mcp_server.call_tool("search_documents", {"query": "test"})
    assert result is not None

Requirements

Python Version

• Python 3.8+

Dependencies

• pytest>=7.0.0
• pytest-asyncio>=0.21.0 (for MCP server tests)
• pytest-cov (optional, for coverage)

Models

• spaCy English model: python -m spacy download en_core_web_sm
• SentenceTransformers: Downloads automatically
• MCP package: pip install mcp (for MCP server tests)

Skipped Tests

Some tests may be skipped if optional dependencies are not installed:

• MCP server tests: Skipped if mcp package not installed
• spaCy tests: Skipped if spaCy model not available
• OpenAI tests: Skipped if OpenAI API key not configured

Test Maintenance

• Update tests when features change
• Add new tests for new features
• Review and refine tests regularly
• Keep test data updated

Troubleshooting

Import Errors

# Ensure you're in the project root
cd /path/to/auto_tagging_rag
pytest tests/ -v

MCP Tests Fail

# Install MCP package
pip install mcp
pytest tests/test_mcp_server.py -v

Model Not Found Errors

# Download required models
python -m spacy download en_core_web_sm

Session Errors

• Tests use temporary directories (auto-cleanup)
• If tests fail, check temp directory permissions

---

Last Updated: 2024
Test Suite Version: 1.0