docs/TESTING.md

Testing Guide

Quick Test

Run the automated test suite:

python tests/test_rag_system.py

Or with venv:

source venv/bin/activate
python tests/test_rag_system.py

Manual Testing

1. Test Modal Connection

source venv/bin/activate
modal --version

Expected: Shows Modal version number

2. Test Indexing

source venv/bin/activate
python src/web/query_product_design.py --index

Expected:

• ✅ Loads Word/PDF/Excel files
• ✅ Creates embeddings
• ✅ Stores in ChromaDB
• ✅ Shows success message

Time: 2-5 minutes

3. Test Query (CLI)

source venv/bin/activate
python src/web/query_product_design.py --query "What are the three product tiers?"

Expected:

• ✅ Returns answer with product tier information
• ✅ Shows retrieval and generation times
• ✅ Lists sources

Time: 3-10 seconds (first query may be slower due to cold start)

4. Test Query (Interactive Mode)

source venv/bin/activate
python src/web/query_product_design.py --interactive

Then type questions like:

• "What are the three product tiers?"
• "What is the Year 3 premium volume?"
• "What coverage does the Standard tier include?"

5. Test Web Interface

source venv/bin/activate
python src/web/web_app.py

Then:

1. Open browser: http://127.0.0.1:5000
2. Enter a question
3. Click "Ask Question"
4. Verify answer appears

Test Cases

Product Information Queries

• ✅ "What are the three product tiers?"
• ✅ "What are the premium ranges for each tier?"
• ✅ "What coverage does the Standard tier include?"
• ✅ "What are the unique features of TokyoDrive Insurance?"

Financial Queries

• ✅ "What is the Year 3 premium volume projection?"
• ✅ "What is the target loss ratio?"
• ✅ "What are the break-even projections?"

Market & Strategy Queries

• ✅ "What is the target market size in Tokyo?"
• ✅ "Who are the main competitors?"
• ✅ "What are the key value propositions?"

Compliance Queries

• ✅ "What are the FSA licensing requirements?"
• ✅ "What is the minimum capital requirement?"
• ✅ "What data privacy requirements apply?"

Expected Behavior

Successful Query

• Answer appears within 10-30 seconds
• Answer is relevant and accurate
• Sources are listed
• Timing information is shown

Failed Query

• Clear error message
• Suggestion for fixing the issue
• No crash or hang

Troubleshooting Tests

"Modal command not found"

source venv/bin/activate
pip install modal

"No documents found"

# Upload documents first
modal volume put mcp-hack-ins-products \
  docs/product-design/tokyo_auto_insurance_product_design.docx \
  docs/product-design/tokyo_auto_insurance_product_design.docx

# Then index
python src/web/query_product_design.py --index

"Collection not found"

# Run indexing first
python src/web/query_product_design.py --index

Slow queries

• First query: 10-30 seconds (cold start)
• Subsequent queries: 3-10 seconds (warm container)

Performance Benchmarks

Operation	Expected Time
Indexing (first time)	2-5 minutes
Query (cold start)	10-30 seconds
Query (warm)	3-10 seconds
Web app startup	< 5 seconds

File Format Tests

Should Work

• ✅ .docx files (Word)
• ✅ .pdf files (PDF)
• ✅ .xlsx files (Excel 2007+)
• ✅ .xls files (Excel 97-2003)

Should Be Ignored

• ❌ .md files (markdown)
• ❌ .txt files
• ❌ .csv files
• ❌ Other formats

Integration Tests

End-to-End Test

1. Upload Word/PDF/Excel document
2. Index documents
3. Query via CLI
4. Query via web interface
5. Verify answers are consistent

Error Handling Test

1. Query without indexing → Should show helpful error
2. Query with invalid question → Should handle gracefully
3. Network issues → Should show timeout/connection error

Continuous Testing

For development, run tests after:

• Code changes
• Dependency updates
• Configuration changes
• Document updates