tests/e2e/README.md

# End-to-End Testing Suite for Atom

Comprehensive E2E tests validating Atom's critical workflows with real service integration (PostgreSQL, Redis, LLM providers).

## Overview

This directory contains end-to-end tests that validate complete user journeys with **real services** (not mocked). E2E tests catch integration issues between components and ensure the system works as users expect.

**Testing Philosophy:**
- **Real Service Integration**: PostgreSQL, Redis, Docker, LLM providers (not mocked)
- **Complete User Journeys**: Test workflows end-to-end, not individual components
- **Production Confidence**: Validate actual behavior, not mocked responses
- **Performance Validation**: Ensure workflows complete within time thresholds
- **Coverage Targets**: 60-70% for MCP service (vs 26.56% with mocks)

### Why E2E Testing Matters

Unit tests validate individual components in isolation. Integration tests validate component interactions. **E2E tests validate complete user journeys** from start to finish, catching integration issues that unit/integration tests miss.

**Example**: Unit test might verify `AgentGovernanceService.can_perform_action()` returns `True`. E2E test verifies that an agent can actually execute a workflow end-to-end, including database writes, audit trails, and canvas presentations.

### Test Categories

| Category | File | Purpose | Tests |
|----------|------|---------|-------|
| **MCP Tools** | `test_mcp_tools_e2e.py` | MCP tool integration with real PostgreSQL | 66 |
| **Database** | `test_database_integration_e2e.py` | PostgreSQL, SQLite, migrations, pooling | 31 |
| **LLM Providers** | `test_llm_providers_e2e.py` | OpenAI, Anthropic, DeepSeek, BYOK | 36 |
| **Critical Workflows** | `test_critical_workflows_e2e.py` | Complete user journey validation | 20 |
| **Governance** | `test_scenario_01_governance.py` | Agent governance & routing | 8 |
| **Streaming** | `test_scenario_02_streaming.py` | Multi-provider LLM streaming | 6 |
| **Canvas** | `test_scenario_03_canvas.py` | Canvas presentations with governance | 5 |
| **Guidance** | `test_scenario_04_guidance.py` | Real-time agent guidance | 7 |
| **Browser** | `test_scenario_05_browser.py` | Browser automation with Playwright | 6 |
| **Episodes** | `test_scenario_06_episodes.py` | Episodic memory & retrieval | 8 |
| **Graduation** | `test_scenario_07_graduation.py` | Agent graduation framework | 7 |
| **Training** | `test_scenario_08_training.py` | Student agent training | 8 |
| **Device** | `test_scenario_09_device.py` | Device capabilities & permissions | 9 |
| **Deeplinks** | `test_scenario_10_deeplinks_feedback.py` | Deep linking & feedback | 8 |

## Setup Instructions

### Prerequisites

```bash
# 1. Install Python dependencies
cd /Users/rushiparikh/projects/atom/backend
pip install -e .

# 2. Install Docker (required for PostgreSQL and Redis)
# Download from: https://www.docker.com/products/docker-desktop

# 3. Install Playwright for browser automation tests (optional)
playwright install chromium
```

### Environment Configuration

```bash
# Required for E2E testing with real services
export E2E_TESTING=true

# Database configuration (default: PostgreSQL on port 5433)
export DATABASE_URL="postgresql://atom:atom@localhost:5433/atom_e2e"
# Or for Personal Edition testing:
# export DATABASE_URL="sqlite:///./atom_e2e.db"

# Redis configuration (default: localhost:6380)
export REDIS_URL="redis://localhost:6380"

# LLM Provider API Keys (optional - tests skip if not set)
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export DEEPSEEK_API_KEY="sk-..."
export GEMINI_API_KEY="..."

# Feature flags (automatically set by conftest.py)
export STREAMING_GOVERNANCE_ENABLED=true
export CANVAS_GOVERNANCE_ENABLED=true
export BROWSER_AUTOMATION_ENABLED=true
export EPISODIC_MEMORY_ENABLED=true
```

### Docker Service Setup

```bash
# Start PostgreSQL and Redis for E2E testing
cd /Users/rushiparikh/projects/atom
docker-compose -f docker-compose-e2e.yml up -d

# Verify services are running
docker ps

# Stop services when done
docker-compose -f docker-compose-e2e.yml down
```

**Services:**
- **PostgreSQL 16**: `localhost:5433` (user: `atom`, pass: `atom`, db: `atom_e2e`)
- **Valkey 8** (Redis-compatible): `localhost:6380`

## Test Execution Guide

### Running All E2E Tests

```bash
# Run all E2E tests with real services
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v

# Run with timeout enforcement (10 minutes total)
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v --timeout=600

# Run with coverage (MCP service target: 60-70%)
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v --cov=integrations/mcp_service --cov-report=term-missing
```

### Running Specific Test Suites

```bash
# MCP Tool E2E Tests (PostgreSQL integration)
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/test_mcp_tools_e2e.py -v

# Database Integration E2E Tests (PostgreSQL, SQLite, migrations)
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/test_database_integration_e2e.py -v

# LLM Provider E2E Tests (requires API keys)
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/test_llm_providers_e2e.py -v

# Critical Workflow E2E Tests (complete user journeys)
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/test_critical_workflows_e2e.py -v
```

### Running with Coverage

```bash
# Generate HTML coverage report
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v \
  --cov=integrations/mcp_service \
  --cov=core \
  --cov-report=html \
  --cov-report=term-missing

# View coverage report
open htmlcov/index.html

# Check coverage for specific module
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v \
  --cov=integrations/mcp_service.mcp_service \
  --cov-report=term-missing
```

### Debugging Failed Tests

```bash
# Run with verbose output
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/test_critical_workflows_e2e.py -v -s

# Drop into debugger on failure
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/test_critical_workflows_e2e.py -v -s --pdb

# Run last failed tests
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v --lf

# Run until first failure
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v -x
```

## Test Organization

### E2E Test Files

#### 1. MCP Tool E2E Tests (`test_mcp_tools_e2e.py`)
**Purpose**: Validate MCP tool integration with real PostgreSQL operations

**Test Coverage**:
- CRM tools (Salesforce, HubSpot, Pipedrive): 5 tests
- Task management (Asana, Trello, Jira): 6 tests
- Ticketing systems (Zendesk, Freshdesk): 4 tests
- Knowledge base (Notion, Confluence): 6 tests
- Canvas presentations: 5 tests
- Finance tools (QuickBooks, Xero): 3 tests
- WhatsApp integration: 3 tests
- Shopify integration: 4 tests
- Error handling (404, 500, timeout): 8 tests
- Concurrency (parallel operations): 4 tests
- Performance (bulk operations): 4 tests
- Data validation (SQL injection, XSS): 10 tests

**Key Fixtures**:
- `e2e_docker_compose`: Start/stop Docker services
- `e2e_postgres_db`: PostgreSQL database connection
- `mcp_service`: MCP service instance with DB
- `crm_data_factory`: Create test CRM records

#### 2. Database Integration E2E Tests (`test_database_integration_e2e.py`)
**Purpose**: Validate database operations across PostgreSQL and SQLite

**Test Coverage**:
- PostgreSQL CRUD operations: 4 tests
- PostgreSQL transactions: 3 tests
- PostgreSQL foreign keys: 3 tests
- SQLite Personal Edition: 2 tests
- SQLite WAL mode: 1 test
- Connection pooling: 3 tests
- Alembic migrations: 4 tests
- Backup/restore: 3 tests
- Performance benchmarks: 3 tests

**Key Fixtures**:
- `e2e_postgres_db`: PostgreSQL database
- `e2e_sqlite_db`: SQLite in-memory database
- `migration_runner`: Run Alembic migrations
- `database_backup`: Backup/restore utilities

#### 3. LLM Provider E2E Tests (`test_llm_providers_e2e.py`)
**Purpose**: Validate multi-provider LLM integration with real API calls

**Test Coverage**:
- OpenAI integration: 6 tests
- Anthropic integration: 6 tests
- DeepSeek integration: 5 tests
- BYOK handler: 7 tests
- Context management: 3 tests
- Cross-provider comparison: 2 tests
- Error handling: 3 tests
- Performance benchmarks: 2 tests

**Key Fixtures**:
- `llm_api_keys`: Detect available API keys
- `openai_client`: OpenAI client (skips if no key)
- `anthropic_client`: Anthropic client (skips if no key)
- `byok_handler`: BYOK handler with multiple providers

**Note**: Tests gracefully skip when API keys not configured (CI-friendly)

#### 4. Critical Workflow E2E Tests (`test_critical_workflows_e2e.py`)
**Purpose**: Validate complete user journeys end-to-end

**Test Coverage**:
- **Agent Execution Workflow** (3 tests):
  - Complete agent lifecycle (creation, execution, monitoring, results)
  - Failure scenarios with error handling
  - Multi-agent coordination

- **Skill Loading Workflow** (3 tests):
  - Skill import, security scanning, storage, execution
  - Package dependency detection
  - Invalid SKILL.md handling

- **Package Installation Workflow** (3 tests):
  - Governance approval, vulnerability scanning, installation
  - Dependency resolution
  - Graceful failure handling

- **Multi-Provider LLM Workflow** (3 tests):
  - Provider failure and fallback
  - Cost optimization
  - Budget enforcement

- **Canvas Presentation Workflow** (3 tests):
  - Canvas creation, LLM content generation, presentation
  - LLM-generated content on canvas
  - Feedback loop and linkage

- **End-to-End Smoke Tests** (5 tests):
  - Complete agent-to-canvas journey
  - Skill-to-package integration
  - Performance threshold validation
  - Data integrity checks
  - Error recovery and audit trail validation

**Key Fixtures**:
- `agent_workflow`: Agent execution workflow setup
- `skill_workflow`: Skill loading workflow setup
- `package_workflow`: Package installation workflow setup
- `llm_workflow`: Multi-provider LLM workflow setup
- `canvas_workflow`: Canvas presentation workflow setup
- `composite_workflow`: All workflows combined
- `workflow_performance_thresholds`: Performance validation

### Fixture Modules

#### `conftest.py` - Root E2E Configuration
- Docker Compose lifecycle management
- PostgreSQL and Redis fixtures
- LLM API key detection
- Test data factories
- Performance monitoring

#### `fixtures/workflow_fixtures.py` - Workflow-Specific Fixtures
- Agent workflow fixture
- Skill workflow fixture
- Package workflow fixture
- LLM workflow fixture
- Canvas workflow fixture
- Composite workflow fixture
- Performance threshold fixture
- Audit trail helper

#### `fixtures/database_fixtures.py` - Database Fixtures
- PostgreSQL engine fixture
- SQLite engine fixture
- Migration runner fixture
- Backup/restore fixture
- Connection pool fixture

#### `fixtures/llm_fixtures.py` - LLM Provider Fixtures
- API key detection
- OpenAI client fixture
- Anthropic client fixture
- BYOK handler fixture
- Mock response fixture

#### `fixtures/test_data_factory.py` - Test Data Generation
- CRM data factory
- Task data factory
- Ticket data factory
- Knowledge base factory
- Canvas data factory
- Finance data factory

## CI/CD Integration

### GitHub Actions Configuration

```yaml
name: E2E Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]
  schedule:
    # Run daily at 2 AM UTC
    - cron: '0 2 * * *'

jobs:
  e2e-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_USER: atom
          POSTGRES_PASSWORD: atom
          POSTGRES_DB: atom_e2e
        ports:
          - 5433:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

      redis:
        image: valkey/valkey:8
        ports:
          - 6380:6379
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          cd backend
          pip install -e .
          pip install pytest pytest-timeout pytest-cov

      - name: Run E2E tests
        env:
          E2E_TESTING: true
          DATABASE_URL: postgresql://atom:atom@localhost:5433/atom_e2e
          REDIS_URL: redis://localhost:6380
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          PYTHONPATH=./backend pytest backend/tests/e2e/ -v \
            --timeout=600 \
            --cov=integrations/mcp_service \
            --cov-report=xml \
            --cov-report=term-missing

      - name: Check coverage threshold
        run: |
          coverage=$(coverage report | grep integrations/mcp_service | awk '{print $4}' | sed 's/%//')
          if (( $(echo "$coverage < 60" | bc -l) )); then
            echo "Coverage $coverage% below 60% threshold"
            exit 1
          fi
          echo "Coverage $coverage% meets threshold"

      - name: Upload coverage
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage.xml
          flags: e2e
          name: e2e-coverage

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: e2e-test-results
          path: |
            backend/htmlcov/
            backend/.coverage*
```

### Parallel Execution

```bash
# Run E2E tests in parallel (faster on CI)
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v -n auto

# Run specific suites in parallel
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest \
  backend/tests/e2e/test_mcp_tools_e2e.py \
  backend/tests/e2e/test_database_integration_e2e.py \
  -v -n auto
```

## Performance Benchmarks

### Expected Execution Times

| Test Suite | Tests | Target | Notes |
|------------|-------|--------|-------|
| MCP Tools E2E | 66 | 2-3 min | PostgreSQL operations |
| Database Integration | 31 | 1-2 min | PostgreSQL + SQLite |
| LLM Providers | 36 | 2-4 min | API calls (if keys set) |
| Critical Workflows | 20 | 1-2 min | Workflow validation |
| Scenarios (01-10) | 64 | 3-5 min | Feature scenarios |
| **TOTAL** | **217+** | **<10 min** | **Full suite target** |

### Performance Targets

| Workflow | Target | Validation |
|----------|--------|------------|
| Agent creation | <1s | `test_workflow_performance_within_thresholds` |
| Agent execution | <10s | `test_agent_execution_workflow` |
| Skill import | <5s | `test_skill_loading_workflow` |
| Package install | <60s | `test_python_package_installation` |
| LLM streaming | <5s | `test_multi_provider_fallback` |
| Canvas presentation | <2s | `test_canvas_presentation_workflow` |
| **Total workflow** | **<2 min** | `test_complete_agent_to_canvas_workflow` |

### Timeout Enforcement

```bash
# Set 10-minute timeout for entire suite
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v --timeout=600

# Set per-test timeout
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v --timeout=30
```

## Coverage Targets

### Current Baseline (Phase 62)

| Module | Coverage | Target | Gap |
|--------|----------|--------|-----|
| MCP Service | 26.56% | 60-70% | **-33% to -43%** |
| Core Services | 24.4% | 50-60% | -26% to -36% |
| API Routes | 38.2% | 70-80% | -32% to -42% |

### E2E Test Contribution

E2E tests with real services should improve MCP service coverage from **26.56% to 60-70%**:

**Why E2E Improves Coverage**:
- Unit tests mock external dependencies (database, APIs)
- E2E tests use real PostgreSQL, Redis, LLM providers
- Real service integration covers actual code paths
- No mocking = actual code execution = better coverage

**Validation**:
```bash
# Run E2E tests with coverage
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v \
  --cov=integrations/mcp_service \
  --cov-report=term-missing

# Check MCP service coverage
# Target: 60-70% (vs 26.56% baseline)
```

## Troubleshooting

### Docker Issues

**Issue**: Docker services not starting
```bash
# Check Docker is running
docker ps

# Restart Docker Desktop (macOS/Windows)
# Or restart Docker daemon (Linux)
sudo systemctl restart docker

# Re-create services
docker-compose -f docker-compose-e2e.yml down
docker-compose -f docker-compose-e2e.yml up -d
```

**Issue**: PostgreSQL connection refused
```bash
# Check PostgreSQL is running
docker ps | grep postgres

# Check PostgreSQL logs
docker-compose -f docker-compose-e2e.yml logs postgres

# Verify connection
psql -h localhost -p 5433 -U atom -d atom_e2e
```

**Issue**: Redis connection refused
```bash
# Check Redis is running
docker ps | grep redis

# Check Redis logs
docker-compose -f docker-compose-e2e.yml logs redis

# Verify connection
redis-cli -p 6380 ping
```

### Database Issues

**Issue**: "Database is locked" (SQLite)
```bash
# Remove stale database files
rm -f atom_e2e.db atom_e2e.db-wal atom_e2e.db-shm

# Re-run tests
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v
```

**Issue**: Migration failures
```bash
# Reset database
docker-compose -f docker-compose-e2e.yml down
docker volume rm atom_postgres_data
docker-compose -f docker-compose-e2e.yml up -d

# Run migrations manually
cd backend
alembic upgrade head
```

### API Key Issues

**Issue**: "API key required" (LLM tests)
```bash
# Option 1: Set API keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

# Option 2: Skip LLM tests
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v -k "not llm"
```

### Test Timeout Issues

**Issue**: Tests exceed timeout
```bash
# Increase timeout
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v --timeout=900

# Run tests individually
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/test_critical_workflows_e2e.py -v -s

# Run slow tests last
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v -m "not slow"
```

### Import Errors

**Issue**: "No module named 'core'"
```bash
# Set PYTHONPATH
export PYTHONPATH=/Users/rushiparikh/projects/atom/backend

# Or use inline
PYTHONPATH=/Users/rushiparikh/projects/atom/backend pytest backend/tests/e2e/ -v
```

**Issue**: "No module named 'pytest'"
```bash
# Install pytest
cd /Users/rushiparikh/projects/atom/backend
pip install pytest pytest-timeout pytest-cov
```

## Best Practices

### Writing E2E Tests

1. **Test Complete Workflows**: Don't test individual functions - test user journeys
2. **Use Real Services**: PostgreSQL, Redis, Docker (not mocks)
3. **Validate Audit Trails**: Check database records, logs, metrics
4. **Test Failure Scenarios**: Not just happy path - error handling too
5. **Assert Performance**: Use `workflow_performance_thresholds` fixture
6. **Clean Up Test Data**: Use fixtures with proper teardown

### Example E2E Test

```python
def test_complete_user_workflow(composite_workflow):
    """Test complete workflow from user request to result."""
    agent = composite_workflow["agent"]
    canvas = composite_workflow["canvas"]

    # Step 1: Execute agent
    execution = AgentExecution(
        agent_id=agent["agent"].id,
        user_id=canvas["user_id"],
        task="Generate insights",
        status="in_progress"
    )
    agent["session"].add(execution)
    agent["session"].commit()

    # Step 2: Complete execution
    execution.status = "completed"
    execution.output_data = {"insights": ["insight1", "insight2"]}
    agent["session"].commit()

    # Step 3: Create canvas
    audit = CanvasAudit(
        canvas_id=canvas["canvas_id"],
        user_id=canvas["user_id"],
        action="present",
        canvas_data=execution.output_data
    )
    canvas["session"].add(audit)
    canvas["session"].commit()

    # Step 4: Verify workflow complete
    assert execution.status == "completed"
    assert audit.action == "present"
```

## Contributing

When adding new E2E tests:

1. **Create Test File**: `tests/e2e/test_new_feature_e2e.py`
2. **Use E2E Fixtures**: Leverage existing fixtures from `fixtures/`
3. **Test Real Services**: Use PostgreSQL, Redis, Docker (not mocks)
4. **Validate Workflows**: Test complete user journeys
5. **Add Performance Assertions**: Use `workflow_performance_thresholds`
6. **Update README**: Document new tests in this file
7. **Update conftest.py**: Add new fixtures if needed
8. **Verify Coverage**: Run with `--cov` to check coverage contribution

## Resources

### Documentation
- **Main Documentation**: `/Users/rushiparikh/projects/atom/CLAUDE.md`
- **API Documentation**: `/Users/rushiparikh/projects/atom/backend/docs/API_DOCUMENTATION.md`
- **Deployment Guide**: `/Users/rushiparikh/projects/atom/docs/DEPLOYMENT/DEPLOYMENT_RUNBOOK.md`
- **Operations Guide**: `/Users/rushiparikh/projects/atom/backend/docs/OPERATIONS_GUIDE.md`

### Source Code
- **Core Services**: `/Users/rushiparikh/projects/atom/backend/core/`
- **API Routes**: `/Users/rushiparikh/projects/atom/backend/api/`
- **Integrations**: `/Users/rushiparikh/projects/atom/backend/integrations/`
- **Tools**: `/Users/rushiparikh/projects/atom/backend/tools/`

### Test Infrastructure
- **E2E Fixtures**: `/Users/rushiparikh/projects/atom/backend/tests/e2e/fixtures/`
- **Unit Tests**: `/Users/rushiparikh/projects/atom/backend/tests/unit/`
- **Integration Tests**: `/Users/rushiparikh/projects/atom/backend/tests/integration/`

## Support

For issues or questions about E2E testing:
1. Check test docstrings for details
2. Review fixture definitions in `tests/e2e/fixtures/`
3. Inspect service implementations in `core/`
4. Check API route implementations in `api/`
5. Review CI/CD logs for environment-specific issues
6. Check Docker logs: `docker-compose -f docker-compose-e2e.yml logs`

---

**Last Updated**: 2025-02-20
**E2E Test Count**: 217+ tests across 14 test files
**Coverage Target**: 60-70% for MCP service (vs 26.56% baseline)
**Performance Target**: <10 minutes for full suite execution