test/README.md

NL→SQL Leaderboard Tests

This directory contains all test files for the NL→SQL Leaderboard project.

Test Structure

test/
├── __init__.py              # Test package initialization
├── conftest.py              # Pytest configuration and fixtures
├── test_config.py           # Configuration loading tests
├── test_evaluation.py       # Evaluation pipeline tests
├── test_models.py           # Model testing utilities
├── test_system.py           # System integration tests
└── README.md                # This file

Running Tests

Quick Test Run

python run_tests.py

Using pytest directly

# Run all tests
pytest test/

# Run specific test file
pytest test/test_config.py

# Run with coverage
pytest test/ --cov=src --cov-report=html

# Run only fast tests
pytest test/ -m "not slow"

Test Categories

• Unit Tests: Fast, isolated tests for individual components
• Integration Tests: Tests that verify component interactions
• System Tests: End-to-end tests of the complete system

Test Configuration

Tests are configured to run in mock mode by default:

• MOCK_MODE=true - Uses mock SQL generation
• HF_TOKEN="" - Prevents real API calls
• All external dependencies are mocked

Test Fixtures

• mock_mode: Ensures mock mode is enabled
• test_data_dir: Path to test data directory
• config_dir: Path to configuration directory

Writing New Tests

1. Create test files with test_*.py naming
2. Use descriptive test function names starting with test_
3. Use fixtures from conftest.py when needed
4. Mark slow tests with @pytest.mark.slow
5. Use proper assertions and error messages

Test Coverage

The test suite aims for comprehensive coverage:

• Configuration loading and validation
• Model registry functionality
• Evaluation pipeline
• Scoring and metrics
• UI components
• Error handling

Continuous Integration

Tests are designed to run in CI/CD environments:

• No external dependencies required
• Mock mode prevents API calls
• Fast execution for quick feedback
• Comprehensive coverage reporting