tests/QUICKSTART.md

CLI Test Suite Quickstart

Prerequisites

Ensure you have the conda environment activated:

conda activate conformal-s

Running Tests

Run all CLI tests

cd /groups/doudna/projects/ronb/conformal-protein-retrieval
pytest tests/test_cli.py -v

Expected output:

tests/test_cli.py::test_main_help PASSED                            [  4%]
tests/test_cli.py::test_main_no_command PASSED                      [  8%]
tests/test_cli.py::test_embed_help PASSED                           [ 12%]
tests/test_cli.py::test_search_help PASSED                          [ 16%]
...
======================== 24 passed in 2.34s ========================

Run a single test

pytest tests/test_cli.py::test_search_with_mock_data -v

Run tests with detailed output

pytest tests/test_cli.py -v -s

The -s flag shows print statements from the code.

Run tests and see which code is tested

pytest tests/test_cli.py --cov=protein_conformal.cli --cov-report=term-missing

What Each Test Does

Help Tests (fast, no computation)

# These verify help text is correct
pytest tests/test_cli.py -k "help" -v

Tests: test_*_help (7 tests)

• Verifies all commands have proper documentation
• Checks that all options are listed
• Confirms command structure is correct

Search Tests (uses mock data)

# These test the search functionality
pytest tests/test_cli.py -k "search" -v

Tests: test_search_* (8 tests)

• Creates small mock embeddings (5x128 and 20x128)
• Tests FAISS similarity search
• Tests threshold filtering
• Tests metadata merging
• Tests edge cases

Probability Tests (uses mock calibration)

# These test probability conversion
pytest tests/test_cli.py -k "prob" -v

Tests: test_prob_* (3 tests)

• Creates mock calibration data
• Tests Venn-Abers probability conversion
• Tests CSV input/output

Calibration Tests (uses mock data)

# These test threshold calibration
pytest tests/test_cli.py -k "calibrate" -v

Tests: test_calibrate_* (2 tests)

• Creates mock similarity/label pairs
• Tests FDR/FNR threshold computation
• Tests multiple calibration trials

Example Test Walkthrough

Let's look at test_search_with_mock_data() in detail:

def test_search_with_mock_data(tmp_path):
    """Test search command with small mock embeddings."""
    # 1. Create mock query embeddings (5 proteins, 128-dim)
    query_embeddings = np.random.randn(5, 128).astype(np.float32)

    # 2. Create mock database embeddings (20 proteins, 128-dim)
    db_embeddings = np.random.randn(20, 128).astype(np.float32)

    # 3. Normalize to unit vectors (for cosine similarity)
    query_embeddings = query_embeddings / np.linalg.norm(...)
    db_embeddings = db_embeddings / np.linalg.norm(...)

    # 4. Save to temporary files
    np.save(tmp_path / "query.npy", query_embeddings)
    np.save(tmp_path / "db.npy", db_embeddings)

    # 5. Run CLI command via subprocess
    subprocess.run([
        sys.executable, '-m', 'protein_conformal.cli',
        'search',
        '--query', str(tmp_path / "query.npy"),
        '--database', str(tmp_path / "db.npy"),
        '--output', str(tmp_path / "results.csv"),
        '--k', '3'
    ])

    # 6. Verify output exists and has correct structure
    df = pd.read_csv(tmp_path / "results.csv")
    assert len(df) == 5 * 3  # 5 queries * 3 neighbors
    assert 'similarity' in df.columns

Understanding Test Failures

Import Errors

ModuleNotFoundError: No module named 'faiss'

Solution: Install dependencies

conda install -c conda-forge faiss-cpu

File Not Found

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/...'

Solution: This shouldn't happen with tmp_path fixture. Check that pytest is creating temp directories.

Assertion Errors

AssertionError: assert 8 == 15

Solution: Check if test expectations match actual behavior. This could indicate:

• Bug in code
• Test expectations wrong
• Random seed not working

Subprocess Errors

subprocess.CalledProcessError: Command returned non-zero exit status 1

Solution: Run the command manually to see error:

python -m protein_conformal.cli search --query test.npy --database db.npy ...

Adding Your Own Test

Template for a new CLI test:

def test_my_new_feature(tmp_path):
    """Test description here."""
    # 1. Create test data
    test_data = np.array([1, 2, 3])
    input_file = tmp_path / "input.npy"
    np.save(input_file, test_data)

    # 2. Run CLI command
    result = subprocess.run(
        [sys.executable, '-m', 'protein_conformal.cli',
         'my-command',
         '--input', str(input_file),
         '--output', str(tmp_path / "output.csv")],
        capture_output=True,
        text=True
    )

    # 3. Check return code
    assert result.returncode == 0

    # 4. Verify output
    output_file = tmp_path / "output.csv"
    assert output_file.exists()

    df = pd.read_csv(output_file)
    assert len(df) > 0
    assert 'expected_column' in df.columns

Debugging Tests

Run test with debugger

pytest tests/test_cli.py::test_search_with_mock_data --pdb

This will drop into Python debugger on failure.

Show print statements

pytest tests/test_cli.py::test_search_with_mock_data -s

This shows any print() statements from the code.

Show warnings

pytest tests/test_cli.py -v -W all

This shows all Python warnings (deprecation, etc.)

Keep temporary files

pytest tests/test_cli.py::test_search_with_mock_data --basetemp=./test_tmp

This keeps temp files in ./test_tmp/ for inspection.

Performance

All 24 CLI tests should complete in < 30 seconds:

• Help tests: ~0.1s each (no computation)
• Mock data tests: ~0.5-2s each (small arrays)
• No GPU required
• No large data files

If tests are slow:

1. Check if GPU is being initialized (use --cpu flag)
2. Check calibration data size (should be < 100 samples in tests)
3. Check for network calls (shouldn't happen in these tests)

Next Steps

After CLI tests pass:

1. Run full test suite: pytest tests/ -v
2. Run paper verification: cpr verify --check syn30
3. Try the CLI on real data: cpr search --query ... --database ...
4. Read TEST_SUMMARY.md for complete test documentation