docs/development/testing.md

Testing Guide

This document provides comprehensive information about the testing infrastructure for the PDF-TEI-Editor project.

Table of Contents

1. Testing Architecture
2. Quick Reference
3. Unit Tests
4. API Integration Tests
5. End-to-End Tests
6. Smart Test Runner
7. Writing New Tests
8. Debugging Tests

Testing Architecture

The project uses a multi-tiered testing approach:

• Unit Tests: JavaScript (Node.js test runner) and Python (pytest) for isolated component testing
• API Integration Tests: Node.js-based tests against local or containerized FastAPI backend
• End-to-End Tests: Playwright browser tests against containerized application
• Smart Test Selection: Dependency-aware test execution based on @testCovers annotations

Test Directory Structure

tests/
├── unit/                        # Unit tests
│   ├── js/                      # JavaScript unit tests (*.test.js)
│   ├── fastapi/                 # FastAPI unit tests (test_*.py)
│   └── flask/                   # Legacy Flask unit tests (test_*.py)
├── api/                         # API integration tests
│   ├── v1/                      # API v1 tests (*.test.js)
│   ├── helpers/                 # Shared test utilities
│   └── fixtures/                # Test data fixtures
├── e2e/                         # End-to-end tests
│   ├── tests/                   # Playwright test specs (*.spec.js)
│   ├── tests/helpers/           # E2E test helpers
│   └── fixtures/                # E2E test fixtures
├── lib/                         # Test infrastructure
│   ├── local-server-manager.js  # Local server management
│   ├── container-server-manager.js  # Container management
│   └── ...                      # Other test utilities
├── backend-test-runner.js       # API test runner (local/container)
├── e2e-runner.js               # E2E test runner (local/container)
├── smart-test-runner.js        # Intelligent test selection
├── unit-test-runner.js         # JavaScript unit test runner
└── unit-test-runner.py         # Python unit test runner

Quick Reference

Common Test Commands

# Run all tests
npm test

# Run only changed tests (smart selection)
npm run test:changed

# Unit Tests
npm run test:unit              # All unit tests (JS + Python)
npm run test:unit:js           # JavaScript unit tests only
npm run test:unit:fastapi      # FastAPI Python unit tests

# API Integration Tests (FastAPI backend)
npm run test:api               # Local server (fastest)

# End-to-End Tests (Playwright)
npm run test:e2e               # Local server (fastest)
npm run test:e2e:headed        # Show browser UI
npm run test:e2e:debug         # Step-through debugging
npm run test:e2e:debug-failure # Capture debug artifacts on failure

# Cross-browser E2E tests (real browser engines, no login required)
npm run test:e2e:xmleditor-browsers  # xmlTagSync tests in chromium, firefox, webkit

# Container Tests (runs all tests inside container, same as CI)
npm run test:container                                  # Run with cache
npm run test:container -- --no-cache                    # Rebuild all layers
npm run test:container -- path/to/file.js               # Test specific files
npm run test:container -- --browser firefox             # Use specific browser
npm run test:container -- --browser chromium,firefox,webkit  # Test multiple browsers

# Run specific tests
npm run test:api -- --grep "save"
npm run test:e2e -- --grep "authentication"

Test Runner Options

Local test runners (test:api, test:e2e) support:

# Filter tests by pattern
--grep <pattern>               # Run matching tests
--grep-invert <pattern>        # Exclude matching tests

# Database management
--clean-db                     # Wipe database (default for local)
--keep-db                      # Preserve database between runs

# Browser selection (E2E only)
--browser <name>               # Use specific browser (chromium, firefox, webkit)

# Other options
--verbose                      # Show detailed output
--no-cleanup                   # Keep server running after tests

Container test runner (test:container) options:

--no-cache                     # Rebuild all Docker layers (ignore cache)
--browser <browsers>           # Comma-separated list for E2E tests
--all                          # Run all tests (skip smart selection)
--grep <pattern>               # Filter tests by pattern
# Plus any other smart-test-runner.js options

Unit Tests

Unit tests validate individual components in isolation without external dependencies.

JavaScript Unit Tests

Location: tests/unit/js/ Runner: Node.js built-in test runner Command: npm run test:unit:js

Example:

import { test } from 'node:test';
import assert from 'node:assert';
import { PluginManager } from '../../app/src/modules/plugin-manager.js';

test('plugin manager registration', () => {
  const manager = new PluginManager();
  const plugin = { name: 'test-plugin', install: () => {} };

  manager.register(plugin);
  assert.strictEqual(manager.plugins.length, 1);
});

Python Unit Tests

Location: tests/unit/fastapi/ (FastAPI) and tests/unit/flask/ (legacy) Runner: pytest Command: npm run test:unit:fastapi

Example:

import pytest
from fastapi_app.lib.utils.auth import verify_password, hash_password

def test_password_hashing():
    password = "test123"
    hashed = hash_password(password)
    assert verify_password(password, hashed)
    assert not verify_password("wrong", hashed)

API Integration Tests

API integration tests validate backend endpoints without a browser. They run against a local FastAPI server.

Location: tests/api/v1/ Naming: *.test.js Runner: backend-test-runner.js Command: npm run test:api

Key Features

• Fast Iteration: Local mode starts/stops server automatically
• Database Management: Auto-wipes DB between runs (configurable)
• Fixture Support: Load test data from tests/api/fixtures/
• Authentication Helpers: Built-in session management
• Lock Management: Automatic cleanup between tests

Writing API Tests

API tests use Node.js built-in test runner with helper utilities:

/**
 * @testCovers fastapi_app/routers/files_save.py
 */
import { test, describe } from 'node:test';
import assert from 'node:assert';
import { login, authenticatedApiCall } from '../helpers/test-auth.js';
import { logger } from '../helpers/test-logger.js';

const BASE_URL = process.env.E2E_BASE_URL || 'http://localhost:8000';

describe('Files Save API', () => {
  let session = null;

  test('Setup: login as reviewer', async () => {
    session = await login('reviewer', 'reviewer', BASE_URL);
    assert.ok(session?.sessionId);
  });

  test('should create new gold standard file', async () => {
    const response = await authenticatedApiCall(
      session.sessionId,
      '/files/save',
      'POST',
      {
        file_id: 'test-doc',
        xml_string: '<TEI>...</TEI>'
      },
      BASE_URL
    );

    assert.strictEqual(response.status, 'new_gold');
    assert.ok(response.file_id);
    logger.success(`Created file: ${response.file_id}`);
  });
});

Authentication Helpers

import { login, authenticatedApiCall, createTestSession } from '../helpers/test-auth.js';

// Login with specific user
const session = await login('reviewer', 'reviewer', BASE_URL);

// Make authenticated API call
const result = await authenticatedApiCall(
  session.sessionId,
  '/files/save',
  'POST',
  { file_id: 'test', xml_string: '<TEI/>' },
  BASE_URL
);

// Create session (uses default 'testuser')
const defaultSession = await createTestSession(BASE_URL);

Lock Cleanup

Always clean up locks in test teardown:

import { clearAllLocks } from '../helpers/test-cleanup.js';

test('Cleanup: release locks', async () => {
  await clearAllLocks(BASE_URL);
});

Running API Tests

# Local server (fast iteration)
npm run test:api

# With database preservation
npm run test:api -- --keep-db

# Specific tests
npm run test:api -- --grep "save"

# In container (CI environment)
npm run test:container -- --grep "save"

End-to-End Tests

E2E tests use Playwright to test the full application stack in a browser.

Location: tests/e2e/tests/ Naming: *.spec.js Runner: e2e-runner.js Command: npm run test:e2e (local) or npm run test:container (containerized)

Key Features

• Full Browser Testing: Chromium, Firefox, WebKit support
• Containerized Environment: Isolated test instances
• UI Navigation System: Type-safe access via window.ui
• Test Logging: Structured state verification via testLog()
• Headed Mode: Visual debugging with --headed
• Step-through Debugging: Playwright debugger with --debug
• Isolated Harness Tests: Component-level cross-browser tests without login or application state (see below)

Isolated Component Harness Tests

Some specs test individual components in isolation using a standalone HTML harness page served by the dev server. The harness loads only the component's dependencies via the importmap — no login, no fixtures, no application state.

Use when the component has browser-engine-specific behavior or when you need to reproduce an editor bug without full-application overhead.

Harness HTML	Spec	Tests
tests/e2e/harness/xmleditor-harness.html	tests/e2e/tests/xmleditor-cross-browser.spec.js	xmlTagSync CodeMirror extension

See the Testing Guide for instructions on running and extending harness tests.

Writing E2E Tests

/**
 * @testCovers app/src/plugins/authentication.js
 * @testCovers fastapi_app/routers/auth.py
 */
/** @import { namedElementsTree } from '../../app/src/ui.js' */
import { test, expect } from '../fixtures/debug-on-failure.js';
import { performLogin, performLogout } from './helpers/login-helper.js';

test.describe('Authentication Workflow', () => {
  test('should login successfully', async ({ page }) => {
    await page.goto('http://localhost:8000');

    await performLogin(page, 'testuser', 'testpass');

    // Verify login using UI navigation
    const username = await page.evaluate(() => {
      /** @type {namedElementsTree} */
      const ui = /** @type {any} */(window).ui;
      return ui.toolbar.userMenu.textContent;
    });

    expect(username).toContain('testuser');
  });
});

UI Navigation System

Access UI elements using the typed navigation system:

await page.evaluate(() => {
  /** @type {namedElementsTree} */
  const ui = /** @type {any} */(window).ui;

  // Form interactions
  ui.loginDialog.username.value = 'testuser';
  ui.loginDialog.password.value = 'testpass';
  ui.loginDialog.submit.click();

  // Read state
  return ui.loginDialog.open; // boolean
});

Using the API Client in Browser Context

CRITICAL: Code running in browser context (inside page.evaluate()) should never use manual fetch() calls to the API backend. Instead, always use the client object exposed as a global:

/**
 * @import { api as Client } from '../../../app/src/plugins/client.js'
 */

// Inside page.evaluate():
const result = await page.evaluate(async () => {
  /** @type {Client} */
  const client = /** @type {any} */(window).client;

  // Use the typed API client
  return await client.apiClient.sseTestProgress({
    steps: 3,
    delay_ms: 500,
    label_prefix: 'E2E Test step'
  });
});

Why:

• The client.apiClient provides typed methods for all API endpoints
• Handles authentication headers automatically
• See app/src/modules/api-client-v1.js for all available API methods (auto-generated from OpenAPI schema)

Test Logging System

Use testLog() for state verification instead of DOM queries:

import { setupTestConsoleCapture, waitForTestMessage } from './helpers/test-logging.js';

// Set up console capture
const consoleLogs = setupTestConsoleCapture(page);

// Perform action...
await page.evaluate(() => {
  window.client.saveXml(/* ... */);
});

// Wait for and verify state change
const saveLog = await waitForTestMessage(consoleLogs, 'FILE_SAVED');
expect(saveLog.value.file_id).toBeTruthy();
expect(saveLog.value.status).toBe('saved');

Running E2E Tests

# Local server (fastest)
npm run test:e2e

# Show browser UI
npm run test:e2e:headed

# Step-through debugging
npm run test:e2e:debug

# Specific browser
npm run test:e2e -- --browser firefox

# Specific tests
npm run test:e2e -- --grep "authentication"

# In container (CI environment)
npm run test:container -- --browser chromium
npm run test:container -- --browser chromium,firefox,webkit  # Multiple browsers

Smart Test Runner

The smart test runner automatically selects tests based on file dependencies, dramatically reducing test execution time.

How It Works

1. Scans test files for @testCovers annotations
2. Compares changed files against test dependencies
3. Runs only affected tests plus wildcard tests

Usage

# Run tests for changed files (git diff) - local server
npm run test:changed

# Run tests for changed files - in container (CI environment)
npm run test:container

# Test specific files
npm run test:container -- app/src/plugins/auth.js fastapi_app/routers/auth.py

# Dry run (show which tests would run)
node tests/smart-test-runner.js --changed-files app/src/ui.js --dry-run

# Run all tests
npm test  # Local
npm run test:container -- --all  # Container

Test Coverage Annotations

Add @testCovers comments to link tests to source files:

/**
 * @testCovers app/src/plugins/authentication.js
 * @testCovers fastapi_app/routers/auth.py
 * @testCovers app/src/modules/api-client.js
 */
test('authentication workflow', async ({ page }) => {
  // Test code...
});

/**
 * @testCovers app/src/*
 */
test('frontend smoke test', async ({ page }) => {
  // Runs when any frontend file changes
});

Supported patterns:

• Exact: app/src/ui.js
• Wildcard: app/src/* (all files in directory)
• Recursive: app/src/**/*.js (all JS files recursively)

Writing New Tests

General Guidelines

1. Add @testCovers annotations for smart test selection
2. Clean up after tests - Release locks, delete test files
3. Use helper functions - Don't duplicate authentication/setup code
4. Sequential vs Parallel - Use describe.serial() for dependent tests
5. Meaningful assertions - Test behavior, not implementation details

Test Organization

• Unit tests: Test single functions/classes in isolation
• API tests: Test endpoint behavior and business logic
• E2E tests: Test complete user workflows

Naming Conventions

• Unit tests: feature.test.js or test_feature.py
• API tests: resource_action.test.js (e.g., files_save.test.js)
• E2E tests: workflow-description.spec.js (e.g., auth-workflow.spec.js)

Example Test Structure

describe('Feature Name', () => {
  // Setup
  test('Setup: create test data', async () => {
    // Initialize test state
  });

  // Main tests
  test('should handle success case', async () => {
    // Test implementation
  });

  test('should handle error case', async () => {
    // Test error handling
  });

  // Cleanup
  test('Cleanup: remove test data', async () => {
    // Clean up resources
  });
});

Debugging Tests

API Test Debugging

# Verbose output
npm run test:api -- --verbose --grep "save"

# Keep database for inspection
npm run test:api -- --keep-db --no-cleanup

# Check specific endpoint
curl -X POST http://localhost:8000/api/files/save \
  -H "Content-Type: application/json" \
  -d '{"file_id":"test","xml_string":"<TEI/>"}'

E2E Test Debugging

# Show browser UI
npm run test:e2e:headed -- --grep "auth"

# Step-through debugging (Playwright debugger)
npm run test:e2e:debug -- --grep "auth"

# Capture debug artifacts on failure
npm run test:e2e:debug-failure -- --grep "auth"

# Add breakpoints in test code
await page.pause(); // Pauses execution

Debug-on-Failure Mode: When using npm run test:e2e:debug-failure, failed tests will:

• Stop on first failure
• Capture console messages to console-messages.json
• Capture page errors to page-errors.json
• Take screenshots automatically
• Record video of the test execution
• Save all artifacts to tests/e2e/test-results/<test-name>/

This is particularly useful for debugging failures where you need to understand what happened during test execution.

Implementation: All E2E tests import from ../fixtures/debug-on-failure.js by default, which enables this feature when the --debug-on-failure flag is used. During normal test runs, the fixture has no effect.

Common Issues

Lock Conflicts:

• Use --clean-db to reset locks
• Ensure tests call cleanup helpers

Port Conflicts:

• Test runners auto-select available ports
• Check for stale server processes: lsof -i :8000

Container Issues:

• Rebuild image: remove --no-rebuild flag
• Check logs: docker logs <container-id>

Test Timeouts:

• Increase timeout: --timeout 180 (seconds)
• Check server startup logs

Debug Logging

Enable verbose output:

# API tests
npm run test:api -- --verbose

# E2E tests with debug messages
E2E_DEBUG=true npm run test:e2e

Continuous Integration

Pre-push Hooks

Smart test runner automatically runs on git push:

# Runs affected tests only (local server)
git push

GitHub CI Testing Workflow

The PR testing workflow (.github/workflows/pr-tests.yml) uses an optimized three-path strategy to minimize CI time while ensuring comprehensive test coverage.

Workflow Overview

1. Analyze changed files
2. Determine which tests need to run (--names-only)
3. Choose execution strategy:
   ├─ No tests → Skip (30 seconds)
   ├─ Unit/API only → Native execution (2-5 minutes)
   └─ E2E included → Container execution (10-15 minutes)

Execution Paths

Path 1: No Tests Needed (~30 seconds)

Triggers when: Changed files don't affect any tested code (documentation, configs, etc.)

Steps:
1. Checkout code
2. Install Node.js
3. Run smart test runner with --names-only
4. Detect: No test files in output
5. Skip all test execution
6. Comment PR: "✅ No tests needed!"

Example scenarios:

• Documentation updates (README.md, docs/)
• Configuration changes (.env.example, .gitignore)
• Non-tested utility scripts

Path 2: Native Execution (~2-5 minutes)

Triggers when: Only unit/API tests detected (no E2E tests)

Steps:
1. Checkout code
2. Install Node.js, Python, uv
3. Run smart test runner with --names-only
4. Detect: Only tests/unit/ or tests/api/ in output
5. Install dependencies (npm ci, uv sync)
6. Run tests natively: npm run test:changed
7. Comment PR: "✅ All tests passed! (native execution)"

Example scenarios:

• Backend-only changes (fastapi_app/routers/files.py)
• JavaScript module changes (app/src/modules/state-manager.js)
• Python utility updates (fastapi_app/lib/utils/auth.py)

Path 3: Container Execution (~10-15 minutes)

Triggers when: E2E tests detected in output

Steps:
1. Checkout code
2. Install Node.js, Python, uv
3. Run smart test runner with --names-only
4. Detect: tests/e2e/ in output
5. Build Docker container (with caching)
6. Run all tests in container: docker run pdf-tei-editor:ci
7. Comment PR: "✅ All tests passed! (containerized)"

Example scenarios:

• Frontend UI changes (app/src/plugins/xmleditor.js)
• Full-stack features affecting UI
• Changes explicitly annotated with E2E test coverage

Test Detection Logic

The workflow uses the smart test runner's --names-only option:

# Get list of test files (one per line)
TEST_FILES=$(npm run test:changed -- --names-only <changed-files>)

# Check if E2E tests are present
if echo "$TEST_FILES" | grep -q "tests/e2e/"; then
  # Use container execution (Playwright browsers needed)
  needs_e2e=true
else
  # Use native execution (faster)
  needs_e2e=false
fi

Performance Comparison

PR Type	Changed Files	Tests Detected	Execution	Old Time	New Time	Saved
Docs	README.md	None	Skip	~10 min	~30 sec	~9.5 min
Backend	files.py	Unit + API	Native	~10 min	~3 min	~7 min
Frontend module	state-manager.js	Unit only	Native	~10 min	~2 min	~8 min
UI component	xmleditor.js	Unit + E2E	Container	~10 min	~10 min	0 min

Average time savings: 5-8 minutes per PR (for 70% of PRs that don't need E2E tests)

Native vs Container Execution

Native Execution (Unit/API tests):

# Setup
npm ci                    # Install Node dependencies
uv sync                   # Create Python virtual environment

# Execution
npm run test:changed      # Runs smart test runner
├─ JS unit tests: node tests/unit-test-runner.js
├─ Python unit tests: uv run python tests/unit-test-runner.py
└─ API tests: node tests/backend-test-runner.js (local FastAPI server)

Container Execution (E2E tests):

# Build
docker build --target ci  # Build test container image

# Execution
docker run pdf-tei-editor:ci <changed-files>
├─ All tests run inside container
├─ Playwright browsers pre-installed
└─ Isolated test environment

Local Equivalent

Replicate the CI workflow locally:

# Check which tests would run (like CI does)
npm run test:changed -- --names-only

# Run tests natively (like CI Path 2)
npm run test:changed

# Run tests in container (like CI Path 3)
npm run test:container

# Force all tests in container
npm run test:container -- --all

Debugging CI Failures

If tests pass locally but fail in CI:

1. Check environment differences:

   # CI uses clean install
   npm ci  # vs npm install
   
   # CI uses specific Node/Python versions
   node -v  # Should match workflow (20.x)
   python -v  # Should match workflow (3.11)
   

2. Run in container locally:

   # Exact same environment as CI
   npm run test:container
   
   # Force rebuild (ignore cache)
   npm run test:container -- --no-cache
   

3. Check test isolation:

   # CI always starts with clean state
   # Verify tests clean up properly
   npm run test:changed -- --keep-db  # Check for leaks
   

If container build fails:

1. Check Dockerfile changes:

   # Test build locally
   docker build --target ci -t pdf-tei-editor:ci .
   

2. Check dependency versions:

   # Verify package.json and pyproject.toml
   npm ci
   uv sync
   

CI Architecture

Design Principles:

• Smart test selection: Only run affected tests
• Fail fast: Stop on first failure
• Progressive optimization: Fast path for common cases
• Container when needed: E2E tests require browsers
• Real-time feedback: Stream output, immediate PR comments
• Caching strategy: Docker layer cache, npm/pip caching

Technical Details:

• Node.js 20.x for JavaScript execution
• Python 3.11 for FastAPI backend
• uv for Python dependency management
• Docker Buildx for efficient builds
• GitHub Actions caching for Docker layers
• Concurrent test execution where possible

Test Fixtures

API Test Fixtures

Located in tests/api/fixtures/:

• minimal: Bare minimum config for smoke tests
• standard: Full config with sample data

Fixtures are automatically loaded by backend-test-runner.js.

E2E Test Fixtures

Located in tests/e2e/fixtures/:

• minimal: Basic setup for quick tests
• standard: Complete environment with sample files

Fixtures include:

• Config files (config/)
• Sample PDF/TEI files (files/)
• User credentials

Additional Resources

• Test Infrastructure: See tests/lib/ for server management utilities
• Helper Functions: Check tests/*/helpers/ for shared test utilities
• Example Tests: Review existing tests in each category for patterns