specs/F005-VERIFICATION_SPEC.md

# Verification Specification

**Feature:** F005
**Generated from:** specs/F005-VERIFICATION_INPUT.json
**Generated:** 2026-03-27

---

## 1. Unit Tests

### 1.1 EpisodeResult (frozen dataclass)

| Test | Description | Input | Expected | Category |
|------|-------------|-------|----------|----------|
| test_episode_result_creation | Happy path construction | `EpisodeResult(episode_index=0, correct=True, total_reward=1.0, steps=5, error=None)` | All fields accessible, values match | happy |
| test_episode_result_frozen | Cannot mutate after creation | Attempt `result.correct = False` | `FrozenInstanceError` raised | edge |
| test_episode_result_with_error | Episode that failed | `EpisodeResult(episode_index=1, correct=False, total_reward=0.0, steps=0, error="connection error")` | `error` field is `"connection error"` | error |
| test_episode_result_error_default_none | Error field defaults to None | `EpisodeResult(episode_index=0, correct=True, total_reward=1.0, steps=3)` | `error is None` | happy |

**Run:** `uv run pytest tests/unit/test_evaluation.py -v -k "EpisodeResult"`

### 1.2 EvaluationResult (frozen dataclass)

| Test | Description | Input | Expected | Category |
|------|-------------|-------|----------|----------|
| test_evaluation_result_creation | Happy path with episodes | `EvaluationResult(success_rate=0.5, avg_reward=0.75, avg_steps=3.0, n_episodes=2, n_completed=2, episodes=[...])` | All fields match | happy |
| test_evaluation_result_frozen | Cannot mutate after creation | Attempt `result.success_rate = 1.0` | `FrozenInstanceError` raised | edge |
| test_evaluation_result_empty_episodes | Zero episodes edge case | `EvaluationResult(success_rate=0.0, avg_reward=0.0, avg_steps=0.0, n_episodes=0, n_completed=0, episodes=[])` | Valid construction, all zeros | edge |
| test_evaluation_result_partial_completion | Some episodes failed | `n_episodes=10, n_completed=7` | `n_completed < n_episodes` allowed | edge |
| test_evaluation_result_success_rate_bounds | Success rate between 0 and 1 | `success_rate=0.0` and `success_rate=1.0` | Both valid | edge |

**Run:** `uv run pytest tests/unit/test_evaluation.py -v -k "EvaluationResult"`

### 1.3 Policy Protocol

| Test | Description | Input | Expected | Category |
|------|-------------|-------|----------|----------|
| test_policy_protocol_compliance | Object with select_action satisfies Policy | Custom class with `select_action(obs) -> SQLAction` | `isinstance(obj, Policy)` or structural match | happy |
| test_policy_protocol_missing_method | Object without select_action | Plain object | Does NOT satisfy Protocol | error |

**Run:** `uv run pytest tests/unit/test_evaluation.py -v -k "Policy"`

### 1.4 RandomPolicy.__init__

| Test | Description | Input | Expected | Category |
|------|-------------|-------|----------|----------|
| test_random_policy_default_seed | No seed provided | `RandomPolicy()` | Constructs successfully | happy |
| test_random_policy_with_seed | Explicit seed | `RandomPolicy(seed=42)` | Constructs successfully | happy |
| test_random_policy_none_seed | Explicit None seed | `RandomPolicy(seed=None)` | Constructs successfully | happy |

**Run:** `uv run pytest tests/unit/test_evaluation.py -v -k "random_policy_init or random_policy_default or random_policy_with_seed or random_policy_none"`

### 1.5 RandomPolicy.select_action

| Test | Description | Input | Expected | Category |
|------|-------------|-------|----------|----------|
| test_random_policy_explores_when_budget_gt_1 | Budget > 1 means exploration | Observation with `budget_remaining=10` | Returns SQLAction with `action_type` in `{DESCRIBE, SAMPLE, QUERY}` | happy |
| test_random_policy_answers_when_budget_eq_1 | Budget == 1 forces ANSWER | Observation with `budget_remaining=1` | Returns SQLAction with `action_type == "ANSWER"` | happy |
| test_random_policy_returns_sql_action | Return type is correct | Any valid observation | `isinstance(result, SQLAction)` | happy |
| test_random_policy_deterministic_with_seed | Same seed produces same actions | Two RandomPolicy(seed=42) with identical observations | Same sequence of actions | happy |
| test_random_policy_varies_without_seed | Different runs produce different actions (probabilistic) | Multiple calls without seed | Not all actions identical (run 50 times) | edge |
| test_random_policy_explores_all_action_types | Over many calls, all exploration types appear | Run 100 times with budget > 1 | DESCRIBE, SAMPLE, and QUERY each appear at least once | edge |

**Run:** `uv run pytest tests/unit/test_evaluation.py -v -k "random_policy_select"`

### 1.6 evaluate()

| Test | Description | Input | Expected | Category |
|------|-------------|-------|----------|----------|
| test_evaluate_happy_path | Run N episodes successfully | `evaluate(env, policy, n_episodes=5)` | Returns EvaluationResult with `n_episodes=5, n_completed=5` | happy |
| test_evaluate_returns_evaluation_result | Return type correct | Any valid call | `isinstance(result, EvaluationResult)` | happy |
| test_evaluate_default_n_episodes | Default is 100 | `evaluate(env, policy)` | `result.n_episodes == 100` | happy |
| test_evaluate_n_episodes_zero | Zero episodes | `evaluate(env, policy, n_episodes=0)` | `EvaluationResult` with all zeros, empty episodes list | edge |
| test_evaluate_negative_n_episodes | Negative episodes | `evaluate(env, policy, n_episodes=-1)` | Raises `ValueError` | error |
| test_evaluate_success_rate_calculation | Correct fraction | Policy that answers correctly 3 out of 5 times | `success_rate == 0.6` | happy |
| test_evaluate_avg_reward_calculation | Mean reward correct | Known rewards per episode | `avg_reward` matches manual calculation | happy |
| test_evaluate_avg_steps_calculation | Mean steps correct | Known steps per episode | `avg_steps` matches manual calculation | happy |
| test_evaluate_episodes_list_length | Per-episode breakdown | `n_episodes=5` | `len(result.episodes) == 5` | happy |
| test_evaluate_episode_indices | 0-based episode indices | `n_episodes=3` | `[e.episode_index for e in result.episodes] == [0, 1, 2]` | happy |
| test_evaluate_seed_determinism | Same seed produces same results | Two calls with `seed=42, n_episodes=10` | Both EvaluationResults have identical `success_rate, avg_reward, avg_steps` | happy |
| test_evaluate_seed_per_episode | Episode i uses seed+i | `seed=100, n_episodes=3` | env.reset called with seeds 100, 101, 102 (verify via mock) | happy |
| test_evaluate_no_seed_variation | No seed allows variation | Two calls without seed | Results may differ (non-deterministic) | edge |
| test_evaluate_n_episodes_one | Single episode | `n_episodes=1` | Valid result with 1 episode | edge |
| test_evaluate_large_n_episodes | Large run | `n_episodes=500` | Completes without error, correct counts | edge |

**Run:** `uv run pytest tests/unit/test_evaluation.py -v -k "test_evaluate"`

### 1.7 evaluate() -- Error Handling

| Test | Description | Input | Expected | Category |
|------|-------------|-------|----------|----------|
| test_evaluate_episode_exception_recorded | Exception during episode is caught | Policy that raises on episode 2 | Episode 2 has `correct=False, total_reward=0.0, steps=0, error=<message>` | error |
| test_evaluate_continues_after_exception | Failed episode does not stop evaluation | Exception on episode 1 of 5 | `n_episodes=5`, all 5 episodes in result | error |
| test_evaluate_n_completed_excludes_errors | n_completed counts only successes | 2 out of 5 episodes raise | `n_completed == 3` | error |
| test_evaluate_averages_exclude_failed | avg_reward/avg_steps from completed episodes only | 3 completed with known values, 2 failed | Averages match only the 3 completed | error |
| test_evaluate_env_reset_exception | Exception during env.reset() | Mock env.reset() to raise on episode 3 | Episode 3 recorded with error, others complete | error |
| test_evaluate_policy_exception | Exception during select_action() | Mock policy.select_action() to raise | Episode recorded with error, evaluation continues | error |
| test_evaluate_env_step_exception | Exception during env.step() | Mock env.step() to raise | Episode recorded with error, evaluation continues | error |
| test_evaluate_all_episodes_fail | Every episode fails | Policy that always raises | `n_completed=0`, `success_rate=0.0`, `avg_reward=0.0`, `avg_steps=0.0` | error |

**Run:** `uv run pytest tests/unit/test_evaluation.py -v -k "exception or error or fail"`

### 1.8 evaluate() -- Progress Callback

| Test | Description | Input | Expected | Category |
|------|-------------|-------|----------|----------|
| test_evaluate_progress_callback_called | Callback receives updates | Mock callback, `n_episodes=5` | Callback called with `(1,5), (2,5), (3,5), (4,5), (5,5)` | happy |
| test_evaluate_no_callback | None callback is fine | `progress_callback=None` | No error | happy |
| test_evaluate_callback_receives_correct_total | Total matches n_episodes | `n_episodes=10` | Every callback call has `total=10` | happy |

**Run:** `uv run pytest tests/unit/test_evaluation.py -v -k "callback"`

---

## 2. Integration Tests

### Flow: Full Evaluation with RandomPolicy

| Step | Action | Expected | Verification |
|------|--------|----------|--------------|
| 1 | Create SQLEnvironment with test DB and questions | Environment loads successfully | `len(env.questions) > 0` |
| 2 | Create `RandomPolicy(seed=42)` | Policy created | Object has `select_action` method |
| 3 | Call `evaluate(env, RandomPolicy(seed=42), n_episodes=10, seed=0)` | Returns EvaluationResult | `result.n_episodes == 10` |
| 4 | Verify all episodes recorded | Per-episode breakdown present | `len(result.episodes) == 10` |
| 5 | Verify aggregate metrics are consistent | success_rate matches manual count | `result.success_rate == sum(e.correct for e in result.episodes) / 10` |
| 6 | Verify avg_reward consistent | avg_reward matches manual mean | `result.avg_reward == mean([e.total_reward for e in result.episodes if e.error is None])` |
| 7 | Verify determinism | Repeat with same seed | Identical results |

**Run:** `uv run pytest tests/integration/test_evaluation_integration.py -v -k "full_evaluation"`

### Flow: Evaluation with Partial Failures

| Step | Action | Expected | Verification |
|------|--------|----------|--------------|
| 1 | Create environment and a policy that fails on specific episodes | Setup complete | -- |
| 2 | Call `evaluate(env, flaky_policy, n_episodes=5)` | Returns result with mix of successes and failures | `result.n_completed < result.n_episodes` |
| 3 | Inspect failed episodes | Have error field set | `any(e.error is not None for e in result.episodes)` |
| 4 | Inspect successful episodes | Have error=None | Completed episodes have `error is None` and valid metrics |

**Run:** `uv run pytest tests/integration/test_evaluation_integration.py -v -k "partial_failure"`

### Flow: Zero Episodes

| Step | Action | Expected | Verification |
|------|--------|----------|--------------|
| 1 | Call `evaluate(env, policy, n_episodes=0)` | Returns zero-state result | All aggregate values are 0.0, episodes list is empty |

**Run:** `uv run pytest tests/integration/test_evaluation_integration.py -v -k "zero_episodes"`

---

## 3. API Tests

No API endpoints defined for F005. This section is intentionally empty.

---

## 4. E2E Tests

### Scenario: Single-Command Evaluation of Random Baseline

**Setup:** SQLEnvironment initialized with Spider-format test database and questions file.
**Actions:** Call `evaluate(env, RandomPolicy(seed=42), n_episodes=20, seed=0)` and inspect output.
**Expected:**
- Returns EvaluationResult with `n_episodes=20`
- `success_rate` is a float in [0.0, 1.0]
- `avg_reward` is a float
- `avg_steps` is a positive float
- `n_completed == 20` (no errors with valid env + RandomPolicy)
- All 20 EpisodeResult entries present with valid fields
- Deterministic: re-running with same seeds yields identical results

**Run:** `uv run pytest tests/e2e/test_evaluation_e2e.py -v`

### Scenario: Comparison of Two Policies

**Setup:** SQLEnvironment with test data.
**Actions:**
1. Evaluate RandomPolicy(seed=1) over 20 episodes
2. Evaluate a "always answer immediately" policy over 20 episodes
3. Compare results
**Expected:**
- Both return valid EvaluationResult
- Results are structurally comparable (same fields)
- Metrics differ between policies

**Run:** `uv run pytest tests/e2e/test_evaluation_e2e.py -v -k "comparison"`

---

## 5. Edge Cases Checklist

- [ ] n_episodes = 0 returns zero-valued EvaluationResult with empty episodes list
- [ ] n_episodes = -1 raises ValueError immediately
- [ ] n_episodes = 1 works correctly (single episode)
- [ ] All episodes fail -- n_completed=0, averages are 0.0, success_rate is 0.0
- [ ] Exception during env.reset() is caught and recorded
- [ ] Exception during policy.select_action() is caught and recorded
- [ ] Exception during env.step() is caught and recorded
- [ ] RandomPolicy with budget_remaining=1 always returns ANSWER
- [ ] RandomPolicy with budget_remaining > 1 never returns ANSWER
- [ ] Seed determinism: same seed + same n_episodes = identical EvaluationResult
- [ ] Per-episode seeding: episode i uses seed+i for env.reset()
- [ ] Progress callback receives (current, total) for each episode
- [ ] Progress callback=None does not cause errors
- [ ] EpisodeResult and EvaluationResult are frozen (immutable)
- [ ] Large n_episodes (500+) completes without memory issues
- [ ] success_rate is always in [0.0, 1.0]
- [ ] avg_reward and avg_steps computed only from completed (non-error) episodes

---

## 6. Evidence Requirements

| Category | Evidence Type | Example |
|----------|---------------|---------|
| Unit tests | pytest output | `X passed` from `uv run pytest tests/unit/test_evaluation.py -v` |
| Integration | pytest output | `X passed` from `uv run pytest tests/integration/test_evaluation_integration.py -v` |
| E2E | pytest output | `X passed` from `uv run pytest tests/e2e/test_evaluation_e2e.py -v` |
| Edge cases | pytest output | All edge-case tests in checklist pass |
| Determinism | pytest output | Seed-based tests produce identical results across runs |