specs/F005-RESEARCH_SUMMARY.md

# Research Summary

**Project:** SQLEnv
**Change:** F005 — Green Agent Wrapper (automated evaluation)
**Date:** 2026-03-27
**Status:** Draft

---

## 1. Change Overview

### What We're Changing
Create an automated evaluation wrapper that runs N episodes with a given policy and reports metrics (success_rate, avg_reward, avg_steps). Includes a built-in random baseline policy. Follows the OpenEnv Green Agent pattern.

### Why We're Changing It
Required by competition evaluation criteria. Enables training comparison: "random policy gets 5% success, trained model gets 40%." Single command, structured output.

### Success Criteria
- Single function call: `evaluate(n_episodes=100)` returns clean metrics dict
- Built-in random policy for instant baseline comparison
- Results include per-episode breakdown for analysis
- Doesn't crash partway through and lose results

---

## 2. System Context

### Current Behavior
No evaluation wrapper exists. Manual testing only via `tests/test_smoke.py`.

### Architecture Context
```
evaluate(env, policy, n_episodes)
  ├── for each episode:
  │   ├── env.reset()
  │   ├── while not done: policy.select_action(obs) → env.step(action)
  │   └── collect {correct, total_reward, steps}
  └── aggregate → {success_rate, avg_reward, avg_steps, per_episode}
```

Client-side component — uses environment through public `reset()`/`step()` API.

### Entry Points

| Entry Point | Trigger | Current Flow |
|-------------|---------|--------------|
| `evaluate()` | Training script or CLI | **To be created** |
| `RandomPolicy.select_action()` | Called by evaluate loop | **To be created** |

### Data Flow

| Data | Source | Shape/Type | Destination |
|------|--------|------------|-------------|
| Observation | `env.reset()` / `env.step()` | `SQLObservation` | Policy |
| Action | Policy | `SQLAction` | `env.step()` |
| Episode results | Loop | `list[EpisodeResult]` | Aggregation |
| Metrics | Aggregation | `dict` | Caller |

---

## 3. Dependencies

### Code We Depend On

| Dependency | What We Use | Risk if Changed |
|------------|-------------|-----------------|
| `models.py:SQLAction, SQLObservation` | Action/observation types | Stable (F001 complete) |
| `sql_environment.py:SQLEnvironment` | `reset()`, `step()` API | Stable (F001 complete) |

### Code That Depends On Us

| Dependent | How They Use Us | Impact of Our Change |
|-----------|-----------------|---------------------|
| F006 (GRPO Training) | Baseline comparison + evaluation | Provides metrics API |
| F007 (HF Submission) | Demo results for blog | Produces numbers |

---

## 4. Risks & Edge Cases

### Identified Risks

| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| Evaluation crashes partway | Medium | Loses results | Collect incrementally, return partial on error |
| No progress indicator | Medium | User thinks hung | Optional tqdm or callback |

### Edge Cases to Handle

| Edge Case | Current Behavior | Required Behavior |
|-----------|------------------|-------------------|
| n_episodes=0 | N/A | Return empty metrics |
| Policy exception mid-episode | N/A | Catch, record as failed, continue |
| Environment reset fails | N/A | Skip, log warning, continue |

### Invariants to Preserve

- [ ] Evaluation is read-only — doesn't modify environment between episodes
- [ ] Random policy is deterministic given a seed
- [ ] Metrics match manual calculation

---

## 4b. Code Shape & Design Target

### Target Shape

| Component | Purpose | Why This Boundary |
|-----------|---------|-------------------|
| `evaluate(env, policy, n_episodes, seed)` | Main entry | Single public function |
| `RandomPolicy` | Built-in random baseline | Needed for comparison |
| `Policy` (Protocol) | Type hint for custom policies | Duck typing |
| `EpisodeResult` (dataclass) | Per-episode metrics | Clean structure |

### Abstraction Level

- **Recommendation:** One module `green_agent.py` at project root. Function + dataclass + random policy class.

### Anti-Patterns to Avoid

- Don't create elaborate policy class hierarchy
- Don't couple to WebSocket transport — work with local env directly
- Don't add visualization/plotting (MVP)

---

## 5. Constraints

| Constraint | Requirement | Notes |
|------------|-------------|-------|
| No new heavy deps | tqdm optional | Keep lean |
| Works with local env | Direct SQLEnvironment | Primary use case |
| Seedable | Reproducible results | Random policy + env seed |

---

## 6. Open Questions

| Question | Why It Matters | Who Can Answer |
|----------|----------------|----------------|
| Module location: `green_agent.py` at root? | Naming | Recommend root, matches concept doc |
| Should RandomPolicy use schema info for smarter random? | Baseline quality | Recommend simple random |

---

## 7. Context Sources

| Source | Type | Notes |
|--------|------|-------|
| `docs_draft/SQLEnv_Concept_v1.md` Appendix C | Doc | SQLGreenAgent sketch |
| `server/sql_environment.py` | Code | reset()/step() API |
| `models.py` | Code | SQLAction, SQLObservation |