File size: 5,106 Bytes
5dd1bb4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 | # Research Summary
**Project:** SQLEnv
**Change:** F005 — Green Agent Wrapper (automated evaluation)
**Date:** 2026-03-27
**Status:** Draft
---
## 1. Change Overview
### What We're Changing
Create an automated evaluation wrapper that runs N episodes with a given policy and reports metrics (success_rate, avg_reward, avg_steps). Includes a built-in random baseline policy. Follows the OpenEnv Green Agent pattern.
### Why We're Changing It
Required by competition evaluation criteria. Enables training comparison: "random policy gets 5% success, trained model gets 40%." Single command, structured output.
### Success Criteria
- Single function call: `evaluate(n_episodes=100)` returns clean metrics dict
- Built-in random policy for instant baseline comparison
- Results include per-episode breakdown for analysis
- Doesn't crash partway through and lose results
---
## 2. System Context
### Current Behavior
No evaluation wrapper exists. Manual testing only via `tests/test_smoke.py`.
### Architecture Context
```
evaluate(env, policy, n_episodes)
├── for each episode:
│ ├── env.reset()
│ ├── while not done: policy.select_action(obs) → env.step(action)
│ └── collect {correct, total_reward, steps}
└── aggregate → {success_rate, avg_reward, avg_steps, per_episode}
```
Client-side component — uses environment through public `reset()`/`step()` API.
### Entry Points
| Entry Point | Trigger | Current Flow |
|-------------|---------|--------------|
| `evaluate()` | Training script or CLI | **To be created** |
| `RandomPolicy.select_action()` | Called by evaluate loop | **To be created** |
### Data Flow
| Data | Source | Shape/Type | Destination |
|------|--------|------------|-------------|
| Observation | `env.reset()` / `env.step()` | `SQLObservation` | Policy |
| Action | Policy | `SQLAction` | `env.step()` |
| Episode results | Loop | `list[EpisodeResult]` | Aggregation |
| Metrics | Aggregation | `dict` | Caller |
---
## 3. Dependencies
### Code We Depend On
| Dependency | What We Use | Risk if Changed |
|------------|-------------|-----------------|
| `models.py:SQLAction, SQLObservation` | Action/observation types | Stable (F001 complete) |
| `sql_environment.py:SQLEnvironment` | `reset()`, `step()` API | Stable (F001 complete) |
### Code That Depends On Us
| Dependent | How They Use Us | Impact of Our Change |
|-----------|-----------------|---------------------|
| F006 (GRPO Training) | Baseline comparison + evaluation | Provides metrics API |
| F007 (HF Submission) | Demo results for blog | Produces numbers |
---
## 4. Risks & Edge Cases
### Identified Risks
| Risk | Likelihood | Impact | Mitigation |
|------|------------|--------|------------|
| Evaluation crashes partway | Medium | Loses results | Collect incrementally, return partial on error |
| No progress indicator | Medium | User thinks hung | Optional tqdm or callback |
### Edge Cases to Handle
| Edge Case | Current Behavior | Required Behavior |
|-----------|------------------|-------------------|
| n_episodes=0 | N/A | Return empty metrics |
| Policy exception mid-episode | N/A | Catch, record as failed, continue |
| Environment reset fails | N/A | Skip, log warning, continue |
### Invariants to Preserve
- [ ] Evaluation is read-only — doesn't modify environment between episodes
- [ ] Random policy is deterministic given a seed
- [ ] Metrics match manual calculation
---
## 4b. Code Shape & Design Target
### Target Shape
| Component | Purpose | Why This Boundary |
|-----------|---------|-------------------|
| `evaluate(env, policy, n_episodes, seed)` | Main entry | Single public function |
| `RandomPolicy` | Built-in random baseline | Needed for comparison |
| `Policy` (Protocol) | Type hint for custom policies | Duck typing |
| `EpisodeResult` (dataclass) | Per-episode metrics | Clean structure |
### Abstraction Level
- **Recommendation:** One module `green_agent.py` at project root. Function + dataclass + random policy class.
### Anti-Patterns to Avoid
- Don't create elaborate policy class hierarchy
- Don't couple to WebSocket transport — work with local env directly
- Don't add visualization/plotting (MVP)
---
## 5. Constraints
| Constraint | Requirement | Notes |
|------------|-------------|-------|
| No new heavy deps | tqdm optional | Keep lean |
| Works with local env | Direct SQLEnvironment | Primary use case |
| Seedable | Reproducible results | Random policy + env seed |
---
## 6. Open Questions
| Question | Why It Matters | Who Can Answer |
|----------|----------------|----------------|
| Module location: `green_agent.py` at root? | Naming | Recommend root, matches concept doc |
| Should RandomPolicy use schema info for smarter random? | Baseline quality | Recommend simple random |
---
## 7. Context Sources
| Source | Type | Notes |
|--------|------|-------|
| `docs_draft/SQLEnv_Concept_v1.md` Appendix C | Doc | SQLGreenAgent sketch |
| `server/sql_environment.py` | Code | reset()/step() API |
| `models.py` | Code | SQLAction, SQLObservation |
|