Add training loop, colab notebook, and final README

Docs/implementation_report.md

@@ -0,0 +1,597 @@
+# Policy-to-Logic RL Environment — Complete Implementation Report
+
+> **Purpose**: Exhaustive description of everything implemented, with exact logic, edge cases, and formulas. Intended for AI-assisted gap analysis against the original plan.
+
+---
+
+## 1. Project Architecture
+
+```
+OpenenvHack/
+├── main.py                          # Entry point: uvicorn on port 7860
+├── Dockerfile                       # Docker SDK deployment for HF Spaces
+├── inference.py                     # LLM agent loop (Qwen2.5-72B via OpenAI API)
+├── pyproject.toml                   # UV project: pydantic, fastapi, uvicorn, openai, huggingface-hub
+├── test_hf_spaces.py                # Remote endpoint tests against HF Spaces
+├── test_all.py                      # Local test runner (starts server, runs tests, stops)
+├── test_local.py / test_endpoints.py # Additional test scripts
+├── policy_to_logic_env/
+│   ├── __init__.py                  # Package exports: models + client
+│   ├── models.py                    # Pydantic models: Action, Observation, State, StepResult
+│   ├── client.py                    # HTTP client wrapper for the environment
+│   ├── openenv.yaml                 # OpenEnv specification file
+│   └── server/
+│       ├── app.py                   # FastAPI app with 6 endpoints
+│       ├── environment.py           # Core environment: reset(), step(), state()
+│       ├── policies.py              # 3 task definitions with clarification maps
+│       ├── ground_truth.py          # Programmatic ground truth + clarification oracle
+│       ├── scenario_generator.py    # 4-strategy scenario generation (seeded)
+│       ├── graders.py               # Rule grading against scenarios
+│       ├── dsl_engine.py            # JSON DSL parser, validator, executor
+│       ├── rewards.py               # 4-component reward system
+│       └── requirements.txt         # Server deps: openenv-core, pydantic, fastapi, uvicorn, requests
+```
+
+**Deployment**: Docker on HF Spaces at `https://godreign-policy2logic.hf.space`, port 7860.
+
+---
+
+## 2. HTTP API (app.py)
+
+Single FastAPI app with CORS `allow_origins=["*"]`. One global `PolicyToLogicEnvironment()` instance (single-session).
+
+| Endpoint | Method | Request Body | Response | Purpose |
+|---|---|---|---|---|
+| `/` | GET | — | `{name, version, status, endpoints, docs, redoc}` | Root probe / API info |
+| `/health` | GET | — | `{status: "ok", environment: "policy_to_logic"}` | Health check |
+| `/tasks` | GET | — | `{tasks: {name: {difficulty, max_steps, scenario_count, valid_decisions, variables}}}` | List all 3 tasks |
+| `/reset` | POST | `{task_name: str \| null}` | `StepResult` (observation + reward=0 + done=false) | Start new episode |
+| `/step` | POST | `{action_type: str, content: str}` | `StepResult` (observation + reward + done) | Take an action |
+| `/state` | GET | — | `PolicyToLogicState` (full episode metadata) | Get current state |
+
+If `task_name` is null or invalid in `/reset`, defaults to `"data_access"`.
+
+---
+
+## 3. Data Models (models.py)
+
+### Action
+```
+action_type: Literal["ask_clarification", "propose_rules", "refine_rules"]
+content: str  # JSON string payload
+```
+
+### Observation (returned in every StepResult)
+```
+policy_text: str          # The natural language policy (always present)
+task_name: str
+step_number: int          # 0 on reset, 1+ on steps
+max_steps: int
+clarification_response: str | None    # Oracle answer if ask_clarification
+test_results: dict | None             # {passed, failed, total, score, sample_failures}
+current_accuracy: float               # 0.0-1.0
+available_actions: list[str]          # What the agent can do next
+feedback: str | None                  # Human-readable feedback
+dsl_format: str                       # DSL syntax instructions (always present)
+```
+
+### State
+```
+episode_id: str
+step_count: int
+task_name: str
+current_rules: list | None
+accuracy_history: list[float]
+questions_asked: int
+questions_log: list[str]
+done: bool
+total_reward: float
+```
+
+### StepResult
+```
+observation: Observation
+reward: float    # 0.0-1.0 per step
+done: bool
+info: dict       # Contains reward_breakdown, episode_score, errors, etc.
+```
+
+---
+
+## 4. Episode Lifecycle (environment.py)
+
+### reset(task_name)
+1. Load task config from registry (defaults to `"data_access"`)
+2. Generate scenarios via `generate_scenarios(task_name)` with `seed=42`
+3. Initialize state: `step_count=0`, `accuracy=0`, `done=false`
+4. Return observation with policy text, DSL format, available decisions/variables
+
+### step(action)
+1. Guard: if `state is None` or `done == True` → error result
+2. Increment `step_count`
+3. Dispatch by `action_type`:
+   - `"ask_clarification"` → `_handle_clarification()`
+   - `"propose_rules"` → `_handle_propose()`
+   - `"refine_rules"` → `_handle_refine()`
+
+### Termination Conditions
+Episode ends (`done=True`) when **either**:
+- `accuracy >= 0.9` (success)
+- `step_count >= max_steps` (budget exhausted)
+
+### Clarification Handling
+1. Parse content as JSON to extract `question`, or use raw content as the question
+2. Call `answer_clarification(task_name, question)` → deterministic oracle answer
+3. Usefulness check: `is_useful = "I can provide information" not in answer`
+4. Compute reward (accuracy stays unchanged, clarification component applies)
+5. `refine_rules` is only available after at least one `propose_rules`
+
+### Rule Proposal/Refinement Handling
+1. Parse JSON content via `parse_rules()` → validates DSL structure
+2. If invalid: penalty reward, feedback with parse errors
+3. If valid: grade rules against stored scenarios → accuracy
+4. Compute reward using accuracy delta
+5. Feedback includes: accuracy, improvement direction, passed/total, sample failure
+6. If `accuracy >= 0.9`: feedback says "Target accuracy reached! Episode complete."
+7. On episode end: compute `episode_score` and include in info
+
+---
+
+## 5. The Three Tasks (policies.py)
+
+### Task 1: `data_access` (Easy)
+
+| Property | Value |
+|---|---|
+| Difficulty | easy |
+| Max Steps | 5 |
+| Scenario Count | 30 |
+| Variables | `time` (0-23), `data_type` (sensitive, public, internal) |
+| Valid Decisions | ALLOW, DENY |
+| Hidden Params | `work_start=9`, `work_end=18` |
+
+**Policy Text** (what the agent sees):
+> Employees must not access sensitive data after working hours. Working hours are from 9 AM to 6 PM (9:00 to 18:00). Public data can be accessed at any time. Internal data follows the same rules as sensitive data.
+
+---
+
+### Task 2: `resource_access` (Medium)
+
+| Property | Value |
+|---|---|
+| Difficulty | medium |
+| Max Steps | 7 |
+| Scenario Count | 50 |
+| Variables | `role` (junior, senior, contractor), `time` (0-23), `document_type` (public, internal, confidential) |
+| Valid Decisions | ALLOW, DENY |
+| Hidden Params | `business_start=8`, `business_end=17` |
+
+**Policy Text**:
+> Junior employees cannot access confidential documents outside business hours. Senior employees have unrestricted access to all document types. Contractors can only access public documents, regardless of time. During business hours, junior employees may access public and internal documents.
+
+**Intentional Ambiguity**: The policy says juniors "cannot access confidential documents outside business hours" — implying they CAN during business hours. But the ground truth DENIES confidential for juniors at ALL times. This is a deliberate trap the agent must discover through testing.
+
+---
+
+### Task 3: `transaction_approval` (Hard)
+
+| Property | Value |
+|---|---|
+| Difficulty | hard |
+| Max Steps | 7 |
+| Scenario Count | 80 |
+| Variables | `amount` (100..50000, 12 values), `transfer_type` (domestic, international), `time` (0-23), `initiator_role` (employee, manager, system) |
+| Valid Decisions | APPROVE, REQUIRE_APPROVAL, COMPLIANCE_REVIEW, HOLD |
+| Hidden Params | `standard_limit=5000`, `high_value_threshold=10000`, `business_start=9`, `business_end=17` |
+
+**Policy Text**:
+> Transactions exceeding the standard limit require manager approval. International transfers always need compliance review regardless of amount. High-value domestic transactions during non-business hours are automatically held for review. Routine domestic transactions within limits are auto-approved. Manager-initiated transactions are exempt from the standard limit.
+
+---
+
+## 6. Ground Truth Logic (ground_truth.py)
+
+### Task 1: `_ground_truth_data_access`
+
+```python
+if data_type == "public":           → ALLOW
+if 9 <= time < 18:                  → ALLOW   # sensitive or internal
+else:                               → DENY
+```
+
+**Complete Decision Table**:
+
+| data_type | time | Decision | Why |
+|---|---|---|---|
+| public | any (0-23) | ALLOW | Public is always accessible |
+| sensitive | 0-8 | DENY | Before working hours |
+| sensitive | 9-17 | ALLOW | During working hours |
+| sensitive | 18-23 | DENY | After working hours (18 is OUTSIDE) |
+| internal | 0-8 | DENY | Same rules as sensitive |
+| internal | 9-17 | ALLOW | Same rules as sensitive |
+| internal | 18-23 | DENY | Same rules as sensitive |
+
+> [!IMPORTANT]
+> **Critical boundary**: `time=18` → DENY. The interval is half-open: `[9, 18)`. Hour 18 is the first after-hours hour. Hour 17 is the last working hour.
+
+---
+
+### Task 2: `_ground_truth_resource_access`
+
+```python
+if role == "senior":                                    → ALLOW
+if role == "contractor":
+    if doc_type == "public":                            → ALLOW
+    else:                                               → DENY
+# Junior employee:
+is_business_hours = (8 <= time < 17)
+if doc_type == "public":                                → ALLOW
+if is_business_hours and doc_type == "internal":        → ALLOW
+else:                                                   → DENY
+```
+
+**Complete Decision Table for Junior Employees**:
+
+| document_type | time | Decision | Why |
+|---|---|---|---|
+| public | any (0-23) | ALLOW | Public always allowed for all roles |
+| internal | 0-7 | DENY | Before business hours |
+| internal | 8-16 | ALLOW | During business hours |
+| internal | 17-23 | DENY | After business hours (17 is OUTSIDE) |
+| confidential | any (0-23) | **DENY** | **Always denied for juniors** |
+
+**Senior**: ALLOW for everything, always.
+**Contractor**: ALLOW only for `public`, DENY for `internal` and `confidential`, at all times.
+
+> [!IMPORTANT]
+> **Critical boundary**: `time=17` → outside business hours. Interval: `[8, 17)`. Hour 16 is the last business hour.
+>
+> **Critical trap**: `confidential` is ALWAYS denied for juniors, even during business hours. The policy text misleadingly implies otherwise.
+
+---
+
+### Task 3: `_ground_truth_transaction_approval`
+
+Rules evaluated in strict priority order (first match wins):
+
+```python
+# Rule 1: International → COMPLIANCE_REVIEW (always, regardless of everything)
+if transfer_type == "international":                    → COMPLIANCE_REVIEW
+
+# Rule 2: High-value domestic outside business hours → HOLD
+if amount >= 10000 and not (9 <= time < 17):            → HOLD
+
+# Rule 3: Above standard limit, not manager → REQUIRE_APPROVAL
+if amount > 5000 and initiator_role != "manager":       → REQUIRE_APPROVAL
+
+# Rule 4: Everything else → APPROVE
+else:                                                   → APPROVE
+```
+
+**Critical Edge Cases**:
+
+| amount | transfer_type | time | initiator_role | Decision | Why |
+|---|---|---|---|---|---|
+| 5000 | domestic | 12 | employee | **APPROVE** | At limit, not above (> 5000 fails) |
+| 5001 | domestic | 12 | employee | REQUIRE_APPROVAL | Above limit, not manager |
+| 5001 | domestic | 12 | manager | **APPROVE** | Manager exempt from limit |
+| 10000 | domestic | 20 | employee | **HOLD** | High-value + non-business hours |
+| 10000 | domestic | 12 | employee | REQUIRE_APPROVAL | High-value but business hours (Rule 2 skipped, Rule 3 matches) |
+| 10000 | domestic | 17 | employee | **HOLD** | 17 is non-business hours |
+| 10000 | domestic | 20 | **manager** | **HOLD** | Managers NOT exempt from HOLD rule |
+| 100 | international | 12 | employee | COMPLIANCE_REVIEW | International always |
+| 50000 | international | 3 | manager | COMPLIANCE_REVIEW | International trumps everything |
+| 9999 | domestic | 20 | employee | REQUIRE_APPROVAL | NOT high-value (< 10000), but above limit |
+| 100 | domestic | 3 | employee | APPROVE | Within limit |
+| 100 | domestic | 3 | system | APPROVE | System = employee |
+
+> [!IMPORTANT]
+> **Standard limit comparison**: `amount > 5000` (strict greater than). $5,000 exactly = APPROVE.
+>
+> **High-value comparison**: `amount >= 10000` (greater than or equal). $10,000 exactly = high-value.
+>
+> **Manager exemption scope**: Only exempts from Rule 3 (standard limit). Managers are still subject to Rule 1 (international) and Rule 2 (high-value HOLD).
+>
+> **Business hours**: `[9, 17)`. Hour 17 is non-business.
+
+---
+
+## 7. Clarification Oracle (ground_truth.py)
+
+### Matching Algorithm
+
+```
+Input:  question string (free text from agent)
+Output: best matching answer from task's clarification_map
+
+Algorithm:
+1. Lowercase the question
+2. For each keyword in clarification_map:
+   a. Split keyword into parts by spaces
+   b. Check if ALL parts appear as substrings in the question
+   c. Score = (number_of_parts, total_keyword_length)
+   d. Highest score wins
+3. If no match: return generic fallback (contains "I can provide information")
+```
+
+**Key property**: "junior confidential" matches when BOTH "junior" AND "confidential" appear anywhere in the question (order-independent). This 2-part keyword beats any 1-part keyword like "junior" alone.
+
+### Usefulness Detection
+
+In `environment.py`, line 203:
+```python
+is_useful = "I can provide information" not in answer
+```
+Any answer that matches a keyword entry is "useful". Only the generic fallback is "not useful".
+
+### 3-Tier Progressive Revelation Design
+
+Each task's `clarification_map` has three levels:
+
+| Tier | Keyword Type | Answer Quality | Training Purpose |
+|---|---|---|---|
+| Level 1 | Single short words | Partial truths, technically correct but incomplete/misleading | Agent builds initial (wrong) rules |
+| Level 2 | Common phrases | More detail, boundary still ambiguous | Agent narrows down the problem |
+| Level 3 | Compound/multi-word | Precise, ground-truth-aligned, corrects Level 1 | Agent fixes rules after failures |
+
+**Example — resource_access contradiction**:
+- Agent asks "What can junior employees access?" → matches `"junior"` (Level 1) → *"...but not confidential documents outside business hours"* (implies CAN during hours)
+- Agent proposes rules allowing junior+confidential during hours → **fails**
+- Agent asks "Can junior employees access confidential documents?" → matches `"junior confidential"` (Level 3, 2 parts > 1 part) → *"CANNOT access confidential at ANY time"*
+- Agent refines rules → **passes**
+
+### Clarification Map Entry Counts
+
+| Task | Level 1 | Level 2 | Level 3 | Total |
+|---|---|---|---|---|
+| data_access | 5 | 3 | 6 | 14 |
+| resource_access | 7 | 3 | 8 | 18 |
+| transaction_approval | 9 | 7 | 10 | 26 |
+
+---
+
+## 8. DSL Engine (dsl_engine.py)
+
+### DSL Format
+
+```json
+{
+    "rules": [
+        {
+            "if": [
+                {"field": "<name>", "op": "<operator>", "value": <value>}
+            ],
+            "then": "<DECISION>"
+        }
+    ],
+    "default": "<DEFAULT_DECISION>"
+}
+```
+
+### Supported Operators
+`>`, `<`, `>=`, `<=`, `==`, `!=`
+
+### Validation (`validate_rules`)
+Checks:
+- Root is a dict
+- Has `"rules"` key (must be list)
+- Has `"default"` key (must be string)
+- Each rule has `"if"` (list) and `"then"` (string)
+- Each condition has `"field"` (string), `"op"` (valid operator), `"value"`
+
+Returns `(is_valid: bool, errors: list[str])`.
+
+### Execution (`execute_rules`)
+1. Iterate rules top-to-bottom
+2. For each rule, evaluate ALL conditions (AND logic)
+3. First rule where all conditions match → return its `"then"` decision
+4. If no rules match → return `"default"`
+
+### Type Coercion
+If scenario has `time=9` (int) and rule has `"value": "9"` (str), coerces the string to int. Works both directions. If coercion fails, condition evaluates to `False`.
+
+### Parsing (`parse_rules`)
+1. `json.loads()` the content string
+2. `validate_rules()` on the parsed dict
+3. Returns `(rules_data, [])` on success or `(None, errors)` on failure
+
+---
+
+## 9. Scenario Generator (scenario_generator.py)
+
+### Strategy Allocation
+
+| Strategy | Share | Purpose |
+|---|---|---|
+| Boundary | ~20% | Edge values near hidden param thresholds |
+| Pairwise | ~30% | Systematic variable combinations |
+| Adversarial | ~20% | Hand-crafted traps for common mistakes |
+| Random | remainder | Uniform sampling from variable space |
+
+All seeded with `seed=42` for reproducibility. Scenarios are deduplicated by field values.
+
+### Boundary Strategy
+Extracts numeric hidden params, generates scenarios at `param ± 1` and at variable min/max.
+
+### Pairwise Strategy
+For each pair of variables, samples up to 4 representative values (min, max, middle, random), generates cross-product combinations.
+
+### Adversarial Strategy
+**Hand-crafted per task** — these are the exact scenarios:
+
+#### data_access adversarial:
+| time | data_type | Expected | Tests |
+|---|---|---|---|
+| 9 | sensitive | ALLOW | Start boundary |
+| 18 | sensitive | DENY | End boundary (exclusive) |
+| 8 | sensitive | DENY | Just before start |
+| 17 | sensitive | ALLOW | Just before end |
+| 0 | public | ALLOW | Public at midnight |
+| 23 | internal | DENY | Internal late night |
+| 12 | internal | ALLOW | Internal during hours |
+
+#### resource_access adversarial:
+| role | time | document_type | Expected | Tests |
+|---|---|---|---|---|
+| junior | 8 | confidential | DENY | Confidential at business start |
+| junior | 7 | internal | DENY | Internal before hours |
+| junior | 17 | internal | DENY | Internal at boundary (17=outside) |
+| junior | 16 | internal | ALLOW | Internal just before boundary |
+| contractor | 12 | internal | DENY | Contractor restricted |
+| senior | 2 | confidential | ALLOW | Senior unrestricted |
+| junior | 12 | public | ALLOW | Junior public during hours |
+| contractor | 12 | public | ALLOW | Contractor public |
+
+#### transaction_approval adversarial:
+| amount | transfer | time | role | Expected | Tests |
+|---|---|---|---|---|---|
+| 5000 | domestic | 12 | employee | APPROVE | At limit (not above) |
+| 5001 | domestic | 12 | employee | REQ_APPROVAL | Just above limit |
+| 5001 | domestic | 12 | manager | APPROVE | Manager exempt |
+| 10000 | domestic | 20 | employee | HOLD | High-value non-business |
+| 10000 | domestic | 12 | employee | REQ_APPROVAL | High-value business hours |
+| 100 | international | 12 | employee | COMPLIANCE | International small |
+| 50000 | international | 3 | manager | COMPLIANCE | International manager |
+| 9999 | domestic | 20 | employee | REQ_APPROVAL | Below high-value threshold |
+| 10000 | domestic | 9 | employee | REQ_APPROVAL | High-value at business start |
+| 10000 | domestic | 17 | employee | HOLD | 17=non-business |
+
+---
+
+## 10. Grading (graders.py)
+
+### `grade_task(task_name, rules_data, scenarios)`
+1. Validate rules → if invalid, return `score=0.0`
+2. For each scenario: execute agent's rules, compare to `expected_decision`
+3. Comparison: `actual.upper() == expected.upper()` (case-insensitive)
+4. `score = passed / total`
+5. Returns up to 5 `sample_failures` with scenario details, expected, got
+
+### `quick_grade(task_name, rules_data, scenarios)`
+Same logic, returns only the float score. Used during step processing.
+
+---
+
+## 11. Reward System (rewards.py)
+
+### Per-Step Reward: `compute_reward()`
+
+4 components, clamped to `[0.0, 1.0]`:
+
+| Component | Weight | Formula |
+|---|---|---|
+| **Accuracy** | 0.50 | `current_accuracy × 0.50` |
+| **Improvement** | 0.20 | `min(delta × 2.0, 1.0) × 0.20` if delta > 0; `max(delta × 1.5, -0.5) × 0.20` if delta < 0; `0` if unchanged |
+| **Efficiency** | 0.15 | `max(-0.02 × step_number [+ 0.05 × steps_saved if acc≥0.9], -0.15) × 0.15` |
+| **Clarification** | 0.15 | See below |
+
+**Clarification component details**:
+- `ask_clarification` + useful + questions ≤ 3: `+0.3 × 0.15 = +0.045`
+- `ask_clarification` + useful + questions > 3: `+0.1 × 0.15 = +0.015` (diminishing)
+- `ask_clarification` + not useful: `-0.05 × 0.15 = -0.0075`
+- `propose_rules/refine_rules` + invalid DSL: `-0.1 × 0.15 = -0.015`
+- `propose_rules/refine_rules` + valid DSL: `0`
+
+### Episode Score: `compute_episode_score()`
+
+Used for final grading, `[0.0, 1.0]`:
+
+```
+score = final_accuracy × 0.80
+     + max(0, 1 - steps/max_steps) × 0.10
+     + question_bonus × 0.10
+
+question_bonus = 1.0 if questions ≤ 2
+               = 0.5 if questions ≤ 4
+               = 0.0 if questions > 4
+```
+
+---
+
+## 12. Inference Agent (inference.py)
+
+### Configuration
+- Model: `Qwen/Qwen2.5-72B-Instruct` (via `HF_TOKEN`)
+- API: `https://router.huggingface.co/v1` (OpenAI-compatible)
+- Temperature: 0.3, Max tokens: 1024
+- Env URL: `http://localhost:7860` (configurable via `ENV_BASE_URL`)
+
+### Agent Loop
+```
+for each task in [data_access, resource_access, transaction_approval]:
+    result = env.reset(task)
+    for step in 1..max_steps:
+        if result.done: break
+        action_type, content = get_agent_action(llm, observation, step, history)
+        result = env.step(action)
+        history.append(summary)
+```
+
+### Prompt Design
+- **System prompt**: Describes available actions, DSL format, strategy guidelines
+- **User prompt**: Built per-step with policy text, feedback, clarification answers, test results, sample failures, DSL format, action history (last 3)
+- LLM response parsed as JSON: `{"action_type": "...", "content": "..."}`
+- Handles markdown code blocks (`\`\`\`json ... \`\`\``)
+- Fallback: if unparseable, tries extracting `"rules"`, otherwise submits empty rules
+
+### Output Format
+```
+[START] task=<name> env=policy_to_logic model=<model>
+[STEP]  step=<n> action=<summary> reward=<float> done=<bool> error=<msg|null>
+[END]   success=<bool> steps=<n> score=<float> rewards=<r1,r2,...>
+```
+
+---
+
+## 13. Client Library (client.py)
+
+HTTP client using `requests.Session()`:
+- `reset(task_name)` → POST `/reset` → `PolicyToLogicStepResult`
+- `step(action)` → POST `/step` → `PolicyToLogicStepResult`
+- `state()` → GET `/state` → `PolicyToLogicState`
+- `health()` → GET `/health` → dict
+- `list_tasks()` → GET `/tasks` → dict
+- Context manager support (`with PolicyToLogicEnv() as env:`)
+
+---
+
+## 14. Deployment
+
+### Dockerfile
+```dockerfile
+FROM python:3.11-slim
+WORKDIR /app
+COPY policy_to_logic_env/server/requirements.txt → pip install
+COPY policy_to_logic_env/, main.py, inference.py
+EXPOSE 7860
+HEALTHCHECK: curl -f http://localhost:7860/health
+CMD: python -m uvicorn policy_to_logic_env.server.app:app --host 0.0.0.0 --port 7860
+```
+
+### HF Spaces Config (README.md)
+```yaml
+sdk: docker
+app_port: 7860
+```
+
+Live at: `https://godreign-policy2logic.hf.space`
+
+---
+
+## 15. Known Design Decisions & Limitations
+
+1. **Single-session**: One global environment instance. Concurrent clients will interfere. Suitable for sequential benchmarking, not parallel RL training.
+
+2. **Deterministic scenarios**: `seed=42` always produces the same scenarios. Agent is graded on the same set every episode. Prevents overfitting variance but could lead to memorization.
+
+3. **Stateful server**: The environment holds state in memory. Server restart loses episode state. No persistence layer.
+
+4. **Clarification is keyword-based**: The oracle is not an LLM — it's a deterministic keyword matcher. Agent questions that don't contain any keyword get the generic fallback (penalized as "not useful").
+
+5. **Progressive revelation by design**: Level 1 clarification answers are intentionally misleading partial truths. This is NOT a bug — it's the core RL training signal. Agents that trust Level 1 answers will fail and must learn to ask better (Level 3) questions.
+
+6. **No `refine_rules` before `propose_rules`**: The environment returns a feedback message if the agent tries to refine before proposing. Not an error, just 0 reward + feedback.
+
+7. **Case-insensitive grading**: `actual.upper() == expected.upper()`. Agent can output "allow" or "Allow" or "ALLOW".
+
+8. **DSL type coercion**: Integer-string mismatches are auto-coerced. `"9"` and `9` compare equally.

IMPLEMENTATION_HANDOFF.md

@@ -0,0 +1,1045 @@
+# Policy-to-Logic RL Environment — Final Implementation Handoff
+## 7-Hour Sprint Document (AI-Executable Instructions)
+
+> **Context**: This document is written for an AI-powered IDE to execute. Every instruction is explicit, ordered, and complete. Do not skip steps. Do not reorder steps. Validate each step before moving to the next.
+
+---
+
+## CRITICAL: Validation Checklist (Must Pass All 5 Before Submission)
+
+The hackathon validator checks these automatically. If any fails, the submission is rejected before a human sees it.
+
+| # | Requirement | How to Verify |
+|---|---|---|
+| 1 | Public HF Space at submitted URL | Open in logged-out browser — must load, no 404 |
+| 2 | Valid OpenEnv structure (base class + reset/step/state + openenv.yaml) | Already implemented — verify yaml is parseable |
+| 3 | Training evidence as committed `.png` / `.jpg` files in repo | NOT Wandb links — actual image files in repo |
+| 4 | Runnable training script (Colab notebook preferred) | Must re-execute end-to-end |
+| 5 | README links all deliverables with plots embedded inline | Validator must reach every link from README |
+
+**These 5 items drive the entire 7-hour plan. Every task below serves one or more of them.**
+
+---
+
+## Hour-by-Hour Execution Plan
+
+---
+
+## HOUR 1-2: Build the Reward-Guided Trajectory Training Loop
+
+### What This Is
+
+This is NOT fine-tuning. This is a **reward-guided few-shot accumulation loop** — a legitimate optimization strategy where:
+- The agent runs episodes against the environment
+- Trajectories (full interaction sequences) are stored with their rewards
+- High-reward trajectories become few-shot examples for subsequent episodes
+- Agent performance measurably improves across episodes using the reward signal
+
+**This is your "training loop" for the submission.** It is honest, demonstrable, and buildable.
+
+---
+
+### File to Create: `training/trajectory_optimizer.py`
+
+Create this file in the repo root under a new `training/` directory.
+
+```python
+"""
+Reward-Guided Trajectory Optimization Loop
+==========================================
+Optimizes agent behavior across episodes by accumulating high-reward
+trajectories as few-shot examples. Uses environment reward signal to
+drive improvement — no weight updates required.
+
+This implements a policy improvement loop where:
+  - reward_signal → trajectory_selection → context_construction → improved_policy
+"""
+
+import json
+import os
+import time
+import requests
+from dataclasses import dataclass, field
+from typing import Optional
+from openai import OpenAI
+
+# ── Configuration ────────────────────────────────────────────────────────────
+
+ENV_BASE_URL = os.getenv("ENV_BASE_URL", "http://localhost:7860")
+HF_TOKEN = os.getenv("HF_TOKEN", "")
+MODEL = "Qwen/Qwen2.5-72B-Instruct"
+TEMPERATURE = 0.3
+MAX_TOKENS = 1024
+
+# Training hyperparameters
+NUM_EPISODES_PER_TASK = 8        # Episodes to run per task
+TOP_K_TRAJECTORIES = 3           # Max few-shot examples to keep
+MIN_REWARD_THRESHOLD = 0.3       # Minimum reward to store trajectory
+TASKS = ["data_access", "resource_access", "transaction_approval"]
+
+# ── Data Structures ──────────────────────────────────────────────────────────
+
+@dataclass
+class Step:
+    step_number: int
+    action_type: str
+    action_content: str
+    reward: float
+    accuracy: float
+    feedback: str
+    clarification_response: Optional[str] = None
+
+@dataclass
+class Trajectory:
+    task_name: str
+    episode_id: int
+    steps: list[Step] = field(default_factory=list)
+    total_reward: float = 0.0
+    final_accuracy: float = 0.0
+    success: bool = False
+
+    def to_few_shot_string(self) -> str:
+        """Convert trajectory to a few-shot example string for prompting."""
+        lines = [
+            f"=== Example Episode (reward={self.total_reward:.2f}, accuracy={self.final_accuracy:.2f}) ===",
+        ]
+        for s in self.steps:
+            lines.append(f"Step {s.step_number}: action={s.action_type}")
+            lines.append(f"  Content: {s.action_content[:200]}")
+            lines.append(f"  Result: accuracy={s.accuracy:.2f}, reward={s.reward:.2f}")
+            if s.feedback:
+                lines.append(f"  Feedback: {s.feedback[:150]}")
+        return "\n".join(lines)
+
+# ── Environment Client ────────────────────────────────────────────────────────
+
+class EnvClient:
+    def __init__(self, base_url: str):
+        self.base_url = base_url.rstrip("/")
+        self.session = requests.Session()
+
+    def reset(self, task_name: str) -> dict:
+        r = self.session.post(f"{self.base_url}/reset", json={"task_name": task_name})
+        r.raise_for_status()
+        return r.json()
+
+    def step(self, action_type: str, content: str) -> dict:
+        r = self.session.post(f"{self.base_url}/step", json={
+            "action_type": action_type,
+            "content": content
+        })
+        r.raise_for_status()
+        return r.json()
+
+    def health(self) -> bool:
+        try:
+            r = self.session.get(f"{self.base_url}/health", timeout=5)
+            return r.status_code == 200
+        except Exception:
+            return False
+
+# ── LLM Agent ────────────────────────────────────────────────────────────────
+
+class Agent:
+    def __init__(self, hf_token: str):
+        self.client = OpenAI(
+            base_url="https://router.huggingface.co/v1",
+            api_key=hf_token
+        )
+
+    def get_action(
+        self,
+        observation: dict,
+        step_number: int,
+        episode_history: list[str],
+        few_shot_examples: list[Trajectory]
+    ) -> tuple[str, str]:
+        """
+        Returns (action_type, content_json_string).
+        action_type: one of ask_clarification | propose_rules | refine_rules
+        content: JSON string appropriate for that action
+        """
+        system_prompt = self._build_system_prompt(few_shot_examples)
+        user_prompt = self._build_user_prompt(observation, step_number, episode_history)
+
+        try:
+            response = self.client.chat.completions.create(
+                model=MODEL,
+                messages=[
+                    {"role": "system", "content": system_prompt},
+                    {"role": "user", "content": user_prompt}
+                ],
+                temperature=TEMPERATURE,
+                max_tokens=MAX_TOKENS
+            )
+            raw = response.choices[0].message.content.strip()
+            return self._parse_response(raw, observation)
+        except Exception as e:
+            print(f"    [LLM ERROR] {e}")
+            return "propose_rules", json.dumps({"rules": [], "default": "DENY"})
+
+    def _build_system_prompt(self, few_shot_examples: list[Trajectory]) -> str:
+        base = """You are a policy-to-logic agent. Your job is to convert natural language policies into executable rules.
+
+AVAILABLE ACTIONS:
+1. ask_clarification: {"type": "clarification", "question": "your question"}
+2. propose_rules: {"rules": [...], "default": "DECISION"}
+3. refine_rules: {"rules": [...], "default": "DECISION"}
+
+DSL FORMAT for rules:
+{
+  "rules": [
+    {
+      "if": [
+        {"field": "FIELD_NAME", "op": "OPERATOR", "value": VALUE}
+      ],
+      "then": "DECISION"
+    }
+  ],
+  "default": "FALLBACK_DECISION"
+}
+
+Operators: >, <, >=, <=, ==, !=
+Rules execute top-to-bottom. First match wins. Default applies if no rule matches.
+
+STRATEGY:
+- Step 1: Ask 1-2 targeted clarification questions about ambiguous terms
+- Step 2: Propose initial rules based on policy + clarifications  
+- Step 3+: Refine rules based on failure feedback
+
+OUTPUT FORMAT: Respond ONLY with valid JSON. No markdown. No explanation.
+{"action_type": "propose_rules", "content": "{...escaped json string...}"}
+"""
+        if few_shot_examples:
+            base += "\n\nLEARNED FROM PREVIOUS EPISODES (high-reward strategies):\n"
+            for traj in few_shot_examples[-TOP_K_TRAJECTORIES:]:
+                base += "\n" + traj.to_few_shot_string() + "\n"
+        return base
+
+    def _build_user_prompt(self, obs: dict, step: int, history: list[str]) -> str:
+        lines = [
+            f"TASK: {obs.get('task_name', 'unknown')}",
+            f"STEP: {step} of {obs.get('max_steps', 7)}",
+            f"\nPOLICY:\n{obs.get('policy_text', '')}",
+        ]
+        if obs.get("clarification_response"):
+            lines.append(f"\nLAST CLARIFICATION ANSWER:\n{obs['clarification_response']}")
+        if obs.get("test_results"):
+            tr = obs["test_results"]
+            lines.append(f"\nTEST RESULTS: {tr.get('passed', 0)}/{tr.get('total', 0)} passed (accuracy={obs.get('current_accuracy', 0):.2f})")
+            if tr.get("sample_failures"):
+                lines.append("SAMPLE FAILURES:")
+                for f in tr["sample_failures"][:3]:
+                    lines.append(f"  - {f}")
+        if obs.get("feedback"):
+            lines.append(f"\nFEEDBACK: {obs['feedback']}")
+        if history:
+            lines.append(f"\nACTION HISTORY (last 3):\n" + "\n".join(history[-3:]))
+        lines.append(f"\nAVAILABLE ACTIONS: {obs.get('available_actions', [])}")
+        lines.append("\nRespond with JSON only: {\"action_type\": \"...\", \"content\": \"...\"}")
+        return "\n".join(lines)
+
+    def _parse_response(self, raw: str, obs: dict) -> tuple[str, str]:
+        # Strip markdown code fences if present
+        if "```" in raw:
+            raw = raw.split("```")[1]
+            if raw.startswith("json"):
+                raw = raw[4:]
+        raw = raw.strip()
+
+        try:
+            parsed = json.loads(raw)
+            action_type = parsed.get("action_type", "propose_rules")
+            content = parsed.get("content", "{}")
+
+            # Validate action_type
+            valid_actions = obs.get("available_actions", ["propose_rules", "ask_clarification"])
+            if action_type not in valid_actions:
+                action_type = "propose_rules" if "propose_rules" in valid_actions else valid_actions[0]
+
+            # Ensure content is a string
+            if isinstance(content, dict):
+                content = json.dumps(content)
+            return action_type, content
+        except Exception:
+            return "propose_rules", json.dumps({"rules": [], "default": "DENY"})
+
+# ── Trajectory Bank ───────────────────────────────────────────────────────────
+
+class TrajectoryBank:
+    """Stores and retrieves high-reward trajectories per task."""
+
+    def __init__(self):
+        self.bank: dict[str, list[Trajectory]] = {task: [] for task in TASKS}
+
+    def store(self, trajectory: Trajectory):
+        if trajectory.total_reward >= MIN_REWARD_THRESHOLD:
+            self.bank[trajectory.task_name].append(trajectory)
+            # Keep only top-K by reward
+            self.bank[trajectory.task_name].sort(key=lambda t: t.total_reward, reverse=True)
+            self.bank[trajectory.task_name] = self.bank[trajectory.task_name][:TOP_K_TRAJECTORIES]
+
+    def get_examples(self, task_name: str) -> list[Trajectory]:
+        return self.bank.get(task_name, [])
+
+    def summary(self) -> dict:
+        return {
+            task: {
+                "stored": len(trajs),
+                "best_reward": max((t.total_reward for t in trajs), default=0),
+                "best_accuracy": max((t.final_accuracy for t in trajs), default=0)
+            }
+            for task, trajs in self.bank.items()
+        }
+
+# ── Training Loop ─────────────────────────────────────────────────────────────
+
+class TrainingLoop:
+    def __init__(self, env_url: str, hf_token: str):
+        self.env = EnvClient(env_url)
+        self.agent = Agent(hf_token)
+        self.bank = TrajectoryBank()
+        self.metrics = []  # List of {episode, task, reward, accuracy, success}
+
+    def run_episode(self, task_name: str, episode_id: int) -> Trajectory:
+        """Run a single episode and return the trajectory."""
+        few_shots = self.bank.get_examples(task_name)
+        trajectory = Trajectory(task_name=task_name, episode_id=episode_id)
+
+        # Reset environment
+        result = self.env.reset(task_name)
+        obs = result.get("observation", {})
+        done = result.get("done", False)
+        history = []
+
+        print(f"  [Episode {episode_id}] task={task_name} few_shots={len(few_shots)}")
+
+        step_num = 0
+        while not done and step_num < obs.get("max_steps", 7):
+            step_num += 1
+
+            # Get action from agent
+            action_type, content = self.agent.get_action(
+                observation=obs,
+                step_number=step_num,
+                episode_history=history,
+                few_shot_examples=few_shots
+            )
+
+            # Execute action
+            result = self.env.step(action_type, content)
+            reward = result.get("reward", 0.0)
+            done = result.get("done", False)
+            obs = result.get("observation", {})
+            info = result.get("info", {})
+
+            # Record step
+            step = Step(
+                step_number=step_num,
+                action_type=action_type,
+                action_content=content[:300],
+                reward=reward,
+                accuracy=obs.get("current_accuracy", 0.0),
+                feedback=obs.get("feedback", "") or "",
+                clarification_response=obs.get("clarification_response")
+            )
+            trajectory.steps.append(step)
+            trajectory.total_reward += reward
+
+            # Update history
+            history.append(f"Step {step_num}: {action_type} → reward={reward:.2f} acc={step.accuracy:.2f}")
+
+            print(f"    step={step_num} action={action_type} reward={reward:.3f} acc={step.accuracy:.2f}")
+
+            if done:
+                episode_score = info.get("episode_score", obs.get("current_accuracy", 0.0))
+                trajectory.final_accuracy = episode_score
+                trajectory.success = obs.get("current_accuracy", 0.0) >= 0.9
+                break
+
+        if not trajectory.steps:
+            trajectory.final_accuracy = 0.0
+
+        return trajectory
+
+    def run(self):
+        """Run full training loop across all tasks."""
+        print("=" * 60)
+        print("REWARD-GUIDED TRAJECTORY OPTIMIZATION")
+        print(f"Tasks: {TASKS}")
+        print(f"Episodes per task: {NUM_EPISODES_PER_TASK}")
+        print(f"Top-K trajectories: {TOP_K_TRAJECTORIES}")
+        print("=" * 60)
+
+        # Health check
+        if not self.env.health():
+            raise RuntimeError(f"Environment not reachable at {ENV_BASE_URL}")
+        print(f"Environment: OK ({ENV_BASE_URL})\n")
+
+        global_episode = 0
+
+        for task in TASKS:
+            print(f"\n{'─'*40}")
+            print(f"TASK: {task}")
+            print(f"{'─'*40}")
+
+            task_rewards = []
+            task_accuracies = []
+
+            for ep in range(1, NUM_EPISODES_PER_TASK + 1):
+                global_episode += 1
+                trajectory = self.run_episode(task, ep)
+
+                # Store in bank
+                self.bank.store(trajectory)
+
+                # Record metrics
+                self.metrics.append({
+                    "global_episode": global_episode,
+                    "task": task,
+                    "episode_in_task": ep,
+                    "total_reward": trajectory.total_reward,
+                    "final_accuracy": trajectory.final_accuracy,
+                    "success": trajectory.success,
+                    "num_steps": len(trajectory.steps),
+                    "few_shots_used": len(self.bank.get_examples(task)) - (1 if trajectory.total_reward >= MIN_REWARD_THRESHOLD else 0)
+                })
+
+                task_rewards.append(trajectory.total_reward)
+                task_accuracies.append(trajectory.final_accuracy)
+
+                print(f"  → Episode {ep} complete: reward={trajectory.total_reward:.3f} accuracy={trajectory.final_accuracy:.2f} success={trajectory.success}")
+                time.sleep(0.5)  # Rate limiting
+
+            print(f"\n  Task summary:")
+            print(f"    First episode reward: {task_rewards[0]:.3f}")
+            print(f"    Last episode reward:  {task_rewards[-1]:.3f}")
+            print(f"    Improvement: {task_rewards[-1] - task_rewards[0]:+.3f}")
+
+        print("\n" + "=" * 60)
+        print("TRAINING COMPLETE")
+        print(f"Bank summary: {self.bank.summary()}")
+        print("=" * 60)
+
+        return self.metrics
+
+# ── Plot Generation ───────────────────────────────────────────────────────────
+
+def save_plots(metrics: list[dict]):
+    """
+    Save reward curve and accuracy curve as PNG files.
+    These are REQUIRED for hackathon submission — must be committed to repo.
+    """
+    try:
+        import matplotlib
+        matplotlib.use("Agg")  # Non-interactive backend
+        import matplotlib.pyplot as plt
+        import numpy as np
+    except ImportError:
+        print("matplotlib not installed. Run: pip install matplotlib")
+        return
+
+    os.makedirs("training/plots", exist_ok=True)
+
+    episodes = [m["global_episode"] for m in metrics]
+    rewards = [m["total_reward"] for m in metrics]
+    accuracies = [m["final_accuracy"] for m in metrics]
+    tasks = [m["task"] for m in metrics]
+
+    colors = {
+        "data_access": "#2196F3",
+        "resource_access": "#FF9800",
+        "transaction_approval": "#4CAF50"
+    }
+
+    # ── Plot 1: Reward Curve ──────────────────────────────────────────────────
+    fig, ax = plt.subplots(figsize=(10, 5))
+
+    for task in TASKS:
+        task_eps = [m["global_episode"] for m in metrics if m["task"] == task]
+        task_rews = [m["total_reward"] for m in metrics if m["task"] == task]
+        ax.plot(task_eps, task_rews, marker="o", label=task,
+                color=colors.get(task, "gray"), linewidth=2, markersize=5)
+
+    # Trend line
+    z = np.polyfit(episodes, rewards, 1)
+    p = np.poly1d(z)
+    ax.plot(episodes, p(episodes), "--", color="red", alpha=0.5, linewidth=1.5, label="overall trend")
+
+    ax.set_xlabel("Episode")
+    ax.set_ylabel("Total Reward")
+    ax.set_title("Reward Curve — Reward-Guided Trajectory Optimization")
+    ax.legend()
+    ax.grid(True, alpha=0.3)
+    ax.set_ylim(bottom=0)
+
+    plt.tight_layout()
+    plt.savefig("training/plots/reward_curve.png", dpi=150, bbox_inches="tight")
+    plt.close()
+    print("Saved: training/plots/reward_curve.png")
+
+    # ── Plot 2: Accuracy Curve ────────────────────────────────────────────────
+    fig, ax = plt.subplots(figsize=(10, 5))
+
+    for task in TASKS:
+        task_eps = [m["global_episode"] for m in metrics if m["task"] == task]
+        task_accs = [m["final_accuracy"] for m in metrics if m["task"] == task]
+        ax.plot(task_eps, task_accs, marker="s", label=task,
+                color=colors.get(task, "gray"), linewidth=2, markersize=5)
+
+    ax.axhline(y=0.9, color="red", linestyle="--", alpha=0.7, label="success threshold (0.9)")
+
+    ax.set_xlabel("Episode")
+    ax.set_ylabel("Final Accuracy")
+    ax.set_title("Accuracy Curve — Policy-to-Logic Agent")
+    ax.legend()
+    ax.grid(True, alpha=0.3)
+    ax.set_ylim(0, 1.05)
+
+    plt.tight_layout()
+    plt.savefig("training/plots/accuracy_curve.png", dpi=150, bbox_inches="tight")
+    plt.close()
+    print("Saved: training/plots/accuracy_curve.png")
+
+    # ── Plot 3: Per-Task Improvement Bar Chart ────────────────────────────────
+    fig, ax = plt.subplots(figsize=(8, 5))
+
+    task_names = []
+    improvements = []
+
+    for task in TASKS:
+        task_accs = [m["final_accuracy"] for m in metrics if m["task"] == task]
+        if len(task_accs) >= 2:
+            first = task_accs[0]
+            last = task_accs[-1]
+            task_names.append(task.replace("_", "\n"))
+            improvements.append(last - first)
+
+    bars = ax.bar(task_names, improvements,
+                  color=["#2196F3", "#FF9800", "#4CAF50"][:len(task_names)],
+                  edgecolor="white", linewidth=1.5)
+
+    ax.axhline(y=0, color="black", linewidth=0.8)
+    ax.set_ylabel("Accuracy Improvement (last - first episode)")
+    ax.set_title("Per-Task Improvement from Trajectory Accumulation")
+    ax.grid(True, axis="y", alpha=0.3)
+
+    for bar, val in zip(bars, improvements):
+        ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.01,
+                f"{val:+.2f}", ha="center", va="bottom", fontweight="bold")
+
+    plt.tight_layout()
+    plt.savefig("training/plots/improvement_chart.png", dpi=150, bbox_inches="tight")
+    plt.close()
+    print("Saved: training/plots/improvement_chart.png")
+
+    # ── Save raw metrics as JSON ──────────────────────────────────────────────
+    with open("training/plots/metrics.json", "w") as f:
+        json.dump(metrics, f, indent=2)
+    print("Saved: training/plots/metrics.json")
+
+# ── Entry Point ───────────────────────────────────────────────────────────────
+
+if __name__ == "__main__":
+    hf_token = os.getenv("HF_TOKEN", "")
+    if not hf_token:
+        raise ValueError("HF_TOKEN environment variable not set")
+
+    loop = TrainingLoop(ENV_BASE_URL, hf_token)
+    metrics = loop.run()
+    save_plots(metrics)
+
+    print("\nNext step: commit training/plots/*.png to repo for submission.")
+```
+
+---
+
+## HOUR 2-3: Build the Colab Training Notebook
+
+### File to Create: `training/colab_training.ipynb`
+
+This must be a runnable Colab notebook. Create it with the following cells in order.
+
+**Cell 1 — Install dependencies:**
+```python
+# Cell 1: Install dependencies
+!pip install openai requests matplotlib numpy
+```
+
+**Cell 2 — Configuration:**
+```python
+# Cell 2: Configuration
+import os
+
+# SET THESE BEFORE RUNNING
+HF_TOKEN = ""  # Your Hugging Face token with inference access
+ENV_URL = "https://godreign-policy2logic.hf.space"  # Your deployed environment URL
+
+os.environ["HF_TOKEN"] = HF_TOKEN
+os.environ["ENV_BASE_URL"] = ENV_URL
+
+print(f"Environment URL: {ENV_URL}")
+print(f"HF Token set: {'Yes' if HF_TOKEN else 'NO - MUST SET THIS'}")
+```
+
+**Cell 3 — Health check:**
+```python
+# Cell 3: Verify environment is reachable
+import requests
+
+r = requests.get(f"{ENV_URL}/health")
+print(f"Status: {r.status_code}")
+print(f"Response: {r.json()}")
+
+r2 = requests.get(f"{ENV_URL}/tasks")
+tasks = r2.json()
+print(f"\nAvailable tasks: {list(tasks['tasks'].keys())}")
+```
+
+**Cell 4 — Paste entire `trajectory_optimizer.py` content here as a cell.**
+
+Add this comment at the top of the cell:
+```python
+# Cell 4: Training loop implementation
+# (paste full contents of training/trajectory_optimizer.py here)
+```
+
+**Cell 5 — Run training:**
+```python
+# Cell 5: Run training loop
+loop = TrainingLoop(ENV_URL, HF_TOKEN)
+metrics = loop.run()
+print(f"\nTotal episodes run: {len(metrics)}")
+```
+
+**Cell 6 — Generate and display plots:**
+```python
+# Cell 6: Generate plots and display inline
+save_plots(metrics)
+
+from IPython.display import Image, display
+display(Image("training/plots/reward_curve.png"))
+display(Image("training/plots/accuracy_curve.png"))
+display(Image("training/plots/improvement_chart.png"))
+```
+
+**Cell 7 — Download plots (CRITICAL — these must be committed to repo):**
+```python
+# Cell 7: Download plots to commit to repo
+# After running this, download the files and commit them to your GitHub repo
+from google.colab import files
+
+files.download("training/plots/reward_curve.png")
+files.download("training/plots/accuracy_curve.png")
+files.download("training/plots/improvement_chart.png")
+files.download("training/plots/metrics.json")
+
+print("Downloaded. Now commit these files to: training/plots/ in your repo.")
+```
+
+---
+
+## HOUR 3-4: Run Training and Capture Results
+
+### Steps (execute in order):
+
+1. Start the environment server locally OR use the deployed HF Space URL.
+2. Open the Colab notebook.
+3. Set `HF_TOKEN` in Cell 2.
+4. Set `ENV_URL` to your HF Space URL: `https://godreign-policy2logic.hf.space`
+5. Run all cells top to bottom.
+6. Wait for training to complete (~20-30 minutes for 8 episodes × 3 tasks).
+7. Cell 7 will download the plot PNG files.
+8. **Immediately commit the PNG files to the repo** under `training/plots/`.
+
+### Git commands after downloading plots:
+```bash
+git add training/plots/reward_curve.png
+git add training/plots/accuracy_curve.png
+git add training/plots/improvement_chart.png
+git add training/plots/metrics.json
+git commit -m "Add training evidence: reward and accuracy curves"
+git push
+```
+
+### If training takes too long (fallback):
+Reduce `NUM_EPISODES_PER_TASK = 4` in the configuration. 4 episodes × 3 tasks = 12 total, which is enough to show a trend.
+
+---
+
+## HOUR 4-5: Write the README (CRITICAL — Validator Reads This)
+
+### File to Replace: `README.md`
+
+The README must link every deliverable. The validator traverses links from README. If a link is broken or missing, that deliverable is marked absent.
+
+```markdown
+# Policy-to-Logic RL Environment
+
+> A verifiable reinforcement learning environment for policy-to-logic reasoning,
+> where an agent learns to iteratively convert natural language policies into
+> executable rules through interaction and reward-guided optimization.
+
+---
+
+## 🔗 Deliverables
+
+| Deliverable | Link |
+|---|---|
+| **HF Space (Live Environment)** | [godreign-policy2logic.hf.space](https://godreign-policy2logic.hf.space) |
+| **Training Notebook (Colab)** | [Open in Colab](https://colab.research.google.com/github/YOUR_USERNAME/YOUR_REPO/blob/main/training/colab_training.ipynb) |
+| **Writeup / Slides** | [Link to your blog/slides/video here] |
+
+> Replace `YOUR_USERNAME/YOUR_REPO` with your actual GitHub path.
+
+---
+
+## 📊 Training Results
+
+The agent is trained using a **reward-guided trajectory optimization loop**.
+High-reward interaction sequences are accumulated as few-shot examples,
+improving agent behavior across episodes without weight updates.
+
+### Reward Curve
+![Reward Curve](training/plots/reward_curve.png)
+
+### Accuracy Curve
+![Accuracy Curve](training/plots/accuracy_curve.png)
+
+### Per-Task Improvement
+![Improvement Chart](training/plots/improvement_chart.png)
+
+---
+
+## 🧠 What This Is
+
+This project builds a **verifiable RL environment** where:
+- Policies are stated in natural language
+- An agent converts them to executable JSON rules (DSL)
+- The environment evaluates rules against generated scenarios
+- Reward signals drive measurable improvement across episodes
+
+**This is not a finished product. It is a training and evaluation framework.**
+
+---
+
+## 🏗️ Architecture
+
+```
+Policy → Agent → (Ask / Propose / Refine)
+       → Environment → (Scenarios + Evaluation)
+       → Reward → Trajectory Bank → Improved Agent
+```
+
+### Three Tasks (increasing difficulty)
+
+| Task | Difficulty | Variables | Decisions |
+|---|---|---|---|
+| data_access | Easy | time, data_type | ALLOW, DENY |
+| resource_access | Medium | role, time, document_type | ALLOW, DENY |
+| transaction_approval | Hard | amount, transfer_type, time, role | APPROVE, REQUIRE_APPROVAL, COMPLIANCE_REVIEW, HOLD |
+
+---
+
+## 🎮 Environment API
+
+Live at: `https://godreign-policy2logic.hf.space`
+
+| Endpoint | Method | Purpose |
+|---|---|---|
+| `/health` | GET | Health check |
+| `/tasks` | GET | List available tasks |
+| `/reset` | POST | Start new episode |
+| `/step` | POST | Take action |
+| `/state` | GET | Get episode state |
+
+### Quick Start
+
+```python
+import requests
+
+base = "https://godreign-policy2logic.hf.space"
+
+# Start episode
+result = requests.post(f"{base}/reset", json={"task_name": "data_access"}).json()
+print(result["observation"]["policy_text"])
+
+# Take action
+action = requests.post(f"{base}/step", json={
+    "action_type": "propose_rules",
+    "content": '{"rules": [{"if": [{"field": "time", "op": ">=", "value": 9}, {"field": "time", "op": "<", "value": 18}], "then": "ALLOW"}], "default": "DENY"}'
+}).json()
+print(f"Reward: {action['reward']}, Accuracy: {action['observation']['current_accuracy']}")
+```
+
+---
+
+## 🔁 Training Loop
+
+The training approach uses **reward-guided trajectory accumulation**:
+
+1. Agent runs episode zero-shot
+2. High-reward trajectories stored in trajectory bank
+3. Next episode uses top-K trajectories as few-shot context
+4. Agent performance improves as bank accumulates better examples
+
+**This is a legitimate policy improvement loop driven by environment reward signal.**
+
+### Run Training Locally
+
+```bash
+# Install dependencies
+pip install openai requests matplotlib numpy
+
+# Set environment variables
+export HF_TOKEN=your_token_here
+export ENV_BASE_URL=https://godreign-policy2logic.hf.space
+
+# Run
+python training/trajectory_optimizer.py
+```
+
+---
+
+## 📁 Repository Structure
+
+```
+├── policy_to_logic_env/
+│   ├── server/
+│   │   ├── app.py              # FastAPI endpoints
+│   │   ├── environment.py      # Core RL environment (reset/step/state)
+│   │   ├── policies.py         # 3 task definitions
+│   │   ├── ground_truth.py     # Ground truth + clarification oracle
+│   │   ├── scenario_generator.py  # 4-strategy scenario generation
+│   │   ├── dsl_engine.py       # JSON DSL parser and executor
+│   │   ├── rewards.py          # Multi-component reward system
+│   │   └── graders.py          # Rule evaluation
+│   ├── models.py               # Pydantic data models
+│   ├── client.py               # HTTP client library
+│   └── openenv.yaml            # OpenEnv specification
+├── training/
+│   ├── trajectory_optimizer.py # Training loop
+│   ├── colab_training.ipynb    # Colab notebook
+│   └── plots/
+│       ├── reward_curve.png    # Training evidence (committed)
+│       ├── accuracy_curve.png  # Training evidence (committed)
+│       └── improvement_chart.png
+├── main.py                     # Server entry point
+├── Dockerfile                  # HF Spaces deployment
+└── README.md                   # This file
+```
+
+---
+
+## ⚙️ OpenEnv Compliance
+
+This environment implements the OpenEnv specification:
+- Gym-style `reset()` / `step()` / `state()` interface
+- Valid `openenv.yaml` at `policy_to_logic_env/openenv.yaml`
+- Pydantic models for all inputs/outputs
+- HTTP API for remote agent interaction
+
+---
+
+## ⚠️ Known Limitations
+
+1. Single-session server (sequential episodes only, not parallel)
+2. Deterministic scenario seed — same scenarios every episode
+3. Training loop uses trajectory accumulation, not weight updates
+4. Clarification oracle is keyword-based, not semantic
+
+---
+
+## 🧾 Reward System
+
+| Component | Weight | Signal |
+|---|---|---|
+| Accuracy | 50% | Rules correct vs ground truth |
+| Improvement | 20% | Accuracy delta per step |
+| Efficiency | 15% | Steps used vs budget |
+| Clarification | 15% | Question usefulness |
+```
+
+---
+
+## HOUR 5: Verify openenv.yaml is Valid
+
+### Check existing file: `policy_to_logic_env/openenv.yaml`
+
+Open the file and verify it contains at minimum:
+
+```yaml
+name: policy-to-logic-env
+version: "1.0.0"
+description: "RL environment for converting natural language policies into executable rules"
+
+environment:
+  type: Environment
+  reset_endpoint: /reset
+  step_endpoint: /step
+  state_endpoint: /state
+
+tasks:
+  - name: data_access
+    difficulty: easy
+    max_steps: 5
+  - name: resource_access
+    difficulty: medium
+    max_steps: 7
+  - name: transaction_approval
+    difficulty: hard
+    max_steps: 7
+
+observation_space:
+  policy_text: string
+  current_accuracy: float
+  available_actions: list
+  feedback: string
+
+action_space:
+  - ask_clarification
+  - propose_rules
+  - refine_rules
+
+reward:
+  min: 0.0
+  max: 1.0
+```
+
+If this file exists and is different, do NOT replace it — just verify it is parseable YAML and contains `reset`, `step`, `state` references.
+
+If it is missing these fields, add them.
+
+---
+
+## HOUR 6: Final Verification Pass
+
+Run through every checklist item explicitly. Do not assume. Verify each one.
+
+### Check 1: HF Space is Public and Reachable
+
+```bash
+# Open this URL in a logged-out browser (incognito window)
+# https://godreign-policy2logic.hf.space
+# Must load without login prompt
+# Must return 200 on /health endpoint
+
+curl https://godreign-policy2logic.hf.space/health
+# Expected: {"status": "ok", "environment": "policy_to_logic"}
+```
+
+### Check 2: OpenEnv Structure
+
+```bash
+# Verify openenv.yaml exists and is valid YAML
+python -c "import yaml; yaml.safe_load(open('policy_to_logic_env/openenv.yaml'))"
+# No error = valid
+
+# Verify reset/step/state endpoints respond
+curl -X POST https://godreign-policy2logic.hf.space/reset -H "Content-Type: application/json" -d '{"task_name": "data_access"}'
+curl -X GET https://godreign-policy2logic.hf.space/state
+```
+
+### Check 3: Plot PNG Files Are Committed
+
+```bash
+git ls-files training/plots/
+# Must output:
+# training/plots/reward_curve.png
+# training/plots/accuracy_curve.png
+# training/plots/improvement_chart.png
+
+# Verify they are actual image files (not empty)
+ls -lh training/plots/*.png
+# Each must be > 10KB
+```
+
+### Check 4: Training Script is Runnable
+
+```bash
+# Verify the Python script runs (dry run — just check imports and config)
+python -c "
+import sys
+sys.path.insert(0, '.')
+# Check imports
+import json, os, time, requests
+from openai import OpenAI
+print('All imports OK')
+print('Training script: training/trajectory_optimizer.py — OK')
+"
+
+# Verify Colab notebook exists
+ls -la training/colab_training.ipynb
+```
+
+### Check 5: README Links
+
+Open `README.md` and manually verify:
+- HF Space link is correct and matches actual URL
+- Colab badge link uses correct GitHub username and repo name
+- Both `![Reward Curve](training/plots/reward_curve.png)` images render when viewed on GitHub
+- Writeup/slides link is filled in (not placeholder)
+
+---
+
+## HOUR 7: Buffer — Fix Whatever Failed Check
+
+Use this hour to fix anything that failed the verification pass.
+
+**Most likely failures and fixes:**
+
+| Failure | Fix |
+|---|---|
+| HF Space returns 404 | Rebuild Docker and redeploy to HF Spaces |
+| PNG files not in repo | Download from Colab, `git add`, `git commit`, `git push` |
+| openenv.yaml missing fields | Add missing fields, push |
+| Colab link broken | Fix GitHub path in README |
+| Plots not rendering in README | Verify relative path matches actual file location |
+
+---
+
+## Fallback: If Training Produces Flat Curves
+
+If the reward curve shows no improvement (all episodes get similar rewards), do NOT fabricate results. Instead:
+
+1. Run more episodes — increase `NUM_EPISODES_PER_TASK = 12`
+2. Lower `MIN_REWARD_THRESHOLD = 0.1` to accumulate more examples
+3. If still flat, the submission narrative becomes: *"We demonstrate that the environment produces consistent reward signals and the agent achieves non-trivial baseline performance. Future work includes fine-tuning with TRL/GRPO."*
+
+A flat but honest curve is better than a fabricated improving curve.
+
+---
+
+## What to Say to Judges (Prepared Answers)
+
+**Q: Is this RL training?**
+> "We implement a reward-guided trajectory optimization loop. The environment's reward signal selects high-value interaction trajectories which are accumulated as few-shot context, improving agent policy across episodes. This is a form of in-context policy improvement driven by environment feedback."
+
+**Q: Why not fine-tune the model?**
+> "Given hackathon constraints, we demonstrate the environment's training capability through trajectory accumulation. The environment is fully compatible with TRL/GRPO fine-tuning — the reward signal, episode structure, and action space are all defined. Fine-tuning is the natural next step."
+
+**Q: What does the agent actually learn?**
+> "The agent learns when to ask clarifying questions versus when to propose rules, and how to refine rules based on failure feedback. The trajectory bank accumulates successful strategies that improve decision-making in subsequent episodes."
+
+**Q: Why is the simulation not realistic?**
+> "The environment is a verification harness, not a simulation. It functions like unit testing for policy logic — correctness is the goal, not realism. This gives us objective, programmatic reward signals suitable for RL."
+
+---
+
+## File Checklist (Everything That Must Exist at Submission)
+
+```
+✅ policy_to_logic_env/openenv.yaml          — already exists, verify valid
+✅ policy_to_logic_env/server/environment.py  — already exists
+✅ policy_to_logic_env/server/app.py          — already exists
+✅ training/trajectory_optimizer.py           — CREATE THIS (Hour 1-2)
+✅ training/colab_training.ipynb              — CREATE THIS (Hour 2-3)
+✅ training/plots/reward_curve.png            — GENERATE AND COMMIT (Hour 3-4)
+✅ training/plots/accuracy_curve.png          — GENERATE AND COMMIT (Hour 3-4)
+✅ training/plots/improvement_chart.png       — GENERATE AND COMMIT (Hour 3-4)
+✅ README.md                                  — REWRITE (Hour 4-5)
+```
+
+---
+
+*End of handoff document. Every step above is required. Execute in order.*

README.md

@@ -9,4 +9,185 @@ pinned: false
 short_description: Meta pytorch hugging face hackathon
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Policy-to-Logic RL Environment
+
+> A verifiable reinforcement learning environment for policy-to-logic reasoning,
+> where an agent learns to iteratively convert natural language policies into
+> executable rules through interaction and reward-guided optimization.
+
+---
+
+## 🔗 Deliverables
+
+| Deliverable | Link |
+|---|---|
+| **HF Space (Live Environment)** | [godreign-policy2logic.hf.space](https://godreign-policy2logic.hf.space) |
+| **Training Notebook (Colab)** | [Open in Colab](https://colab.research.google.com/github/GodreignElgin/policy2logic/blob/main/training/colab_training.ipynb) |
+| **Writeup / Slides** | *TBD — add your link here* |
+
+---
+
+## 📊 Training Results
+
+The agent is trained using a **reward-guided trajectory optimization loop**.
+High-reward interaction sequences are accumulated as few-shot examples,
+improving agent behavior across episodes without weight updates.
+
+### Reward Curve
+![Reward Curve](training/plots/reward_curve.png)
+
+### Accuracy Curve
+![Accuracy Curve](training/plots/accuracy_curve.png)
+
+### Per-Task Improvement
+![Improvement Chart](training/plots/improvement_chart.png)
+
+---
+
+## 🧠 What This Is
+
+This project builds a **verifiable RL environment** where:
+- Policies are stated in natural language
+- An agent converts them to executable JSON rules (DSL)
+- The environment evaluates rules against generated scenarios
+- Reward signals drive measurable improvement across episodes
+
+**This is not a finished product. It is a training and evaluation framework.**
+
+---
+
+## 🏗️ Architecture
+
+```
+Policy → Agent → (Ask / Propose / Refine)
+       → Environment → (Scenarios + Evaluation)
+       → Reward → Trajectory Bank → Improved Agent
+```
+
+### Three Tasks (increasing difficulty)
+
+| Task | Difficulty | Variables | Decisions |
+|---|---|---|---|
+| data_access | Easy | time, data_type | ALLOW, DENY |
+| resource_access | Medium | role, time, document_type | ALLOW, DENY |
+| transaction_approval | Hard | amount, transfer_type, time, role | APPROVE, REQUIRE_APPROVAL, COMPLIANCE_REVIEW, HOLD |
+
+---
+
+## 🎮 Environment API
+
+Live at: `https://godreign-policy2logic.hf.space`
+
+| Endpoint | Method | Purpose |
+|---|---|---|
+| `/health` | GET | Health check |
+| `/tasks` | GET | List available tasks |
+| `/reset` | POST | Start new episode |
+| `/step` | POST | Take action |
+| `/state` | GET | Get episode state |
+
+### Quick Start
+
+```python
+import requests
+
+base = "https://godreign-policy2logic.hf.space"
+
+# Start episode
+result = requests.post(f"{base}/reset", json={"task_name": "data_access"}).json()
+print(result["observation"]["policy_text"])
+
+# Take action
+action = requests.post(f"{base}/step", json={
+    "action_type": "propose_rules",
+    "content": '{"rules": [{"if": [{"field": "time", "op": ">=", "value": 9}, {"field": "time", "op": "<", "value": 18}], "then": "ALLOW"}], "default": "DENY"}'
+}).json()
+print(f"Reward: {action['reward']}, Accuracy: {action['observation']['current_accuracy']}")
+```
+
+---
+
+## 🔁 Training Loop
+
+The training approach uses **reward-guided trajectory accumulation**:
+
+1. Agent runs episode zero-shot
+2. High-reward trajectories stored in trajectory bank
+3. Next episode uses top-K trajectories as few-shot context
+4. Agent performance improves as bank accumulates better examples
+
+**This is a legitimate policy improvement loop driven by environment reward signal.**
+
+### Run Training Locally
+
+```bash
+# Install dependencies
+pip install openai requests matplotlib numpy
+
+# Set environment variables
+export HF_TOKEN=your_token_here
+export ENV_BASE_URL=https://godreign-policy2logic.hf.space
+
+# Run
+python training/trajectory_optimizer.py
+```
+
+---
+
+## 📁 Repository Structure
+
+```
+├── policy_to_logic_env/
+│   ├── server/
+│   │   ├── app.py              # FastAPI endpoints
+│   │   ├── environment.py      # Core RL environment (reset/step/state)
+│   │   ├── policies.py         # 3 task definitions
+│   │   ├── ground_truth.py     # Ground truth + clarification oracle
+│   │   ├── scenario_generator.py  # 4-strategy scenario generation
+│   │   ├── dsl_engine.py       # JSON DSL parser and executor
+│   │   ├── rewards.py          # Multi-component reward system
+│   │   └── graders.py          # Rule evaluation
+│   ├── models.py               # Pydantic data models
+│   ├── client.py               # HTTP client library
+│   └── openenv.yaml            # OpenEnv specification
+├── training/
+│   ├── trajectory_optimizer.py # Training loop
+│   ├── colab_training.ipynb    # Colab notebook
+│   └── plots/
+│       ├── reward_curve.png    # Training evidence (committed)
+│       ├─��� accuracy_curve.png  # Training evidence (committed)
+│       └── improvement_chart.png
+├── main.py                     # Server entry point
+├── Dockerfile                  # HF Spaces deployment
+└── README.md                   # This file
+```
+
+---
+
+## ⚙️ OpenEnv Compliance
+
+This environment implements the OpenEnv specification:
+- Gym-style `reset()` / `step()` / `state()` interface
+- Valid `openenv.yaml` at `policy_to_logic_env/openenv.yaml`
+- Pydantic models for all inputs/outputs
+- HTTP API for remote agent interaction
+
+---
+
+## ⚠️ Known Limitations
+
+1. Single-session server (sequential episodes only, not parallel)
+2. Deterministic scenario seed — same scenarios every episode
+3. Training loop uses trajectory accumulation, not weight updates
+4. Clarification oracle is keyword-based, not semantic
+
+---
+
+## 🧾 Reward System
+
+| Component | Weight | Signal |
+|---|---|---|
+| Accuracy | 50% | Rules correct vs ground truth |
+| Improvement | 20% | Accuracy delta per step |
+| Efficiency | 15% | Steps used vs budget |
+| Clarification | 15% | Question usefulness |

implementation_report.md

@@ -0,0 +1,597 @@
+# Policy-to-Logic RL Environment — Complete Implementation Report
+
+> **Purpose**: Exhaustive description of everything implemented, with exact logic, edge cases, and formulas. Intended for AI-assisted gap analysis against the original plan.
+
+---
+
+## 1. Project Architecture
+
+```
+OpenenvHack/
+├── main.py                          # Entry point: uvicorn on port 7860
+├── Dockerfile                       # Docker SDK deployment for HF Spaces
+├── inference.py                     # LLM agent loop (Qwen2.5-72B via OpenAI API)
+├── pyproject.toml                   # UV project: pydantic, fastapi, uvicorn, openai, huggingface-hub
+├── test_hf_spaces.py                # Remote endpoint tests against HF Spaces
+├── test_all.py                      # Local test runner (starts server, runs tests, stops)
+├── test_local.py / test_endpoints.py # Additional test scripts
+├── policy_to_logic_env/
+│   ├── __init__.py                  # Package exports: models + client
+│   ├── models.py                    # Pydantic models: Action, Observation, State, StepResult
+│   ├── client.py                    # HTTP client wrapper for the environment
+│   ├── openenv.yaml                 # OpenEnv specification file
+│   └── server/
+│       ├── app.py                   # FastAPI app with 6 endpoints
+│       ├── environment.py           # Core environment: reset(), step(), state()
+│       ├── policies.py              # 3 task definitions with clarification maps
+│       ├── ground_truth.py          # Programmatic ground truth + clarification oracle
+│       ├── scenario_generator.py    # 4-strategy scenario generation (seeded)
+│       ├── graders.py               # Rule grading against scenarios
+│       ├── dsl_engine.py            # JSON DSL parser, validator, executor
+│       ├── rewards.py               # 4-component reward system
+│       └── requirements.txt         # Server deps: openenv-core, pydantic, fastapi, uvicorn, requests
+```
+
+**Deployment**: Docker on HF Spaces at `https://godreign-policy2logic.hf.space`, port 7860.
+
+---
+
+## 2. HTTP API (app.py)
+
+Single FastAPI app with CORS `allow_origins=["*"]`. One global `PolicyToLogicEnvironment()` instance (single-session).
+
+| Endpoint | Method | Request Body | Response | Purpose |
+|---|---|---|---|---|
+| `/` | GET | — | `{name, version, status, endpoints, docs, redoc}` | Root probe / API info |
+| `/health` | GET | — | `{status: "ok", environment: "policy_to_logic"}` | Health check |
+| `/tasks` | GET | — | `{tasks: {name: {difficulty, max_steps, scenario_count, valid_decisions, variables}}}` | List all 3 tasks |
+| `/reset` | POST | `{task_name: str \| null}` | `StepResult` (observation + reward=0 + done=false) | Start new episode |
+| `/step` | POST | `{action_type: str, content: str}` | `StepResult` (observation + reward + done) | Take an action |
+| `/state` | GET | — | `PolicyToLogicState` (full episode metadata) | Get current state |
+
+If `task_name` is null or invalid in `/reset`, defaults to `"data_access"`.
+
+---
+
+## 3. Data Models (models.py)
+
+### Action
+```
+action_type: Literal["ask_clarification", "propose_rules", "refine_rules"]
+content: str  # JSON string payload
+```
+
+### Observation (returned in every StepResult)
+```
+policy_text: str          # The natural language policy (always present)
+task_name: str
+step_number: int          # 0 on reset, 1+ on steps
+max_steps: int
+clarification_response: str | None    # Oracle answer if ask_clarification
+test_results: dict | None             # {passed, failed, total, score, sample_failures}
+current_accuracy: float               # 0.0-1.0
+available_actions: list[str]          # What the agent can do next
+feedback: str | None                  # Human-readable feedback
+dsl_format: str                       # DSL syntax instructions (always present)
+```
+
+### State
+```
+episode_id: str
+step_count: int
+task_name: str
+current_rules: list | None
+accuracy_history: list[float]
+questions_asked: int
+questions_log: list[str]
+done: bool
+total_reward: float
+```
+
+### StepResult
+```
+observation: Observation
+reward: float    # 0.0-1.0 per step
+done: bool
+info: dict       # Contains reward_breakdown, episode_score, errors, etc.
+```
+
+---
+
+## 4. Episode Lifecycle (environment.py)
+
+### reset(task_name)
+1. Load task config from registry (defaults to `"data_access"`)
+2. Generate scenarios via `generate_scenarios(task_name)` with `seed=42`
+3. Initialize state: `step_count=0`, `accuracy=0`, `done=false`
+4. Return observation with policy text, DSL format, available decisions/variables
+
+### step(action)
+1. Guard: if `state is None` or `done == True` → error result
+2. Increment `step_count`
+3. Dispatch by `action_type`:
+   - `"ask_clarification"` → `_handle_clarification()`
+   - `"propose_rules"` → `_handle_propose()`
+   - `"refine_rules"` → `_handle_refine()`
+
+### Termination Conditions
+Episode ends (`done=True`) when **either**:
+- `accuracy >= 0.9` (success)
+- `step_count >= max_steps` (budget exhausted)
+
+### Clarification Handling
+1. Parse content as JSON to extract `question`, or use raw content as the question
+2. Call `answer_clarification(task_name, question)` → deterministic oracle answer
+3. Usefulness check: `is_useful = "I can provide information" not in answer`
+4. Compute reward (accuracy stays unchanged, clarification component applies)
+5. `refine_rules` is only available after at least one `propose_rules`
+
+### Rule Proposal/Refinement Handling
+1. Parse JSON content via `parse_rules()` → validates DSL structure
+2. If invalid: penalty reward, feedback with parse errors
+3. If valid: grade rules against stored scenarios → accuracy
+4. Compute reward using accuracy delta
+5. Feedback includes: accuracy, improvement direction, passed/total, sample failure
+6. If `accuracy >= 0.9`: feedback says "Target accuracy reached! Episode complete."
+7. On episode end: compute `episode_score` and include in info
+
+---
+
+## 5. The Three Tasks (policies.py)
+
+### Task 1: `data_access` (Easy)
+
+| Property | Value |
+|---|---|
+| Difficulty | easy |
+| Max Steps | 5 |
+| Scenario Count | 30 |
+| Variables | `time` (0-23), `data_type` (sensitive, public, internal) |
+| Valid Decisions | ALLOW, DENY |
+| Hidden Params | `work_start=9`, `work_end=18` |
+
+**Policy Text** (what the agent sees):
+> Employees must not access sensitive data after working hours. Working hours are from 9 AM to 6 PM (9:00 to 18:00). Public data can be accessed at any time. Internal data follows the same rules as sensitive data.
+
+---
+
+### Task 2: `resource_access` (Medium)
+
+| Property | Value |
+|---|---|
+| Difficulty | medium |
+| Max Steps | 7 |
+| Scenario Count | 50 |
+| Variables | `role` (junior, senior, contractor), `time` (0-23), `document_type` (public, internal, confidential) |
+| Valid Decisions | ALLOW, DENY |
+| Hidden Params | `business_start=8`, `business_end=17` |
+
+**Policy Text**:
+> Junior employees cannot access confidential documents outside business hours. Senior employees have unrestricted access to all document types. Contractors can only access public documents, regardless of time. During business hours, junior employees may access public and internal documents.
+
+**Intentional Ambiguity**: The policy says juniors "cannot access confidential documents outside business hours" — implying they CAN during business hours. But the ground truth DENIES confidential for juniors at ALL times. This is a deliberate trap the agent must discover through testing.
+
+---
+
+### Task 3: `transaction_approval` (Hard)
+
+| Property | Value |
+|---|---|
+| Difficulty | hard |
+| Max Steps | 7 |
+| Scenario Count | 80 |
+| Variables | `amount` (100..50000, 12 values), `transfer_type` (domestic, international), `time` (0-23), `initiator_role` (employee, manager, system) |
+| Valid Decisions | APPROVE, REQUIRE_APPROVAL, COMPLIANCE_REVIEW, HOLD |
+| Hidden Params | `standard_limit=5000`, `high_value_threshold=10000`, `business_start=9`, `business_end=17` |
+
+**Policy Text**:
+> Transactions exceeding the standard limit require manager approval. International transfers always need compliance review regardless of amount. High-value domestic transactions during non-business hours are automatically held for review. Routine domestic transactions within limits are auto-approved. Manager-initiated transactions are exempt from the standard limit.
+
+---
+
+## 6. Ground Truth Logic (ground_truth.py)
+
+### Task 1: `_ground_truth_data_access`
+
+```python
+if data_type == "public":           → ALLOW
+if 9 <= time < 18:                  → ALLOW   # sensitive or internal
+else:                               → DENY
+```
+
+**Complete Decision Table**:
+
+| data_type | time | Decision | Why |
+|---|---|---|---|
+| public | any (0-23) | ALLOW | Public is always accessible |
+| sensitive | 0-8 | DENY | Before working hours |
+| sensitive | 9-17 | ALLOW | During working hours |
+| sensitive | 18-23 | DENY | After working hours (18 is OUTSIDE) |
+| internal | 0-8 | DENY | Same rules as sensitive |
+| internal | 9-17 | ALLOW | Same rules as sensitive |
+| internal | 18-23 | DENY | Same rules as sensitive |
+
+> [!IMPORTANT]
+> **Critical boundary**: `time=18` → DENY. The interval is half-open: `[9, 18)`. Hour 18 is the first after-hours hour. Hour 17 is the last working hour.
+
+---
+
+### Task 2: `_ground_truth_resource_access`
+
+```python
+if role == "senior":                                    → ALLOW
+if role == "contractor":
+    if doc_type == "public":                            → ALLOW
+    else:                                               → DENY
+# Junior employee:
+is_business_hours = (8 <= time < 17)
+if doc_type == "public":                                → ALLOW
+if is_business_hours and doc_type == "internal":        → ALLOW
+else:                                                   → DENY
+```
+
+**Complete Decision Table for Junior Employees**:
+
+| document_type | time | Decision | Why |
+|---|---|---|---|
+| public | any (0-23) | ALLOW | Public always allowed for all roles |
+| internal | 0-7 | DENY | Before business hours |
+| internal | 8-16 | ALLOW | During business hours |
+| internal | 17-23 | DENY | After business hours (17 is OUTSIDE) |
+| confidential | any (0-23) | **DENY** | **Always denied for juniors** |
+
+**Senior**: ALLOW for everything, always.
+**Contractor**: ALLOW only for `public`, DENY for `internal` and `confidential`, at all times.
+
+> [!IMPORTANT]
+> **Critical boundary**: `time=17` → outside business hours. Interval: `[8, 17)`. Hour 16 is the last business hour.
+>
+> **Critical trap**: `confidential` is ALWAYS denied for juniors, even during business hours. The policy text misleadingly implies otherwise.
+
+---
+
+### Task 3: `_ground_truth_transaction_approval`
+
+Rules evaluated in strict priority order (first match wins):
+
+```python
+# Rule 1: International → COMPLIANCE_REVIEW (always, regardless of everything)
+if transfer_type == "international":                    → COMPLIANCE_REVIEW
+
+# Rule 2: High-value domestic outside business hours → HOLD
+if amount >= 10000 and not (9 <= time < 17):            → HOLD
+
+# Rule 3: Above standard limit, not manager → REQUIRE_APPROVAL
+if amount > 5000 and initiator_role != "manager":       → REQUIRE_APPROVAL
+
+# Rule 4: Everything else → APPROVE
+else:                                                   → APPROVE
+```
+
+**Critical Edge Cases**:
+
+| amount | transfer_type | time | initiator_role | Decision | Why |
+|---|---|---|---|---|---|
+| 5000 | domestic | 12 | employee | **APPROVE** | At limit, not above (> 5000 fails) |
+| 5001 | domestic | 12 | employee | REQUIRE_APPROVAL | Above limit, not manager |
+| 5001 | domestic | 12 | manager | **APPROVE** | Manager exempt from limit |
+| 10000 | domestic | 20 | employee | **HOLD** | High-value + non-business hours |
+| 10000 | domestic | 12 | employee | REQUIRE_APPROVAL | High-value but business hours (Rule 2 skipped, Rule 3 matches) |
+| 10000 | domestic | 17 | employee | **HOLD** | 17 is non-business hours |
+| 10000 | domestic | 20 | **manager** | **HOLD** | Managers NOT exempt from HOLD rule |
+| 100 | international | 12 | employee | COMPLIANCE_REVIEW | International always |
+| 50000 | international | 3 | manager | COMPLIANCE_REVIEW | International trumps everything |
+| 9999 | domestic | 20 | employee | REQUIRE_APPROVAL | NOT high-value (< 10000), but above limit |
+| 100 | domestic | 3 | employee | APPROVE | Within limit |
+| 100 | domestic | 3 | system | APPROVE | System = employee |
+
+> [!IMPORTANT]
+> **Standard limit comparison**: `amount > 5000` (strict greater than). $5,000 exactly = APPROVE.
+>
+> **High-value comparison**: `amount >= 10000` (greater than or equal). $10,000 exactly = high-value.
+>
+> **Manager exemption scope**: Only exempts from Rule 3 (standard limit). Managers are still subject to Rule 1 (international) and Rule 2 (high-value HOLD).
+>
+> **Business hours**: `[9, 17)`. Hour 17 is non-business.
+
+---
+
+## 7. Clarification Oracle (ground_truth.py)
+
+### Matching Algorithm
+
+```
+Input:  question string (free text from agent)
+Output: best matching answer from task's clarification_map
+
+Algorithm:
+1. Lowercase the question
+2. For each keyword in clarification_map:
+   a. Split keyword into parts by spaces
+   b. Check if ALL parts appear as substrings in the question
+   c. Score = (number_of_parts, total_keyword_length)
+   d. Highest score wins
+3. If no match: return generic fallback (contains "I can provide information")
+```
+
+**Key property**: "junior confidential" matches when BOTH "junior" AND "confidential" appear anywhere in the question (order-independent). This 2-part keyword beats any 1-part keyword like "junior" alone.
+
+### Usefulness Detection
+
+In `environment.py`, line 203:
+```python
+is_useful = "I can provide information" not in answer
+```
+Any answer that matches a keyword entry is "useful". Only the generic fallback is "not useful".
+
+### 3-Tier Progressive Revelation Design
+
+Each task's `clarification_map` has three levels:
+
+| Tier | Keyword Type | Answer Quality | Training Purpose |
+|---|---|---|---|
+| Level 1 | Single short words | Partial truths, technically correct but incomplete/misleading | Agent builds initial (wrong) rules |
+| Level 2 | Common phrases | More detail, boundary still ambiguous | Agent narrows down the problem |
+| Level 3 | Compound/multi-word | Precise, ground-truth-aligned, corrects Level 1 | Agent fixes rules after failures |
+
+**Example — resource_access contradiction**:
+- Agent asks "What can junior employees access?" → matches `"junior"` (Level 1) → *"...but not confidential documents outside business hours"* (implies CAN during hours)
+- Agent proposes rules allowing junior+confidential during hours → **fails**
+- Agent asks "Can junior employees access confidential documents?" → matches `"junior confidential"` (Level 3, 2 parts > 1 part) → *"CANNOT access confidential at ANY time"*
+- Agent refines rules → **passes**
+
+### Clarification Map Entry Counts
+
+| Task | Level 1 | Level 2 | Level 3 | Total |
+|---|---|---|---|---|
+| data_access | 5 | 3 | 6 | 14 |
+| resource_access | 7 | 3 | 8 | 18 |
+| transaction_approval | 9 | 7 | 10 | 26 |
+
+---
+
+## 8. DSL Engine (dsl_engine.py)
+
+### DSL Format
+
+```json
+{
+    "rules": [
+        {
+            "if": [
+                {"field": "<name>", "op": "<operator>", "value": <value>}
+            ],
+            "then": "<DECISION>"
+        }
+    ],
+    "default": "<DEFAULT_DECISION>"
+}
+```
+
+### Supported Operators
+`>`, `<`, `>=`, `<=`, `==`, `!=`
+
+### Validation (`validate_rules`)
+Checks:
+- Root is a dict
+- Has `"rules"` key (must be list)
+- Has `"default"` key (must be string)
+- Each rule has `"if"` (list) and `"then"` (string)
+- Each condition has `"field"` (string), `"op"` (valid operator), `"value"`
+
+Returns `(is_valid: bool, errors: list[str])`.
+
+### Execution (`execute_rules`)
+1. Iterate rules top-to-bottom
+2. For each rule, evaluate ALL conditions (AND logic)
+3. First rule where all conditions match → return its `"then"` decision
+4. If no rules match → return `"default"`
+
+### Type Coercion
+If scenario has `time=9` (int) and rule has `"value": "9"` (str), coerces the string to int. Works both directions. If coercion fails, condition evaluates to `False`.
+
+### Parsing (`parse_rules`)
+1. `json.loads()` the content string
+2. `validate_rules()` on the parsed dict
+3. Returns `(rules_data, [])` on success or `(None, errors)` on failure
+
+---
+
+## 9. Scenario Generator (scenario_generator.py)
+
+### Strategy Allocation
+
+| Strategy | Share | Purpose |
+|---|---|---|
+| Boundary | ~20% | Edge values near hidden param thresholds |
+| Pairwise | ~30% | Systematic variable combinations |
+| Adversarial | ~20% | Hand-crafted traps for common mistakes |
+| Random | remainder | Uniform sampling from variable space |
+
+All seeded with `seed=42` for reproducibility. Scenarios are deduplicated by field values.
+
+### Boundary Strategy
+Extracts numeric hidden params, generates scenarios at `param ± 1` and at variable min/max.
+
+### Pairwise Strategy
+For each pair of variables, samples up to 4 representative values (min, max, middle, random), generates cross-product combinations.
+
+### Adversarial Strategy
+**Hand-crafted per task** — these are the exact scenarios:
+
+#### data_access adversarial:
+| time | data_type | Expected | Tests |
+|---|---|---|---|
+| 9 | sensitive | ALLOW | Start boundary |
+| 18 | sensitive | DENY | End boundary (exclusive) |
+| 8 | sensitive | DENY | Just before start |
+| 17 | sensitive | ALLOW | Just before end |
+| 0 | public | ALLOW | Public at midnight |
+| 23 | internal | DENY | Internal late night |
+| 12 | internal | ALLOW | Internal during hours |
+
+#### resource_access adversarial:
+| role | time | document_type | Expected | Tests |
+|---|---|---|---|---|
+| junior | 8 | confidential | DENY | Confidential at business start |
+| junior | 7 | internal | DENY | Internal before hours |
+| junior | 17 | internal | DENY | Internal at boundary (17=outside) |
+| junior | 16 | internal | ALLOW | Internal just before boundary |
+| contractor | 12 | internal | DENY | Contractor restricted |
+| senior | 2 | confidential | ALLOW | Senior unrestricted |
+| junior | 12 | public | ALLOW | Junior public during hours |
+| contractor | 12 | public | ALLOW | Contractor public |
+
+#### transaction_approval adversarial:
+| amount | transfer | time | role | Expected | Tests |
+|---|---|---|---|---|---|
+| 5000 | domestic | 12 | employee | APPROVE | At limit (not above) |
+| 5001 | domestic | 12 | employee | REQ_APPROVAL | Just above limit |
+| 5001 | domestic | 12 | manager | APPROVE | Manager exempt |
+| 10000 | domestic | 20 | employee | HOLD | High-value non-business |
+| 10000 | domestic | 12 | employee | REQ_APPROVAL | High-value business hours |
+| 100 | international | 12 | employee | COMPLIANCE | International small |
+| 50000 | international | 3 | manager | COMPLIANCE | International manager |
+| 9999 | domestic | 20 | employee | REQ_APPROVAL | Below high-value threshold |
+| 10000 | domestic | 9 | employee | REQ_APPROVAL | High-value at business start |
+| 10000 | domestic | 17 | employee | HOLD | 17=non-business |
+
+---
+
+## 10. Grading (graders.py)
+
+### `grade_task(task_name, rules_data, scenarios)`
+1. Validate rules → if invalid, return `score=0.0`
+2. For each scenario: execute agent's rules, compare to `expected_decision`
+3. Comparison: `actual.upper() == expected.upper()` (case-insensitive)
+4. `score = passed / total`
+5. Returns up to 5 `sample_failures` with scenario details, expected, got
+
+### `quick_grade(task_name, rules_data, scenarios)`
+Same logic, returns only the float score. Used during step processing.
+
+---
+
+## 11. Reward System (rewards.py)
+
+### Per-Step Reward: `compute_reward()`
+
+4 components, clamped to `[0.0, 1.0]`:
+
+| Component | Weight | Formula |
+|---|---|---|
+| **Accuracy** | 0.50 | `current_accuracy × 0.50` |
+| **Improvement** | 0.20 | `min(delta × 2.0, 1.0) × 0.20` if delta > 0; `max(delta × 1.5, -0.5) × 0.20` if delta < 0; `0` if unchanged |
+| **Efficiency** | 0.15 | `max(-0.02 × step_number [+ 0.05 × steps_saved if acc≥0.9], -0.15) × 0.15` |
+| **Clarification** | 0.15 | See below |
+
+**Clarification component details**:
+- `ask_clarification` + useful + questions ≤ 3: `+0.3 × 0.15 = +0.045`
+- `ask_clarification` + useful + questions > 3: `+0.1 × 0.15 = +0.015` (diminishing)
+- `ask_clarification` + not useful: `-0.05 × 0.15 = -0.0075`
+- `propose_rules/refine_rules` + invalid DSL: `-0.1 × 0.15 = -0.015`
+- `propose_rules/refine_rules` + valid DSL: `0`
+
+### Episode Score: `compute_episode_score()`
+
+Used for final grading, `[0.0, 1.0]`:
+
+```
+score = final_accuracy × 0.80
+     + max(0, 1 - steps/max_steps) × 0.10
+     + question_bonus × 0.10
+
+question_bonus = 1.0 if questions ≤ 2
+               = 0.5 if questions ≤ 4
+               = 0.0 if questions > 4
+```
+
+---
+
+## 12. Inference Agent (inference.py)
+
+### Configuration
+- Model: `Qwen/Qwen2.5-72B-Instruct` (via `HF_TOKEN`)
+- API: `https://router.huggingface.co/v1` (OpenAI-compatible)
+- Temperature: 0.3, Max tokens: 1024
+- Env URL: `http://localhost:7860` (configurable via `ENV_BASE_URL`)
+
+### Agent Loop
+```
+for each task in [data_access, resource_access, transaction_approval]:
+    result = env.reset(task)
+    for step in 1..max_steps:
+        if result.done: break
+        action_type, content = get_agent_action(llm, observation, step, history)
+        result = env.step(action)
+        history.append(summary)
+```
+
+### Prompt Design
+- **System prompt**: Describes available actions, DSL format, strategy guidelines
+- **User prompt**: Built per-step with policy text, feedback, clarification answers, test results, sample failures, DSL format, action history (last 3)
+- LLM response parsed as JSON: `{"action_type": "...", "content": "..."}`
+- Handles markdown code blocks (`\`\`\`json ... \`\`\``)
+- Fallback: if unparseable, tries extracting `"rules"`, otherwise submits empty rules
+
+### Output Format
+```
+[START] task=<name> env=policy_to_logic model=<model>
+[STEP]  step=<n> action=<summary> reward=<float> done=<bool> error=<msg|null>
+[END]   success=<bool> steps=<n> score=<float> rewards=<r1,r2,...>
+```
+
+---
+
+## 13. Client Library (client.py)
+
+HTTP client using `requests.Session()`:
+- `reset(task_name)` → POST `/reset` → `PolicyToLogicStepResult`
+- `step(action)` → POST `/step` → `PolicyToLogicStepResult`
+- `state()` → GET `/state` → `PolicyToLogicState`
+- `health()` → GET `/health` → dict
+- `list_tasks()` → GET `/tasks` → dict
+- Context manager support (`with PolicyToLogicEnv() as env:`)
+
+---
+
+## 14. Deployment
+
+### Dockerfile
+```dockerfile
+FROM python:3.11-slim
+WORKDIR /app
+COPY policy_to_logic_env/server/requirements.txt → pip install
+COPY policy_to_logic_env/, main.py, inference.py
+EXPOSE 7860
+HEALTHCHECK: curl -f http://localhost:7860/health
+CMD: python -m uvicorn policy_to_logic_env.server.app:app --host 0.0.0.0 --port 7860
+```
+
+### HF Spaces Config (README.md)
+```yaml
+sdk: docker
+app_port: 7860
+```
+
+Live at: `https://godreign-policy2logic.hf.space`
+
+---
+
+## 15. Known Design Decisions & Limitations
+
+1. **Single-session**: One global environment instance. Concurrent clients will interfere. Suitable for sequential benchmarking, not parallel RL training.
+
+2. **Deterministic scenarios**: `seed=42` always produces the same scenarios. Agent is graded on the same set every episode. Prevents overfitting variance but could lead to memorization.
+
+3. **Stateful server**: The environment holds state in memory. Server restart loses episode state. No persistence layer.
+
+4. **Clarification is keyword-based**: The oracle is not an LLM — it's a deterministic keyword matcher. Agent questions that don't contain any keyword get the generic fallback (penalized as "not useful").
+
+5. **Progressive revelation by design**: Level 1 clarification answers are intentionally misleading partial truths. This is NOT a bug — it's the core RL training signal. Agents that trust Level 1 answers will fail and must learn to ask better (Level 3) questions.
+
+6. **No `refine_rules` before `propose_rules`**: The environment returns a feedback message if the agent tries to refine before proposing. Not an error, just 0 reward + feedback.
+
+7. **Case-insensitive grading**: `actual.upper() == expected.upper()`. Agent can output "allow" or "Allow" or "ALLOW".
+
+8. **DSL type coercion**: Integer-string mismatches are auto-coerced. `"9"` and `9` compare equally.

policy_to_logic_env/openenv.yaml

@@ -15,6 +15,9 @@ tags:
 environment:
   entry_point: "policy_to_logic_env.server.app:app"
   python_version: ">=3.10"
+  reset_endpoint: /reset
+  step_endpoint: /step
+  state_endpoint: /state
 
 tasks:
   - name: data_access

pyproject.toml

@@ -11,6 +11,8 @@ dependencies = [
     "openai>=1.0.0",
     "huggingface>=0.0.1",
     "huggingface-hub>=1.12.0",
+    "matplotlib>=3.7.0",
+    "numpy>=1.24.0",
 ]
 
 [project.optional-dependencies]

training/colab_training.ipynb

@@ -0,0 +1,378 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+  "colab": {
+   "provenance": [],
+   "name": "Policy-to-Logic Training"
+  },
+  "kernelspec": {
+   "name": "python3",
+   "display_name": "Python 3"
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Policy-to-Logic RL Environment — Training Notebook\n",
+    "\n",
+    "This notebook runs the **reward-guided trajectory optimization loop** against the deployed environment.\n",
+    "\n",
+    "**What it does:**\n",
+    "1. Connects to the live HF Spaces environment\n",
+    "2. Runs 8 episodes per task (3 tasks = 24 total episodes)\n",
+    "3. Accumulates high-reward trajectories as few-shot examples\n",
+    "4. Generates training evidence plots (reward curve, accuracy curve, improvement chart)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cell 1: Install dependencies\n",
+    "!pip install openai requests matplotlib numpy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cell 2: Configuration\n",
+    "import os\n",
+    "\n",
+    "# SET THESE BEFORE RUNNING\n",
+    "HF_TOKEN = \"\"  # Your Hugging Face token with inference access\n",
+    "ENV_URL = \"https://godreign-policy2logic.hf.space\"  # Your deployed environment URL\n",
+    "\n",
+    "os.environ[\"HF_TOKEN\"] = HF_TOKEN\n",
+    "os.environ[\"ENV_BASE_URL\"] = ENV_URL\n",
+    "\n",
+    "print(f\"Environment URL: {ENV_URL}\")\n",
+    "print(f\"HF Token set: {'Yes' if HF_TOKEN else 'NO - MUST SET THIS'}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cell 3: Verify environment is reachable\n",
+    "import requests\n",
+    "\n",
+    "r = requests.get(f\"{ENV_URL}/health\")\n",
+    "print(f\"Status: {r.status_code}\")\n",
+    "print(f\"Response: {r.json()}\")\n",
+    "\n",
+    "r2 = requests.get(f\"{ENV_URL}/tasks\")\n",
+    "tasks = r2.json()\n",
+    "print(f\"\\nAvailable tasks: {list(tasks['tasks'].keys())}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cell 4: Training loop implementation\n",
+    "# Full contents of training/trajectory_optimizer.py\n",
+    "\n",
+    "import json\n",
+    "import os\n",
+    "import time\n",
+    "import requests\n",
+    "from dataclasses import dataclass, field\n",
+    "from typing import Optional\n",
+    "from openai import OpenAI\n",
+    "\n",
+    "# ── Configuration ────────────────────────────────────────────────────────────\n",
+    "\n",
+    "ENV_BASE_URL = os.getenv(\"ENV_BASE_URL\", \"http://localhost:7860\")\n",
+    "HF_TOKEN = os.getenv(\"HF_TOKEN\", \"\")\n",
+    "MODEL = \"Qwen/Qwen2.5-72B-Instruct\"\n",
+    "TEMPERATURE = 0.3\n",
+    "MAX_TOKENS = 1024\n",
+    "\n",
+    "NUM_EPISODES_PER_TASK = 8\n",
+    "TOP_K_TRAJECTORIES = 3\n",
+    "MIN_REWARD_THRESHOLD = 0.3\n",
+    "TASKS = [\"data_access\", \"resource_access\", \"transaction_approval\"]\n",
+    "\n",
+    "@dataclass\n",
+    "class Step:\n",
+    "    step_number: int\n",
+    "    action_type: str\n",
+    "    action_content: str\n",
+    "    reward: float\n",
+    "    accuracy: float\n",
+    "    feedback: str\n",
+    "    clarification_response: Optional[str] = None\n",
+    "\n",
+    "@dataclass\n",
+    "class Trajectory:\n",
+    "    task_name: str\n",
+    "    episode_id: int\n",
+    "    steps: list[Step] = field(default_factory=list)\n",
+    "    total_reward: float = 0.0\n",
+    "    final_accuracy: float = 0.0\n",
+    "    success: bool = False\n",
+    "\n",
+    "    def to_few_shot_string(self) -> str:\n",
+    "        lines = [f\"=== Example Episode (reward={self.total_reward:.2f}, accuracy={self.final_accuracy:.2f}) ===\"]\n",
+    "        for s in self.steps:\n",
+    "            lines.append(f\"Step {s.step_number}: action={s.action_type}\")\n",
+    "            lines.append(f\"  Content: {s.action_content[:200]}\")\n",
+    "            lines.append(f\"  Result: accuracy={s.accuracy:.2f}, reward={s.reward:.2f}\")\n",
+    "            if s.feedback:\n",
+    "                lines.append(f\"  Feedback: {s.feedback[:150]}\")\n",
+    "        return \"\\n\".join(lines)\n",
+    "\n",
+    "class EnvClient:\n",
+    "    def __init__(self, base_url: str):\n",
+    "        self.base_url = base_url.rstrip(\"/\")\n",
+    "        self.session = requests.Session()\n",
+    "\n",
+    "    def reset(self, task_name: str) -> dict:\n",
+    "        r = self.session.post(f\"{self.base_url}/reset\", json={\"task_name\": task_name})\n",
+    "        r.raise_for_status()\n",
+    "        return r.json()\n",
+    "\n",
+    "    def step(self, action_type: str, content: str) -> dict:\n",
+    "        r = self.session.post(f\"{self.base_url}/step\", json={\"action_type\": action_type, \"content\": content})\n",
+    "        r.raise_for_status()\n",
+    "        return r.json()\n",
+    "\n",
+    "    def health(self) -> bool:\n",
+    "        try:\n",
+    "            r = self.session.get(f\"{self.base_url}/health\", timeout=5)\n",
+    "            return r.status_code == 200\n",
+    "        except Exception:\n",
+    "            return False\n",
+    "\n",
+    "class Agent:\n",
+    "    def __init__(self, hf_token: str):\n",
+    "        self.client = OpenAI(base_url=\"https://router.huggingface.co/v1\", api_key=hf_token)\n",
+    "\n",
+    "    def get_action(self, observation, step_number, episode_history, few_shot_examples):\n",
+    "        system_prompt = self._build_system_prompt(few_shot_examples)\n",
+    "        user_prompt = self._build_user_prompt(observation, step_number, episode_history)\n",
+    "        try:\n",
+    "            response = self.client.chat.completions.create(\n",
+    "                model=MODEL,\n",
+    "                messages=[{\"role\": \"system\", \"content\": system_prompt}, {\"role\": \"user\", \"content\": user_prompt}],\n",
+    "                temperature=TEMPERATURE, max_tokens=MAX_TOKENS\n",
+    "            )\n",
+    "            raw = response.choices[0].message.content.strip()\n",
+    "            return self._parse_response(raw, observation)\n",
+    "        except Exception as e:\n",
+    "            print(f\"    [LLM ERROR] {e}\")\n",
+    "            return \"propose_rules\", json.dumps({\"rules\": [], \"default\": \"DENY\"})\n",
+    "\n",
+    "    def _build_system_prompt(self, few_shot_examples):\n",
+    "        base = \"\"\"You are a policy-to-logic agent. Convert natural language policies into executable rules.\n",
+    "\n",
+    "AVAILABLE ACTIONS:\n",
+    "1. ask_clarification: {\"type\": \"clarification\", \"question\": \"your question\"}\n",
+    "2. propose_rules: {\"rules\": [...], \"default\": \"DECISION\"}\n",
+    "3. refine_rules: {\"rules\": [...], \"default\": \"DECISION\"}\n",
+    "\n",
+    "DSL FORMAT: {\"rules\": [{\"if\": [{\"field\": \"NAME\", \"op\": \"OP\", \"value\": VAL}], \"then\": \"DECISION\"}], \"default\": \"FALLBACK\"}\n",
+    "Operators: >, <, >=, <=, ==, !=. Rules execute top-to-bottom, first match wins.\n",
+    "\n",
+    "STRATEGY: Ask 1-2 clarifications first, then propose rules, then refine based on failures.\n",
+    "OUTPUT: Respond ONLY with valid JSON: {\"action_type\": \"...\", \"content\": \"...\"}\"\"\"\n",
+    "        if few_shot_examples:\n",
+    "            base += \"\\n\\nLEARNED FROM PREVIOUS EPISODES:\\n\"\n",
+    "            for traj in few_shot_examples[-TOP_K_TRAJECTORIES:]:\n",
+    "                base += \"\\n\" + traj.to_few_shot_string() + \"\\n\"\n",
+    "        return base\n",
+    "\n",
+    "    def _build_user_prompt(self, obs, step, history):\n",
+    "        lines = [f\"TASK: {obs.get('task_name', 'unknown')}\", f\"STEP: {step} of {obs.get('max_steps', 7)}\", f\"\\nPOLICY:\\n{obs.get('policy_text', '')}\"]\n",
+    "        if obs.get(\"clarification_response\"): lines.append(f\"\\nLAST CLARIFICATION:\\n{obs['clarification_response']}\")\n",
+    "        if obs.get(\"test_results\"):\n",
+    "            tr = obs[\"test_results\"]\n",
+    "            lines.append(f\"\\nTEST RESULTS: {tr.get('passed',0)}/{tr.get('total',0)} (acc={obs.get('current_accuracy',0):.2f})\")\n",
+    "            if tr.get(\"sample_failures\"): lines.extend([f\"  - {f}\" for f in tr[\"sample_failures\"][:3]])\n",
+    "        if obs.get(\"feedback\"): lines.append(f\"\\nFEEDBACK: {obs['feedback']}\")\n",
+    "        if history: lines.append(f\"\\nHISTORY:\\n\" + \"\\n\".join(history[-3:]))\n",
+    "        lines.append(f\"\\nAVAILABLE: {obs.get('available_actions', [])}\")\n",
+    "        lines.append(\"\\nRespond with JSON only.\")\n",
+    "        return \"\\n\".join(lines)\n",
+    "\n",
+    "    def _parse_response(self, raw, obs):\n",
+    "        if \"```\" in raw:\n",
+    "            raw = raw.split(\"```\")[1]\n",
+    "            if raw.startswith(\"json\"): raw = raw[4:]\n",
+    "        raw = raw.strip()\n",
+    "        try:\n",
+    "            parsed = json.loads(raw)\n",
+    "            action_type = parsed.get(\"action_type\", \"propose_rules\")\n",
+    "            content = parsed.get(\"content\", \"{}\")\n",
+    "            valid = obs.get(\"available_actions\", [\"propose_rules\"])\n",
+    "            if action_type not in valid: action_type = valid[0]\n",
+    "            if isinstance(content, dict): content = json.dumps(content)\n",
+    "            return action_type, content\n",
+    "        except: return \"propose_rules\", json.dumps({\"rules\": [], \"default\": \"DENY\"})\n",
+    "\n",
+    "class TrajectoryBank:\n",
+    "    def __init__(self): self.bank = {task: [] for task in TASKS}\n",
+    "    def store(self, t):\n",
+    "        if t.total_reward >= MIN_REWARD_THRESHOLD:\n",
+    "            self.bank[t.task_name].append(t)\n",
+    "            self.bank[t.task_name].sort(key=lambda x: x.total_reward, reverse=True)\n",
+    "            self.bank[t.task_name] = self.bank[t.task_name][:TOP_K_TRAJECTORIES]\n",
+    "    def get_examples(self, task): return self.bank.get(task, [])\n",
+    "    def summary(self): return {t: {\"stored\": len(v), \"best_reward\": max((x.total_reward for x in v), default=0)} for t,v in self.bank.items()}\n",
+    "\n",
+    "class TrainingLoop:\n",
+    "    def __init__(self, env_url, hf_token):\n",
+    "        self.env = EnvClient(env_url)\n",
+    "        self.agent = Agent(hf_token)\n",
+    "        self.bank = TrajectoryBank()\n",
+    "        self.metrics = []\n",
+    "\n",
+    "    def run_episode(self, task_name, episode_id):\n",
+    "        few_shots = self.bank.get_examples(task_name)\n",
+    "        traj = Trajectory(task_name=task_name, episode_id=episode_id)\n",
+    "        result = self.env.reset(task_name)\n",
+    "        obs, done, history = result.get(\"observation\", {}), result.get(\"done\", False), []\n",
+    "        print(f\"  [Episode {episode_id}] task={task_name} few_shots={len(few_shots)}\")\n",
+    "        step_num = 0\n",
+    "        while not done and step_num < obs.get(\"max_steps\", 7):\n",
+    "            step_num += 1\n",
+    "            action_type, content = self.agent.get_action(obs, step_num, history, few_shots)\n",
+    "            result = self.env.step(action_type, content)\n",
+    "            reward, done = result.get(\"reward\", 0.0), result.get(\"done\", False)\n",
+    "            obs, info = result.get(\"observation\", {}), result.get(\"info\", {})\n",
+    "            step = Step(step_num, action_type, content[:300], reward, obs.get(\"current_accuracy\", 0.0), obs.get(\"feedback\", \"\") or \"\", obs.get(\"clarification_response\"))\n",
+    "            traj.steps.append(step); traj.total_reward += reward\n",
+    "            history.append(f\"Step {step_num}: {action_type} -> reward={reward:.2f} acc={step.accuracy:.2f}\")\n",
+    "            print(f\"    step={step_num} action={action_type} reward={reward:.3f} acc={step.accuracy:.2f}\")\n",
+    "            if done:\n",
+    "                traj.final_accuracy = info.get(\"episode_score\", obs.get(\"current_accuracy\", 0.0))\n",
+    "                traj.success = obs.get(\"current_accuracy\", 0.0) >= 0.9\n",
+    "                break\n",
+    "        if not traj.steps: traj.final_accuracy = 0.0\n",
+    "        return traj\n",
+    "\n",
+    "    def run(self):\n",
+    "        print(\"=\" * 60)\n",
+    "        print(\"REWARD-GUIDED TRAJECTORY OPTIMIZATION\")\n",
+    "        print(f\"Tasks: {TASKS}, Episodes/task: {NUM_EPISODES_PER_TASK}\")\n",
+    "        print(\"=\" * 60)\n",
+    "        if not self.env.health(): raise RuntimeError(f\"Env not reachable at {ENV_BASE_URL}\")\n",
+    "        print(f\"Environment: OK\\n\")\n",
+    "        global_ep = 0\n",
+    "        for task in TASKS:\n",
+    "            print(f\"\\n--- TASK: {task} ---\")\n",
+    "            task_rewards = []\n",
+    "            for ep in range(1, NUM_EPISODES_PER_TASK + 1):\n",
+    "                global_ep += 1\n",
+    "                traj = self.run_episode(task, ep)\n",
+    "                self.bank.store(traj)\n",
+    "                self.metrics.append({\"global_episode\": global_ep, \"task\": task, \"episode_in_task\": ep, \"total_reward\": traj.total_reward, \"final_accuracy\": traj.final_accuracy, \"success\": traj.success, \"num_steps\": len(traj.steps)})\n",
+    "                task_rewards.append(traj.total_reward)\n",
+    "                print(f\"  -> Ep {ep}: reward={traj.total_reward:.3f} acc={traj.final_accuracy:.2f} success={traj.success}\")\n",
+    "                time.sleep(0.5)\n",
+    "            print(f\"  Improvement: {task_rewards[-1] - task_rewards[0]:+.3f}\")\n",
+    "        print(\"\\n\" + \"=\" * 60 + \"\\nTRAINING COMPLETE\\n\" + \"=\" * 60)\n",
+    "        return self.metrics\n",
+    "\n",
+    "def save_plots(metrics):\n",
+    "    import matplotlib; matplotlib.use(\"Agg\")\n",
+    "    import matplotlib.pyplot as plt; import numpy as np\n",
+    "    os.makedirs(\"training/plots\", exist_ok=True)\n",
+    "    episodes = [m[\"global_episode\"] for m in metrics]\n",
+    "    rewards = [m[\"total_reward\"] for m in metrics]\n",
+    "    colors = {\"data_access\": \"#2196F3\", \"resource_access\": \"#FF9800\", \"transaction_approval\": \"#4CAF50\"}\n",
+    "    # Plot 1: Reward\n",
+    "    fig, ax = plt.subplots(figsize=(10, 5))\n",
+    "    for task in TASKS:\n",
+    "        te = [m[\"global_episode\"] for m in metrics if m[\"task\"]==task]\n",
+    "        tr = [m[\"total_reward\"] for m in metrics if m[\"task\"]==task]\n",
+    "        ax.plot(te, tr, marker=\"o\", label=task, color=colors.get(task), linewidth=2, markersize=5)\n",
+    "    z = np.polyfit(episodes, rewards, 1); p = np.poly1d(z)\n",
+    "    ax.plot(episodes, p(episodes), \"--\", color=\"red\", alpha=0.5, label=\"trend\")\n",
+    "    ax.set_xlabel(\"Episode\"); ax.set_ylabel(\"Total Reward\"); ax.set_title(\"Reward Curve\"); ax.legend(); ax.grid(True, alpha=0.3); ax.set_ylim(bottom=0)\n",
+    "    plt.tight_layout(); plt.savefig(\"training/plots/reward_curve.png\", dpi=150); plt.close()\n",
+    "    # Plot 2: Accuracy\n",
+    "    fig, ax = plt.subplots(figsize=(10, 5))\n",
+    "    for task in TASKS:\n",
+    "        te = [m[\"global_episode\"] for m in metrics if m[\"task\"]==task]\n",
+    "        ta = [m[\"final_accuracy\"] for m in metrics if m[\"task\"]==task]\n",
+    "        ax.plot(te, ta, marker=\"s\", label=task, color=colors.get(task), linewidth=2, markersize=5)\n",
+    "    ax.axhline(y=0.9, color=\"red\", linestyle=\"--\", alpha=0.7, label=\"threshold\")\n",
+    "    ax.set_xlabel(\"Episode\"); ax.set_ylabel(\"Accuracy\"); ax.set_title(\"Accuracy Curve\"); ax.legend(); ax.grid(True, alpha=0.3); ax.set_ylim(0, 1.05)\n",
+    "    plt.tight_layout(); plt.savefig(\"training/plots/accuracy_curve.png\", dpi=150); plt.close()\n",
+    "    # Plot 3: Improvement\n",
+    "    fig, ax = plt.subplots(figsize=(8, 5))\n",
+    "    tnames, imps = [], []\n",
+    "    for task in TASKS:\n",
+    "        accs = [m[\"final_accuracy\"] for m in metrics if m[\"task\"]==task]\n",
+    "        if len(accs) >= 2: tnames.append(task.replace(\"_\",\"\\n\")); imps.append(accs[-1]-accs[0])\n",
+    "    bars = ax.bar(tnames, imps, color=[\"#2196F3\",\"#FF9800\",\"#4CAF50\"][:len(tnames)])\n",
+    "    ax.axhline(y=0, color=\"black\"); ax.set_ylabel(\"Improvement\"); ax.set_title(\"Per-Task Improvement\"); ax.grid(True, axis=\"y\", alpha=0.3)\n",
+    "    for bar, val in zip(bars, imps): ax.text(bar.get_x()+bar.get_width()/2, bar.get_height()+0.01, f\"{val:+.2f}\", ha=\"center\", fontweight=\"bold\")\n",
+    "    plt.tight_layout(); plt.savefig(\"training/plots/improvement_chart.png\", dpi=150); plt.close()\n",
+    "    with open(\"training/plots/metrics.json\", \"w\") as f: json.dump(metrics, f, indent=2)\n",
+    "    print(\"All plots saved to training/plots/\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cell 5: Run training loop\n",
+    "loop = TrainingLoop(ENV_URL, HF_TOKEN)\n",
+    "metrics = loop.run()\n",
+    "print(f\"\\nTotal episodes run: {len(metrics)}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cell 6: Generate plots and display inline\n",
+    "save_plots(metrics)\n",
+    "\n",
+    "from IPython.display import Image, display\n",
+    "display(Image(\"training/plots/reward_curve.png\"))\n",
+    "display(Image(\"training/plots/accuracy_curve.png\"))\n",
+    "display(Image(\"training/plots/improvement_chart.png\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Cell 7: Download plots to commit to repo\n",
+    "# After running this, download the files and commit them to your GitHub repo\n",
+    "from google.colab import files\n",
+    "\n",
+    "files.download(\"training/plots/reward_curve.png\")\n",
+    "files.download(\"training/plots/accuracy_curve.png\")\n",
+    "files.download(\"training/plots/improvement_chart.png\")\n",
+    "files.download(\"training/plots/metrics.json\")\n",
+    "\n",
+    "print(\"Downloaded. Now commit these files to: training/plots/ in your repo.\")"
+   ]
+  }
+ ]
+}

training/trajectory_optimizer.py

@@ -0,0 +1,506 @@
+"""
+Reward-Guided Trajectory Optimization Loop
+==========================================
+Optimizes agent behavior across episodes by accumulating high-reward
+trajectories as few-shot examples. Uses environment reward signal to
+drive improvement — no weight updates required.
+
+This implements a policy improvement loop where:
+  - reward_signal → trajectory_selection → context_construction → improved_policy
+"""
+
+import json
+import os
+import time
+import requests
+from dataclasses import dataclass, field
+from typing import Optional
+from openai import OpenAI
+
+# ── Configuration ────────────────────────────────────────────────────────────
+
+ENV_BASE_URL = os.getenv("ENV_BASE_URL", "http://localhost:7860")
+HF_TOKEN = os.getenv("HF_TOKEN", "")
+MODEL = "Qwen/Qwen2.5-72B-Instruct"
+TEMPERATURE = 0.3
+MAX_TOKENS = 1024
+
+# Training hyperparameters
+NUM_EPISODES_PER_TASK = 8        # Episodes to run per task
+TOP_K_TRAJECTORIES = 3           # Max few-shot examples to keep
+MIN_REWARD_THRESHOLD = 0.3       # Minimum reward to store trajectory
+TASKS = ["data_access", "resource_access", "transaction_approval"]
+
+# ── Data Structures ──────────────────────────────────────────────────────────
+
+@dataclass
+class Step:
+    step_number: int
+    action_type: str
+    action_content: str
+    reward: float
+    accuracy: float
+    feedback: str
+    clarification_response: Optional[str] = None
+
+@dataclass
+class Trajectory:
+    task_name: str
+    episode_id: int
+    steps: list[Step] = field(default_factory=list)
+    total_reward: float = 0.0
+    final_accuracy: float = 0.0
+    success: bool = False
+
+    def to_few_shot_string(self) -> str:
+        """Convert trajectory to a few-shot example string for prompting."""
+        lines = [
+            f"=== Example Episode (reward={self.total_reward:.2f}, accuracy={self.final_accuracy:.2f}) ===",
+        ]
+        for s in self.steps:
+            lines.append(f"Step {s.step_number}: action={s.action_type}")
+            lines.append(f"  Content: {s.action_content[:200]}")
+            lines.append(f"  Result: accuracy={s.accuracy:.2f}, reward={s.reward:.2f}")
+            if s.feedback:
+                lines.append(f"  Feedback: {s.feedback[:150]}")
+        return "\n".join(lines)
+
+# ── Environment Client ────────────────────────────────────────────────────────
+
+class EnvClient:
+    def __init__(self, base_url: str):
+        self.base_url = base_url.rstrip("/")
+        self.session = requests.Session()
+
+    def reset(self, task_name: str) -> dict:
+        r = self.session.post(f"{self.base_url}/reset", json={"task_name": task_name})
+        r.raise_for_status()
+        return r.json()
+
+    def step(self, action_type: str, content: str) -> dict:
+        r = self.session.post(f"{self.base_url}/step", json={
+            "action_type": action_type,
+            "content": content
+        })
+        r.raise_for_status()
+        return r.json()
+
+    def health(self) -> bool:
+        try:
+            r = self.session.get(f"{self.base_url}/health", timeout=5)
+            return r.status_code == 200
+        except Exception:
+            return False
+
+# ── LLM Agent ────────────────────────────────────────────────────────────────
+
+class Agent:
+    def __init__(self, hf_token: str):
+        self.client = OpenAI(
+            base_url="https://router.huggingface.co/v1",
+            api_key=hf_token
+        )
+
+    def get_action(
+        self,
+        observation: dict,
+        step_number: int,
+        episode_history: list[str],
+        few_shot_examples: list[Trajectory]
+    ) -> tuple[str, str]:
+        """
+        Returns (action_type, content_json_string).
+        action_type: one of ask_clarification | propose_rules | refine_rules
+        content: JSON string appropriate for that action
+        """
+        system_prompt = self._build_system_prompt(few_shot_examples)
+        user_prompt = self._build_user_prompt(observation, step_number, episode_history)
+
+        try:
+            response = self.client.chat.completions.create(
+                model=MODEL,
+                messages=[
+                    {"role": "system", "content": system_prompt},
+                    {"role": "user", "content": user_prompt}
+                ],
+                temperature=TEMPERATURE,
+                max_tokens=MAX_TOKENS
+            )
+            raw = response.choices[0].message.content.strip()
+            return self._parse_response(raw, observation)
+        except Exception as e:
+            print(f"    [LLM ERROR] {e}")
+            return "propose_rules", json.dumps({"rules": [], "default": "DENY"})
+
+    def _build_system_prompt(self, few_shot_examples: list[Trajectory]) -> str:
+        base = """You are a policy-to-logic agent. Your job is to convert natural language policies into executable rules.
+
+AVAILABLE ACTIONS:
+1. ask_clarification: {"type": "clarification", "question": "your question"}
+2. propose_rules: {"rules": [...], "default": "DECISION"}
+3. refine_rules: {"rules": [...], "default": "DECISION"}
+
+DSL FORMAT for rules:
+{
+  "rules": [
+    {
+      "if": [
+        {"field": "FIELD_NAME", "op": "OPERATOR", "value": VALUE}
+      ],
+      "then": "DECISION"
+    }
+  ],
+  "default": "FALLBACK_DECISION"
+}
+
+Operators: >, <, >=, <=, ==, !=
+Rules execute top-to-bottom. First match wins. Default applies if no rule matches.
+
+STRATEGY:
+- Step 1: Ask 1-2 targeted clarification questions about ambiguous terms
+- Step 2: Propose initial rules based on policy + clarifications  
+- Step 3+: Refine rules based on failure feedback
+
+OUTPUT FORMAT: Respond ONLY with valid JSON. No markdown. No explanation.
+{"action_type": "propose_rules", "content": "{...escaped json string...}"}
+"""
+        if few_shot_examples:
+            base += "\n\nLEARNED FROM PREVIOUS EPISODES (high-reward strategies):\n"
+            for traj in few_shot_examples[-TOP_K_TRAJECTORIES:]:
+                base += "\n" + traj.to_few_shot_string() + "\n"
+        return base
+
+    def _build_user_prompt(self, obs: dict, step: int, history: list[str]) -> str:
+        lines = [
+            f"TASK: {obs.get('task_name', 'unknown')}",
+            f"STEP: {step} of {obs.get('max_steps', 7)}",
+            f"\nPOLICY:\n{obs.get('policy_text', '')}",
+        ]
+        if obs.get("clarification_response"):
+            lines.append(f"\nLAST CLARIFICATION ANSWER:\n{obs['clarification_response']}")
+        if obs.get("test_results"):
+            tr = obs["test_results"]
+            lines.append(f"\nTEST RESULTS: {tr.get('passed', 0)}/{tr.get('total', 0)} passed (accuracy={obs.get('current_accuracy', 0):.2f})")
+            if tr.get("sample_failures"):
+                lines.append("SAMPLE FAILURES:")
+                for f in tr["sample_failures"][:3]:
+                    lines.append(f"  - {f}")
+        if obs.get("feedback"):
+            lines.append(f"\nFEEDBACK: {obs['feedback']}")
+        if history:
+            lines.append(f"\nACTION HISTORY (last 3):\n" + "\n".join(history[-3:]))
+        lines.append(f"\nAVAILABLE ACTIONS: {obs.get('available_actions', [])}")
+        lines.append("\nRespond with JSON only: {\"action_type\": \"...\", \"content\": \"...\"}")
+        return "\n".join(lines)
+
+    def _parse_response(self, raw: str, obs: dict) -> tuple[str, str]:
+        # Strip markdown code fences if present
+        if "```" in raw:
+            raw = raw.split("```")[1]
+            if raw.startswith("json"):
+                raw = raw[4:]
+        raw = raw.strip()
+
+        try:
+            parsed = json.loads(raw)
+            action_type = parsed.get("action_type", "propose_rules")
+            content = parsed.get("content", "{}")
+
+            # Validate action_type
+            valid_actions = obs.get("available_actions", ["propose_rules", "ask_clarification"])
+            if action_type not in valid_actions:
+                action_type = "propose_rules" if "propose_rules" in valid_actions else valid_actions[0]
+
+            # Ensure content is a string
+            if isinstance(content, dict):
+                content = json.dumps(content)
+            return action_type, content
+        except Exception:
+            return "propose_rules", json.dumps({"rules": [], "default": "DENY"})
+
+# ── Trajectory Bank ───────────────────────────────────────────────────────────
+
+class TrajectoryBank:
+    """Stores and retrieves high-reward trajectories per task."""
+
+    def __init__(self):
+        self.bank: dict[str, list[Trajectory]] = {task: [] for task in TASKS}
+
+    def store(self, trajectory: Trajectory):
+        if trajectory.total_reward >= MIN_REWARD_THRESHOLD:
+            self.bank[trajectory.task_name].append(trajectory)
+            # Keep only top-K by reward
+            self.bank[trajectory.task_name].sort(key=lambda t: t.total_reward, reverse=True)
+            self.bank[trajectory.task_name] = self.bank[trajectory.task_name][:TOP_K_TRAJECTORIES]
+
+    def get_examples(self, task_name: str) -> list[Trajectory]:
+        return self.bank.get(task_name, [])
+
+    def summary(self) -> dict:
+        return {
+            task: {
+                "stored": len(trajs),
+                "best_reward": max((t.total_reward for t in trajs), default=0),
+                "best_accuracy": max((t.final_accuracy for t in trajs), default=0)
+            }
+            for task, trajs in self.bank.items()
+        }
+
+# ── Training Loop ────────────��────────────────────────────────────────────────
+
+class TrainingLoop:
+    def __init__(self, env_url: str, hf_token: str):
+        self.env = EnvClient(env_url)
+        self.agent = Agent(hf_token)
+        self.bank = TrajectoryBank()
+        self.metrics = []  # List of {episode, task, reward, accuracy, success}
+
+    def run_episode(self, task_name: str, episode_id: int) -> Trajectory:
+        """Run a single episode and return the trajectory."""
+        few_shots = self.bank.get_examples(task_name)
+        trajectory = Trajectory(task_name=task_name, episode_id=episode_id)
+
+        # Reset environment
+        result = self.env.reset(task_name)
+        obs = result.get("observation", {})
+        done = result.get("done", False)
+        history = []
+
+        print(f"  [Episode {episode_id}] task={task_name} few_shots={len(few_shots)}")
+
+        step_num = 0
+        while not done and step_num < obs.get("max_steps", 7):
+            step_num += 1
+
+            # Get action from agent
+            action_type, content = self.agent.get_action(
+                observation=obs,
+                step_number=step_num,
+                episode_history=history,
+                few_shot_examples=few_shots
+            )
+
+            # Execute action
+            result = self.env.step(action_type, content)
+            reward = result.get("reward", 0.0)
+            done = result.get("done", False)
+            obs = result.get("observation", {})
+            info = result.get("info", {})
+
+            # Record step
+            step = Step(
+                step_number=step_num,
+                action_type=action_type,
+                action_content=content[:300],
+                reward=reward,
+                accuracy=obs.get("current_accuracy", 0.0),
+                feedback=obs.get("feedback", "") or "",
+                clarification_response=obs.get("clarification_response")
+            )
+            trajectory.steps.append(step)
+            trajectory.total_reward += reward
+
+            # Update history
+            history.append(f"Step {step_num}: {action_type} → reward={reward:.2f} acc={step.accuracy:.2f}")
+
+            print(f"    step={step_num} action={action_type} reward={reward:.3f} acc={step.accuracy:.2f}")
+
+            if done:
+                episode_score = info.get("episode_score", obs.get("current_accuracy", 0.0))
+                trajectory.final_accuracy = episode_score
+                trajectory.success = obs.get("current_accuracy", 0.0) >= 0.9
+                break
+
+        if not trajectory.steps:
+            trajectory.final_accuracy = 0.0
+
+        return trajectory
+
+    def run(self):
+        """Run full training loop across all tasks."""
+        print("=" * 60)
+        print("REWARD-GUIDED TRAJECTORY OPTIMIZATION")
+        print(f"Tasks: {TASKS}")
+        print(f"Episodes per task: {NUM_EPISODES_PER_TASK}")
+        print(f"Top-K trajectories: {TOP_K_TRAJECTORIES}")
+        print("=" * 60)
+
+        # Health check
+        if not self.env.health():
+            raise RuntimeError(f"Environment not reachable at {ENV_BASE_URL}")
+        print(f"Environment: OK ({ENV_BASE_URL})\n")
+
+        global_episode = 0
+
+        for task in TASKS:
+            print(f"\n{'─'*40}")
+            print(f"TASK: {task}")
+            print(f"{'─'*40}")
+
+            task_rewards = []
+            task_accuracies = []
+
+            for ep in range(1, NUM_EPISODES_PER_TASK + 1):
+                global_episode += 1
+                trajectory = self.run_episode(task, ep)
+
+                # Store in bank
+                self.bank.store(trajectory)
+
+                # Record metrics
+                self.metrics.append({
+                    "global_episode": global_episode,
+                    "task": task,
+                    "episode_in_task": ep,
+                    "total_reward": trajectory.total_reward,
+                    "final_accuracy": trajectory.final_accuracy,
+                    "success": trajectory.success,
+                    "num_steps": len(trajectory.steps),
+                    "few_shots_used": len(self.bank.get_examples(task)) - (1 if trajectory.total_reward >= MIN_REWARD_THRESHOLD else 0)
+                })
+
+                task_rewards.append(trajectory.total_reward)
+                task_accuracies.append(trajectory.final_accuracy)
+
+                print(f"  → Episode {ep} complete: reward={trajectory.total_reward:.3f} accuracy={trajectory.final_accuracy:.2f} success={trajectory.success}")
+                time.sleep(0.5)  # Rate limiting
+
+            print(f"\n  Task summary:")
+            print(f"    First episode reward: {task_rewards[0]:.3f}")
+            print(f"    Last episode reward:  {task_rewards[-1]:.3f}")
+            print(f"    Improvement: {task_rewards[-1] - task_rewards[0]:+.3f}")
+
+        print("\n" + "=" * 60)
+        print("TRAINING COMPLETE")
+        print(f"Bank summary: {self.bank.summary()}")
+        print("=" * 60)
+
+        return self.metrics
+
+# ── Plot Generation ───────────────────────────────────────────────────────────
+
+def save_plots(metrics: list[dict]):
+    """
+    Save reward curve and accuracy curve as PNG files.
+    These are REQUIRED for hackathon submission — must be committed to repo.
+    """
+    try:
+        import matplotlib
+        matplotlib.use("Agg")  # Non-interactive backend
+        import matplotlib.pyplot as plt
+        import numpy as np
+    except ImportError:
+        print("matplotlib not installed. Run: pip install matplotlib")
+        return
+
+    os.makedirs("training/plots", exist_ok=True)
+
+    episodes = [m["global_episode"] for m in metrics]
+    rewards = [m["total_reward"] for m in metrics]
+    accuracies = [m["final_accuracy"] for m in metrics]
+    tasks = [m["task"] for m in metrics]
+
+    colors = {
+        "data_access": "#2196F3",
+        "resource_access": "#FF9800",
+        "transaction_approval": "#4CAF50"
+    }
+
+    # ── Plot 1: Reward Curve ──────────────────────────────────────────────────
+    fig, ax = plt.subplots(figsize=(10, 5))
+
+    for task in TASKS:
+        task_eps = [m["global_episode"] for m in metrics if m["task"] == task]
+        task_rews = [m["total_reward"] for m in metrics if m["task"] == task]
+        ax.plot(task_eps, task_rews, marker="o", label=task,
+                color=colors.get(task, "gray"), linewidth=2, markersize=5)
+
+    # Trend line
+    z = np.polyfit(episodes, rewards, 1)
+    p = np.poly1d(z)
+    ax.plot(episodes, p(episodes), "--", color="red", alpha=0.5, linewidth=1.5, label="overall trend")
+
+    ax.set_xlabel("Episode")
+    ax.set_ylabel("Total Reward")
+    ax.set_title("Reward Curve — Reward-Guided Trajectory Optimization")
+    ax.legend()
+    ax.grid(True, alpha=0.3)
+    ax.set_ylim(bottom=0)
+
+    plt.tight_layout()
+    plt.savefig("training/plots/reward_curve.png", dpi=150, bbox_inches="tight")
+    plt.close()
+    print("Saved: training/plots/reward_curve.png")
+
+    # ── Plot 2: Accuracy Curve ────────────────────────────────────────────────
+    fig, ax = plt.subplots(figsize=(10, 5))
+
+    for task in TASKS:
+        task_eps = [m["global_episode"] for m in metrics if m["task"] == task]
+        task_accs = [m["final_accuracy"] for m in metrics if m["task"] == task]
+        ax.plot(task_eps, task_accs, marker="s", label=task,
+                color=colors.get(task, "gray"), linewidth=2, markersize=5)
+
+    ax.axhline(y=0.9, color="red", linestyle="--", alpha=0.7, label="success threshold (0.9)")
+
+    ax.set_xlabel("Episode")
+    ax.set_ylabel("Final Accuracy")
+    ax.set_title("Accuracy Curve — Policy-to-Logic Agent")
+    ax.legend()
+    ax.grid(True, alpha=0.3)
+    ax.set_ylim(0, 1.05)
+
+    plt.tight_layout()
+    plt.savefig("training/plots/accuracy_curve.png", dpi=150, bbox_inches="tight")
+    plt.close()
+    print("Saved: training/plots/accuracy_curve.png")
+
+    # ── Plot 3: Per-Task Improvement Bar Chart ────────────────────────────────
+    fig, ax = plt.subplots(figsize=(8, 5))
+
+    task_names = []
+    improvements = []
+
+    for task in TASKS:
+        task_accs = [m["final_accuracy"] for m in metrics if m["task"] == task]
+        if len(task_accs) >= 2:
+            first = task_accs[0]
+            last = task_accs[-1]
+            task_names.append(task.replace("_", "\n"))
+            improvements.append(last - first)
+
+    bars = ax.bar(task_names, improvements,
+                  color=["#2196F3", "#FF9800", "#4CAF50"][:len(task_names)],
+                  edgecolor="white", linewidth=1.5)
+
+    ax.axhline(y=0, color="black", linewidth=0.8)
+    ax.set_ylabel("Accuracy Improvement (last - first episode)")
+    ax.set_title("Per-Task Improvement from Trajectory Accumulation")
+    ax.grid(True, axis="y", alpha=0.3)
+
+    for bar, val in zip(bars, improvements):
+        ax.text(bar.get_x() + bar.get_width() / 2, bar.get_height() + 0.01,
+                f"{val:+.2f}", ha="center", va="bottom", fontweight="bold")
+
+    plt.tight_layout()
+    plt.savefig("training/plots/improvement_chart.png", dpi=150, bbox_inches="tight")
+    plt.close()
+    print("Saved: training/plots/improvement_chart.png")
+
+    # ── Save raw metrics as JSON ──────────────────────────────────────────────
+    with open("training/plots/metrics.json", "w") as f:
+        json.dump(metrics, f, indent=2)
+    print("Saved: training/plots/metrics.json")
+
+# ── Entry Point ─────────────────────────────────���─────────────────────────────
+
+if __name__ == "__main__":
+    hf_token = os.getenv("HF_TOKEN", "")
+    if not hf_token:
+        raise ValueError("HF_TOKEN environment variable not set")
+
+    loop = TrainingLoop(ENV_BASE_URL, hf_token)
+    metrics = loop.run()
+    save_plots(metrics)
+
+    print("\nNext step: commit training/plots/*.png to repo for submission.")