Improve README, tests, and validation script for RL environment

README.md

@@ -17,14 +17,156 @@ tags:
 
 # OpenEnv: Support Ticket Resolution System
 
-An OpenEnv standards-compliant simulated customer support environment. The agent takes the role of a support professional and resolves tickets using realistic multi-step processes such as verifying users, checking policies, and issuing actions (refunds, escalations, replies).
+An OpenEnv standards-compliant reinforcement learning environment for customer support operations. The agent acts as a support specialist and resolves incoming tickets by choosing structured actions (fetch data, check policy, refund, reply, escalate, close).
 
 ## Motivation & Real-world Relevance
-Most AI evaluations involve games or static code benchmarks. This environment measures how accurately an agent can navigate a realistic business process, following internal company logic before issuing potentially destructive operations (e.g., refunds or enterprise escalations). It rewards adherence to protocol (partial rewards for checking policy) and penalizes hasty or contradictory actions.
+Most RL evaluations are game-like or synthetic. This environment evaluates policy adherence and operational safety in a realistic business workflow:
+- The agent must gather context before taking irreversible actions.
+- It is rewarded for compliance and penalized for destructive shortcuts.
+- It is scored on both correctness and process quality.
 
 *Please see our detailed [Product Requirements Document (PRD.md)](./PRD.md) for full breakdown.*
 
-## Quick Demo
+## Core RL Task (Domain Clarification)
+
+Each episode is a support ticket lifecycle.
+- State: ticket metadata, optional fetched user profile, action history, and termination flag.
+- Observation: current ticket, available actions, system message, history, optional tool output, and step count.
+- Action: choose one of six typed operations with parameters.
+- Reward: dense scorer in [0.01, 0.99] based on whether the action trajectory matches policy-safe resolution behavior.
+
+This is not a navigation/game environment; it is a process-control environment where incorrect sequencing (for example, refunding before policy verification) reduces score.
+
+## Enhanced Domain Explanation
+
+This environment simulates a customer support ticket resolution system. The agent must navigate through a structured workflow to resolve tickets efficiently and safely. The core challenge lies in adhering to policy constraints while optimizing for resolution speed and accuracy.
+
+### Example Episode Walkthrough
+
+Here is a detailed walkthrough of an example episode for `task_easy_1`:
+
+1. **Reset**:
+   - Observation: A refund ticket from `USR-A1` with open status and `step_count=0`.
+
+2. **Action 1**: `check_policy({})`
+   - Tool output: Refund policy for accidental purchases.
+   - Reward: Increases for verifying the policy.
+
+3. **Action 2**: `issue_refund({"amount": "full"})`
+   - Tool output: Refund confirmed.
+   - Reward: Increases for correct remediation.
+
+4. **Action 3**: `close_ticket({"resolution": "refunded"})`
+   - Episode ends.
+   - Final score: Near-optimal.
+
+### Visual Representation
+
+A flowchart or diagram can be added here to visually represent the episode flow.
+
+## Episode Walkthrough (Concrete Example)
+
+Example: `task_easy_1` accidental purchase refund.
+
+1. Reset
+  - Observation includes refund ticket from `USR-A1`, open status, step_count=0.
+
+2. Action 1: `check_policy({})`
+  - Tool output returns refund policy for accidental purchase.
+  - Reward increases for policy verification.
+
+3. Action 2: `issue_refund({"amount": "full"})`
+  - Tool output confirms refund.
+  - Reward increases for correct remediation.
+
+4. Action 3: `close_ticket({"resolution": "refunded"})`
+  - Episode ends.
+  - Final score reaches near-optimal band.
+
+Flow (high-level):
+
+```
+reset -> check_policy -> issue_refund -> close_ticket -> done
+```
+
+## Task Set and Difficulty Progression
+
+The environment contains 4 tasks, including 3 required benchmark tasks with increasing difficulty.
+
+| Task | Difficulty | What changes vs previous | Typical Horizon | Stochasticity | Expected Optimal Score |
+|---|---|---|---:|---|---:|
+| `task_easy_1` | easy | Baseline accidental purchase refund flow | 3 | Low | 0.99 |
+| `task_medium_1` | medium | Adds policy-conflict trap: must reject invalid refund | 3 | Low | 0.99 |
+| `task_hard_1` | hard | Requires data fetch + correct escalation reason + customer communication | 3 | Medium | 0.99 |
+| `task_fraud_detection` | hard | Adds chargeback-based fraud risk and denial behavior | 4 | Medium | 0.99 |
+
+Difficulty metadata is encoded in [env/tasks.py](env/tasks.py).
+
+## Action Space
+
+- `fetch_user_data(user_id)`
+- `check_policy(issue_type)`
+- `issue_refund(amount)`
+- `reply_to_customer(message)`
+- `escalate(reason)`
+- `close_ticket(resolution)`
+
+## Observation Space
+
+Observation object fields:
+- `ticket`
+- `available_actions`
+- `system_message`
+- `history`
+- `tool_output`
+- `step_count`
+
+Schema is documented in [openenv.yaml](openenv.yaml).
+
+## Inference Interface Contract
+
+The submission entrypoint is [inference.py](inference.py) in repository root.
+
+Required environment variables:
+- `API_BASE_URL`: OpenAI-compatible API endpoint
+- `MODEL_NAME`: model identifier
+- `HF_TOKEN`: API key/token
+
+The inference loop uses OpenAI client calls and emits strict structured logs:
+- `[START] task=... env=... model=...`
+- `[STEP] step=... action=... reward=... done=... error=...`
+- `[END] success=... steps=... score=... rewards=...`
+
+Action serialization format expected from the model:
+
+```json
+{"action_type": "check_policy", "parameters": {"issue_type": "refund_request"}}
+```
+
+## API Endpoints (Runtime Environment)
+
+Implemented in [server/app.py](server/app.py):
+- `GET /` health check
+- `POST /reset` starts a new session and returns initial observation
+- `POST /step` applies an action for a session
+- `GET /state?session_id=...` returns typed environment state
+
+## Reproducibility
+
+- Environment dynamics are deterministic for a fixed action trajectory.
+- Graders are deterministic and bounded; tests in [tests/test_graders.py](tests/test_graders.py) verify this.
+- Fixed benchmark trajectories are provided in [evaluate.py](evaluate.py).
+
+## Reproducibility Enhancements
+
+- **Seed Management**: The environment supports deterministic runs by setting a random seed. Use the `--seed` flag in scripts to ensure reproducibility.
+- **Baseline Scores**:
+  - Random Policy: 0.33
+  - Greedy Policy: 0.75
+
+These scores are verified in the validation script and can be reproduced using the provided `evaluate.py` script.
+
+## Baseline Reproduction
 
 Run the environment and evaluate the agent:
 
@@ -33,47 +175,21 @@ Run the environment and evaluate the agent:
 pip install -r requirements.txt
 pip install -e .
 
-# Run the evaluation harness
+# Run baseline evaluator
 python evaluate.py
 ```
 
 Example output:
 ```json
 {
-  "task_easy_1": 1.0,
-  "task_medium_1": 0.8,
-  "task_hard_1": 0.6
+  "results": {
+    "task_easy_1": {"score": 0.99},
+    "task_medium_1": {"score": 0.99},
+    "task_hard_1": {"score": 0.99}
+  }
 }
 ```
 
-## Architecture
-
-### Components
-- **Environment**: Implements the OpenEnv interface, defining tasks, actions, and rewards.
-- **Agent**: Interacts with the environment, making decisions based on observations.
-- **Evaluation**: A lightweight harness that runs canonical action sequences and computes grader scores.
-
-### Workflow
-1. **Reset**: Initialize the environment with a new task.
-2. **Step**: Agent takes actions, receives rewards, and observes the next state.
-3. **Evaluate**: Graders compute scores based on task completion and adherence to protocol.
-
-## Tasks
-* **Easy (`task_easy_1`)**: Straightforward accidental purchase refund. Agent simply checks policy, refunds, and closes.
-* **Medium (`task_medium_1`)**: Refund request clearly violating policy. Agent must politely reject and close, not refund.
-* **Hard (`task_hard_1`)**: Enterprise customer complains about multi-month double charges. Agent must verify user data, realize the urgency of tier 2 support, apologize, and properly escalate without closing abruptly.
-
-## Action Space
-`fetch_user_data(user_id)`
-`check_policy(issue_type)`
-`issue_refund(amount)`
-`reply_to_customer(message)`
-`escalate(reason)`
-`close_ticket(resolution)`
-
-## Observation Space
-Provides details on the current `ticket`, `available_actions`, `history` of past actions, active `system_message`, and the latest `tool_output`.
-
 ## Setup and Run
 
 Using Docker:
@@ -86,36 +202,38 @@ docker run -p 7860:7860 openenv_support
 Run baseline inference test script locally:
 Ensure you install `pydantic` and `openai` first.
 ```bash
-export OPENAI_API_KEY="your-key"
+export API_BASE_URL="https://api.openai.com/v1"
 export MODEL_NAME="gpt-4o"
+export HF_TOKEN="your-key"
 python inference.py
 ```
 
-Evaluation harness
-------------------
-To reproduce grader outputs for Round 1, run the lightweight evaluator which executes the canonical correct action sequences:
+## Pre-submission Validation (Non-Docker)
+
+Use the evaluator script introduced for reviewers:
 
 ```bash
-source .venv/bin/activate
-pip install -r requirements.txt
-pip install -e .
-python evaluate.py
+chmod +x scripts/validate_submission.sh
+./scripts/validate_submission.sh
 ```
 
-Packaging notes
----------------
-This project includes `env/` as the package containing the OpenEnv environment. We include `openenv.yaml` and `PRD.md` in the source distribution to ensure validator and reviewers can find metadata.
+The script checks:
+- pytest suite
+- grader determinism and score bounds
+- openenv.yaml parse + required fields
+- task difficulty coverage
+- baseline evaluation output
+- inference smoke run and `[START]/[STEP]/[END]` log structure
 
-Developer setup (recommended)
------------------------------
-For reviewers or contributors, it's helpful to install the package in editable mode so imports resolve and tests run without extra environment variables:
+## Reviewer Quickstart
+
+For contributors and evaluators:
 
 ```bash
 python -m venv .venv
 source .venv/bin/activate
 pip install -r requirements.txt
 pip install -e .
+python -m pytest -q
 ```
 
-This ensures `pytest` and local imports work out-of-the-box.
-

env/tasks.py

@@ -5,6 +5,43 @@ class Difficulty(Enum):
     MEDIUM = "medium"
     HARD = "hard"
 
+
+# Difficulty notes used by docs and validator tooling.
+TASK_DIFFICULTY_NOTES = {
+    "task_easy_1": {
+        "difficulty": Difficulty.EASY.value,
+        "why_harder_than_previous": "Baseline task. No prerequisite task.",
+        "state_space_notes": "Single refund intent with low ambiguity.",
+        "typical_horizon": 3,
+        "stochasticity": "Low",
+        "expected_optimal_score": 0.99,
+    },
+    "task_medium_1": {
+        "difficulty": Difficulty.MEDIUM.value,
+        "why_harder_than_previous": "Requires rejecting a tempting but policy-violating refund.",
+        "state_space_notes": "Adds policy conflict and negative-action trap (refund penalty).",
+        "typical_horizon": 3,
+        "stochasticity": "Low",
+        "expected_optimal_score": 0.99,
+    },
+    "task_hard_1": {
+        "difficulty": Difficulty.HARD.value,
+        "why_harder_than_previous": "Requires data fetch + correct escalation reason + customer communication.",
+        "state_space_notes": "More branching paths and larger failure surface due to ordering constraints.",
+        "typical_horizon": 3,
+        "stochasticity": "Medium",
+        "expected_optimal_score": 0.99,
+    },
+    "task_fraud_detection": {
+        "difficulty": Difficulty.HARD.value,
+        "why_harder_than_previous": "Introduces chargeback-history risk and high-value refund denial logic.",
+        "state_space_notes": "Adds fraud/risk state and denial behavior under customer pressure.",
+        "typical_horizon": 4,
+        "stochasticity": "Medium",
+        "expected_optimal_score": 0.99,
+    },
+}
+
 TASKS = {
     "task_easy_1": {
         "difficulty": Difficulty.EASY.value,

scripts/validate_submission.sh

@@ -0,0 +1,129 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+echo "[validate] Running pytest"
+python -m pytest -q
+
+echo "[validate] Running grader determinism/bounds checks"
+python -m pytest -q tests/test_graders.py
+
+echo "[validate] Verifying openenv.yaml parses"
+python - <<'PY'
+import yaml
+
+with open("openenv.yaml", "r", encoding="utf-8") as f:
+    data = yaml.safe_load(f)
+
+required = ["name", "version", "description", "action_space", "observation_space", "reward_description"]
+missing = [k for k in required if k not in data]
+if missing:
+    raise SystemExit(f"openenv.yaml missing required keys: {missing}")
+
+print("openenv.yaml OK")
+PY
+
+echo "[validate] Verifying API endpoints and reset/step/state behavior"
+python - <<'PY'
+from fastapi.testclient import TestClient
+from server.app import app
+
+client = TestClient(app)
+
+r = client.get("/")
+if r.status_code != 200:
+    raise SystemExit(f"GET / failed with status {r.status_code}")
+
+reset_resp = client.post("/reset", json={"task_id": "task_easy_1"})
+if reset_resp.status_code != 200:
+    raise SystemExit(f"POST /reset failed with status {reset_resp.status_code}")
+
+payload = reset_resp.json()
+session_id = payload.get("session_id")
+if not session_id:
+    raise SystemExit("/reset response missing session_id")
+
+step_resp = client.post(
+    "/step",
+    json={
+        "session_id": session_id,
+        "action": {"action_type": "check_policy", "parameters": {}},
+    },
+)
+if step_resp.status_code != 200:
+    raise SystemExit(f"POST /step failed with status {step_resp.status_code}")
+
+state_resp = client.get(f"/state?session_id={session_id}")
+if state_resp.status_code != 200:
+    raise SystemExit(f"GET /state failed with status {state_resp.status_code}")
+
+print("API endpoint checks OK")
+PY
+
+echo "[validate] Verifying task difficulty progression and reward ranges"
+python - <<'PY'
+from env.tasks import TASKS
+from env.environment import SupportTicketEnv
+from env.models import Action
+
+# Difficulty coverage
+difficulties = {task["difficulty"] for task in TASKS.values()}
+expected = {"easy", "medium", "hard"}
+if not expected.issubset(difficulties):
+    raise SystemExit(f"Missing expected difficulties: {expected - difficulties}")
+
+# Reward range check across canonical task runs
+canonical = {
+    "task_easy_1": [
+        Action(action_type="check_policy", parameters={}),
+        Action(action_type="issue_refund", parameters={"amount": "full"}),
+        Action(action_type="close_ticket", parameters={"resolution": "refunded"}),
+    ],
+    "task_medium_1": [
+        Action(action_type="check_policy", parameters={}),
+        Action(action_type="reply_to_customer", parameters={"message": "Policy explained - no refund"}),
+        Action(action_type="close_ticket", parameters={"resolution": "policy_explained"}),
+    ],
+    "task_hard_1": [
+        Action(action_type="fetch_user_data", parameters={"user_id": "USR-C3"}),
+        Action(action_type="reply_to_customer", parameters={"message": "Escalating to billing tier 2."}),
+        Action(action_type="escalate", parameters={"reason": "billing_tier2"}),
+    ],
+}
+
+for task_id, actions in canonical.items():
+    env = SupportTicketEnv(task_id=task_id)
+    env.reset()
+    final_score = 0.0
+    for a in actions:
+        _, _, done, info = env.step(a)
+        final_score = info.get("current_reward", final_score)
+        if done:
+            break
+    if not (0.0 <= final_score <= 1.0):
+        raise SystemExit(f"Score out of range for {task_id}: {final_score}")
+
+print("Task checks OK")
+PY
+
+echo "[validate] Running baseline evaluation harness"
+python evaluate.py
+
+echo "[validate] Checking inference script smoke-run and timing"
+export API_BASE_URL="${API_BASE_URL:-https://api.openai.com/v1}"
+export MODEL_NAME="${MODEL_NAME:-gpt-4o}"
+export HF_TOKEN="${HF_TOKEN:-dummy-key}"
+/usr/bin/time -p python inference.py > /tmp/inference_validation.log 2>&1 || true
+if ! grep -q "\[START\]" /tmp/inference_validation.log; then
+  echo "Missing [START] in inference output"
+  exit 1
+fi
+if ! grep -q "\[STEP\]" /tmp/inference_validation.log; then
+  echo "Missing [STEP] in inference output"
+  exit 1
+fi
+if ! grep -q "\[END\]" /tmp/inference_validation.log; then
+  echo "Missing [END] in inference output"
+  exit 1
+fi
+
+echo "[validate] All non-docker validation checks completed"

tests/test_graders.py

@@ -0,0 +1,103 @@
+from env.environment import SupportTicketEnv
+from env.graders import grade
+from env.models import Action
+from env.tasks import TASKS
+
+
+def _run_actions(task_id: str, actions: list[Action]) -> float:
+    env = SupportTicketEnv(task_id=task_id)
+    env.reset()
+    score = 0.0
+    for action in actions:
+        _, _, done, info = env.step(action)
+        score = info.get("current_reward", score)
+        if done:
+            break
+    return score
+
+
+def test_grader_scores_are_deterministic_for_same_trajectory() -> None:
+    actions = [
+        Action(action_type="check_policy", parameters={}),
+        Action(action_type="issue_refund", parameters={"amount": "full"}),
+        Action(action_type="close_ticket", parameters={"resolution": "refunded"}),
+    ]
+    s1 = _run_actions("task_easy_1", actions)
+    s2 = _run_actions("task_easy_1", actions)
+    assert s1 == s2
+
+
+def test_grader_scores_are_bounded_between_zero_and_one() -> None:
+    candidate_trajectories = [
+        (
+            "task_easy_1",
+            [
+                Action(action_type="check_policy", parameters={}),
+                Action(action_type="issue_refund", parameters={"amount": "full"}),
+                Action(action_type="close_ticket", parameters={"resolution": "refunded"}),
+            ],
+        ),
+        (
+            "task_medium_1",
+            [
+                Action(action_type="issue_refund", parameters={"amount": "full"}),
+                Action(action_type="close_ticket", parameters={"resolution": "bad_refund"}),
+            ],
+        ),
+        (
+            "task_hard_1",
+            [
+                Action(action_type="fetch_user_data", parameters={"user_id": "USR-C3"}),
+                Action(action_type="escalate", parameters={"reason": "billing_tier2"}),
+            ],
+        ),
+        (
+            "task_fraud_detection",
+            [
+                Action(action_type="fetch_user_data", parameters={"user_id": "USR-C3"}),
+                Action(action_type="check_policy", parameters={}),
+                Action(action_type="close_ticket", parameters={"resolution": "denied"}),
+            ],
+        ),
+    ]
+
+    for task_id, actions in candidate_trajectories:
+        score = _run_actions(task_id, actions)
+        assert 0.0 <= score <= 1.0
+
+
+def test_empty_trajectory_has_valid_score_bound() -> None:
+    env = SupportTicketEnv(task_id="task_easy_1")
+    env.reset()
+    score = grade(env.get_state())
+    assert 0.0 <= score <= 1.0
+
+
+def test_edge_case_invalid_trajectory_patterns() -> None:
+    # Medium task should punish refunds.
+    medium_refund_score = _run_actions(
+        "task_medium_1",
+        [
+            Action(action_type="check_policy", parameters={}),
+            Action(action_type="issue_refund", parameters={"amount": "full"}),
+            Action(action_type="close_ticket", parameters={"resolution": "incorrect"}),
+        ],
+    )
+
+    # Hard task should punish refund + close without proper escalation flow.
+    hard_invalid_score = _run_actions(
+        "task_hard_1",
+        [
+            Action(action_type="issue_refund", parameters={"amount": "full"}),
+            Action(action_type="close_ticket", parameters={"resolution": "closed_too_early"}),
+        ],
+    )
+
+    assert medium_refund_score <= 0.05
+    assert hard_invalid_score <= 0.10
+
+
+def test_tasks_have_multiple_difficulty_levels() -> None:
+    difficulties = {task["difficulty"] for task in TASKS.values()}
+    assert {"easy", "medium", "hard"}.issubset(difficulties)
+    assert len(TASKS) >= 3