hf done

.dockerignore

@@ -0,0 +1,24 @@
+# Secrets — never bake into image
+.env
+.env.*
+!.env.example
+
+# Git / IDE
+.git
+.gitignore
+.cursor
+
+# Python noise
+__pycache__
+*.py[cod]
+*$py.class
+.pytest_cache
+.mypy_cache
+.ruff_cache
+.venv
+venv
+*.egg-info
+
+# Local / OS
+Thumbs.db
+.DS_Store

.env.example

@@ -0,0 +1,29 @@
+# Copy this file to `.env` and fill in your values.
+# Never commit `.env` — it is listed in `.gitignore`.
+
+# --- Hackathon / inference.py (OpenAI-compatible client) ---
+# Primary key (Hugging Face token OR OpenAI key, depending on API_BASE_URL)
+HF_TOKEN=
+
+# If you use OpenAI directly instead of HF router:
+# OPENAI_API_KEY=
+
+# Generic fallback used by some samples:
+# API_KEY=
+
+# LLM endpoint (OpenAI: https://api.openai.com/v1 | HF router: https://router.huggingface.co/v1)
+API_BASE_URL=https://api.openai.com/v1
+
+# Model id for chat completions (must match your provider)
+MODEL_NAME=gpt-4o-mini
+
+# --- Optional inference tuning ---
+# OPENAI_SEED=42
+# MAX_STEPS=24
+# ENV_CONTAINER_START=false
+# IMAGE_NAME=sql-agent-env:latest
+# SQL_AGENT_TASK=sql_analyst_episode
+# SQL_AGENT_BENCHMARK=sql_agent_openenv
+
+# --- Hugging Face Space / server ---
+# PORT=7860

.gitignore

@@ -0,0 +1,11 @@
+.env
+.env.local
+.env.*.local
+__pycache__/
+*.pyc
+.Python
+*.py[cod]
+.venv/
+venv/
+*.egg-info/
+.pytest_cache/

Dockerfile

@@ -14,12 +14,14 @@ COPY . /app/
 
 # Environment configurations setup
 ENV PYTHONPATH="/app"
-
-# Health check
-HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
-    CMD curl -f http://localhost:7860/health || exit 1
+# Hugging Face Spaces may set PORT; must match README `app_port` (7860).
+ENV PORT=7860
 
 EXPOSE 7860
 
-# Run server with updated root OOP Path
-CMD ["uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "7860"]
+# Health check uses same default PORT as CMD (Spaces usually leave PORT=7860)
+HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=5 \
+    CMD sh -c 'curl -fsS "http://127.0.0.1:${PORT}/health" || exit 1'
+
+# OpenEnv app: FastAPI ASGI at server.app:app (see openenv.yaml)
+CMD ["sh", "-c", "exec uvicorn server.app:app --host 0.0.0.0 --port ${PORT}"]

README.md

@@ -67,31 +67,59 @@ Episode ends when all tasks are solved or max attempts for a task are exhausted.
    ```bash
    pip install -r requirements.txt
    ```
-2. Set API key for baseline:
+2. Set **hackathon-required** variables (OpenAI-compatible client):
    ```bash
-   export OPENAI_API_KEY="your-key"
+   export HF_TOKEN="your-api-key"
+   export API_BASE_URL="https://api.openai.com/v1"
+   export MODEL_NAME="gpt-4o-mini"
    ```
-3. Run baseline:
+   `HF_TOKEN` is preferred; `OPENAI_API_KEY` or `API_KEY` are accepted as fallbacks.
+3. Run baseline (root `inference.py`):
    ```bash
    python inference.py
    ```
 
-The baseline uses the OpenAI API client with deterministic settings:
-- `temperature=0.0`
-- fixed seed (`OPENAI_SEED`, default `42`)
-- fixed task order and deterministic grader
+The baseline uses the **OpenAI Python client** (`openai.OpenAI`) with `base_url=API_BASE_URL` and `api_key=HF_TOKEN` (or fallback). For reproducibility on the official OpenAI API, `temperature=0.0` and `seed=OPENAI_SEED` (default `42`) are used when `API_BASE_URL` points at OpenAI.
 
-## Baseline Score Reporting
-`inference.py` prints:
-- per-task best score
-- average task score
-- episode return
-- model and seed used
+### Mandatory stdout format (for automated judging)
+`inference.py` prints **only** these structured lines to **stdout** (debug goes to **stderr**):
+
+- `[START] task=<name> env=<benchmark> model=<model>`
+- `[STEP] step=<n> action=<sql> reward=<0.00> done=<true|false> error=<msg|null>`
+- `[END] success=<true|false> steps=<n> score=<0.000> rewards=<r1,r2,...>`
+
+`[END] score` is the mean of best grader scores seen for the three tasks (each in `[0, 1]`). `success` is `true` when the episode ends with `done` and that mean score is ≥ `0.95`.
+
+## Pre-submission validation
+Before submitting, run:
+```bash
+python hf/pre-validation-script.py
+```
+This checks files, the stdout contract in `inference.py`, syntax, and `openenv validate` (if the CLI is installed).
 
-This creates reproducible benchmark records for easy/medium/hard tasks.
+## Baseline Score Reporting
+Structured `[END]` line carries the aggregate score and per-step rewards; use stderr `[DEBUG]` lines only for local troubleshooting.
+
+## Deploy to Hugging Face Spaces (spec checklist)
+1. **Create a Space** → **Docker** template, or link this GitHub repo to a Space.
+2. **README frontmatter** (top of this file) must stay valid YAML:
+   - `sdk: docker`
+   - `app_port: 7860` (must match `openenv.yaml` `port` and the container listen port)
+   - `tags:` includes `openenv`
+3. **Build**: HF runs `docker build` on the repo root; entrypoint is `Dockerfile` `CMD` → Uvicorn on `0.0.0.0:$PORT` (default **7860**).
+4. **Health**: platform probes your app; this repo exposes **`GET /health`** and OpenEnv **`POST /reset`** for automated checks.
+5. **Push** the same commit you validated locally (`openenv validate`, `python hf/pre-validation-script.py`).
+
+Local smoke test (matches CI-style build):
+```bash
+docker build -t sql-agent-env .
+docker run --rm -p 7860:7860 -e PORT=7860 sql-agent-env
+# Then open http://localhost:7860/health and http://localhost:7860/
+```
 
 ## Docker and Hugging Face Space
-- `Dockerfile` is included and starts `uvicorn server.app:app`.
+- `Dockerfile` runs `uvicorn server.app:app` with **`PORT`** from the environment (default **7860**).
+- `.dockerignore` excludes `.env` and build junk so secrets are not copied into the image.
 - Space metadata is configured in this `README.md` frontmatter with `sdk: docker`.
 - Health endpoint: `/health`
 - **Web playground:** open the Space URL (`/`). Click **Connect session** to open a WebSocket to `/ws`, then **Start episode (reset)** and **Run query (step)**. OpenEnv’s HTTP `POST /reset` and `POST /step` each use a fresh environment instance (stateless); the WebSocket session keeps one episode alive for the UI.

hf/pre-validation-script.py

@@ -0,0 +1,142 @@
+#!/usr/bin/env python3
+"""
+Pre-submission checks for the OpenEnv hackathon SQL agent project.
+
+Run from repo root:
+  python hf/pre-validation-script.py
+
+Or:
+  cd hf && python pre-validation-script.py
+
+Checks:
+  - Required files exist (inference.py, Dockerfile, openenv.yaml, README, models, etc.)
+  - inference.py contains mandatory stdout format helpers and tags
+  - Python syntax compiles
+  - openenv validate (if CLI installed)
+
+Does not call external LLM APIs (no HF_TOKEN required for this script).
+"""
+
+from __future__ import annotations
+
+import ast
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parent.parent
+
+
+def ok(msg: str) -> None:
+    print(f"[OK] {msg}", flush=True)
+
+
+def fail(msg: str) -> bool:
+    print(f"[FAIL] {msg}", flush=True)
+    return False
+
+
+def check_files() -> bool:
+    good = True
+    required = [
+        ROOT / "inference.py",
+        ROOT / "Dockerfile",
+        ROOT / "openenv.yaml",
+        ROOT / "README.md",
+        ROOT / "models.py",
+        ROOT / "client.py",
+        ROOT / "server" / "app.py",
+        ROOT / "server" / "environment.py",
+        ROOT / "core" / "tasks.py",
+        ROOT / "core" / "grader.py",
+    ]
+    for p in required:
+        if not p.is_file():
+            good &= fail(f"Missing file: {p.relative_to(ROOT)}")
+    if good:
+        ok("Required project files present")
+    return good
+
+
+def check_inference_stdout_contract() -> bool:
+    path = ROOT / "inference.py"
+    text = path.read_text(encoding="utf-8")
+    tokens = [
+        "def log_start",
+        "def log_step",
+        "def log_end",
+        "[START]",
+        "[STEP]",
+        "[END]",
+        "OpenAI(",
+        "chat.completions.create",
+        "HF_TOKEN",
+        "API_BASE_URL",
+        "MODEL_NAME",
+    ]
+    good = True
+    for t in tokens:
+        if t not in text:
+            good &= fail(f"inference.py missing required fragment: {t!r}")
+    if good:
+        ok("inference.py stdout contract + OpenAI client config markers")
+    return good
+
+
+def check_syntax() -> bool:
+    good = True
+    for py in [
+        ROOT / "inference.py",
+        ROOT / "server" / "app.py",
+        ROOT / "server" / "environment.py",
+        ROOT / "models.py",
+    ]:
+        try:
+            ast.parse(py.read_text(encoding="utf-8"), filename=str(py))
+        except SyntaxError as e:
+            good &= fail(f"Syntax error in {py.relative_to(ROOT)}: {e}")
+    if good:
+        ok("Core Python files parse (syntax)")
+    return good
+
+
+def check_openenv_validate() -> bool:
+    try:
+        r = subprocess.run(
+            ["openenv", "validate"],
+            cwd=str(ROOT),
+            capture_output=True,
+            text=True,
+            timeout=120,
+        )
+    except FileNotFoundError:
+        print(
+            "[WARN] openenv CLI not on PATH — install openenv-core and ensure `openenv` is available",
+            flush=True,
+        )
+        return True
+    sys.stdout.write(r.stdout)
+    sys.stderr.write(r.stderr)
+    if r.returncode != 0:
+        return fail("openenv validate returned non-zero")
+    ok("openenv validate")
+    return True
+
+
+def main() -> int:
+    print("=== Pre-submission validation ===", flush=True)
+    print(f"ROOT: {ROOT}", flush=True)
+    all_ok = True
+    all_ok &= check_files()
+    all_ok &= check_inference_stdout_contract()
+    all_ok &= check_syntax()
+    all_ok &= check_openenv_validate()
+    if all_ok:
+        print("\n=== All automated checks passed ===", flush=True)
+        return 0
+    print("\n=== Fix failures above before submitting ===", flush=True)
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

hf/simple-interface-script.py

@@ -0,0 +1,188 @@
+"""
+Inference Script Example
+===================================
+MANDATORY
+- Before submitting, ensure the following variables are defined in your environment configuration:
+    API_BASE_URL   The API endpoint for the LLM.
+    MODEL_NAME     The model identifier to use for inference.
+    HF_TOKEN       Your Hugging Face / API key.
+    LOCAL_IMAGE_NAME The name of the local image to use for the environment if you are using from_docker_image()
+                     method
+
+- Defaults are set only for API_BASE_URL and MODEL_NAME 
+    (and should reflect your active inference setup):
+    API_BASE_URL = os.getenv("API_BASE_URL", "<your-active-endpoint>")
+    MODEL_NAME = os.getenv("MODEL_NAME", "<your-active-model>")
+    
+- The inference script must be named `inference.py` and placed in the root directory of the project
+- Participants must use OpenAI Client for all LLM calls using above variables
+
+STDOUT FORMAT
+- The script must emit exactly three line types to stdout, in this order:
+
+    [START] task=<task_name> env=<benchmark> model=<model_name>
+    [STEP]  step=<n> action=<action_str> reward=<0.00> done=<true|false> error=<msg|null>
+    [END]   success=<true|false> steps=<n> score=<score> rewards=<r1,r2,...,rn>
+
+  Rules:
+    - One [START] line at episode begin.
+    - One [STEP] line per step, immediately after env.step() returns.
+    - One [END] line after env.close(), always emitted (even on exception).
+    - reward and rewards are formatted to 2 decimal places.
+    - done and success are lowercase booleans: true or false.
+    - error is the raw last_action_error string, or null if none.
+    - All fields on a single line with no newlines within a line.
+    - Each tasks should return score in [0, 1]
+
+  Example:
+    [START] task=click-test env=miniwob model=Qwen3-VL-30B
+    [STEP] step=1 action=click('123') reward=0.00 done=false error=null
+    [STEP] step=2 action=fill('456','text') reward=0.00 done=false error=null
+    [STEP] step=3 action=click('789') reward=1.00 done=true error=null
+    [END] success=true steps=3 score=1.00 rewards=0.00,0.00,1.00
+"""
+
+import asyncio
+import os
+import textwrap
+from typing import List, Optional
+
+from openai import OpenAI
+
+from my_env_v4 import MyEnvV4Action, MyEnvV4Env
+IMAGE_NAME = os.getenv("IMAGE_NAME") # If you are using docker image 
+API_KEY = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
+
+API_BASE_URL = os.getenv("API_BASE_URL") or "https://router.huggingface.co/v1"
+MODEL_NAME = os.getenv("MODEL_NAME") or "Qwen/Qwen2.5-72B-Instruct"
+TASK_NAME = os.getenv("MY_ENV_V4_TASK", "echo")
+BENCHMARK = os.getenv("MY_ENV_V4_BENCHMARK", "my_env_v4")
+MAX_STEPS = 8
+TEMPERATURE = 0.7
+MAX_TOKENS = 150
+SUCCESS_SCORE_THRESHOLD = 0.1  # normalized score in [0, 1]
+
+# Max possible reward: each token contributes 0.1, across all steps
+_MAX_REWARD_PER_STEP = MAX_TOKENS * 0.1
+MAX_TOTAL_REWARD = MAX_STEPS * _MAX_REWARD_PER_STEP
+
+SYSTEM_PROMPT = textwrap.dedent(
+    """
+    You are interacting with a simple echo environment.
+    Each turn you must send a message. The environment will echo it back.
+    Reward is proportional to message length: reward = len(message) * 0.1
+    Your goal is to maximize total reward by sending meaningful, substantive messages.
+    Reply with exactly one message string — no quotes, no prefixes, just the message text.
+    """
+).strip()
+
+
+def log_start(task: str, env: str, model: str) -> None:
+    print(f"[START] task={task} env={env} model={model}", flush=True)
+
+
+def log_step(step: int, action: str, reward: float, done: bool, error: Optional[str]) -> None:
+    error_val = error if error else "null"
+    done_val = str(done).lower()
+    print(
+        f"[STEP] step={step} action={action} reward={reward:.2f} done={done_val} error={error_val}",
+        flush=True,
+    )
+
+
+def log_end(success: bool, steps: int, score: float, rewards: List[float]) -> None:
+    rewards_str = ",".join(f"{r:.2f}" for r in rewards)
+    print(f"[END] success={str(success).lower()} steps={steps} score={score:.3f} rewards={rewards_str}", flush=True)
+
+
+def build_user_prompt(step: int, last_echoed: str, last_reward: float, history: List[str]) -> str:
+    history_block = "\n".join(history[-4:]) if history else "None"
+    return textwrap.dedent(
+        f"""
+        Step: {step}
+        Last echoed message: {last_echoed!r}
+        Last reward: {last_reward:.2f}
+        Previous steps:
+        {history_block}
+        Send your next message.
+        """
+    ).strip()
+
+
+def get_model_message(client: OpenAI, step: int, last_echoed: str, last_reward: float, history: List[str]) -> str:
+    user_prompt = build_user_prompt(step, last_echoed, last_reward, history)
+    try:
+        completion = client.chat.completions.create(
+            model=MODEL_NAME,
+            messages=[
+                {"role": "system", "content": SYSTEM_PROMPT},
+                {"role": "user", "content": user_prompt},
+            ],
+            temperature=TEMPERATURE,
+            max_tokens=MAX_TOKENS,
+            stream=False,
+        )
+        text = (completion.choices[0].message.content or "").strip()
+        return text if text else "hello"
+    except Exception as exc:
+        print(f"[DEBUG] Model request failed: {exc}", flush=True)
+        return "hello"
+
+
+async def main() -> None:
+    client = OpenAI(base_url=API_BASE_URL, api_key=API_KEY)
+
+    env = await MyEnvV4Env.from_docker_image(IMAGE_NAME)
+
+    history: List[str] = []
+    rewards: List[float] = []
+    steps_taken = 0
+    score = 0.0
+    success = False
+
+    log_start(task=TASK_NAME, env=BENCHMARK, model=MODEL_NAME)
+
+    try:
+        result = await env.reset() # OpenENV.reset()
+        last_echoed = result.observation.echoed_message
+        last_reward = 0.0
+
+        for step in range(1, MAX_STEPS + 1):
+            if result.done:
+                break
+
+            message = get_model_message(client, step, last_echoed, last_reward, history)
+
+            result = await env.step(MyEnvV4Action(message=message))
+            obs = result.observation
+
+            reward = result.reward or 0.0
+            done = result.done
+            error = None
+
+            rewards.append(reward)
+            steps_taken = step
+            last_echoed = obs.echoed_message
+            last_reward = reward
+
+            log_step(step=step, action=message, reward=reward, done=done, error=error)
+
+            history.append(f"Step {step}: {message!r} -> reward {reward:+.2f}")
+
+            if done:
+                break
+
+        score = sum(rewards) / MAX_TOTAL_REWARD if MAX_TOTAL_REWARD > 0 else 0.0
+        score = min(max(score, 0.0), 1.0)  # clamp to [0, 1]
+        success = score >= SUCCESS_SCORE_THRESHOLD
+
+    finally:
+        try:
+            await env.close()
+        except Exception as e:
+            print(f"[DEBUG] env.close() error (container cleanup): {e}", flush=True)
+        log_end(success=success, steps=steps_taken, score=score, rewards=rewards)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

inference.py

@@ -1,60 +1,141 @@
+"""
+SQL Data Analyst OpenEnv — baseline inference (hackathon format).
+
+Environment variables (set before running):
+  HF_TOKEN         Primary API key (Hugging Face / OpenAI-compatible).
+  API_BASE_URL     LLM base URL (e.g. https://api.openai.com/v1 or HF router).
+  MODEL_NAME       Model id for chat completions.
+  OPENAI_API_KEY   Optional fallback if HF_TOKEN is unset.
+  API_KEY          Optional second fallback.
+
+Optional:
+  SQL_AGENT_TASK      Logged as task= in [START] (default: sql_analyst_episode).
+  SQL_AGENT_BENCHMARK Logged as env= in [START] (default: sql_agent_openenv).
+  MAX_STEPS           Max env.step calls (default: 24).
+  OPENAI_SEED         Passed to OpenAI only when base URL looks like OpenAI.
+  ENV_CONTAINER_START true to use SqlEnvClient.from_docker_image (default: false).
+  IMAGE_NAME / LOCAL_IMAGE_NAME / ENV_IMAGE_NAME  Docker image tag when using container.
+"""
+
+from __future__ import annotations
+
 import os
+import re
+import sys
 import textwrap
-from openai import OpenAI
-from typing import List, Dict, Any
+from typing import Any, Dict, List, Optional
+
 from dotenv import load_dotenv
+from openai import OpenAI
 
 load_dotenv()
 
 from client import SqlEnvClient
 from models import SqlAction
 
-# OpenEnv execution environment config
-ENV_CONTAINER_START = os.getenv("ENV_CONTAINER_START", "false").lower() == "true"
-ENV_IMAGE_NAME = "sql-agent-env:latest"
-
-# LLM inference config as per instructions
+# --- Mandatory hackathon configuration ---
 API_BASE_URL = os.getenv("API_BASE_URL", "https://api.openai.com/v1")
-API_KEY = os.getenv("OPENAI_API_KEY")
 MODEL_NAME = os.getenv("MODEL_NAME", "gpt-4o-mini")
-MAX_STEPS = int(os.getenv("MAX_STEPS", "20"))
+API_KEY = (
+    os.getenv("HF_TOKEN")
+    or os.getenv("OPENAI_API_KEY")
+    or os.getenv("API_KEY")
+)
+
+TASK_NAME = os.getenv("SQL_AGENT_TASK", "sql_analyst_episode")
+BENCHMARK = os.getenv("SQL_AGENT_BENCHMARK", "sql_agent_openenv")
+
+MAX_STEPS = int(os.getenv("MAX_STEPS", "24"))
 OPENAI_SEED = int(os.getenv("OPENAI_SEED", "42"))
 
+ENV_CONTAINER_START = os.getenv("ENV_CONTAINER_START", "false").lower() == "true"
+ENV_IMAGE_NAME = (
+    os.getenv("LOCAL_IMAGE_NAME")
+    or os.getenv("IMAGE_NAME")
+    or os.getenv("ENV_IMAGE_NAME", "sql-agent-env:latest")
+)
+
+# Grader task ids (must match core/tasks.py) for normalized [END] score in [0, 1]
+TASK_IDS = [
+    "employee_payroll_overview",
+    "department_budget_summary",
+    "senior_engineering_comp_review",
+]
+
 SYSTEM_PROMPT = textwrap.dedent(
     """
     You are an expert Data Analyst and SQL Agent.
     You will be provided with a SQL Schema and an instruction.
-    Reply with exactly one JSON block containing the SQL query to execute.
-    Do NOT provide markdown formatting or explanations. DO NOT wrap in ```sql.
-    The response should be the pure string of the query itself.
+    Reply with exactly one SQL query string to execute.
+    Do NOT use markdown fences or explanations — only the raw SQL text.
     """
 ).strip()
 
-def build_user_prompt(step: int, observation, history: List[str]) -> str:
+
+def log_start(task: str, env: str, model: str) -> None:
+    print(f"[START] task={task} env={env} model={model}", flush=True)
+
+
+def log_step(
+    step: int,
+    action: str,
+    reward: float,
+    done: bool,
+    error: Optional[str],
+) -> None:
+    err = error if error else "null"
+    err_one = sanitize_one_line(err) if err != "null" else "null"
+    act_one = sanitize_one_line(action)
+    done_val = str(done).lower()
+    print(
+        f"[STEP] step={step} action={act_one} reward={reward:.2f} done={done_val} error={err_one}",
+        flush=True,
+    )
+
+
+def log_end(success: bool, steps: int, score: float, rewards: List[float]) -> None:
+    # Evaluators expect a comma-separated list; use 0.00 if no steps ran.
+    rewards_str = ",".join(f"{r:.2f}" for r in rewards) if rewards else "0.00"
+    print(
+        f"[END] success={str(success).lower()} steps={steps} score={score:.3f} rewards={rewards_str}",
+        flush=True,
+    )
+
+
+def sanitize_one_line(s: str) -> str:
+    if not s:
+        return ""
+    return " ".join(s.split())
+
+
+def build_user_prompt(step: int, observation: Any, history: List[str]) -> str:
     schema = observation.schema_info
     instruction = observation.current_task_instruction
     exec_result = observation.execution_result or "None"
     error = observation.execution_error or "None"
-    
-    prompt = textwrap.dedent(
+    history_block = "\n".join(history[-6:]) if history else "None"
+    return textwrap.dedent(
         f"""
         Step: {step}
         Database Schema:
         {schema}
-        
+
         Task Instruction:
         {instruction}
-        
+
         Previous Execution Result: {exec_result}
         Previous Error: {error}
-        
+
+        Recent history:
+        {history_block}
+
         Write the precise SQL query string to accomplish the task instruction. Return nothing else.
         """
     ).strip()
-    return prompt
+
 
 def parse_model_action(response_text: str) -> str:
-    query = response_text.strip()
+    query = (response_text or "").strip()
     if query.startswith("```sql"):
         query = query[6:]
     if query.startswith("```"):
@@ -63,120 +144,141 @@ def parse_model_action(response_text: str) -> str:
         query = query[:-3]
     return query.strip()
 
-def main():
+
+def _make_direct_client():
+    from server.environment import SqlEnvironment
+
+    base_env = SqlEnvironment()
+
+    class DirectClient:
+        def __init__(self, target):
+            self.target = target
+
+        def reset(self):
+            obs = self.target.reset()
+            return type(
+                "StepResult",
+                (),
+                {"observation": obs, "reward": 0.0, "done": False},
+            )()
+
+        def step(self, action):
+            obs = self.target.step(action)
+            return type(
+                "StepResult",
+                (),
+                {
+                    "observation": obs,
+                    "reward": obs.reward if obs.reward is not None else 0.0,
+                    "done": obs.done,
+                },
+            )()
+
+        def close(self):
+            pass
+
+    return DirectClient(base_env)
+
+
+def main() -> None:
     if not API_KEY:
-        raise RuntimeError("OPENAI_API_KEY is required for reproducible baseline runs.")
+        print(
+            "[DEBUG] Set HF_TOKEN (or OPENAI_API_KEY) before running.",
+            file=sys.stderr,
+            flush=True,
+        )
+        raise RuntimeError(
+            "Missing API key: set HF_TOKEN or OPENAI_API_KEY for OpenAI-compatible client."
+        )
 
     client = OpenAI(base_url=API_BASE_URL, api_key=API_KEY)
 
-    print("--- SQL Data Analyst OpenEnv Baseline ---")
-    
     if ENV_CONTAINER_START:
-        print(f"Starting docker image: {ENV_IMAGE_NAME}")
+        print(f"[DEBUG] Using docker image: {ENV_IMAGE_NAME}", file=sys.stderr, flush=True)
         env = SqlEnvClient.from_docker_image(ENV_IMAGE_NAME).sync()
     else:
-        # Fallback to local memory-bound initialization
-        from server.environment import SqlEnvironment
-        base_env = SqlEnvironment()
-        
-        # Direct wrapper
-        class DirectClient:
-            def __init__(self, target):
-                self.target = target
-            def reset(self):
-                obs = self.target.reset()
-                return type('StepResult', (), {'observation': obs, 'reward': 0.0, 'done': False})()
-            def step(self, action):
-                obs = self.target.step(action)
-                return type(
-                    'StepResult',
-                    (),
-                    {
-                        'observation': obs,
-                        'reward': obs.reward if obs.reward is not None else 0.0,
-                        'done': obs.done,
-                    },
-                )()
-            def close(self):
-                pass
-        env = DirectClient(base_env)
+        env = _make_direct_client()
 
     history: List[str] = []
-    task_scores: Dict[str, float] = {}
-    run_trace: List[Dict[str, Any]] = []
-    
+    rewards: List[float] = []
+    best_task_score: Dict[str, float] = {tid: 0.0 for tid in TASK_IDS}
+
+    steps_taken = 0
+    score = 0.0
+    success = False
+    last_done = False
+
+    log_start(task=TASK_NAME, env=BENCHMARK, model=MODEL_NAME)
+
     try:
         result = env.reset()
         observation = result.observation
-        print(f"\n[Episode Start] Initial Task: {observation.current_task_instruction}")
-        
+
         for step in range(1, MAX_STEPS + 1):
             if result.done:
-                print("Environment signalled done. Stopping early.")
                 break
-                
+
             user_prompt = build_user_prompt(step, observation, history)
-            
             messages = [
                 {"role": "system", "content": SYSTEM_PROMPT},
                 {"role": "user", "content": user_prompt},
             ]
-            
-            completion = client.chat.completions.create(
-                model=MODEL_NAME,
-                messages=messages,
-                temperature=0.0,
-                seed=OPENAI_SEED,
-                stream=False,
-            )
-            response_text = completion.choices[0].message.content or "SELECT 1"
+
+            create_kwargs: Dict[str, Any] = {
+                "model": MODEL_NAME,
+                "messages": messages,
+                "temperature": 0.0,
+                "stream": False,
+            }
+            if re.search(r"openai\.com", API_BASE_URL, re.I):
+                create_kwargs["seed"] = OPENAI_SEED
+
+            try:
+                completion = client.chat.completions.create(**create_kwargs)
+                response_text = completion.choices[0].message.content or "SELECT 1"
+            except Exception as exc:
+                print(f"[DEBUG] Model request failed: {exc}", file=sys.stderr, flush=True)
+                response_text = "SELECT 1"
 
             action_str = parse_model_action(response_text)
-            print(f"\n[Step {step}] Model suggested: {action_str}")
-            current_task_id = observation.task_id
             result = env.step(SqlAction(query=action_str))
             observation = result.observation
-            reward = result.reward
-            done = result.done
-
-            print(f"  Reward: {reward:+.2f} | Done: {done}")
-            print(f"  Task Score: {observation.task_score:.3f} | Difficulty: {observation.difficulty}")
-            if observation.grader_feedback:
-                print(f"  Grader: {observation.grader_feedback}")
-            if observation.execution_error:
-                print(f"  Error: {observation.execution_error[:200]}")
-
-            if observation.task_score > 0:
-                task_scores[current_task_id] = max(task_scores.get(current_task_id, 0.0), observation.task_score)
-                
-            history.append(f"Q: {action_str} -> R: {reward:+.2f}")
-            run_trace.append(
-                {
-                    "step": step,
-                    "task_id": current_task_id,
-                    "action": action_str,
-                    "reward": reward,
-                    "task_score": observation.task_score,
-                    "done": done,
-                }
-            )
-            
+            reward = float(result.reward if result.reward is not None else 0.0)
+            done = bool(result.done)
+            last_done = done
+
+            err_raw = observation.execution_error
+            rewards.append(reward)
+            steps_taken = step
+
+            tid = getattr(observation, "task_id", None)
+            if tid in best_task_score:
+                best_task_score[tid] = max(
+                    best_task_score[tid], float(observation.task_score or 0.0)
+                )
+
+            log_step(step=step, action=action_str, reward=reward, done=done, error=err_raw)
+
+            history.append(f"Q: {action_str[:200]} -> R: {reward:+.2f}")
+
             if done:
-                print("\nAll tasks complete!")
                 break
 
-        print("\n=== Baseline Summary ===")
-        for task_id in sorted(task_scores):
-            print(f"{task_id}: {task_scores[task_id]:.3f}")
-        task_avg = (sum(task_scores.values()) / len(task_scores)) if task_scores else 0.0
-        total_return = sum(item["reward"] for item in run_trace)
-        print(f"average_task_score: {task_avg:.3f}")
-        print(f"episode_return: {total_return:.3f}")
-        print(f"model: {MODEL_NAME} | seed: {OPENAI_SEED}")
-        print("reproducibility_note: deterministic prompt, temperature=0.0, fixed seed")
+        score = sum(best_task_score[t] for t in TASK_IDS) / len(TASK_IDS)
+        score = min(max(score, 0.0), 1.0)
+        # Success: episode ended in terminal success state with strong average task score
+        success = last_done and score >= 0.95
 
+    except Exception as exc:
+        print(f"[DEBUG] Episode error: {exc}", file=sys.stderr, flush=True)
+        success = False
     finally:
-        env.close()
+        try:
+            env.close()
+        except Exception as exc:
+            print(f"[DEBUG] env.close() error: {exc}", file=sys.stderr, flush=True)
+        log_end(success=success, steps=steps_taken, score=score, rewards=rewards)
+
 
 if __name__ == "__main__":
     main()

openenv.yaml

@@ -1,5 +1,6 @@
 spec_version: 1
 name: sql-agent-env
+description: OpenEnv SQL data analyst — schema → SQL tasks with deterministic graders
 type: space
 runtime: fastapi
 app: server.app:app