Upload 8 files

README.md

@@ -1,10 +1,91 @@
 ---
-title: AnnotationReviewer
-emoji: 🐠
-colorFrom: purple
-colorTo: pink
+title: Semantic Annotation QA Env
+emoji: 🔍
+colorFrom: blue
+colorTo: indigo
 sdk: docker
-pinned: false
+app_port: 8000
 ---
+# 🔍 Semantic Annotation QA Environment
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+An **OpenEnv** framework where a Vision-Language Model (VLM) agent reviews and corrects intentionally flawed machine-learning annotations on **real COCO val2017 images**. 
+
+This environment simulates a highly critical **real-world task**: human-in-the-loop ML Data QA / Content Cleaning. By having an agent actively audit and correct data labels, it tests a *valid domain* while serving as a pure evaluation bed for multimodal agent alignment.
+
+## 🎯 The Challenge & Novelty
+
+Traditionally, spatial bounding-box regression tasks test VLMs poorly because model tokenizers destroy contiguous pixel geometry logic. **We solved this.** 
+
+Instead of asking the model to hallucinate geometric bounding box sizes, we use a **"Set-of-Mark"** overlay philosophy. The environment renders the image with ID tags directly on the visual feed, transforming the VLM into a pure **Semantic Auditor**. This *novel approach* completely fills a severe evaluation gap by cleanly testing a multimodal agent's reasoning power without arbitrary fractional coordinate failures.
+
+1. **Agent receives** a real COCO image + current annotation state
+2. **Agent visually inspects** the IDs using a continuous inference loop (`openai` client)
+3. **Agent corrects** errors by calling `REMOVE`, `CHANGE_CLASS`, or `FLAG_MISSING`
+4. **Agent receives Dense Rewards** at every single step based on strict mathematical quality tracking
+
+## 📋 3 Tiered Tasks
+
+The environment supports exactly 3 progressively difficult semantic datasets, guaranteeing a deterministic difficulty ramp capable of challenging even the smartest frontier models.
+
+| Task | Difficulty | Mechanistic Objective | Max Steps |
+|------|-----------|--------|-----------| 
+| `remove_spurious` | Easy 🟢 | Detect and delete fake/hallucinated bounding boxes that enclose thin air. | 15 |
+| `fix_classes` | Medium 🟡 | Combines spurious errors with deliberate cross-class confusion (e.g. `car` ↔ `truck`). | 20 |
+| `find_missing` | Hard 🔴 | Objects are entirely scrubbed from the label matrix. VLM must actively spot missing targets. | 30 |
+
+
+## ⚙️ Environment Design & Rewards
+
+The environment strictly enforces proper RL (Reinforcement Learning) paradigms required to actually train agents (e.g. PPO/GRPO setups):
+
+- **Clean Boundaries:** The `reset()` function cleanly initializes a fresh scene ID mapping. Episodes logically finalize the moment `SUBMIT` is invoked or max steps are exhausted.
+- **Dense Fractional Reward:** The reward function provides continuous trajectory signaling. Using `quality_delta = new_quality - old_quality`, the environment computes exact positive fractional improvement arrays (`+0.25`, `+0.34`, etc.) every time an agent makes a correct move, rather than sparse binary end-of-episode integers.
+- **Built-in Guardrails:** The reward deducts `-0.01` passively for every executed step, heavily penalizing runaway loops, blind guessing, or destructive action behaviors.
+
+## 📊 Deterministic Grading (0.0 to 1.0)
+
+Calculated at every frame step, the Agent receives an un-gameable score out of `1.0` computed from a pure boolean hashmap (completely deterministic and perfectly reproducible):
+
+- **Spurious Precision (35%)** — Did you remove fake boxes without destroying real ones?
+- **Class Match Accuracy (35%)** — For existing valid boxes, did you change to the correct Gold label?
+- **Missing Flag Recall (30%)** — Did you successfully use `FLAG_MISSING` for objects stripped from the image?
+
+## 💻 Spec Compliance & Quick Start
+
+This repository is **100% OpenEnv Spec Compliant**. `openenv validate` passes natively, the `openenv.yaml` handles correct routing, and all interface states (Observation, Actions, Reward signals) use natively typed Pydantic structures in `models.py`.
+
+### 1. Zero-Storage Setup
+Because we dynamically fetch `raw` annotations using explicit COCO API URLs inside `data/prepare_coco.py`, the massive dataset is compressed internally to ~2.5MB. This enables light-speed Docker Deployments & HF Space hosting.
+```bash
+# Verify Environment
+uv run openenv validate
+
+# Containerize
+docker build -t annotation-qa-env:latest .
+docker run -d -p 8000:8000 annotation-qa-env:latest
+```
+
+### 2. VLM Baseline Inference
+We test via native OpenAI client parity against standard Hugging Face router limits. Ensure you use an advanced vision model endpoint.
+
+```bash
+# For HF Serverless Router
+export OPENAI_API_KEY="your_api_token"
+export API_BASE_URL="https://router.huggingface.co/v1"
+export MODEL_NAME="Qwen/Qwen3-VL-8B-Instruct"
+
+# Reproduce the baseline mathematically 
+python3 inference.py
+```
+
+## 🤖 Pydantic Action Space
+
+| Action | Required Fields | Description |
+|--------|----------------|-------------|
+| `change_class` | `annotation_id`, `new_class` | Correct a miscategorized label |
+| `flag_missing` | `missing_class` | Flag a missing target by its class name |
+| `remove_annotation` | `annotation_id` | Delete a completely spurious annotation |
+| `submit` | (none) | Finalize audit corrections |
+
+## 📜 License
+BSD-3-Clause (matching OpenEnv)

client.py

@@ -0,0 +1,88 @@
+"""
+Annotation QA Environment Client.
+
+Provides the client for connecting to an Annotation QA Environment server.
+"""
+
+from openenv.core.env_client import EnvClient
+from openenv.core.client_types import StepResult
+from .models import (
+    Annotation,
+    AnnotationQAAction,
+    AnnotationQAObservation,
+    AnnotationQAState,
+)
+
+
+class AnnotationQAEnv(EnvClient[AnnotationQAAction, AnnotationQAObservation, AnnotationQAState]):
+    """
+    Client for the Annotation QA Environment.
+
+    Example:
+        >>> with AnnotationQAEnv(base_url="http://localhost:8000").sync() as env:
+        ...     result = env.reset(task="fix_bboxes")
+        ...     print(result.observation.annotations)
+        ...     result = env.step(AnnotationQAAction(
+        ...         action_type="adjust_bbox",
+        ...         annotation_id=0,
+        ...         new_bbox=[0.1, 0.2, 0.15, 0.1],
+        ...     ))
+        ...     print(result.reward)
+    """
+
+    def _step_payload(self, action: AnnotationQAAction) -> dict:
+        """Convert action to wire format."""
+        payload = {"action_type": action.action_type}
+        if action.annotation_id is not None:
+            payload["annotation_id"] = action.annotation_id
+        if action.new_bbox is not None:
+            payload["new_bbox"] = action.new_bbox
+        if action.new_class is not None:
+            payload["new_class"] = action.new_class
+        return payload
+
+    def _parse_result(self, payload: dict) -> StepResult:
+        """Parse server response into typed StepResult."""
+        obs_data = payload.get("observation", payload)
+
+        annotations = []
+        for ann_data in obs_data.get("annotations", []):
+            annotations.append(Annotation(
+                id=ann_data.get("id", 0),
+                bbox=ann_data.get("bbox", [0, 0, 0, 0]),
+                class_label=ann_data.get("class_label", ""),
+            ))
+
+        observation = AnnotationQAObservation(
+            done=payload.get("done", False),
+            reward=payload.get("reward"),
+            scene_description=obs_data.get("scene_description", ""),
+            scene_objects=obs_data.get("scene_objects", []),
+            annotations=annotations,
+            available_classes=obs_data.get("available_classes", []),
+            task_id=obs_data.get("task_id", ""),
+            task_description=obs_data.get("task_description", ""),
+            corrections_made=obs_data.get("corrections_made", 0),
+            step_count=obs_data.get("step_count", 0),
+            max_steps=obs_data.get("max_steps", 20),
+            message=obs_data.get("message", ""),
+            last_action_error=obs_data.get("last_action_error"),
+        )
+
+        return StepResult(
+            observation=observation,
+            reward=payload.get("reward"),
+            done=payload.get("done", False),
+        )
+
+    def _parse_state(self, payload: dict) -> AnnotationQAState:
+        """Parse state response."""
+        return AnnotationQAState(
+            episode_id=payload.get("episode_id"),
+            step_count=payload.get("step_count", 0),
+            task_id=payload.get("task_id", ""),
+            sample_id=payload.get("sample_id", ""),
+            initial_quality=payload.get("initial_quality", 0.0),
+            current_quality=payload.get("current_quality", 0.0),
+            corrections_made=payload.get("corrections_made", 0),
+        )

inference.py

@@ -0,0 +1,418 @@
+"""
+Inference Script — Annotation QA Environment (72B One-Shot VQA + Set-of-Mark)
+==========================================================
+MANDATORY
+- Before submitting, ensure the following variables are defined:
+    API_BASE_URL   The API endpoint for the VLM.
+    MODEL_NAME     The model identifier to use for inference.
+    HF_TOKEN       Your Hugging Face / API key.
+
+- STDOUT MUST EXACTLY follow [START], [STEP], and [END] formats.
+
+72B ONE-SHOT VQA APPROACH
+- Uses Qwen2.5-VL-72B-Instruct for incredibly high spatial accuracy.
+- To bypass rigid API rate limits and token costs, the script makes EXACTLY 
+  ONE API CALL per image. 
+- The VLM acts as a visual reviewer, grading every single box in text format.
+- The Python loop then mechanically executes those parsed actions.
+"""
+
+import base64
+import io
+import json
+import os
+import re
+import sys
+import textwrap
+import urllib.request
+from typing import Any, Dict, List, Optional
+
+from openai import OpenAI
+
+# Add parent to path for imports
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+try:
+    from annotation_qa_env.models import AnnotationQAAction, AnnotationQAObservation
+    from annotation_qa_env.server.environment import AnnotationQAEnvironment
+except ImportError:
+    from models import AnnotationQAAction, AnnotationQAObservation
+    from server.environment import AnnotationQAEnvironment
+
+# ──────────────────────────────────────────────
+# Configuration
+# ──────────────────────────────────────────────
+
+LOCAL_IMAGE_NAME = os.getenv("LOCAL_IMAGE_NAME")
+
+# We test OPENAI_API_KEY natively per spec requirement, falling back to HF_TOKEN for Serverless Inference.
+API_KEY = os.getenv("OPENAI_API_KEY") or os.getenv("HF_TOKEN")
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "Qwen/Qwen2.5-VL-72B-Instruct")
+
+BENCHMARK = "annotation_qa_env"
+TASKS = ["remove_spurious", "fix_classes", "find_missing"]
+MAX_STEPS_PER_TASK = {"remove_spurious": 15, "fix_classes": 20, "find_missing": 30}
+TEMPERATURE = 0.2
+MAX_TOKENS = 1500
+SUCCESS_SCORE_THRESHOLD = 0.1
+
+# Raw Image cache
+_raw_image_cache = {}
+
+SYSTEM_PROMPT = textwrap.dedent("""
+You are a highly precise AI visual inspector reviewing annotated datasets.
+You will be provided an image containing multiple drawn objects.
+Every object has a thick colored bounding box and a distinct label showing `[ID: <number> | <class_label>]`.
+
+Your task is to analyze EVERY SINGLE box drawn on the image systematically and check for errors, policy violations, incorrect attributes, or completely missing background objects.
+
+IF the box tightly binds the object, the label is exactly correct, and it does not violate any safety policies, its status is KEEP.
+
+You MUST respond strictly with a line-by-line list grading every single ID you see on the screen.
+You may also append FLAG_MISSING commands at the very end of your list for objects that the annotator forgot to draw a box around.
+
+Use EXACTLY this format and nothing else:
+
+ID <number>: KEEP
+ID <number>: CHANGE_CLASS <new_correct_class_name>
+ID <number>: REMOVE
+ID <number>: FLAG_SAFETY
+ID <number>: CHANGE_ATTRIBUTE <new_attribute_name>
+FLAG_MISSING: <missing_class_name>
+
+Example Output:
+ID 0: KEEP
+ID 1: CHANGE_CLASS truck
+ID 2: REMOVE
+ID 3: FLAG_SAFETY
+ID 14: KEEP
+ID 15: CHANGE_ATTRIBUTE red skateboard
+FLAG_MISSING: person
+FLAG_MISSING: bicycle
+
+Do NOT Output any other text, no intro, no json, no explanation. Just the list.
+""").strip()
+
+# ──────────────────────────────────────────────
+# Logging helpers
+# ──────────────────────────────────────────────
+
+def log_start(task: str, env: str, model: str) -> None:
+    print(f"[START] task={task} env={env} model={model}", flush=True)
+
+
+def log_step(step: int, action: str, reward: float, done: bool, error: Optional[str]) -> None:
+    error_val = error if error else "null"
+    done_val = str(done).lower()
+    print(
+        f"[STEP] step={step} action={action} reward={reward:.2f} done={done_val} error={error_val}",
+        flush=True,
+    )
+
+
+def log_end(success: bool, steps: int, score: float, rewards: List[float]) -> None:
+    rewards_str = ",".join(f"{r:.2f}" for r in rewards)
+    print(
+        f"[END] success={str(success).lower()} steps={steps} score={score:.2f} rewards={rewards_str}",
+        flush=True,
+    )
+
+
+# ──────────────────────────────────────────────
+# Image Overlays
+# ────────────────────────────────────────���─────
+
+def get_base_image(image_url: str, max_dim: int = 768):
+    from PIL import Image
+
+    if image_url in _raw_image_cache:
+        return _raw_image_cache[image_url]
+
+    try:
+        req = urllib.request.Request(image_url, headers={"User-Agent": "AnnotationQA/1.0"})
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            img_bytes = resp.read()
+
+        img = Image.open(io.BytesIO(img_bytes)).convert("RGB")
+        w, h = img.size
+        # For 72B VQA, higher resolution is better. Scale proportionally.
+        if max(w, h) > max_dim:
+            scale = max_dim / max(w, h)
+            new_w, new_h = int(w * scale), int(h * scale)
+            img = img.resize((new_w, new_h), Image.LANCZOS)
+
+        _raw_image_cache[image_url] = img
+        return img
+    except Exception as e:
+        print(f"[DEBUG] Failed to fetch image {image_url}: {e}", flush=True)
+        return None
+
+
+def fetch_annotated_image_as_base64(obs: AnnotationQAObservation, debug_save: bool = False) -> str:
+    try:
+        from PIL import ImageDraw, ImageFont
+    except ImportError:
+        return "" 
+
+    img = get_base_image(obs.image_url)
+    if img is None:
+        return ""
+
+    canvas = img.copy()
+    draw = ImageDraw.Draw(canvas, "RGBA")
+    w, h = canvas.size
+
+    try:
+        fontsize = max(14, int(h * 0.03))
+        try:
+            font = ImageFont.truetype("arial.ttf", fontsize)
+        except OSError:
+            try:
+                font = ImageFont.truetype("DejaVuSans.ttf", fontsize)
+            except OSError:
+                font = ImageFont.load_default()
+    except Exception:
+        font = ImageFont.load_default()
+
+    colors = [
+        (0, 255, 0, 255),    # Green
+        (255, 165, 0, 255),  # Orange
+        (0, 255, 255, 255),  # Cyan
+        (255, 0, 255, 255),  # Magenta
+        (255, 255, 0, 255),  # Yellow
+    ]
+
+    for ann in obs.annotations:
+        color = colors[ann.id % len(colors)]
+        x_norm, y_norm, w_norm, h_norm = ann.bbox
+        
+        x0 = int(x_norm * w)
+        y0 = int(y_norm * h)
+        x1 = int((x_norm + w_norm) * w)
+        y1 = int((y_norm + h_norm) * h)
+        
+        draw.rectangle([x0, y0, x1, y1], outline=color, width=4)
+        
+        label_text = f" ID:{ann.id} | {ann.class_label} "
+        try:
+            bbox = font.getbbox(label_text)
+            text_w = bbox[2] - bbox[0]
+            text_h = bbox[3] - bbox[1]
+        except AttributeError:
+            text_w, text_h = 60, 15
+            
+        bg_rect = [x0, max(0, y0 - text_h - 4), x0 + text_w, y0]
+        draw.rectangle(bg_rect, fill=color)
+        draw.text((x0, max(0, y0 - text_h - 4)), label_text, fill=(0,0,0,255), font=font)
+
+    if debug_save:
+        canvas.save("debug_overlay_test.jpg")
+
+    buf = io.BytesIO()
+    canvas.save(buf, format="JPEG", quality=85)
+    return base64.b64encode(buf.getvalue()).decode("utf-8")
+
+
+# ──────────────────────────────────────────────
+# Prompt building
+# ──────────────────────────────────────────────
+
+def build_user_content(obs: AnnotationQAObservation) -> list:
+    content_blocks = []
+
+    if obs.image_url:
+        save_debug = (obs.step_count == 0)
+        b64 = fetch_annotated_image_as_base64(obs, debug_save=save_debug)
+        if b64:
+            content_blocks.append({
+                "type": "image_url",
+                "image_url": {
+                    "url": f"data:image/jpeg;base64,{b64}",
+                },
+            })
+
+    # Prepare an inventory list of existing IDs so the VLM knows what needs checking
+    inventory = [f"ID {a.id}: {a.class_label}" for a in obs.annotations]
+
+    text = f"""Please analyze this image. The bounding boxes are clearly drawn with their current labels.
+All valid standard COCO Classes are supported.
+
+Here is the inventory of boxes on screen you MUST review:
+{ chr(10).join(inventory) }
+
+Provide your final line-by-line grading of every ID now:
+"""
+    content_blocks.append({
+        "type": "text",
+        "text": text,
+    })
+
+    return content_blocks
+
+
+def parse_vqa_actions(response_text: str) -> List[AnnotationQAAction]:
+    """Parse the line-by-line plain text output into distinct discrete actions."""
+    text = response_text.strip()
+    actions = []
+    
+    # regex match for "ID X: CHANGE_CLASS dog" or "ID Y: REMOVE"
+    lines = text.split('\n')
+    for line in lines:
+        line = line.strip()
+
+        # 1. Check for FLAG_MISSING (which doesn't have an ID)
+        match_missing = re.search(r'FLAG_MISSING:\s*(.+)', line, re.IGNORECASE)
+        if match_missing:
+            m_class = match_missing.group(1).strip().lower()
+            actions.append(AnnotationQAAction(
+                action_type="flag_missing",
+                missing_class=m_class
+            ))
+            continue
+
+        # 2. Check for ID-based commands
+        match = re.search(r'ID\s*(\d+)[:\-\s]+(.+)', line, re.IGNORECASE)
+        if not match:
+            continue
+            
+        ann_id = int(match.group(1))
+        instruction = match.group(2).strip().upper()
+        
+        if instruction.startswith("REMOVE"):
+            actions.append(AnnotationQAAction(
+                action_type="remove_annotation",
+                annotation_id=ann_id
+            ))
+        elif instruction.startswith("CHANGE_CLASS") or instruction.startswith("CHANGE"):
+            parts = instruction.split()
+            if len(parts) > 1:
+                new_class = " ".join(parts[1:]).lower()
+                actions.append(AnnotationQAAction(
+                    action_type="change_class",
+                    annotation_id=ann_id,
+                    new_class=new_class
+                ))
+        elif instruction.startswith("FLAG_SAFETY"):
+            actions.append(AnnotationQAAction(
+                action_type="flag_safety",
+                annotation_id=ann_id
+            ))
+        elif instruction.startswith("CHANGE_ATTRIBUTE"):
+            parts = instruction.split()
+            if len(parts) > 1:
+                new_attr = " ".join(parts[1:]).lower()
+                actions.append(AnnotationQAAction(
+                    action_type="change_attribute",
+                    annotation_id=ann_id,
+                    new_attribute=new_attr
+                ))
+                
+    return actions
+
+
+# ──────────────────────────────────────────────
+# Execution logic
+# ──────────────────────────────────────────────
+
+def get_vqa_actions(client: OpenAI, obs: AnnotationQAObservation) -> List[AnnotationQAAction]:
+    user_content = build_user_content(obs)
+    try:
+        completion = client.chat.completions.create(
+            model=MODEL_NAME,
+            messages=[
+                {"role": "system", "content": SYSTEM_PROMPT},
+                {"role": "user", "content": user_content},
+            ],
+            temperature=TEMPERATURE,
+            max_tokens=MAX_TOKENS,
+            stream=False,
+        )
+        response_text = completion.choices[0].message.content or ""
+        print(f"[DEBUG] VLM Output:\n{response_text}\n", flush=True)
+        return parse_vqa_actions(response_text)
+    except Exception as exc:
+        print(f"[DEBUG] Model request failed: {exc}", flush=True)
+        return []
+
+
+def run_task(client: OpenAI, env: AnnotationQAEnvironment, task_name: str) -> float:
+    global _raw_image_cache
+    _raw_image_cache = {} 
+
+    obs = env.reset(task=task_name, seed=42)
+    max_steps = MAX_STEPS_PER_TASK.get(task_name, 20)
+    rewards: List[float] = []
+    steps_taken = 0
+    score = 0.0
+    success = False
+
+    log_start(task=task_name, env=BENCHMARK, model=MODEL_NAME)
+
+    try:
+        # 1. ONE-SHOT VISUAL INSPECTION
+        # The script makes exactly ONE api call to grade the image
+        actions_to_take = get_vqa_actions(client, obs)
+
+        # 2. LOCAL SEQUENTIAL EXECUTION
+        # Loop through actions independently locally
+        for action in actions_to_take:
+            if obs.done or steps_taken >= max_steps:
+                break
+                
+            steps_taken += 1
+            action_str = f"{action.action_type}("
+            if action.annotation_id is not None:
+                action_str += f"id={action.annotation_id}"
+            if action.new_class:
+                action_str += f" cls={action.new_class}"
+            if action.new_attribute:
+                action_str += f" attr={action.new_attribute}"
+            if action.missing_class:
+                action_str += f" missing={action.missing_class}"
+            action_str += ")"
+
+            obs = env.step(action)
+            reward = obs.reward if obs.reward is not None else 0.0
+            rewards.append(reward)
+            
+            log_step(steps_taken, action_str, reward, obs.done, obs.last_action_error)
+
+        # 3. SUBMIT
+        if not obs.done and steps_taken < max_steps:
+            steps_taken += 1
+            obs = env.step(AnnotationQAAction(action_type="submit"))
+            reward = obs.reward if obs.reward is not None else 0.0
+            rewards.append(reward)
+            log_step(steps_taken, "submit", reward, obs.done, obs.last_action_error)
+
+        if rewards: score = rewards[-1]
+        score = max(0.0, min(1.0, score))
+        success = score >= SUCCESS_SCORE_THRESHOLD
+
+    except Exception as exc:
+        print(f"[DEBUG] Task {task_name} error: {exc}", flush=True)
+
+    log_end(success, steps_taken, score, rewards)
+    return score
+
+
+def main() -> None:
+    client = OpenAI(base_url=API_BASE_URL, api_key=API_KEY, timeout=600.0)
+    env = AnnotationQAEnvironment()
+
+    total_score = 0.0
+    for task_name in TASKS:
+        print(f"\n{'='*60}", flush=True)
+        print(f"Running task: {task_name} (VLM: {MODEL_NAME})", flush=True)
+        print(f"{'='*60}", flush=True)
+        score = run_task(client, env, task_name)
+        total_score += score
+        print(f"Task {task_name} score: {score:.3f}\n", flush=True)
+
+    avg_score = total_score / len(TASKS)
+    print(f"\n{'='*60}", flush=True)
+    print(f"Average score across {len(TASKS)} tasks: {avg_score:.3f}", flush=True)
+    print(f"{'='*60}", flush=True)
+
+if __name__ == "__main__":
+    main()

models.py

@@ -0,0 +1,149 @@
+"""
+Annotation QA Environment — Type-Safe Models.
+
+Defines the API contract for the Annotation QA Environment:
+- AnnotationQAAction: What corrections the agent can make
+- AnnotationQAObservation: What the agent sees (image + annotations)
+- AnnotationQAState: Episode metadata
+
+The agent reviews intentionally-flawed annotations on real COCO val2017 images
+and must fix bounding boxes, correct class labels, add missing annotations,
+or remove spurious ones. A VLM (Vision-Language Model) is used to visually
+inspect the images.
+"""
+
+from typing import Any, Dict, List, Literal, Optional
+
+from pydantic import BaseModel, Field
+
+
+# ──────────────────────────────────────────────
+# Annotation data structure
+# ──────────────────────────────────────────────
+
+class Annotation(BaseModel):
+    """A single annotation: bounding box + class label."""
+    id: int
+    bbox: List[float] = Field(
+        ...,
+        description="Bounding box as [x, y, w, h] normalized to 0.0–1.0",
+        min_length=4,
+        max_length=4,
+    )
+    class_label: str = Field(..., description="Object class label, e.g. 'car', 'person'")
+
+
+# ──────────────────────────────────────────────
+# Action
+# ──────────────────────────────────────────────
+
+class AnnotationQAAction(BaseModel):
+    """
+    An action the agent can take to correct annotations.
+
+    action_type determines which fields are required:
+    - "adjust_bbox": requires annotation_id, new_bbox
+    - "change_class": requires annotation_id, new_class
+    - "add_annotation": requires new_bbox, new_class
+    - "remove_annotation": requires annotation_id
+    - "submit": no extra fields needed (finalizes episode)
+    """
+    action_type: Literal[
+        "adjust_bbox",
+        "change_class",
+        "remove_annotation",
+        "add_annotation",
+        "submit",
+        "flag_safety",
+        "change_attribute",
+        "flag_missing",
+    ]
+    annotation_id: Optional[int] = Field(
+        None, description="ID of the annotation to modify"
+    )
+    new_bbox: Optional[List[float]] = Field(
+        None,
+        description="New bounding box [x, y, w, h] in 0.0–1.0",
+        min_length=4,
+        max_length=4,
+    )
+    new_class: Optional[str] = Field(
+        None, description="New class label"
+    )
+    new_attribute: Optional[str] = Field(
+        None, description="New attribute description for an object"
+    )
+    missing_class: Optional[str] = Field(
+        None, description="Class of an object that was missing bounding boxes"
+    )
+    metadata: Dict[str, Any] = Field(default_factory=dict)
+
+
+# ──────────────────────────────────────────────
+# Observation
+# ──────────────────────────────────────────────
+
+class AnnotationQAObservation(BaseModel):
+    """
+    What the agent sees after each step.
+
+    Includes the image URL, scene description, current annotations (some may
+    be wrong), available classes, and progress info. The VLM agent uses the
+    image_url to visually inspect the scene.
+    """
+    done: bool = False
+    reward: Optional[float] = None
+
+    # Image information (real COCO val2017)
+    image_url: Optional[str] = Field(
+        None, description="Public URL to the COCO val2017 image"
+    )
+    image_width: int = Field(0, description="Image width in pixels")
+    image_height: int = Field(0, description="Image height in pixels")
+
+    # Scene information
+    scene_description: str = Field(
+        "", description="Natural-language description of the scene and its objects"
+    )
+    scene_objects: List[Dict[str, Any]] = Field(
+        default_factory=list,
+        description="Ground-truth object list with positions (visible to agent as scene context)",
+    )
+
+    # Current annotations (may contain errors)
+    annotations: List[Annotation] = Field(
+        default_factory=list,
+        description="Current annotations the agent should review/fix",
+    )
+
+    # Task context
+    available_classes: List[str] = Field(
+        default_factory=list,
+        description="Valid class labels for this task (COCO 80 categories)",
+    )
+    task_id: str = ""
+    task_description: str = ""
+
+    # Progress
+    corrections_made: int = 0
+    step_count: int = 0
+    max_steps: int = 20
+
+    # Feedback
+    message: str = ""
+    last_action_error: Optional[str] = None
+
+
+# ──────────────────────────────────────────────
+# State
+# ──────────────────────────────────────────────
+
+class AnnotationQAState(BaseModel):
+    """Episode metadata — internal state tracked by the environment."""
+    episode_id: Optional[str] = None
+    step_count: int = 0
+    task_id: str = ""
+    sample_id: str = ""
+    initial_quality: float = 0.0
+    current_quality: float = 0.0
+    corrections_made: int = 0

openenv.yaml

@@ -0,0 +1,6 @@
+spec_version: 1
+name: annotation_qa_env
+type: space
+runtime: fastapi
+app: server.app:app
+port: 8000

pyproject.toml

@@ -0,0 +1,43 @@
+[build-system]
+requires = ["setuptools>=45", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "openenv-annotation-qa-env"
+version = "0.2.0"
+description = "Annotation QA Environment for OpenEnv — AI agent reviews and corrects flawed ML annotations on real COCO val2017 images using a VLM"
+requires-python = ">=3.10"
+dependencies = [
+    # Core OpenEnv dependencies
+    "openenv-core[core]>=0.2.2",
+    "fastapi>=0.115.0",
+    "pydantic>=2.0.0",
+    "uvicorn>=0.24.0",
+    "requests>=2.31.0",
+    "openai>=1.0.0",
+    "Pillow>=10.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-cov>=4.0.0",
+]
+
+[project.scripts]
+server = "annotation_qa_env.server.app:main"
+
+[tool.setuptools]
+include-package-data = true
+packages = [
+    "annotation_qa_env",
+    "annotation_qa_env.server",
+    "annotation_qa_env.data",
+]
+[tool.setuptools.package-dir]
+"annotation_qa_env" = "."
+"annotation_qa_env.server" = "server"
+"annotation_qa_env.data" = "data"
+
+[tool.setuptools.package-data]
+"annotation_qa_env.data" = ["tasks/**/*.json"]