spec.md

# Bug Triage OpenEnv - Specification

1) Goal

Build a complete OpenEnv environment that simulates real-world software bug triage and supports learning through:

• reset()
• step(action)
• state()

The environment must include deterministic tasks, programmatic graders, reward shaping, reproducible baseline scoring, and deployment artifacts for Hugging Face Spaces.

2) Real-World Simulation

Domain

A software organization receives incoming bug reports from users, QA, and automated monitoring.
The agent acts as a triage engineer responsible for:

• assigning severity and priority
• identifying duplicates
• routing bugs to correct team/component
• requesting missing information when needed
• deciding whether to close/defer/escalate

Why this is real-world

Bug triage is a common operational workflow in engineering teams and has concrete measurable outcomes: SLA compliance, issue resolution speed, and triage correctness.

3) Environment API (OpenEnv Spec)

Implement typed Pydantic models and standard methods.

3.1 Typed Models

Create:

• ObservationModel
• ActionModel
• RewardModel
• StateModel (for state())

All models should be explicit and strictly typed (Literal, Enum, bounded numeric fields where possible).

3.2 Method Contracts

• reset(task_id: str | None = None, seed: int | None = None) -> ObservationModel
• step(action: ActionModel) -> tuple[ObservationModel, RewardModel, bool, dict]
• state() -> StateModel

3.3 Step Semantics

Each step processes exactly one action against the current ticket queue and returns:

• updated observation
• reward details (dense + cumulative)
• done when success/failure/step-budget reached
• info with grader signals and diagnostics

4) Data Model

Each ticket record should include:

• ticket_id: str
• title: str
• description: str
• reporter_type: Literal["user","qa","monitoring"]
• service: str
• component_candidates: list[str]
• created_at: datetime
• customer_tier: Literal["free","pro","enterprise"]
• repro_steps_present: bool
• logs_present: bool
• attachments_count: int
• suspected_duplicate_ids: list[str]

Hidden ground truth per ticket:

• true_severity: Literal["sev0","sev1","sev2","sev3"]
• true_priority: Literal["p0","p1","p2","p3"]
• true_component: str
• true_assignee_team: str
• duplicate_of: str | null
• needs_more_info: bool

5) Action Space

Use a single structured action schema with action_type plus typed payload:

• classify: set severity + priority + component
• assign: set owner team
• mark_duplicate: set canonical ticket id
• request_info: ask for missing repro/logs
• defer: move to backlog with reason
• close: close with reason (invalid, won't_fix, cannot_reproduce, resolved)
• escalate_incident: escalate urgent production impact
• next_ticket: move focus pointer

Validation rules:

• invalid field combos rejected with penalty
• references to unknown ticket IDs rejected with penalty
• destructive actions (close, defer) penalized if inconsistent with truth

6) Observation Space

Observation should include:

• focused ticket summary (current_ticket)
• queue stats (remaining_count, urgent_count, SLA-at-risk count)
• prior actions and outcomes (last_action_result)
• available teams/components
• step budget (steps_used, steps_remaining)
• partial score breakdown (optional but recommended for agent learning)

Observation must not expose hidden labels directly.

7) Reward Function (Meaningful + Dense)

Reward range per step: clamp to [-1.0, 1.0].
Episode final score normalized to [0.0, 1.0].

7.1 Dense Progress Signals

Suggested additive components:

• +0.20 correct severity
• +0.15 correct priority
• +0.15 correct component
• +0.10 correct team assignment
• +0.15 correct duplicate linking
• +0.10 correct request for more info
• +0.15 correct escalation for sev0/sev1 production incident

7.2 Negative Signals

• -0.20 incorrect close/defer on valid bug
• -0.15 missed critical escalation
• -0.05 invalid action schema
• -0.02 repeated no-op / loop behavior
• -0.01 unnecessary ticket switches

7.3 Terminal Bonus/Penalty

• +0.10 if all critical tickets triaged within budget
• -0.10 if step budget exhausted with high-severity untriaged tickets

8) Tasks and Graders (Easy / Medium / Hard)

Implement 3 fixed tasks with deterministic fixtures and grader logic.

8.1 Task A - Easy (bug_triage_easy)

Scenario:

• 8 tickets
• mostly clear reports
• minimal duplicates
• obvious component mapping

Grader criteria:

• weighted accuracy on severity/priority/component/team
• no more than 1 major mistake
• deterministic score in [0,1]

8.2 Task B - Medium (bug_triage_medium)

Scenario:

• 15 tickets
• mixed-quality reports
• several duplicates
• some missing repro details
• limited step budget

Grader criteria:

• weighted label correctness
• duplicate resolution F1-like score
• correct use of request_info
• loop/destructive penalties applied

8.3 Task C - Hard (bug_triage_hard)

Scenario:

• 25 tickets
• noisy and ambiguous text
• conflicting signals
• strict SLA pressure for enterprise + high severity tickets
• trade-offs under capacity constraints

Grader criteria:

• strict weighting on sev0/sev1 correctness
• SLA-risk handling score
• escalation correctness
• global policy quality (avoid destructive shortcuts)

8.4 Grader Output Contract

Each grader returns:

• score: float in [0.0, 1.0]
• subscores: dict[str, float]
• mistakes: list[str]
• passed: bool (optional threshold, e.g., >= 0.75)

9) Determinism and Reproducibility

• Fixed dataset fixtures per task (data/tasks/*.json)
• Deterministic shuffling controlled by seed
• deterministic grader logic (no LLM-in-the-loop grading)
• baseline run outputs JSON artifact with exact scores

Recommended defaults:

• DEFAULT_SEED=42
• model temperature 0

10) Baseline Inference Script

Create scripts/baseline_inference.py:

• Reads OPENAI_API_KEY from environment
• Runs all 3 tasks end-to-end
• Uses fixed prompt template and deterministic model params
• Writes:
  • console table of per-task score
  • artifacts/baseline_scores.json
  • aggregate mean score

CLI example:

python scripts/baseline_inference.py --model gpt-5-mini --seed 42

11) openenv.yaml (Required)

Include metadata and task registration:

id: bug-triage-openenv
name: Bug Triage OpenEnv
version: 0.1.0
entrypoint: openenv_bug_triage.env:BugTriageEnv
tags:
  - openenv
  - bug-triage
  - real-world
tasks:
  - id: bug_triage_easy
    difficulty: easy
  - id: bug_triage_medium
    difficulty: medium
  - id: bug_triage_hard
    difficulty: hard

Then validate with:

openenv validate

12) Suggested Project Structure

openenv-bug-triage/
  openenv_bug_triage/
    __init__.py
    env.py
    models.py
    grader.py
    reward.py
    tasks.py
    data/
      tasks/
        bug_triage_easy.json
        bug_triage_medium.json
        bug_triage_hard.json
  scripts/
    baseline_inference.py
  tests/
    test_env_api.py
    test_graders.py
    test_determinism.py
  openenv.yaml
  Dockerfile
  requirements.txt
  README.md

13) Docker + HF Space Deployment

13.1 Dockerfile Requirements

• Base image: python:3.11-slim
• install dependencies from requirements.txt
• copy source + data
• expose runtime command for environment app or API wrapper
• must run successfully with:

docker build -t bug-triage-openenv .
docker run --rm -p 7860:7860 bug-triage-openenv

13.2 Hugging Face Space

• Space type: Docker
• include README.md and openenv tag in metadata
• startup should pass health check and expose expected endpoint/process

14) README Requirements

README must include:

• environment motivation and real-world relevance
• observation and action schema definitions
• task descriptions + difficulty rationale
• reward design and scoring explanation
• local setup instructions
• validation command (openenv validate)
• baseline command and expected reproducible scores
• Docker and HF Space deployment steps

15) Acceptance Criteria Checklist

• Real-world bug triage simulation implemented
• Full OpenEnv API with typed models
• openenv.yaml present and valid
• 3 tasks (easy/medium/hard) with deterministic graders
• Dense reward shaping with partial progress + penalties
• Baseline inference script with reproducible results
• Working Dockerfile
• Deployable HF Space configuration
• README complete per requirements

16) Implementation Sequence (How to Build)

1. Define typed models in models.py.
2. Implement environment lifecycle in env.py (reset/step/state).
3. Add fixed task fixtures and loader in tasks.py.
4. Implement reward shaping logic in reward.py.
5. Implement deterministic grader in grader.py.
6. Register metadata in openenv.yaml; run openenv validate.
7. Build baseline runner and lock seed/model params.
8. Add tests for API contract, graders, determinism.
9. Add Dockerfile and verify container startup.
10. Publish to HF Space and verify runtime.

This sequence is the recommended path for turning the spec into a working submission.

1. Start Docker Desktop engine
2. Install validator tool: pip install openenv-core
3. Run: python precheck.py --space-url https://.hf.space