Spaces:
Sleeping
Sleeping
| # Bug Triage OpenEnv - Specification | |
| ## 1) Goal | |
| Build a complete OpenEnv environment that simulates **real-world software bug triage** and supports learning through: | |
| - `reset()` | |
| - `step(action)` | |
| - `state()` | |
| The environment must include deterministic tasks, programmatic graders, reward shaping, reproducible baseline scoring, and deployment artifacts for Hugging Face Spaces. | |
| ## 2) Real-World Simulation | |
| ### Domain | |
| A software organization receives incoming bug reports from users, QA, and automated monitoring. | |
| The agent acts as a triage engineer responsible for: | |
| - assigning severity and priority | |
| - identifying duplicates | |
| - routing bugs to correct team/component | |
| - requesting missing information when needed | |
| - deciding whether to close/defer/escalate | |
| ### Why this is real-world | |
| Bug triage is a common operational workflow in engineering teams and has concrete measurable outcomes: SLA compliance, issue resolution speed, and triage correctness. | |
| ## 3) Environment API (OpenEnv Spec) | |
| Implement typed Pydantic models and standard methods. | |
| ### 3.1 Typed Models | |
| Create: | |
| - `ObservationModel` | |
| - `ActionModel` | |
| - `RewardModel` | |
| - `StateModel` (for `state()`) | |
| All models should be explicit and strictly typed (`Literal`, `Enum`, bounded numeric fields where possible). | |
| ### 3.2 Method Contracts | |
| - `reset(task_id: str | None = None, seed: int | None = None) -> ObservationModel` | |
| - `step(action: ActionModel) -> tuple[ObservationModel, RewardModel, bool, dict]` | |
| - `state() -> StateModel` | |
| ### 3.3 Step Semantics | |
| Each step processes exactly one action against the current ticket queue and returns: | |
| - updated observation | |
| - reward details (dense + cumulative) | |
| - `done` when success/failure/step-budget reached | |
| - `info` with grader signals and diagnostics | |
| ## 4) Data Model | |
| Each ticket record should include: | |
| - `ticket_id: str` | |
| - `title: str` | |
| - `description: str` | |
| - `reporter_type: Literal["user","qa","monitoring"]` | |
| - `service: str` | |
| - `component_candidates: list[str]` | |
| - `created_at: datetime` | |
| - `customer_tier: Literal["free","pro","enterprise"]` | |
| - `repro_steps_present: bool` | |
| - `logs_present: bool` | |
| - `attachments_count: int` | |
| - `suspected_duplicate_ids: list[str]` | |
| Hidden ground truth per ticket: | |
| - `true_severity: Literal["sev0","sev1","sev2","sev3"]` | |
| - `true_priority: Literal["p0","p1","p2","p3"]` | |
| - `true_component: str` | |
| - `true_assignee_team: str` | |
| - `duplicate_of: str | null` | |
| - `needs_more_info: bool` | |
| ## 5) Action Space | |
| Use a single structured action schema with `action_type` plus typed payload: | |
| - `classify`: set severity + priority + component | |
| - `assign`: set owner team | |
| - `mark_duplicate`: set canonical ticket id | |
| - `request_info`: ask for missing repro/logs | |
| - `defer`: move to backlog with reason | |
| - `close`: close with reason (`invalid`, `won't_fix`, `cannot_reproduce`, `resolved`) | |
| - `escalate_incident`: escalate urgent production impact | |
| - `next_ticket`: move focus pointer | |
| Validation rules: | |
| - invalid field combos rejected with penalty | |
| - references to unknown ticket IDs rejected with penalty | |
| - destructive actions (`close`, `defer`) penalized if inconsistent with truth | |
| ## 6) Observation Space | |
| Observation should include: | |
| - focused ticket summary (`current_ticket`) | |
| - queue stats (`remaining_count`, `urgent_count`, SLA-at-risk count) | |
| - prior actions and outcomes (`last_action_result`) | |
| - available teams/components | |
| - step budget (`steps_used`, `steps_remaining`) | |
| - partial score breakdown (optional but recommended for agent learning) | |
| Observation must not expose hidden labels directly. | |
| ## 7) Reward Function (Meaningful + Dense) | |
| Reward range per step: clamp to `[-1.0, 1.0]`. | |
| Episode final score normalized to `[0.0, 1.0]`. | |
| ### 7.1 Dense Progress Signals | |
| Suggested additive components: | |
| - `+0.20` correct severity | |
| - `+0.15` correct priority | |
| - `+0.15` correct component | |
| - `+0.10` correct team assignment | |
| - `+0.15` correct duplicate linking | |
| - `+0.10` correct request for more info | |
| - `+0.15` correct escalation for sev0/sev1 production incident | |
| ### 7.2 Negative Signals | |
| - `-0.20` incorrect close/defer on valid bug | |
| - `-0.15` missed critical escalation | |
| - `-0.05` invalid action schema | |
| - `-0.02` repeated no-op / loop behavior | |
| - `-0.01` unnecessary ticket switches | |
| ### 7.3 Terminal Bonus/Penalty | |
| - `+0.10` if all critical tickets triaged within budget | |
| - `-0.10` if step budget exhausted with high-severity untriaged tickets | |
| ## 8) Tasks and Graders (Easy / Medium / Hard) | |
| Implement 3 fixed tasks with deterministic fixtures and grader logic. | |
| ## 8.1 Task A - Easy (`bug_triage_easy`) | |
| Scenario: | |
| - 8 tickets | |
| - mostly clear reports | |
| - minimal duplicates | |
| - obvious component mapping | |
| Grader criteria: | |
| - weighted accuracy on severity/priority/component/team | |
| - no more than 1 major mistake | |
| - deterministic score in `[0,1]` | |
| ## 8.2 Task B - Medium (`bug_triage_medium`) | |
| Scenario: | |
| - 15 tickets | |
| - mixed-quality reports | |
| - several duplicates | |
| - some missing repro details | |
| - limited step budget | |
| Grader criteria: | |
| - weighted label correctness | |
| - duplicate resolution F1-like score | |
| - correct use of `request_info` | |
| - loop/destructive penalties applied | |
| ## 8.3 Task C - Hard (`bug_triage_hard`) | |
| Scenario: | |
| - 25 tickets | |
| - noisy and ambiguous text | |
| - conflicting signals | |
| - strict SLA pressure for enterprise + high severity tickets | |
| - trade-offs under capacity constraints | |
| Grader criteria: | |
| - strict weighting on sev0/sev1 correctness | |
| - SLA-risk handling score | |
| - escalation correctness | |
| - global policy quality (avoid destructive shortcuts) | |
| ## 8.4 Grader Output Contract | |
| Each grader returns: | |
| - `score: float` in `[0.0, 1.0]` | |
| - `subscores: dict[str, float]` | |
| - `mistakes: list[str]` | |
| - `passed: bool` (optional threshold, e.g., `>= 0.75`) | |
| ## 9) Determinism and Reproducibility | |
| - Fixed dataset fixtures per task (`data/tasks/*.json`) | |
| - Deterministic shuffling controlled by seed | |
| - deterministic grader logic (no LLM-in-the-loop grading) | |
| - baseline run outputs JSON artifact with exact scores | |
| Recommended defaults: | |
| - `DEFAULT_SEED=42` | |
| - model temperature `0` | |
| ## 10) Baseline Inference Script | |
| Create `scripts/baseline_inference.py`: | |
| - Reads `OPENAI_API_KEY` from environment | |
| - Runs all 3 tasks end-to-end | |
| - Uses fixed prompt template and deterministic model params | |
| - Writes: | |
| - console table of per-task score | |
| - `artifacts/baseline_scores.json` | |
| - aggregate mean score | |
| CLI example: | |
| ```bash | |
| python scripts/baseline_inference.py --model gpt-5-mini --seed 42 | |
| ``` | |
| ## 11) openenv.yaml (Required) | |
| Include metadata and task registration: | |
| ```yaml | |
| id: bug-triage-openenv | |
| name: Bug Triage OpenEnv | |
| version: 0.1.0 | |
| entrypoint: openenv_bug_triage.env:BugTriageEnv | |
| tags: | |
| - openenv | |
| - bug-triage | |
| - real-world | |
| tasks: | |
| - id: bug_triage_easy | |
| difficulty: easy | |
| - id: bug_triage_medium | |
| difficulty: medium | |
| - id: bug_triage_hard | |
| difficulty: hard | |
| ``` | |
| Then validate with: | |
| ```bash | |
| openenv validate | |
| ``` | |
| ## 12) Suggested Project Structure | |
| ```text | |
| openenv-bug-triage/ | |
| openenv_bug_triage/ | |
| __init__.py | |
| env.py | |
| models.py | |
| grader.py | |
| reward.py | |
| tasks.py | |
| data/ | |
| tasks/ | |
| bug_triage_easy.json | |
| bug_triage_medium.json | |
| bug_triage_hard.json | |
| scripts/ | |
| baseline_inference.py | |
| tests/ | |
| test_env_api.py | |
| test_graders.py | |
| test_determinism.py | |
| openenv.yaml | |
| Dockerfile | |
| requirements.txt | |
| README.md | |
| ``` | |
| ## 13) Docker + HF Space Deployment | |
| ### 13.1 Dockerfile Requirements | |
| - Base image: `python:3.11-slim` | |
| - install dependencies from `requirements.txt` | |
| - copy source + data | |
| - expose runtime command for environment app or API wrapper | |
| - must run successfully with: | |
| ```bash | |
| docker build -t bug-triage-openenv . | |
| docker run --rm -p 7860:7860 bug-triage-openenv | |
| ``` | |
| ### 13.2 Hugging Face Space | |
| - Space type: Docker | |
| - include `README.md` and `openenv` tag in metadata | |
| - startup should pass health check and expose expected endpoint/process | |
| ## 14) README Requirements | |
| README must include: | |
| - environment motivation and real-world relevance | |
| - observation and action schema definitions | |
| - task descriptions + difficulty rationale | |
| - reward design and scoring explanation | |
| - local setup instructions | |
| - validation command (`openenv validate`) | |
| - baseline command and expected reproducible scores | |
| - Docker and HF Space deployment steps | |
| ## 15) Acceptance Criteria Checklist | |
| - [ ] Real-world bug triage simulation implemented | |
| - [ ] Full OpenEnv API with typed models | |
| - [ ] `openenv.yaml` present and valid | |
| - [ ] 3 tasks (easy/medium/hard) with deterministic graders | |
| - [ ] Dense reward shaping with partial progress + penalties | |
| - [ ] Baseline inference script with reproducible results | |
| - [ ] Working Dockerfile | |
| - [ ] Deployable HF Space configuration | |
| - [ ] README complete per requirements | |
| ## 16) Implementation Sequence (How to Build) | |
| 1. Define typed models in `models.py`. | |
| 2. Implement environment lifecycle in `env.py` (`reset/step/state`). | |
| 3. Add fixed task fixtures and loader in `tasks.py`. | |
| 4. Implement reward shaping logic in `reward.py`. | |
| 5. Implement deterministic grader in `grader.py`. | |
| 6. Register metadata in `openenv.yaml`; run `openenv validate`. | |
| 7. Build baseline runner and lock seed/model params. | |
| 8. Add tests for API contract, graders, determinism. | |
| 9. Add Dockerfile and verify container startup. | |
| 10. Publish to HF Space and verify runtime. | |
| This sequence is the recommended path for turning the spec into a working submission. | |
| 1. Start Docker Desktop engine | |
| 2. Install validator tool: pip install openenv-core | |
| 3. Run: python precheck.py --space-url https://<your-space>.hf.space |