Spaces:
Sleeping
Sleeping
| title: PostMortem Incident Triage OpenEnv | |
| emoji: 🚨 | |
| colorFrom: red | |
| colorTo: yellow | |
| sdk: docker | |
| app_port: 8000 | |
| pinned: false | |
| license: bsd-3-clause | |
| tags: | |
| - openenv | |
| - rl | |
| - sre | |
| - incident-response | |
| base_path: /web | |
| # PostMortem — Live Incident Triage Environment | |
| An OpenEnv environment where an LLM agent plays an on-call SRE responding to a | |
| live production incident. Real-world task, typed OpenEnv spec, deterministic | |
| grader, three difficulty tiers, dense process-reward signal. | |
| ## The task | |
| On each episode the agent receives an alert. It must: | |
| 1. **ack** the incident (accept ownership) | |
| 2. **query_logs / query_metrics / query_traces** on services to gather evidence | |
| 3. **scope** the blast radius | |
| 4. **hypothesize** the root cause | |
| 5. **mitigate** (propose a concrete remediation) | |
| 6. **write_status** (post a customer-facing update) | |
| All six verbs are exposed as a single typed action: | |
| ```python | |
| PostmortemAction(tool="query_logs", args={"service": "api"}) | |
| ``` | |
| ## Action space | |
| | tool | args | effect | | |
| |----------------|-----------------------------------|-------------------------------------| | |
| | `ack` | `{}` | accept the incident (sub-goal 1) | | |
| | `query_logs` | `{"service": str}` | return recent log lines | | |
| | `query_metrics`| `{"service": str}` | return latest metrics | | |
| | `query_traces` | `{"trace_id": str}` | return distributed trace spans | | |
| | `scope` | `{"services": list[str]}` | declare blast radius (sub-goal 2) | | |
| | `hypothesize` | `{"root_cause": str}` | declare root cause (sub-goal 3) | | |
| | `mitigate` | `{"action": str}` | apply mitigation (sub-goal 4) | | |
| | `write_status` | `{"text": str}` | publish update, ends ep (sub-goal 5)| | |
| ## Observation space | |
| Key fields of `PostmortemObservation`: | |
| - `task_id`, `task_description`, `available_services`, `available_trace_ids` | |
| - `tool_result` — free text result of the last tool call | |
| - `subgoals` — bool dict `{acked, scoped, hypothesized, mitigated, written}` | |
| - `reward_so_far` — cumulative reward in [0, 1] | |
| - `steps_remaining`, `last_error` | |
| - `done`, `reward` (current step) | |
| ## Tasks (3 difficulty tiers) | |
| On each `reset()` the env rotates to the next scenario. Running three resets in | |
| a row covers all three tiers in order. | |
| | task_id | difficulty | incident | | |
| |-------------------|------------|--------------------------------------------------------------| | |
| | `easy_oom` | easy | `api` OOM-killed; cause directly visible in logs | | |
| | `medium_cascade` | medium | checkout latency cascade; must correlate trace across 3 svcs | | |
| | `hard_dns` | hard | 503s blamed on fresh `api` deploy, real cause is upstream DNS| | |
| ## Reward design | |
| The reward is a **5-stage process-reward ladder** in `[0, 1]`: | |
| ``` | |
| ack +0.10 (granted on first successful ack) | |
| scope +0.20 × Jaccard(agent_services, gold_services) | |
| hypothesize +0.20 × keyword_fraction(agent_text, gold_hypothesis_keywords) | |
| mitigate +0.20 × keyword_fraction(agent_text, gold_mitigation_keywords) | |
| write_status +0.30 × keyword_fraction(agent_text, gold_writeup_keywords) | |
| ``` | |
| Each sub-goal is awarded once. The grader is fully **deterministic** — no LLM | |
| judge, no randomness. Partial credit gives a smooth gradient. The episode | |
| terminates when `write_status` fires or after `MAX_STEPS = 12`. | |
| ## Setup | |
| ```bash | |
| pip install openenv-core | |
| openenv build . # build Docker image | |
| python inference.py # run baseline (3 scenarios) | |
| ``` | |
| ### Required environment variables | |
| | var | default | notes | | |
| |----------------|------------------------------------------------|-------| | |
| | `HF_TOKEN` | (required) | HuggingFace token, also used as the OpenAI client API key | | |
| | `API_BASE_URL` | `https://router.huggingface.co/v1` | any OpenAI-compatible endpoint | | |
| | `MODEL_NAME` | `Qwen/Qwen2.5-72B-Instruct` | any chat model | | |
| | `IMAGE_NAME` | `postmortem_env-env:latest` | docker tag of the env image | | |
| ## Baseline reproduction | |
| ```bash | |
| export HF_TOKEN=hf_... | |
| export IMAGE_NAME=postmortem_env-env:latest | |
| python inference.py | |
| ``` | |
| Emits strict `[START] / [STEP] / [END]` lines, one `[END]` per task. | |
| ## Resource budget | |
| Well within the hackathon limits of **2 vCPU / 8 GB RAM**, and completes the | |
| 3-task sweep in **well under 20 minutes** (dominated by LLM latency, ≤ 36 LLM | |
| calls total). | |
| ## License | |
| BSD-3-Clause (matches OpenEnv core). | |