Initial submission: OpenEnv-Sentinel SRE triage environment

.dockerignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+outputs/
+.env
+.venv/
+venv/
+*.log
+.DS_Store

.gitignore

@@ -0,0 +1,17 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+outputs/
+.env
+.venv/
+venv/
+*.log
+.DS_Store
+.inference_run_count
+Pipfile.lock
+inference_azure_local.py
+pat_token

Dockerfile

@@ -0,0 +1,27 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install curl for healthcheck
+RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
+
+# Install Python dependencies
+COPY server/requirements.txt /tmp/requirements.txt
+RUN pip install --no-cache-dir -r /tmp/requirements.txt && rm /tmp/requirements.txt
+
+# Copy environment code
+COPY models.py /app/models.py
+COPY __init__.py /app/__init__.py
+COPY client.py /app/client.py
+COPY server/ /app/server/
+COPY scenarios/ /app/scenarios/
+COPY tools/ /app/tools/
+COPY grading/ /app/grading/
+COPY openenv.yaml /app/openenv.yaml
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
+    CMD curl -f http://localhost:8000/health || exit 1
+
+EXPOSE 8000
+CMD ["uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "8000"]

Pipfile

@@ -0,0 +1,16 @@
+[[source]]
+url = "https://pypi.org/simple"
+verify_ssl = true
+name = "pypi"
+
+[packages]
+openenv-sentinel = {extras = ["dev"], file = ".", editable = true}
+openai = "*"
+httpx = "*"
+websockets = "*"
+
+[dev-packages]
+
+[requires]
+python_version = "3.12"
+python_full_version = "3.12.4"

README.md

@@ -0,0 +1,141 @@
+---
+title: OpenEnv-Sentinel
+emoji: 🚨
+colorFrom: red
+colorTo: yellow
+sdk: docker
+pinned: false
+app_port: 8000
+tags:
+  - openenv
+---
+
+# OpenEnv-Sentinel: SRE Incident Triage Environment
+
+An OpenEnv environment that simulates SRE incident triage. An AI agent receives a degraded system state and must use diagnostic tools to identify the root cause and recommend a fix.
+
+## Quick Start
+
+```bash
+pip install -e .
+uvicorn server.app:app --host 0.0.0.0 --port 8000
+```
+
+Or with Docker:
+
+```bash
+docker build -t sentinel-env -f server/Dockerfile .
+docker run -p 8000:8000 sentinel-env
+```
+
+## Action Space
+
+```python
+class SentinelAction(Action):
+    tool_name: str    # Tool to invoke
+    parameters: dict  # Tool-specific parameters
+```
+
+### Available Tools
+
+| Tool | Parameters | Description |
+|---|---|---|
+| `query_logs` | `service`, `query`, `severity` | Search service logs |
+| `query_metrics` | `service`, `metric` | Get time-series metrics (cpu/memory/error_rate/latency/connections) |
+| `get_service_status` | `service` | Service health, uptime, errors |
+| `get_dependency_map` | `service` (optional) | Service dependency graph |
+| `consult_runbook` | `topic` | SOP/runbook lookup |
+| `check_recent_changes` | `service` (optional) | Recent deployments/config changes |
+| `submit_resolution` | `root_cause`, `affected_service`, `recommendation` | Submit final answer (ends episode) |
+
+## Observation Space
+
+```python
+class SentinelObservation(Observation):
+    incident_summary: str       # Alert description
+    tool_output: str            # Result from last tool call
+    available_tools: list[str]  # Available tool names
+    step_number: int            # Current step (0-indexed)
+    max_steps: int              # Episode limit (20)
+    cumulative_reward: float    # Running reward total
+    last_action_error: str      # Error message if action was invalid
+    done: bool                  # Episode finished?
+    reward: float | None        # Per-step reward
+```
+
+## Tasks
+
+### Task 1 — The Smoking Gun (Easy)
+**Alert:** payment-api returning HTTP 500 errors. Straightforward single-service crash with a clear root cause in logs and deploy history. Optimal: 2–3 tool calls.
+
+### Task 2 — The Upstream Culprit (Medium)
+**Alert:** checkout-service p99 latency > 5 seconds. Requires tracing a dependency chain to find the real culprit (inventory-service OOM). Optimal: 4–6 tool calls.
+
+### Task 3 — The Cascading Failure (Hard)
+**Alert:** Multiple services degraded simultaneously. A long-running analytics query exhausts the PostgreSQL connection pool, cascading through auth, user-profile, and notification services. Includes red herrings. Optimal: 6–10 tool calls.
+
+## Scoring
+
+Each task is scored 0.0–1.0 using deterministic keyword-based grading:
+- **Root cause identification** (weighted by task)
+- **Correct affected service** identification
+- **Actionable recommendation**
+- **Efficiency bonus** (fewer steps = higher score)
+- **Destructive penalty** (recommending harmful actions = score deduction)
+
+Per-step rewards provide partial credit signal:
+- Relevant tool call: +0.12
+- Irrelevant tool call: −0.02
+- Repeated call: −0.05
+- Invalid action: −0.03
+- Step cost: −0.01
+
+## Running Inference
+
+Uses `OpenAI(base_url=...)` — compatible with HF Inference, OpenAI, and any
+OpenAI-compatible API. Azure OpenAI is supported via `AzureOpenAI` client.
+
+```bash
+# Environment server URL
+export ENV_URL=http://localhost:8000
+
+# LLM config (defaults to HF router)
+export API_BASE_URL=https://router.huggingface.co/v1  # default, can omit
+export MODEL_NAME=openai/gpt-oss-120b:novita           # default, can omit
+export API_KEY=your-key      # or HF_TOKEN or OPENAI_API_KEY
+
+# Azure OpenAI (optional, set AZURE_OPENAI_ENDPOINT to enable)
+# export AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
+# export AZURE_OPENAI_API_KEY=your-azure-key
+# export MODEL_NAME=your-deployment-name
+
+pip install openai websockets
+python inference.py
+```
+
+Output:
+```
+Task 1: 0.85
+Task 2: 0.65
+Task 3: 0.40
+Average: 0.63
+```
+
+## Baseline Scores
+
+| Task | GPT-4o (expected) | Open LLM (expected) |
+|---|---|---|
+| Task 1 (Easy) | 0.80–0.95 | 0.60–0.80 |
+| Task 2 (Medium) | 0.60–0.80 | 0.40–0.60 |
+| Task 3 (Hard) | 0.30–0.60 | 0.15–0.35 |
+
+## API Endpoints
+
+| Endpoint | Method | Description |
+|---|---|---|
+| `/health` | GET | Health check |
+| `/reset` | POST | Reset environment (`{"task_id": 1\|2\|3}`) |
+| `/step` | POST | Execute action (`{"action": {...}}`) |
+| `/state` | GET | Get current state |
+| `/schema` | GET | JSON schemas for action/observation/state |
+| `/ws` | WebSocket | Persistent session |

USER_GUIDE.md

@@ -0,0 +1,563 @@
+# OpenEnv-Sentinel — User Guide
+
+Step-by-step guide for running, validating, and deploying the SRE Incident Triage environment.
+
+---
+
+## Table of Contents
+
+1. [Prerequisites](#1-prerequisites)
+2. [Local Setup](#2-local-setup)
+3. [Running the Server Locally](#3-running-the-server-locally)
+4. [Manual Validation — Local Server](#4-manual-validation--local-server)
+5. [Docker Build & Validation](#5-docker-build--validation)
+6. [Running Inference (LLM Agent)](#6-running-inference-llm-agent)
+7. [OpenEnv Validate](#7-openenv-validate)
+8. [Deploy to Hugging Face Spaces](#8-deploy-to-hugging-face-spaces)
+9. [Troubleshooting](#9-troubleshooting)
+
+---
+
+## 1. Prerequisites
+
+| Tool | Version | Purpose |
+|---|---|---|
+| Python | ≥ 3.10 | Runtime |
+| pip / pipenv | Latest | Dependency management |
+| Docker | Latest | Container build & test |
+| Git | Latest | Version control, HF push |
+| huggingface-cli | Latest | HF Spaces deployment |
+| openenv-core CLI | ≥ 0.2.3 | `openenv validate` / `openenv push` |
+
+Install the OpenEnv CLI and Hugging Face CLI:
+
+```bash
+pip install openenv-core huggingface-hub[cli]
+```
+
+---
+
+## 2. Local Setup
+
+### Option A — pip (quick)
+
+```bash
+cd openenv-sentinel
+pip install -e ".[dev,inference]"
+```
+
+### Option B — pipenv (isolated)
+
+```bash
+cd openenv-sentinel
+pipenv install --python 3.12
+pipenv install -e ".[dev]"
+pipenv install openai httpx websockets
+pipenv shell
+```
+
+All subsequent commands assume you are inside the virtual environment.
+
+---
+
+## 3. Running the Server Locally
+
+Start the FastAPI server on port 8000:
+
+```bash
+uvicorn server.app:app --host 0.0.0.0 --port 8000
+```
+
+You should see:
+
+```
+INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
+```
+
+Verify with:
+
+```bash
+curl http://localhost:8000/health
+# → {"status":"ok"}
+
+curl http://localhost:8000/schema
+# → JSON with action, observation, state schemas
+```
+
+---
+
+## 4. Manual Validation — Local Server
+
+With the server running (from step 3), validate the environment in a **second terminal**.
+
+### 4.1 Automated test script
+
+The quickest way to validate all 3 tasks end-to-end:
+
+```bash
+pip install websockets httpx   # if not already installed
+python test_local.py
+```
+
+Expected output:
+
+```
+Health: {'status': 'ok'}
+Schema: action fields=['tool_name', 'parameters']
+
+==================================================
+TASK 1
+==================================================
+  Reset OK: CRITICAL: payment-api returning HTTP 500 errors...
+  Step 1 (status payment-api): reward=0.11
+  Step 2 (logs payment-api): reward=0.11
+  Resolution: score=0.75, done=True
+  State: final_score=1.0, root_cause_correct=True, recommendation_correct=True
+
+... (Tasks 2 & 3 similar) ...
+
+✅ ALL TESTS PASSED
+```
+
+### 4.2 Manual cURL validation (HTTP endpoints)
+
+> **Note:** HTTP endpoints are stateless — each request creates a fresh environment
+> instance. Use these for single-shot checks only. For multi-step episodes, use
+> the WebSocket endpoint (section 4.3).
+
+**Health check:**
+
+```bash
+curl http://localhost:8000/health
+```
+
+**Schema check:**
+
+```bash
+curl http://localhost:8000/schema | python -m json.tool
+```
+
+**Reset (single-shot):**
+
+```bash
+curl -X POST http://localhost:8000/reset \
+  -H "Content-Type: application/json" \
+  -d '{"task_id": 1}'
+```
+
+### 4.3 Manual WebSocket validation (stateful sessions)
+
+Multi-step episodes require WebSocket because the server maintains session state
+across messages. Install `websocat` or use Python:
+
+**Using Python interactively:**
+
+```python
+import asyncio, json, websockets
+
+async def manual_test():
+    async with websockets.connect("ws://localhost:8000/ws") as ws:
+        # 1. Reset to Task 1
+        await ws.send(json.dumps({"type": "reset", "data": {"task_id": 1}}))
+        resp = json.loads(await ws.recv())
+        print("Reset:", json.dumps(resp["data"]["observation"]["incident_summary"]))
+
+        # 2. Call a diagnostic tool
+        await ws.send(json.dumps({
+            "type": "step",
+            "data": {
+                "tool_name": "get_service_status",
+                "parameters": {"service": "payment-api"}
+            }
+        }))
+        resp = json.loads(await ws.recv())
+        print("Step 1:", resp["data"]["observation"]["tool_output"][:200])
+
+        # 3. Submit resolution
+        await ws.send(json.dumps({
+            "type": "step",
+            "data": {
+                "tool_name": "submit_resolution",
+                "parameters": {
+                    "root_cause": "Missing DB_CONNECTION_STRING after v2.3.1 deploy",
+                    "affected_service": "payment-api",
+                    "recommendation": "Rollback to v2.3.0 or set the env var"
+                }
+            }
+        }))
+        resp = json.loads(await ws.recv())
+        print("Done:", resp["data"]["done"], "Score:", resp["data"]["reward"])
+
+        # 4. Get final state
+        await ws.send(json.dumps({"type": "state"}))
+        resp = json.loads(await ws.recv())
+        print("Final score:", resp["data"]["final_score"])
+
+asyncio.run(manual_test())
+```
+
+**Using websocat (CLI tool):**
+
+```bash
+brew install websocat   # macOS
+websocat ws://localhost:8000/ws
+```
+
+Then type JSON messages line by line:
+
+```json
+{"type": "reset", "data": {"task_id": 1}}
+{"type": "step", "data": {"tool_name": "get_service_status", "parameters": {"service": "payment-api"}}}
+{"type": "step", "data": {"tool_name": "submit_resolution", "parameters": {"root_cause": "Missing DB_CONNECTION_STRING", "affected_service": "payment-api", "recommendation": "Rollback to v2.3.0"}}}
+{"type": "state"}
+```
+
+### 4.4 What to check
+
+| Check | Expected |
+|---|---|
+| `/health` returns 200 | `{"status": "ok"}` |
+| `/schema` returns action/observation/state schemas | Three top-level keys with JSON Schema properties |
+| Reset with `task_id` 1, 2, 3 | Returns `incident_summary`, `available_tools` (7 tools), `done: false` |
+| Diagnostic tool steps | Returns `tool_output` (non-empty), per-step `reward` |
+| `submit_resolution` | Sets `done: true`, returns graded `reward` |
+| State after resolution | `final_score` between 0.0–1.0, `root_cause_correct` bool |
+| All 3 tasks produce scores > 0.0 with good resolutions | Task 1 ≈ 1.0, Task 2 ≈ 1.0, Task 3 ≈ 1.0 (with ideal answers) |
+
+---
+
+## 5. Docker Build & Validation
+
+### 5.1 Build the image
+
+```bash
+docker build -t sentinel-env:latest -f server/Dockerfile .
+```
+
+### 5.2 Run the container
+
+```bash
+docker run -p 8000:8000 sentinel-env:latest
+```
+
+The server starts on port 8000 inside the container, mapped to your host.
+
+### 5.3 Validate against the container
+
+Once the container is running, all the same validation steps from section 4 work:
+
+```bash
+# Health check
+curl http://localhost:8000/health
+
+# Run the automated test suite
+python test_local.py
+
+# Or run inference against the containerised server
+ENV_URL=http://localhost:8000 python inference.py
+```
+
+### 5.4 Docker — useful commands
+
+```bash
+# Build with no cache (clean rebuild)
+docker build --no-cache -t sentinel-env:latest -f server/Dockerfile .
+
+# Run in background
+docker run -d --name sentinel -p 8000:8000 sentinel-env:latest
+
+# View logs
+docker logs -f sentinel
+
+# Stop and remove
+docker stop sentinel && docker rm sentinel
+
+# Check image size (should be < 500MB)
+docker images sentinel-env
+```
+
+---
+
+## 6. Running Inference (LLM Agent)
+
+The inference script drives an LLM through all 3 tasks via WebSocket.
+
+### 6.1 Set environment variables
+
+The inference script supports **HF Inference** (default), **OpenAI**, and **Azure OpenAI** endpoints.
+
+> **Important:** `ENV_URL` is the Sentinel environment server. `API_BASE_URL` is
+> the LLM API endpoint (matching the official OpenEnv inference pattern).
+
+**Option A — HF Inference API (default, for hackathon submission):**
+
+```bash
+export ENV_URL=http://localhost:8000                        # env server
+export API_BASE_URL=https://router.huggingface.co/v1        # default, can omit
+export MODEL_NAME=openai/gpt-oss-120b:novita                # default, can omit
+export HF_TOKEN=hf_...                                      # or API_KEY
+```
+
+**Option B — OpenAI:**
+
+```bash
+export ENV_URL=http://localhost:8000
+export API_BASE_URL=https://api.openai.com/v1
+export MODEL_NAME=gpt-4o
+export API_KEY=sk-...
+```
+
+**Option C — Azure OpenAI (for local/enterprise testing):**
+
+```bash
+export ENV_URL=http://localhost:8000
+export AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
+export AZURE_OPENAI_API_KEY=your-azure-key
+export MODEL_NAME=your-deployment-name      # Azure deployment name
+export AZURE_OPENAI_API_VERSION=2024-12-01-preview  # optional, this is the default
+```
+
+> When `AZURE_OPENAI_ENDPOINT` is set, the script uses `AzureOpenAI` client.
+> Otherwise it uses `OpenAI(base_url=API_BASE_URL, api_key=...)` — which
+> covers both HF router and direct OpenAI.
+
+### 6.2 Install inference dependencies
+
+```bash
+pip install openai websockets
+```
+
+### 6.3 Run
+
+```bash
+python inference.py
+```
+
+Expected output:
+
+```
+==================================================
+Running Task 1...
+==================================================
+Task 1: 0.85
+
+==================================================
+Running Task 2...
+==================================================
+Task 2: 0.65
+
+==================================================
+Running Task 3...
+==================================================
+Task 3: 0.40
+
+==================================================
+Task 1: 0.85
+Task 2: 0.65
+Task 3: 0.40
+Average: 0.63
+==================================================
+```
+
+### 6.4 Inference against a remote HF Space
+
+```bash
+# HF model via HF router (hackathon default)
+export ENV_URL=https://your-username-sentinel-env.hf.space
+export HF_TOKEN=hf_...
+python inference.py
+
+# OpenAI model
+export ENV_URL=https://your-username-sentinel-env.hf.space
+export API_BASE_URL=https://api.openai.com/v1
+export MODEL_NAME=gpt-4o
+export API_KEY=sk-...
+python inference.py
+
+# Azure OpenAI
+export ENV_URL=https://your-username-sentinel-env.hf.space
+export AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
+export AZURE_OPENAI_API_KEY=your-azure-key
+export MODEL_NAME=your-deployment-name
+python inference.py
+```
+
+---
+
+## 7. OpenEnv Validate
+
+Run the official OpenEnv validation to confirm spec compliance:
+
+```bash
+openenv validate
+```
+
+This checks:
+- `openenv.yaml` manifest is valid
+- The app entry point (`server.app:app`) is importable
+- A `main()` function exists in the script entry point
+- `uv.lock` is present and up to date
+
+If `uv.lock` is missing or stale:
+
+```bash
+pip install uv
+uv lock
+openenv validate
+```
+
+---
+
+## 8. Deploy to Hugging Face Spaces
+
+### 8.1 Login to Hugging Face
+
+```bash
+huggingface-cli login
+# Paste your HF token when prompted (needs write access)
+```
+
+### 8.2 Option A — `openenv push` (recommended)
+
+```bash
+openenv push
+```
+
+This reads `openenv.yaml` and pushes the environment as a Docker Space tagged
+with `openenv`.
+
+### 8.3 Option B — Manual HF Spaces deployment
+
+**Step 1: Create the Space**
+
+Go to https://huggingface.co/new-space and create a new Space:
+
+- **Space name:** `sentinel-env` (or any name)
+- **SDK:** Docker
+- **Hardware:** CPU basic (2 vCPU, 16GB RAM — free tier)
+- **Visibility:** Public
+
+**Step 2: Clone the Space repo**
+
+```bash
+git clone https://huggingface.co/spaces/YOUR_USERNAME/sentinel-env hf-space
+cd hf-space
+```
+
+**Step 3: Copy project files**
+
+```bash
+# Copy all source files
+cp -r /path/to/openenv-sentinel/{models.py,__init__.py,client.py,inference.py} .
+cp -r /path/to/openenv-sentinel/{server,scenarios,tools,grading} .
+cp /path/to/openenv-sentinel/openenv.yaml .
+cp /path/to/openenv-sentinel/pyproject.toml .
+cp /path/to/openenv-sentinel/README.md .
+
+# The Dockerfile must be at the repo root for HF Spaces
+cp /path/to/openenv-sentinel/server/Dockerfile .
+```
+
+> **Important:** HF Spaces expects `Dockerfile` at the repository root. The
+> COPY paths inside the Dockerfile already reference files relative to the
+> build context (repo root), so no changes are needed.
+
+**Step 4: Push to HF**
+
+```bash
+git add .
+git commit -m "Deploy OpenEnv-Sentinel"
+git push
+```
+
+**Step 5: Verify deployment**
+
+The Space builds automatically. Once running:
+
+```bash
+curl https://YOUR_USERNAME-sentinel-env.hf.space/health
+# → {"status": "ok"}
+```
+
+### 8.4 Verify the deployed Space
+
+```bash
+# Health
+curl https://YOUR_USERNAME-sentinel-env.hf.space/health
+
+# Schema
+curl https://YOUR_USERNAME-sentinel-env.hf.space/schema
+
+# Run test_local.py against the Space (edit BASE_HTTP/BASE_WS in the file)
+# Or run inference:
+ENV_URL=https://YOUR_USERNAME-sentinel-env.hf.space \
+HF_TOKEN=hf_... \
+python inference.py
+```
+
+### 8.5 HF Spaces tips
+
+- **Cold starts:** Free-tier Spaces sleep after inactivity. First request takes ~30s.
+- **Logs:** View build & runtime logs in the Space's "Logs" tab on HF.
+- **Environment variables:** Set secrets (like API keys) in Space Settings → Repository secrets.
+- **Tags:** Ensure the README frontmatter includes `tags: [openenv]` for hackathon discovery.
+- **Port:** The `app_port: 8000` in README frontmatter must match the `EXPOSE` in the Dockerfile.
+
+---
+
+## 9. Troubleshooting
+
+| Problem | Solution |
+|---|---|
+| `ModuleNotFoundError: No module named 'openenv'` | Run `pip install -e .` or `pip install openenv-core>=0.2.3` |
+| `openenv validate` fails with "no main() found" | Ensure `server/app.py` has a `def main()` function and `[project.scripts]` in pyproject.toml |
+| `openenv validate` fails with "uv.lock not found" | Run `pip install uv && uv lock` |
+| WebSocket connection refused | Server must be running (`uvicorn server.app:app --port 8000`) |
+| HTTP `/step` returns fresh state (not continuing episode) | HTTP endpoints are stateless. Use WebSocket `/ws` for multi-step episodes |
+| Docker build fails on `COPY` | Run `docker build` from the project root (not from `server/`) |
+| Docker healthcheck failing | Ensure `curl` is installed in the image (the Dockerfile does this) |
+| `inference.py` error: "ENV_URL required" | `export ENV_URL=http://localhost:8000` |
+| Azure OpenAI 401 / auth error | Verify `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_KEY`, and that `MODEL_NAME` matches your deployment name |
+| HF Space shows "Building" forever | Check the Logs tab for build errors. Common: missing files in COPY |
+| HF Space returns 502 | The app hasn't started yet (cold start) or crashed. Check runtime logs |
+| Task score is 0.0 | The resolution keywords didn't match. Check grading criteria in HACKATHON_PLAN.md §5 |
+| `websockets` not installed | `pip install websockets` |
+
+---
+
+## Quick Reference
+
+```bash
+# ── Local development ──
+pip install -e ".[dev,inference]"
+uvicorn server.app:app --port 8000          # start server
+python test_local.py                         # validate all 3 tasks
+openenv validate                             # check spec compliance
+
+# ── Docker ──
+docker build -t sentinel-env -f server/Dockerfile .
+docker run -p 8000:8000 sentinel-env
+
+# ── Inference (HF router — hackathon default) ──
+export ENV_URL=http://localhost:8000
+export HF_TOKEN=hf_...
+python inference.py
+
+# ── Inference (OpenAI) ──
+export ENV_URL=http://localhost:8000
+export API_BASE_URL=https://api.openai.com/v1
+export MODEL_NAME=gpt-4o
+export API_KEY=sk-...
+python inference.py
+
+# ── Inference (Azure OpenAI) ──
+export ENV_URL=http://localhost:8000
+export AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
+export AZURE_OPENAI_API_KEY=your-azure-key
+export MODEL_NAME=your-deployment-name
+python inference.py
+
+# ── Deploy ──
+huggingface-cli login
+openenv push
+```

__init__.py

@@ -0,0 +1,17 @@
+"""OpenEnv-Sentinel: SRE Incident Triage Environment."""
+
+__all__: list = []
+
+try:
+    from models import SentinelAction, SentinelObservation, SentinelState  # noqa: F401
+
+    __all__ += ["SentinelAction", "SentinelObservation", "SentinelState"]
+except Exception:
+    pass
+
+try:
+    from client import SentinelEnv  # noqa: F401
+
+    __all__ += ["SentinelEnv"]
+except Exception:
+    pass

client.py

@@ -0,0 +1,25 @@
+"""SentinelEnv client — connects to the Sentinel environment server."""
+
+from openenv.core import EnvClient, StepResult
+from .models import SentinelAction, SentinelObservation, SentinelState
+
+
+class SentinelEnv(EnvClient[SentinelAction, SentinelObservation, SentinelState]):
+    """Client for the Sentinel SRE Incident Triage Environment."""
+
+    def _step_payload(self, action: SentinelAction) -> dict:
+        return {
+            "tool_name": action.tool_name,
+            "parameters": action.param_dict(),
+        }
+
+    def _parse_result(self, payload: dict) -> StepResult[SentinelObservation]:
+        obs = SentinelObservation(**payload["observation"])
+        return StepResult(
+            observation=obs,
+            reward=payload.get("reward"),
+            done=payload.get("done", False),
+        )
+
+    def _parse_state(self, payload: dict) -> SentinelState:
+        return SentinelState(**payload)

conftest.py

@@ -0,0 +1 @@
+collect_ignore_glob = ["__init__.py", "client.py", "inference.py", "test_local.py"]

grading/__init__.py

@@ -0,0 +1 @@
+"""Grading package."""

grading/grader.py

@@ -0,0 +1,15 @@
+"""Terminal grader — delegates to the scenario's grade_resolution method."""
+
+
+def normalize_service_name(name: str) -> str:
+    """Normalize an affected_service string for comparison."""
+    return name.lower().strip().replace("_", "-")
+
+
+def grade(scenario, resolution: dict, step_count: int) -> dict:
+    """Grade a resolution using the scenario's grading logic.
+
+    Returns dict with keys: score (float 0-1), root_cause_correct (bool),
+    recommendation_correct (bool).
+    """
+    return scenario.grade_resolution(resolution, step_count)

grading/rewards.py

@@ -0,0 +1,61 @@
+"""Per-step reward calculator."""
+
+from typing import List
+
+from tools.registry import make_relevance_key
+
+REWARD_RELEVANT = 0.12
+REWARD_IRRELEVANT = -0.02
+REWARD_REPEATED = -0.05
+REWARD_INVALID = -0.03
+REWARD_STEP_COST = -0.01
+
+
+def _call_signature(tool_name: str, params: dict) -> str:
+    """Create a hashable signature for a tool call to detect repeats."""
+    return make_relevance_key(tool_name, params)
+
+
+def _is_relevant(
+    tool_name: str,
+    params: dict,
+    relevant_tools: List[str],
+) -> bool:
+    """Check if a tool call matches the scenario's relevance list.
+
+    Each entry is a colon-joined string like "query_logs:auth-service".
+    A match requires the computed key to appear in the relevance list.
+    """
+    key = make_relevance_key(tool_name, params)
+    return key in relevant_tools
+
+
+def compute_step_reward(
+    tool_name: str,
+    params: dict,
+    is_valid: bool,
+    relevant_tools: List[str],
+    previous_calls: List[str],
+) -> float:
+    """Compute the reward for a single step.
+
+    Returns the reward value.  Caller is responsible for appending the call
+    signature to previous_calls after calling this.
+    """
+    reward = REWARD_STEP_COST
+
+    if not is_valid:
+        reward += REWARD_INVALID
+        return reward
+
+    sig = _call_signature(tool_name, params)
+    if sig in previous_calls:
+        reward += REWARD_REPEATED
+        return reward
+
+    if _is_relevant(tool_name, params, relevant_tools):
+        reward += REWARD_RELEVANT
+    else:
+        reward += REWARD_IRRELEVANT
+
+    return reward

inference.py

@@ -0,0 +1,359 @@
+"""Baseline inference script for OpenEnv-Sentinel.
+
+Drives an LLM agent through all 3 SRE incident triage tasks.
+
+Environment variables:
+  ENV_URL          - Sentinel environment server URL (e.g. http://localhost:8000)
+  API_BASE_URL     - LLM API base URL (default: https://router.huggingface.co/v1)
+  MODEL_NAME       - Model or deployment name (default: openai/gpt-oss-120b:novita)
+  HF_TOKEN         - Hugging Face token (used as API key)
+  LOCAL_IMAGE_NAME - Docker image name when using from_docker_image() (optional)
+
+  # Azure OpenAI (optional, for local testing)
+  AZURE_OPENAI_ENDPOINT    - Set to enable AzureOpenAI client
+  AZURE_OPENAI_API_KEY     - Azure API key
+  AZURE_OPENAI_API_VERSION - Azure API version (default: 2024-12-01-preview)
+"""
+
+import asyncio
+import json
+import os
+import re
+import sys
+import time
+
+from openai import OpenAI, AzureOpenAI
+
+# ── configuration ───────────────────────────────────────────────────
+
+# Environment server URL (where the Sentinel env is running)
+ENV_URL = os.getenv("ENV_URL", "")
+
+# LLM configuration (aligned with official OpenEnv inference examples)
+API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
+MODEL_NAME = os.getenv("MODEL_NAME", "openai/gpt-oss-120b:novita")
+HF_TOKEN = os.getenv("HF_TOKEN")
+
+# Optional — if you use from_docker_image():
+LOCAL_IMAGE_NAME = os.getenv("LOCAL_IMAGE_NAME")
+
+# API key: prefer API_KEY, fall back to HF_TOKEN
+API_KEY = os.getenv("API_KEY") or HF_TOKEN or os.getenv("OPENAI_API_KEY", "")
+
+# Azure OpenAI (optional, for local testing with enterprise deployments)
+AZURE_OPENAI_ENDPOINT = os.getenv("AZURE_OPENAI_ENDPOINT", "")
+AZURE_OPENAI_API_KEY = os.getenv("AZURE_OPENAI_API_KEY", "")
+AZURE_OPENAI_API_VERSION = os.getenv("AZURE_OPENAI_API_VERSION", "2024-12-01-preview")
+AZURE_OPENAI_DEPLOYMENT = os.getenv("AZURE_OPENAI_DEPLOYMENT", "")
+
+TASK_TIMEOUT = 360  # 6 minutes per task
+MAX_PARSE_RETRIES = 3
+MAX_COMPLETION_TOKENS = 16384  # reasoning models need room for chain-of-thought
+
+SYSTEM_PROMPT = """You are an expert SRE agent triaging a production incident.
+You have access to diagnostic tools. Respond with ONLY a single JSON object — no markdown, no explanation, no extra text.
+
+Available tools:
+- get_service_status: {"tool_name": "get_service_status", "parameters": {"service": "<name>"}}
+- query_logs: {"tool_name": "query_logs", "parameters": {"service": "<name>", "query": "<text filter, use empty string for all logs>"}}
+- query_metrics: {"tool_name": "query_metrics", "parameters": {"service": "<name>", "metric": "<cpu|memory|error_rate|latency|connections>"}}
+- get_dependency_map: {"tool_name": "get_dependency_map", "parameters": {"service": "<name or omit for full map>"}}
+- consult_runbook: {"tool_name": "consult_runbook", "parameters": {"topic": "<search_topic>"}}
+- check_recent_changes: {"tool_name": "check_recent_changes", "parameters": {"service": "<name or omit for all>"}}
+- submit_resolution: {"tool_name": "submit_resolution", "parameters": {"root_cause": "<detailed explanation>", "affected_service": "<primary ROOT CAUSE service>", "recommendation": "<specific actionable fix>"}}
+
+INVESTIGATION PLAN — you have only 20 steps total, be extremely efficient:
+Step 1: get_dependency_map (no service param) to see full architecture
+Step 2: check_recent_changes (no service param) to see all recent deploys and changes
+Step 3-4: get_service_status for the UNHEALTHY/DEGRADED services mentioned in the incident
+Step 5-6: query_logs for unhealthy services (use "" as query to get all logs)
+Step 7-8: query_metrics for the suspicious root-cause service (error_rate, memory, connections)
+Step 9: submit_resolution with your findings
+
+CRITICAL RULES:
+- The ROOT CAUSE is often UPSTREAM — a dependency of the symptomatic service, not the alerted service itself
+- Look for: bad deployments, missing env vars, OOM/memory issues, connection pool exhaustion, long-running queries
+- affected_service MUST be the root-cause service, NOT the symptom service
+- root_cause must mention specific service names, error types, versions, and technical details
+- recommendation must be specific and actionable (e.g. rollback, increase memory limit, kill query, set timeout)
+- Do NOT repeat the same tool call — you already have that data
+- You MUST call submit_resolution by step 10 at the latest — do not keep investigating
+- Respond with ONLY a JSON object. No markdown fences, no explanation."""
+
+FORCE_RESOLUTION_PROMPT = """URGENT: You MUST call submit_resolution NOW. No more investigation.
+Synthesize everything you have gathered. Your response MUST be ONLY:
+{"tool_name": "submit_resolution", "parameters": {"root_cause": "<detailed with service names, errors, versions>", "affected_service": "<the root cause service>", "recommendation": "<specific fix>"}}
+Do NOT call any other tool. Submit NOW."""
+
+
+# ── action parsing ──────────────────────────────────────────────────
+
+def parse_action(text: str) -> dict | None:
+    """Parse LLM output into an action dict with multiple fallbacks."""
+    # 1. Direct JSON parse
+    try:
+        obj = json.loads(text.strip())
+        if isinstance(obj, dict) and "tool_name" in obj:
+            return obj
+    except json.JSONDecodeError:
+        pass
+
+    # 2. Extract from markdown code fence
+    fence_match = re.search(r"```(?:json)?\s*\n?(.*?)\n?\s*```", text, re.DOTALL)
+    if fence_match:
+        try:
+            obj = json.loads(fence_match.group(1).strip())
+            if isinstance(obj, dict) and "tool_name" in obj:
+                return obj
+        except json.JSONDecodeError:
+            pass
+
+    # 3. Regex: first {...} block
+    brace_match = re.search(r"\{[^{}]*(?:\{[^{}]*\}[^{}]*)*\}", text, re.DOTALL)
+    if brace_match:
+        try:
+            obj = json.loads(brace_match.group(0))
+            if isinstance(obj, dict) and "tool_name" in obj:
+                return obj
+        except json.JSONDecodeError:
+            pass
+
+    return None
+
+
+# ── history management ──────────────────────────────────────────────
+
+def build_initial_prompt(observation: dict) -> str:
+    """Build the first user prompt from the reset observation."""
+    parts = []
+    parts.append(f"INCIDENT: {observation.get('incident_summary', '')}")
+
+    tool_descs = observation.get("tool_descriptions")
+    if tool_descs:
+        parts.append("\n--- AVAILABLE TOOL PARAMETERS ---")
+        for tool, meta in tool_descs.items():
+            parts.append(f"  {tool}: {json.dumps(meta)}")
+
+    parts.append(
+        f"\nStep {observation.get('step_number', 0)}/{observation.get('max_steps', 20)}"
+    )
+    parts.append("\nBegin your investigation. Respond with your first action as JSON:")
+    return "\n".join(parts)
+
+
+def build_tool_response_prompt(observation: dict) -> str:
+    """Build a follow-up user prompt after a tool call."""
+    parts = []
+
+    tool_output = observation.get("tool_output", "")
+    if tool_output:
+        parts.append(f"Tool output:\n{tool_output}")
+
+    if observation.get("last_action_error"):
+        parts.append(f"\n⚠ ERROR: {observation['last_action_error']}")
+
+    step_num = observation.get("step_number", 0)
+    max_steps = observation.get("max_steps", 20)
+    parts.append(
+        f"\nStep {step_num}/{max_steps} | "
+        f"Cumulative reward: {observation.get('cumulative_reward', 0.0):.2f}"
+    )
+
+    # Add urgency nudges based on step progress
+    if step_num >= max_steps - 5:
+        parts.append("\n⚠⚠ CRITICAL: You MUST call submit_resolution NOW! No more investigation!")
+    elif step_num >= max_steps - 8:
+        parts.append("\n⚠ WARNING: Submit your resolution NOW. Call submit_resolution with your best analysis.")
+    elif step_num >= max_steps - 12:
+        parts.append("\nNote: Start forming your resolution. You should submit within the next 2-3 steps.")
+
+    parts.append("\nRespond with your next action as JSON:")
+    return "\n".join(parts)
+
+
+# ── main loop ───────────────────────────────────────────────────────
+
+async def run_task(task_id: int, base_url: str, client: OpenAI) -> float:
+    """Run a single task against the environment via WebSocket. Returns grader score."""
+    import websockets
+
+    ws_url = base_url.replace("http://", "ws://").replace("https://", "wss://")
+    ws_url = ws_url.rstrip("/") + "/ws"
+
+    async with websockets.connect(ws_url, ping_interval=120, ping_timeout=300) as ws:
+        # Reset
+        await ws.send(json.dumps({"type": "reset", "data": {"task_id": task_id}}))
+        resp = json.loads(await ws.recv())
+        data = resp["data"]
+        observation = data.get("observation", data)
+        done = data.get("done", False)
+
+        # [START] — mandatory structured log
+        print(f"[START] task=task_{task_id} env=sentinel_env model={MODEL_NAME}")
+
+        # Build multi-turn conversation
+        messages: list[dict] = [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": build_initial_prompt(observation)},
+        ]
+        final_score = 0.0
+        local_step = 0  # track client-side loop iterations
+        rewards_list: list[str] = []  # collect per-step rewards for [END] line
+
+        while not done:
+            local_step += 1
+            step_num = observation.get("step_number", local_step)
+            max_steps = observation.get("max_steps", 20)
+
+            # Safety: break if we've looped too many times without env advancing
+            if local_step > max_steps + 10:
+                print(f"  Exceeded max loop iterations ({local_step}), breaking", flush=True)
+                break
+
+            print(f"  Step {step_num} (iter {local_step}): calling LLM...", flush=True)
+
+            # Force resolution when nearing step limit
+            force_resolution = step_num >= max_steps - 8
+            if force_resolution:
+                print(f"  Step {step_num}: FORCING resolution submission", flush=True)
+                # Add force prompt as an additional system nudge in the conversation
+                force_msg = {"role": "system", "content": FORCE_RESOLUTION_PROMPT}
+                call_messages = messages + [force_msg]
+            else:
+                call_messages = messages
+
+            # Trim conversation if too long (keep system + first user + last 20 messages)
+            if len(call_messages) > 30:
+                call_messages = call_messages[:2] + call_messages[-20:]
+
+            # Try to get a valid action from the LLM
+            action_dict = None
+            for attempt in range(MAX_PARSE_RETRIES):
+                try:
+                    # Run sync LLM call in a thread so the event loop can
+                    # still handle WebSocket pings during long reasoning calls
+                    response = await asyncio.to_thread(
+                        client.chat.completions.create,
+                        model=MODEL_NAME,
+                        messages=call_messages,
+                        max_completion_tokens=MAX_COMPLETION_TOKENS,
+                    )
+                    raw = response.choices[0].message.content or ""
+                    action_dict = parse_action(raw)
+                    if action_dict:
+                        # Store the assistant response in conversation
+                        messages.append({"role": "assistant", "content": raw})
+                        print(f"  Step {step_num}: action={action_dict.get('tool_name', '?')}", flush=True)
+                        break
+                    else:
+                        print(f"  Step {step_num}: parse failed (attempt {attempt + 1}), raw={raw[:200]}", flush=True)
+                        # On parse failure, add a nudge and retry
+                        if attempt < MAX_PARSE_RETRIES - 1:
+                            call_messages = call_messages + [
+                                {"role": "assistant", "content": raw},
+                                {"role": "user", "content": "That was not valid JSON. Respond with ONLY a JSON object like {\"tool_name\": \"...\", \"parameters\": {...}}"},
+                            ]
+                except Exception as e:
+                    print(f"  LLM error (attempt {attempt + 1}): {e}", file=sys.stderr)
+
+            if action_dict is None:
+                # Fallback: send an invalid action to let the env handle it
+                action_dict = {"tool_name": "_invalid_", "parameters": {}}
+                messages.append({"role": "assistant", "content": json.dumps(action_dict)})
+
+            # Normalize "all" query param to empty string (handler uses it as substring filter)
+            if action_dict.get("parameters", {}).get("query") == "all":
+                action_dict["parameters"]["query"] = ""
+            if action_dict.get("parameters", {}).get("severity") == "all":
+                action_dict["parameters"].pop("severity", None)
+
+            # Step the environment via WebSocket
+            await ws.send(json.dumps({"type": "step", "data": action_dict}))
+            resp = json.loads(await ws.recv())
+            data = resp["data"]
+            observation = data.get("observation", data)
+            done = data.get("done", False)
+
+            if observation.get("done"):
+                done = True
+
+            step_reward = observation.get("reward", 0.0)
+            rewards_list.append(f"{step_reward:.2f}")
+
+            # [STEP] — mandatory structured log
+            print(f"[STEP] step={step_num} action={action_dict.get('tool_name')} "
+                  f"reward={step_reward:.2f} done={str(done).lower()} "
+                  f"error={observation.get('last_action_error', 'null')}")
+
+            # Add tool response as user message in the conversation
+            if not done:
+                messages.append({"role": "user", "content": build_tool_response_prompt(observation)})
+
+        # Get final state for score
+        await ws.send(json.dumps({"type": "state"}))
+        resp = json.loads(await ws.recv())
+        state_data = resp["data"]
+        final_score = state_data.get("final_score", 0.0)
+
+        # [END] — mandatory structured log
+        print(f"[END] success={str(final_score > 0).lower()} steps={local_step} "
+              f"score={final_score:.2f} rewards={','.join(rewards_list)}")
+
+        return final_score
+
+
+async def main() -> None:
+    if not ENV_URL:
+        print("ERROR: ENV_URL environment variable is required.", file=sys.stderr)
+        print("Set it to the environment server URL (e.g., http://localhost:8000)", file=sys.stderr)
+        sys.exit(1)
+
+    if AZURE_OPENAI_ENDPOINT:
+        llm_client = AzureOpenAI(
+            azure_endpoint=AZURE_OPENAI_ENDPOINT,
+            api_key=AZURE_OPENAI_API_KEY or API_KEY,
+            api_version=AZURE_OPENAI_API_VERSION,
+        )
+        if AZURE_OPENAI_DEPLOYMENT:
+            global MODEL_NAME
+            MODEL_NAME = AZURE_OPENAI_DEPLOYMENT
+        print(f"Using Azure OpenAI: {AZURE_OPENAI_ENDPOINT} / deployment={MODEL_NAME}")
+    else:
+        llm_client = OpenAI(
+            base_url=API_BASE_URL,
+            api_key=API_KEY,
+        )
+        print(f"Using LLM API: {API_BASE_URL} / model={MODEL_NAME}")
+
+    scores: dict[int, float] = {}
+    for task_id in [1, 2, 3]:
+        print(f"\n{'='*50}")
+        print(f"Running Task {task_id}...")
+        print(f"{'='*50}")
+
+        try:
+            score = await asyncio.wait_for(
+                run_task(task_id, ENV_URL, llm_client),
+                timeout=TASK_TIMEOUT,
+            )
+            scores[task_id] = score
+        except asyncio.TimeoutError:
+            print(f"  Task {task_id} timed out after {TASK_TIMEOUT}s", file=sys.stderr)
+            scores[task_id] = 0.0
+        except Exception as e:
+            print(f"  Task {task_id} failed: {e}", file=sys.stderr)
+            scores[task_id] = 0.0
+
+        print(f"Task {task_id}: {scores[task_id]:.2f}")
+
+    avg = sum(scores.values()) / len(scores) if scores else 0.0
+    print(f"\n{'='*50}")
+    print(f"Task 1: {scores.get(1, 0.0):.2f}")
+    print(f"Task 2: {scores.get(2, 0.0):.2f}")
+    print(f"Task 3: {scores.get(3, 0.0):.2f}")
+    print(f"Average: {avg:.2f}")
+    print(f"{'='*50}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

models.py

@@ -0,0 +1,160 @@
+"""Pydantic models for OpenEnv-Sentinel.
+
+Typed discriminated union for actions — each tool has its own action + params class.
+"""
+
+from typing import Annotated, List, Literal, Union
+
+from openenv.core.env_server.types import Action, Observation, State
+from pydantic import BaseModel, Field, RootModel
+
+
+# ── Parameter models ────────────────────────────────────────────────
+
+
+class QueryLogsParams(BaseModel):
+    service: str
+    query: str = "all"
+    severity: str = "all"
+
+
+class QueryMetricsParams(BaseModel):
+    service: str
+    metric: str
+
+
+class GetServiceStatusParams(BaseModel):
+    service: str
+
+
+class GetDependencyMapParams(BaseModel):
+    service: str = ""
+
+
+class ConsultRunbookParams(BaseModel):
+    topic: str
+
+
+class CheckRecentChangesParams(BaseModel):
+    service: str = ""
+
+
+class SubmitResolutionParams(BaseModel):
+    root_cause: str
+    affected_service: str
+    recommendation: str
+
+
+# ── Action models ───────────────────────────────────────────────────
+
+
+class QueryLogsAction(Action):
+    tool_name: Literal["query_logs"] = "query_logs"
+    parameters: QueryLogsParams
+
+    def param_dict(self) -> dict:
+        return self.parameters.model_dump()
+
+
+class QueryMetricsAction(Action):
+    tool_name: Literal["query_metrics"] = "query_metrics"
+    parameters: QueryMetricsParams
+
+    def param_dict(self) -> dict:
+        return self.parameters.model_dump()
+
+
+class GetServiceStatusAction(Action):
+    tool_name: Literal["get_service_status"] = "get_service_status"
+    parameters: GetServiceStatusParams
+
+    def param_dict(self) -> dict:
+        return self.parameters.model_dump()
+
+
+class GetDependencyMapAction(Action):
+    tool_name: Literal["get_dependency_map"] = "get_dependency_map"
+    parameters: GetDependencyMapParams
+
+    def param_dict(self) -> dict:
+        return self.parameters.model_dump()
+
+
+class ConsultRunbookAction(Action):
+    tool_name: Literal["consult_runbook"] = "consult_runbook"
+    parameters: ConsultRunbookParams
+
+    def param_dict(self) -> dict:
+        return self.parameters.model_dump()
+
+
+class CheckRecentChangesAction(Action):
+    tool_name: Literal["check_recent_changes"] = "check_recent_changes"
+    parameters: CheckRecentChangesParams
+
+    def param_dict(self) -> dict:
+        return self.parameters.model_dump()
+
+
+class SubmitResolutionAction(Action):
+    tool_name: Literal["submit_resolution"] = "submit_resolution"
+    parameters: SubmitResolutionParams
+
+    def param_dict(self) -> dict:
+        return self.parameters.model_dump()
+
+
+# ── Discriminated union ─────────────────────────────────────────────
+
+_ActionUnion = Annotated[
+    Union[
+        QueryLogsAction,
+        QueryMetricsAction,
+        GetServiceStatusAction,
+        GetDependencyMapAction,
+        ConsultRunbookAction,
+        CheckRecentChangesAction,
+        SubmitResolutionAction,
+    ],
+    Field(discriminator="tool_name"),
+]
+
+
+class SentinelAction(RootModel[_ActionUnion]):
+    """Discriminated union action — delegates to the matched concrete action."""
+
+    @property
+    def tool_name(self) -> str:
+        return self.root.tool_name
+
+    def param_dict(self) -> dict:
+        return self.root.param_dict()
+
+
+# ── Observation & State ─────────────────────────────────────────────
+
+
+class SentinelObservation(Observation):
+    """What the agent sees after each step."""
+
+    incident_summary: str = Field(default="", description="Initial alert / ongoing context")
+    tool_output: str = Field(default="", description="Result from the last tool call")
+    available_tools: List[str] = Field(default_factory=list, description="Tools the agent can use")
+    step_number: int = Field(default=0, description="Current step number")
+    max_steps: int = Field(default=20, description="Maximum steps per episode")
+    cumulative_reward: float = Field(default=0.0, description="Running total of per-step rewards")
+    last_action_error: str = Field(default="", description="Error from last invalid action")
+    tool_descriptions: dict = Field(default_factory=dict, description="Parameter metadata (populated on reset only)")
+
+
+class SentinelState(State):
+    """Internal environment state."""
+
+    task_id: int = 1
+    task_name: str = ""
+    tools_called: List[str] = Field(default_factory=list)
+    relevant_tools_called: List[str] = Field(default_factory=list)
+    resolution_submitted: bool = False
+    root_cause_correct: bool = False
+    recommendation_correct: bool = False
+    final_score: float = 0.0

openenv.yaml

@@ -0,0 +1,6 @@
+spec_version: 1
+name: sentinel_env
+type: space
+runtime: fastapi
+app: server.app:app
+port: 8000

pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "openenv-sentinel"
+version = "0.1.0"
+description = "SRE Incident Triage Environment for OpenEnv"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.10"
+dependencies = [
+    "openenv-core>=0.2.3",
+    "fastapi>=0.104.0",
+    "uvicorn>=0.24.0",
+    "pydantic>=2.0.0",
+]
+
+[project.optional-dependencies]
+inference = [
+    "openai>=1.0.0",
+]
+dev = [
+    "pytest>=7.0",
+    "httpx>=0.25.0",
+]
+
+[project.scripts]
+server = "server.app:main"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.setuptools.packages.find]
+include = ["openenv_sentinel*", "server*", "scenarios*", "tools*", "grading*"]

scenarios/__init__.py

@@ -0,0 +1,6 @@
+"""Scenario package."""
+
+from .base import BaseScenario
+
+__all__ = ["BaseScenario"]
+

scenarios/base.py

@@ -0,0 +1,96 @@
+"""Base scenario ABC for SRE incident triage tasks."""
+
+from abc import ABC, abstractmethod
+from typing import Dict, List
+
+
+class BaseScenario(ABC):
+    """Abstract base for all incident scenarios."""
+
+    @abstractmethod
+    def get_incident_summary(self) -> str:
+        """Return the initial alert text for this incident."""
+
+    @abstractmethod
+    def get_services(self) -> Dict[str, dict]:
+        """Return service status data."""
+
+    @abstractmethod
+    def get_tool_response(self, tool_name: str, parameters: dict) -> str:
+        """Return a pre-scripted response for the given tool call."""
+
+    @abstractmethod
+    def get_relevant_tools(self) -> List[str]:
+        """Return list of 'tool_name:param' string keys considered relevant."""
+
+    @abstractmethod
+    def grade_resolution(self, resolution: dict, step_count: int) -> dict:
+        """Grade a submitted resolution. Returns dict with score, root_cause_correct, recommendation_correct."""
+
+    @abstractmethod
+    def get_tool_descriptions(self) -> dict:
+        """Return parameter metadata for LLM context. Called once on reset."""
+
+    # ── shared helpers ──────────────────────────────────────────────
+
+    def _format_logs(self, entries: List[dict]) -> str:
+        lines = []
+        for e in entries:
+            lines.append(f"[{e['timestamp']}] {e['level']} - {e['message']}")
+            if "source" in e:
+                lines.append(f"    at {e['source']}")
+        return "\n".join(lines) if lines else "No matching log entries found."
+
+    def _format_metrics(self, metric_name: str, data: dict) -> str:
+        lines = [f"Metric: {metric_name}"]
+        if "values" in data:
+            lines.append(f"  Values (recent): {data['values']}")
+        if "unit" in data:
+            lines.append(f"  Unit: {data['unit']}")
+        if "annotation" in data:
+            lines.append(f"  ⚠ {data['annotation']}")
+        return "\n".join(lines)
+
+    def _format_service_status(self, svc: dict) -> str:
+        lines = [
+            f"Service: {svc['name']}",
+            f"  Status: {svc['status']}",
+            f"  Error Rate: {svc.get('error_rate', 'N/A')}",
+            f"  Uptime: {svc.get('uptime', 'N/A')}",
+        ]
+        if "last_deploy" in svc:
+            lines.append(f"  Last Deploy: {svc['last_deploy']}")
+        if "restarts" in svc:
+            lines.append(f"  Restarts (30min): {svc['restarts']}")
+        if "latency_p99" in svc:
+            lines.append(f"  Latency (p99): {svc['latency_p99']}")
+        if "queue_depth" in svc:
+            lines.append(f"  Queue Depth: {svc['queue_depth']}")
+        if "connections" in svc:
+            lines.append(f"  Connections: {svc['connections']}")
+        return "\n".join(lines)
+
+    def _format_dependency_map(self, deps: dict) -> str:
+        lines = []
+        for svc, info in deps.items():
+            depends_on = ", ".join(info.get("depends_on", []))
+            depended_by = ", ".join(info.get("depended_by", []))
+            lines.append(f"{svc}:")
+            if depends_on:
+                lines.append(f"  Depends on: {depends_on}")
+            if depended_by:
+                lines.append(f"  Depended on by: {depended_by}")
+        return "\n".join(lines) if lines else "No dependency data available."
+
+    def _format_changes(self, changes: List[dict]) -> str:
+        if not changes:
+            return "No recent changes found."
+        lines = []
+        for c in changes:
+            lines.append(f"[{c['timestamp']}] {c['service']} - {c['description']}")
+            if "changelog" in c:
+                lines.append(f"  Changelog: {c['changelog']}")
+        return "\n".join(lines)
+
+    def _format_runbook(self, content: str) -> str:
+        return content if content else "No matching runbook found for this topic."

scenarios/task1_smoking_gun.py

@@ -0,0 +1,405 @@
+"""Task 1 — The Smoking Gun: a clear-cut deployment-caused outage."""
+
+import re
+
+from .base import BaseScenario
+
+
+class SmokingGunScenario(BaseScenario):
+    """Payment-API HTTP 500s caused by a missing env var after deploy v2.3.1."""
+
+    def __init__(self) -> None:
+        self.incident_text = (
+            "CRITICAL: payment-api returning HTTP 500 errors. "
+            "Customer checkout failing. Started 10 minutes ago."
+        )
+
+        self.services = {
+            "payment-api": {
+                "name": "payment-api",
+                "status": "UNHEALTHY",
+                "error_rate": "92%",
+                "uptime": "99.3% (30d)",
+                "last_deploy": "v2.3.1 deployed 8 minutes ago",
+                "restarts": 0,
+                "latency_p99": "timeout",
+                "connections": "0 active DB connections",
+            },
+            "order-service": {
+                "name": "order-service",
+                "status": "HEALTHY",
+                "error_rate": "0.1%",
+                "uptime": "99.99% (30d)",
+                "last_deploy": "v4.1.0 deployed 3 days ago",
+                "latency_p99": "120ms",
+            },
+            "user-service": {
+                "name": "user-service",
+                "status": "HEALTHY",
+                "error_rate": "0.05%",
+                "uptime": "99.99% (30d)",
+                "last_deploy": "v1.8.2 deployed 5 days ago",
+                "latency_p99": "45ms",
+            },
+            "postgres-primary": {
+                "name": "postgres-primary",
+                "status": "HEALTHY",
+                "error_rate": "0%",
+                "uptime": "99.999% (30d)",
+                "connections": "42 active connections",
+            },
+        }
+
+        self.log_entries = {
+            "payment-api": [
+                {
+                    "timestamp": "2026-04-01T10:32:14Z",
+                    "level": "ERROR",
+                    "message": "NullPointerException: DB_CONNECTION_STRING is null",
+                    "source": "config.DatabaseConfig.getConnection(DatabaseConfig.java:42)",
+                },
+                {
+                    "timestamp": "2026-04-01T10:32:16Z",
+                    "level": "ERROR",
+                    "message": "NullPointerException: DB_CONNECTION_STRING is null",
+                    "source": "config.DatabaseConfig.getConnection(DatabaseConfig.java:42)",
+                },
+                {
+                    "timestamp": "2026-04-01T10:32:18Z",
+                    "level": "ERROR",
+                    "message": "NullPointerException: DB_CONNECTION_STRING is null",
+                    "source": "config.DatabaseConfig.getConnection(DatabaseConfig.java:42)",
+                },
+                {
+                    "timestamp": "2026-04-01T10:32:20Z",
+                    "level": "ERROR",
+                    "message": "Failed to handle request POST /api/v1/payments: HTTP 500",
+                    "source": "handler.PaymentHandler.processPayment(PaymentHandler.java:87)",
+                },
+                {
+                    "timestamp": "2026-04-01T10:32:22Z",
+                    "level": "ERROR",
+                    "message": "NullPointerException: DB_CONNECTION_STRING is null",
+                    "source": "config.DatabaseConfig.getConnection(DatabaseConfig.java:42)",
+                },
+                {
+                    "timestamp": "2026-04-01T10:32:24Z",
+                    "level": "WARN",
+                    "message": "Health check failed: unable to reach database",
+                    "source": "health.HealthCheckService.check(HealthCheckService.java:23)",
+                },
+                {
+                    "timestamp": "2026-04-01T10:32:26Z",
+                    "level": "ERROR",
+                    "message": "NullPointerException: DB_CONNECTION_STRING is null",
+                    "source": "config.DatabaseConfig.getConnection(DatabaseConfig.java:42)",
+                },
+                {
+                    "timestamp": "2026-04-01T10:32:28Z",
+                    "level": "ERROR",
+                    "message": "Failed to handle request POST /api/v1/payments: HTTP 500",
+                    "source": "handler.PaymentHandler.processPayment(PaymentHandler.java:87)",
+                },
+                {
+                    "timestamp": "2026-04-01T10:32:30Z",
+                    "level": "INFO",
+                    "message": "Application started: payment-api v2.3.1",
+                    "source": "main.Application.start(Application.java:15)",
+                },
+                {
+                    "timestamp": "2026-04-01T10:32:32Z",
+                    "level": "ERROR",
+                    "message": "NullPointerException: DB_CONNECTION_STRING is null",
+                    "source": "config.DatabaseConfig.getConnection(DatabaseConfig.java:42)",
+                },
+            ],
+        }
+
+        self.metrics = {
+            "payment-api": {
+                "error_rate": {
+                    "values": [
+                        ("2026-04-01T10:20:00Z", "0.1%"),
+                        ("2026-04-01T10:22:00Z", "0.1%"),
+                        ("2026-04-01T10:24:00Z", "0.1%"),
+                        ("2026-04-01T10:25:00Z", "92.0%"),
+                        ("2026-04-01T10:26:00Z", "92.3%"),
+                        ("2026-04-01T10:28:00Z", "91.8%"),
+                        ("2026-04-01T10:30:00Z", "92.1%"),
+                        ("2026-04-01T10:32:00Z", "92.0%"),
+                    ],
+                    "unit": "percent",
+                    "annotation": "Spike at 10:25:00Z correlates with deploy v2.3.1",
+                },
+                "latency_p99": {
+                    "values": [
+                        ("2026-04-01T10:20:00Z", "85ms"),
+                        ("2026-04-01T10:22:00Z", "82ms"),
+                        ("2026-04-01T10:24:00Z", "88ms"),
+                        ("2026-04-01T10:25:00Z", "timeout"),
+                        ("2026-04-01T10:26:00Z", "timeout"),
+                        ("2026-04-01T10:28:00Z", "timeout"),
+                        ("2026-04-01T10:30:00Z", "timeout"),
+                        ("2026-04-01T10:32:00Z", "timeout"),
+                    ],
+                    "unit": "milliseconds",
+                    "annotation": "Latency spiked to timeout at deploy timestamp",
+                },
+                "request_count": {
+                    "values": [
+                        ("2026-04-01T10:20:00Z", "1200"),
+                        ("2026-04-01T10:22:00Z", "1180"),
+                        ("2026-04-01T10:24:00Z", "1210"),
+                        ("2026-04-01T10:25:00Z", "1195"),
+                        ("2026-04-01T10:26:00Z", "1050"),
+                        ("2026-04-01T10:28:00Z", "870"),
+                        ("2026-04-01T10:30:00Z", "620"),
+                        ("2026-04-01T10:32:00Z", "430"),
+                    ],
+                    "unit": "requests/min",
+                    "annotation": "Request volume dropping as clients receive errors",
+                },
+            },
+        }
+
+        self.dependency_map = {
+            "payment-api": {
+                "depends_on": ["postgres-primary", "user-service"],
+                "depended_by": ["order-service"],
+            },
+            "order-service": {
+                "depends_on": ["payment-api", "postgres-primary"],
+                "depended_by": [],
+            },
+            "user-service": {
+                "depends_on": ["postgres-primary"],
+                "depended_by": ["payment-api"],
+            },
+            "postgres-primary": {
+                "depends_on": [],
+                "depended_by": ["payment-api", "order-service", "user-service"],
+            },
+        }
+
+        self.recent_changes = {
+            "payment-api": [
+                {
+                    "timestamp": "2026-04-01T10:24:30Z",
+                    "service": "payment-api",
+                    "description": "Deployed v2.3.1",
+                    "changelog": "Refactored config loader to use new ConfigManager class. Removed legacy environment variable fallback.",
+                },
+            ],
+            "order-service": [
+                {
+                    "timestamp": "2026-03-29T14:00:00Z",
+                    "service": "order-service",
+                    "description": "Deployed v4.1.0",
+                    "changelog": "Added bulk order endpoint.",
+                },
+            ],
+        }
+
+        self.runbooks = {
+            "payment-api": (
+                "## Runbook: payment-api\n\n"
+                "### Common Issues\n\n"
+                "1. **HTTP 500 errors**\n"
+                "   - Check database connectivity: verify DB_CONNECTION_STRING env var is set.\n"
+                "   - Check connection pool exhaustion: review active connections on postgres-primary.\n"
+                "   - If caused by a bad deploy, rollback to previous version.\n\n"
+                "2. **High latency**\n"
+                "   - Check postgres-primary slow query log.\n"
+                "   - Review connection pool settings.\n\n"
+                "### Rollback Procedure\n"
+                "   ```\n"
+                "   kubectl rollout undo deployment/payment-api\n"
+                "   ```\n\n"
+                "### Escalation\n"
+                "   Contact: #payments-oncall in Slack\n"
+            ),
+        }
+
+    # ── abstract method implementations ─────────────────────────────
+
+    def get_incident_summary(self) -> str:
+        return self.incident_text
+
+    def get_services(self) -> dict:
+        return self.services
+
+    def get_tool_response(self, tool_name: str, parameters: dict) -> str:
+        dispatch = {
+            "get_service_status": self._handle_service_status,
+            "query_logs": self._handle_query_logs,
+            "query_metrics": self._handle_query_metrics,
+            "get_dependency_map": self._handle_dependency_map,
+            "consult_runbook": self._handle_runbook,
+            "check_recent_changes": self._handle_recent_changes,
+        }
+        handler = dispatch.get(tool_name)
+        if handler is None:
+            return f"Unknown tool: {tool_name}"
+        return handler(parameters)
+
+    def get_relevant_tools(self) -> list[str]:
+        return [
+            "get_service_status:payment-api",
+            "query_logs:payment-api",
+            "check_recent_changes:payment-api",
+            "query_metrics:payment-api:error_rate",
+        ]
+
+    def get_tool_descriptions(self) -> dict:
+        return {
+            "query_logs": {
+                "services": list(self.services.keys()),
+                "severity_options": ["all", "error", "warning", "info"],
+            },
+            "query_metrics": {
+                "services": list(self.metrics.keys()),
+                "metrics": ["error_rate", "latency_p99", "request_count", "cpu", "memory"],
+            },
+            "get_service_status": {
+                "services": list(self.services.keys()),
+            },
+            "get_dependency_map": {
+                "services": list(self.dependency_map.keys()),
+                "note": "Omit service for full map",
+            },
+            "consult_runbook": {
+                "topics": list(self.runbooks.keys()),
+            },
+            "check_recent_changes": {
+                "services": list(self.recent_changes.keys()),
+                "note": "Omit service for all changes",
+            },
+        }
+
+    def grade_resolution(self, resolution: dict, step_count: int) -> dict:
+        score = 0.0
+        root_cause_text = str(resolution.get("root_cause", "")).lower()
+        affected = str(resolution.get("affected_service", "")).strip().lower()
+        recommendation_text = str(resolution.get("recommendation", "")).lower()
+
+        # Root cause mentions config/env var issue (0.30)
+        config_match = False
+        if "db_connection_string" in root_cause_text:
+            config_match = True
+        elif any(kw in root_cause_text for kw in ["environment variable", "env var", "config"]):
+            if any(kw in root_cause_text for kw in ["missing", "null", "empty"]):
+                config_match = True
+        if config_match:
+            score += 0.30
+
+        # Root cause mentions deployment (0.20)
+        deploy_match = any(
+            kw in root_cause_text for kw in ["deploy", "v2.3.1", "release", "v2.3"]
+        )
+        if deploy_match:
+            score += 0.20
+
+        # Affected service is payment-api (0.15)
+        if affected == "payment-api":
+            score += 0.15
+
+        # Recommendation includes rollback or fix (0.15)
+        rec_match = False
+        if any(kw in recommendation_text for kw in ["rollback", "revert"]):
+            rec_match = True
+        elif re.search(r"set.*db_connection", recommendation_text):
+            rec_match = True
+        elif re.search(r"configure.*db_connection", recommendation_text):
+            rec_match = True
+        if rec_match:
+            score += 0.15
+
+        # Efficiency bonus (0.10)
+        if step_count <= 5:
+            score += 0.10
+        elif step_count <= 8:
+            score += 0.05
+
+        # No destructive recommendations (0.10) — present by default
+        destructive = any(
+            kw in recommendation_text
+            for kw in ["restart-all", "drop database", "delete", "truncate"]
+        )
+        if not destructive:
+            score += 0.10
+
+        root_cause_correct = config_match and deploy_match
+        recommendation_correct = rec_match
+
+        score = max(0.0, min(1.0, score))
+        return {
+            "score": score,
+            "root_cause_correct": root_cause_correct,
+            "recommendation_correct": recommendation_correct,
+        }
+
+    # ── internal tool handlers ──────────────────────────────────────
+
+    def _handle_service_status(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        svc = self.services.get(service)
+        if svc is None:
+            return f"Service '{service}' not found."
+        return self._format_service_status(svc)
+
+    def _handle_query_logs(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        query = parameters.get("query", "").lower()
+        entries = self.log_entries.get(service)
+        if entries is None:
+            return f"No logs available for service '{service}'."
+        if query:
+            matched = [
+                e
+                for e in entries
+                if query in e["message"].lower()
+                or query in e.get("level", "").lower()
+                or query in e.get("source", "").lower()
+            ]
+        else:
+            matched = entries
+        return self._format_logs(matched)
+
+    def _handle_query_metrics(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        metric = parameters.get("metric", "").lower()
+        svc_metrics = self.metrics.get(service)
+        if svc_metrics is None:
+            return f"No metrics available for service '{service}'."
+        # Fuzzy substring match on metric name
+        for name, data in svc_metrics.items():
+            if metric in name.lower() or name.lower() in metric:
+                return self._format_metrics(name, data)
+        return f"No matching metric '{metric}' for service '{service}'."
+
+    def _handle_dependency_map(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        if service and service in self.dependency_map:
+            subset = {service: self.dependency_map[service]}
+            return self._format_dependency_map(subset)
+        return self._format_dependency_map(self.dependency_map)
+
+    def _handle_runbook(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        topic = parameters.get("topic", "").lower()
+        # Try service-specific runbook first
+        content = self.runbooks.get(service)
+        if content:
+            return self._format_runbook(content)
+        # Try topic match
+        for key, value in self.runbooks.items():
+            if topic and topic in key.lower():
+                return self._format_runbook(value)
+        return self._format_runbook("")
+
+    def _handle_recent_changes(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        changes = self.recent_changes.get(service)
+        if changes is None:
+            return f"No recent changes found for service '{service}'."
+        return self._format_changes(changes)

scenarios/task2_upstream_culprit.py

@@ -0,0 +1,621 @@
+"""Task 2 — The Upstream Culprit: checkout latency caused by an upstream OOM."""
+
+import re
+
+from .base import BaseScenario
+
+
+class UpstreamCulpritScenario(BaseScenario):
+    """Checkout-service p99 latency caused by inventory-service OOMKilled restarts."""
+
+    def __init__(self) -> None:
+        self.incident_text = (
+            "WARNING: checkout-service p99 latency > 5 seconds. "
+            "Customer-facing degradation. SLA breach imminent."
+        )
+
+        self.services = {
+            "checkout-service": {
+                "name": "checkout-service",
+                "status": "DEGRADED",
+                "error_rate": "12%",
+                "uptime": "99.7% (30d)",
+                "last_deploy": "v3.8.0 deployed 2 days ago",
+                "restarts": 0,
+                "latency_p99": "5.2s",
+                "connections": "38 active DB connections",
+            },
+            "inventory-service": {
+                "name": "inventory-service",
+                "status": "UNHEALTHY",
+                "error_rate": "45%",
+                "uptime": "94.2% (30d)",
+                "last_deploy": "v2.1.4 deployed 5 days ago",
+                "restarts": 4,
+                "latency_p99": "timeout",
+                "connections": "12 active DB connections",
+            },
+            "redis-cache": {
+                "name": "redis-cache",
+                "status": "HEALTHY",
+                "error_rate": "0%",
+                "uptime": "99.999% (30d)",
+                "connections": "64 active connections",
+            },
+            "postgres-primary": {
+                "name": "postgres-primary",
+                "status": "HEALTHY",
+                "error_rate": "0%",
+                "uptime": "99.999% (30d)",
+                "connections": "55 active connections",
+            },
+            "api-gateway": {
+                "name": "api-gateway",
+                "status": "HEALTHY",
+                "error_rate": "2.1%",
+                "uptime": "99.98% (30d)",
+                "last_deploy": "v5.0.2 deployed 1 week ago",
+                "latency_p99": "5.4s",
+            },
+        }
+
+        self.log_entries = {
+            "checkout-service": [
+                {
+                    "timestamp": "2026-04-01T09:31:04Z",
+                    "level": "ERROR",
+                    "message": "TimeoutException: inventory-service did not respond within 3000ms",
+                    "source": "client.InventoryClient.checkStock(InventoryClient.java:67)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:31:18Z",
+                    "level": "ERROR",
+                    "message": "TimeoutException: inventory-service did not respond within 3000ms",
+                    "source": "client.InventoryClient.checkStock(InventoryClient.java:67)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:31:22Z",
+                    "level": "WARN",
+                    "message": "Retry attempt 2/3 for inventory-service call failed",
+                    "source": "client.InventoryClient.checkStock(InventoryClient.java:74)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:31:35Z",
+                    "level": "ERROR",
+                    "message": "TimeoutException: inventory-service did not respond within 3000ms",
+                    "source": "client.InventoryClient.reserveItem(InventoryClient.java:112)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:31:48Z",
+                    "level": "ERROR",
+                    "message": "Failed to handle request POST /api/v1/checkout: HTTP 504 Gateway Timeout",
+                    "source": "handler.CheckoutHandler.processCheckout(CheckoutHandler.java:91)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:32:01Z",
+                    "level": "ERROR",
+                    "message": "TimeoutException: inventory-service did not respond within 3000ms",
+                    "source": "client.InventoryClient.checkStock(InventoryClient.java:67)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:32:15Z",
+                    "level": "WARN",
+                    "message": "Circuit breaker OPEN for inventory-service — too many failures",
+                    "source": "circuitbreaker.CircuitBreakerManager.trip(CircuitBreakerManager.java:43)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:32:30Z",
+                    "level": "INFO",
+                    "message": "Circuit breaker HALF-OPEN for inventory-service — attempting probe",
+                    "source": "circuitbreaker.CircuitBreakerManager.probe(CircuitBreakerManager.java:58)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:32:44Z",
+                    "level": "ERROR",
+                    "message": "TimeoutException: inventory-service did not respond within 3000ms",
+                    "source": "client.InventoryClient.checkStock(InventoryClient.java:67)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:32:58Z",
+                    "level": "WARN",
+                    "message": "Circuit breaker OPEN for inventory-service — probe failed",
+                    "source": "circuitbreaker.CircuitBreakerManager.trip(CircuitBreakerManager.java:43)",
+                },
+            ],
+            "inventory-service": [
+                {
+                    "timestamp": "2026-04-01T09:15:02Z",
+                    "level": "INFO",
+                    "message": "Starting catalog sync batch job — processing 12,400 items",
+                    "source": "jobs.CatalogSyncJob.execute(CatalogSyncJob.java:38)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:18:33Z",
+                    "level": "WARN",
+                    "message": "GC overhead limit exceeded — 98% of time spent in GC",
+                    "source": "runtime.GarbageCollector",
+                },
+                {
+                    "timestamp": "2026-04-01T09:19:47Z",
+                    "level": "ERROR",
+                    "message": "java.lang.OutOfMemoryError: Java heap space",
+                    "source": "jobs.CatalogSyncJob.processBatch(CatalogSyncJob.java:85)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:19:48Z",
+                    "level": "ERROR",
+                    "message": "OOMKilled: Container exceeded memory limit 512Mi",
+                    "source": "kubernetes.ContainerRuntime",
+                },
+                {
+                    "timestamp": "2026-04-01T09:19:50Z",
+                    "level": "INFO",
+                    "message": "Container restarting (restart count: 1)",
+                    "source": "kubernetes.ContainerRuntime",
+                },
+                {
+                    "timestamp": "2026-04-01T09:20:15Z",
+                    "level": "INFO",
+                    "message": "Application started: inventory-service v2.1.4",
+                    "source": "main.Application.start(Application.java:15)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:20:18Z",
+                    "level": "INFO",
+                    "message": "Resuming catalog sync batch job — 8,200 items remaining",
+                    "source": "jobs.CatalogSyncJob.execute(CatalogSyncJob.java:42)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:24:11Z",
+                    "level": "WARN",
+                    "message": "GC overhead limit exceeded — 97% of time spent in GC",
+                    "source": "runtime.GarbageCollector",
+                },
+                {
+                    "timestamp": "2026-04-01T09:25:03Z",
+                    "level": "ERROR",
+                    "message": "java.lang.OutOfMemoryError: Java heap space",
+                    "source": "jobs.CatalogSyncJob.processBatch(CatalogSyncJob.java:85)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:25:04Z",
+                    "level": "ERROR",
+                    "message": "OOMKilled: Container exceeded memory limit 512Mi",
+                    "source": "kubernetes.ContainerRuntime",
+                },
+                {
+                    "timestamp": "2026-04-01T09:25:06Z",
+                    "level": "INFO",
+                    "message": "Container restarting (restart count: 2)",
+                    "source": "kubernetes.ContainerRuntime",
+                },
+                {
+                    "timestamp": "2026-04-01T09:25:30Z",
+                    "level": "INFO",
+                    "message": "Application started: inventory-service v2.1.4",
+                    "source": "main.Application.start(Application.java:15)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:29:41Z",
+                    "level": "ERROR",
+                    "message": "java.lang.OutOfMemoryError: Java heap space",
+                    "source": "jobs.CatalogSyncJob.processBatch(CatalogSyncJob.java:85)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:29:42Z",
+                    "level": "ERROR",
+                    "message": "OOMKilled: Container exceeded memory limit 512Mi",
+                    "source": "kubernetes.ContainerRuntime",
+                },
+                {
+                    "timestamp": "2026-04-01T09:29:44Z",
+                    "level": "INFO",
+                    "message": "Container restarting (restart count: 3)",
+                    "source": "kubernetes.ContainerRuntime",
+                },
+                {
+                    "timestamp": "2026-04-01T09:30:10Z",
+                    "level": "INFO",
+                    "message": "Application started: inventory-service v2.1.4",
+                    "source": "main.Application.start(Application.java:15)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:33:58Z",
+                    "level": "ERROR",
+                    "message": "java.lang.OutOfMemoryError: Java heap space",
+                    "source": "jobs.CatalogSyncJob.processBatch(CatalogSyncJob.java:85)",
+                },
+                {
+                    "timestamp": "2026-04-01T09:33:59Z",
+                    "level": "ERROR",
+                    "message": "OOMKilled: Container exceeded memory limit 512Mi",
+                    "source": "kubernetes.ContainerRuntime",
+                },
+                {
+                    "timestamp": "2026-04-01T09:34:01Z",
+                    "level": "INFO",
+                    "message": "Container restarting (restart count: 4)",
+                    "source": "kubernetes.ContainerRuntime",
+                },
+            ],
+        }
+
+        self.metrics = {
+            "inventory-service": {
+                "memory": {
+                    "values": [
+                        ("2026-04-01T09:15:00Z", "210Mi"),
+                        ("2026-04-01T09:16:00Z", "280Mi"),
+                        ("2026-04-01T09:17:00Z", "380Mi"),
+                        ("2026-04-01T09:18:00Z", "450Mi"),
+                        ("2026-04-01T09:19:00Z", "510Mi"),
+                        ("2026-04-01T09:19:47Z", "OOMKilled"),
+                        ("2026-04-01T09:20:15Z", "120Mi"),
+                        ("2026-04-01T09:22:00Z", "260Mi"),
+                        ("2026-04-01T09:24:00Z", "440Mi"),
+                        ("2026-04-01T09:25:00Z", "510Mi"),
+                        ("2026-04-01T09:25:03Z", "OOMKilled"),
+                        ("2026-04-01T09:25:30Z", "125Mi"),
+                        ("2026-04-01T09:28:00Z", "390Mi"),
+                        ("2026-04-01T09:29:41Z", "OOMKilled"),
+                        ("2026-04-01T09:30:10Z", "118Mi"),
+                        ("2026-04-01T09:33:00Z", "480Mi"),
+                        ("2026-04-01T09:33:58Z", "OOMKilled"),
+                    ],
+                    "unit": "mebibytes",
+                    "annotation": "RSS climbing: 380Mi → 450Mi → 510Mi → OOMKilled → restart → climb again. Memory limit: 512Mi.",
+                },
+                "cpu": {
+                    "values": [
+                        ("2026-04-01T09:15:00Z", "15%"),
+                        ("2026-04-01T09:16:00Z", "32%"),
+                        ("2026-04-01T09:17:00Z", "58%"),
+                        ("2026-04-01T09:18:00Z", "74%"),
+                        ("2026-04-01T09:19:00Z", "91%"),
+                        ("2026-04-01T09:19:47Z", "99%"),
+                        ("2026-04-01T09:20:15Z", "8%"),
+                        ("2026-04-01T09:22:00Z", "35%"),
+                        ("2026-04-01T09:24:00Z", "72%"),
+                        ("2026-04-01T09:25:03Z", "98%"),
+                        ("2026-04-01T09:25:30Z", "10%"),
+                        ("2026-04-01T09:29:41Z", "97%"),
+                        ("2026-04-01T09:30:10Z", "9%"),
+                        ("2026-04-01T09:33:58Z", "96%"),
+                    ],
+                    "unit": "percent",
+                    "annotation": "CPU spikes correlated with OOM events — GC thrashing before each kill.",
+                },
+            },
+            "checkout-service": {
+                "latency_p99": {
+                    "values": [
+                        ("2026-04-01T09:10:00Z", "180ms"),
+                        ("2026-04-01T09:15:00Z", "190ms"),
+                        ("2026-04-01T09:19:00Z", "3200ms"),
+                        ("2026-04-01T09:20:00Z", "5100ms"),
+                        ("2026-04-01T09:22:00Z", "1400ms"),
+                        ("2026-04-01T09:25:00Z", "5200ms"),
+                        ("2026-04-01T09:27:00Z", "2100ms"),
+                        ("2026-04-01T09:29:00Z", "4800ms"),
+                        ("2026-04-01T09:31:00Z", "5200ms"),
+                        ("2026-04-01T09:33:00Z", "5400ms"),
+                    ],
+                    "unit": "milliseconds",
+                    "annotation": "Latency spikes correlate with inventory-service restart cycles.",
+                },
+                "error_rate": {
+                    "values": [
+                        ("2026-04-01T09:10:00Z", "0.3%"),
+                        ("2026-04-01T09:15:00Z", "0.4%"),
+                        ("2026-04-01T09:19:00Z", "8.2%"),
+                        ("2026-04-01T09:20:00Z", "14.1%"),
+                        ("2026-04-01T09:22:00Z", "5.3%"),
+                        ("2026-04-01T09:25:00Z", "15.0%"),
+                        ("2026-04-01T09:27:00Z", "6.1%"),
+                        ("2026-04-01T09:29:00Z", "11.8%"),
+                        ("2026-04-01T09:31:00Z", "12.0%"),
+                        ("2026-04-01T09:33:00Z", "13.5%"),
+                    ],
+                    "unit": "percent",
+                    "annotation": "Timeout errors from inventory-service dependency.",
+                },
+            },
+        }
+
+        self.dependency_map = {
+            "api-gateway": {
+                "depends_on": ["checkout-service"],
+                "depended_by": [],
+            },
+            "checkout-service": {
+                "depends_on": ["inventory-service", "redis-cache"],
+                "depended_by": ["api-gateway"],
+            },
+            "inventory-service": {
+                "depends_on": ["redis-cache", "postgres-primary"],
+                "depended_by": ["checkout-service"],
+            },
+            "redis-cache": {
+                "depends_on": [],
+                "depended_by": ["checkout-service", "inventory-service"],
+            },
+            "postgres-primary": {
+                "depends_on": [],
+                "depended_by": ["inventory-service"],
+            },
+        }
+
+        self.recent_changes = {
+            "checkout-service": [
+                {
+                    "timestamp": "2026-03-30T11:00:00Z",
+                    "service": "checkout-service",
+                    "description": "Deployed v3.8.0",
+                    "changelog": "Added coupon validation at checkout. Minor UI fixes.",
+                },
+            ],
+            "inventory-service": [
+                {
+                    "timestamp": "2026-03-27T16:30:00Z",
+                    "service": "inventory-service",
+                    "description": "Deployed v2.1.4",
+                    "changelog": "Bumped catalog sync batch size from 500 to 5000 items per batch for throughput improvement.",
+                },
+                {
+                    "timestamp": "2026-04-01T09:14:00Z",
+                    "service": "inventory-service",
+                    "description": "Scheduled catalog sync job triggered",
+                    "changelog": "Cron job started full catalog sync — 12,400 items to process.",
+                },
+            ],
+            "api-gateway": [
+                {
+                    "timestamp": "2026-03-25T09:00:00Z",
+                    "service": "api-gateway",
+                    "description": "Deployed v5.0.2",
+                    "changelog": "Updated rate limiting configuration.",
+                },
+            ],
+        }
+
+        self.runbooks = {
+            "inventory-service": (
+                "## Runbook: inventory-service\n\n"
+                "### Common Issues\n\n"
+                "1. **OOMKilled / Memory Issues**\n"
+                "   - Check current memory usage: `kubectl top pod -l app=inventory-service`\n"
+                "   - Review Java heap settings: `-Xmx` should be ~75% of container memory limit.\n"
+                "   - If caused by batch processing, reduce batch size or increase memory limit.\n"
+                "   - Current memory limit: 512Mi. Recommended: 1Gi for large catalog syncs.\n"
+                "   - To increase limit: `kubectl set resources deployment/inventory-service --limits=memory=1Gi`\n\n"
+                "2. **High Latency / Timeouts**\n"
+                "   - Check downstream dependencies (redis-cache, postgres-primary).\n"
+                "   - Verify connection pool is not exhausted.\n"
+                "   - If service is in restart loop, check for OOMKilled events.\n\n"
+                "### Scaling\n"
+                "   ```\n"
+                "   kubectl scale deployment/inventory-service --replicas=3\n"
+                "   ```\n\n"
+                "### Escalation\n"
+                "   Contact: #inventory-oncall in Slack\n"
+            ),
+            "checkout-service": (
+                "## Runbook: checkout-service\n\n"
+                "### Common Issues\n\n"
+                "1. **High Latency**\n"
+                "   - Check upstream dependencies: inventory-service, redis-cache.\n"
+                "   - Verify circuit breaker state.\n"
+                "   - If inventory-service is down, checkout will degrade.\n\n"
+                "2. **Timeout Errors**\n"
+                "   - Default timeout to inventory-service: 3000ms.\n"
+                "   - If inventory-service is restarting frequently, consider enabling graceful degradation mode.\n\n"
+                "### Escalation\n"
+                "   Contact: #checkout-oncall in Slack\n"
+            ),
+            "oom troubleshooting": (
+                "## Runbook: OOM Troubleshooting (General)\n\n"
+                "### Symptoms\n"
+                "- Container status: OOMKilled\n"
+                "- Repeated restarts in `kubectl get pods`\n"
+                "- Java services: `java.lang.OutOfMemoryError: Java heap space`\n\n"
+                "### Investigation Steps\n"
+                "1. Check container memory limit vs actual usage: `kubectl top pod`\n"
+                "2. Review JVM heap settings: `-Xmx`, `-Xms`\n"
+                "3. Look for memory leak patterns: steady RSS climb → OOM → restart → climb\n"
+                "4. Check for recent changes that increased memory footprint (batch sizes, cache sizes).\n\n"
+                "### Immediate Mitigation\n"
+                "- Increase memory limit: `kubectl set resources deployment/<name> --limits=memory=<new_limit>`\n"
+                "- Reduce workload (e.g. smaller batch sizes).\n"
+                "- Restart with profiling enabled to capture heap dump: `-XX:+HeapDumpOnOutOfMemoryError`\n\n"
+                "### Long-term Fix\n"
+                "- Profile heap usage and fix leak.\n"
+                "- Use streaming/pagination instead of loading full batches into memory.\n"
+            ),
+        }
+
+    # ── abstract method implementations ─────────────────────────────
+
+    def get_incident_summary(self) -> str:
+        return self.incident_text
+
+    def get_services(self) -> dict:
+        return self.services
+
+    def get_tool_response(self, tool_name: str, parameters: dict) -> str:
+        dispatch = {
+            "get_service_status": self._handle_service_status,
+            "query_logs": self._handle_query_logs,
+            "query_metrics": self._handle_query_metrics,
+            "get_dependency_map": self._handle_dependency_map,
+            "consult_runbook": self._handle_runbook,
+            "check_recent_changes": self._handle_recent_changes,
+        }
+        handler = dispatch.get(tool_name)
+        if handler is None:
+            return f"Unknown tool: {tool_name}"
+        return handler(parameters)
+
+    def get_relevant_tools(self) -> list[str]:
+        return [
+            "get_service_status:checkout-service",
+            "query_logs:checkout-service",
+            "get_dependency_map:checkout-service",
+            "get_service_status:inventory-service",
+            "query_logs:inventory-service",
+            "query_metrics:inventory-service:memory",
+        ]
+
+    def get_tool_descriptions(self) -> dict:
+        return {
+            "query_logs": {
+                "services": list(self.services.keys()),
+                "severity_options": ["all", "error", "warning", "info"],
+            },
+            "query_metrics": {
+                "services": list(self.metrics.keys()),
+                "metrics": ["cpu", "memory", "error_rate", "latency", "connections"],
+            },
+            "get_service_status": {
+                "services": list(self.services.keys()),
+            },
+            "get_dependency_map": {
+                "services": list(self.dependency_map.keys()),
+                "note": "Omit service for full map",
+            },
+            "consult_runbook": {
+                "topics": list(self.runbooks.keys()),
+            },
+            "check_recent_changes": {
+                "services": list(self.recent_changes.keys()),
+                "note": "Omit service for all changes",
+            },
+        }
+
+    def grade_resolution(self, resolution: dict, step_count: int) -> dict:
+        score = 0.0
+        root_cause_text = str(resolution.get("root_cause", "")).lower()
+        affected = str(resolution.get("affected_service", "")).strip().lower()
+        recommendation_text = str(resolution.get("recommendation", "")).lower()
+
+        # Root cause identifies OOM/memory in inventory-service (0.30)
+        memory_keywords = ["memory", "oom", "outofmemory", "outofmemoryerror", "oomkilled"]
+        has_memory = any(kw in root_cause_text for kw in memory_keywords)
+        has_inventory = "inventory" in root_cause_text
+        root_cause_memory = has_memory and has_inventory
+        if root_cause_memory:
+            score += 0.30
+
+        # Identifies checkout latency caused by inventory (0.15)
+        upstream_keywords = ["upstream", "dependency"]
+        inventory_checkout_keywords_a = ["inventory"]
+        inventory_checkout_keywords_b = ["checkout", "timeout", "latency"]
+        upstream_match = any(kw in root_cause_text for kw in upstream_keywords)
+        inv_checkout_match = (
+            any(kw in root_cause_text for kw in inventory_checkout_keywords_a)
+            and any(kw in root_cause_text for kw in inventory_checkout_keywords_b)
+        )
+        if upstream_match or inv_checkout_match:
+            score += 0.15
+
+        # Affected service is inventory-service (0.15)
+        affected_normalized = re.sub(r"[^a-z0-9-]", "", affected)
+        if affected_normalized == "inventory-service":
+            score += 0.15
+        # Score 0.0 if they say checkout-service (no partial credit)
+
+        # Recommendation addresses memory (0.20)
+        rec_keywords = [
+            "memory limit", "heap", "batch", "stream", "1gi",
+            "increase memory", "reduce batch", "xmx",
+        ]
+        rec_match = any(kw in recommendation_text for kw in rec_keywords)
+        if rec_match:
+            score += 0.20
+
+        # Efficiency bonus (0.10)
+        if step_count <= 8:
+            score += 0.10
+        elif step_count <= 12:
+            score += 0.05
+
+        # No destructive recommendations (0.10) — present by default
+        destructive_keywords = ["restart-all", "drop", "delete"]
+        destructive = any(kw in recommendation_text for kw in destructive_keywords)
+        if not destructive:
+            score += 0.10
+
+        root_cause_correct = root_cause_memory
+        recommendation_correct = rec_match
+
+        score = max(0.0, min(1.0, score))
+        return {
+            "score": score,
+            "root_cause_correct": root_cause_correct,
+            "recommendation_correct": recommendation_correct,
+        }
+
+    # ── internal tool handlers ──────────────────────────────────────
+
+    def _handle_service_status(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        svc = self.services.get(service)
+        if svc is None:
+            return f"Service '{service}' not found."
+        return self._format_service_status(svc)
+
+    def _handle_query_logs(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        query = parameters.get("query", "").lower()
+        entries = self.log_entries.get(service)
+        if entries is None:
+            return f"No logs available for service '{service}'."
+        if query:
+            matched = [
+                e
+                for e in entries
+                if query in e["message"].lower()
+                or query in e.get("level", "").lower()
+                or query in e.get("source", "").lower()
+            ]
+        else:
+            matched = entries
+        return self._format_logs(matched)
+
+    def _handle_query_metrics(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        metric = parameters.get("metric", "").lower()
+        svc_metrics = self.metrics.get(service)
+        if svc_metrics is None:
+            return f"No metrics available for service '{service}'."
+        # Fuzzy substring match on metric name
+        for name, data in svc_metrics.items():
+            if metric in name.lower() or name.lower() in metric:
+                return self._format_metrics(name, data)
+        return f"No matching metric '{metric}' for service '{service}'."
+
+    def _handle_dependency_map(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        if service and service in self.dependency_map:
+            subset = {service: self.dependency_map[service]}
+            return self._format_dependency_map(subset)
+        return self._format_dependency_map(self.dependency_map)
+
+    def _handle_runbook(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        topic = parameters.get("topic", "").lower()
+        # Try service-specific runbook first
+        content = self.runbooks.get(service)
+        if content:
+            return self._format_runbook(content)
+        # Try topic match
+        for key, value in self.runbooks.items():
+            if topic and topic in key.lower():
+                return self._format_runbook(value)
+        return self._format_runbook("")
+
+    def _handle_recent_changes(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        changes = self.recent_changes.get(service)
+        if changes is None:
+            return f"No recent changes found for service '{service}'."
+        return self._format_changes(changes)

scenarios/task3_cascading_failure.py

@@ -0,0 +1,560 @@
+"""Task 3 — The Cascading Failure: analytics-worker long query exhausts postgres pool."""
+
+import re
+from typing import Dict, List
+
+from .base import BaseScenario
+
+
+class CascadingFailureScenario(BaseScenario):
+    """Multiple services degraded due to postgres connection pool exhaustion caused by analytics-worker."""
+
+    def __init__(self) -> None:
+        self.incident_text = (
+            "CRITICAL: Multiple services degraded. auth-service, user-profile-service, "
+            "notification-service all reporting errors. PagerDuty escalation triggered."
+        )
+
+        self.services: Dict[str, dict] = {
+            "auth-service": {
+                "name": "auth-service",
+                "status": "UNHEALTHY",
+                "error_rate": "98%",
+                "uptime": "87.2% (24h)",
+                "last_deploy": "v5.2.0 deployed 2 days ago",
+                "latency_p99": "timeout",
+            },
+            "user-profile-service": {
+                "name": "user-profile-service",
+                "status": "UNHEALTHY",
+                "error_rate": "85%",
+                "uptime": "91.0% (24h)",
+                "last_deploy": "v2.4.1 deployed 1 week ago",
+                "latency_p99": "timeout",
+            },
+            "notification-service": {
+                "name": "notification-service",
+                "status": "DEGRADED",
+                "error_rate": "45%",
+                "uptime": "94.5% (24h)",
+                "last_deploy": "v3.1.0 deployed 30 minutes ago",
+                "queue_depth": "15420 (normal < 100)",
+            },
+            "postgres-primary": {
+                "name": "postgres-primary",
+                "status": "DEGRADED",
+                "error_rate": "0%",
+                "uptime": "99.999% (30d)",
+                "connections": "50/50 active (pool exhausted), 23 waiting",
+            },
+            "analytics-worker": {
+                "name": "analytics-worker",
+                "status": "HEALTHY",
+                "error_rate": "0%",
+                "uptime": "99.9% (30d)",
+                "last_deploy": "v1.9.0 deployed 1 week ago",
+            },
+            "api-gateway": {
+                "name": "api-gateway",
+                "status": "DEGRADED",
+                "error_rate": "62%",
+                "uptime": "93.1% (24h)",
+                "last_deploy": "v8.0.3 deployed 3 days ago",
+                "latency_p99": "12.4s",
+            },
+        }
+
+        self.log_entries: Dict[str, List[dict]] = {
+            "auth-service": [
+                {
+                    "timestamp": "2026-04-01T14:32:10Z",
+                    "level": "ERROR",
+                    "message": "PSQLException: Cannot acquire connection from pool. Pool exhausted (max=50, active=50, idle=0)",
+                    "source": "com.auth.db.ConnectionManager.getConnection()",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:12Z",
+                    "level": "ERROR",
+                    "message": "PSQLException: Cannot acquire connection from pool. Pool exhausted (max=50, active=50, idle=0)",
+                    "source": "com.auth.db.ConnectionManager.getConnection()",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:14Z",
+                    "level": "ERROR",
+                    "message": "Failed to validate authentication token: database connection unavailable",
+                    "source": "com.auth.service.TokenValidator.validate()",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:16Z",
+                    "level": "WARN",
+                    "message": "Health check failing: cannot reach database",
+                    "source": "com.auth.health.DatabaseHealthCheck",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:18Z",
+                    "level": "ERROR",
+                    "message": "PSQLException: Cannot acquire connection from pool. Pool exhausted (max=50, active=50, idle=0)",
+                    "source": "com.auth.db.ConnectionManager.getConnection()",
+                },
+            ],
+            "user-profile-service": [
+                {
+                    "timestamp": "2026-04-01T14:32:11Z",
+                    "level": "ERROR",
+                    "message": "AuthenticationException: Token validation failed - auth-service unreachable",
+                    "source": "com.userprofile.auth.AuthClient.validateToken()",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:13Z",
+                    "level": "ERROR",
+                    "message": "HTTP 401 Unauthorized: auth-service returned connection timeout",
+                    "source": "com.userprofile.auth.AuthClient.validateToken()",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:15Z",
+                    "level": "WARN",
+                    "message": "Falling back to cached user profile for user_id=8821 (auth unavailable)",
+                    "source": "com.userprofile.service.ProfileService.getProfile()",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:17Z",
+                    "level": "ERROR",
+                    "message": "AuthenticationException: Token validation failed - auth-service unreachable",
+                    "source": "com.userprofile.auth.AuthClient.validateToken()",
+                },
+            ],
+            "notification-service": [
+                {
+                    "timestamp": "2026-04-01T14:32:12Z",
+                    "level": "ERROR",
+                    "message": "UserProfileException: Cannot fetch user preferences - user-profile-service returned 401",
+                    "source": "com.notification.client.UserProfileClient.getPreferences()",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:14Z",
+                    "level": "WARN",
+                    "message": "Notification queue depth: 15420 (threshold: 1000). Processing stalled.",
+                    "source": "com.notification.queue.QueueMonitor",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:16Z",
+                    "level": "ERROR",
+                    "message": "UserProfileException: Cannot fetch user preferences - user-profile-service returned 401",
+                    "source": "com.notification.client.UserProfileClient.getPreferences()",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:18Z",
+                    "level": "INFO",
+                    "message": "v3.1.0 deployment completed successfully. New feature: batch notification grouping.",
+                    "source": "com.notification.deploy.DeploymentHook",
+                },
+            ],
+            "postgres-primary": [
+                {
+                    "timestamp": "2026-04-01T14:10:05Z",
+                    "level": "LOG",
+                    "message": "duration: 1245032.456 ms  statement: SELECT e.event_id, e.event_type, e.created_at, u.user_name, u.email, s.session_data FROM events e JOIN users u ON e.user_id = u.id JOIN sessions s ON e.session_id = s.id WHERE e.created_at > '2026-01-01' AND e.event_type IN ('purchase', 'refund', 'chargeback') ORDER BY e.created_at  -- analytics_worker scheduled_report",
+                    "source": "postgres/slow_query_log",
+                },
+                {
+                    "timestamp": "2026-04-01T14:25:30Z",
+                    "level": "WARN",
+                    "message": "connection pool near capacity: 48/50 active connections",
+                    "source": "postgres/connection_monitor",
+                },
+                {
+                    "timestamp": "2026-04-01T14:28:00Z",
+                    "level": "ERROR",
+                    "message": "connection pool exhausted: 50/50 active connections, 12 clients waiting",
+                    "source": "postgres/connection_monitor",
+                },
+                {
+                    "timestamp": "2026-04-01T14:30:15Z",
+                    "level": "ERROR",
+                    "message": "connection pool exhausted: 50/50 active connections, 23 clients waiting. Longest query running for 1221s (analytics_worker)",
+                    "source": "postgres/connection_monitor",
+                },
+            ],
+            "analytics-worker": [
+                {
+                    "timestamp": "2026-04-01T14:10:00Z",
+                    "level": "INFO",
+                    "message": "Starting scheduled report: quarterly_event_analysis. Estimated rows: 4.2M",
+                    "source": "com.analytics.scheduler.ReportRunner",
+                },
+                {
+                    "timestamp": "2026-04-01T14:10:05Z",
+                    "level": "INFO",
+                    "message": "Executing query against postgres-primary (no read replica configured)",
+                    "source": "com.analytics.db.QueryExecutor",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:00Z",
+                    "level": "INFO",
+                    "message": "Query still running. Rows processed so far: 2.1M of estimated 4.2M",
+                    "source": "com.analytics.scheduler.ReportRunner",
+                },
+            ],
+            "api-gateway": [
+                {
+                    "timestamp": "2026-04-01T14:32:10Z",
+                    "level": "ERROR",
+                    "message": "Upstream timeout: auth-service did not respond within 5000ms",
+                    "source": "com.gateway.proxy.UpstreamHandler",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:12Z",
+                    "level": "ERROR",
+                    "message": "Upstream timeout: user-profile-service did not respond within 5000ms",
+                    "source": "com.gateway.proxy.UpstreamHandler",
+                },
+                {
+                    "timestamp": "2026-04-01T14:32:15Z",
+                    "level": "WARN",
+                    "message": "Circuit breaker OPEN for auth-service (failure rate: 98%)",
+                    "source": "com.gateway.circuit.CircuitBreaker",
+                },
+            ],
+        }
+
+        self.metrics: Dict[str, Dict[str, dict]] = {
+            "postgres-primary": {
+                "connections": {
+                    "values": [42, 44, 46, 48, 49, 50, 50, 50, 50, 50],
+                    "unit": "active connections (max 50)",
+                    "annotation": "Connection pool saturated at 14:28. 48 connections held by analytics_worker query. 23 queries waiting.",
+                },
+                "cpu": {
+                    "values": [15, 18, 22, 35, 42, 55, 58, 60, 62, 61],
+                    "unit": "percent",
+                    "annotation": "CPU elevated due to long-running analytics query (full table scan + joins).",
+                },
+            },
+            "analytics-worker": {
+                "cpu": {
+                    "values": [5, 5, 8, 12, 15, 15, 14, 15, 14, 15],
+                    "unit": "percent",
+                    "annotation": "Stable elevated CPU — analytics query in progress since 14:10.",
+                },
+                "memory": {
+                    "values": [210, 215, 220, 225, 230, 235, 240, 245, 248, 250],
+                    "unit": "Mi",
+                    "annotation": "Memory slowly climbing as query result set grows. Within limits (512Mi).",
+                },
+            },
+            "auth-service": {
+                "error_rate": {
+                    "values": [0.1, 0.1, 0.2, 5.0, 45.0, 88.0, 95.0, 97.0, 98.0, 98.0],
+                    "unit": "percent",
+                    "annotation": "Error rate spike correlates with postgres pool exhaustion at 14:28.",
+                },
+                "latency": {
+                    "values": [50, 55, 120, 800, 3000, 5000, 5000, 5000, 5000, 5000],
+                    "unit": "ms (p99)",
+                    "annotation": "Latency spike to timeout threshold. Auth requests waiting for DB connections.",
+                },
+            },
+            "notification-service": {
+                "queue_depth": {
+                    "values": [45, 50, 80, 350, 1200, 4500, 8200, 11000, 13800, 15420],
+                    "unit": "messages",
+                    "annotation": "Queue backlog growing rapidly. Processing stalled due to user-profile-service failures.",
+                },
+            },
+        }
+
+        self.dependency_map: Dict[str, dict] = {
+            "api-gateway": {
+                "depends_on": ["auth-service", "user-profile-service", "notification-service"],
+                "depended_by": [],
+            },
+            "auth-service": {
+                "depends_on": ["postgres-primary"],
+                "depended_by": ["api-gateway", "user-profile-service"],
+            },
+            "user-profile-service": {
+                "depends_on": ["auth-service"],
+                "depended_by": ["api-gateway", "notification-service"],
+            },
+            "notification-service": {
+                "depends_on": ["user-profile-service"],
+                "depended_by": ["api-gateway"],
+            },
+            "postgres-primary": {
+                "depends_on": [],
+                "depended_by": ["auth-service", "analytics-worker"],
+            },
+            "analytics-worker": {
+                "depends_on": ["postgres-primary"],
+                "depended_by": [],
+            },
+        }
+
+        self.recent_changes: Dict[str, List[dict]] = {
+            "notification-service": [
+                {
+                    "timestamp": "2026-04-01T14:02:00Z",
+                    "service": "notification-service",
+                    "description": "Deployed v3.1.0 — batch notification grouping feature",
+                    "changelog": "Added batch grouping for push notifications. No database schema changes.",
+                },
+            ],
+            "analytics-worker": [
+                {
+                    "timestamp": "2026-04-01T14:10:00Z",
+                    "service": "analytics-worker",
+                    "description": "Scheduled job started: quarterly_event_analysis",
+                    "changelog": "Automated quarterly report. Runs against postgres-primary (no read replica configured).",
+                },
+            ],
+            "": [  # all recent changes
+                {
+                    "timestamp": "2026-04-01T14:02:00Z",
+                    "service": "notification-service",
+                    "description": "Deployed v3.1.0 — batch notification grouping feature",
+                    "changelog": "Added batch grouping for push notifications. No database schema changes.",
+                },
+                {
+                    "timestamp": "2026-04-01T14:10:00Z",
+                    "service": "analytics-worker",
+                    "description": "Scheduled job started: quarterly_event_analysis",
+                    "changelog": "Automated quarterly report. Runs against postgres-primary (no read replica configured).",
+                },
+            ],
+        }
+
+        self.runbooks: Dict[str, str] = {
+            "connection pool exhausted": (
+                "Runbook: PostgreSQL Connection Pool Exhausted\n"
+                "1. Check active connections: SELECT count(*) FROM pg_stat_activity;\n"
+                "2. Identify long-running queries: SELECT pid, now()-query_start AS duration, query FROM pg_stat_activity WHERE state='active' ORDER BY duration DESC;\n"
+                "3. Kill the offending query: SELECT pg_terminate_backend(<pid>);\n"
+                "4. Monitor pool recovery — connections should free up within seconds.\n"
+                "5. Prevent recurrence: Set statement_timeout for batch jobs. Consider using a read replica for analytics."
+            ),
+            "connection pool": (
+                "Runbook: PostgreSQL Connection Pool Exhausted\n"
+                "1. Check active connections: SELECT count(*) FROM pg_stat_activity;\n"
+                "2. Identify long-running queries: SELECT pid, now()-query_start AS duration, query FROM pg_stat_activity WHERE state='active' ORDER BY duration DESC;\n"
+                "3. Kill the offending query: SELECT pg_terminate_backend(<pid>);\n"
+                "4. Monitor pool recovery — connections should free up within seconds.\n"
+                "5. Prevent recurrence: Set statement_timeout for batch jobs. Consider using a read replica for analytics."
+            ),
+            "authentication failure": (
+                "Runbook: Authentication Service Failures\n"
+                "1. Check auth-service health: GET /health\n"
+                "2. Verify database connectivity from auth-service.\n"
+                "3. Check for recent auth-service deployments.\n"
+                "4. If DB connection issue, check postgres connection pool status.\n"
+                "5. Escalate to DBA if postgres is the bottleneck."
+            ),
+            "notification queue": (
+                "Runbook: Notification Queue Backlog\n"
+                "1. Check queue depth and processing rate.\n"
+                "2. Verify upstream dependencies (user-profile-service).\n"
+                "3. Check for recent notification-service deployments.\n"
+                "4. If upstream is down, queue will naturally drain once restored.\n"
+                "5. Do NOT restart notification-service — this will lose queued messages."
+            ),
+        }
+
+    # ── public interface ────────────────────────────────────────────
+
+    def get_incident_summary(self) -> str:
+        return self.incident_text
+
+    def get_services(self) -> Dict[str, dict]:
+        return self.services
+
+    def get_tool_response(self, tool_name: str, parameters: dict) -> str:
+        handlers = {
+            "get_service_status": self._handle_service_status,
+            "query_logs": self._handle_query_logs,
+            "query_metrics": self._handle_query_metrics,
+            "get_dependency_map": self._handle_dependency_map,
+            "consult_runbook": self._handle_runbook,
+            "check_recent_changes": self._handle_recent_changes,
+        }
+        handler = handlers.get(tool_name)
+        if handler is None:
+            return f"Tool '{tool_name}' is not available."
+        return handler(parameters)
+
+    def get_relevant_tools(self) -> list[str]:
+        return [
+            "get_service_status:auth-service",
+            "query_logs:auth-service",
+            "get_service_status:postgres-primary",
+            "query_metrics:postgres-primary:connections",
+            "query_logs:postgres-primary",
+            "get_dependency_map",
+            "consult_runbook:connection pool",
+            "get_service_status:analytics-worker",
+            "query_metrics:analytics-worker",
+            "check_recent_changes",
+        ]
+
+    def get_tool_descriptions(self) -> dict:
+        return {
+            "query_logs": {
+                "services": list(self.services.keys()),
+                "severity_options": ["all", "error", "warning", "info"],
+            },
+            "query_metrics": {
+                "services": list(self.metrics.keys()),
+                "metrics": ["connections", "cpu", "memory", "error_rate", "latency", "queue_depth"],
+            },
+            "get_service_status": {
+                "services": list(self.services.keys()),
+            },
+            "get_dependency_map": {
+                "services": list(self.dependency_map.keys()),
+                "note": "Omit service for full map",
+            },
+            "consult_runbook": {
+                "topics": list(self.runbooks.keys()),
+            },
+            "check_recent_changes": {
+                "services": list(self.recent_changes.keys()),
+                "note": "Omit service for all changes",
+            },
+        }
+
+    def grade_resolution(self, resolution: dict, step_count: int) -> dict:
+        root_cause = resolution.get("root_cause", "").lower()
+        affected = resolution.get("affected_service", "").lower().strip().replace("_", "-")
+        recommendation = resolution.get("recommendation", "").lower()
+
+        score = 0.0
+
+        # Root cause identifies postgres connection pool exhaustion (0.20)
+        pool_terms = any(t in root_cause for t in ["connection pool", "connections"])
+        exhausted_terms = any(t in root_cause for t in ["exhausted", "maxed", "full", "saturated"])
+        pool_match = pool_terms and exhausted_terms
+        if pool_match:
+            score += 0.20
+
+        # Root cause identifies analytics-worker / long-running query (0.20)
+        analytics_match_a = "analytics" in root_cause
+        analytics_match_b = any(t in root_cause for t in ["query", "long-running", "holding connections", "long query"])
+        analytics_match = analytics_match_a and analytics_match_b
+        if analytics_match:
+            score += 0.20
+
+        # Understands cascade chain (0.10)
+        cascade_kw = "cascade" in root_cause
+        # Or mentions 3+ services in the chain
+        chain_services = ["auth", "user-profile", "notification", "postgres"]
+        chain_count = sum(1 for s in chain_services if s in root_cause)
+        if cascade_kw or chain_count >= 3:
+            score += 0.10
+
+        # Affected service is postgres-primary or analytics-worker (0.10)
+        valid_affected = {"postgres-primary", "postgres", "analytics-worker", "analytics"}
+        bad_affected = {"auth-service", "notification-service", "user-profile-service"}
+        if affected in valid_affected:
+            score += 0.10
+        elif affected in bad_affected:
+            score += 0.0  # explicit 0
+
+        # Does NOT blame notification-service deploy (0.05) — present by default
+        notif_blamed = ("notification" in root_cause) and any(
+            t in root_cause for t in ["deploy", "v3.1"]
+        )
+        if not notif_blamed:
+            score += 0.05
+
+        # Recommendation: kill query + prevent recurrence (0.15)
+        kill_match = any(t in recommendation for t in ["kill", "terminate"]) and "query" in recommendation
+        prevent_match = any(
+            t in recommendation
+            for t in ["timeout", "read replica", "pgbouncer", "statement_timeout", "pool"]
+        )
+        if kill_match and prevent_match:
+            score += 0.15
+        elif kill_match or prevent_match:
+            score += 0.07
+
+        # Efficiency bonus (0.10)
+        if step_count <= 12:
+            score += 0.10
+        elif step_count <= 16:
+            score += 0.05
+
+        # No destructive recommendations (0.10) — present by default
+        has_kill_query = bool(re.search(r"kill.*query|terminate.*query", recommendation))
+        destructive_terms = ["restart postgres", "drop connections", "restart-all"]
+        is_destructive = any(t in recommendation for t in destructive_terms) and not has_kill_query
+        if not is_destructive:
+            score += 0.10
+
+        root_cause_correct = pool_match and analytics_match
+        recommendation_correct = kill_match and prevent_match
+
+        score = max(0.0, min(1.0, score))
+        return {
+            "score": score,
+            "root_cause_correct": root_cause_correct,
+            "recommendation_correct": recommendation_correct,
+        }
+
+    # ── internal tool handlers ──────────────────────────────────────
+
+    def _handle_service_status(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        svc = self.services.get(service)
+        if svc is None:
+            return f"Service '{service}' not found in this environment."
+        return self._format_service_status(svc)
+
+    def _handle_query_logs(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        query = parameters.get("query", "").lower()
+        entries = self.log_entries.get(service)
+        if entries is None:
+            return f"No logs available for service '{service}'."
+        if query:
+            matched = [
+                e
+                for e in entries
+                if query in e["message"].lower()
+                or query in e.get("level", "").lower()
+                or query in e.get("source", "").lower()
+            ]
+        else:
+            matched = entries
+        if not matched:
+            return f"No matching log entries found for '{service}' with query '{query}'."
+        return self._format_logs(matched)
+
+    def _handle_query_metrics(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        metric = parameters.get("metric", "").lower()
+        svc_metrics = self.metrics.get(service)
+        if svc_metrics is None:
+            return f"No metrics available for service '{service}'."
+        for name, data in svc_metrics.items():
+            if metric in name.lower() or name.lower() in metric:
+                return self._format_metrics(name, data)
+        return f"No matching metric '{metric}' for service '{service}'."
+
+    def _handle_dependency_map(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        if service and service in self.dependency_map:
+            subset = {service: self.dependency_map[service]}
+            return self._format_dependency_map(subset)
+        return self._format_dependency_map(self.dependency_map)
+
+    def _handle_runbook(self, parameters: dict) -> str:
+        topic = parameters.get("topic", "").lower()
+        for key, value in self.runbooks.items():
+            if topic and topic in key.lower():
+                return self._format_runbook(value)
+        return self._format_runbook("")
+
+    def _handle_recent_changes(self, parameters: dict) -> str:
+        service = parameters.get("service", "")
+        if service and service in self.recent_changes:
+            return self._format_changes(self.recent_changes[service])
+        # No service specified — return all
+        return self._format_changes(self.recent_changes.get("", []))

server/__init__.py

@@ -0,0 +1 @@
+"""Server package."""

server/app.py

@@ -0,0 +1,24 @@
+"""FastAPI application for the Sentinel Environment."""
+
+from openenv.core.env_server import create_app
+
+from models import SentinelAction, SentinelObservation
+from server.sentinel_environment import SentinelEnvironment
+
+app = create_app(
+    SentinelEnvironment,
+    SentinelAction,
+    SentinelObservation,
+    env_name="sentinel_env",
+)
+
+
+def main():
+    """Entry point for direct execution."""
+    import uvicorn
+
+    uvicorn.run(app, host="0.0.0.0", port=8000)
+
+
+if __name__ == "__main__":
+    main()

server/requirements.txt

@@ -0,0 +1,4 @@
+openenv-core>=0.2.3
+fastapi>=0.104.0
+uvicorn>=0.24.0
+pydantic>=2.0.0

server/sentinel_environment.py

@@ -0,0 +1,198 @@
+"""SentinelEnvironment — core OpenEnv Environment for SRE incident triage."""
+
+import uuid
+from typing import Any, Optional
+
+from openenv.core.env_server.interfaces import Environment
+from openenv.core.env_server.types import Observation
+
+from grading.grader import grade
+from grading.rewards import compute_step_reward, _call_signature, _is_relevant
+from models import SentinelAction, SentinelObservation, SentinelState
+from scenarios.task1_smoking_gun import SmokingGunScenario
+from scenarios.task2_upstream_culprit import UpstreamCulpritScenario
+from scenarios.task3_cascading_failure import CascadingFailureScenario
+from tools.registry import AVAILABLE_TOOLS, dispatch, make_relevance_key
+
+MAX_STEPS = 20
+MAX_CONSECUTIVE_INVALID = 5
+
+SCENARIOS = {
+    1: SmokingGunScenario,
+    2: UpstreamCulpritScenario,
+    3: CascadingFailureScenario,
+}
+
+TASK_NAMES = {
+    1: "The Smoking Gun",
+    2: "The Upstream Culprit",
+    3: "The Cascading Failure",
+}
+
+
+class SentinelEnvironment(Environment):
+    """OpenEnv Environment for SRE incident triage."""
+
+    SUPPORTS_CONCURRENT_SESSIONS = True
+
+    def __init__(self) -> None:
+        super().__init__()
+        self._scenario = None
+        self._state = SentinelState()
+        self._cumulative_reward: float = 0.0
+        self._previous_calls: list[str] = []
+        self._consecutive_invalid: int = 0
+
+    # ── helpers ──────────────────────────────────────────────────────
+
+    def _make_obs(
+        self,
+        *,
+        tool_output: str = "",
+        error: str = "",
+        done: bool = False,
+        reward: Optional[float] = None,
+        tool_descriptions: Optional[dict] = None,
+    ) -> SentinelObservation:
+        return SentinelObservation(
+            incident_summary=self._scenario.get_incident_summary() if self._scenario else "",
+            tool_output=tool_output,
+            available_tools=AVAILABLE_TOOLS,
+            step_number=self._state.step_count,
+            max_steps=MAX_STEPS,
+            cumulative_reward=self._cumulative_reward,
+            last_action_error=error,
+            done=done,
+            reward=reward,
+            tool_descriptions=tool_descriptions or {},
+        )
+
+    def _handle_submit(self, params: dict, step_num: int) -> SentinelObservation:
+        required = ("root_cause", "affected_service", "recommendation")
+        missing = [f for f in required if not params.get(f)]
+        if missing:
+            return self._make_obs(
+                error=f"submit_resolution requires: {', '.join(missing)}",
+                done=False,
+                reward=0.0,
+            )
+
+        self._consecutive_invalid = 0
+        result = grade(self._scenario, params, step_num)
+        self._state.resolution_submitted = True
+        self._state.root_cause_correct = result["root_cause_correct"]
+        self._state.recommendation_correct = result["recommendation_correct"]
+        self._state.final_score = result["score"]
+        self._state.tools_called.append("submit_resolution")
+
+        return self._make_obs(
+            tool_output=f"Resolution graded. Score: {result['score']:.2f}",
+            done=True,
+            reward=result["score"],
+        )
+
+    # ── Environment interface ────────────────────────────────────────
+
+    def reset(
+        self,
+        seed: Optional[int] = None,
+        episode_id: Optional[str] = None,
+        **kwargs: Any,
+    ) -> SentinelObservation:
+        task_id = kwargs.get("task_id", 1)
+        if task_id not in SCENARIOS:
+            task_id = 1
+
+        self._scenario = SCENARIOS[task_id]()
+        self._cumulative_reward = 0.0
+        self._previous_calls = []
+        self._consecutive_invalid = 0
+
+        self._state = SentinelState(
+            episode_id=episode_id or str(uuid.uuid4()),
+            step_count=0,
+            task_id=task_id,
+            task_name=TASK_NAMES[task_id],
+        )
+
+        return self._make_obs(
+            done=False,
+            reward=None,
+            tool_descriptions=self._scenario.get_tool_descriptions(),
+        )
+
+    def step(
+        self,
+        action: SentinelAction,
+        timeout_s: Optional[float] = None,
+        **kwargs: Any,
+    ) -> SentinelObservation:
+        if self._scenario is None:
+            return SentinelObservation(
+                incident_summary="",
+                last_action_error="Environment not reset. Call reset() first.",
+                done=True,
+                reward=0.0,
+            )
+
+        self._state.step_count += 1
+        step_num = self._state.step_count
+
+        tool_name = action.tool_name
+        params = action.param_dict()
+
+        # ── submit_resolution ───────────────────────────────────────
+        if tool_name == "submit_resolution":
+            return self._handle_submit(params, step_num)
+
+        # ── regular tool dispatch ───────────────────────────────────
+        output, is_valid = dispatch(tool_name, params, self._scenario)
+
+        if not is_valid:
+            self._consecutive_invalid += 1
+        else:
+            self._consecutive_invalid = 0
+
+        # compute reward
+        reward = compute_step_reward(
+            tool_name,
+            params,
+            is_valid,
+            self._scenario.get_relevant_tools(),
+            self._previous_calls,
+        )
+        self._cumulative_reward += reward
+
+        # track call
+        sig = _call_signature(tool_name, params)
+        self._previous_calls.append(sig)
+        self._state.tools_called.append(f"{tool_name}({params})")
+
+        # track relevant
+        if is_valid and _is_relevant(tool_name, params, self._scenario.get_relevant_tools()):
+            self._state.relevant_tools_called.append(f"{tool_name}({params})")
+
+        # ── check termination ───────────────────────────────────────
+        done = False
+        error_msg = "" if is_valid else output
+
+        if self._consecutive_invalid >= MAX_CONSECUTIVE_INVALID:
+            done = True
+            output = f"{output}\n\nEpisode terminated: {MAX_CONSECUTIVE_INVALID} consecutive invalid actions."
+            self._state.final_score = 0.0
+
+        if step_num >= MAX_STEPS:
+            done = True
+            output = f"{output}\n\nEpisode terminated: maximum steps ({MAX_STEPS}) reached."
+            self._state.final_score = 0.0
+
+        return self._make_obs(
+            tool_output=output,
+            error=error_msg,
+            done=done,
+            reward=reward,
+        )
+
+    @property
+    def state(self) -> SentinelState:
+        return self._state

tests/conftest.py

@@ -0,0 +1,28 @@
+"""Shared fixtures for Sentinel tests."""
+
+import pytest
+
+from scenarios.task1_smoking_gun import SmokingGunScenario
+from scenarios.task2_upstream_culprit import UpstreamCulpritScenario
+from scenarios.task3_cascading_failure import CascadingFailureScenario
+from server.sentinel_environment import SentinelEnvironment
+
+
+@pytest.fixture
+def task1():
+    return SmokingGunScenario()
+
+
+@pytest.fixture
+def task2():
+    return UpstreamCulpritScenario()
+
+
+@pytest.fixture
+def task3():
+    return CascadingFailureScenario()
+
+
+@pytest.fixture
+def env():
+    return SentinelEnvironment()

tests/test_environment.py

@@ -0,0 +1,102 @@
+"""Tests for SentinelEnvironment reset, step, submit, and termination."""
+
+import pytest
+
+from models import SentinelAction
+from server.sentinel_environment import SentinelEnvironment, MAX_STEPS, MAX_CONSECUTIVE_INVALID
+
+
+def _action(tool_name: str, **params):
+    return SentinelAction.model_validate({"tool_name": tool_name, "parameters": params})
+
+
+class TestReset:
+    def test_reset_returns_observation(self, env):
+        obs = env.reset(task_id=1)
+        assert obs.done is False
+        assert obs.incident_summary != ""
+        assert obs.step_number == 0
+
+    def test_reset_includes_tool_descriptions(self, env):
+        obs = env.reset(task_id=1)
+        assert obs.tool_descriptions != {}
+        assert "query_logs" in obs.tool_descriptions
+
+    def test_reset_invalid_task_defaults_to_1(self, env):
+        obs = env.reset(task_id=999)
+        assert obs.done is False
+        assert env.state.task_id == 1
+
+
+class TestStep:
+    def test_step_without_reset(self, env):
+        action = _action("get_service_status", service="auth")
+        obs = env.step(action)
+        assert obs.done is True
+        assert "not reset" in obs.last_action_error.lower()
+
+    def test_valid_step_returns_output(self, env):
+        env.reset(task_id=1)
+        action = _action("get_service_status", service="payment-api")
+        obs = env.step(action)
+        assert obs.tool_output != ""
+        assert obs.done is False
+        assert obs.step_number == 1
+
+    def test_step_no_tool_descriptions(self, env):
+        env.reset(task_id=1)
+        action = _action("get_service_status", service="payment-api")
+        obs = env.step(action)
+        assert obs.tool_descriptions == {}
+
+    def test_invalid_tool(self, env):
+        env.reset(task_id=1)
+        # Use a raw dict bypass since pydantic rejects unknown tools
+        # Instead, test via unknown service which is still valid dispatch
+        action = _action("get_service_status", service="nonexistent")
+        obs = env.step(action)
+        assert obs.done is False  # valid tool, just unknown service
+
+
+class TestSubmit:
+    def test_submit_resolution_grades(self, env):
+        env.reset(task_id=1)
+        action = _action(
+            "submit_resolution",
+            root_cause="Missing DB_CONNECTION_STRING after v2.3.1 deploy",
+            affected_service="payment-api",
+            recommendation="Rollback to v2.3.0",
+        )
+        obs = env.step(action)
+        assert obs.done is True
+        assert obs.reward is not None
+        assert obs.reward > 0
+
+    def test_submit_missing_fields(self, env):
+        env.reset(task_id=1)
+        action = _action(
+            "submit_resolution",
+            root_cause="",
+            affected_service="",
+            recommendation="",
+        )
+        obs = env.step(action)
+        assert obs.last_action_error != ""
+        assert obs.done is False
+
+
+class TestTermination:
+    def test_max_steps(self, env):
+        env.reset(task_id=1)
+        for _ in range(MAX_STEPS):
+            action = _action("get_service_status", service="payment-api")
+            obs = env.step(action)
+        assert obs.done is True
+        assert "maximum steps" in obs.tool_output.lower()
+
+    def test_consecutive_invalid_not_triggered_by_valid(self, env):
+        env.reset(task_id=1)
+        for _ in range(MAX_CONSECUTIVE_INVALID + 1):
+            action = _action("get_service_status", service="payment-api")
+            obs = env.step(action)
+        assert obs.done is False  # valid actions don't trigger termination

tests/test_grading.py

@@ -0,0 +1,96 @@
+"""Tests for per-task grading: perfect scores, partial credit, wrong service, destructive penalty."""
+
+import pytest
+
+
+class TestTask1Grading:
+    def test_perfect_resolution(self, task1):
+        result = task1.grade_resolution(
+            {
+                "root_cause": "Missing DB_CONNECTION_STRING env var after deploy v2.3.1",
+                "affected_service": "payment-api",
+                "recommendation": "Rollback to v2.3.0 or set the DB_CONNECTION_STRING env var",
+            },
+            step_count=3,
+        )
+        assert result["score"] >= 0.80
+
+    def test_wrong_service(self, task1):
+        result = task1.grade_resolution(
+            {
+                "root_cause": "Missing DB_CONNECTION_STRING env var after deploy v2.3.1",
+                "affected_service": "order-service",
+                "recommendation": "Rollback",
+            },
+            step_count=3,
+        )
+        # Should lose the affected_service points
+        assert result["score"] <= 0.85
+
+    def test_empty_resolution(self, task1):
+        result = task1.grade_resolution(
+            {"root_cause": "", "affected_service": "", "recommendation": ""},
+            step_count=1,
+        )
+        assert result["score"] <= 0.20
+
+
+class TestTask2Grading:
+    def test_perfect_resolution(self, task2):
+        result = task2.grade_resolution(
+            {
+                "root_cause": "inventory-service OOM memory leak from batch processing causing checkout-service timeout",
+                "affected_service": "inventory-service",
+                "recommendation": "Increase memory limit to 1Gi and reduce batch size or stream results",
+            },
+            step_count=4,
+        )
+        assert result["score"] >= 0.70
+
+    def test_wrong_root_cause(self, task2):
+        result = task2.grade_resolution(
+            {
+                "root_cause": "Network partition",
+                "affected_service": "inventory-service",
+                "recommendation": "Restart networking",
+            },
+            step_count=3,
+        )
+        assert result["score"] < 0.40
+
+
+class TestTask3Grading:
+    def test_perfect_resolution(self, task3):
+        result = task3.grade_resolution(
+            {
+                "root_cause": "analytics-worker long-running query exhausted the connection pool, cascade to auth-service, user-profile-service, notification-service",
+                "affected_service": "postgres-primary",
+                "recommendation": "Kill the query and set statement_timeout, use read replica for analytics",
+            },
+            step_count=5,
+        )
+        assert result["score"] >= 0.70
+
+    def test_blames_notification_deploy(self, task3):
+        result = task3.grade_resolution(
+            {
+                "root_cause": "notification-service deploy v3.1 caused the failure",
+                "affected_service": "notification-service",
+                "recommendation": "Rollback notification-service",
+            },
+            step_count=3,
+        )
+        # Should score poorly — wrong root cause and wrong affected service
+        assert result["score"] <= 0.20
+
+    def test_partial_credit_pool_only(self, task3):
+        result = task3.grade_resolution(
+            {
+                "root_cause": "postgres connection pool exhausted and full",
+                "affected_service": "postgres-primary",
+                "recommendation": "Increase pool size",
+            },
+            step_count=3,
+        )
+        # Pool identified but not analytics-worker
+        assert 0.20 <= result["score"] <= 0.65

tests/test_models.py

@@ -0,0 +1,60 @@
+"""Tests for typed action models and discriminated union."""
+
+import pytest
+from pydantic import ValidationError
+
+from models import (
+    SentinelAction,
+    QueryLogsAction,
+    SubmitResolutionAction,
+    SentinelObservation,
+)
+
+
+class TestDeserialization:
+    def test_query_logs(self):
+        raw = {"tool_name": "query_logs", "parameters": {"service": "auth", "query": "error"}}
+        action = SentinelAction.model_validate(raw)
+        assert action.tool_name == "query_logs"
+        assert action.param_dict() == {"service": "auth", "query": "error", "severity": "all"}
+
+    def test_submit_resolution(self):
+        raw = {
+            "tool_name": "submit_resolution",
+            "parameters": {"root_cause": "x", "affected_service": "y", "recommendation": "z"},
+        }
+        action = SentinelAction.model_validate(raw)
+        assert action.param_dict()["root_cause"] == "x"
+
+    def test_get_dependency_map_defaults(self):
+        raw = {"tool_name": "get_dependency_map", "parameters": {}}
+        action = SentinelAction.model_validate(raw)
+        assert action.param_dict() == {"service": ""}
+
+
+class TestInvalidRejection:
+    def test_unknown_tool_name(self):
+        raw = {"tool_name": "hack_server", "parameters": {}}
+        with pytest.raises(ValidationError):
+            SentinelAction.model_validate(raw)
+
+    def test_missing_required_param(self):
+        raw = {"tool_name": "query_logs", "parameters": {}}
+        with pytest.raises(ValidationError):
+            SentinelAction.model_validate(raw)
+
+
+class TestDiscriminator:
+    def test_schema_has_discriminator(self):
+        schema = SentinelAction.model_json_schema()
+        assert "$defs" in schema
+
+
+class TestObservation:
+    def test_tool_descriptions_default_empty(self):
+        obs = SentinelObservation()
+        assert obs.tool_descriptions == {}
+
+    def test_tool_descriptions_populated(self):
+        obs = SentinelObservation(tool_descriptions={"query_logs": {"services": ["a"]}})
+        assert "query_logs" in obs.tool_descriptions

tests/test_tools.py

@@ -0,0 +1,41 @@
+"""Tests for tool dispatch and make_relevance_key."""
+
+import pytest
+
+from tools.registry import dispatch, make_relevance_key
+
+
+class TestMakeRelevanceKey:
+    def test_tool_with_service(self):
+        assert make_relevance_key("query_logs", {"service": "auth"}) == "query_logs:auth"
+
+    def test_tool_with_service_and_metric(self):
+        key = make_relevance_key("query_metrics", {"service": "pg", "metric": "cpu"})
+        assert key == "query_metrics:pg:cpu"
+
+    def test_tool_with_topic(self):
+        key = make_relevance_key("consult_runbook", {"topic": "connection pool"})
+        assert key == "consult_runbook:connection pool"
+
+    def test_tool_no_params(self):
+        assert make_relevance_key("check_recent_changes", {}) == "check_recent_changes"
+
+    def test_tool_empty_service(self):
+        assert make_relevance_key("get_dependency_map", {"service": ""}) == "get_dependency_map"
+
+
+class TestDispatch:
+    def test_unknown_tool(self, task1):
+        output, is_valid = dispatch("hack_server", {}, task1)
+        assert is_valid is False
+        assert "Unknown tool" in output
+
+    def test_valid_tool(self, task1):
+        output, is_valid = dispatch("get_service_status", {"service": "payment-api"}, task1)
+        assert is_valid is True
+        assert output != ""
+
+    def test_submit_resolution_passthrough(self, task1):
+        output, is_valid = dispatch("submit_resolution", {}, task1)
+        assert is_valid is True
+        assert output == ""

tools/__init__.py

@@ -0,0 +1 @@
+"""Tool registry package."""

tools/registry.py

@@ -0,0 +1,49 @@
+"""Tool registry — validates and dispatches tool calls to the active scenario."""
+
+from typing import Tuple
+
+from scenarios.base import BaseScenario
+
+AVAILABLE_TOOLS = [
+    "query_logs",
+    "query_metrics",
+    "get_service_status",
+    "get_dependency_map",
+    "consult_runbook",
+    "check_recent_changes",
+    "submit_resolution",
+]
+
+
+def make_relevance_key(tool_name: str, params: dict) -> str:
+    """Build a colon-joined relevance key from tool name and significant params.
+
+    Examples:
+        make_relevance_key("query_logs", {"service": "auth"}) -> "query_logs:auth"
+        make_relevance_key("get_dependency_map", {}) -> "get_dependency_map"
+    """
+    parts = [tool_name]
+    for k in ("service", "metric", "topic"):
+        v = params.get(k)
+        if v:
+            parts.append(str(v))
+    return ":".join(parts)
+
+
+def dispatch(tool_name: str, params: dict, scenario: BaseScenario) -> Tuple[str, bool]:
+    """Dispatch a tool call to the scenario.
+
+    Returns (output_text, is_valid).
+    is_valid=False means the tool_name was unknown or parameters were malformed.
+    """
+    if tool_name not in AVAILABLE_TOOLS:
+        return (
+            f"Unknown tool '{tool_name}'. Available tools: {', '.join(AVAILABLE_TOOLS)}",
+            False,
+        )
+
+    if tool_name == "submit_resolution":
+        # submit_resolution is handled by the environment directly, not here
+        return ("", True)
+
+    return (scenario.get_tool_response(tool_name, params), True)