README.md

metadata

title: IncidentOps SRE Environment
emoji: 🚨
colorFrom: red
colorTo: gray
sdk: docker
app_port: 8000

IncidentOps — AI Incident Response Training Environment

IncidentOps is a production-grade OpenEnv environment that trains AI agents to respond to real-world production incidents, just like an on-call Site Reliability Engineer (SRE).

The agent interacts with a simulated microservice infrastructure through a realistic text-based terminal, executing commands to triage alerts, investigate logs and metrics, identify root causes, and apply remediations.

Why this matters: Incident response is one of the highest-stakes, time-sensitive tasks in software engineering. Training AI agents to perform it well has immediate real-world value for DevOps/SRE teams.

---

Environment Description

The agent acts as an on-call SRE engineer who receives a pager alert. They must:

1. Triage — Check active alerts and system-wide dashboard to understand impact
2. Investigate — Dig into logs, metrics, distributed traces, and diagnostics
3. Identify Root Cause — Determine which service failed and why
4. Remediate — Apply the correct fix (restart, rollback, failover, config change)
5. Resolve — Confirm recovery and close the incident

The environment simulates realistic microservice architectures with:

• Service dependency graphs (5–8 services per scenario)
• Timestamped log streams showing the failure progression
• Performance metrics (CPU, memory, latency percentiles, error rates)
• Distributed request traces across the service mesh
• Multiple alerts with varying severity — including red-herring alerts

---

Action Space

The agent issues text commands through a terminal-style interface:

Command	Description	Example
help	List all available commands	help
status	System-wide service dashboard	status
alerts	Active alerts with severity and details	alerts
logs <service>	Recent log entries for a service	logs api-gateway
metrics <service>	CPU / memory / latency / error-rate	metrics database-primary
trace <request_id>	Follow a request across services	trace req-48219
diagnose <service>	Deep diagnostic with recommendations	diagnose payment-processor
restart <service>	Restart a service	restart payment-processor
scale <service> <n>	Scale replica count	scale read-service 6
rollback <service>	Roll back last deployment	rollback api-gateway
failover <service>	Promote standby to primary	failover database-primary
config <service> <key> <val>	Update live configuration	config api-gateway pool_size 100
notify <channel> <msg>	Post status update to team	notify oncall investigating
resolve	Declare the incident resolved (ends episode)	resolve

Action Model

class IncidentAction(Action):
    command: str    # Full command string including arguments

---

Observation Space

After each action, the environment returns:

class IncidentObservation(Observation):
    output: str              # Terminal-style output from the command
    timestamp: str           # Current simulation time (ISO-8601)
    alert_count: int         # Number of active alerts remaining
    severity: str            # Incident severity: "critical" | "high" | "medium" | "low" | "none"
    affected_services: list  # Services currently degraded or down
    done: bool               # True when episode has ended
    reward: float            # Score in open interval (0.01, 0.99)
    metadata: dict           # task_name, episode_id, step, final_score

---

Tasks

Three tasks of increasing difficulty. Select at reset time via task_name parameter.

Task 1: service-restart — OOM Service Crash (Easy)

Scenario: A payment-processor microservice has been OOM-killed (Out of Memory). It has entered CrashLoopBackOff. Three alerts are firing including a critical alert.

Objective: Identify the crashing service and restart it.

Expected episode length: 3–6 steps

Grading:

• Checked alerts: +5%
• Checked payment-processor logs/metrics: +12%
• Found root cause (payment-processor): +30%
• Applied correct fix (restart payment-processor): +30%
• Sent a status notification: +5%
• Efficiency bonus (≤ 4 steps): +5%

Baseline score: ~0.65

---

Task 2: config-drift — Connection Pool Exhaustion (Medium)

Scenario: A recent deployment to api-gateway (v3.1.5) introduced a config regression — the connection pool size was set to 5 (was 100). This causes connection pool exhaustion, cascading timeouts across 3 downstream services, and a critical + high alert combo.

Objective: Identify the config regression and fix it (rollback or config api-gateway pool_size 100).

Expected episode length: 6–12 steps

Grading:

• Investigated 2+ services: +10%
• Diagnosed api-gateway: +30%
• Applied correct fix (rollback or config): +30%
• Sent notification: +5%
• Efficiency bonus: +5%

Baseline score: ~0.58

---

Task 3: cascading-failure — Disk I/O → Cache Stampede → API Overload (Hard)

Scenario: A complex multi-service cascading failure:

• database-primary disk I/O saturated (iowait 82%)
• Replication lag builds to 182 seconds
• cache-layer gets a stampede (cache hit rate drops from 94% to 16%, CPU hits 99.8%)
• api-gateway becomes overloaded (72% error rate)
• Two red-herring alerts are included (notification-service delay, inventory maintenance window)

Objective: Full incident lifecycle — identify disk I/O root cause, execute failover database-primary then restart cache-layer, verify recovery.

Expected episode length: 10–20 steps

Grading:

• Explored full causal chain (DB → cache → API): +30%
• Identified database-primary as root cause: +25%
• DB failover: +18%
• Cache restart: +12%
• Both steps executed: +30% total (instead of +18+12)
• Notification: +5%
• Efficiency + avoided red herrings: up to +10%

Baseline score: ~0.48

---

Reward Function

Rewards are dense — they provide signal on every step, not just at episode end.

Phase	Range	Signal
Investigation steps	0.01–0.05 incremental	+0.05 for root-cause service, +0.03 for affected, +0.01 for healthy
Root cause identification	0.05–0.30	Based on depth of investigation
Correct remediation	0.30	On correct fix applied to correct service
Wrong remediation	−0.05	Restarting healthy services penalised
Communication	0.05	At least one notify issued
Efficiency	0.00–0.10	Inverse of excess steps taken

All rewards are clamped to the open interval (0.01, 0.99) to satisfy the OpenEnv validator.

---

Setup & Usage

Prerequisites

• Python 3.10+
• pip install openenv-core

Local Development (without Docker)

# Install
pip install -e ".[dev]"

# Run the server locally
uvicorn server.app:app --host 0.0.0.0 --port 8000 --reload

# In a separate terminal — connect and interact
python - <<'EOF'
from incident_ops_env import IncidentAction, IncidentOpsEnv

with IncidentOpsEnv(base_url="http://localhost:8000").sync() as env:
    result = env.reset(task_name="service-restart")
    print(result.observation.output)

    for cmd in ["alerts", "logs payment-processor", "restart payment-processor", "resolve"]:
        result = env.step(IncidentAction(command=cmd))
        print(result.observation.output)
        if result.observation.done:
            print(f"\nFinal score: {result.reward:.3f}")
            break
EOF

Docker

# Build
docker build -f server/Dockerfile -t incident-ops-env:latest .

# Run
docker run -p 8000:8000 incident-ops-env:latest

# Validate
openenv validate

Running the Baseline Inference Script

export HF_TOKEN="your-token"
export API_BASE_URL="https://router.huggingface.co/v1"
export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
export INCIDENT_BASE_URL="http://localhost:8000"

python inference.py

Running Tests

pip install ".[dev]"
pytest tests/test_env.py -v

---

Baseline Scores

Scores achieved by Qwen/Qwen2.5-72B-Instruct via HuggingFace Inference:

Task	Difficulty	Baseline Score
service-restart	Easy	~0.65
config-drift	Medium	~0.58
cascading-failure	Hard	~0.48

A perfect agent would score ~0.95 on all tasks. These baseline scores leave significant headroom for RL training to improve agent performance.

---

Project Structure

.
├── openenv.yaml          # OpenEnv manifest
├── pyproject.toml        # Package and dependency configuration
├── README.md             # This file
├── inference.py          # Baseline inference script
├── __init__.py           # Package exports
├── models.py             # Pydantic Action/Observation models
├── client.py             # EnvClient subclass
├── tests/
│   └── test_env.py       # Unit tests
└── server/
    ├── __init__.py
    ├── app.py             # FastAPI application
    ├── Dockerfile         # Container image
    ├── environment.py     # Core IncidentOpsEnvironment class
    ├── scenarios.py       # Three incident scenario definitions
    ├── graders.py         # Task-specific scoring functions
    └── simulation.py      # System simulation engine

---

HuggingFace Space

URL: https://chandan123467896uyjh.hf.space

Team: CodeBlockers

---

License

MIT License. See LICENSE file.