docs/ARCHITECTURE.md

# Architecture

> **The single insight that ties the whole pitch together: each tier escalates a *different* dimension — compute (Basic) → horizon (Advanced) → realism (Max) — not just scenario count.** If you only read one paragraph of this document, read this one.

This document explains why sre-gym ships three tiers, what each tier defends as a research question, and how the package is structured to make that defence visible from a 30-second skim of the repo.

---

## 1. The dimensional-escalation insight

The standard pattern in agentic-LLM evaluation, surveyed across SWE-bench Lite/Verified/Pro, MLE-bench Low/Med/High, ITBench-static/live, WebArena/-Verified/-Hard, and CRMArena/-Pro, is to escalate a single axis (volume, complexity, dataset size, horizon length) across difficulty bands. That works for benchmarks whose underlying capability is one-dimensional. SRE is not one-dimensional.

A junior on-call engineer learning to triage faces a fundamentally different bottleneck (cognitive efficiency under tight context) than a senior SRE running a multi-incident postmortem (state tracking across long horizons), which faces a fundamentally different bottleneck than an enterprise platform team operating against an actively chaos-engineered cluster (operating in a partially-observable, adversarial, irreversible world). Their training signals, episode shapes, observation richness, and reward structures should not look the same.

sre-gym takes that observation seriously and stratifies its tiers along *the dimension that actually limits the persona's training loop*:

| Tier | Bottleneck | Persona implication | Episode shape |
|---|---|---|---|
| Basic | **Compute** ($30 of HF credits, 1 A100 ~12h) | Pre-digested observations, dense reward shaping, 8K context, 11-action space | 8–13 ticks |
| Advanced | **Horizon** ($300–500 budget, 1–2 A100-days) | Multi-incident chains, partial observability, 28-action space, on-call peer | 60–90 ticks |
| Max | **Realism** (8×A100/H100, real chaos eng) | Ephemeral docker-compose / k3d, real `kubectl` / Vercel / Stripe APIs, subprocess shell, learned-critic rewards | 110–180+ actions, real wall-clock minutes |

This framing is *defensible as research*: the claim is that "training on a tier-1 environment that's causally rich but compute-cheap will produce a 3B specialist that beats Haiku on incident triage" is testable in 12 hours. The claim that "training on a tier-3 environment that includes real subprocess-shell access will produce an SRE agent that can actually go on-call" is testable in *months* and tens of thousands of dollars. Those are different research questions and the environment should make them visibly different.

---

## 2. Repository layout, with the design intent annotated

```
sre-enginnerllm/
├── sre_gym/                          # Tier-aware public package
│   ├── env.py                        # SREGym(tier=...) factory; Basic delegates,
│   │                                 # Advanced/Max raise TierNotRunnableError
│   │                                 # carrying a docs pointer.
│   ├── tier.py                       # Tier enum + TierConfig; the
│   │                                 # escalation_dimension field is the
│   │                                 # load-bearing piece of the pitch.
│   ├── advanced/scenarios/*.yaml     # 3 reference scenarios, real YAML, real
│   │                                 # topology, real reward dimensions, real
│   │                                 # reference traces — but the simulator
│   │                                 # backing them is intentionally not built.
│   └── max/                          # 1 fully-specced family, real
│       ├── families/*.yaml           # docker-compose, real chaos library,
│       ├── chaos/*.yaml              # real workload generator config.
│       └── compose/*.yaml            # Provisioning is left to the operator.
│
└── unified_incident_env/             # Basic-tier core; the v2 surface that the
                                      # HF Space serves and openenv.yaml declares.
```

Two design choices worth calling out:

**(a) Basic delegates to `unified_incident_env` rather than reimplementing.** The Basic tier's runnable surface is the existing v2 environment, kept verbatim — including its 36+ test suite, scripted-optimal baselines, and HF Space deployment. The `sre_gym` wrapper is intentionally thin: it adds the tier flag, the introspection methods, and the YAML-spec loader for the design-only tiers. This is the difference between "a single-tier env that's hard to extend" and "an env that visibly carries the three-tier story while still passing every Basic-tier test it ever passed."

**(b) Advanced and Max are shipped as data, not as code.** The YAML specs in `sre_gym/advanced/scenarios/` and `sre_gym/max/families/` are *real*: they reference real topologies, real action sets, real reward dimensions, real chaos patterns, and real reference traces. They're loaded by `SREGym.list_scenarios()` and renderable by the playground. What's missing is the simulator backing them — and that's deliberate. Building a credible Max-tier simulator in 36 hours is a fantasy; building a credible Max-tier *spec* that downstream operators can lift into a $40-150/day cluster is achievable. The cost of pretending Max is shipped runnable would be zero credibility with judges who have run real chaos engineering. The cost of shipping it as a credible vision is one extra YAML file.

---

## 3. The Basic tier in detail

The Basic tier is the only tier we trained against, so it gets the most concrete defence.

### 3.1 What it is

12 base templates × 5 procgen variants = 72 deterministic scenarios. Procgen jitters metric values, deploy timestamps, and noise-service rotation while preserving the causal structure — so a trained agent can't memorize fingerprints. Holding out one variant per template gives a 60-train / 12-eval split.

### 3.2 The 12 templates and what each one teaches

| # | Template | Skill | Decoy / red herring |
|---|---|---|---|
| 1 | `worker_deploy_cascade` | deploy-history reasoning | none — the easy entry point |
| 2 | `db_config_rollout` | config-vs-code disambiguation | concurrent worker deploy |
| 3 | `gateway_auth_rollout` | wrong-loud-service trap | worker queue-depth alert |
| 4 | `payment_webhook_misconfig` | downstream symptom (Stripe) | DB write-rate drop |
| 5 | `schema_drift_missing_migration` | application vs DB blame | DB looks healthy |
| 6 | `cache_stale_state` | metrics-look-good-but-customers-don't | cache hit rate is *up* |
| 7 | `dep_degradation` | "your service vs theirs" | worker CPU is loud |
| 8 | `memory_leak_oom` | restart count > error count | DB CPU spikes look like DB fault |
| 9 | `auth_token_expiry` | cross-service credential propagation | gateway is the loudest service |
| 10 | `network_partition` | trust connectivity, not self-reports | cache reports healthy in own metrics |
| 11 | `rate_limit_retry_storm` | counterintuitive (more retries = worse) | DB CPU/connections look pathological |
| 12 | `migration_lock` | lock contention without crash | worker errors look like a worker fault |

Each template contains a different **cognitive failure mode**. A 12-template catalogue with 12 different failure modes is a denser training signal than a 60-template catalogue that all reduce to "look at the deploy that just happened" — that's the depth-not-quantity argument.

### 3.3 Why these compute knobs

- **8K context** — fits the entire Basic episode (12 ticks × ~600 tokens of observation + ~80 tokens of action) inside the trained model's working set without truncation. Letting the trajectory spill into 16K context would force a smaller model (more A100 hours per token) or a longer training run (more wall-clock).
- **11 actions** — small enough that GRPO group-relative advantages converge in 600–1000 steps. Add 5 more actions and the policy has 50% more dimensions to explore at the same compute budget, which usually means a noisier gradient.
- **5-component dense reward** — recovery + containment + verification + impact + efficiency, with shaped intermediate signal (potential-function differences). Pure terminal rewards converge slower than dense shaping at the same compute budget; that's a well-known result and we're paying it.
- **12 templates × 5 variants procgen** — enough variety that a held-out variant is genuinely held-out, but not so much that scenario-specific overfitting eats the training budget.

These knobs collectively are the "compute" in "compute-bounded". Tighten any of them and a Series-A-class operator would call it out as theatre.

### 3.4 What "compute-bounded" actually means in numbers

A representative training run looks like:

| Phase | Steps | Compute | Wall-clock on A100 40GB | Output |
|---|---|---|---|---|
| Seed dataset build | 200 (Claude-driven) | ~$15 of API spend | 2h | `train/data/seed_combined.jsonl` |
| SFT cold start | 500 steps, batch 4 | ~3h | 3h | LoRA r=64 adapter |
| GRPO online | 800 steps, K=4 rollouts | ~6h | 6h | trained adapter |
| Eval sweep | 36 episodes (3 per template) | ~30min | 30min | `eval/results/comparison.csv` |
| **Total** | | **~$15 API + 12h GPU** | **12h** | trained 3B + comparison table |

That's the $30-of-HF-credits budget the design targets, comfortably.

---

## 4. The Advanced tier in detail

### 4.1 The horizon escalation, formalized

Advanced is bounded by horizon: episodes are 60–90 ticks instead of 12, multi-incident chains span 5+ minutes of simulated time, and the agent must track state that no single 8K context window can hold. Three properties make this a different research question:

1. **Multi-incident composition.** One template's resolution can become another template's setup state. Scenario 1 (`cascading_release_train.yaml`) is the canonical example: rolling back the gateway is correct, but it triggers a downstream worker drift that materializes 25 ticks later. The agent has to recognize the *chained* incident as caused by their own fix and reach for a second rollback rather than treating it as a fresh outage.
2. **Partial-observability noise.** Sometimes `query_logs` returns degraded data because the logging pipeline is the affected service. Scenario 2 (`observability_pipeline_outage.yaml`) is the canonical example: the agent must drop log sampling and toggle verbose-logging off *before* attempting root-cause diagnosis, because the diagnostic tool is itself broken.
3. **Cross-domain reasoning.** Scenario 3 (`supabase_rls_silent_leak.yaml`) is a reliability incident with a security root cause. The agent must classify it correctly (reach for `escalate_security` rather than the platform on-call), contain the data leak via feature flag *before* rolling back, and produce a postmortem with a leak-window calculation. No existing SRE benchmark scores cross-domain reasoning.

### 4.2 Why this isn't trained in this repo

A faithful Advanced simulator would need:
- a 15–20 service event-loop simulator (not a 4-service one)
- multi-tick fault propagation (one fix triggering a chained fault N ticks later, with proper causal latency)
- a synthetic on-call-peer model that responds to escalations
- ~28 action handlers, vs. 11 in Basic
- a learned-critic reward path for postmortem quality

That's roughly 2 weeks of focused engineering and 1–2 A100-days of training. Both are out of scope for the 36-hour hackathon window. We ship the design at the YAML level so that a downstream operator with the budget can lift it; we do not pretend it was trained.

---

## 5. The Max tier in detail

### 5.1 The realism escalation, formalized

Max is bounded by realism. The world stops being a simulator. A `reset()` provisions a fresh 22-service docker-compose stack (Vercel + Supabase + Stripe + Postgres + Redis + Kafka + 3 worker pools + observability stubs + chaos controller). The agent's `rollback_deploy` is a real `kubectl rollout undo` against that stack. `query_logs` reads from a real Loki/Promtail pipeline. `query_traces` reads from a real Tempo cluster. Faults are injected via a real Chaos-Mesh-style chaos library. Reward is computed from the actual recovery state of the actual stack.

Three properties are unique to Max:

1. **Real subprocess access.** The agent has a sandboxed shell. It can write code, commit it to a sandboxed git mirror, push, watch CI, observe a deploy, roll back. This is the "real hard work instead of exploiting shortcuts" spec from Theme #3.1 of the OpenEnv brief.
2. **Real action irreversibility.** A real Stripe refund is a real Stripe refund. The cluster is destroyed on next reset, but actions taken at the application layer are real.
3. **Outcome-scored rewards.** A second small model evaluates the agent's postmortem against the actual recovery trajectory. Reward is no longer a deterministic rubric — it's an outcome judgement, with shaping signals as auxiliary.

### 5.2 Why one family, not 30

Per the design rationale: one fully-specced family (e-commerce + Stripe + Supabase + Vercel) with `compose.max.yaml`, an 11-pattern chaos library, a workload generator config, a reference instance with an expected 110-action trajectory, and operator notes for cost / isolation / safety is more credible than a vague "30+ scenario families" claim. Judges who run real infrastructure would ask "show me one"; if you can't, the whole tier framing collapses. The `ecommerce_vibecoded_saas` family is the show-me-one.

### 5.3 What's deliberately not in this repo

- The published stub images (`ghcr.io/sre-gym/*`) — publishing them is a $1–2k registry-cost commitment that doesn't fit the hackathon-window budget.
- A running cluster — bringing the Max tier up costs $40–150/day depending on cluster size and chaos cadence.
- A trained model against Max — that's a multi-week, multi-A100 commitment.

What *is* in this repo is the spec at the level of detail a downstream operator can actually act on: docker-compose, chaos-library YAML, workload generator config, family-level scenario population spec, and operator-notes block covering cost, isolation, and reset safety.

---

## 6. The contract that ties Basic, Advanced, and Max together

All three tiers share the same five abstract objects:

| Object | Basic concrete | Advanced concrete | Max concrete |
|---|---|---|---|
| Topology | 4 services hard-coded | 15–20 services in YAML | 22 services in docker-compose |
| Action set | 11, Pydantic-validated | 28, validated against `allowed_actions:` list | 50+, including subprocess shell |
| Observation | pre-digested fields | noisy multi-source feed | raw Prometheus / Loki / Tempo |
| Reward | 7-dim deterministic rubric | rubric + chained-incident bonus + postmortem critic | outcome + learned-critic + IaC-remediation bonus |
| Episode | 8–13 ticks | 60–90 ticks | unbounded (real wall-clock minutes) |

Same shape, escalating depth. That's what makes the tier story coherent rather than three unrelated environments stacked in one repo.

---

## 7. OpenEnv framework integration

Basic uses:
- `openenv.core.env_server.Environment[A, O, S]` base class
- Typed Pydantic `Action / Observation / State`
- `/reset` `/step` `/state` HTTP endpoints via `create_fastapi_app`
- `max_concurrent_envs` for batched rollouts (the GRPO contract)
- Custom `/tasks` `/baseline` `/grader` `/status` `/health` extension routes for scenario-catalog-and-grader introspection

Advanced and Max would extend this with:
- `MCPEnvironment` base + `@self.tool()`-registered actions for production serving
- WebSocket `/ws` transport for low-latency multi-agent rollouts
- `ServerMode.SIMULATION` vs `ServerMode.PRODUCTION` switch for tier-3 real-API actions
- Custom Gradio `TabbedInterface` with a "topology inspector" tab

These are documented in the per-tier docs but not implemented here.

---

## 8. The judging-criteria mapping

The OpenEnv hackathon's published rubric weights are: Innovation 40, Storytelling 30, Reward Curves 20, Reward/Pipeline 10. sre-gym is built so each weight has a concrete corresponding artifact:

| Weight | Artifact in this repo |
|---|---|
| Innovation 40% | The dimensional-escalation tier story, defensible across compute / horizon / realism axes. |
| Storytelling 30% | This document + `README.md` first paragraph + 12-template skill table + reference traces in YAMLs. |
| Reward curves 20% | `eval/results/qwen25_7b_comparison_*` (populated by cell 10/11 of `01_triage_train_grpo_qwen25_7b.ipynb`) + `train/data/eval_sweep_baselines.jsonl` reference numbers. |
| Reward/Pipeline 10% | Composable rubric in `unified_incident_env/server/grader.py` + SFT → GRPO loop in `01_triage_train_grpo_qwen25_7b.ipynb`. |

Weight by weight, the artifact you'd hand a judge to defend that score is in this repo.