AtlasOps — Architecture
System Overview
graph TB
subgraph GCP["Google Cloud Platform — us-central1"]
subgraph GKE["GKE Standard Cluster (atlasops)"]
OB["Online Boutique<br/>11 microservices<br/>gRPC + protobuf"]
CM["Chaos Mesh<br/>PodChaos · NetworkChaos<br/>StressChaos · DNSChaos<br/>IOChaos · TimeChaos"]
PROM["Prometheus + Grafana<br/>kube-prometheus-stack"]
JAEGER["Jaeger + OTel Collector<br/>Distributed tracing"]
ARGO["Argo CD<br/>GitOps rollbacks"]
AM["Alertmanager<br/>Webhook → coordinator"]
end
CSQL["Cloud SQL<br/>Postgres 15"]
PS["Cloud PubSub<br/>cloudsre-checkout-events"]
CMON["Cloud Monitoring API<br/>GCP-native metrics"]
CLOG["Cloud Logging<br/>Structured logs"]
end
subgraph MI300X["AMD MI300X — 192 GB HBM3"]
COORD["Coordinator<br/>FastAPI :9099"]
subgraph AGENTS["4 Specialized Agents (Qwen2.5-7B + QLoRA)"]
T["🔴 Triage"]
D["🔍 Diagnosis"]
R["🔧 Remediation"]
C["📣 Comms"]
end
JUDGE["72B Judge<br/>Qwen2.5-72B<br/>Adversarial designer<br/>+ Evaluator"]
DASH["Gradio Dashboard<br/>:7860"]
end
CM -->|injects faults| OB
OB -->|metrics| PROM
OB -->|traces| JAEGER
PROM -->|alert fires| AM
AM -->|webhook| COORD
COORD --> T --> D --> R --> C
T & D & R & C <-->|"20 real SRE tools<br/>(kubectl · promql · jaeger<br/>argocd · gcloud · alertmanager)"| GKE
T & D & R & C <-->|Cloud APIs| CSQL & PS & CMON & CLOG
JUDGE -->|generates scenarios| CM
JUDGE -->|scores actions| COORD
DASH -->|live thoughts SSE| COORD
Agent Chain
sequenceDiagram
participant AM as Alertmanager
participant CO as Coordinator
participant TR as Triage Agent
participant DG as Diagnosis Agent
participant RM as Remediation Agent
participant CM as Comms Agent
participant GKE as Real GKE Cluster
AM->>CO: POST /webhook (alert fired)
CO->>TR: {incident_id, alert}
TR->>GKE: kubectl_top_pods()
TR->>GKE: promql_query("rate(5xx[1m])")
TR-->>CO: {severity: P1, blast_radius: [...]}
CO->>DG: {triage_output}
DG->>GKE: jaeger_search(service, min_duration=500ms)
DG->>GKE: promql_query_range(query, last_15m)
DG->>GKE: kubectl_logs(bottleneck_pod)
DG->>GKE: argocd_app_history(app)
DG-->>CO: {root_cause: {...}, recommended_actions: [...]}
CO->>RM: {triage, diagnosis}
RM->>GKE: argocd_rollback(app, revision)
RM->>GKE: promql_query("rate(5xx[1m])")
Note over RM: verify error_rate < 1%
RM-->>CO: {outcome: resolved, ttr: 187s}
CO->>CM: {full incident chain}
CM->>CM: slack_post_update(...)
CM->>CM: postmortem_draft(incident)
CM-->>CO: {postmortem_path: docs/postmortems/...}
AMD MI300X Co-hosting
AMD MI300X (192 GB HBM3)
┌─────────────────────────────────────────────────────────┐
│ │
│ Qwen2.5-7B base (4-bit NF4) ~4 GB │
│ ├─ triage_adapter (LoRA r=16) ~40 MB │
│ ├─ diagnosis_adapter (LoRA r=16) ~40 MB │
│ ├─ remediation_adapter (LoRA r=16) ~40 MB │
│ └─ comms_adapter (LoRA r=16) ~40 MB │
│ │
│ Qwen2.5-72B (4-bit NF4) ~37 GB │
│ └─ Adversarial designer + judge │
│ │
│ Total used: ~41 GB Available: ~151 GB │
│ │
│ ❌ A100 (80 GB): can fit judge OR agents, not both │
│ ❌ T4 (16 GB): can't fit even the 7B base │
│ ✅ MI300X: all 5 models + room to spare │
└─────────────────────────────────────────────────────────┘
Training Pipeline
flowchart LR
A["Real GKE Cluster<br/>11 live microservices"] -->|Chaos Mesh injects faults| B
B["Alertmanager fires<br/>real webhook"] --> C
C["4 agents run<br/>against live cluster"] --> D
D["72B judge scores<br/>tool calls + outcomes"] --> E
E["5k trajectory corpus<br/>data/sft_corpus.jsonl<br/>reward_contract included"]
E --> F["Stage 1: QLoRA SFT<br/>Qwen2.5-7B + LoRA r=16<br/>4-bit NF4 on MI300X"]
F --> G["Stage 2: GRPO<br/>Optuna HP search (6 trials)<br/>num_gen=8, cosine LR<br/>tier-aware reward contract"]
G --> H["4 role adapters<br/>~40 MB each<br/>checkpoints/grpo_v3/"]
H -->|"hot-swap adapters<br/>per agent call"| I["Production serving<br/>vLLM on MI300X"]
20 Real SRE Tools
kubectl (7) promql (2) jaeger (2)
───────────── ────────── ──────────
kubectl_get promql_query jaeger_search
kubectl_describe promql_query_range jaeger_get_trace
kubectl_logs
kubectl_top_pods argocd (3) gcloud (2)
kubectl_top_nodes ───────── ──────────
kubectl_rollout argocd_list_apps gcloud_logs_read
kubectl_scale argocd_app_history cloud_monitoring_query
kubectl_exec argocd_rollback
alertmanager (2) comms (2)
──────────────── ──────────
alertmanager_silence slack_post_update
alertmanager_list postmortem_draft
AtlasOps exposes 20 production-relevant SRE tools across Kubernetes, tracing, metrics, GitOps, and comms workflows.
Reward Contract (Anti-Gaming)
reward = 0.35 × resolve # did the incident get fixed?
+ 0.20 × evidence # was the root cause proven with data?
+ 0.20 × safety # was the action minimum-blast-radius?
+ 0.15 × speed # logistic decay — fast is good, race-to-zero is penalised
+ 0.10 × comms # was a postmortem generated?
- 0.10 if turns > 40 # command spam
- 0.25 if claimed resolved but wasn't # false resolution
- 0.20 if efficiency < 0.3 # unsafe shortcut
- 0.20 if reasoning AND correctness both low # hallucinated evidence
- 0.10 if silenced an alert without resolving # over-silencing
tier adjustments:
cascade: r_evidence weight ↑ (tracing the chain matters more)
multi_fault: r_safety weight ↑ (conservative action matters more)
adversarial: all penalties × 1.25 (harder tier, stricter scoring)