deploy/aws/ARCHITECTURE.md

# AntiAtropos Architecture Guide

A complete explanation of how AntiAtropos works across Hugging Face Spaces and AWS, written for someone who is technically strong but new to Kubernetes.

---

## The Big Picture

AntiAtropos trains AI agents to be Site Reliability Engineers (SREs). An SRE agent watches a simulated microservice cluster and decides when to scale services, reroute traffic, or shed load to keep things running smoothly.

The system is split across two platforms:

```
Hugging Face Spaces                      AWS
=====================                    ======================
The "brain"                              The "muscle"

AntiAtropos FastAPI server               EKS (Kubernetes cluster)
  - Runs the simulator                     - Runs the actual microservice pods
  - Runs the SRE agent logic               - The agent scales these pods
  - Queries Prometheus for metrics         - Prometheus Agent scrapes metrics
  - Sends scale commands to K8s            - Metrics flow to AMP
                                           - Grafana (AMG) visualizes it all
```

Why split? HF Spaces is free/cheap for running the Python server. AWS EKS is where the real infrastructure lives that the agent practices on.

---

## Kubernetes Concepts You Need

### Pod

The smallest unit in Kubernetes. A pod is one or more containers that run together. In our case, each pod runs a single nginx container that simulates a microservice (like "payments" or "checkout").

Think of it as: one running instance of a service.

### Deployment

A Deployment is a recipe that tells Kubernetes "keep N copies of this pod running at all times." If a pod dies, the Deployment automatically replaces it.

The key field is `spec.replicas` — this is the number the SRE agent changes when it scales a service up or down.

```
Deployment: payments
  replicas: 3         <-- the agent changes this number
    |
    +-- Pod: payments-abc123   (running)
    +-- Pod: payments-def456   (running)
    +-- Pod: payments-ghi789   (running)
```

**The agent scales replicas, not pods.** When it sets `replicas: 5`, Kubernetes creates 5 pods. When it sets `replicas: 2`, Kubernetes kills 3 pods.

### Service

A Service gives pods a stable network name. Instead of connecting to `payments-abc123` directly (which changes when the pod is recreated), you connect to `payments` (the Service), which routes to whichever pods are healthy.

### Namespace

A namespace is a folder for organizing resources. We use:
- `prod-sre` — where the 5 microservice Deployments live
- `monitoring` — where the Prometheus Agent pod lives
- `kube-system` — where AWS/EKS system pods live

### Node

A node is one EC2 virtual machine in the EKS cluster. Our cluster has 2-4 nodes. Each node runs multiple pods. When all nodes are full and the agent wants to scale up, Kubernetes adds more nodes (up to `maxSize: 4` in our config).

```
EKS Cluster
  Node 1 (t3.medium - 4 vCPU, 8GB RAM)
    Pod: payments-abc123
    Pod: checkout-def456
    Pod: catalog-ghi789
    Pod: prometheus-agent-xyz
  Node 2 (t3.medium - 4 vCPU, 8GB RAM)
    Pod: payments-jkl012    <-- agent scaled payments from 1 to 2
    Pod: cart-mno345
    Pod: auth-pqr678
```

### ResourceQuota

A hard limit on how many resources a namespace can use. We set one on `prod-sre` that caps total pods at 30. This is a safety net — even if the Python code cap fails, Kubernetes itself will refuse to create more than 30 pods.

---

## How the SRE Agent Works

### The Loop

Every "tick" (one step of the simulation), the agent goes through this cycle:

```
1. OBSERVE  -- Read telemetry (CPU, latency, queue depth) from Prometheus
2. DECIDE   -- Choose an action (SCALE_UP, SCALE_DOWN, REROUTE_TRAFFIC, SHED_LOAD, NO_OP)
3. ACT      -- Send the action to KubernetesExecutor
4. REWARD   -- Compute Lyapunov stability reward (was the cluster more or less stable?)
5. REPEAT
```

### How Each Action Works

| Action | What the Agent Decides | What Happens on EKS |
|---|---|---|
| `SCALE_UP` | "node-0 needs more capacity" | `KubernetesExecutor` patches `payments` Deployment: `replicas: 2 -> 5` |
| `SCALE_DOWN` | "node-3 is over-provisioned" | `KubernetesExecutor` patches `cart` Deployment: `replicas: 4 -> 1` |
| `REROUTE_TRAFFIC` | "Move traffic away from node-2" | `KubernetesExecutor` scales DOWN target deployment and redistributes replicas to healthy peer deployments |
| `SHED_LOAD` | "Drop 50% of traffic to node-3" | `KubernetesExecutor` scales DOWN target deployment by `parameter * current_replicas` |
| `NO_OP` | "Do nothing this tick" | Nothing changes on EKS |

### The SCALE_UP Flow in Detail

Here is exactly what happens when the agent decides to scale up `node-0` (the payments service):

```
HF Spaces                                    AWS EKS
----------                                   --------

Agent: "SCALE_UP, node-0, parameter=0.5"
  |
  v
AntiAtroposEnvironment.step()
  |
  v
KubernetesExecutor.execute_with_metadata()
  |
  v
_load_node_workload_map()
  reads: node-0 -> {"deployment": "payments", "namespace": "prod-sre"}
  |
  v
_scale_deployment("SCALE_UP", "node-0", 0.5)
  |
  +-- 1. Read current replicas: apps_v1.read_namespaced_deployment_scale("payments", "prod-sre")
  |      Current replicas = 2
  |
  +-- 2. Calculate delta: max(1, int(0.5 * 3)) = 1
  |      Desired = min(6, 2 + 1) = 3        <-- max_replicas cap from env var
  |
  +-- 3. Patch: apps_v1.patch_namespaced_deployment_scale("payments", "prod-sre",
  |         body={"spec": {"replicas": 3}})
  |
  v                                                     +---------------------------+
Returns: "Ack: SCALE_UP for node-0 -                    | K8s creates 1 new pod:    |
  deployment payments in namespace                       |   payments-newpod-xyz     |
  prod-sre scaled 2->3"                                  +---------------------------+
```

### The Telemetry Flow in Detail

How the agent reads metrics from the real cluster:

```
EKS Cluster                              AMP                          HF Spaces
-----------                              ---                          ----------

Workload pods                            AMP Workspace                AntiAtropos
(payments, checkout...)                  stores all metrics           PrometheusClient
  |                                           ^                        |
  | /metrics (scraped every 15s)              |                        |
  v                                           |                        |
Prometheus Agent                             |                        |
  |                                           |                        |
  | remote-write (SigV4 auth)                 |                        |
  +------------------------------------------->                        |
                                              |                        |
                                              |  HTTPS query           |
                                              +------------------------>
                                              (PROMETHEUS_URL env var)
                                                                       |
                                                                       v
                                                                 _fetch_real_metrics()
                                                                 runs PromQL like:
                                                                   sum(rate(http_requests_total[1m])) by (pod)
                                                                 returns: TelemetryRecord for each node
```

---

## The Three Layers of Scaling Caps

This is the most important thing to understand for cost control. There are **three** independent limits:

### Layer 1: Python Code Cap (Soft)

**Where:** `ANTIATROPOS_MAX_REPLICAS` env var on HF Spaces, read by `kubernetes_executor.py` line 18.

**How it works:** The `_scale_deployment()` method calculates `desired = min(self.max_replicas, current + delta)`. If the agent tries to scale above 6, it gets:

```
Ack: SCALE_UP for node-0 - replicas unchanged at 6 (bounds 1-6)
```

**Can it be bypassed?** Yes. A bug in the code, or someone running `kubectl scale deployment payments --replicas=50` directly.

**Set to:** `6` on HF Spaces.

### Layer 2: Kubernetes ResourceQuota (Hard)

**Where:** `k8s-workloads.yaml` — ResourceQuota on the `prod-sre` namespace.

**How it works:** Kubernetes itself refuses to schedule pods that would exceed the quota. If the namespace already has 30 pods and something tries to create a 31st:

```
Error from server (Forbidden): pods "payments-new" is forbidden:
exceeded quota: prod-sre-quota, requested: pods=1, used: pods=30, limited: pods=30
```

**Can it be bypassed?** Only by someone with cluster-admin access who deletes or edits the ResourceQuota.

**Set to:** 30 pods total, 8 CPU, 8GB RAM.

### Layer 3: EKS Node Group Max Size (Hard)

**Where:** `eksctl-cluster.yaml` — `managedNodeGroups[0].maxSize: 4`.

**How it works:** The Cluster Autoscaler will never add more than 4 nodes. Even if there are 100 pending pods, it stops at 4 nodes. Pending pods just wait.

**Can it be bypassed?** Only by someone editing the node group in the AWS console.

**Set to:** 4 nodes (4 x t3.medium = 8 vCPU, 16GB RAM max).

### How the Three Layers Work Together

```
Agent wants to scale all 5 deployments to 20 replicas each:

Layer 1 (Python cap):      6 replicas max per deployment  -> agent gets "unchanged at 6"
                           5 x 6 = 30 pods maximum

Layer 2 (ResourceQuota):   30 pods max in namespace       -> 31st pod is Forbidden

Layer 3 (Node group):      4 nodes max                     -> if 30 pods don't fit on 4 nodes,
                                                            some stay Pending (no cost)

Worst case with all caps:  30 pods on 4 nodes = ~$160/month
Without any caps:          100 pods on 25 nodes = ~$1,800/month
```

---

## The Mapping: Simulator Nodes to Real Deployments

The simulator has 5 abstract nodes (node-0 through node-4). The `ANTIATROPOS_WORKLOAD_MAP` env var tells the system which K8s Deployment each simulator node maps to:

```
Simulator Node    K8s Deployment    Namespace    Notes
-------------     ---------------   ---------    -----
node-0            payments          prod-sre     VIP (4x importance weight)
node-1            checkout          prod-sre     Critical (no SHED_LOAD)
node-2            catalog           prod-sre     Critical (no SHED_LOAD)
node-3            cart              prod-sre     Non-critical (sheddable)
node-4            auth              prod-sre     Non-critical (sheddable)
```

When the simulator says "SCALE_UP node-0 by 0.5", the system:
1. Looks up node-0 in the workload map -> `payments` in `prod-sre`
2. Calls `patch_namespaced_deployment_scale("payments", "prod-sre", ...)`
3. Kubernetes creates/destroys pods to match the new replica count

---

## What Runs Where (Complete List)

### On Hugging Face Spaces

| Component | What It Does | Port |
|---|---|---|
| FastAPI server (`server/app.py`) | HTTP API for the agent | 7860 (via NGINX) |
| Simulator (`simulator.py`) | 5-node microservice cluster simulation | Internal |
| PrometheusClient (`telemetry/prometheus_client.py`) | Queries AMP for real metrics | Outbound HTTPS |
| KubernetesExecutor (`control/kubernetes_executor.py`) | Sends scale commands to EKS | Outbound HTTPS |
| Prometheus metrics exporter | Serves `/metrics` for HF's monitoring | 8000 |
| Grafana + local Prometheus | Local dashboards (from the Dockerfile) | 3000, 9090 |

### On AWS EKS

| Component | Namespace | What It Does |
|---|---|---|
| payments Deployment | prod-sre | 2 nginx pods (scales with agent) |
| checkout Deployment | prod-sre | 1 nginx pod (scales with agent) |
| catalog Deployment | prod-sre | 1 nginx pod (scales with agent) |
| cart Deployment | prod-sre | 1 nginx pod (scales with agent) |
| auth Deployment | prod-sre | 1 nginx pod (scales with agent) |
| Prometheus Agent | monitoring | Scrapes workload pods, remote-writes to AMP |
| Cluster Autoscaler | kube-system | Adds/removes EC2 nodes based on demand |

### On AWS Managed Services

| Service | What It Does |
|---|---|
| AMP (Amazon Managed Prometheus) | Stores all metrics. Queried by HF Spaces. |
| AMG (Amazon Managed Grafana) | Visualizes metrics in dashboards. Accessed via browser. |

---

## The Simulator vs Real Cluster

AntiAtropos has three modes controlled by `ANTIATROPOS_ENV_MODE`:

### Simulated Mode (`simulated`)

Everything is fake. The simulator generates synthetic metrics (random CPU, latency, etc.). No K8s, no Prometheus. The agent practices in a safe sandbox.

This is the default on HF Spaces without AWS configured.

### Hybrid Mode (`hybrid`)

The simulator runs, but it pulls real metrics from AMP to calibrate itself. If AMP says `payments` pods have 80% CPU, the simulator adjusts its internal model to match. The agent can read real data but actions only affect the simulator, not real pods.

### Live Mode (`live`)

The real deal. The agent reads real metrics from AMP and sends real scale commands to EKS. When it says `SCALE_UP`, actual pods get created on actual EC2 instances that cost actual money.

**Set `ANTIATROPOS_ENV_MODE=live` on HF Spaces to enable this.**

---

## Cost Flow

Every pod on EKS costs money. Here is how costs flow based on the agent's actions:

```
Agent action: SCALE_UP node-0
  -> payments Deployment: replicas 2 -> 5
  -> 3 new pods created
  -> If existing nodes are full, Cluster Autoscaler adds a node
  -> New node = another t3.medium EC2 instance = ~$0.04/hr
  -> 3 pods running = 3 x (0.1 CPU + 64MB RAM) from the quota

Agent action: SCALE_DOWN node-3
  -> cart Deployment: replicas 4 -> 1
  -> 3 pods terminated
  -> If nodes are now underutilized, Cluster Autoscaler removes a node (after 10 min)
  -> One fewer EC2 instance = saves ~$0.04/hr
```

The Lyapunov reward function penalizes the agent for both instability AND cost, so a well-trained agent should learn to scale efficiently:

```
R_t = -(alpha * delta_V  +  beta * cost  +  gamma * SLA_violation)
                                  ^^^^
                           beta=0.01 penalizes over-provisioning
```

---

## Quick Reference: Key Files

| File | Purpose |
|---|---|
| `kubernetes_executor.py` | Translates agent actions to K8s API calls |
| `prometheus_client.py` | Queries AMP for real metrics |
| `simulator.py` | 5-node fluid-queue simulation |
| `stability.py` | Lyapunov reward computation |
| `deploy/aws/k8s-workloads.yaml` | The 5 Deployments + ResourceQuota on EKS |
| `deploy/aws/eksctl-cluster.yaml` | EKS cluster definition (nodes, caps) |
| `deploy/aws/prometheus-agent-values.yaml` | Helm config for Prometheus Agent |
| `deploy/aws/generate-kubeconfig.sh` | Creates kubeconfig for HF Spaces |