docs/walkthrough.md

# OpenSecOpsEnv — Full Project Walkthrough

## What Is This?

**OpenSecOpsEnv** is an OpenEnv-compliant reinforcement learning environment where an AI agent acts as an on-call security engineer. The agent must investigate and resolve production incidents across 4 escalating scenarios — from a simple memory leak to a disguised data exfiltration attack buried under 55% noise.

---

## 🤖 What Your Agent Does — Full Pipeline

```
┌────────────────────────────────────────────────────────────────┐
│                     Episode Flow                               │
│                                                                │
│  env.reset(task_id)                                            │
│       │                                                        │
│       ▼                                                        │
│  HiddenState (true root cause — NEVER shown to agent)         │
│       │ drives                                                 │
│       ▼                                                        │
│  SecOpsObservation (alerts + metrics + logs + topology)        │
│       │                                                        │
│       ▼                                                        │
│  LLM reads observation as text prompt                          │
│       │                                                        │
│       ▼                                                        │
│  LLM outputs JSON: {"action_type": "...", "parameters": {...}} │
│       │                                                        │
│       ▼                                                        │
│  env.step(action) → (new_obs, reward, done, info)             │
│       │                                                        │
│  GRPO uses reward to update model weights                      │
│       │                                                        │
│       └──── loop until done or max_steps ────────────────────┘
│                                                                │
│  grade(state) → score [0, 1]                                  │
└────────────────────────────────────────────────────────────────┘
```

### The 4 Tasks

| Task | Incident | Root Cause | Noise | Max Steps |
|------|----------|------------|-------|-----------|
| easy_memory_leak | Auth service heap overflow | `infra_failure:memory_leak` | 5% | 30 |
| medium_ddos_cascade | DDoS from 2 IP ranges → gateway → api → auth | `cyber_attack:ddos` | 25% | 40 |
| medium_hard_bad_deployment | Bad Redis config in api v2.4.1 → cache reconnect storm | `misconfiguration:bad_config` | 35% | 45 |
| hard_data_exfiltration | Compromised `reports_bot` exfiltrating 4GB via DB, false cache alert | `cyber_attack:data_exfiltration` | 55% | 50 |

### Reward Design (Dense)

| Action | Reward |
|--------|--------|
| Investigate affected service (first time) | +0.20 |
| Security scan hits cyber attack target | +0.30 |
| Correct mitigation step | +0.50 |
| Correct diagnosis | +1.00 |
| Investigate irrelevant service | −0.05 |
| Wrong mitigation (e.g., restart wrong service) | −0.10 |
| Harmful action (block legit IP, isolate healthy service) | −0.50 |
| Wrong diagnosis | −1.00 |

### Grader (Episode Score [0, 1])

```
score = 0.5 × diagnosis_correct
      + 0.3 × action_efficiency
      + 0.2 × investigation_quality
```

---

## 🧠 Training: GRPO with Unsloth

**Model:** Qwen2.5-7B-Instruct (4-bit QLoRA via Unsloth)  
**Algorithm:** Group Relative Policy Optimization (GRPO)  
**Training file:** `colab_training.ipynb` (Google Colab, T4 GPU)

### What GRPO does
For each observation prompt, it generates **4 completions** (different JSON actions), runs each in the environment, gets rewards, then updates the model to increase probability of higher-reward completions relative to the group average.

```
Prompt (observation) → [action_1, action_2, action_3, action_4]
                              ↓         ↓         ↓         ↓
                           r=0.20    r=-0.05   r=0.50   r=-0.50
                                              ↑ winner
GRPO update: increase P(action_3) relative to group mean
```

### What the agent learns
- Start with `inspect_metrics({})` to get overview (not random)
- Follow the topology: gateway → api → auth (upstream = root cause)
- Run `run_security_scan` before isolating services
- Ignore false alerts (the cache in hard task is a decoy)
- Submit exact diagnosis labels

---

## 🎨 Dashboard Changes (Round 2 Retheme)

The dashboard was redesigned from **cyberpunk neon** to **clean minimal professional**:

| Before | After |
|--------|-------|
| Dark background `#050811` | Light background `#f8fafc` |
| Neon cyan/green glows | Clean blue `#2563eb`, green `#16a34a`, red `#dc2626` |
| Animated grid background | Clean white cards with subtle shadows |
| `alert()` for comparison hint | Non-blocking toast notification |

### Bugs Fixed (10 total)
1. ✅ Topology nodes now highlight in red when they're affected services
2. ✅ Log stream clears properly on episode reset
3. ✅ `alert()` replaced with `showToast()` (non-blocking)
4. ✅ Progress bar colors are now dynamic (green/orange/red based on score)
5. ✅ Score total color resets to neutral on UI reset
6. ✅ `comparisonScores` preserved across `resetUI()` so Compare works after replay
7. ✅ Latency bar scale fixed (was /5000, now /2000 for better visibility)
8. ✅ Chart theme updated to match clean palette
9. ✅ Panel backgrounds consistent across all three columns
10. ✅ Topology node `topo-node.affected` CSS was always defined but JS never applied the class — fixed

---

## 📦 Files Overview

```
opensecops_env/
├── env.py              # Core environment (reset/step/state)
├── multi_agent_env.py  # ← NEW: Red vs Blue adversarial environment
├── curriculum.py       # ← NEW: Procedural task generation & level ups
├── models.py           # SecOpsAction, SecOpsObservation, SecOpsState
├── grader.py           # Deterministic episode grader [0,1]
├── tasks/
│   └── task_definitions.py  # 4 base task configs
└── server/
    └── app.py          # FastAPI server + endpoints + dashboard

training/
├── train_grpo.py       # GRPO training script (local)
└── plot_rewards.py     # Reward curve generation

colab_training.ipynb    # Full Colab notebook for HF credits
inference.py            # Baseline heuristic agent
```

---

## 🏆 Round 2 Theme Alignment

| Theme | Status | Evidence |
|-------|--------|---------|
| **Long-Horizon Planning** | ✅ Strong | 30–50 step episodes, sparse final reward, causal chain reasoning |
| **World Modeling: Professional** | ✅ Strong | Real SecOps tools (block_ip, rollback_deployment), realistic metrics |
| **Multi-Agent** | ✅ Strong | **Red vs Blue**: An attacker (Red) interleaves turns with the defender (Blue). Red actively obfuscates logs, spikes metrics, and injects false alerts while Blue investigates. |
| **Self-Improvement** | ✅ Strong | **Curriculum Manager**: Environment dynamically scales difficulty. When agent scores > 0.72 consistently, it unlocks Level 2 (compound failures), Level 3 (unknown procedurally generated topologies), and Level 4 (Adversarial Red). |

---

## 🚀 How to Execute & Test

### 1. Test the API Endpoints (Curriculum & Multi-Agent)
Start the server locally:
```bash
uvicorn opensecops_env.server.app:app --reload
```

In another terminal, test the Curriculum endpoint:
```bash
curl http://localhost:8000/curriculum/summary
# You will see the current level (1) and performance stats.
```

Test the Multi-Agent environment (Blue step, then Red step):
```bash
# Reset with "auto" to use curriculum
curl -X POST http://localhost:8000/multi/reset \
  -H "Content-Type: application/json" -d '{"task_id":"auto"}'

# Blue takes a step
curl -X POST http://localhost:8000/multi/blue/step \
  -H "Content-Type: application/json" \
  -d '{"action_type":"inspect_metrics", "parameters":{}}'

# Red reacts (heuristic attacker)
curl -X POST http://localhost:8000/multi/red/step \
  -H "Content-Type: application/json" -d '{}'
```

### 2. View the Dashboard
Go to `http://localhost:8000/dashboard` in your browser.
Click **Run Episode** and you will see the new clean, minimal theme, functional topology highlighting, and fixed progress bars.

### 3. Run the Colab Notebook (The Core Training Evidence)
1. Upload `colab_training.ipynb` to Google Colab.
2. Select a **T4 GPU** runtime (or A100 if using HF credits).
3. Add your `HF_TOKEN` in Colab's Secrets tab.
4. Run all cells. It will:
   - Download the model and environment
   - Run GRPO training using Unsloth
   - Automatically generate `training_results.png` (learning curves)
   - Push the fine-tuned adapter to Hugging Face.

**Primary pitch to judges:** This environment isn't just a static puzzle. It actively fights back (Multi-Agent) and gets harder as the model learns (Self-Improvement), making it a true testbed for next-generation reasoning models.