docs/walkthrough.md

OpenSecOpsEnv — Full Project Walkthrough

What Is This?

OpenSecOpsEnv is an OpenEnv-compliant reinforcement learning environment where an AI agent acts as an on-call security engineer. The agent must investigate and resolve production incidents across 4 escalating scenarios — from a simple memory leak to a disguised data exfiltration attack buried under 55% noise.

---

🤖 What Your Agent Does — Full Pipeline

┌────────────────────────────────────────────────────────────────┐
│                     Episode Flow                               │
│                                                                │
│  env.reset(task_id)                                            │
│       │                                                        │
│       ▼                                                        │
│  HiddenState (true root cause — NEVER shown to agent)         │
│       │ drives                                                 │
│       ▼                                                        │
│  SecOpsObservation (alerts + metrics + logs + topology)        │
│       │                                                        │
│       ▼                                                        │
│  LLM reads observation as text prompt                          │
│       │                                                        │
│       ▼                                                        │
│  LLM outputs JSON: {"action_type": "...", "parameters": {...}} │
│       │                                                        │
│       ▼                                                        │
│  env.step(action) → (new_obs, reward, done, info)             │
│       │                                                        │
│  GRPO uses reward to update model weights                      │
│       │                                                        │
│       └──── loop until done or max_steps ────────────────────┘
│                                                                │
│  grade(state) → score [0, 1]                                  │
└────────────────────────────────────────────────────────────────┘

The 4 Tasks

Task	Incident	Root Cause	Noise	Max Steps
easy_memory_leak	Auth service heap overflow	infra_failure:memory_leak	5%	30
medium_ddos_cascade	DDoS from 2 IP ranges → gateway → api → auth	cyber_attack:ddos	25%	40
medium_hard_bad_deployment	Bad Redis config in api v2.4.1 → cache reconnect storm	misconfiguration:bad_config	35%	45
hard_data_exfiltration	Compromised reports_bot exfiltrating 4GB via DB, false cache alert	cyber_attack:data_exfiltration	55%	50

Reward Design (Dense)

Action	Reward
Investigate affected service (first time)	+0.20
Security scan hits cyber attack target	+0.30
Correct mitigation step	+0.50
Correct diagnosis	+1.00
Investigate irrelevant service	−0.05
Wrong mitigation (e.g., restart wrong service)	−0.10
Harmful action (block legit IP, isolate healthy service)	−0.50
Wrong diagnosis	−1.00

Grader (Episode Score [0, 1])

score = 0.5 × diagnosis_correct
      + 0.3 × action_efficiency
      + 0.2 × investigation_quality

---

🧠 Training: GRPO with Unsloth

Model: Qwen2.5-7B-Instruct (4-bit QLoRA via Unsloth)
Algorithm: Group Relative Policy Optimization (GRPO)
Training file: colab_training.ipynb (Google Colab, T4 GPU)

What GRPO does

For each observation prompt, it generates 4 completions (different JSON actions), runs each in the environment, gets rewards, then updates the model to increase probability of higher-reward completions relative to the group average.

Prompt (observation) → [action_1, action_2, action_3, action_4]
                              ↓         ↓         ↓         ↓
                           r=0.20    r=-0.05   r=0.50   r=-0.50
                                              ↑ winner
GRPO update: increase P(action_3) relative to group mean

What the agent learns

• Start with inspect_metrics({}) to get overview (not random)
• Follow the topology: gateway → api → auth (upstream = root cause)
• Run run_security_scan before isolating services
• Ignore false alerts (the cache in hard task is a decoy)
• Submit exact diagnosis labels

---

🎨 Dashboard Changes (Round 2 Retheme)

The dashboard was redesigned from cyberpunk neon to clean minimal professional:

Before	After
Dark background #050811	Light background #f8fafc
Neon cyan/green glows	Clean blue #2563eb, green #16a34a, red #dc2626
Animated grid background	Clean white cards with subtle shadows
alert() for comparison hint	Non-blocking toast notification

Bugs Fixed (10 total)

1. ✅ Topology nodes now highlight in red when they're affected services
2. ✅ Log stream clears properly on episode reset
3. ✅ alert() replaced with showToast() (non-blocking)
4. ✅ Progress bar colors are now dynamic (green/orange/red based on score)
5. ✅ Score total color resets to neutral on UI reset
6. ✅ comparisonScores preserved across resetUI() so Compare works after replay
7. ✅ Latency bar scale fixed (was /5000, now /2000 for better visibility)
8. ✅ Chart theme updated to match clean palette
9. ✅ Panel backgrounds consistent across all three columns
10. ✅ Topology node topo-node.affected CSS was always defined but JS never applied the class — fixed

---

📦 Files Overview

opensecops_env/
├── env.py              # Core environment (reset/step/state)
├── multi_agent_env.py  # ← NEW: Red vs Blue adversarial environment
├── curriculum.py       # ← NEW: Procedural task generation & level ups
├── models.py           # SecOpsAction, SecOpsObservation, SecOpsState
├── grader.py           # Deterministic episode grader [0,1]
├── tasks/
│   └── task_definitions.py  # 4 base task configs
└── server/
    └── app.py          # FastAPI server + endpoints + dashboard

training/
├── train_grpo.py       # GRPO training script (local)
└── plot_rewards.py     # Reward curve generation

colab_training.ipynb    # Full Colab notebook for HF credits
inference.py            # Baseline heuristic agent

---

🏆 Round 2 Theme Alignment

Theme	Status	Evidence
Long-Horizon Planning	✅ Strong	30–50 step episodes, sparse final reward, causal chain reasoning
World Modeling: Professional	✅ Strong	Real SecOps tools (block_ip, rollback_deployment), realistic metrics
Multi-Agent	✅ Strong	Red vs Blue: An attacker (Red) interleaves turns with the defender (Blue). Red actively obfuscates logs, spikes metrics, and injects false alerts while Blue investigates.
Self-Improvement	✅ Strong	Curriculum Manager: Environment dynamically scales difficulty. When agent scores > 0.72 consistently, it unlocks Level 2 (compound failures), Level 3 (unknown procedurally generated topologies), and Level 4 (Adversarial Red).

---

🚀 How to Execute & Test

1. Test the API Endpoints (Curriculum & Multi-Agent)

Start the server locally:

uvicorn opensecops_env.server.app:app --reload

In another terminal, test the Curriculum endpoint:

curl http://localhost:8000/curriculum/summary
# You will see the current level (1) and performance stats.

Test the Multi-Agent environment (Blue step, then Red step):

# Reset with "auto" to use curriculum
curl -X POST http://localhost:8000/multi/reset \
  -H "Content-Type: application/json" -d '{"task_id":"auto"}'

# Blue takes a step
curl -X POST http://localhost:8000/multi/blue/step \
  -H "Content-Type: application/json" \
  -d '{"action_type":"inspect_metrics", "parameters":{}}'

# Red reacts (heuristic attacker)
curl -X POST http://localhost:8000/multi/red/step \
  -H "Content-Type: application/json" -d '{}'

2. View the Dashboard

Go to http://localhost:8000/dashboard in your browser. Click Run Episode and you will see the new clean, minimal theme, functional topology highlighting, and fixed progress bars.

3. Run the Colab Notebook (The Core Training Evidence)

1. Upload colab_training.ipynb to Google Colab.
2. Select a T4 GPU runtime (or A100 if using HF credits).
3. Add your HF_TOKEN in Colab's Secrets tab.
4. Run all cells. It will:
   • Download the model and environment
   • Run GRPO training using Unsloth
   • Automatically generate training_results.png (learning curves)
   • Push the fine-tuned adapter to Hugging Face.

Primary pitch to judges: This environment isn't just a static puzzle. It actively fights back (Multi-Agent) and gets harder as the model learns (Self-Improvement), making it a true testbed for next-generation reasoning models.