---
title: CyberSOC Upgraded RLVR
emoji: 🛡️
colorFrom: red
colorTo: purple
sdk: docker
pinned: false
---

# CyberSOC: Complete Project Review

> **New — Hotseat Multiplayer Mode**: A human can now play the Red Team live from the dashboard. After every Blue action the UI pauses, enables the **🔴 Red Team Toolkit**, shows a `BLUE / RED TEAM TURN` indicator in the header, and resumes Blue auto-play only after the Red player submits their move. Backend now runs in FSP (Fictitious Self-Play) mode by default.

> **New — 4 Architectural Upgrades**: Lazy Adversary (stall punishment + 15 % autonomous pivot), Rigid Bureaucracy emergency-isolation gate (UNJUSTIFIED\_EMERGENCY penalty), Siloed Intelligence external intel feed injection, and Depressed Analyst doomsday-clock direct modifier + negligence penalty.

## 1. Project Overview — RLVR Positioning

CyberSOC (CyberSOCEnv) is an **RLVR-stage reinforcement learning environment** that sits at the final rung of the model-maturation arc: **Random Init → Pretraining → SFT/IFT → Preference FT → RLVR**. It does not pretrain, supervise, or preference-align — it consumes a base model that has already been through those stages and turns its agentic actions into a dense, verifiable, 10-dimensional reward signal that GRPO can train on.

It assumes a base model that has already been SFT-aligned. The environment itself does not perform SFT; it is an **RL-only artifact**. This satisfies **Daniel's "Law of RL"**: *The base model must get non-zero reward on Easy before it can meaningfully learn Hard.*

**Built for**: The OpenEnv Hackathon (Meta Platforms)
**Framework**: OpenEnv (Meta's RL environment framework)
**License**: BSD-style (Meta Platforms, Inc.)

### Guide Alignment Summary

| Guide Section | Requirement | Our Implementation |
|---|---|---|
| §1 | Step-by-step, programmatic verification, hard-but-possible | 10 typed actions, 10-dim deterministic grader, Easy→Hard curriculum |
| §4 | Design env before trainer | Env designed first; reset/step/state as first-class artifacts |
| §6 | Keep task simple at first | 1000+ scenarios across 3 difficulty tiers enable curriculum learning |
| §7 | Multiple independent reward functions | 10 dimensions consumed as `reward_funcs=[...]` by GRPOTrainer |
| §8 | Protect against reward hacking | 8 distinct defenses mapped to guide's attack vectors |
| §10 | Right training stack | Unsloth (QLoRA) + TRL (GRPO) + OpenEnv (transport) |
| §11 | Prefer GRPO/RLVR | RLVR throughout; every reward is deterministic code (zero LLM-as-judge) |
| §12 | Keep inference fast | Graph-delta injection + sparse nodes = rollout-latency optimizations |
| §14 | Scale only after stable | All 9 components passed integration before any GRPO rollout |

---

## 2. Core Idea & Innovation

### The Problem
Traditional cybersecurity training environments use static puzzles with fixed answers. Real SOC work requires dynamic reasoning under time pressure with incomplete information.

### The Solution
CyberSOC creates a **fully dynamic, deterministic SOC simulation** with:

1. **Procedural Scenario Generation** — 1,003 unique attack scenarios (3 curated + 1,000 generated) from seed-based deterministic generation. Same seed = same scenario, enabling reproducible RL training.
2. **13 Threat Categories** — Ransomware, Phishing, Credential Theft, Lateral Movement, C2 Communication, Privilege Escalation, Data Exfiltration, Cryptomining, Supply Chain, Insider Threat, Webshell, Botnet, Malware.
3. **Adaptive Red Team** — An adversary that reacts to agent actions: if you isolate a host, the attacker may pivot laterally. If you kill a process without blocking IOCs, it may reinfect with a `_v2` variant.
4. **10-Dimensional Grading** — Not a binary pass/fail. Agents are scored across 10 weighted dimensions for nuanced RL credit assignment. **Zero LLM-as-a-judge.**
5. **Business Continuity Constraints** — Rash actions (isolating clean subnets, killing legitimate processes) cause business downtime penalties.
6. **TRL GRPO Integration** — 10 reward functions that plug directly into Hugging Face's TRL `GRPOTrainer` for RL fine-tuning.

---

## 3. Architecture

```
MetaRound2/
├── models.py              # Pydantic data models (Observation, Action, State)
├── client.py              # WebSocket client for agent interaction
├── __init__.py            # Package exports
├── inference.py           # LLM baseline inference script
├── dashboard_server.py    # Dashboard + API server launcher
├── pyproject.toml         # Python package config
├── Dockerfile             # HuggingFace Spaces deployment
├── openenv.yaml           # 1003 task manifest
├── validate_submission.sh # Hackathon submission validator
│
├── server/                # Backend environment engine
│   ├── app.py             # FastAPI application entry point
│   ├── play_environment.py # Core environment (1284 lines)
│   ├── tasks.py           # Hand-crafted task definitions (easy/medium/hard)
│   ├── task_generator.py  # Procedural generation engine (1000+ tasks)
│   ├── graders.py         # 10-dimensional grading system
│   ├── threat_graph.py    # Typed knowledge graph
│   ├── soar_playbooks.py  # 5 SOAR playbook definitions
│   ├── action_validation.py # 3-gate action validation middleware
│   ├── tool_router.py     # Phase state machine + triage solver
│   ├── episode_sandbox.py # Wall-clock + step-limit guard
│   ├── visualize_graph.py # PNG graph renderer (matplotlib/networkx)
│   └── Dockerfile         # Multi-stage Docker build
│
├── training/              # RL training integration
│   └── reward_funcs.py    # 10 TRL GRPO reward functions
│
├── dashboard/             # Real-time web dashboard
│   ├── index.html         # Main HTML (6 panels)
│   ├── css/styles.css     # Dark theme CSS (25KB)
│   └── js/
│       ├── app.js         # Main dashboard logic (45KB)
│       ├── graphs.js      # D3.js threat graph + Chart.js (31KB)
│       ├── api.js         # REST API client
│       └── animations.js  # Micro-animations & effects
│
└── tests/                 # 10 test files + integration suite
    ├── test_integration.py
    └── test_task1.py ... test_task9.py
```

---

## 4. Backend (Server)

### 4.1 Core Environment — `play_environment.py`

The heart of the project. `CyberSOCEnvironment` extends OpenEnv's `Environment` interface.

**Key features:**
- **`reset(task_id)`** — Builds the network, injects attack chains, initializes alert queue, seeds the ThreatGraph
- **`step(action)`** — Processes one agent action, computes rewards, updates state, triggers adaptive adversary
- **Concurrent sessions** — Each WebSocket connection gets its own environment instance
- **ActionMiddleware** — Pre-flight validation (phase violations, graph-groundedness) before consuming a step

**10 Agent Actions:**

| # | Action | Purpose | Reward Range |
|---|--------|---------|-------------|
| 1 | `query_host` | Map architecture, get endpoint info | -0.05 to +0.05 |
| 2 | `run_forensics` | Deep system artifact extraction | -0.02 to +0.10 |
| 3 | `kill_process` | Terminate malicious execution | -0.08 to +0.25 |
| 4 | `block_ioc` | Blacklist IOCs network-wide | -0.03 to +0.15 |
| 5 | `isolate_segment` | Quarantine subnet or host | -0.10 to +0.15 |
| 6 | `correlate_alerts` | Find shared entities across alerts | ±0.05 |
| 7 | `enrich_ioc` | Threat-intel enrichment (actor, TTPs) | ±0.05 |
| 8 | `scan_host_vulnerabilities` | Discover CVEs on a host | ±0.05 |
| 9 | `trigger_playbook` | Execute SOAR automated response | ±0.10 |
| 10 | `submit_containment_plan` | Final report — ends episode | 0.0 to 1.0 |

### 4.2 Data Models — `models.py`

All data flows through strict **Pydantic models** (429 lines):

- **Enums**: `Severity`, `ThreatType` (13 types), `HostStatus`, `SubnetRole` (6 roles)
- **Sub-models**: `Alert`, `HostInfo`, `NetworkTopology`, `ForensicsResult`, `TimelineEntry`
- **`SOCObservation`** (extends OpenEnv `Observation`): 20+ fields including `alert_queue`, `network_topology`, `host_forensics`, `threat_graph_summary`, `reward_dimensions`, `available_playbooks`
- **Actions**: Discriminated union of 10 action types via `SOCActionWrapper`
- **`SOCState`** (internal): Tracks all episode state — killed processes, blocked IOCs, isolated subnets, etc.

### 4.3 Task Definitions — `tasks.py`

Three hand-crafted benchmark scenarios:

| Task | Threats | Hosts | Max Steps | Description |
|------|---------|-------|-----------|-------------|
| **Easy** | 1 | 1 | 15 | Single ransomware on WS-042 |
| **Medium** | 3 | 4 | 25 | Phishing → credential theft → lateral movement across 3 subnets |
| **Hard** | 5 | 7 | 30 | Full APT: phishing → C2 → privesc → exfil → ransomware |

**Network**: ~75 active hosts across 6 subnets (corporate, engineering, finance, DMZ, datacenter, executive) with realistic processes, ports, and criticality scores.

### 4.4 Procedural Task Generator — `task_generator.py`

Generates **1,000+ unique deterministic scenarios** from a seed:

- `hash(task_id)` → deterministic `random.Random` seed → drives ALL choices
- **Template pools**: 90+ malware process names, 40 C2 domains, 36 C2 IPs, 12 ransomware extensions, 12 data types
- **3 difficulty tiers**: Easy (1 threat), Medium (2-3 threats, multi-stage chains), Hard (3-6 threats, APT campaigns)
- **Alert generation**: Templated descriptions with randomized details (timestamps, file counts, data sizes)

### 4.5 Grading System — `graders.py`

**10-dimensional weighted grading:**

| Dimension | Weight | What It Measures |
|-----------|--------|-----------------|
| `threat_containment` | 0.20 | Fraction of required process kills completed |
| `ioc_blocking` | 0.12 | Fraction of known IOCs blocked (penalizes blind blocking) |
| `forensic_investigation` | 0.10 | Compromised hosts examined |
| `siem_correlation` | 0.08 | Whether alerts were correlated (bonus for early correlation) |
| `threat_intel_usage` | 0.08 | IOCs enriched with threat intel |
| `vuln_root_cause` | 0.08 | CVE root causes discovered (bonus if cited in plan) |
| `business_impact` | 0.10 | Penalizes unnecessary isolation and over-isolation (>20% = -0.30) |
| `step_efficiency` | 0.07 | Rewards SOAR playbook usage, penalizes step overrun |
| `plan_coverage` | 0.10 | Threats addressed in final plan |
| `plan_evidence_quality` | 0.07 | Evidence confidence from ThreatGraph |

**Anti-gaming**: Per-occurrence penalty cap (±0.15), blind-blocking penalties, normalized evidence confidence.

### 4.6 Threat Graph — `threat_graph.py`

A **typed knowledge graph** tracking all SOC entities:

- **5 Node Types**: `HostNode`, `ProcessNode`, `IOCNode`, `VulnerabilityNode`, `AlertNode`
- **6 Edge Types**: `runs_on`, `involves`, `communicates_with`, `pivoted_from`, `part_of_chain`, `exploits`
- **200-node cap** with LRU IOC pruning
- **Version tracking** with changelog for delta queries
- **Evidence confidence** computation for plan quality scoring
- **Context summary** generation for LLM injection

### 4.7 SOAR Playbooks — `soar_playbooks.py`

5 automated response playbooks with prerequisite validation:

| Playbook | Prerequisites | Sub-Actions |
|----------|--------------|-------------|
| `ransomware_containment` | Forensics run, process identified | kill_process, block_ioc |
| `c2_disruption` | IOC enriched, C2 IP identified | block_ioc, isolate_segment |
| `lateral_movement_lockdown` | Forensics run, lateral movement detected | kill_process, isolate_segment |
| `phishing_response` | Phishing vector confirmed | enrich_ioc, block_ioc |
| `data_exfil_stop` | Forensics run, exfil destination identified | block_ioc, kill_process |

### 4.8 Action Validation — `action_validation.py`

**3-gate middleware:**
1. **Phase whitelist** — Actions restricted by phase (triage/investigation/remediation/report)
2. **Schema validation** — Required arguments checked
3. **Graph groundedness** — Actions must reference discovered entities (can't block an IOC you haven't seen)

### 4.9 Tool Router — `tool_router.py`

**Deterministic phase state machine:**
- Phases: `triage` → `investigation` → `remediation` → `report` → `done`
- Loop limits: max 4 investigation loops, 3 remediation loops
- Supports **pushback** — agent can justify staying in a phase with graph references

**Triage Solver**: Priority = `severity_weight × criticality_weight × (1 + blast_radius/10)`

### 4.10 Episode Sandbox — `episode_sandbox.py`

**Safety guardrails:**
- **120-second wall-clock timeout** per episode
- **20-step hard limit** per episode
- **State integrity protection** — Protected fields (`_task_def`, `_live_requirements`, `_threat_graph`) are snapshot-hashed; mutations are detected and rolled back
- **Hacking detection** — Reports any external state tampering

### 4.11 Adaptive Red Team

Two mechanisms in `play_environment.py`:

1. **Reinfection** (`_maybe_reinfect`): 30% chance when killing a process if IOCs in the chain are unblocked → spawns `process_v2` variant + CRITICAL alert
2. **Lateral Pivot** (`_execute_lateral_pivot`): Triggered by isolate/kill actions on hard tasks → copies malware to adjacent healthy host, adds `pivoted_from` edge, emits PIVOT alert, updates live requirements

**Escalation**: Probability increases when agent is slow (step > 10 with 0 containments).

### 4.12 Server Application — `app.py`

FastAPI app created via OpenEnv's `create_app()`:
- **POST /reset** — Reset environment with task_id
- **POST /step** — Execute an action
- **GET /state** — Get current state
- **WS /ws** — WebSocket for persistent sessions
- CORS enabled for dashboard communication
- Supports 4 concurrent environment instances

---

## 5. Frontend (Dashboard)

### 5.1 Overview

A real-time **"CyberSOC Command Center"** web dashboard with 6 panels, built with vanilla HTML/CSS/JS + D3.js + Chart.js.

### 5.2 Six Dashboard Panels

1. **Alert Queue** — Live SIEM/EDR alerts with severity badges and IOC indicators
2. **Live Threat Graph** — D3.js force-directed graph with 5 node types, drag/zoom, glow effects, pivot animation
3. **Agent Actions** — Chronological action log with reward tracking
4. **Network Topology** — Visual subnet map with compromised/isolated counts
5. **Performance Metrics** — Chart.js radar chart (10 dimensions) + cumulative reward timeline
6. **Mission Status** — Containment progress bars, business impact gauge, active threat list, episode controls, **🔴 Red Team Toolkit** (hotseat multiplayer)

### 5.3 Visual Design

- **Dark theme** with glassmorphism panels
- **Typography**: Inter (UI) + JetBrains Mono (data)
- **Color system**: Accent colors for cyan, green, amber, red, purple
- **Animations**: Count-up numbers, scale bounces, pulse glows, screen flashes
- **Red Team pivot**: Screen border flash, toast notification, traveling dot animation on pivot edges

### 5.4 Key Frontend Components

**`graphs.js` (881 lines)**:
- `ClientThreatGraph` — Client-side graph state manager synced from observations
- `ThreatGraphViz` — D3.js v7 force simulation with SVG glow filters, curved edges, node symbols (circle/diamond/triangle/square/wye), click-to-highlight, drag behavior
- `RadarChart` — Chart.js 10-axis radar for live grading dimensions
- `RewardTimeline` — Gradient-filled cumulative reward line chart

**`app.js` (45KB)** — Main orchestrator handling episode lifecycle, API calls, UI updates, phase indicator tracking

**`api.js`** — REST client with auto-detection of server origin, session management, response parsing

**`animations.js`** — Utility library for count-up, screen flash, toast notifications, scale bounce, pulse glow, dramatic final score reveal

### 5.5 Dashboard Server — `dashboard_server.py`

Wraps the FastAPI app to also serve the dashboard as static files at `/dashboard/`. Prints a styled ASCII banner on startup.

---

## 6. Inference & Training

### 6.1 Inference Script — `inference.py`

LLM baseline agent using **OpenAI-compatible API**:
- System prompt defines SOC analyst role with all 6 core actions
- Formats observations into structured text for the LLM
- Parses JSON actions from LLM responses (with fallback extraction)
- Runs episodes across easy/medium/hard tasks
- Emits structured stdout logs: `[START]`, `[STEP]`, `[END]` (hackathon requirement)
- Default model: `Qwen/Qwen2.5-72B-Instruct` via HuggingFace Router

### 6.2 GRPO Reward Functions — `training/reward_funcs.py`

10 TRL-compatible reward functions for **Group Relative Policy Optimization**:

```python
from training.reward_funcs import make_soc_reward_funcs
reward_fns = make_soc_reward_funcs("http://localhost:8000")
trainer = GRPOTrainer(model=model, reward_funcs=reward_fns, args=GRPOConfig(...))
```

Each function:
1. Parses completion as JSON action list
2. Replays actions against live environment server
3. Returns the specific dimension's score from `grade_breakdown`
4. Non-parseable completions return 0.0

### 6.3 Per-Step Reward Dimensions

The environment computes **live partial scores** every step (`_compute_reward_dimensions`) for GRPO credit assignment without waiting for the terminal grade. These are exposed in `SOCObservation.reward_dimensions`.

---

## 7. Testing

**11 test files** covering all major components:

| File | Focus |
|------|-------|
| `test_integration.py` | Full episode flows, phase violations, adaptive pivots, 10-dim grading, sandbox limits |
| `test_task1.py` - `test_task9.py` | Individual task-specific validations |

Key integration tests:
- Easy/medium episodes complete without crashes
- All 10 action types can be exercised in a single episode
- Phase violations return negative reward (not crash)
- Adaptive pivot fires on hard tasks
- Step rewards accumulate correctly and are idempotent
- Grader returns exactly 10 dimensions
- Sandbox step limit raises `EpisodeTimeout`

---

## 8. Deployment & DevOps

### Docker
- **Root Dockerfile** — Slim Python 3.10, serves on port 7860 (HuggingFace Spaces)
- **Server Dockerfile** — Multi-stage build from `ghcr.io/meta-pytorch/openenv-base`, uses `uv` for dependency management, health check on `/health`

### Validation
`validate_submission.sh` — 3-step validator:
1. Ping HF Space `/reset` endpoint
2. Docker build succeeds
3. `openenv validate` passes

### OpenEnv Manifest
`openenv.yaml` — 1,003 task definitions with descriptions, max steps, and difficulty tags. Used by the OpenEnv framework for task discovery and benchmarking.

---

## 9. Environment Variables

| Variable | Purpose | Default |
|----------|---------|---------|
| `API_BASE_URL` | LLM API endpoint | `https://router.huggingface.co/v1` |
| `MODEL_NAME` | Model identifier | `Qwen/Qwen2.5-72B-Instruct` |
| `HF_TOKEN` | HuggingFace API key | — |

---

## 10. Data Flow

```mermaid
sequenceDiagram
    participant Agent as LLM Agent
    participant Inf as inference.py
    participant Env as CyberSOCEnvironment
    participant TG as ThreatGraph
    participant Gr as Grader

    Inf->>Env: reset(task_id="hard")
    Env->>TG: populate from task_def
    Env-->>Inf: SOCObservation (alerts, topology)
    
    loop Each Step
        Inf->>Agent: format_observation → LLM prompt
        Agent-->>Inf: JSON action
        Inf->>Env: step(SOCActionWrapper)
        Env->>Env: ActionMiddleware.validate()
        Env->>Env: Handle action (query/forensics/kill/etc)
        Env->>TG: Update graph nodes/edges
        Env->>Env: _adversary_react() (adaptive pivot)
        Env->>Env: _compute_reward_dimensions()
        Env-->>Inf: SOCObservation (updated state)
    end
    
    Inf->>Env: step(submit_containment_plan)
    Env->>Gr: grade_episode(actions, plan, graph, task_def, state)
    Gr-->>Env: {final_score, breakdown[10], penalties, bonuses}
    Env-->>Inf: SOCObservation (done=true, final_score)
```

---

## 11. Red Team Design Philosophy

The Red Team is NOT a separate LLM agent. It is a **deterministic adversarial dynamics engine** that defines the environment's state transition function.

### 7 Behavioral Mechanisms
1. **Reactive Pivoting**: Triggers on `isolate_segment` and `kill_process` (copy-not-move spread)
2. **Persistence**: Reinfection triggers when a process is killed but its root IOC remains unblocked (teaches causal reasoning)
3. **Time Pressure**: Pivot probability escalates +0.2 after step 10 if zero containments are achieved
4. **Controlled Randomness**: Uses an episode-scoped `self._rng` (seeded by `task_id`) to ensure deterministic rollouts
5. **Noisy Observations**: Benign processes mixed in host data
6. **Escalation**: Pivot probabilities scale with difficulty (`Easy: 0.0`, `Medium: 0.3`, `Hard: 0.8`)
7. **Stall Punishment** *(new)*: If Blue makes 3+ consecutive passive actions (`query_host` / `pass_turn`) without containment, Red immediately deploys ransomware; plus a 15 % chance to spread laterally even on passive Blue turns

### Attack Lifecycle Model (MITRE-aligned)
`Phase 1: Compromise` → `Phase 2: Lateral Movement` → `Phase 3: Persistence` → `Phase 4: Escalation` → `Phase 5: Impact`

---

## 12. Reward-Hacking Defense Map

Per guide §8, we implemented specific defenses against the known RL exploit vectors:

| Guide Attack Vector | Our Defense |
|---|---|
| Editing timers | `EpisodeSandbox` wall-clock enforcement |
| Caching results | Idempotent step rewards via `_fired_step_rewards` |
| Abusing globals | Instance-scoped RNG + episode-scoped `self._rng` |
| Mutating protected state | Sandbox hash-snapshot + rollback |
| Exploiting env bugs | 3-gate validation middleware |
| Reward-function gaming | Evidence confidence normalization |
| Cheating via blind remediation | Graph-groundedness gate |
| Blind IOC blocking | Enrichment-before-block penalty |

---

## 13. Curriculum Learning Strategy

The 1000+ deterministic scenarios generated by `task_generator.py` are explicitly divided into three difficulty tiers to support Curriculum Learning (Guide §6).

This exists precisely to satisfy **Daniel's Law of RL**: *The base model must get non-zero reward on Easy before it can meaningfully learn Hard.*

- **Phase 1 (Warm-Start)**: `gen_0001`–`gen_0333` (Easy). Single threat, 15 max steps, 0.0 pivot probability.
- **Phase 2 (Scaling)**: `gen_0334`–`gen_0666` (Medium). Multi-stage, 25 max steps, 0.3 pivot probability.
- **Phase 3 (Stress-Test)**: `gen_0667`–`gen_1000` (Hard). APT, 30 max steps, 0.8 pivot probability.

The adaptive pivot probability is itself a curriculum signal; the environment gets harder as the agent gets better.

---

## 14. Intended Training Stack

CyberSOCEnv is designed for the canonical stack specified in Guide §10:

1. **Unsloth**: 4-bit QLoRA loading and efficient inference
2. **TRL**: `GRPOTrainer` consuming our 10 independent callable functions via `reward_funcs=[...]`
3. **OpenEnv**: WebSocket transport and session isolation
4. **vLLM**: Serving the rollout workers for maximum throughput

A reference adapter module exists at `training/reward_funcs.py` that mirrors the Unsloth 2048 notebook structure 1:1, allowing plug-and-play GRPO training.

---

## 15. Anti-Patterns Avoided

How we avoided the 7 common mistakes listed in Guide §21:

1. **Building before designing env**: Env, types, and sandbox were built and tested completely offline before any trainer was attached.
2. **LLM-as-a-judge**: CyberSOCEnv uses zero LLM-as-judge signals. Everything is deterministic code against the ThreatGraph.
3. **Single monolithic reward**: We use a 10-dimensional verifiable rubric, fed independently into TRL.
4. **Ignoring inference latency**: We implemented Graph Delta Injection (~10x fewer tokens) and a sparse-node generation strategy (~75 active nodes) specifically to optimize GRPO rollout latency.
5. **No abuse prevention**: 3-gate middleware + EpisodeSandbox explicitly prevent out-of-band cheating.
6. **Delayed deployment**: Environment was packaged with Docker and deployed to HF Spaces early.
7. **Scaling prematurely**: All 9 components passed integration testing (`test_integration.py` through `test_task9.py`) before scaling to 1000 tasks.

---

## 16. Known Deviations & Alignment Items

While we strive to match the OpenEnv canonical scaffolding (Guide §5), there are a few intentional architectural differences:

1. **Action Dispatch**: We use a discriminated union wrapper (`SOCActionWrapper` with a `type` field) rather than a single flat action class. This matches the MCP ToolCall pattern and real SOC work better than a flat action space.
2. **Decoupled Engine**: The core logic lives in `server/play_environment.py`, completely separate from the FastAPI transport layer in `server/app.py`. This ensures we can run headless parallel environments during GRPO without HTTP overhead if needed.

---

## 17. Team Structure & Role Split

Per guide §17, responsibilities are split across three functional roles to execute the RL pipeline effectively.

### Role 1: Environment Engineer
**Mission**: Build a deterministic, unhackable, fast environment.
**Owns**: `play_environment.py`, `tasks.py`, `threat_graph.py`, `episode_sandbox.py`, `action_validation.py`, `tests/`
- **Scope**: Implements the state machine, Red Team behavior, and validates actions. Owns the core `step()` and `reset()` loops. Ensures the environment parses valid inputs and securely handles invalid ones.
- **Hackathon Focus**: Bug fixes, latency optimization (graph deltas), sandbox integrity, and procedural scenario generation.

### Role 2: Reward Engineer
**Mission**: Design the mathematical signals that shape model behavior.
**Owns**: `graders.py`, `training/reward_funcs.py`, `models.py`
- **Scope**: Creates the 10-dimensional verifiable grading logic. Plumbs the environment outputs into TRL-compatible `reward_funcs`. Tunes the penalties to prevent reward hacking (e.g., punishing blind IOC blocking).
- **Hackathon Focus**: Ensuring the model gets positive step-rewards early on to prevent it from collapsing, while preventing it from finding "lazy" exploits.

### Role 3: Training Engineer
**Mission**: Execute the GRPO curriculum and produce the final model.
**Owns**: `training/` directory, Colab notebooks, `inference.py`
- **Scope**: Sets up the actual training loops using Unsloth and TRL. Manages the hyperparameter tuning, LoRA checkpointing, and vLLM inference configuration. Runs the curriculum from Easy to Hard.
- **Hackathon Focus**: Capturing the before/after learning curves on held-out tasks to prove to the judges that the environment actually works to train a model.

---

## 18. Key Innovations Summary

| Innovation | Description |
|-----------|-------------|
| **Procedural Generation** | SHA-256 seeded RNG generates 1000+ unique deterministic scenarios |
| **ThreatGraph** | Typed knowledge graph with version tracking, evidence confidence, and LRU pruning |
| **10-Dim Grading** | Weighted multi-dimensional scoring replacing binary pass/fail |
| **Adaptive Red Team** | Attacker reacts to defender actions — lateral pivots and reinfection |
| **SOAR Playbooks** | Prerequisite-gated automated response workflows |
| **3-Gate Validation** | Phase whitelist + schema + graph-groundedness prevents invalid actions |
| **Episode Sandbox** | State integrity protection with hash-based tampering detection |
| **Live GRPO Signals** | Per-step reward dimensions for RL credit assignment |
| **Anti-Gaming** | Blind-blocking penalties, over-isolation cap, idempotent step rewards (0.40 cap) |
| **Real-time Dashboard** | D3.js threat graph with pivot animations and 10-dim radar chart |
| **Hotseat Multiplayer** | Human Red Team player via in-dashboard toolkit; FSP backend; per-turn UI lock |
| **Stall Punishment** | 3 consecutive passive Blue actions triggers ransomware deploy + 15 % autonomous pivot |
| **Emergency Gate** | `isolate_segment` in triage requires a critical alert; `UNJUSTIFIED_EMERGENCY` → -0.15 penalty |
| **External Intel Feed** | `task_def.external_intel_feed` IOCs injected at reset — immediately blockable/enrichable |
| **Doomsday Clock** | `state.business_impact × 0.30` applied as a direct score modifier; 90 % negligence crush when no threats contained |

---

## 19. Technology Stack

| Layer | Technologies |
|-------|-------------|
| **Backend** | Python 3.10+, FastAPI, Uvicorn, Pydantic v2, OpenEnv Core |
| **Frontend** | Vanilla HTML/CSS/JS, D3.js v7, Chart.js v4, Inter/JetBrains Mono fonts |
| **Inference** | OpenAI Python SDK, asyncio |
| **Training** | TRL (Hugging Face), GRPO |
| **DevOps** | Docker (multi-stage), uv package manager, pytest |
| **Deployment** | HuggingFace Spaces (Docker SDK) |
| **Visualization** | NetworkX + Matplotlib (server-side PNG), D3.js (client-side interactive) |