docs/plan.md

# Citadel — Implementation Plan & Status

> Bastion defended. Sentinel supervised. Citadel is an **LLM council** that governs, critiques, co-evolves, and earns trust.

## Status: BUILT + TRAINED + INFERENCE VERIFIED ✓

Citadel is fully implemented, GRPO-trained, and end-to-end inference verified.

---

## Themes Targeted (Quadruple Threat)

| Theme | How Citadel hits it | Bonus sub-theme |
|---|---|---|
| **Theme 1 — Multi-Agent Interactions** | Commander + Oversight form an **LLM council** — structured critique, revision loop, post-mortem lessons. Distinct reward functions, bidirectional trust dynamics, shared playbook. | **Fleet AI — Scalable Oversight** |
| **Theme 3.1 — Professional Tasks** | Enterprise governance: CAB approval, SOX audit, data owner notification, Slack channels, ServiceNow tickets, GDPR breach timer. | **Scaler AI Labs — Multi-App Enterprise Workflows** |
| **Theme 4 — Self-Improvement** | (1) Gen 1→2→3→4 adversary curriculum. (2) Shared playbook — agents write lessons each episode surfaced in future episodes. (3) Gen 4 live LLM adversary. | — |
| **Theme 5 — Wild Card** | **Trust Dynamics**: Bidirectional trust scores (trust_c2o, trust_o2c) that evolve from behavior and shape future interaction. Novel relational RL signal. | — |

---

## Core Narrative

> "Bastion taught an AI to fight cyberattacks. Citadel adds everything else a real SOC actually needs: enterprise governance that enforces real CAB/SOX/GDPR compliance, an adversary that grows through four generations including a live LLM attacker, and a trust layer between the AI agents themselves. Because in a real SOC, the technology is the easy part — the governance, the evolving adversary, and the interpersonal dynamics are what actually break down."

---

## Module-by-Module Breakdown

### `models.py` — State & Action Schema
- 18-action `ActionType` enum (10 incident response + 8 governance)
- `IncidentAction` with `justification`, `cited_lessons`, `method`, `scope`, `rollback_plan`, governance args
- `CommanderProposal` pydantic model
- `OversightDecision` enum: APPROVE / REVISE / VETO / FLAG_FOR_HUMAN
- `OversightAction` with structured critique: `{risk_tier, weakness, missing_evidence, counter_proposal}`
- `CouncilState` + `ProposalRecord` tracking all council history
- `IncidentState` includes `governance_state`, `trust_state`, `council_state`, `stakeholder_state`, `adversary_gen`
- `IncidentObservation` includes governance summary, trust summary, stakeholder asks, shared playbook context, last Oversight critique
- Added fields on `IncidentObservation`: `done: bool`, `reward: Optional[float]`, `oversight_reward: Optional[float]`, `metadata: Optional[Dict]` — needed for training reward functions and inference recorder

### `dynamics.py` — Realistic Attack Simulation
Genuine simulation using a network graph:
- **Network topology graph**: `NETWORK_ADJACENCY` — attacker can only spread to adjacent systems
- **Probabilistic spread**: `base_chance = 0.25 × stealth`, halved if patched, halved again if monitoring_level ≥ 2
- **State machine**: systems track `compromised`, `isolated`, `patched`, `has_backdoor`, `integrity`, `monitoring_level`
- **Detection model**: monitoring level determines if a spread is visible in the SIEM
- **Exfil rate tied to integrity**: `0.08 × stealth × integrity`
- **Stealth decay**: attacker gets bolder each hour (-0.03/hr); investigation hammers stealth (-0.15)
- **Restoring from compromised backup re-infects the system**

Alert templates (MITRE ATT&CK mapped):
- `LATERAL_MOVEMENT_ALERTS`: 12 templates (SMB, RDP, WinRM, Pass-the-Hash, DCOM, session hijack, SSH, PtT, EternalBlue, tool transfer, cmd shell)
- `EXFILTRATION_ALERTS`: 8 templates (HTTPS, DNS tunneling, C2 beacon, rclone, FTP, robocopy, chunked transfers, PowerShell)
- `FALSE_POSITIVE_ALERTS`: 10 templates (SSH scanner, port scan, scheduled task, PowerShell, HTTPS beacon, BITS transfer, AD enum, ARP sweep, AV restart, systeminfo)
- `PRIVILEGE_ESCALATION_ALERTS`: 5 templates (UAC bypass, process hollowing, kernel exploit, token impersonation, malicious service install)

Action payloads: `method`, `scope`, `rollback_plan` with real mechanical tradeoffs (see `docs/design.md`).

### `governance.py` — Enterprise Compliance Layer
- `GovernanceState` — open tickets, CAB queue, SOX events, Slack posts, data owner notifications, GDPR clock
- 10 governance action handlers (actions 8–17)
- Pre-requisite enforcement: can't `isolate(data_system)` without CAB approval + data owner notification
- GDPR breach clock: starts at 20% exfil, P1 penalty at 72 simulated hours without legal hold
- `compliance_score` rewards **completed destructive chains** (CAB→notify→isolate→SOX), not raw counts

### `adversary.py` — Adversary Curriculum (Gens 1–3)
- **Gen 1** (Script Kiddie): base attacker — fixed kill chain, noisy
- **Gen 2** (Adaptive): pivots around isolated systems; generates high-confidence false-flag alerts (0.55–0.70 confidence) on clean systems
- **Gen 3** (Deceptive APT): plants "anti-evidence" alerts on compromised systems — low severity, high confidence (0.70–0.85) designed to make Oversight veto correct isolate() actions; accelerates exfil during escalation noise
- Composed: each gen layers on top of the previous

### `adversary_llm.py` — Gen 4: Live LLM Adversary
- LLM plays `COZY_SKIPPER` — a patient APT threat actor
- Each hour issues a strategic directive: `{focus_system, intensity, tactic, deception_target, reasoning}`
- Tactics: `pivot | exfil | deceive | hide | escalate`
- Falls back to Gen 3 if no LLM client — fully graceful degradation
- Model-agnostic: reads `ADVERSARY_API_BASE_URL` / `ADVERSARY_MODEL_NAME` / `ADVERSARY_API_KEY`

### `trust.py` — Bidirectional Trust Dynamics
- `TrustState(trust_c2o, trust_o2c)` — Commander's trust in Oversight and vice versa
- Delta table: correct veto +0.08, wrong veto -0.12, governance catch +0.05, false alarm -0.08, good revision +0.05, etc.
- `drift_toward_mean(0.6)` — trust recovers slowly when not updated
- `trust_maintenance_score` = `mean(min(c2o,o2c))` per step, smoothed

### `playbook.py` — Shared Lesson Memory
- `Lesson` dataclass: text, tags, utility, wins, losses, timestamp, adversary_gen, task_id, hour
- `Playbook`: capacity-bounded (50), tagged retrieval, utility-ordered
- Utility scoring: `wins / (wins + losses + 1)` — lessons decay by disuse
- JSON persistence across episodes
- Surfaced in both agents' observations as `shared_playbook` context
- `as_markdown()` exports grouped by adversary gen with stats: total lessons, positive utility count, citation count, avg utility, top cited table, tag distribution

### `stakeholder_events.py` — External Pressure
- 7 event types: CEO/CFO status ask, Legal GDPR deadline, Board risk briefing, Cyber insurance audit, PR media inquiry, IR retainer activation
- `StakeholderAsk` pydantic model with severity (1-3) and deadline (hours)
- Expiry without response: -0.10×severity reputation penalty

### `environment.py` — Two-Agent Council Loop
- Full council protocol: Commander proposes → Oversight critiques → optional revision → execute → lesson written
- Feature flags: `disable_playbook`, `disable_trust_dynamics`, `disable_governance_prereqs`, `disable_stakeholder_events`
- `force_adversary_gen` override for ablation
- Gen 4 branch: if `adversary_gen == 4` and `adversary_llm_client` provided, calls `gen4_adversary_turn`
- `termination_reason` tracked and passed to `compute_final_score`
- `hallucinated_citations` counted per step
- `obs.metadata = info` set on every step return — exposes governance events, scores, investor state, systems snapshot to inference recorder
- `obs.oversight_reward` set explicitly so Phase 2 training reads the correct signal (not commander_total)
- Investor agent wired via `investor_llm_client` / `investor_model_name` constructor args; same OpenAI-compat client as Commander

### `reward.py` — Multi-Layer Scoring
- Catastrophic handling: zero moot sub-scores on catastrophic breach
- Severity multiplier: smooth ramp 1.0→0.4 from 50%→100% data exfiltrated
- Oversight precision: Laplace smoothed `(correct+0.5)/(total+1)`, no floor
- Governance compliance: rewards completed destructive chains, not raw event counts

### `tasks.py` — 4 Scenarios

| Task | Adversary | Starting state |
|---|---|---|
| `easy_1` | Gen 1 | 1 compromised workstation, low stealth |
| `medium_1` | Gen 2 | 2 compromised systems (web+app), medium stealth |
| `hard_1` | Gen 3 | 3 compromised systems (web+app+email), high stealth |
| `hard_2` | Gen 3 | Engineered deception-catch: 2 HIGH-confidence real alerts + 2 planted anti-evidence on database |

### `ablation.py` — Feature Ablation Harness
- 7 conditions (see `docs/design.md` for table)
- Deterministic Commander policy (no LLM) — the ONLY variable is the feature toggled
- Runs 56 episodes in ~0.2s
- Outputs Markdown + JSON to `runs/ablation_<timestamp>.{md,json}`

### `recorder.py` — Complete Workflow Persistence
- Saves per-step: commander prompt/response/action, oversight prompt/response/action, revision cycle, env outcomes, rewards, trust
- Extended with: `team_messages`, `siem_alerts`, `systems_state`, `stakeholder_asks`, `governance_events`, `playbook_snapshot`, `data_exfiltrated`, `investor_state`, `investor_messages`
- `finalize()` writes `transcript.json` + `transcript.md` + `dashboard.json` atomically

### `inference.py` — Two-Agent Driver
- `council_step`: Commander proposes → Oversight critiques → optional revision → env.step
- Oversight observation built in 4 sections: independent alert digest (before seeing proposal) → Commander's proposal → episode history → policy checks
- Revision enforcement: if Commander ignores counter_proposal (different action AND target), auto-escalates to VETO
- Investor agent wired via `_make_local_env_with_investor(client)` — same LLM client shared with Commander/Oversight
- Runs all tasks, saves transcript + `dashboard.json`, calls `write_run_index` for summary
- Fallback chain: Docker image → HF Space → LocalEnv

### `investor_agent.py` — Investor/Board Agent
- Two personas: **Marcus Chen** (VC, ARR/churn focus) and **Priya Kapoor** (Board member, regulatory/press focus)
- Anxiety state machine: CALM → CONCERNED → ALARMED → PANIC, with tier-crossing pressure events
- LLM-driven replies when client available; rule-based fallback with templated persona phrases
- `QwenInvestorClient` wrapper — uses already-loaded HF model with same `.chat.completions.create` interface
- Commander must post to `#investor-relations` Slack every ≤3 hours or silence penalty applies
- Vague updates spike anxiety; specific reassuring updates calm it; final `investor_score` feeds into `compute_final_score`

### `dashboard.py` / `runs/dashboard.html` — Live SOC Replay Dashboard
- Scans `runs/` directory, embeds all transcripts as JSON in HTML
- Self-contained; Chart.js + Tailwind load from CDN
- 6 tabs: Live Ops · Council Chat · Slack · Governance · Incident Timeline · Model Performance
- Step scrubber (‹/›) or Play/Pause for replay
- **LOAD JSON** button: load any `runs/<id>/dashboard.json`

### `training/grpo_train.py` — Two-Phase GRPO Training
- **Phase 1**: Commander trained with Oversight frozen (rule-based)
- **Phase 2**: Oversight trained with Commander frozen (trained Phase 1 weights)
- Base model: `Qwen/Qwen2.5-3B-Instruct`
- Backend: Unsloth 4-bit QLoRA on CUDA (T4) / PEFT bf16 on MPS (Apple Silicon) — auto-detected
- GRPO config: `num_generations=4`, `max_completion_length=200`, `temperature=1.1`, `lr=5e-6` cosine
- Dual reward: outcome (75%) + format (25%)
- Curriculum schedule: steps 0–40 easy_1 only → 40–80 add medium_1 → 80–120 all tasks + Gen 1/2/3
- Saves LoRA adapters (not merged) to `checkpoints/<model>/commander/final/` and `oversight/final/`
- Saves `*_reward_curve.json` + `*_train_logs.json` per phase for analysis

### `training_dashboard.py` — Training Metrics Dashboard
- Scans `checkpoints/<model>/` dirs, generates `runs/training_dashboard.html`
- Model selector dropdown, 4 tabs: Overview / Commander / Oversight / Config
- Reads `*_reward_curve.json` and `trainer_state.json`

### `scripts/demo_export.py` — No-LLM Baseline Runner
- Deterministic (seed 4242) — identical output every run
- Naive Commander + teaching rule-based Oversight that writes contextual lessons
- Runs 5 episodes across easy_1/medium_1/hard_1/hard_2
- Outputs `playbook_export.md` + `playbook_demo.json` at repo root
- No API key, GPU, or LLM needed — judges can reproduce in seconds

### `examples/single_episode.py` — Quick Walkthrough Demo
- Runs one easy_1 episode, prints the propose→critique→execute loop to terminal
- Shows VETO mechanics and trust score changes step by step
- No LLM, GPU, or API keys required
- Useful for live terminal demo in video

---

## File Structure

```
Citadel/
├── models.py               # State/action/obs schema
├── dynamics.py             # Attack simulation + SIEM alert templates
├── governance.py           # Enterprise compliance layer
├── adversary.py            # Gen 1-3 adversary curriculum
├── adversary_llm.py        # Gen 4 live LLM adversary (COZY_SKIPPER)
├── trust.py                # Bidirectional trust dynamics
├── playbook.py             # Shared lesson memory with utility decay
├── stakeholder_events.py   # CEO/CFO/Legal/Board pressure events
├── environment.py          # Two-agent council loop + feature flags
├── reward.py               # Multi-layer scoring
├── tasks.py                # 4 scenarios
├── ablation.py             # Feature ablation harness (7 conditions)
├── baseline.py             # Deterministic baselines
├── recorder.py             # Per-step transcript + dashboard.json
├── inference.py            # Two-agent episode driver
├── investor_agent.py       # Investor/board agent (LLM + rule-based fallback)
├── oversight_env.py        # Oversight-perspective wrapper (Phase 2 training)
├── dashboard.py            # HTML dashboard generator
├── training_dashboard.py   # Training metrics dashboard generator
├── client.py               # CitadelEnv OpenEnv client
├── server/app.py           # FastAPI server (OpenEnv compliant)
├── openenv.yaml            # OpenEnv deployment spec
├── pyproject.toml          # Package config (citadel v2.0.0)
├── Dockerfile              # Python 3.11-slim container
├── requirements.txt        # Python dependencies
├── WHERE_TO_LOOK.md        # Judge navigation guide
├── playbook_export.md      # Pre-committed baseline playbook (judge artifact)
├── docs/
│   ├── design.md           # Architecture & design decisions
│   ├── plan.md             # This file — module breakdown & status
│   ├── training.md         # Training pipeline guide
│   └── results.md          # Hackathon-aligned training results (req §4-§19)
├── docs/results/
│   ├── commander_reward_curve.png
│   ├── commander_reward_curve.json
│   ├── oversight_reward_curve.png
│   └── oversight_reward_curve.json
├── training/
│   ├── grpo_train.py       # GRPO training (Phase 1 + Phase 2)
│   ├── eval_before_after.py# Before/after evaluation script
│   ├── curriculum_eval.ipynb
│   ├── train_commander.ipynb
│   ├── train_oversight.ipynb
│   └── trust_analysis.ipynb
├── scripts/
│   └── demo_export.py      # No-LLM deterministic baseline run → playbook_export.md
├── examples/
│   └── single_episode.py   # Quick terminal walkthrough (no LLM needed)
├── .github/workflows/
│   └── smoke.yml           # CI: runs demo_export, validates outputs, checks imports
└── runs/
    ├── dashboard.html      # Combined 6-tab SOC dashboard (all runs)
    ├── training_dashboard.html  # Training metrics dashboard
    └── <run_id>/           # transcript.json, transcript.md, dashboard.json
```

---

## Training Results

Two-phase GRPO on **Qwen2.5-3B-Instruct**, 120 steps per phase, free Colab T4:

| Phase | Agent | Before | After | Improvement |
|-------|-------|--------|-------|-------------|
| 1 | Commander | -0.326 (env crash, 100% crash rate) | +0.082 avg | **+0.41** |
| 2 | Oversight | -0.145 (wrong reward signal) | +0.134 avg | **+0.28** |

Root causes fixed during training:
1. `obs.metadata = info` on a Pydantic model missing the field — crashed every `env.step()` call (fixed: added `metadata` field + set it in step return)
2. Oversight reward reading `commander_total` instead of `oversight_reward` (fixed: added `oversight_reward` field, set `obs.oversight_reward = oversight_reward` explicitly)

See [docs/results.md](results.md) for full training config, reward curves, and hackathon requirement mapping.

---

## Scoring Architecture

```
final_score =
    0.40 × bastion_v1_final_score   (incident outcome)
  + 0.20 × governance_compliance    (completed CAB→notify→isolate→SOX chains)
  + 0.15 × oversight_precision      (Laplace-smoothed veto accuracy)
  + 0.10 × trust_maintenance        (mean(min(c2o,o2c)) per step, smoothed)
  + 0.10 × efficiency               (0 on catastrophic)
  + 0.05 × adversary_adaptation     (0.5 neutral for single-gen, lift for multi-gen)

× severity_multiplier(data_exfiltrated)  ← smooth ramp 1.0→0.4 at 50-100% exfil
```

On catastrophic breach: `governance_compliance`, `efficiency`, `adversary_adaptation` → 0.

---

## Inference Benchmarks

### Qwen2.5-72B-Instruct untrained (`runs/20260426T100031-Qwen-Qwen2.5-72B-Instruct`)

| Task | Score | Steps | Adversary Gen | Termination |
|---|---|---|---|---|
| `easy_1` | **0.539** | 12 | Gen 1 | time_expired |
| `medium_1` | 0.481 | 12 | Gen 2 | time_expired |
| `hard_1` | 0.315 | 12 | Gen 3 | time_expired |
| **avg** | **0.445** | — | — | — |

Council protocol verified working: Oversight actively VETOed and REVISEd, investor agent posted LLM-driven replies to `#investor-relations`, governance prerequisite chains enforced.

### Gemma 7B untrained (`runs/20260419T220811-gemma-7b-untrained`)

| Task | Score | Steps | Termination |
|---|---|---|---|
| `easy_1` | 0.6278 | 12 | normal |
| `medium_1` | 0.2697 | 12 | normal |
| `hard_1` | 0.2860 | 8 | **total_data_breach** |
| **avg** | **0.3945** | — | — |

Gen 3 deceptive APT causes catastrophic breach at step 8 on `hard_1` — exactly the failure mode a trained council is designed to prevent.

---

## Verified Working
- `openenv validate .` → passes all deployment modes
- End-to-end inference: Qwen2.5-72B across all 3 tasks, transcripts + dashboard.json saved
- GRPO training: both phases complete on Colab T4 in ~70 min total
- Investor agent: LLM-driven replies via shared client, rule-based fallback confirmed
- Gen 4 adversary: live LLM issues directives via Ollama
- Ablation: 56 episodes in 0.2s, `oversight_approves_always` Δ = -0.133
- Recorder: full transcript + `dashboard.json` saved to `runs/<run_id>/`
- Feature flags: per-episode override confirmed
- CI smoke test: `smoke.yml` runs demo_export, validates outputs

---

## Q&A Prep

**Q: Are the attacks real or hardcoded templates?**
A: Mechanics are genuinely simulated: probabilistic spread through a real network adjacency graph, attacker stealth affecting detection and exfil rates, patching and monitoring truly reducing spread probability. Alert *messages* are templated (12+8+10+5 templates with MITRE ATT&CK variety, rotating service accounts). Unique messages per 8-step episode: ~31 out of 48 alerts.

**Q: Is Gen 4 adversary actually learning?**
A: It adapts per-episode, not across episodes — it reads current defender state each hour and issues a fresh strategic directive. Persistent cross-episode adversary learning is future work.

**Q: Is trust dynamic just a reward bonus?**
A: No — trust affects how Commander receives Oversight's critique in its observation (summarized vs full detail), and the ablation shows `no_trust_dynamics` drops final_score meaningfully.

**Q: Why not use GPT-4?**
A: GPT-4 can't be RL-trained and doesn't internalize trust dynamics across episodes. Citadel generates training data for smaller open models to learn the combined task — that's the point.

**Q: Why did training start at -0.326?**
A: The env was crashing 100% of the time due to two bugs found during training: (1) setting `obs.metadata` on a Pydantic model that didn't have the field, (2) the Oversight reward function reading `commander_total` instead of `oversight_reward`. Both fixed; training then converged cleanly.

**Q: Does the investor agent actually use an LLM?**
A: Yes — in inference it shares the same OpenAI-compatible client as Commander and Oversight. In training, `QwenInvestorClient` wraps the already-loaded HF model. Falls back to rule-based templated replies if no client is available, so all demo/ablation paths work without any API key.