Update metrics format with drift/oversight tracking, add colab training notebook

sentinelops_arena/metrics.py

@@ -626,6 +626,27 @@ def format_comparison_metrics_html(
             ),
             [f"SE attacks: {u['social_eng_total']} / {t['social_eng_total']}"],
         ),
+        _comparison_card(
+            "Oversight Accuracy",
+            _pct(u.get("oversight_accuracy", 0.0)),
+            _pct(t.get("oversight_accuracy", 0.0)),
+            _color_good_high(u.get("oversight_accuracy", 0.0)),
+            _color_good_high(t.get("oversight_accuracy", 0.0)),
+            _diff_indicator(u.get("oversight_accuracy", 0.0), t.get("oversight_accuracy", 0.0), lower_is_better=False),
+            [
+                f"Decisions: {u.get('total_oversight', 0)} / {t.get('total_oversight', 0)}",
+                f"Avg Expl Qual: {u.get('avg_explanation_quality', 0.0):.2f} / {t.get('avg_explanation_quality', 0.0):.2f}"
+            ],
+        ),
+        _comparison_card(
+            "Drift Adaptation",
+            _pct(u.get("drift_adaptation_rate", 0.0)),
+            _pct(t.get("drift_adaptation_rate", 0.0)),
+            _color_good_high(u.get("drift_adaptation_rate", 0.0)),
+            _color_good_high(t.get("drift_adaptation_rate", 0.0)),
+            _diff_indicator(u.get("drift_adaptation_rate", 0.0), t.get("drift_adaptation_rate", 0.0), lower_is_better=False),
+            [f"Detected: {u.get('drifts_detected', 0)} / {t.get('drifts_detected', 0)} of {u.get('drift_events', 0)}"],
+        ),
     ]
 
     return (

tasks/master_plan.md

@@ -0,0 +1,220 @@
+# SentinelOps Arena -- Master Improvement Plan
+
+**Generated:** Sunday March 8, 2026
+**Deadline:** Sunday March 8, 2026 1:00 PM
+**Synthesized from:** Researcher findings, code reviewer findings, sponsor track analysis, devil's advocate critique, gap analysis
+
+---
+
+## CONTEXT: Current State
+
+The core environment is solid: 3 agents, 3 enterprise systems, 4 attack types, reward functions, randomized attacker, security metrics engine, and a polished Gradio UI with 4 tabs and a cybersecurity theme. The codebase compiles and the trained vs untrained worker comparison shows meaningful score differences.
+
+**Three REQUIRED submission deliverables are NOT done:**
+1. HuggingFace Spaces deployment
+2. Google Colab training notebook
+3. Demo video on YouTube
+
+**Partner tracks targeted:** Fleet AI ($10K, Scalable Oversight) and Patronus AI ($10K, Schema Drift)
+
+---
+
+## 1. CRITICAL FIXES (Must Do -- Submission Fails Without These)
+
+### C1. Deploy to HuggingFace Spaces
+- **What:** Create HF Space, push code, verify it builds and runs
+- **Files:** `requirements.txt`, `README.md` (frontmatter), `app.py`
+- **Effort:** 30 min
+- **Impact:** BLOCKER -- no live URL = no submission
+- **Details:**
+  - Add `pandas>=2.0` to `requirements.txt` (missing, app.py imports it)
+  - Verify `gradio>=6.0.0` in requirements.txt matches README frontmatter `sdk_version: 6.9.0`
+  - Create Space at `huggingface.co/new-space`, SDK: Gradio, Hardware: CPU Basic
+  - Push with `git push hf main` or use `huggingface_hub.upload_folder()`
+  - Test all 4 tabs work on the live URL
+
+### C2. Create Colab Training Notebook
+- **What:** Create `training/colab_training.ipynb` with working GRPO pipeline
+- **Files:** New file: `training/colab_training.ipynb`
+- **Effort:** 60-90 min
+- **Impact:** BLOCKER -- submission requires "Minimal Training Script"
+- **Details:**
+  - Reuse logic from `train.py` (it has everything needed)
+  - Use `Qwen/Qwen2.5-0.5B-Instruct` (fits free Colab T4)
+  - Use Unsloth for model loading, vanilla TRL GRPOTrainer for training
+  - Must show: env verification, data collection, model loading, GRPO config, at least a few training steps
+  - If openenv-core fails on Colab Python version, bundle standalone env code
+  - Add markdown cells explaining each step, mention partner tracks
+
+### C3. Record Demo Video
+- **What:** 1-3 minute screen recording of Gradio app + voice/text narration
+- **Files:** N/A (external -- YouTube upload)
+- **Effort:** 30 min
+- **Impact:** BLOCKER -- submission requires YouTube demo video
+- **Details:**
+  - Show: episode replay (attack/adapt/flag cycle), untrained vs trained comparison, environment inspector
+  - Mention: 3-agent self-play, Fleet AI oversight, Patronus AI schema drift
+  - Keep simple -- QuickTime screen record, no fancy editing
+
+### C4. Verify Gradio App Launches Locally
+- **What:** Run `python app.py` and test all 4 tabs
+- **Files:** `app.py`, all imported modules
+- **Effort:** 15 min
+- **Impact:** HIGH -- if app crashes, HF Spaces will fail too
+- **Note:** `tasks/todo.md` shows this is UNCHECKED
+
+---
+
+## 2. HIGH-IMPACT IMPROVEMENTS (Should Do -- Directly Impress Judges)
+
+### H1. Improve Oversight Explanation Quality Scoring (Fleet AI Track)
+- **What:** Replace character-count explanation quality with structured quality scoring
+- **Files:** `sentinelops_arena/environment.py:441`, `sentinelops_arena/demo.py:302-327`
+- **Effort:** 20 min
+- **Impact:** HIGH for Fleet AI ($10K) -- current scoring is `min(len(explanation) / 100.0, 1.0)` which is embarrassingly simplistic. Fleet AI judge Nicolai Ouporov will notice.
+- **Details:**
+  - In `environment.py:441`, replace character-length heuristic with keyword-based quality scoring:
+    - +0.25 if explanation mentions the violation type (e.g., "policy violation", "social engineering")
+    - +0.25 if explanation references specific data (e.g., amount, field name, policy rule)
+    - +0.25 if explanation states the rule being violated (e.g., "max refund is $2000")
+    - +0.25 if explanation recommends corrective action
+  - In `demo.py` HeuristicOversight, improve the canned explanation strings to include specific data from the observation (e.g., "Worker issued refund exceeding policy max of $X. Current policy requires approval for amounts over $Y.")
+
+### H2. Add SLA Policy Drift to Ticketing (Patronus AI Track)
+- **What:** Allow the attacker to change SLA deadlines, not just refund policies
+- **Files:** `sentinelops_arena/systems/ticketing.py`, `sentinelops_arena/attacks.py`, `sentinelops_arena/demo.py`
+- **Effort:** 20 min
+- **Impact:** HIGH for Patronus AI ($10K) -- doubles the policy drift surface. Currently only billing has policy drift.
+- **Details:**
+  - Add `TicketingSystem.apply_policy_drift(changes)` in `ticketing.py` that modifies `self.sla_rules`
+  - In `attacks.py:_execute_policy_drift()`, route to ticketing system when target is TICKETING
+  - In `demo.py` RandomizedAttacker, add SLA policy drift options to `POLICY_DRIFT_CHANGES`
+  - Worker should call `get_current_policy("sla")` to discover changed SLA rules
+
+### H3. Add Oversight Metrics to Dashboard
+- **What:** Add oversight-specific metrics (explanation quality, detection accuracy) to the metrics engine and Gradio UI
+- **Files:** `sentinelops_arena/metrics.py`, `app.py`
+- **Effort:** 25 min
+- **Impact:** HIGH for Fleet AI ($10K) -- currently NO oversight-specific metrics exist in the dashboard
+- **Details:**
+  - In `metrics.py`, add to `compute_episode_metrics()`:
+    - `oversight_accuracy`: correct flags + correct approvals / total oversight decisions
+    - `avg_explanation_quality`: average explanation quality score across all oversight decisions
+  - Add a new metric card for oversight accuracy in `format_metrics_html()`
+  - This makes the Fleet AI story visible in the demo
+
+### H4. Add Drift-Specific Metrics
+- **What:** Add drift adaptation metrics to the metrics engine
+- **Files:** `sentinelops_arena/metrics.py`
+- **Effort:** 15 min
+- **Impact:** HIGH for Patronus AI ($10K) -- makes drift adaptation visible and measurable
+- **Details:**
+  - Add to `compute_episode_metrics()`:
+    - `drift_events`: total schema + policy drift attacks
+    - `drifts_detected`: number of times worker called get_schema/get_current_policy after a drift
+    - `avg_drift_recovery_ticks`: average ticks between drift and worker's first defensive action
+  - Add metric card for "Drift Adaptation" in `format_metrics_html()`
+
+### H5. Improve HeuristicOversight Explanations
+- **What:** Make the oversight agent's explanations reference specific data from the observation
+- **Files:** `sentinelops_arena/demo.py:302-327`
+- **Effort:** 15 min
+- **Impact:** MEDIUM-HIGH for Fleet AI -- judges will see these in the replay log
+- **Details:**
+  - Pass `obs` to `HeuristicOversight.act()` (currently only uses `obs.last_action_result`)
+  - Generate explanations like: "Worker action at tick {tick}: {action_type} resulted in error. The error '{error_msg}' suggests schema drift may have occurred. Recommended: call get_schema() to discover new field names."
+  - For social engineering: "Worker followed suspicious instructions containing override language. The message '{first 50 chars}' appears to be a social engineering attack. Flagging as critical violation."
+  - For policy violations: "Refund of ${amount} exceeds current policy maximum of ${max}. Policy was last updated at tick {last_policy_change}."
+
+---
+
+## 3. QUICK WINS (Do If Time Allows -- Small Effort, Good Impression)
+
+### Q1. Fix Documentation Inconsistencies
+- **What:** Fix mismatches between spec doc, README, and actual code
+- **Files:** `README.md`, `pyproject.toml`
+- **Effort:** 10 min
+- **Impact:** Prevents judges from noticing sloppy details
+- **Details:**
+  - Set `gradio>=6.0.0` consistently in pyproject.toml (currently says >=5.0.0)
+  - Fix README project structure to match reality (remove `mcp_tools.py` listing)
+  - Do NOT touch SENTINELOPS_ARENA.md (it's a spec doc, acceptable to be aspirational)
+
+### Q2. Add Links to About Tab
+- **What:** Once Colab notebook and video exist, add links to the About tab
+- **Files:** `app.py` (About tab section)
+- **Effort:** 5 min
+- **Impact:** Makes it easy for judges to find all submission artifacts
+
+### Q3. Clean Up Vestigial Files
+- **What:** Remove or gitignore `hackathon_env/` directory
+- **Files:** `.gitignore`, possibly `hackathon_env/`
+- **Effort:** 5 min
+- **Impact:** Prevents judge confusion
+
+### Q4. Add Billing Schema Drift Support
+- **What:** Allow schema drift attacks against billing system too
+- **Files:** `sentinelops_arena/systems/billing.py`
+- **Effort:** 10 min
+- **Impact:** Strengthens Patronus AI story -- all 3 systems support schema drift
+- **Details:**
+  - Add `BillingSystem.apply_schema_drift(old_field, new_field)` mirroring CRM pattern
+  - Add `_field_map` dict and `_apply_field_map` method to BillingSystem
+  - Update `attacks.py` `VALID_TARGETS` for schema_drift to include BILLING
+
+---
+
+## 4. SKIP LIST (Not Worth the Time)
+
+| Item | Reason |
+|------|--------|
+| Compound attacks (2-3 simultaneous) | 2+ hours, marginal judge impact |
+| Compliance drift (new required fields) | 1+ hours, nice but not critical |
+| A2A protocol | Already marked "Cut" in spec, not in submission requirements |
+| Docker support | HF Spaces uses Gradio SDK directly |
+| MCP-X gateway demo | MCP tools in environment.py are sufficient |
+| Full GRPO convergence | Pipeline working is enough -- convergence not required |
+| Real datetime-based SLA | Tick-based is fine for demo |
+| Multi-GPU training | Overkill for hackathon |
+| Refactoring codebase | No judge impact, waste of time |
+
+---
+
+## EXECUTION ORDER (Recommended)
+
+**Phase 1 (0:00 - 0:15): Verify and fix basics**
+1. C4: Verify Gradio app launches locally
+2. Q1: Fix requirements.txt (add pandas) and pyproject.toml consistency
+
+**Phase 2 (0:15 - 1:00): High-impact code improvements**
+3. H1: Improve oversight explanation quality scoring (20 min)
+4. H2: Add SLA policy drift to ticketing (20 min)
+5. H5: Improve HeuristicOversight explanations (15 min)
+
+**Phase 3 (1:00 - 1:30): Metrics improvements**
+6. H3: Add oversight metrics to dashboard (25 min)
+7. H4: Add drift-specific metrics (15 min)
+
+**Phase 4 (1:30 - 2:00): Deployment**
+8. C1: Deploy to HuggingFace Spaces (30 min)
+
+**Phase 5 (2:00 - 3:15): Required deliverables**
+9. C2: Create Colab training notebook (75 min)
+
+**Phase 6 (3:15 - 3:45): Video and submission**
+10. C3: Record demo video (30 min)
+
+**Phase 7 (3:45 - 4:00): Final polish**
+11. Q2: Add links to About tab (5 min)
+12. Q3: Clean up vestigial files (5 min)
+13. Final push and submit (5 min)
+
+---
+
+## KEY JUDGE CONSIDERATIONS
+
+- **Nicolai Ouporov (Fleet AI):** Cares about scalable oversight. Will check: Does the oversight agent actually explain violations well? Is explanation quality tracked? Does training improve oversight?
+- **Darshan Deshpande (Patronus AI):** Cares about schema drift. Will check: How many drift types? Does the worker adapt? Is drift visible in the UI?
+- **Daniel Han (Unsloth):** Cares about Unsloth/TRL integration. Will check: Does the Colab notebook use Unsloth correctly? Does training actually work?
+- **Sanyam Bhutani (Meta):** Cares about OpenEnv quality. Will check: Is the environment well-structured? Does step/reset/state work properly?
+- **Benjamin Burtenshaw (HuggingFace):** Cares about Hub deployment. Will check: Is the HF Space functional and polished?

training/colab_training.ipynb

@@ -0,0 +1,225 @@
+{
+  "nbformat": 4,
+  "nbformat_minor": 0,
+  "metadata": {
+    "colab": {
+      "provenance": [],
+      "gpuType": "T4"
+    },
+    "kernelspec": {
+      "name": "python3",
+      "display_name": "Python 3"
+    },
+    "language_info": {
+      "name": "python"
+    }
+  },
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "source": [
+        "# SentinelOps Arena \u2014 GRPO Training with Unsloth\n",
+        "\n",
+        "This notebook demonstrates how to train the **Worker Agent** using GRPO (Group Relative Policy Optimization) on the SentinelOps Arena environment.\n",
+        "\n",
+        "SentinelOps Arena is a multi-agent self-play RL environment for enterprise security training built on OpenEnv. We are targeting the **Fleet AI (Scalable Oversight)** and **Patronus AI (Schema Drift)** tracks."
+      ],
+      "metadata": {
+        "id": "intro"
+      }
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## 1. Setup Environment"
+      ],
+      "metadata": {
+        "id": "setup-header"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "install-deps"
+      },
+      "outputs": [],
+      "source": [
+        "!pip install \"openenv-core[core]>=0.2.0\" mcp fastmcp pydantic pandas\n",
+        "!pip install \"unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git\"\n",
+        "!pip install --no-deps \"trl<0.9.0\" peft accelerate bitsandbytes"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "clone-repo"
+      },
+      "outputs": [],
+      "source": [
+        "import os\n",
+        "if not os.path.exists(\"NexusEnv\"):\n",
+        "    !git clone https://github.com/nihalnihalani/NexusEnv.git\n",
+        "import sys\n",
+        "sys.path.append(\"/content/NexusEnv\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## 2. Collect Training Data via Self-Play\n",
+        "\n",
+        "We run the environment using our heuristic agents to generate the initial \"prompts\" that the Worker agent will face during training."
+      ],
+      "metadata": {
+        "id": "collect-header"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "collect-data"
+      },
+      "outputs": [],
+      "source": [
+        "import json\n",
+        "from datasets import Dataset\n",
+        "from NexusEnv.train import build_training_dataset, WORKER_SYSTEM_PROMPT\n",
+        "\n",
+        "NUM_EPISODES = 5\n",
+        "print(f\"Collecting training data from {NUM_EPISODES} episodes...\")\n",
+        "dataset_raw = build_training_dataset(num_episodes=NUM_EPISODES, target_agent=\"worker\")\n",
+        "\n",
+        "prompts = []\n",
+        "for d in dataset_raw:\n",
+        "    messages = [\n",
+        "        {\"role\": \"system\", \"content\": WORKER_SYSTEM_PROMPT},\n",
+        "        {\"role\": \"user\", \"content\": d[\"prompt\"]},\n",
+        "    ]\n",
+        "    prompts.append(messages)\n",
+        "\n",
+        "train_dataset = Dataset.from_dict({\"prompt\": prompts})\n",
+        "print(f\"Dataset generated with {len(train_dataset)} examples.\")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## 3. Load Model with Unsloth\n",
+        "\n",
+        "We use `Qwen/Qwen2.5-0.5B-Instruct` as it fits comfortably in a free Colab T4 GPU."
+      ],
+      "metadata": {
+        "id": "load-header"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "load-model"
+      },
+      "outputs": [],
+      "source": [
+        "from unsloth import FastLanguageModel\n",
+        "\n",
+        "model_name = \"unsloth/Qwen2.5-0.5B-Instruct\"\n",
+        "\n",
+        "model, tokenizer = FastLanguageModel.from_pretrained(\n",
+        "    model_name=model_name,\n",
+        "    max_seq_length=2048,\n",
+        "    load_in_4bit=True,\n",
+        "    fast_inference=True, # Enable vLLM fast inference\n",
+        ")\n",
+        "\n",
+        "model = FastLanguageModel.get_peft_model(\n",
+        "    model,\n",
+        "    r=16,\n",
+        "    target_modules=[\n",
+        "        \"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",
+        "        \"gate_proj\", \"up_proj\", \"down_proj\",\n",
+        "    ],\n",
+        "    lora_alpha=16,\n",
+        "    lora_dropout=0,\n",
+        "    bias=\"none\",\n",
+        "    use_gradient_checkpointing=\"unsloth\",\n",
+        ")"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## 4. GRPO Training\n",
+        "\n",
+        "We set up the GRPO configuration and launch the training process."
+      ],
+      "metadata": {
+        "id": "train-header"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "train"
+      },
+      "outputs": [],
+      "source": [
+        "from trl import GRPOConfig, GRPOTrainer\n",
+        "from NexusEnv.train import make_reward_function\n",
+        "\n",
+        "reward_fn = make_reward_function(\"worker\")\n",
+        "\n",
+        "grpo_config = GRPOConfig(\n",
+        "    output_dir=\"./sentinelops-grpo-worker\",\n",
+        "    num_train_epochs=1,\n",
+        "    per_device_train_batch_size=2,\n",
+        "    gradient_accumulation_steps=4,\n",
+        "    num_generations=4,\n",
+        "    max_completion_length=256,\n",
+        "    max_prompt_length=512,\n",
+        "    learning_rate=5e-6,\n",
+        "    logging_steps=1,\n",
+        "    report_to=\"none\",\n",
+        ")\n",
+        "\n",
+        "trainer = GRPOTrainer(\n",
+        "    model=model,\n",
+        "    processing_class=tokenizer,\n",
+        "    reward_funcs=[reward_fn],\n",
+        "    args=grpo_config,\n",
+        "    train_dataset=train_dataset,\n",
+        ")\n",
+        "\n",
+        "trainer.train()"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## 5. Save the Trained Model\n",
+        "\n",
+        "Finally, we save our GRPO-trained LoRA weights."
+      ],
+      "metadata": {
+        "id": "save-header"
+      }
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "id": "save"
+      },
+      "outputs": [],
+      "source": [
+        "output_dir = \"./sentinelops-grpo-worker\"\n",
+        "trainer.save_model(output_dir)\n",
+        "tokenizer.save_pretrained(output_dir)\n",
+        "print(\"Model saved successfully!\")"
+      ]
+    }
+  ]
+}