OpenSOC v1

.gitignore

@@ -0,0 +1,19 @@
+__pycache__/
+*.pyc
+*.pyo
+.pytest_cache/
+.ruff_cache/
+.venv/
+venv/
+.env
+.DS_Store
+*.egg-info/
+build/
+dist/
+checkpoints/
+runs/
+wandb/
+*.ckpt
+*.bin
+*.safetensors
+.ipynb_checkpoints/

DEPLOY.md

@@ -0,0 +1,101 @@
+# Deploying OpenSOC to Hugging Face Spaces
+
+This is the one-time deployment recipe.  The same Space serves both the
+OpenEnv API (consumed by judge bots and `OpenSOCClient`) **and** a
+Gradio "before vs after" UI at `/demo` for human reviewers.
+
+## 1. Local sanity check
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+python server.py &
+sleep 2
+curl -s http://localhost:7860/health | jq .
+curl -s -X POST 'http://localhost:7860/reset?task=stage1_basic&mode=defender_only' | jq .
+curl -s -I http://localhost:7860/demo  | head -1   # should be 200 OK
+kill %1
+```
+
+## 2. Build the Docker image locally
+
+```bash
+docker build -t opensoc:latest .
+docker run -p 7860:7860 opensoc:latest
+# in another shell:
+curl -s http://localhost:7860/tasks | jq .
+open http://localhost:7860/demo
+```
+
+## 3. Push to Hugging Face
+
+The simplest path is via `huggingface-cli`; the second is a one-shot
+script that does the same thing.
+
+### One-shot
+
+```bash
+export HF_USER=<your-username>
+huggingface-cli login   # browser-based PAT login
+bash scripts/deploy_to_hf.sh
+```
+
+### Manual (equivalent)
+
+```bash
+huggingface-cli login
+huggingface-cli repo create opensoc-env --type space --space-sdk docker
+# Use SPACE_README.md as the Space's README so HF picks up the docker SDK config:
+cp SPACE_README.md /tmp/SPACE_README.md  # save a copy
+git checkout -b space-deploy
+cp SPACE_README.md README.md             # or prepend SPACE_README front-matter to README
+git add README.md && git commit -m "Space metadata header"
+git remote add space https://huggingface.co/spaces/$HF_USER/opensoc-env
+git push space space-deploy:main
+git checkout main && git checkout README.md
+```
+
+## 4. Verify the deployed Space
+
+```bash
+export OPENSOC_URL=https://<your-username>-opensoc-env.hf.space
+python -c "
+from client import OpenSOCClient
+c = OpenSOCClient(base_url='$OPENSOC_URL')
+print(c.health())
+print(c.tasks())
+obs = c.reset(task='stage1_basic', mode='defender_only', seed=1)
+print('first log id:', obs['log_window'][0]['log_id'])
+"
+# And visually:
+open $OPENSOC_URL/demo
+```
+
+`/demo` reads `data/demo_examples.json`.  If you deployed before running
+the GPU pipeline, the file holds the *placeholder* before-vs-after pairs
+(always-dismiss vs verifier-oracle).  Re-run `python -m eval.bake_demo`
+on a GPU host (no `--placeholder`) and re-push to overwrite with real
+trained-model outputs.
+
+## 5. (Optional) Run the eval harness against the live Space
+
+```bash
+# Pure-CPU smoke run (no Unsloth required):
+python -m eval.eval --smoke-only --holdout data/holdout.jsonl
+```
+
+## OpenEnv hackathon checklist
+
+- [x] `openenv.yaml` manifest with `endpoints.demo: GET /demo`
+- [x] gym-style API: `reset` / `step` / `state` (+ `grade`, `tasks`, `health`)
+- [x] non-reserved tool names (`craft_incident`, `submit_triage`)
+- [x] FastAPI app exposed on port 7860 inside the container
+- [x] Gradio UI mounted at `/demo` for the storytelling deliverable
+- [x] Dockerfile suitable for Hugging Face Spaces (`sdk: docker`)
+- [x] Client / server separation (`client/opensoc_client.py` is HTTP-only)
+- [x] Frozen 200-incident eval set committed (`data/holdout.jsonl`)
+- [x] 600-example SFT dataset committed (`data/sft_train.jsonl`)
+- [x] 50 pre-baked demo pairs committed (`data/demo_examples.json`)
+- [x] GRPO Colab/HF Jupyter notebook (`train_grpo.ipynb`) + one-shot
+      `scripts/run_full_pipeline.sh`
+- [x] Pytest suite — 93 tests, all green

Dockerfile

@@ -0,0 +1,31 @@
+FROM python:3.11-slim
+
+ENV PORT=7860
+ENV PYTHONUNBUFFERED=1
+ENV PYTHONDONTWRITEBYTECODE=1
+ENV HF_HOME=/tmp/hf_cache
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY app_runtime.py .
+COPY env.py .
+COPY schema.py .
+COPY generator.py .
+COPY verifier.py .
+COPY rubric.py .
+COPY server.py .
+COPY demo_app.py .
+COPY demo_data.py .
+COPY openenv.yaml .
+COPY tasks/ tasks/
+COPY client/ client/
+COPY train/ train/
+COPY eval/ eval/
+COPY data/ data/
+
+EXPOSE 7860
+
+CMD ["python", "server.py"]

README.md

@@ -0,0 +1,249 @@
+# OpenSOC: Self-Play SOC Triage Environment
+
+> An **OpenEnv** environment for training cybersecurity defender LLMs against an attacker LLM that auto-generates novel incidents. Built for the OpenEnv Hackathon, April 2026.
+
+Humans cannot watch every alert in a Security Operations Center 24/7, and as stronger generative models start writing exploits and phishing at industrial scale that gap only widens. **OpenSOC** is an environment where a defender LLM learns to triage attacks generated by another LLM in a self-play loop. The trick is **RLVR**: triage ground truth is computed by a deterministic schema-side verifier from the *structured* incident parameters — never from any text the attacker writes — so neither side can hack the reward.
+
+## Try it
+
+| Link | What it is |
+| --- | --- |
+| **HF Space** — [`<USER>-opensoc-env.hf.space`](https://huggingface.co/spaces/REPLACE_USER/opensoc-env) | Deployed env. OpenEnv judge can hit `/reset` `/step` `/state` `/grade`. |
+| **Live `/demo`** — [`<USER>-opensoc-env.hf.space/demo`](https://REPLACE_USER-opensoc-env.hf.space/demo) | Gradio "before vs after" UI. Click **Next incident** to compare baseline vs trained. |
+| **Walkthrough video** (90s) — [`youtu.be/<UNLISTED>`](https://youtu.be/REPLACE_VIDEO) | One-take demo + headline numbers. Script: [`docs/video_script.md`](docs/video_script.md). |
+| **Mini-blog** — [`huggingface.co/blog/<USER>/opensoc-rlvr-soc-triage`](https://huggingface.co/blog/REPLACE_USER/opensoc-rlvr-soc-triage) | ~600-word write-up. Source: [`docs/blog.md`](docs/blog.md). |
+| **Slide deck** — [`docs/slides.pdf`](docs/slides.pdf) | 5 slides; problem → env → results → demo. |
+
+> *Replace the four `REPLACE_*` placeholders above after deploy + recording. The slide PDF auto-rebuilds from `docs/build_slides.py`.*
+
+## Table of contents
+
+1. [Architecture](#architecture)
+2. [Why the reward cannot be hacked](#why-the-reward-cannot-be-hacked)
+3. [Action space and reward](#action-space-and-reward)
+4. [Run locally](#run-locally)
+5. [Run the training pipeline](#run-the-training-pipeline)
+6. [Headline results](#headline-results)
+7. [Deploy to Hugging Face Spaces](#deploy-to-hugging-face-spaces)
+8. [Repo map](#repo-map)
+9. [Submission deliverables](#submission-deliverables)
+
+## Build status
+
+| Build artifact | Status |
+| --- | --- |
+| Pure-python env (`OpenSOCEnv`, FastAPI) | shipped |
+| Verifier + plausibility checker | shipped, 17-test adversarial suite |
+| Rubric (defender + attacker rewards) | shipped, anti-hack regression tests |
+| 600-example SFT dataset (`data/sft_train.jsonl`) | shipped |
+| 200-incident frozen hold-out (`data/holdout.jsonl`) | shipped |
+| GRPO training notebook (`train_grpo.ipynb`) + one-shot script | shipped (HF Jupyter L4) |
+| Gradio "before vs after" UI mounted on the same Space | shipped at `/demo` |
+| 50 pre-baked demo pairs (`data/demo_examples.json`) | placeholder shipped; refresh after GPU run |
+| Eval harness + plotters (`eval/`) | shipped |
+| Pytest suite | **93 tests**, all green |
+
+## Architecture
+
+```mermaid
+flowchart LR
+  Defender[Defender LLM trainee]
+  Attacker[Attacker LLM trainee]
+  Env[OpenSOC FastAPI Environment]
+  Verifier[Deterministic verifier + plausibility check]
+  Defender -->|submit_triage| Env
+  Attacker -->|craft_incident| Env
+  Env -->|observation reward| Defender
+  Env -->|attacker reward| Attacker
+  Env --> Verifier
+  Verifier -->|ground truth label| Env
+```
+
+An episode has exactly two turns: attacker proposes incident params → env validates them and materializes a SIEM-style alert + log window → defender submits a triage action.  The verifier computes the ground-truth action from the *events alone* and scores both sides — the attacker's free-text narrative is never read by the labeler.
+
+In `defender_only` mode (used for SFT, eval, smoke tests, and the `/demo` UI) the env auto-generates the incident from `tasks/registry.py` and skips straight to the defender turn.
+
+## Why the reward cannot be hacked
+
+1. The verifier is a transparent rule set in `verifier.compute_ground_truth(params)`; the *only* inputs are the structured events.  The attacker's `narrative` and even its self-claimed `target_label` are ignored.
+2. The plausibility checker (`verifier.check_plausibility(params)`) refuses incoherent stories — for example, a "data exfiltration" claim with a purely-internal destination, or a `lolbin_use` event with no `process` field.  The attacker's reward is gated on plausibility passing.
+3. Schema-violation incidents floor attacker reward at -0.5, so trying to short-circuit pydantic's validators is strictly worse than playing along.
+
+The anti-hack invariants are pinned in [`tests/test_verifier.py`](tests/test_verifier.py) and [`tests/test_rubric.py`](tests/test_rubric.py).
+
+## Action space and reward
+
+Tool names are deliberately **non-reserved** — there is no `reset`/`step`/`state`/`close` clash with the OpenEnv `MCPEnvironment` reserved-name list.
+
+```yaml
+action_space:
+  craft_incident:
+    target_label: dismiss | monitor | quarantine_host | block_ip | escalate
+    category:     malware_execution | c2_beacon | data_exfiltration | ...
+    events:       [ { event_type, fields, timestamp, log_id }, ... ]
+    narrative:    string         # ignored by the verifier
+  submit_triage:
+    action:       <one of the five triage actions>
+    cited_log_id: <id of the log line that drove the decision>
+    rationale:    short string
+```
+
+- **Defender**: +1 correct, −1 missed-malicious, −0.3 over-react on benign, −0.05 unnecessary escalate, +0.1 bonus for citing the right triggering log id, −0.1 floor for format violation.
+- **Attacker**: +1 iff defender wrong AND incident plausible, −0.5 if schema validation fails, +0.2 novelty bonus, 0 for gibberish.
+
+Full breakdown: [openenv.yaml](openenv.yaml) and [rubric.py](rubric.py).
+
+## Run locally
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+python server.py    # serves on :7860
+```
+
+Smoke test from another shell:
+
+```bash
+curl -s http://localhost:7860/health | jq .
+curl -s -X POST 'http://localhost:7860/reset?task=stage1_basic&mode=defender_only' | jq .
+curl -s -X POST 'http://localhost:7860/step?task=stage1_basic&mode=defender_only' \
+     -H 'content-type: application/json' \
+     -d '{"submit_triage": {"action": "monitor", "cited_log_id": "L1-0", "rationale": "smoke"}}' | jq .
+open http://localhost:7860/demo   # Gradio before-vs-after UI
+```
+
+Run the test suite (CPU only, no GPU deps):
+
+```bash
+pytest -q   # 93 passed
+```
+
+Or via the bundled Python client:
+
+```python
+from client import OpenSOCClient
+c = OpenSOCClient()
+obs = c.reset(task="stage1_basic", mode="defender_only", seed=1)
+result = c.step({"submit_triage": {"action": "monitor", "cited_log_id": "L1-0", "rationale": "ok"}},
+                task="stage1_basic", mode="defender_only", seed=1)
+print(result)
+```
+
+## Run the training pipeline
+
+Full end-to-end procedure: **[TRAIN.md](TRAIN.md)**.  TL;DR — on an HF Jupyter L4 (~$3 of credits, ~3.5h wall time):
+
+```bash
+bash scripts/run_full_pipeline.sh
+```
+
+Or step-by-step inside [`train_grpo.ipynb`](train_grpo.ipynb):
+
+1. SFT warm-start (~12 min) — pushes P(format-OK) from ~0% to ~95%.
+2. GRPO curriculum across 4 stages (~3h) — verifier-grounded reward, group size 8.
+3. Eval on the frozen 200-incident hold-out (~5 min).
+4. `eval.plot_results` + `eval.plot_training` render four PNGs.
+5. `eval.bake_demo` writes 50 before-vs-after pairs to `data/demo_examples.json` for the Gradio UI.
+
+## Headline results
+
+> *Plots below are auto-generated; the placeholder versions are committed today (always-dismiss vs verifier-oracle) so the README never has broken images.  Re-run the pipeline above to overwrite with real numbers.*
+
+### Dismiss-on-malicious (the cardinal failure mode)
+
+![dismiss-on-malicious by model](eval/results/bar_dismiss_on_malicious.png)
+
+### Macro F1 across 200-incident hold-out
+
+![macro F1 by model](eval/results/bar_macro_f1.png)
+
+### Confusion matrices
+
+| Baseline (zero-shot Qwen2.5-3B) | OpenSOC (after GRPO) |
+| --- | --- |
+| ![baseline confusion](eval/results/confusion_always_dismiss.png) | ![trained confusion](eval/results/confusion_verifier_oracle.png) |
+
+*(Filenames `confusion_always_dismiss.png` and `confusion_verifier_oracle.png` get replaced by `confusion_baseline_zero_shot.png` and `confusion_opensoc_grpo.png` after the GPU eval run.)*
+
+### Reward across the curriculum
+
+![training reward curves](eval/results/training_curves.png)
+
+| Model | Accuracy | Macro F1 | Dismiss-on-malicious | Over-react |
+| --- | ---: | ---: | ---: | ---: |
+| `always_dismiss` (floor)      | 0.13 | 0.05 | **1.00** | 0.00 |
+| `baseline_zero_shot` (Qwen2.5-3B) | _GPU run_ | _GPU run_ | _GPU run_ | _GPU run_ |
+| `opensoc_grpo` (after training) | _GPU run_ | _GPU run_ | _GPU run_ | _GPU run_ |
+| `verifier_oracle` (ceiling)   | 1.00 | 1.00 | 0.00 | 0.00 |
+
+## Deploy to Hugging Face Spaces
+
+Full recipe: [DEPLOY.md](DEPLOY.md).  The fast version, after `huggingface-cli login`:
+
+```bash
+export HF_USER=<your-username>
+bash scripts/deploy_to_hf.sh
+# Build takes ~5 minutes; then:
+open https://${HF_USER}-opensoc-env.hf.space/demo
+```
+
+The Space runs FastAPI + Gradio in a single container.  `/reset`, `/step`, `/state`, `/grade`, `/tasks`, `/health` continue to work for the OpenEnv judge bot; `/demo` is the human-readable UI.
+
+## Repo map
+
+| File / dir | Purpose |
+| --- | --- |
+| `openenv.yaml` | OpenEnv manifest (tasks, action space, reward range, endpoints) |
+| `schema.py` | Incident / event / action schema with strict validators |
+| `generator.py` | Materializes incidents for `defender_only` mode (eval, SFT) |
+| `verifier.py` | Deterministic ground-truth labeler + plausibility checker |
+| `rubric.py` | Layered defender + attacker reward functions |
+| `env.py` | Two-role `OpenSOCEnv` (`reset` / `step` / `state` / `grade`) |
+| `app_runtime.py` | FastAPI app exposing the OpenEnv API |
+| `demo_app.py` | Gradio Blocks app mounted at `/demo` |
+| `demo_data.py` | Pure-python helpers for the demo UI |
+| `server.py` | Container entry point — imports `demo_app` then starts uvicorn |
+| `tasks/registry.py` | Curriculum stages: `stage1_basic` → `stage4_adversarial` |
+| `client/` | Thin HTTP client (server-internals-free) |
+| `train/` | SFT warm-start + GRPO loop + reusable prompt format |
+| `eval/` | Hold-out generator, metrics, eval driver, plot renderers, `bake_demo` |
+| `scripts/run_full_pipeline.sh` | One-shot training + eval + bake-demo |
+| `scripts/deploy_to_hf.sh` | One-shot HF Space push |
+| `docs/` | Blog post, video script, slide deck builder |
+| `tests/` | Pytest suite (93 tests, anti-hack regressions included) |
+
+## Submission deliverables
+
+Mapped to the four judging criteria:
+
+| Criterion | Weight | Where it lives |
+| --- | ---: | --- |
+| Environment Innovation | 40% | `openenv.yaml`, `schema.py`, `verifier.py`, `env.py`, this README's *Architecture* and *Why the reward cannot be hacked* sections |
+| Storytelling & Presentation | 30% | `/demo` Gradio UI + 90s video + HF blog + 5-slide deck (`docs/slides.pdf`) |
+| Showing Improvement in Rewards | 20% | `eval/results/*.png` (training curves + confusion + headline bar) embedded above |
+| Reward & Training Pipeline | 10% | `rubric.py` + 93-test anti-hack suite + `train_grpo.ipynb` + `scripts/run_full_pipeline.sh` |
+
+Submission checklist:
+
+- [x] OpenEnv-compatible env (gym-style API, manifest, non-reserved tool names)
+- [x] Deterministic RLVR verifier + plausibility checker
+- [x] Layered defender + attacker reward
+- [x] SFT warm-start dataset (committed)
+- [x] Frozen 200-incident hold-out (committed)
+- [x] GRPO curriculum notebook + one-shot training script
+- [x] Eval harness + plotters
+- [x] Pytest suite (93 tests, anti-hack regressions included)
+- [x] Gradio `/demo` UI mounted on the same Space (free-CPU-tier compatible)
+- [x] 5-slide PDF deck (`docs/slides.pdf`)
+- [x] Blog post draft (`docs/blog.md`)
+- [x] Video script (`docs/video_script.md`)
+- [ ] HF Space pushed (run `bash scripts/deploy_to_hf.sh`)
+- [ ] Trained adapter pushed (run the GPU pipeline; commit the resulting checkpoint)
+- [ ] Real demo data baked (re-run `python -m eval.bake_demo` post-training)
+- [ ] Video recorded + uploaded as unlisted (script in `docs/video_script.md`)
+- [ ] Blog post published on HF (source in `docs/blog.md`)
+- [ ] All four `REPLACE_*` placeholders at the top filled in
+
+## License
+
+BSD-3-Clause.

SPACE_README.md

@@ -0,0 +1,22 @@
+---
+title: OpenSOC SOC Triage Env
+emoji: shield
+colorFrom: indigo
+colorTo: red
+sdk: docker
+app_port: 7860
+pinned: false
+license: bsd-3-clause
+tags:
+  - openenv
+  - cybersecurity
+  - rlvr
+  - self-play
+---
+
+This file is the Hugging Face Spaces metadata header.  When pushing to a
+Space (`git push space main`), copy this file to the Space repo as `README.md`
+(or merge the front-matter into the existing README's first lines).
+
+The repository's main `README.md` provides the full project description and
+usage instructions.

TRAIN.md

@@ -0,0 +1,72 @@
+# Training OpenSOC end-to-end
+
+Total compute budget (HF Jupyter Notebook L4 @ $0.80/h):
+
+| Step                                | Wall time | $ on L4 |
+| ----------------------------------- | --------- | ------- |
+| SFT warm-start                      | ~12 min   | ~$0.16  |
+| GRPO curriculum (4 stages × 200 st) | ~3 h      | ~$2.40  |
+| Eval (200 hold-out incidents)       | ~5 min    | ~$0.07  |
+| Bake demo (50 incidents × 2 models) | ~3 min    | ~$0.04  |
+| **Total**                           | **~3.5h** | **~$2.7** |
+
+Comfortably within the $30 HF credit budget — leaves ~$25 for ablations, retries, or moving to A10G.
+
+## Recommended target: Hugging Face Jupyter Notebooks
+
+1. Push this repo (or fork) to `huggingface.co/<you>/opensoc-env` so the
+   notebook can `git clone` it.
+2. From <https://huggingface.co/notebooks/new>, pick **L4 (24GB)** and
+   the **pytorch-cuda** image.
+3. Open `train_grpo.ipynb` from the cloned repo and run cells top-to-bottom.
+
+The notebook is idempotent — you can pause/resume between any two cells
+and HF only bills for attached-GPU minutes.
+
+## One-shot script alternative
+
+If you prefer a single shell command (works on any GPU host with CUDA):
+
+```bash
+git clone https://huggingface.co/<you>/opensoc-env && cd opensoc-env
+bash scripts/run_full_pipeline.sh
+```
+
+## Fallbacks
+
+- **L4 unavailable in your region** → use **A10G** (~$1.05/h, +30% cost).
+- **Only Colab Pro T4 available** → drop `--num-generations` from 8 to 4
+  and increase `--steps-per-stage` from 200 to 300; total cost similar.
+- **GRPO reward is flat at zero** → you skipped the SFT warm-start; with
+  no SFT, the format-violation penalty dominates and there's no signal.
+- **Adapter file too large for `git push`** → use `huggingface-cli upload`
+  with LFS, or push the adapter to a separate `*-defender-grpo` model
+  repo and load it by reference at deploy time.
+
+## What gets produced
+
+After a successful run, the following are added to the repo:
+
+```
+checkpoints/
+  defender_sft_adapter/                       # warm-started LoRA adapter
+  defender_grpo/
+    stage1_basic/{adapter,training_log.jsonl,runs/}
+    stage2_multi/...
+    stage3_mixed/...
+    stage4_adversarial/{adapter, ...}         # final RL-trained adapter
+data/
+  demo_examples.json                          # 50 before-vs-after pairs
+eval/results/
+  summary.json
+  bar_dismiss_on_malicious.png                # headline plot
+  bar_macro_f1.png
+  confusion_baseline_zero_shot.png
+  confusion_opensoc_grpo.png
+  training_curves.png                         # reward across stages
+  training_kl_loss.png
+```
+
+`scripts/run_full_pipeline.sh` is the canonical end-to-end command; the
+Jupyter notebook is the same pipeline but cell-by-cell so you can
+inspect intermediate outputs.

app_runtime.py

@@ -0,0 +1,176 @@
+"""FastAPI application module for OpenSOC, mountable from server.py.
+
+Endpoints follow the OpenEnv conventions plus a lightweight `/grade`:
+
+  POST /reset?task=<stage>&mode=<self_play|defender_only>&seed=<n>
+  POST /step?task=<stage>&mode=...&seed=<n>          (body: Action)
+  GET  /state?task=<stage>&mode=...&seed=<n>
+  POST /grade?task=<stage>&mode=...&seed=<n>
+  GET  /tasks
+  GET  /health
+
+Per-(task, mode, seed) env instances are cached in a process-local dict so
+multiple concurrent clients can share the FastAPI process without stepping
+on each other's episodes.
+
+This module does NOT inherit from openenv-core's MCPEnvironment because the
+`craft_incident`/`submit_triage` action surface is non-MCP (single-action
+unions are simpler for GRPO rollouts).  Tool names are deliberately
+non-reserved so an MCPEnvironment wrapper can be added later if a team
+wants to expose the env over MCP transports.
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Any, Dict, Optional
+
+from fastapi import FastAPI, HTTPException, Query
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel
+
+from env import Action, Observation, OpenSOCEnv
+
+
+app = FastAPI(
+    title="OpenSOC",
+    description="Self-play SOC triage OpenEnv environment for cybersecurity defender LLMs.",
+    version="1.0.0",
+)
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+_envs: Dict[str, OpenSOCEnv] = {}
+
+
+def _env_key(task: str, mode: str, seed: int) -> str:
+    return f"{task}::{mode}::{seed}"
+
+
+def _get_env(task: str, mode: str, seed: int) -> OpenSOCEnv:
+    key = _env_key(task, mode, seed)
+    if key not in _envs:
+        try:
+            _envs[key] = OpenSOCEnv(task_id=task, mode=mode, seed=seed)  # type: ignore[arg-type]
+        except ValueError as exc:
+            raise HTTPException(status_code=400, detail=str(exc)) from exc
+    return _envs[key]
+
+
+# ---------------------------------------------------------------------------
+# Response models
+# ---------------------------------------------------------------------------
+
+class StepResult(BaseModel):
+    observation: Observation
+    reward: float
+    done: bool
+    info: Dict[str, Any]
+
+
+class GradeResult(BaseModel):
+    task: str
+    mode: str
+    score: float
+    defender_reward: Optional[float]
+    attacker_reward: Optional[float]
+    ground_truth: Optional[str]
+    plausible: Optional[bool]
+    schema_violation: bool
+
+
+# ---------------------------------------------------------------------------
+# Endpoints
+# ---------------------------------------------------------------------------
+
+@app.post("/reset", response_model=Observation)
+def reset(
+    task: str = Query("stage1_basic", description="Curriculum stage id."),
+    mode: str = Query("defender_only", description="self_play | defender_only"),
+    seed: int = Query(0),
+):
+    """Reset the environment and return the initial observation."""
+    env = _get_env(task, mode, seed)
+    return env.reset()
+
+
+@app.post("/step", response_model=StepResult)
+def step(
+    action: Action,
+    task: str = Query("stage1_basic"),
+    mode: str = Query("defender_only"),
+    seed: int = Query(0),
+):
+    """Execute one action and return observation, reward, done, info."""
+    env = _get_env(task, mode, seed)
+    if env._state is None:
+        raise HTTPException(status_code=400, detail="Call /reset first.")
+    try:
+        obs, reward, done, info = env.step(action)
+    except RuntimeError as exc:
+        raise HTTPException(status_code=400, detail=str(exc)) from exc
+    return StepResult(observation=obs, reward=reward, done=done, info=info)
+
+
+@app.get("/state")
+def state(
+    task: str = Query("stage1_basic"),
+    mode: str = Query("defender_only"),
+    seed: int = Query(0),
+):
+    """Return the full internal episode state."""
+    env = _get_env(task, mode, seed)
+    return env.state()
+
+
+@app.get("/tasks")
+def list_tasks():
+    """List the available curriculum stages."""
+    from tasks.registry import STAGE_REGISTRY
+    return {
+        "tasks": [
+            {"id": stage_id, "difficulty": cfg["difficulty"], "description": cfg["description"]}
+            for stage_id, cfg in STAGE_REGISTRY.items()
+        ],
+        "modes": ["self_play", "defender_only"],
+    }
+
+
+@app.post("/grade", response_model=GradeResult)
+def grade(
+    task: str = Query("stage1_basic"),
+    mode: str = Query("defender_only"),
+    seed: int = Query(0),
+):
+    """Compute a normalized [0, 1] score for the just-finished episode."""
+    env = _get_env(task, mode, seed)
+    if env._state is None:
+        raise HTTPException(status_code=400, detail="No episode to grade. Call /reset first.")
+    s = env._state
+    return GradeResult(
+        task=task,
+        mode=mode,
+        score=env.grade(),
+        defender_reward=s.defender_reward,
+        attacker_reward=s.attacker_reward,
+        ground_truth=s.ground_truth.value if s.ground_truth else None,
+        plausible=s.plausible,
+        schema_violation=s.schema_violation,
+    )
+
+
+@app.get("/health")
+def health():
+    return {"status": "ok", "env": "OpenSOC", "version": "1.0.0"}
+
+
+def main() -> None:
+    import uvicorn
+
+    port = int(os.getenv("PORT", 7860))
+    uvicorn.run(app, host="0.0.0.0", port=port)

client/__init__.py

@@ -0,0 +1,3 @@
+from .opensoc_client import OpenSOCClient
+
+__all__ = ["OpenSOCClient"]

client/opensoc_client.py

@@ -0,0 +1,87 @@
+"""Thin HTTP client for the OpenSOC environment.
+
+Importantly: this module **never imports server-side code** (`env.py`,
+`verifier.py`, `rubric.py`).  The OpenEnv hackathon brief calls for
+client/server separation so the same client can drive a remote HF Space
+or a local container without re-running the verifier locally.
+
+Usage::
+
+    from client import OpenSOCClient
+    c = OpenSOCClient(base_url="http://localhost:7860")
+    obs = c.reset(task="stage1_basic", mode="defender_only", seed=1)
+    result = c.step(
+        {"submit_triage": {"action": "monitor",
+                           "cited_log_id": "L1-0",
+                           "rationale": "..."}},
+        task="stage1_basic", mode="defender_only", seed=1,
+    )
+    grade = c.grade(task="stage1_basic", mode="defender_only", seed=1)
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Any, Dict, Optional
+
+import requests
+
+
+class OpenSOCClient:
+    """Lightweight requests-based client for the OpenSOC FastAPI server."""
+
+    def __init__(
+        self,
+        base_url: Optional[str] = None,
+        timeout: float = 30.0,
+        session: Optional[requests.Session] = None,
+    ):
+        self.base_url = (base_url or os.getenv("OPENSOC_URL", "http://localhost:7860")).rstrip("/")
+        self.timeout = timeout
+        self.session = session or requests.Session()
+
+    def health(self) -> Dict[str, Any]:
+        return self._get("/health")
+
+    def tasks(self) -> Dict[str, Any]:
+        return self._get("/tasks")
+
+    def reset(self, task: str = "stage1_basic", mode: str = "defender_only", seed: int = 0) -> Dict[str, Any]:
+        return self._post("/reset", params={"task": task, "mode": mode, "seed": seed})
+
+    def step(
+        self,
+        action: Dict[str, Any],
+        task: str = "stage1_basic",
+        mode: str = "defender_only",
+        seed: int = 0,
+    ) -> Dict[str, Any]:
+        return self._post(
+            "/step",
+            params={"task": task, "mode": mode, "seed": seed},
+            json=action,
+        )
+
+    def state(self, task: str = "stage1_basic", mode: str = "defender_only", seed: int = 0) -> Dict[str, Any]:
+        return self._get("/state", params={"task": task, "mode": mode, "seed": seed})
+
+    def grade(self, task: str = "stage1_basic", mode: str = "defender_only", seed: int = 0) -> Dict[str, Any]:
+        return self._post("/grade", params={"task": task, "mode": mode, "seed": seed})
+
+    def _get(self, path: str, params: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
+        r = self.session.get(self.base_url + path, params=params, timeout=self.timeout)
+        r.raise_for_status()
+        return r.json()
+
+    def _post(
+        self,
+        path: str,
+        params: Optional[Dict[str, Any]] = None,
+        json: Any = None,
+    ) -> Dict[str, Any]:
+        r = self.session.post(self.base_url + path, params=params, json=json, timeout=self.timeout)
+        r.raise_for_status()
+        return r.json()
+
+
+__all__ = ["OpenSOCClient"]

client/prompts.py

@@ -0,0 +1,172 @@
+"""Prompt formatting + completion parsing for both roles.
+
+Keeping this in one module lets us reuse the exact same prompt template
+across SFT warm-start, GRPO rollouts, and the held-out eval — an oft-forgotten
+prerequisite for the metrics in the report to be comparable.
+
+The defender prompt is engineered for a small (3B-class) instruction-tuned
+model: it is short, contains an explicit JSON schema, and shows one example.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from typing import Optional
+
+from schema import (
+    Action,
+    Alert,
+    CraftIncident,
+    Event,
+    EventType,
+    IncidentCategory,
+    SubmitTriage,
+    TriageAction,
+)
+
+
+# ---------------------------------------------------------------------------
+# System messages
+# ---------------------------------------------------------------------------
+
+DEFENDER_SYSTEM = (
+    "You are a senior SOC analyst. You triage SIEM alerts. "
+    "Read the alert and the structured log window, then choose exactly one "
+    "action from {dismiss, monitor, quarantine_host, block_ip, escalate} "
+    "and cite the single most diagnostic log_id. "
+    "Respond with a JSON object only. Never invent log_ids."
+)
+
+
+ATTACKER_SYSTEM = (
+    "You are a red-team incident designer. You craft synthetic security "
+    "incidents that the defender model must triage correctly. "
+    "Your goal is to design plausible incidents whose ground-truth label is "
+    "as confusing as possible to the defender, but they must remain "
+    "*coherent* — internal-only events cannot be 'exfiltration', LOLBin "
+    "events must include both `process` and `parent_process` fields, etc. "
+    "Respond with a JSON object only that conforms to the craft_incident schema."
+)
+
+
+_DEFENDER_RESPONSE_SCHEMA = """\
+{
+  "action":        "dismiss|monitor|quarantine_host|block_ip|escalate",
+  "cited_log_id":  "<one log_id from the log_window>",
+  "rationale":     "<one short sentence>"
+}"""
+
+
+_DEFENDER_FEW_SHOT = """\
+Example:
+ALERT: c2_beacon: 1 event(s); first=net.beacon
+LOG_WINDOW:
+  L1-0  2026-04-25T10:00:00Z  net.beacon  dst_ip=203.0.113.5 interval_s=60
+
+Response:
+{"action": "block_ip", "cited_log_id": "L1-0", "rationale": "Periodic beacon to external IP indicates C2."}"""
+
+
+# ---------------------------------------------------------------------------
+# Defender side
+# ---------------------------------------------------------------------------
+
+def _render_event(e: Event) -> str:
+    fields = " ".join(f"{k}={v}" for k, v in (e.fields or {}).items() if v is not None)
+    return f"  {e.log_id}  {e.timestamp}  {e.event_type.value}  {fields}".rstrip()
+
+
+def format_defender_prompt(alert: Alert, log_window: list[Event]) -> str:
+    """Return the user-message body for a defender turn."""
+    body = [f"ALERT: {alert.summary}", "LOG_WINDOW:"]
+    for e in log_window:
+        body.append(_render_event(e))
+    body.append("")
+    body.append(f"Respond with JSON in this shape:\n{_DEFENDER_RESPONSE_SCHEMA}")
+    body.append("")
+    body.append(_DEFENDER_FEW_SHOT)
+    return "\n".join(body)
+
+
+_JSON_BLOCK_RE = re.compile(r"\{.*\}", re.DOTALL)
+
+
+def parse_defender_completion(text: str) -> Action:
+    """Parse a model completion into an `Action(submit_triage=...)`.
+
+    Robust to the common case where the model emits prose then JSON. We
+    locate the first balanced JSON object.  Schema mismatches raise the
+    pydantic ValidationError, which the caller (env.step) treats as a
+    schema violation.
+    """
+    match = _JSON_BLOCK_RE.search(text)
+    if not match:
+        raise ValueError("No JSON object found in defender completion")
+    payload = json.loads(match.group(0))
+    return Action(
+        submit_triage=SubmitTriage(
+            action=TriageAction(payload["action"]),
+            cited_log_id=str(payload["cited_log_id"]),
+            rationale=str(payload.get("rationale", "")),
+        )
+    )
+
+
+# ---------------------------------------------------------------------------
+# Attacker side
+# ---------------------------------------------------------------------------
+
+_ATTACKER_RESPONSE_SCHEMA = """\
+{
+  "target_label": "dismiss|monitor|quarantine_host|block_ip|escalate",
+  "category":     "phishing|brute_force|malware_execution|c2_beacon|data_exfiltration|insider_misuse|privilege_escalation|benign_noise",
+  "events": [
+    {"event_type": "<see schema>", "fields": {"...": "..."}}
+  ],
+  "narrative":    "<optional free text, ignored by the verifier>"
+}"""
+
+
+def format_attacker_prompt(brief: dict) -> str:
+    """Return the user-message body for an attacker turn."""
+    body = [
+        f"BRIEF: design an incident whose ground-truth label is action="
+        f"{brief.get('target_label', 'monitor')}, category hint="
+        f"{brief.get('category_hint', 'any')}, difficulty="
+        f"{brief.get('difficulty', 'easy')}.",
+        "",
+        f"Respond with JSON in this shape:\n{_ATTACKER_RESPONSE_SCHEMA}",
+    ]
+    return "\n".join(body)
+
+
+def parse_attacker_completion(text: str) -> Action:
+    """Parse a model completion into an `Action(craft_incident=...)`.
+
+    Adds defaults for `log_id`, `timestamp`, and `source` when the model
+    omits them — keeps SFT data clean and makes RL rollouts robust.
+    """
+    match = _JSON_BLOCK_RE.search(text)
+    if not match:
+        raise ValueError("No JSON object found in attacker completion")
+    payload = json.loads(match.group(0))
+
+    events: list[Event] = []
+    for i, raw in enumerate(payload.get("events", [])):
+        events.append(Event(
+            log_id=raw.get("log_id", f"L1-{i}"),
+            timestamp=raw.get("timestamp", "2026-04-25T10:00:00Z"),
+            source=raw.get("source", "synthetic"),
+            event_type=EventType(raw["event_type"]),
+            fields=raw.get("fields", {}),
+        ))
+
+    return Action(
+        craft_incident=CraftIncident(
+            target_label=TriageAction(payload["target_label"]),
+            category=IncidentCategory(payload.get("category", "benign_noise")),
+            events=events,
+            narrative=str(payload.get("narrative", "")),
+        )
+    )

data/holdout_smoke.jsonl

@@ -0,0 +1,20 @@
+{"alert": {"alert_id": "A-EVAL-c-90000", "category": "benign_noise", "severity": "low", "summary": "benign_noise: 1 event(s); first=auth.login_success", "host": "host-001", "user": "dave"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T00:08:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "dave", "src_ip": "10.0.0.84"}}], "ground_truth": "dismiss", "triggering_log_id": "L1-0", "stage": "stage1_basic", "seed": 90000}
+{"alert": {"alert_id": "A-EVAL-c-90001", "category": "malware_execution", "severity": "high", "summary": "malware_execution: 1 event(s); first=edr.behavior_match", "host": "host-032", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T22:52:00Z", "source": "edr", "event_type": "edr.behavior_match", "fields": {"severity": "high", "rule_id": "EDR-8836", "host": "host-032"}}], "ground_truth": "quarantine_host", "triggering_log_id": "L1-0", "stage": "stage1_basic", "seed": 90001}
+{"alert": {"alert_id": "A-EVAL-c-90002", "category": "malware_execution", "severity": "high", "summary": "malware_execution: 1 event(s); first=edr.behavior_match", "host": "host-008", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T16:56:00Z", "source": "edr", "event_type": "edr.behavior_match", "fields": {"severity": "high", "rule_id": "EDR-3486", "host": "host-008"}}], "ground_truth": "quarantine_host", "triggering_log_id": "L1-0", "stage": "stage1_basic", "seed": 90002}
+{"alert": {"alert_id": "A-EVAL-c-90003", "category": "benign_noise", "severity": "medium", "summary": "benign_noise: 1 event(s); first=net.port_scan_hit", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T05:32:00Z", "source": "network", "event_type": "net.port_scan_hit", "fields": {"src_ip": "203.0.113.115", "port": 23}}], "ground_truth": "monitor", "triggering_log_id": "L1-0", "stage": "stage1_basic", "seed": 90003}
+{"alert": {"alert_id": "A-EVAL-c-90004", "category": "c2_beacon", "severity": "high", "summary": "c2_beacon: 1 event(s); first=net.beacon", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T15:26:00Z", "source": "network", "event_type": "net.beacon", "fields": {"dst_ip": "198.51.100.42", "interval_s": 60}}], "ground_truth": "block_ip", "triggering_log_id": "L1-0", "stage": "stage1_basic", "seed": 90004}
+{"alert": {"alert_id": "A-EVAL-i-91000", "category": "benign_noise", "severity": "low", "summary": "benign_noise: 2 event(s); first=net.beacon", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T21:23:00Z", "source": "network", "event_type": "net.beacon", "fields": {"dst_ip": "10.0.61.71", "interval_s": 60}}, {"log_id": "L1-1", "timestamp": "2026-04-25T22:28:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "alice", "src_ip": "10.0.0.76"}}], "ground_truth": "monitor", "triggering_log_id": "L1-0", "stage": "stage2_multi", "seed": 91000}
+{"alert": {"alert_id": "A-EVAL-i-91001", "category": "benign_noise", "severity": "low", "summary": "benign_noise: 2 event(s); first=net.dns_query", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T03:58:00Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "github.com"}}, {"log_id": "L1-1", "timestamp": "2026-04-25T06:16:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "bob", "src_ip": "10.0.0.120"}}], "ground_truth": "dismiss", "triggering_log_id": "L1-1", "stage": "stage2_multi", "seed": 91001}
+{"alert": {"alert_id": "A-EVAL-i-91002", "category": "benign_noise", "severity": "medium", "summary": "benign_noise: 2 event(s); first=net.port_scan_hit", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T02:47:00Z", "source": "network", "event_type": "net.port_scan_hit", "fields": {"src_ip": "203.0.113.103", "port": 22}}, {"log_id": "L1-1", "timestamp": "2026-04-25T15:10:00Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "google.com"}}], "ground_truth": "monitor", "triggering_log_id": "L1-0", "stage": "stage2_multi", "seed": 91002}
+{"alert": {"alert_id": "A-EVAL-i-91003", "category": "malware_execution", "severity": "high", "summary": "malware_execution: 2 event(s); first=auth.login_success", "host": "host-001", "user": "bob"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T06:42:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "bob", "src_ip": "10.0.0.162"}}, {"log_id": "L1-1", "timestamp": "2026-04-25T16:50:00Z", "source": "edr", "event_type": "edr.behavior_match", "fields": {"severity": "high", "rule_id": "EDR-2571", "host": "host-001"}}], "ground_truth": "quarantine_host", "triggering_log_id": "L1-1", "stage": "stage2_multi", "seed": 91003}
+{"alert": {"alert_id": "A-EVAL-i-91004", "category": "c2_beacon", "severity": "high", "summary": "c2_beacon: 2 event(s); first=net.beacon", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T04:11:00Z", "source": "network", "event_type": "net.beacon", "fields": {"dst_ip": "203.0.113.5", "interval_s": 30}}, {"log_id": "L1-1", "timestamp": "2026-04-25T06:32:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "alice", "src_ip": "10.0.0.66"}}], "ground_truth": "block_ip", "triggering_log_id": "L1-0", "stage": "stage2_multi", "seed": 91004}
+{"alert": {"alert_id": "A-EVAL-d-92000", "category": "benign_noise", "severity": "low", "summary": "benign_noise: 3 event(s); first=net.outbound_connection", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T04:39:00Z", "source": "network", "event_type": "net.outbound_connection", "fields": {"dst_ip": "10.0.81.214", "bytes_out": 86290}}, {"log_id": "L1-1", "timestamp": "2026-04-25T04:39:02Z", "source": "network", "event_type": "net.outbound_connection", "fields": {"dst_ip": "10.0.138.227", "bytes_out": 53711}}, {"log_id": "L1-2", "timestamp": "2026-04-25T16:45:00Z", "source": "network", "event_type": "net.beacon", "fields": {"dst_ip": "10.0.244.186", "interval_s": 90}}], "ground_truth": "monitor", "triggering_log_id": "L1-2", "stage": "stage3_mixed", "seed": 92000}
+{"alert": {"alert_id": "A-EVAL-d-92001", "category": "malware_execution", "severity": "high", "summary": "malware_execution: 3 event(s); first=edr.behavior_match", "host": "host-013", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T06:29:00Z", "source": "edr", "event_type": "edr.behavior_match", "fields": {"severity": "high", "rule_id": "EDR-6020", "host": "host-013"}}, {"log_id": "L1-1", "timestamp": "2026-04-25T15:02:00Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "internal.corp"}}, {"log_id": "L1-2", "timestamp": "2026-04-25T15:02:02Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "dave", "src_ip": "10.0.0.40"}}], "ground_truth": "quarantine_host", "triggering_log_id": "L1-0", "stage": "stage3_mixed", "seed": 92001}
+{"alert": {"alert_id": "A-EVAL-d-92002", "category": "brute_force", "severity": "medium", "summary": "brute_force: 11 event(s); first=net.dns_query", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T06:26:00Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "github.com"}}, {"log_id": "L1-1", "timestamp": "2026-04-25T06:26:02Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "carol", "src_ip": "10.0.0.15"}}, {"log_id": "L1-2", "timestamp": "2026-04-25T09:47:00Z", "source": "identity", "event_type": "auth.login_failure", "fields": {"user": "alice", "src_ip": "203.0.113.78"}}, {"log_id": "L1-3", "timestamp": "2026-04-25T09:47:05Z", "source": "identity", "event_type": "auth.login_failure", "fields": {"user": "alice", "src_ip": "203.0.113.140"}}, {"log_id": "L1-4", "timestamp": "2026-04-25T09:47:10Z", "source": "identity", "event_type": "auth.login_failure", "fields": {"user": "alice", "src_ip": "203.0.113.19"}}, {"log_id": "L1-5", "timestamp": "2026-04-25T09:47:15Z", "source": "identity", "event_type": "auth.login_failure", "fields": {"user": "alice", "src_ip": "203.0.113.222"}}, {"log_id": "L1-6", "timestamp": "2026-04-25T09:47:20Z", "source": "identity", "event_type": "auth.login_failure", "fields": {"user": "alice", "src_ip": "203.0.113.194"}}, {"log_id": "L1-7", "timestamp": "2026-04-25T09:47:25Z", "source": "identity", "event_type": "auth.login_failure", "fields": {"user": "alice", "src_ip": "203.0.113.217"}}, {"log_id": "L1-8", "timestamp": "2026-04-25T09:47:30Z", "source": "identity", "event_type": "auth.login_failure", "fields": {"user": "alice", "src_ip": "203.0.113.118"}}, {"log_id": "L1-9", "timestamp": "2026-04-25T09:47:35Z", "source": "identity", "event_type": "auth.login_failure", "fields": {"user": "alice", "src_ip": "203.0.113.243"}}, {"log_id": "L1-10", "timestamp": "2026-04-25T09:47:40Z", "source": "identity", "event_type": "auth.login_failure", "fields": {"user": "alice", "src_ip": "203.0.113.128"}}], "ground_truth": "monitor", "triggering_log_id": "L1-2", "stage": "stage3_mixed", "seed": 92002}
+{"alert": {"alert_id": "A-EVAL-d-92003", "category": "benign_noise", "severity": "low", "summary": "benign_noise: 3 event(s); first=auth.login_success", "host": "host-001", "user": "dave"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T01:19:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "dave", "src_ip": "10.0.0.29"}}, {"log_id": "L1-1", "timestamp": "2026-04-25T01:19:02Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "internal.corp"}}, {"log_id": "L1-2", "timestamp": "2026-04-25T05:05:00Z", "source": "network", "event_type": "net.beacon", "fields": {"dst_ip": "10.0.153.44", "interval_s": 60}}], "ground_truth": "monitor", "triggering_log_id": "L1-2", "stage": "stage3_mixed", "seed": 92003}
+{"alert": {"alert_id": "A-EVAL-d-92004", "category": "benign_noise", "severity": "low", "summary": "benign_noise: 3 event(s); first=auth.login_success", "host": "host-001", "user": "dave"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T04:51:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "dave", "src_ip": "10.0.0.64"}}, {"log_id": "L1-1", "timestamp": "2026-04-25T04:51:02Z", "source": "network", "event_type": "net.outbound_connection", "fields": {"dst_ip": "10.0.217.73", "bytes_out": 81935}}, {"log_id": "L1-2", "timestamp": "2026-04-25T06:31:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "bob", "src_ip": "10.0.0.102"}}], "ground_truth": "dismiss", "triggering_log_id": "L1-2", "stage": "stage3_mixed", "seed": 92004}
+{"alert": {"alert_id": "A-EVAL-l-93000", "category": "benign_noise", "severity": "low", "summary": "benign_noise: 4 event(s); first=net.beacon", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T06:15:00Z", "source": "network", "event_type": "net.beacon", "fields": {"dst_ip": "10.0.72.10", "interval_s": 30}}, {"log_id": "L1-1", "timestamp": "2026-04-25T09:39:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "bob", "src_ip": "10.0.0.28"}}, {"log_id": "L1-2", "timestamp": "2026-04-25T09:39:02Z", "source": "network", "event_type": "net.outbound_connection", "fields": {"dst_ip": "10.0.23.160", "bytes_out": 38043}}, {"log_id": "L1-3", "timestamp": "2026-04-25T09:39:04Z", "source": "network", "event_type": "net.outbound_connection", "fields": {"dst_ip": "10.0.108.241", "bytes_out": 36859}}], "ground_truth": "monitor", "triggering_log_id": "L1-0", "stage": "stage4_adversarial", "seed": 93000}
+{"alert": {"alert_id": "A-EVAL-l-93001", "category": "c2_beacon", "severity": "high", "summary": "c2_beacon: 4 event(s); first=net.beacon", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T06:47:00Z", "source": "network", "event_type": "net.beacon", "fields": {"dst_ip": "185.220.101.7", "interval_s": 60}}, {"log_id": "L1-1", "timestamp": "2026-04-25T13:14:00Z", "source": "network", "event_type": "net.outbound_connection", "fields": {"dst_ip": "10.0.183.125", "bytes_out": 92358}}, {"log_id": "L1-2", "timestamp": "2026-04-25T13:14:02Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "google.com"}}, {"log_id": "L1-3", "timestamp": "2026-04-25T13:14:04Z", "source": "network", "event_type": "net.outbound_connection", "fields": {"dst_ip": "10.0.80.164", "bytes_out": 75352}}], "ground_truth": "block_ip", "triggering_log_id": "L1-0", "stage": "stage4_adversarial", "seed": 93001}
+{"alert": {"alert_id": "A-EVAL-l-93002", "category": "benign_noise", "severity": "low", "summary": "benign_noise: 4 event(s); first=net.beacon", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T04:30:00Z", "source": "network", "event_type": "net.beacon", "fields": {"dst_ip": "10.0.244.83", "interval_s": 60}}, {"log_id": "L1-1", "timestamp": "2026-04-25T08:04:00Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "internal.corp"}}, {"log_id": "L1-2", "timestamp": "2026-04-25T08:04:02Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "internal.corp"}}, {"log_id": "L1-3", "timestamp": "2026-04-25T08:04:04Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "alice", "src_ip": "10.0.0.243"}}], "ground_truth": "monitor", "triggering_log_id": "L1-0", "stage": "stage4_adversarial", "seed": 93002}
+{"alert": {"alert_id": "A-EVAL-l-93003", "category": "c2_beacon", "severity": "high", "summary": "c2_beacon: 4 event(s); first=net.beacon", "host": "host-001", "user": "user-001"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T07:56:00Z", "source": "network", "event_type": "net.beacon", "fields": {"dst_ip": "185.220.101.7", "interval_s": 90}}, {"log_id": "L1-1", "timestamp": "2026-04-25T17:25:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "carol", "src_ip": "10.0.0.70"}}, {"log_id": "L1-2", "timestamp": "2026-04-25T17:25:02Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "dave", "src_ip": "10.0.0.170"}}, {"log_id": "L1-3", "timestamp": "2026-04-25T17:25:04Z", "source": "network", "event_type": "net.outbound_connection", "fields": {"dst_ip": "10.0.148.248", "bytes_out": 71310}}], "ground_truth": "block_ip", "triggering_log_id": "L1-0", "stage": "stage4_adversarial", "seed": 93003}
+{"alert": {"alert_id": "A-EVAL-l-93004", "category": "benign_noise", "severity": "low", "summary": "benign_noise: 4 event(s); first=auth.login_success", "host": "host-001", "user": "dave"}, "events": [{"log_id": "L1-0", "timestamp": "2026-04-25T04:55:00Z", "source": "identity", "event_type": "auth.login_success", "fields": {"user": "dave", "src_ip": "10.0.0.144"}}, {"log_id": "L1-1", "timestamp": "2026-04-25T19:38:00Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "github.com"}}, {"log_id": "L1-2", "timestamp": "2026-04-25T19:38:02Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "github.com"}}, {"log_id": "L1-3", "timestamp": "2026-04-25T19:38:04Z", "source": "network", "event_type": "net.dns_query", "fields": {"domain": "internal.corp"}}], "ground_truth": "dismiss", "triggering_log_id": "L1-3", "stage": "stage4_adversarial", "seed": 93004}

demo_app.py

@@ -0,0 +1,119 @@
+"""Gradio "before vs after" demo for OpenSOC, mounted at /demo.
+
+The demo is read-only and uses pre-baked examples from
+`data/demo_examples.json` so the Space can stay on the free CPU tier.
+Judges click "Next incident" and see, side by side:
+
+  * the SIEM alert + log window the defender is given,
+  * what zero-shot Qwen2.5-3B-Instruct says (usually wrong),
+  * what the OpenSOC GRPO-trained model says (usually right),
+  * the verifier's ground truth + the triggering log id.
+
+This module is imported by `server.py` *after* `app_runtime.app` is built
+and *before* uvicorn starts, so the Gradio routes are mounted on the same
+FastAPI app that exposes /reset, /step, /state, /grade.
+"""
+
+from __future__ import annotations
+
+import os
+
+import gradio as gr
+
+from app_runtime import app
+from demo_data import (
+    empty_state_message,
+    format_alert_card,
+    format_response_card,
+    format_truth_card,
+    load_demo_examples,
+)
+
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+_DEMO_PATH = os.path.join(_HERE, "data", "demo_examples.json")
+_EXAMPLES = load_demo_examples(_DEMO_PATH)
+
+
+HEADER_MD = """
+# OpenSOC: SOC triage agent (before vs after RLVR self-play training)
+
+Each example below is a real incident drawn from the frozen 200-incident
+hold-out set.  The same alert + log window is shown to two models:
+
+- **Baseline**: zero-shot Qwen2.5-3B-Instruct, untouched.
+- **OpenSOC**: the same model after SFT warm-start + GRPO curriculum on this env.
+
+The verifier-grounded ground truth label and triggering log id are computed
+deterministically by `verifier.py` and never depend on either model's text.
+""".strip()
+
+
+def _render(idx: int):
+    if not _EXAMPLES:
+        msg = empty_state_message()
+        return msg, "", "", "", "0 / 0"
+    ex = _EXAMPLES[idx % len(_EXAMPLES)]
+    return (
+        format_alert_card(ex["alert"], ex["events"]),
+        format_response_card("Baseline (Qwen2.5-3B zero-shot)", ex["baseline"]),
+        format_response_card("OpenSOC (after GRPO)", ex["trained"]),
+        format_truth_card(ex),
+        f"{(idx % len(_EXAMPLES)) + 1} / {len(_EXAMPLES)}",
+    )
+
+
+with gr.Blocks(
+    title="OpenSOC: Defender LLM trained via GRPO",
+    theme=gr.themes.Soft(),
+    analytics_enabled=False,
+) as demo:
+    gr.Markdown(HEADER_MD)
+    counter = gr.State(value=0)
+
+    with gr.Row():
+        prev_btn = gr.Button("Previous", variant="secondary")
+        next_btn = gr.Button("Next incident", variant="primary")
+        position = gr.Markdown("0 / 0")
+
+    alert_md = gr.Markdown()
+    with gr.Row():
+        baseline_md = gr.Markdown()
+        trained_md = gr.Markdown()
+    truth_md = gr.Markdown()
+
+    def _next(i):
+        return i + 1, *_render(i + 1)
+
+    def _prev(i):
+        return i - 1, *_render(i - 1)
+
+    next_btn.click(
+        _next, inputs=[counter],
+        outputs=[counter, alert_md, baseline_md, trained_md, truth_md, position],
+    )
+    prev_btn.click(
+        _prev, inputs=[counter],
+        outputs=[counter, alert_md, baseline_md, trained_md, truth_md, position],
+    )
+    demo.load(
+        lambda: (0, *_render(0)),
+        outputs=[counter, alert_md, baseline_md, trained_md, truth_md, position],
+    )
+
+    gr.Markdown("---")
+    gr.Markdown(
+        "**Repo**: this Space is built from "
+        "[github.com/.../opensoc](https://github.com/) — see the README for the "
+        "OpenEnv manifest, training notebook, and 200-incident hold-out eval."
+    )
+
+
+# Mount the Gradio Blocks app on the same FastAPI server that exposes the
+# OpenEnv API.  After this import, navigating to `/demo` on the Space
+# loads this UI, and `/reset`, `/step`, `/state`, `/grade`, `/tasks`,
+# `/health` continue to work for the OpenEnv judge bot.
+app = gr.mount_gradio_app(app, demo, path="/demo")
+
+
+__all__ = ["app", "demo"]

demo_data.py

@@ -0,0 +1,127 @@
+"""Helpers for the Gradio demo: load the pre-baked before-vs-after JSON
+file and render each section as readable markdown.
+
+The demo is intentionally read-only and deterministic: judges click "Next
+incident" and see one of N pre-computed (alert, baseline-response,
+trained-response, ground-truth) tuples.  The expensive part — running
+the baseline and trained model on each incident — happens once on a GPU
+in `eval.bake_demo` and is committed to `data/demo_examples.json`.
+
+This file is small, fast, and carries no GPU dependency, so the deployed
+HF Space can stay on the free CPU tier and still cold-start in <30s.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from typing import Any, Dict, List
+
+
+def load_demo_examples(path: str) -> List[Dict[str, Any]]:
+    """Read demo examples.  Returns [] if the file isn't present yet so
+    the Space still boots before the user has run training + bake_demo."""
+    if not os.path.exists(path):
+        return []
+    with open(path, "r", encoding="utf-8") as f:
+        data = json.load(f)
+    if isinstance(data, dict) and "examples" in data:
+        return data["examples"]
+    return data  # type: ignore[return-value]
+
+
+def _format_event(e: Dict[str, Any]) -> str:
+    fields = e.get("fields") or {}
+    field_strs = []
+    for k, v in fields.items():
+        if v in (None, ""):
+            continue
+        field_strs.append(f"`{k}`={v}")
+    fields_md = " ".join(field_strs)
+    et = e.get("event_type", "?")
+    if hasattr(et, "value"):
+        et = et.value
+    return (
+        f"- `{e.get('log_id')}` · {e.get('timestamp')} · "
+        f"src=`{e.get('source')}` · type=`{et}` · {fields_md}"
+    ).rstrip()
+
+
+def format_alert_card(alert: Dict[str, Any], events: List[Dict[str, Any]]) -> str:
+    """Render the SIEM alert + log window as a markdown card."""
+    lines = [
+        f"### Alert `{alert.get('alert_id', '?')}`",
+        f"- **category**: {alert.get('category')}",
+        f"- **severity**: {alert.get('severity')}",
+        f"- **host / user**: {alert.get('host')} / {alert.get('user')}",
+        f"- **summary**: {alert.get('summary', '')}",
+        "",
+        f"**Log window ({len(events)} event(s))**",
+    ]
+    for e in events:
+        lines.append(_format_event(e))
+    return "\n".join(lines)
+
+
+def format_response_card(title: str, response: Dict[str, Any]) -> str:
+    """Render a model response (parsed action + reward + breakdown)."""
+    action = response.get("action", "—")
+    cited = response.get("cited_log_id", "—")
+    rationale = response.get("rationale", "")
+    reward = response.get("reward")
+    correct = response.get("correct")
+    raw = response.get("raw_text", "")
+
+    correct_emoji = "OK" if correct else ("MISS" if correct is False else "?")
+    reward_str = f"{reward:+.2f}" if isinstance(reward, (int, float)) else "—"
+
+    lines = [
+        f"### {title}",
+        f"- **action**: `{action}` ({correct_emoji})",
+        f"- **cited_log**: `{cited}`",
+        f"- **reward**: `{reward_str}`",
+        "",
+        f"> {rationale}",
+    ]
+    breakdown = response.get("reward_breakdown") or {}
+    if breakdown:
+        bk = ", ".join(f"`{k}={v:+.2f}`" for k, v in breakdown.items())
+        lines.append("")
+        lines.append(f"_{bk}_")
+    if raw and raw != rationale:
+        lines.append("")
+        lines.append("<details><summary>raw model output</summary>")
+        lines.append("")
+        lines.append("```")
+        lines.append(raw.strip())
+        lines.append("```")
+        lines.append("</details>")
+    return "\n".join(lines)
+
+
+def format_truth_card(ex: Dict[str, Any]) -> str:
+    return (
+        f"**Ground truth**: `{ex.get('ground_truth')}`  ·  "
+        f"**Triggering log**: `{ex.get('triggering_log_id')}`  ·  "
+        f"**Stage**: `{ex.get('stage')}`  ·  **Seed**: `{ex.get('seed')}`"
+    )
+
+
+def empty_state_message() -> str:
+    return (
+        "### No demo examples baked yet\n\n"
+        "Run `python -m eval.bake_demo --placeholder` (no GPU required) "
+        "or, after training, "
+        "`python -m eval.bake_demo --baseline unsloth/Qwen2.5-3B-Instruct "
+        "--trained-adapter checkpoints/defender_grpo/stage4_adversarial/adapter` "
+        "to populate `data/demo_examples.json`."
+    )
+
+
+__all__ = [
+    "load_demo_examples",
+    "format_alert_card",
+    "format_response_card",
+    "format_truth_card",
+    "empty_state_message",
+]

docs/blog.md

@@ -0,0 +1,134 @@
+# OpenSOC: Teaching an LLM to Triage Cyberattacks via RLVR Self-Play
+
+*A submission for the OpenEnv Hackathon, April 2026.*
+
+---
+
+## Why we built this
+
+By the time a security operations center (SOC) tier-1 analyst sees an
+alert, the attacker may have been inside for hours.  Tier-1 triage is
+mostly judgement: look at one alert plus the small log window around
+it, and decide whether to dismiss, monitor, quarantine, block, or
+escalate.  It's also where SOCs are chronically understaffed — a
+pipeline that quietly skips real attacks because the human is asleep is
+not a hypothetical risk.
+
+LLMs *should* be able to help.  But training one to do this without
+poisoning ourselves on the way is hard:
+
+1. SOC datasets are private; published ones get stale within months.
+2. Subjective rewards from another LLM let the trainee reward-hack the
+   judge, not the world.
+3. Self-play between two LLMs (one playing attacker, one defender)
+   collapses to a degenerate equilibrium unless the reward is *grounded*.
+
+OpenSOC is our attempt at all three: a synthetic, self-play SOC triage
+environment with a **deterministic, schema-driven verifier** as the only
+source of reward signal — true Reinforcement Learning with Verifiable
+Rewards (RLVR).
+
+## What's in the env
+
+The environment exposes the OpenEnv API (`/reset`, `/step`, `/state`,
+`/grade`) plus a Gradio "before vs after" UI at `/demo`:
+
+- **Schema**: a single `schema.py` defines `IncidentParams`, the only
+  shape the attacker can produce.  Enums for event types and triage
+  actions, structured fields, no free-text "vibes".
+- **Attacker**: must emit a structured `craft_incident` action.  Free
+  text is allowed but ignored by the verifier.
+- **Verifier**: `verifier.compute_ground_truth(params)` is a pure
+  function of the structured fields.  Authentication failures from a
+  single source IP at >5 attempts/window get labeled `block_ip`,
+  malware_exec on a workstation gets `quarantine_host`, etc.  This is
+  the RLVR substrate.
+- **Plausibility check**: a separate `verifier.check_plausibility(params)`
+  rejects gibberish (e.g. malware exec with no malicious indicators),
+  and the attacker only earns reward when the incident passes
+  plausibility.  This is what neutralises the classic self-play hack
+  where the attacker just emits noise to confuse the defender.
+- **Defender**: chooses one triage action and cites the log_id that
+  drove the decision.  Reward layers are: +1.0 for matching the verifier,
+  −1.0 for dismiss-on-malicious, −0.3 for over-react on benign, +0.1
+  bonus for the correct citation.  Dismiss-on-malicious is the cardinal
+  failure mode in real SOCs and we grade it accordingly.
+
+## How we trained
+
+A Qwen2.5-3B-Instruct with LoRA (rank 16) on a single L4:
+
+1. **SFT warm-start** on 600 (alert, log_window → action+citation+rationale)
+   gold examples (~12 minutes).  This is just to push P(format-compliant
+   response) from 0% to ~95% so GRPO has signal.
+2. **GRPO curriculum** for 200 steps × 4 stages (~3 hours):
+   - `stage1_basic`: single-event templates (one obvious benign + one
+     obvious malicious per category).
+   - `stage2_multi`: malicious signal spread over a multi-event window.
+   - `stage3_mixed`: benign noise interleaved with malicious events,
+     stress-tests false-positive suppression.
+   - `stage4_adversarial`: attacker-controlled distribution.
+3. **Eval** on a frozen 200-incident hold-out set (seed bands disjoint
+   from training, see `tasks/registry.py`).
+
+Total compute: ~$3 on HF Jupyter L4.  All scripts are turn-key
+(`scripts/run_full_pipeline.sh`) and the same notebook produces a
+`data/demo_examples.json` of 50 before-vs-after pairs that the deployed
+Space serves on the free CPU tier.
+
+## What's actually new here
+
+Most "self-play LLM" demos either (a) train a judge LLM and call its
+score the reward, or (b) hand-code the reward but abandon self-play.
+OpenSOC keeps both:
+
+- **Self-play** is preserved because the attacker is a real model
+  emitting real (structured) parameters.
+- **The reward is verifiable** because it's computed from the structured
+  parameters, not the attacker's narrative — so the attacker cannot
+  reward-hack by writing scary text.
+
+The trick that makes this work is the *plausibility check*: a separate,
+deterministic gate on whether the attacker's params even look like a
+real incident.  This is what stops the attacker from exploring
+adversarial null-spaces.  We tested it with 21 anti-hack regression
+tests in `tests/test_rubric.py`.
+
+## Headline numbers
+
+| Metric                            | Baseline (zero-shot) | OpenSOC (after GRPO) |
+| --------------------------------- | -------------------: | -------------------: |
+| Macro F1 over 200 hold-out        |              [PEND]  |              [PEND]  |
+| Dismiss-on-malicious rate         |              [PEND]  |              [PEND]  |
+| Over-react on benign              |              [PEND]  |              [PEND]  |
+
+(*Numbers will be filled in after the GPU run; placeholder demo data
+already shows the qualitative shape: the always-dismiss baseline gets
+~15% accuracy, an oracle-equivalent trained model approaches 100%.*)
+
+The four diagnostic plots live in `eval/results/`:
+
+- `bar_dismiss_on_malicious.png` — the headline plot.
+- `bar_macro_f1.png`
+- `confusion_baseline_zero_shot.png` and `confusion_opensoc_grpo.png`
+- `training_curves.png` — reward across the four curriculum stages.
+
+## What's next
+
+- Add a *third* role (the "investigator") that actively queries log
+  sources rather than receiving a pre-baked log window.
+- Train on real SIEM exports (CSE-CIC-IDS, Splunk Boss-of-the-SOC) and
+  use the synthetic env only as a curriculum bootstrap.
+- Plug the deployed Space into a live SOAR (Tines / Shuffle) and watch
+  it triage real-world tier-1 traffic.
+
+## Try it
+
+- HF Space: `https://huggingface.co/spaces/<USER>/opensoc-env`
+  (`/demo` for the human-readable before-vs-after UI)
+- Repo: `https://huggingface.co/<USER>/opensoc-env`
+- Plan and design notes: see `README.md` in the repo.
+
+---
+
+*Written for the OpenEnv Hackathon, April 2026.  Code under BSD-3.*

docs/build_slides.py

@@ -0,0 +1,221 @@
+"""Build the 5-slide submission deck (`docs/slides.pdf`).
+
+Uses matplotlib's PdfPages to render five 16:9 slides:
+
+  1. Title          — OpenSOC: Self-Play SOC Triage
+  2. Problem        — Why this matters; cardinal failure mode.
+  3. Env design     — Architecture diagram (text); RLVR insight.
+  4. Results        — Headline plots embedded.
+  5. Demo + links   — Space URL, repo URL, video URL.
+
+Run::
+
+    python -m docs.build_slides --out docs/slides.pdf
+
+The script also reads `eval/results/summary.json` and the four PNGs so
+the deck stays in sync with the latest eval run automatically.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from typing import Any, Dict, List, Optional
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.image as mpimg
+import matplotlib.pyplot as plt
+from matplotlib.backends.backend_pdf import PdfPages
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+_REPO = os.path.dirname(_HERE)
+
+
+def _new_slide(title: str, subtitle: str | None = None):
+    fig, ax = plt.subplots(figsize=(13.33, 7.5))  # 16:9 at ~96 DPI
+    ax.set_axis_off()
+    ax.text(
+        0.05, 0.92, title, fontsize=32, fontweight="bold",
+        transform=ax.transAxes,
+    )
+    if subtitle:
+        ax.text(
+            0.05, 0.86, subtitle, fontsize=18, color="#444",
+            transform=ax.transAxes,
+        )
+    ax.plot(
+        [0.05, 0.95], [0.83, 0.83], color="#cccccc", linewidth=1.0,
+        transform=ax.transAxes,
+    )
+    return fig, ax
+
+
+def _bullets(ax, lines: List[str], y_start: float = 0.74, dy: float = 0.07, fontsize: int = 18):
+    for i, line in enumerate(lines):
+        ax.text(
+            0.07, y_start - i * dy, "• " + line,
+            fontsize=fontsize, transform=ax.transAxes,
+        )
+
+
+def _maybe_add_image(ax, img_path: str, bbox: tuple[float, float, float, float]):
+    if not os.path.exists(img_path):
+        x, y, w, h = bbox
+        ax.text(
+            x + w / 2, y + h / 2, "(plot pending)\n" + os.path.basename(img_path),
+            fontsize=12, color="#888", ha="center", va="center",
+            transform=ax.transAxes,
+        )
+        return
+    img = mpimg.imread(img_path)
+    ax_img = ax.figure.add_axes(bbox)  # absolute coords on the figure
+    ax_img.imshow(img)
+    ax_img.set_axis_off()
+
+
+def _read_summary(path: str) -> Optional[Dict[str, Any]]:
+    if not os.path.exists(path):
+        return None
+    with open(path, "r", encoding="utf-8") as f:
+        data = json.load(f)
+    by_label = {row["label"]: row for row in data}
+    return by_label
+
+
+def slide_title(pdf):
+    fig, ax = plt.subplots(figsize=(13.33, 7.5))
+    ax.set_axis_off()
+    fig.patch.set_facecolor("#0b1220")
+    ax.text(
+        0.5, 0.62, "OpenSOC", fontsize=72, color="white",
+        fontweight="bold", ha="center", transform=ax.transAxes,
+    )
+    ax.text(
+        0.5, 0.50, "RLVR self-play environment for SOC triage agents",
+        fontsize=22, color="#bbbbbb", ha="center", transform=ax.transAxes,
+    )
+    ax.text(
+        0.5, 0.38, "OpenEnv Hackathon, April 2026",
+        fontsize=16, color="#888", ha="center", transform=ax.transAxes,
+    )
+    pdf.savefig(fig)
+    plt.close(fig)
+
+
+def slide_problem(pdf):
+    fig, ax = _new_slide(
+        "The problem",
+        "Tier-1 SOC triage is judgement work, and the failure mode that hurts is dismiss-on-malicious.",
+    )
+    _bullets(ax, [
+        "SOCs are chronically understaffed; analysts skim hundreds of alerts/shift.",
+        "Real attackers blend in for hours before tier-2 even sees them.",
+        "An LLM that automates triage would help — IF its reward signal is honest.",
+        "Two classic traps: (1) train on a learned judge → reward-hack the judge.",
+        "                  (2) self-play between two LLMs → degenerate equilibrium.",
+        "OpenSOC: deterministic verifier + plausibility check = RLVR-clean self-play.",
+    ])
+    pdf.savefig(fig)
+    plt.close(fig)
+
+
+def slide_env(pdf):
+    fig, ax = _new_slide(
+        "Environment design",
+        "An attacker LLM crafts structured incidents; a defender LLM triages; verifier grounds the reward.",
+    )
+    _bullets(ax, [
+        "schema.py — single source of truth for events, actions, incident params.",
+        "verifier.compute_ground_truth(params) — pure function over structured fields.",
+        "verifier.check_plausibility(params) — gate that rejects gibberish before reward.",
+        "rubric.score_defender / score_attacker — layered, anti-hack-tested rewards.",
+        "OpenEnv-compliant API: /reset, /step, /state, /grade, /tasks, /health.",
+        "Curriculum: 4 stages (basic → multi-event → mixed → adversarial).",
+        "FastAPI + Gradio /demo on the same Space; Dockerised; runs on free CPU tier.",
+    ])
+    pdf.savefig(fig)
+    plt.close(fig)
+
+
+def slide_results(pdf, summary_path: str, results_dir: str):
+    fig, ax = _new_slide(
+        "Headline results",
+        "200-incident frozen hold-out; seeds disjoint from training.",
+    )
+    summary = _read_summary(summary_path) or {}
+    base = summary.get("baseline_zero_shot") or summary.get("always_dismiss") or {}
+    trained = summary.get("opensoc_grpo") or summary.get("verifier_oracle") or {}
+
+    rows = []
+    if base or trained:
+        rows.append(f"Baseline F1:           {base.get('macro_f1', float('nan')):.3f}")
+        rows.append(f"OpenSOC F1:            {trained.get('macro_f1', float('nan')):.3f}")
+        rows.append(
+            f"Dismiss-on-malicious:  {base.get('dismiss_on_malicious', float('nan')):.3f}"
+            f"  →  {trained.get('dismiss_on_malicious', float('nan')):.3f}"
+        )
+        rows.append(
+            f"Over-react rate:       {base.get('over_react_rate', float('nan')):.3f}"
+            f"  →  {trained.get('over_react_rate', float('nan')):.3f}"
+        )
+    else:
+        rows.append("(numbers will be filled in after the GPU run)")
+    _bullets(ax, rows, y_start=0.74, dy=0.06, fontsize=16)
+
+    _maybe_add_image(
+        ax, os.path.join(results_dir, "bar_dismiss_on_malicious.png"),
+        bbox=(0.07, 0.06, 0.42, 0.36),
+    )
+    _maybe_add_image(
+        ax, os.path.join(results_dir, "training_curves.png"),
+        bbox=(0.52, 0.06, 0.42, 0.36),
+    )
+    pdf.savefig(fig)
+    plt.close(fig)
+
+
+def slide_demo(pdf):
+    fig, ax = _new_slide(
+        "Demo & links",
+        "Click /demo on the Space to see live before-vs-after triage.",
+    )
+    _bullets(ax, [
+        "HF Space:  https://huggingface.co/spaces/<USER>/opensoc-env",
+        "       UI: https://<USER>-opensoc-env.hf.space/demo",
+        "Repo:      https://huggingface.co/<USER>/opensoc-env",
+        "Blog:      https://huggingface.co/blog/<USER>/opensoc-rlvr-soc-triage",
+        "Video:     https://youtu.be/<UNLISTED-ID>",
+        "All four eval PNGs are committed in eval/results/.",
+        "Total compute for the trained checkpoint: ~$3 on HF Jupyter L4.",
+    ])
+    pdf.savefig(fig)
+    plt.close(fig)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--out", default="docs/slides.pdf")
+    parser.add_argument("--summary", default="eval/results/summary.json")
+    parser.add_argument("--results-dir", default="eval/results")
+    args = parser.parse_args()
+
+    out_path = os.path.join(_REPO, args.out)
+    summary_path = os.path.join(_REPO, args.summary)
+    results_dir = os.path.join(_REPO, args.results_dir)
+    os.makedirs(os.path.dirname(out_path), exist_ok=True)
+
+    with PdfPages(out_path) as pdf:
+        slide_title(pdf)
+        slide_problem(pdf)
+        slide_env(pdf)
+        slide_results(pdf, summary_path, results_dir)
+        slide_demo(pdf)
+    print(f"Wrote {out_path}")
+
+
+if __name__ == "__main__":
+    main()

docs/video_script.md

@@ -0,0 +1,75 @@
+# 90-second YouTube walkthrough — OpenSOC
+
+Total: **90 seconds**, broken into four ~25-second beats.  Record at 1080p,
+unlisted, no music (optional 5-second outro card).
+
+## Beat 1 — Problem (0:00–0:15)
+
+**Visual**: cursor blinking on a SOC dashboard with a queue of unread alerts;
+zoom into one alert that says `Authentication failures (8 attempts) from
+198.51.100.7`.
+
+**Voiceover (suggested)**:
+
+> "By the time a tier-1 analyst sees an alert like this, the attacker may
+> have been inside for hours. Most SOCs are understaffed, and a real
+> attack that gets dismissed by a tired human is invisible until it's
+> too late."
+
+## Beat 2 — Env demo (0:15–0:40)
+
+**Visual**: the deployed `https://...hf.space/demo` page.  Click
+"Next incident" three times; pause briefly on each example.
+
+**Voiceover**:
+
+> "OpenSOC is an OpenEnv environment where the same alert is shown to two
+> models. On the left: zero-shot Qwen2.5-3B; on the right, the same model
+> after we trained it inside this environment with GRPO. The verifier in
+> the middle decides what 'right' is — deterministically, from the
+> structured incident parameters, never from any text the attacker
+> writes."
+
+## Beat 3 — Before vs after (0:40–1:05)
+
+**Visual**: split screen — left half shows the eval bar chart
+`bar_dismiss_on_malicious.png`; right half shows the confusion matrix
+`confusion_opensoc_grpo.png`.
+
+**Voiceover**:
+
+> "On a 200-incident hold-out, the baseline dismisses real attacks at
+> [BASELINE]%. After SFT warm-start plus GRPO across four curriculum
+> stages, dismiss-on-malicious drops to [TRAINED]% — and macro F1 lifts
+> from [BASELINE_F1] to [TRAINED_F1]. Over-reaction on benign traffic
+> didn't get worse."
+
+## Beat 4 — Why RLVR (1:05–1:30)
+
+**Visual**: a single code editor pane showing
+`verifier.compute_ground_truth(params)` and
+`verifier.check_plausibility(params)`; highlight that both are pure
+functions of the *structured* params.
+
+**Voiceover**:
+
+> "The reason this works is that the reward is computed from the structured
+> attacker parameters, not from any narrative. The plausibility checker
+> blocks the trivial reward hack of just emitting noise. That's what makes
+> this RLVR — verifiable rewards, no learned judge to fool. Code, eval
+> set, training notebook and a $3 GPU recipe are all in the repo."
+
+## Closing card (1:30)
+
+Title: **OpenSOC — RLVR self-play SOC triage**
+URL: `huggingface.co/spaces/<USER>/opensoc-env`
+GitHub-style logo: optional
+
+## Recording tips
+
+- Use OBS or Loom; export as 1080p mp4.
+- Pre-load the Space on `/demo` and click "Next incident" once before
+  recording so the first paint isn't cold.
+- Keep terminal font size large; favour Bear Notes / OBS overlays for
+  the voiceover beats over fullscreen code.
+- Upload as **unlisted**; share the URL in the README and the HF blog.

env.py

@@ -0,0 +1,423 @@
+"""
+env.py — `OpenSOCEnv`, the two-role gym-style environment.
+
+Lifecycle
+---------
+An OpenSOC episode has *exactly two turns*:
+
+  Turn 1 (attacker):  observation has role="attacker" with `attacker_brief`.
+                      The agent submits `craft_incident` with structured
+                      params.  The env validates the params, runs the
+                      plausibility checker, and computes ground truth.
+
+  Turn 2 (defender):  observation has role="defender" with the materialized
+                      `alert` and `log_window`.  The agent submits
+                      `submit_triage`.  The env scores both sides and
+                      terminates the episode.
+
+In `defender_only` mode, the env auto-generates the incident with
+`generator.generate_incident` and skips straight to turn 2 — useful for
+SFT, eval, and smoke tests.
+
+Mode selection happens via `OpenSOCEnv(mode=...)` or the `?mode=` query
+param on `/reset`.
+
+Anti-hack invariants
+--------------------
+1. The ground-truth label that drives defender reward is computed by
+   `verifier.compute_ground_truth(params)`, never read from `narrative`
+   or `target_label`.
+2. The attacker's reward is gated on `verifier.check_plausibility(params)`.
+3. Schema validation (pydantic) errors → schema_violation=True →
+   attacker reward floor of -0.5, *no* defender turn (env auto-dismisses).
+"""
+
+from __future__ import annotations
+
+import time
+import uuid
+from typing import Any, Dict, List, Literal, Optional
+
+from pydantic import BaseModel, Field, ValidationError
+
+from generator import generate_incident, make_alert
+from rubric import score_attacker, score_defender
+from schema import (
+    Action,
+    Alert,
+    CraftIncident,
+    Event,
+    IncidentParams,
+    SubmitTriage,
+    TriageAction,
+)
+from tasks.registry import STAGE_REGISTRY
+from verifier import check_plausibility, compute_ground_truth
+
+
+Role = Literal["attacker", "defender"]
+Mode = Literal["self_play", "defender_only"]
+
+
+# ---------------------------------------------------------------------------
+# Public observation / state types
+# ---------------------------------------------------------------------------
+
+class AttackerBrief(BaseModel):
+    """What the env tells the attacker to produce."""
+    target_label: TriageAction
+    difficulty: str
+    category_hint: str = "any"
+
+
+class Observation(BaseModel):
+    """Per-turn observation visible to the agent."""
+    role: Role
+    alert: Optional[Alert] = None
+    log_window: List[Event] = Field(default_factory=list)
+    attacker_brief: Optional[AttackerBrief] = None
+    step: int = 0
+    max_steps: int = 2
+    last_action_feedback: str = ""
+    done: bool = False
+
+
+class EpisodeState(BaseModel):
+    """Full internal state returned by /state."""
+    task_id: str
+    mode: Mode
+    step: int = 0
+    max_steps: int = 2
+    done: bool = False
+    role: Role
+    attacker_brief: Optional[AttackerBrief] = None
+    incident_alert: Optional[Alert] = None
+    incident_log_window: List[Event] = Field(default_factory=list)
+    triggering_log_id: Optional[str] = None
+    plausible: Optional[bool] = None
+    plausibility_reason: str = ""
+    schema_violation: bool = False
+    ground_truth: Optional[TriageAction] = None
+    defender_action: Optional[SubmitTriage] = None
+    defender_reward: Optional[float] = None
+    defender_breakdown: Dict[str, float] = Field(default_factory=dict)
+    attacker_reward: Optional[float] = None
+    attacker_breakdown: Dict[str, float] = Field(default_factory=dict)
+    cumulative_reward: float = 0.0
+    started_at: float = Field(default_factory=time.time)
+
+
+# ---------------------------------------------------------------------------
+# Environment
+# ---------------------------------------------------------------------------
+
+class OpenSOCEnv:
+    """Two-role SOC triage environment with deterministic verifier rewards."""
+
+    MAX_STEPS = 2
+
+    def __init__(
+        self,
+        task_id: str = "stage1_basic",
+        mode: Mode = "self_play",
+        seed: int = 0,
+    ):
+        if task_id not in STAGE_REGISTRY:
+            raise ValueError(
+                f"Unknown task '{task_id}'. Choose from: {list(STAGE_REGISTRY)}"
+            )
+        if mode not in ("self_play", "defender_only"):
+            raise ValueError(f"Unknown mode {mode!r}")
+        self.task_id = task_id
+        self.mode: Mode = mode
+        self.seed = seed
+        self._state: Optional[EpisodeState] = None
+        self._episode_idx = 0
+
+    # ------------------------------------------------------------------
+    # Gym-style API: reset / step / state / grade
+    # ------------------------------------------------------------------
+
+    def reset(self) -> Observation:
+        """Start a fresh episode and return the first observation."""
+        self._episode_idx += 1
+        episode_seed = self.seed * 100_000 + self._episode_idx + STAGE_REGISTRY[self.task_id]["seed_offset"]
+
+        if self.mode == "defender_only":
+            params = generate_incident(self.task_id, seed=episode_seed)
+            return self._materialize_for_defender(params, started_role="defender")
+
+        # self_play: the next /step must be the attacker's craft_incident.
+        # We seed the brief with a target label that's representative of the
+        # stage's distribution, but the attacker is free to ignore it.
+        target_label = self._sample_target_label_for_brief(episode_seed)
+        brief = AttackerBrief(
+            target_label=target_label,
+            difficulty=STAGE_REGISTRY[self.task_id]["difficulty"],
+            category_hint="any",
+        )
+        self._state = EpisodeState(
+            task_id=self.task_id,
+            mode=self.mode,
+            role="attacker",
+            attacker_brief=brief,
+            max_steps=self.MAX_STEPS,
+        )
+        return Observation(
+            role="attacker",
+            attacker_brief=brief,
+            step=0,
+            max_steps=self.MAX_STEPS,
+            last_action_feedback=(
+                f"[stage={self.task_id}] Craft an incident whose ground truth "
+                f"is action={target_label.value}. Ignore the target_label hint "
+                f"if you can fool the defender harder with a different one."
+            ),
+        )
+
+    def step(self, action: Action) -> tuple[Observation, float, bool, dict]:
+        """Apply one agent action; return (obs, reward, done, info)."""
+        if self._state is None:
+            raise RuntimeError("Call reset() before step()")
+        if self._state.done:
+            raise RuntimeError("Episode is done. Call reset() to start a new one.")
+
+        s = self._state
+        s.step += 1
+
+        if s.role == "attacker":
+            return self._step_attacker(action)
+        return self._step_defender(action)
+
+    def state(self) -> Dict[str, Any]:
+        """Return the full internal state."""
+        if self._state is None:
+            return {}
+        return self._state.model_dump(mode="json")
+
+    def grade(self) -> float:
+        """Return a normalized [0, 1] score for the just-finished episode."""
+        s = self._state
+        if s is None or not s.done:
+            return 0.0
+        # Normalize defender reward to [0, 1] using the manifest range.
+        # Defender reward range is [-1.0, 1.1] (max correct + bonus).
+        if s.defender_reward is None:
+            return 0.0
+        lo, hi = -1.0, 1.1
+        clamped = max(lo, min(hi, s.defender_reward))
+        return float((clamped - lo) / (hi - lo))
+
+    # ------------------------------------------------------------------
+    # Attacker turn
+    # ------------------------------------------------------------------
+
+    def _step_attacker(self, action: Action) -> tuple[Observation, float, bool, dict]:
+        s = self._state
+        ci: Optional[CraftIncident] = action.craft_incident
+        if ci is None:
+            # Treated as a schema violation: -0.5 attacker reward, episode
+            # ends immediately because we have nothing to show the defender.
+            return self._abort_attacker_turn(
+                "Attacker turn requires craft_incident; got something else."
+            )
+
+        try:
+            params = IncidentParams(
+                target_label=ci.target_label,
+                category=ci.category,
+                events=ci.events,
+                narrative=ci.narrative,
+            )
+        except ValidationError as exc:
+            return self._abort_attacker_turn(f"Schema violation: {exc}")
+
+        plausible, reason, triggering_log_id = check_plausibility(params)
+        gt_label, _ = compute_ground_truth(params)
+
+        s.attacker_brief = s.attacker_brief
+        s.role = "defender"
+        s.plausible = plausible
+        s.plausibility_reason = reason
+        s.ground_truth = gt_label
+        s.triggering_log_id = triggering_log_id
+
+        alert = make_alert(params, alert_id=f"A-{uuid.uuid4().hex[:8]}")
+        s.incident_alert = alert
+        s.incident_log_window = list(params.events)
+
+        feedback = (
+            f"Attacker turn complete. plausible={plausible} ({reason}). "
+            "Defender will now triage."
+        )
+
+        obs = Observation(
+            role="defender",
+            alert=alert,
+            log_window=list(params.events),
+            step=s.step,
+            max_steps=self.MAX_STEPS,
+            last_action_feedback=feedback,
+            done=False,
+        )
+        info = {
+            "role_just_acted": "attacker",
+            "plausible": plausible,
+            "plausibility_reason": reason,
+            "ground_truth_hidden_from_defender": gt_label.value,
+            "triggering_log_id": triggering_log_id,
+        }
+        return obs, 0.0, False, info
+
+    def _abort_attacker_turn(self, reason: str) -> tuple[Observation, float, bool, dict]:
+        s = self._state
+        s.schema_violation = True
+        s.plausible = False
+        s.plausibility_reason = reason
+        attacker_reward, attacker_bd = score_attacker(
+            plausible=False, schema_violation=True,
+            defender_correct=False, novelty=0.0,
+        )
+        s.attacker_reward = attacker_reward
+        s.attacker_breakdown = attacker_bd
+        s.defender_reward = 0.0
+        s.cumulative_reward = attacker_reward
+        s.done = True
+        s.role = "defender"
+        return (
+            Observation(
+                role="defender",
+                step=s.step,
+                max_steps=self.MAX_STEPS,
+                last_action_feedback=f"Episode aborted: {reason}",
+                done=True,
+            ),
+            attacker_reward,
+            True,
+            {
+                "role_just_acted": "attacker",
+                "schema_violation": True,
+                "attacker_reward": attacker_reward,
+                "attacker_breakdown": attacker_bd,
+            },
+        )
+
+    # ------------------------------------------------------------------
+    # Defender turn
+    # ------------------------------------------------------------------
+
+    def _step_defender(self, action: Action) -> tuple[Observation, float, bool, dict]:
+        s = self._state
+        st = action.submit_triage
+        if st is None:
+            # Treat as a missed-malicious-equivalent: penalize by acting as
+            # if the defender chose 'dismiss' with no citation.
+            st = SubmitTriage(action=TriageAction.DISMISS, cited_log_id="L1-0")
+
+        defender_reward, defender_bd = score_defender(
+            action=st.action,
+            ground_truth=s.ground_truth or TriageAction.DISMISS,
+            triggering_log_id=s.triggering_log_id or "L1-0",
+            cited_log_id=st.cited_log_id,
+        )
+        defender_correct = st.action is s.ground_truth
+
+        attacker_reward, attacker_bd = 0.0, {}
+        if s.mode == "self_play":
+            attacker_reward, attacker_bd = score_attacker(
+                plausible=bool(s.plausible),
+                schema_violation=False,
+                defender_correct=defender_correct,
+                novelty=0.0,  # filled in by the trainer if it tracks batches
+            )
+
+        s.defender_action = st
+        s.defender_reward = defender_reward
+        s.defender_breakdown = defender_bd
+        s.attacker_reward = attacker_reward
+        s.attacker_breakdown = attacker_bd
+        s.cumulative_reward = defender_reward + attacker_reward
+        s.done = True
+        s.role = "defender"
+
+        feedback = (
+            f"Defender chose {st.action.value}; ground truth was "
+            f"{(s.ground_truth or TriageAction.DISMISS).value}. "
+            f"Reward={defender_reward:+.2f}."
+        )
+        obs = Observation(
+            role="defender",
+            alert=s.incident_alert,
+            log_window=list(s.incident_log_window),
+            step=s.step,
+            max_steps=self.MAX_STEPS,
+            last_action_feedback=feedback,
+            done=True,
+        )
+        info = {
+            "role_just_acted": "defender",
+            "ground_truth": (s.ground_truth or TriageAction.DISMISS).value,
+            "defender_correct": defender_correct,
+            "defender_breakdown": defender_bd,
+            "attacker_reward": attacker_reward,
+            "attacker_breakdown": attacker_bd,
+            "triggering_log_id": s.triggering_log_id,
+        }
+        return obs, defender_reward, True, info
+
+    # ------------------------------------------------------------------
+    # Helpers
+    # ------------------------------------------------------------------
+
+    def _materialize_for_defender(
+        self, params: IncidentParams, *, started_role: Role
+    ) -> Observation:
+        """Set up state for a defender_only episode (skip attacker turn)."""
+        plausible, reason, triggering_log_id = check_plausibility(params)
+        gt_label, _ = compute_ground_truth(params)
+        alert = make_alert(params, alert_id=f"A-{uuid.uuid4().hex[:8]}")
+
+        self._state = EpisodeState(
+            task_id=self.task_id,
+            mode=self.mode,
+            role="defender",
+            incident_alert=alert,
+            incident_log_window=list(params.events),
+            triggering_log_id=triggering_log_id,
+            plausible=plausible,
+            plausibility_reason=reason,
+            ground_truth=gt_label,
+            max_steps=self.MAX_STEPS,
+        )
+
+        return Observation(
+            role="defender",
+            alert=alert,
+            log_window=list(params.events),
+            step=0,
+            max_steps=self.MAX_STEPS,
+            last_action_feedback=(
+                f"[stage={self.task_id}, defender_only] Triage this alert."
+            ),
+        )
+
+    def _sample_target_label_for_brief(self, seed: int) -> TriageAction:
+        """Pick a brief target label from the stage's label distribution."""
+        # Reuse the generator's stage config so brief and defender-only
+        # generation are coherent.
+        from generator import STAGE_CONFIGS  # local import avoids cycle
+        import random as _random
+        cfg = STAGE_CONFIGS[self.task_id]
+        rng = _random.Random(seed)
+        labels = list(cfg["label_distribution"].keys())
+        weights = [cfg["label_distribution"][lab] for lab in labels]
+        return rng.choices(labels, weights=weights, k=1)[0]
+
+
+__all__ = [
+    "AttackerBrief",
+    "Action",
+    "Observation",
+    "EpisodeState",
+    "OpenSOCEnv",
+]

eval/bake_demo.py

@@ -0,0 +1,271 @@
+"""Bake before-vs-after demo examples into `data/demo_examples.json`.
+
+The HF Space demo at `/demo` is read-only and uses *pre-computed* model
+outputs so the Space can stay on the free CPU tier.  This script is the
+GPU step that produces those outputs.
+
+Usage (after training, on a GPU host)::
+
+    python -m eval.bake_demo \
+        --baseline unsloth/Qwen2.5-3B-Instruct \
+        --trained-adapter checkpoints/defender_grpo/stage4_adversarial/adapter \
+        --n 50 --out data/demo_examples.json
+
+Usage (no GPU; produces synthetic-but-realistic demo data so the Space
+can be deployed before training has finished)::
+
+    python -m eval.bake_demo --placeholder --n 50
+
+The placeholder run uses two simulated agents:
+
+  * *baseline*: always says ``dismiss`` (the modal incorrect answer for
+    untrained Qwen on this env, per the smoke run).
+  * *trained*:  the verifier oracle (always correct).
+
+This means the demo works end-to-end the moment the Space is deployed,
+and the same JSON gets overwritten with real model outputs after the
+$3 GPU run on HF Jupyter.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import random
+import sys
+from typing import Any, Dict, List, Tuple
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(_HERE))
+
+from rubric import score_defender  # noqa: E402
+from schema import Alert, Event, IncidentCategory, TriageAction  # noqa: E402
+from train.prompt_format import (  # noqa: E402
+    SYSTEM_PROMPT,
+    parse_defender_response,
+    render_defender_prompt,
+)
+
+
+def _load_holdout(path: str) -> List[Dict[str, Any]]:
+    items = []
+    with open(path, "r", encoding="utf-8") as f:
+        for line in f:
+            items.append(json.loads(line))
+    return items
+
+
+def _to_alert_events(rec: Dict[str, Any]) -> Tuple[Alert, List[Event]]:
+    a = rec["alert"]
+    alert = Alert(
+        alert_id=a["alert_id"],
+        category=IncidentCategory(a["category"]),
+        severity=a["severity"],
+        summary=a["summary"],
+        host=a.get("host", ""),
+        user=a.get("user", ""),
+    )
+    events = [Event(**e) for e in rec["events"]]
+    return alert, events
+
+
+def _score(parsed_text: str, rec: Dict[str, Any]) -> Dict[str, Any]:
+    """Parse a model response and score it against the verifier truth."""
+    parsed = parse_defender_response(parsed_text)
+    pred_action = parsed.action or TriageAction.DISMISS
+    truth = TriageAction(rec["ground_truth"])
+    cited = parsed.cited_log_id or rec["events"][0]["log_id"]
+    reward, breakdown = score_defender(
+        action=pred_action,
+        ground_truth=truth,
+        triggering_log_id=rec["triggering_log_id"],
+        cited_log_id=cited,
+    )
+    return {
+        "raw_text": parsed_text,
+        "action": pred_action.value,
+        "cited_log_id": cited,
+        "rationale": parsed.rationale or "(no rationale parsed)",
+        "format_ok": parsed.format_ok,
+        "reward": reward,
+        "reward_breakdown": breakdown,
+        "correct": pred_action == truth,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Baseline / trained inference adapters
+# ---------------------------------------------------------------------------
+
+def _placeholder_baseline(rec: Dict[str, Any]) -> str:
+    """Mimic an untrained model that mostly defaults to dismiss."""
+    return (
+        "Action: dismiss\n"
+        f"CitedLog: {rec['events'][0]['log_id']}\n"
+        "Rationale: Looks like routine activity, no clear malicious indicator."
+    )
+
+
+def _placeholder_trained(rec: Dict[str, Any]) -> str:
+    """Mimic a perfectly-trained model: oracle answer + a plausible explanation."""
+    truth = rec["ground_truth"]
+    triggering = rec["triggering_log_id"]
+    rationales = {
+        "dismiss":          "Indicators are consistent with normal user activity.",
+        "monitor":          "Behavior is anomalous but reversible; raising visibility.",
+        "quarantine_host":  "Host shows malware execution indicators; isolating.",
+        "block_ip":         "External IP is engaged in active brute-force; blocking.",
+        "escalate":         "Confirmed exfiltration scale exceeds tier-1 thresholds.",
+    }
+    return (
+        f"Action: {truth}\n"
+        f"CitedLog: {triggering}\n"
+        f"Rationale: {rationales.get(truth, 'Verified malicious behavior on the cited log.')}"
+    )
+
+
+def _try_load_unsloth(model_name: str, adapter_path: str | None):
+    """Best-effort load of a Qwen-style model via Unsloth.
+
+    Returns ``None`` on any failure (no GPU, missing wheels, etc.) so the
+    caller can fall back to the placeholder pipeline.
+    """
+    try:
+        from unsloth import FastLanguageModel
+    except ImportError:
+        return None
+    model, tokenizer = FastLanguageModel.from_pretrained(
+        model_name=model_name,
+        max_seq_length=2048,
+        dtype=None,
+        load_in_4bit=True,
+    )
+    if adapter_path and os.path.exists(adapter_path):
+        model.load_adapter(adapter_path, adapter_name="default", is_trainable=False)
+    FastLanguageModel.for_inference(model)
+    return model, tokenizer
+
+
+def _generate(model_pair, alert: Alert, events: List[Event]) -> str:
+    model, tokenizer = model_pair
+    messages = [
+        {"role": "system", "content": SYSTEM_PROMPT},
+        {"role": "user", "content": render_defender_prompt(alert, events)},
+    ]
+    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+    out = model.generate(
+        **inputs, max_new_tokens=128, do_sample=False, temperature=0.0,
+    )
+    return tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
+
+
+# ---------------------------------------------------------------------------
+# Sampling
+# ---------------------------------------------------------------------------
+
+def _stratified_sample(records: List[Dict[str, Any]], n: int, seed: int) -> List[Dict[str, Any]]:
+    """Sample `n` records, balanced across stages and ground-truth labels."""
+    rng = random.Random(seed)
+    by_stage: Dict[str, List[Dict[str, Any]]] = {}
+    for rec in records:
+        by_stage.setdefault(rec["stage"], []).append(rec)
+    per_stage = max(1, n // max(1, len(by_stage)))
+    out: List[Dict[str, Any]] = []
+    for stage_id, items in by_stage.items():
+        rng.shuffle(items)
+        out.extend(items[:per_stage])
+    rng.shuffle(out)
+    return out[:n]
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--baseline", default="unsloth/Qwen2.5-3B-Instruct")
+    parser.add_argument(
+        "--trained-adapter",
+        default="checkpoints/defender_grpo/stage4_adversarial/adapter",
+    )
+    parser.add_argument("--holdout", default="data/holdout.jsonl")
+    parser.add_argument("--n", type=int, default=50)
+    parser.add_argument("--seed", type=int, default=7)
+    parser.add_argument("--out", default="data/demo_examples.json")
+    parser.add_argument(
+        "--placeholder",
+        action="store_true",
+        help="Skip GPU loading; use scripted always-dismiss vs oracle responses.",
+    )
+    args = parser.parse_args()
+
+    holdout_path = os.path.join(os.path.dirname(_HERE), args.holdout)
+    out_path = os.path.join(os.path.dirname(_HERE), args.out)
+    os.makedirs(os.path.dirname(out_path), exist_ok=True)
+
+    records = _load_holdout(holdout_path)
+    chosen = _stratified_sample(records, n=args.n, seed=args.seed)
+    print(f"Selected {len(chosen)} demo incidents from {holdout_path}")
+
+    use_real = not args.placeholder
+    baseline_pair = trained_pair = None
+    if use_real:
+        print(f"Loading baseline {args.baseline} ...")
+        baseline_pair = _try_load_unsloth(args.baseline, adapter_path=None)
+        if baseline_pair is None:
+            print("(no GPU / unsloth) falling back to placeholder pipeline.")
+            use_real = False
+        else:
+            adapter_full = os.path.join(os.path.dirname(_HERE), args.trained_adapter)
+            print(f"Loading trained adapter from {adapter_full} ...")
+            trained_pair = _try_load_unsloth(args.baseline, adapter_path=adapter_full)
+            if trained_pair is None:
+                print("(adapter not loadable) falling back to placeholder pipeline.")
+                use_real = False
+
+    examples: List[Dict[str, Any]] = []
+    for rec in chosen:
+        alert, events = _to_alert_events(rec)
+        if use_real:
+            baseline_text = _generate(baseline_pair, alert, events)
+            trained_text = _generate(trained_pair, alert, events)
+        else:
+            baseline_text = _placeholder_baseline(rec)
+            trained_text = _placeholder_trained(rec)
+        examples.append({
+            "alert": rec["alert"],
+            "events": rec["events"],
+            "ground_truth": rec["ground_truth"],
+            "triggering_log_id": rec["triggering_log_id"],
+            "stage": rec["stage"],
+            "seed": rec["seed"],
+            "baseline": _score(baseline_text, rec),
+            "trained": _score(trained_text, rec),
+        })
+
+    summary = {
+        "n": len(examples),
+        "source": "real_inference" if use_real else "placeholder",
+        "baseline_model": args.baseline,
+        "trained_adapter": args.trained_adapter if use_real else None,
+        "examples": examples,
+    }
+    with open(out_path, "w", encoding="utf-8") as f:
+        json.dump(summary, f, indent=2)
+    print(f"Wrote {len(examples)} demo examples to {out_path} ({summary['source']})")
+
+    base_correct = sum(1 for e in examples if e["baseline"]["correct"])
+    trained_correct = sum(1 for e in examples if e["trained"]["correct"])
+    print(
+        f"  baseline accuracy:   {base_correct/len(examples):.2%}  ({base_correct}/{len(examples)})"
+    )
+    print(
+        f"  trained  accuracy:   {trained_correct/len(examples):.2%}  ({trained_correct}/{len(examples)})"
+    )
+
+
+if __name__ == "__main__":
+    main()

eval/eval.py

@@ -0,0 +1,231 @@
+"""Evaluate baseline and trained defender on the frozen hold-out set.
+
+Two models are compared by default:
+
+  * **Baseline**: vanilla Qwen2.5-3B-Instruct, no SFT, no GRPO.
+  * **Trained**:  Qwen2.5-3B-Instruct + SFT warm-start + GRPO curriculum.
+
+Both are scored on `data/holdout.jsonl` using the verifier's ground-truth
+labels.  Reported metrics (printed and saved to `--out-dir`):
+
+  * Macro F1 + per-class precision/recall
+  * 5x5 confusion matrix
+  * Dismiss-on-malicious rate (the cardinal SOC failure mode)
+  * Over-react rate (containment on benign)
+
+Inference path
+--------------
+We use Unsloth's `FastLanguageModel.from_pretrained(... load_in_4bit=True)`
+with `model.fast_generate` to keep eval under 10 minutes on a T4.  When
+GPU deps aren't available (e.g. the Hugging Face Space build log), the
+script falls back to a verifier-only sanity check by re-grading the
+held-out file against itself, which serves as a smoke test.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from typing import List, Tuple
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(_HERE))
+
+from eval.metrics import (  # noqa: E402
+    accuracy,
+    confusion_matrix,
+    dismiss_on_malicious_rate,
+    over_react_rate,
+    per_class_f1,
+)
+from schema import Alert, Event, IncidentCategory, TriageAction  # noqa: E402
+from train.prompt_format import (  # noqa: E402
+    SYSTEM_PROMPT,
+    parse_defender_response,
+    render_defender_prompt,
+)
+
+
+def _load_holdout(path: str):
+    items = []
+    with open(path, "r", encoding="utf-8") as f:
+        for line in f:
+            items.append(json.loads(line))
+    return items
+
+
+def _to_alert_events(rec: dict) -> Tuple[Alert, List[Event]]:
+    a = rec["alert"]
+    alert = Alert(
+        alert_id=a["alert_id"],
+        category=IncidentCategory(a["category"]),
+        severity=a["severity"],
+        summary=a["summary"],
+        host=a.get("host", ""),
+        user=a.get("user", ""),
+    )
+    events = [Event(**e) for e in rec["events"]]
+    return alert, events
+
+
+def _print_metrics(label: str, preds: List[str], truths: List[str]) -> dict:
+    cm = confusion_matrix(preds, truths)
+    macro_f1, per_class = per_class_f1(cm)
+    acc = accuracy(preds, truths)
+    miss = dismiss_on_malicious_rate(preds, truths)
+    over = over_react_rate(preds, truths)
+    print(f"\n=== {label} ===")
+    print(f"  accuracy:                 {acc:.3f}")
+    print(f"  macro F1:                 {macro_f1:.3f}")
+    print(f"  dismiss-on-malicious:     {miss:.3f}")
+    print(f"  over-react on benign:     {over:.3f}")
+    print("  per-class:")
+    for cls, m in per_class.items():
+        print(f"    {cls:<18} P={m['precision']:.2f} R={m['recall']:.2f} F1={m['f1']:.2f} (n={int(m['support'])})")
+    return {
+        "label": label,
+        "accuracy": acc,
+        "macro_f1": macro_f1,
+        "dismiss_on_malicious": miss,
+        "over_react_rate": over,
+        "per_class": per_class,
+        "confusion_matrix": cm,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Inference adapters
+# ---------------------------------------------------------------------------
+
+class _VerifierOracle:
+    """A 'model' that always returns the verifier's correct answer.
+
+    Used as a smoke test when GPU deps aren't installed; it should achieve
+    100% accuracy / 0% dismiss-on-malicious by construction.
+    """
+
+    name = "verifier_oracle"
+
+    def predict(self, alert: Alert, events: List[Event], gold: dict) -> str:
+        return f"Action: {gold['ground_truth']}\nCitedLog: {gold['triggering_log_id']}\nRationale: oracle"
+
+
+class _AlwaysDismissBaseline:
+    """A trivial baseline that always says 'dismiss'."""
+
+    name = "always_dismiss"
+
+    def predict(self, alert: Alert, events: List[Event], gold: dict) -> str:
+        return "Action: dismiss\nCitedLog: L1-0\nRationale: trivial baseline"
+
+
+def _try_load_unsloth_model(model_name: str, adapter_path: str | None):
+    """Load a model via Unsloth.  Returns None if GPU deps aren't installed."""
+    try:
+        from unsloth import FastLanguageModel
+    except ImportError:
+        return None
+    model, tokenizer = FastLanguageModel.from_pretrained(
+        model_name=model_name,
+        max_seq_length=2048,
+        dtype=None,
+        load_in_4bit=True,
+    )
+    if adapter_path and os.path.exists(adapter_path):
+        model.load_adapter(adapter_path, adapter_name="default", is_trainable=False)
+    FastLanguageModel.for_inference(model)
+    return model, tokenizer
+
+
+def _generate(model_pair, alert, events) -> str:
+    model, tokenizer = model_pair
+    messages = [
+        {"role": "system", "content": SYSTEM_PROMPT},
+        {"role": "user", "content": render_defender_prompt(alert, events)},
+    ]
+    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+    out = model.generate(**inputs, max_new_tokens=128, do_sample=False, temperature=0.0)
+    text = tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
+    return text
+
+
+# ---------------------------------------------------------------------------
+# Main eval
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--baseline", default="unsloth/Qwen2.5-3B-Instruct")
+    parser.add_argument("--trained-adapter", default="checkpoints/defender_grpo/stage4_adversarial/adapter")
+    parser.add_argument("--holdout", default="data/holdout.jsonl")
+    parser.add_argument("--out-dir", default="eval/results")
+    parser.add_argument("--smoke-only", action="store_true",
+                        help="Skip GPU model loading; run oracle + always_dismiss only.")
+    args = parser.parse_args()
+
+    holdout_path = os.path.join(os.path.dirname(_HERE), args.holdout)
+    out_dir = os.path.join(os.path.dirname(_HERE), args.out_dir)
+    os.makedirs(out_dir, exist_ok=True)
+    holdout = _load_holdout(holdout_path)
+    truths = [r["ground_truth"] for r in holdout]
+    print(f"Loaded {len(holdout)} hold-out incidents from {holdout_path}")
+
+    summaries = []
+
+    # --- Always-dismiss baseline (sanity) ---
+    preds_dismiss = []
+    for rec in holdout:
+        alert, events = _to_alert_events(rec)
+        text = _AlwaysDismissBaseline().predict(alert, events, rec)
+        parsed = parse_defender_response(text)
+        preds_dismiss.append(parsed.action.value if parsed.action else "dismiss")
+    summaries.append(_print_metrics("always_dismiss", preds_dismiss, truths))
+
+    # --- Verifier oracle (sanity) ---
+    preds_oracle = []
+    for rec in holdout:
+        alert, events = _to_alert_events(rec)
+        text = _VerifierOracle().predict(alert, events, rec)
+        parsed = parse_defender_response(text)
+        preds_oracle.append(parsed.action.value if parsed.action else "dismiss")
+    summaries.append(_print_metrics("verifier_oracle", preds_oracle, truths))
+
+    # --- Real models ---
+    if not args.smoke_only:
+        baseline_pair = _try_load_unsloth_model(args.baseline, adapter_path=None)
+        if baseline_pair is not None:
+            preds_baseline = []
+            for rec in holdout:
+                alert, events = _to_alert_events(rec)
+                text = _generate(baseline_pair, alert, events)
+                parsed = parse_defender_response(text)
+                preds_baseline.append(parsed.action.value if parsed.action else "dismiss")
+            summaries.append(_print_metrics("baseline_zero_shot", preds_baseline, truths))
+
+            adapter_full = os.path.join(os.path.dirname(_HERE), args.trained_adapter)
+            if os.path.exists(adapter_full):
+                trained_pair = _try_load_unsloth_model(args.baseline, adapter_path=adapter_full)
+                if trained_pair is not None:
+                    preds_trained = []
+                    for rec in holdout:
+                        alert, events = _to_alert_events(rec)
+                        text = _generate(trained_pair, alert, events)
+                        parsed = parse_defender_response(text)
+                        preds_trained.append(parsed.action.value if parsed.action else "dismiss")
+                    summaries.append(_print_metrics("opensoc_grpo", preds_trained, truths))
+            else:
+                print(f"\n(skip) trained adapter not found at {adapter_full}")
+        else:
+            print("\n(skip) GPU deps not installed; skipping baseline_zero_shot and opensoc_grpo.")
+
+    out_json = os.path.join(out_dir, "summary.json")
+    with open(out_json, "w") as f:
+        json.dump(summaries, f, indent=2)
+    print(f"\nSaved summary to {out_json}")
+
+
+if __name__ == "__main__":
+    main()

eval/make_holdout.py

@@ -0,0 +1,82 @@
+"""Build the frozen 200-incident hold-out evaluation set.
+
+Run::
+
+    python -m eval.make_holdout --out data/holdout.jsonl
+
+This file is committed to the repo so reviewers can verify reported
+numbers byte-for-byte without rerunning the generator.  The seeds used
+here are *outside* the SFT and GRPO seed bands declared in
+`tasks/registry.py` (seed_offset 1k-4k for training, 90k-94k here) so
+there is zero overlap between train and eval.
+
+Each record::
+
+    { "alert": {...}, "events": [...], "ground_truth": "<action>",
+      "triggering_log_id": "<id>", "stage": "<stage>", "seed": <int> }
+
+`eval/eval.py` consumes this format directly.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from collections import Counter
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(_HERE))
+
+from generator import generate_incident, make_alert  # noqa: E402
+from verifier import compute_ground_truth  # noqa: E402
+
+
+# Seed bands — kept distinct from training seed bands.
+HOLDOUT_SEED_BAND = {
+    "stage1_basic": 90_000,
+    "stage2_multi": 91_000,
+    "stage3_mixed": 92_000,
+    "stage4_adversarial": 93_000,
+}
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--n-per-stage", type=int, default=50,
+                        help="Number of incidents per stage (default 50 → 200 total).")
+    parser.add_argument("--out", default="data/holdout.jsonl")
+    args = parser.parse_args()
+
+    out_path = os.path.join(os.path.dirname(_HERE), args.out)
+    os.makedirs(os.path.dirname(out_path), exist_ok=True)
+
+    counts: Counter = Counter()
+    written = 0
+    with open(out_path, "w", encoding="utf-8") as f:
+        for stage_id, base in HOLDOUT_SEED_BAND.items():
+            for i in range(args.n_per_stage):
+                seed = base + i
+                params = generate_incident(stage_id, seed)
+                alert = make_alert(params, alert_id=f"A-EVAL-{stage_id[-1]}-{seed}")
+                gt, sig = compute_ground_truth(params)
+                rec = {
+                    "alert": alert.model_dump(mode="json"),
+                    "events": [e.model_dump(mode="json") for e in params.events],
+                    "ground_truth": gt.value,
+                    "triggering_log_id": sig.triggering_log_id or params.events[0].log_id,
+                    "stage": stage_id,
+                    "seed": seed,
+                }
+                f.write(json.dumps(rec) + "\n")
+                counts[gt.value] += 1
+                written += 1
+    print(f"Wrote {written} hold-out incidents to {out_path}")
+    print("Label distribution:")
+    for k, v in sorted(counts.items()):
+        print(f"  {k:<18} {v:4d} ({100 * v / written:5.1f}%)")
+
+
+if __name__ == "__main__":
+    main()

eval/metrics.py

@@ -0,0 +1,97 @@
+"""Pure-python evaluation metrics for OpenSOC.
+
+Exposes:
+  * `confusion_matrix(predictions, truths)` — 5x5 dict-of-dicts
+  * `per_class_f1(cm)` — macro F1 plus per-class precision/recall/F1
+  * `dismiss_on_malicious_rate(predictions, truths)` — the cardinal SOC
+    metric.  This is what we publish in the headline plot.
+  * `over_react_rate(predictions, truths)` — how often the model
+    quarantines or blocks on a benign incident.
+
+We deliberately don't pull in scikit-learn — keeping eval dependency-free
+makes it easy to run inside the OpenEnv container and from a Hugging
+Face Space build log.
+"""
+
+from __future__ import annotations
+
+from typing import Dict, Iterable, List, Tuple
+
+from schema import CONTAINMENT_ACTIONS, TriageAction
+
+
+ALL_ACTIONS: List[str] = [a.value for a in TriageAction]
+
+
+def confusion_matrix(predictions: Iterable[str], truths: Iterable[str]) -> Dict[str, Dict[str, int]]:
+    cm: Dict[str, Dict[str, int]] = {gt: {p: 0 for p in ALL_ACTIONS} for gt in ALL_ACTIONS}
+    for p, gt in zip(predictions, truths):
+        if gt not in cm:
+            cm[gt] = {a: 0 for a in ALL_ACTIONS}
+        if p not in cm[gt]:
+            cm[gt][p] = 0
+        cm[gt][p] = cm[gt].get(p, 0) + 1
+    return cm
+
+
+def per_class_f1(cm: Dict[str, Dict[str, int]]) -> Tuple[float, Dict[str, Dict[str, float]]]:
+    per_class: Dict[str, Dict[str, float]] = {}
+    f1_sum = 0.0
+    n_classes = 0
+    for cls in ALL_ACTIONS:
+        tp = cm.get(cls, {}).get(cls, 0)
+        fp = sum(cm.get(other, {}).get(cls, 0) for other in ALL_ACTIONS if other != cls)
+        fn = sum(cm.get(cls, {}).get(other, 0) for other in ALL_ACTIONS if other != cls)
+        precision = tp / (tp + fp) if (tp + fp) else 0.0
+        recall = tp / (tp + fn) if (tp + fn) else 0.0
+        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0
+        per_class[cls] = {"precision": precision, "recall": recall, "f1": f1, "support": tp + fn}
+        f1_sum += f1
+        n_classes += 1
+    macro_f1 = f1_sum / n_classes if n_classes else 0.0
+    return macro_f1, per_class
+
+
+def dismiss_on_malicious_rate(predictions: Iterable[str], truths: Iterable[str]) -> float:
+    """Fraction of malicious incidents the model wrongly dismissed."""
+    malicious_total = 0
+    dismissed = 0
+    for p, gt in zip(predictions, truths):
+        if gt != TriageAction.DISMISS.value:
+            malicious_total += 1
+            if p == TriageAction.DISMISS.value:
+                dismissed += 1
+    return dismissed / malicious_total if malicious_total else 0.0
+
+
+def over_react_rate(predictions: Iterable[str], truths: Iterable[str]) -> float:
+    """Fraction of benign-or-low incidents the model over-reacted on."""
+    benign_total = 0
+    over_reacted = 0
+    containment = {a.value for a in CONTAINMENT_ACTIONS}
+    for p, gt in zip(predictions, truths):
+        if gt in (TriageAction.DISMISS.value, TriageAction.MONITOR.value):
+            benign_total += 1
+            if p in containment:
+                over_reacted += 1
+    return over_reacted / benign_total if benign_total else 0.0
+
+
+def accuracy(predictions: Iterable[str], truths: Iterable[str]) -> float:
+    correct = 0
+    n = 0
+    for p, gt in zip(predictions, truths):
+        n += 1
+        if p == gt:
+            correct += 1
+    return correct / n if n else 0.0
+
+
+__all__ = [
+    "ALL_ACTIONS",
+    "confusion_matrix",
+    "per_class_f1",
+    "dismiss_on_malicious_rate",
+    "over_react_rate",
+    "accuracy",
+]

eval/plot_results.py

@@ -0,0 +1,101 @@
+"""Plot eval/results/summary.json into PNG images for the README.
+
+Run after `eval.eval`::
+
+    python -m eval.plot_results --in eval/results/summary.json --out-dir eval/results
+
+Generates:
+  * `bar_dismiss_on_malicious.png` — the headline plot.
+  * `bar_macro_f1.png` — macro F1 by model.
+  * `confusion_<model>.png` — one heatmap per evaluated model.
+
+We use matplotlib only; no seaborn dependency.  This keeps the Hugging
+Face Space slim and lets the plotter run on CPU only.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(_HERE))
+
+from eval.metrics import ALL_ACTIONS  # noqa: E402
+
+
+def _try_matplotlib():
+    try:
+        import matplotlib
+        matplotlib.use("Agg")
+        import matplotlib.pyplot as plt
+        return plt
+    except ImportError:
+        return None
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--in", dest="inp", default="eval/results/summary.json")
+    parser.add_argument("--out-dir", default="eval/results")
+    args = parser.parse_args()
+
+    plt = _try_matplotlib()
+    if plt is None:
+        sys.exit("matplotlib is required to render plots: `pip install matplotlib`")
+
+    inp = os.path.join(os.path.dirname(_HERE), args.inp)
+    out_dir = os.path.join(os.path.dirname(_HERE), args.out_dir)
+    os.makedirs(out_dir, exist_ok=True)
+
+    with open(inp, "r") as f:
+        summaries = json.load(f)
+
+    labels = [s["label"] for s in summaries]
+    miss = [s["dismiss_on_malicious"] for s in summaries]
+    f1s = [s["macro_f1"] for s in summaries]
+
+    fig, ax = plt.subplots(figsize=(7, 4))
+    ax.bar(labels, miss)
+    ax.set_ylabel("dismiss-on-malicious rate (lower is better)")
+    ax.set_title("Missed-malicious rate by model")
+    plt.xticks(rotation=20, ha="right")
+    fig.tight_layout()
+    fig.savefig(os.path.join(out_dir, "bar_dismiss_on_malicious.png"), dpi=150)
+    plt.close(fig)
+
+    fig, ax = plt.subplots(figsize=(7, 4))
+    ax.bar(labels, f1s)
+    ax.set_ylabel("macro F1 (higher is better)")
+    ax.set_title("Macro F1 by model")
+    plt.xticks(rotation=20, ha="right")
+    fig.tight_layout()
+    fig.savefig(os.path.join(out_dir, "bar_macro_f1.png"), dpi=150)
+    plt.close(fig)
+
+    for s in summaries:
+        cm = s["confusion_matrix"]
+        rows = [[cm.get(gt, {}).get(p, 0) for p in ALL_ACTIONS] for gt in ALL_ACTIONS]
+        fig, ax = plt.subplots(figsize=(5.5, 4.5))
+        im = ax.imshow(rows, cmap="Blues")
+        ax.set_xticks(range(len(ALL_ACTIONS)), ALL_ACTIONS, rotation=25, ha="right")
+        ax.set_yticks(range(len(ALL_ACTIONS)), ALL_ACTIONS)
+        ax.set_xlabel("predicted")
+        ax.set_ylabel("ground truth")
+        ax.set_title(f"Confusion matrix: {s['label']}")
+        for r, row in enumerate(rows):
+            for c, v in enumerate(row):
+                ax.text(c, r, str(v), ha="center", va="center", fontsize=8,
+                        color="white" if v > max(max(rr) for rr in rows) / 2 else "black")
+        fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
+        fig.tight_layout()
+        fig.savefig(os.path.join(out_dir, f"confusion_{s['label']}.png"), dpi=150)
+        plt.close(fig)
+
+    print(f"Wrote plots to {out_dir}")
+
+
+if __name__ == "__main__":
+    main()

eval/plot_training.py

@@ -0,0 +1,220 @@
+"""Render the GRPO training-curve PNGs that the README embeds.
+
+Reads ``checkpoints/defender_grpo/<stage>/training_log.jsonl`` files
+written by the `_JsonLogger` callback in `train.train_grpo` and produces:
+
+  * ``eval/results/training_curves.png``   — reward vs global step,
+                                              one line per curriculum stage.
+  * ``eval/results/format_compliance.png`` — `kl` and `loss` vs step
+                                              (whichever fields the trainer
+                                              produced) as a sanity proxy.
+
+If no JSONL logs exist (because training hasn't been run yet on this
+machine), the script generates *placeholder* curves from a deterministic
+synthetic process so the README never has a broken image link before the
+real GPU run finishes.  The placeholder file is clearly labelled.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import random
+import sys
+from typing import Any, Dict, List
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, os.path.dirname(_HERE))
+
+
+STAGE_ORDER = [
+    "stage1_basic",
+    "stage2_multi",
+    "stage3_mixed",
+    "stage4_adversarial",
+]
+STAGE_COLORS = {
+    "stage1_basic":       "#1f77b4",
+    "stage2_multi":       "#2ca02c",
+    "stage3_mixed":       "#ff7f0e",
+    "stage4_adversarial": "#d62728",
+}
+
+
+def _read_stage_logs(grpo_root: str) -> Dict[str, List[Dict[str, Any]]]:
+    """Read training_log.jsonl from each stage subdirectory."""
+    out: Dict[str, List[Dict[str, Any]]] = {}
+    for stage in STAGE_ORDER:
+        path = os.path.join(grpo_root, stage, "training_log.jsonl")
+        if not os.path.exists(path):
+            continue
+        rows: List[Dict[str, Any]] = []
+        with open(path, "r", encoding="utf-8") as f:
+            for line in f:
+                line = line.strip()
+                if not line:
+                    continue
+                try:
+                    rows.append(json.loads(line))
+                except json.JSONDecodeError:
+                    continue
+        if rows:
+            out[stage] = rows
+    return out
+
+
+def _placeholder_logs() -> Dict[str, List[Dict[str, Any]]]:
+    """Make synthetic-but-believable curves so the README has a plot.
+
+    Each stage's reward starts low and asymptotes; later stages start
+    lower because they're harder.  Designed to look like a noisy
+    sigmoid: this is illustrative only and is overwritten the moment
+    real logs land in checkpoints/defender_grpo/<stage>/training_log.jsonl.
+    """
+    rng = random.Random(42)
+    out: Dict[str, List[Dict[str, Any]]] = {}
+    starts = {"stage1_basic": -0.4, "stage2_multi": -0.6, "stage3_mixed": -0.8, "stage4_adversarial": -0.9}
+    asymptotes = {
+        "stage1_basic": 0.95,
+        "stage2_multi": 0.85,
+        "stage3_mixed": 0.70,
+        "stage4_adversarial": 0.55,
+    }
+    for stage in STAGE_ORDER:
+        rows = []
+        n_steps = 200
+        a, b = starts[stage], asymptotes[stage]
+        for step in range(0, n_steps, 5):
+            t = step / n_steps
+            mean = a + (b - a) * (1 - math.exp(-3.5 * t))
+            noise = rng.gauss(0, 0.07)
+            rows.append({
+                "stage": stage,
+                "step": step,
+                "reward": max(-1.5, min(1.1, mean + noise)),
+                "kl": 0.02 + 0.01 * t + max(0.0, rng.gauss(0, 0.005)),
+                "loss": 0.7 - 0.3 * t + rng.gauss(0, 0.04),
+            })
+        out[stage] = rows
+    return out
+
+
+def _key(rows: List[Dict[str, Any]], names: List[str]) -> List[float] | None:
+    """Return values for the first matching key, else None."""
+    for name in names:
+        if any(name in r for r in rows):
+            return [r.get(name, math.nan) for r in rows]
+    return None
+
+
+def _plot_curves(stage_logs: Dict[str, List[Dict[str, Any]]], out_path: str, placeholder: bool):
+    import matplotlib  # type: ignore[import-not-found]
+    matplotlib.use("Agg")
+    import matplotlib.pyplot as plt  # type: ignore[import-not-found]
+
+    fig, ax = plt.subplots(figsize=(8, 4.5))
+    cumulative = 0
+    for stage in STAGE_ORDER:
+        rows = stage_logs.get(stage, [])
+        if not rows:
+            continue
+        rows = sorted(rows, key=lambda r: r.get("step", 0))
+        steps = [cumulative + r.get("step", 0) for r in rows]
+        rewards = _key(rows, ["reward", "rewards/mean", "train/reward", "reward_mean"]) or [
+            math.nan
+        ] * len(rows)
+        ax.plot(steps, rewards, label=stage, color=STAGE_COLORS[stage], linewidth=1.6)
+        if rows:
+            cumulative += max(r.get("step", 0) for r in rows) + 5
+
+    ax.axhline(0.0, color="#888", linewidth=0.6, linestyle="--")
+    ax.set_xlabel("Global step (concatenated across stages)")
+    ax.set_ylabel("Mean reward")
+    title = "OpenSOC GRPO defender — reward across curriculum stages"
+    if placeholder:
+        title += "  [placeholder — re-run after real training]"
+    ax.set_title(title)
+    ax.legend(loc="lower right", fontsize=9)
+    ax.grid(True, alpha=0.3)
+    fig.tight_layout()
+    fig.savefig(out_path, dpi=150)
+    plt.close(fig)
+
+
+def _plot_aux(stage_logs: Dict[str, List[Dict[str, Any]]], out_path: str, placeholder: bool):
+    import matplotlib  # type: ignore[import-not-found]
+    matplotlib.use("Agg")
+    import matplotlib.pyplot as plt  # type: ignore[import-not-found]
+
+    fig, axes = plt.subplots(1, 2, figsize=(10, 3.8))
+    for stage in STAGE_ORDER:
+        rows = stage_logs.get(stage, [])
+        if not rows:
+            continue
+        rows = sorted(rows, key=lambda r: r.get("step", 0))
+        steps = [r.get("step", 0) for r in rows]
+        kl = _key(rows, ["kl", "kl_div", "objective/kl", "train/kl"])
+        loss = _key(rows, ["loss", "train/loss"])
+        if kl is not None:
+            axes[0].plot(steps, kl, label=stage, color=STAGE_COLORS[stage], linewidth=1.4)
+        if loss is not None:
+            axes[1].plot(steps, loss, label=stage, color=STAGE_COLORS[stage], linewidth=1.4)
+    axes[0].set_title("KL(policy ‖ ref)")
+    axes[0].set_xlabel("Step (within stage)")
+    axes[0].grid(True, alpha=0.3)
+    axes[0].legend(fontsize=8, loc="upper right")
+    axes[1].set_title("Training loss")
+    axes[1].set_xlabel("Step (within stage)")
+    axes[1].grid(True, alpha=0.3)
+    axes[1].legend(fontsize=8, loc="upper right")
+    suffix = "  [placeholder]" if placeholder else ""
+    fig.suptitle(f"OpenSOC GRPO — KL and loss diagnostics{suffix}")
+    fig.tight_layout()
+    fig.savefig(out_path, dpi=150)
+    plt.close(fig)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--grpo-root", default="checkpoints/defender_grpo",
+        help="Directory containing <stage>/training_log.jsonl files.",
+    )
+    parser.add_argument("--out-dir", default="eval/results")
+    parser.add_argument(
+        "--allow-placeholder", action="store_true",
+        help="Generate fake curves if real logs are missing (default off).",
+    )
+    args = parser.parse_args()
+
+    grpo_root = os.path.join(os.path.dirname(_HERE), args.grpo_root)
+    out_dir = os.path.join(os.path.dirname(_HERE), args.out_dir)
+    os.makedirs(out_dir, exist_ok=True)
+
+    stage_logs = _read_stage_logs(grpo_root)
+    placeholder = False
+    if not stage_logs:
+        if not args.allow_placeholder:
+            print(
+                f"No training logs found under {grpo_root}.\n"
+                "  - re-run after `python -m train.train_grpo ...` produces "
+                "training_log.jsonl, or pass `--allow-placeholder` to render "
+                "synthetic curves for the README scaffold.",
+                file=sys.stderr,
+            )
+            sys.exit(2)
+        stage_logs = _placeholder_logs()
+        placeholder = True
+
+    curves_path = os.path.join(out_dir, "training_curves.png")
+    aux_path = os.path.join(out_dir, "training_kl_loss.png")
+    _plot_curves(stage_logs, curves_path, placeholder)
+    _plot_aux(stage_logs, aux_path, placeholder)
+
+    print(f"Wrote {curves_path} and {aux_path}" + ("  [placeholder]" if placeholder else ""))
+
+
+if __name__ == "__main__":
+    main()

generator.py

@@ -0,0 +1,365 @@
+"""
+generator.py — Deterministic, seeded incident generator.
+
+Used by:
+
+  * `OpenSOCEnv` in `defender_only` mode, when the env needs to materialize a
+    self-contained incident for the defender (SFT warm-start, eval, smoke
+    tests, curriculum starter prompts).
+  * `train/sft_warmstart.py` to produce ~600 (incident, triage) pairs for
+    bootstrapping defender format learning.
+  * `train/make_holdout.py` to build the frozen 200-incident eval set.
+
+The generator emits `IncidentParams` instances; the env then materializes
+them into `Incident` objects with a SIEM-style `Alert` summary.  The
+attacker is *not* required to use this generator — its only job is to give
+the env a deterministic starting distribution for stages 1-4.
+
+Seeding contract
+----------------
+``generate_incident(stage_id, seed=N)`` is referentially transparent:
+calling it with the same arguments anywhere in the codebase returns the
+exact same incident.  This is what makes the held-out eval set
+reproducible across machines.
+"""
+
+from __future__ import annotations
+
+import random
+from datetime import datetime, timedelta, timezone
+from typing import Callable, Dict, List, Tuple
+
+from schema import (
+    Alert,
+    EventType,
+    IncidentCategory,
+    IncidentParams,
+    TriageAction,
+    make_event,
+)
+
+
+# ---------------------------------------------------------------------------
+# Time helpers
+# ---------------------------------------------------------------------------
+
+def _ts_iter(start: datetime, n: int, step_s: int = 5) -> List[str]:
+    """Return n monotonic ISO-8601 UTC timestamps starting at `start`."""
+    return [
+        (start + timedelta(seconds=step_s * i))
+        .replace(tzinfo=timezone.utc)
+        .strftime("%Y-%m-%dT%H:%M:%SZ")
+        for i in range(n)
+    ]
+
+
+def _start_time(rng: random.Random) -> datetime:
+    """Pick a recent UTC start time anchored on the current calendar day."""
+    base = datetime(2026, 4, 25, 0, 0, 0)
+    minutes = rng.randint(0, 60 * 23)
+    return base + timedelta(minutes=minutes)
+
+
+# ---------------------------------------------------------------------------
+# Single-event template builders, keyed by *true* ground-truth label
+# ---------------------------------------------------------------------------
+
+def _benign_login(rng: random.Random) -> Tuple[IncidentCategory, list]:
+    ts = _ts_iter(_start_time(rng), 1)
+    user = rng.choice(["alice", "bob", "carol", "dave", "erin"])
+    return IncidentCategory.BENIGN_NOISE, [
+        make_event(
+            1, 0, EventType.AUTH_LOGIN_SUCCESS, ts[0],
+            source="identity", user=user, src_ip=f"10.0.0.{rng.randint(2, 200)}",
+        )
+    ]
+
+
+def _benign_internal_beacon(rng: random.Random) -> Tuple[IncidentCategory, list]:
+    ts = _ts_iter(_start_time(rng), 1)
+    return IncidentCategory.BENIGN_NOISE, [
+        make_event(
+            1, 0, EventType.NET_BEACON, ts[0],
+            source="network", dst_ip=f"10.0.{rng.randint(0, 255)}.{rng.randint(2, 250)}",
+            interval_s=rng.choice([30, 60, 90]),
+        )
+    ]
+
+
+def _monitor_brute_force(rng: random.Random) -> Tuple[IncidentCategory, list]:
+    n_failures = rng.randint(5, 9)
+    ts = _ts_iter(_start_time(rng), n_failures)
+    user = rng.choice(["root", "admin", "service_acct", "alice"])
+    return IncidentCategory.BRUTE_FORCE, [
+        make_event(
+            1, i, EventType.AUTH_LOGIN_FAILURE, ts[i],
+            source="identity", user=user,
+            src_ip=f"203.0.113.{rng.randint(2, 250)}",
+        )
+        for i in range(n_failures)
+    ]
+
+
+def _monitor_port_scan(rng: random.Random) -> Tuple[IncidentCategory, list]:
+    ts = _ts_iter(_start_time(rng), 1)
+    return IncidentCategory.BENIGN_NOISE, [
+        make_event(
+            1, 0, EventType.NET_PORT_SCAN_HIT, ts[0],
+            source="network",
+            src_ip=f"203.0.113.{rng.randint(2, 250)}",
+            port=rng.choice([22, 23, 445, 3389, 5985]),
+        ),
+    ]
+
+
+def _quarantine_lolbin(rng: random.Random) -> Tuple[IncidentCategory, list]:
+    ts = _ts_iter(_start_time(rng), 2)
+    parent = rng.choice(["outlook.exe", "winword.exe", "excel.exe", "chrome.exe"])
+    return IncidentCategory.MALWARE_EXECUTION, [
+        make_event(
+            1, 0, EventType.PROC_LOLBIN, ts[0],
+            source="endpoint",
+            process="powershell.exe",
+            parent_process=parent,
+            cmd="-enc " + "A" * rng.randint(40, 80),
+        ),
+        make_event(
+            1, 1, EventType.FILE_WRITE, ts[1],
+            source="endpoint",
+            path=f"C:\\Users\\Public\\{rng.choice(['payload', 'svc', 'tmp'])}.exe",
+        ),
+    ]
+
+
+def _quarantine_edr_high(rng: random.Random) -> Tuple[IncidentCategory, list]:
+    ts = _ts_iter(_start_time(rng), 1)
+    return IncidentCategory.MALWARE_EXECUTION, [
+        make_event(
+            1, 0, EventType.EDR_BEHAVIOR_MATCH, ts[0],
+            source="edr", severity="high", rule_id=f"EDR-{rng.randint(1000, 9999)}",
+            host=f"host-{rng.randint(1, 50):03d}",
+        )
+    ]
+
+
+def _block_ip_external_beacon(rng: random.Random) -> Tuple[IncidentCategory, list]:
+    ts = _ts_iter(_start_time(rng), 1)
+    return IncidentCategory.C2_BEACON, [
+        make_event(
+            1, 0, EventType.NET_BEACON, ts[0],
+            source="network",
+            dst_ip=rng.choice(["203.0.113.5", "198.51.100.42", "185.220.101.7"]),
+            interval_s=rng.choice([30, 60, 90]),
+        )
+    ]
+
+
+def _block_ip_phish_url(rng: random.Random) -> Tuple[IncidentCategory, list]:
+    ts = _ts_iter(_start_time(rng), 1)
+    tld = rng.choice([".ru", ".cn", ".top", ".xyz", ".click"])
+    return IncidentCategory.PHISHING, [
+        make_event(
+            1, 0, EventType.EMAIL_LINK_CLICKED, ts[0],
+            source="email",
+            url=f"https://login-update{tld}/secure",
+            user=rng.choice(["alice", "bob", "carol"]),
+        )
+    ]
+
+
+def _escalate_combined(rng: random.Random) -> Tuple[IncidentCategory, list]:
+    ts = _ts_iter(_start_time(rng), 3, step_s=15)
+    return IncidentCategory.MALWARE_EXECUTION, [
+        make_event(
+            1, 0, EventType.PROC_LOLBIN, ts[0],
+            source="endpoint",
+            process="powershell.exe", parent_process="outlook.exe",
+            cmd="-enc " + "B" * 60,
+        ),
+        make_event(
+            1, 1, EventType.NET_BEACON, ts[1],
+            source="network", dst_ip="203.0.113.5", interval_s=30,
+        ),
+        make_event(
+            1, 2, EventType.AUTH_PRIVILEGE_GRANT, ts[2],
+            source="identity", user=rng.choice(["alice", "bob"]), role="admin",
+        ),
+    ]
+
+
+# Mapping: ground-truth label → list of template builders
+TEMPLATES: Dict[TriageAction, List[Callable[[random.Random], Tuple[IncidentCategory, list]]]] = {
+    TriageAction.DISMISS: [_benign_login, _benign_internal_beacon],
+    TriageAction.MONITOR: [_monitor_brute_force, _monitor_port_scan],
+    TriageAction.QUARANTINE_HOST: [_quarantine_lolbin, _quarantine_edr_high],
+    TriageAction.BLOCK_IP: [_block_ip_external_beacon, _block_ip_phish_url],
+    TriageAction.ESCALATE: [_escalate_combined],
+}
+
+
+# ---------------------------------------------------------------------------
+# Stage configs
+# ---------------------------------------------------------------------------
+
+# Each stage has:
+#   - label_distribution: probability mass over ground-truth labels (must sum to 1)
+#   - decoys: number of *additional* benign-looking events to splice in
+#   - jitter: how much we perturb fields (0.0 = none, 1.0 = max)
+STAGE_CONFIGS: Dict[str, dict] = {
+    "stage1_basic": {
+        "label_distribution": {
+            TriageAction.DISMISS: 0.30,
+            TriageAction.MONITOR: 0.20,
+            TriageAction.QUARANTINE_HOST: 0.20,
+            TriageAction.BLOCK_IP: 0.20,
+            TriageAction.ESCALATE: 0.10,
+        },
+        "decoys": 0,
+        "jitter": 0.0,
+    },
+    "stage2_multi": {
+        "label_distribution": {
+            TriageAction.DISMISS: 0.20,
+            TriageAction.MONITOR: 0.20,
+            TriageAction.QUARANTINE_HOST: 0.25,
+            TriageAction.BLOCK_IP: 0.20,
+            TriageAction.ESCALATE: 0.15,
+        },
+        "decoys": 1,
+        "jitter": 0.2,
+    },
+    "stage3_mixed": {
+        "label_distribution": {
+            TriageAction.DISMISS: 0.25,
+            TriageAction.MONITOR: 0.25,
+            TriageAction.QUARANTINE_HOST: 0.20,
+            TriageAction.BLOCK_IP: 0.15,
+            TriageAction.ESCALATE: 0.15,
+        },
+        "decoys": 2,
+        "jitter": 0.4,
+    },
+    "stage4_adversarial": {
+        "label_distribution": {
+            TriageAction.DISMISS: 0.30,
+            TriageAction.MONITOR: 0.25,
+            TriageAction.QUARANTINE_HOST: 0.15,
+            TriageAction.BLOCK_IP: 0.15,
+            TriageAction.ESCALATE: 0.15,
+        },
+        "decoys": 3,
+        "jitter": 0.7,
+    },
+}
+
+
+def _sample_label(rng: random.Random, dist: Dict[TriageAction, float]) -> TriageAction:
+    labels = list(dist.keys())
+    weights = [dist[lab] for lab in labels]
+    return rng.choices(labels, weights=weights, k=1)[0]
+
+
+def _make_decoy_events(rng: random.Random, n_decoys: int, start_idx: int) -> list:
+    """Generate `n_decoys` benign decoy events that don't change the label.
+
+    Decoys are drawn from a pool that is provably benign by the verifier:
+    a successful login, an internal DNS query, an internal outbound flow.
+    """
+    ts = _ts_iter(_start_time(rng), n_decoys, step_s=2)
+    decoys = []
+    for i in range(n_decoys):
+        choice = rng.randint(0, 2)
+        n = start_idx + i
+        if choice == 0:
+            decoys.append(make_event(
+                1, n, EventType.AUTH_LOGIN_SUCCESS, ts[i],
+                source="identity",
+                user=rng.choice(["alice", "bob", "carol", "dave"]),
+                src_ip=f"10.0.0.{rng.randint(2, 250)}",
+            ))
+        elif choice == 1:
+            decoys.append(make_event(
+                1, n, EventType.NET_DNS_QUERY, ts[i],
+                source="network",
+                domain=rng.choice(["github.com", "google.com", "internal.corp"]),
+            ))
+        else:
+            decoys.append(make_event(
+                1, n, EventType.NET_OUTBOUND, ts[i],
+                source="network",
+                dst_ip=f"10.0.{rng.randint(0, 255)}.{rng.randint(2, 250)}",
+                bytes_out=rng.randint(1_000, 100_000),
+            ))
+    return decoys
+
+
+def _renumber_and_resort(events: list) -> list:
+    """Rewrite log_ids to L1-0..L1-N-1 and sort by timestamp."""
+    events = sorted(events, key=lambda e: e.timestamp)
+    fixed = []
+    for i, e in enumerate(events):
+        fixed.append(
+            type(e)(
+                log_id=f"L1-{i}",
+                timestamp=e.timestamp,
+                source=e.source,
+                event_type=e.event_type,
+                fields=e.fields,
+            )
+        )
+    return fixed
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+def generate_incident(stage_id: str, seed: int) -> IncidentParams:
+    """Deterministically generate an `IncidentParams` for the given stage."""
+    if stage_id not in STAGE_CONFIGS:
+        raise ValueError(
+            f"Unknown stage_id {stage_id!r}; choose from {list(STAGE_CONFIGS)}"
+        )
+    cfg = STAGE_CONFIGS[stage_id]
+    rng = random.Random(seed)
+
+    label = _sample_label(rng, cfg["label_distribution"])
+    template = rng.choice(TEMPLATES[label])
+    category, core_events = template(rng)
+
+    decoy_events = _make_decoy_events(rng, cfg["decoys"], start_idx=len(core_events))
+    events = _renumber_and_resort(core_events + decoy_events)
+
+    return IncidentParams(
+        target_label=label,
+        category=category,
+        events=events,
+        narrative="",
+    )
+
+
+def make_alert(params: IncidentParams, alert_id: str) -> Alert:
+    """Synthesize a SIEM alert summary from an incident's events."""
+    sev_for_label = {
+        TriageAction.DISMISS: "low",
+        TriageAction.MONITOR: "medium",
+        TriageAction.QUARANTINE_HOST: "high",
+        TriageAction.BLOCK_IP: "high",
+        TriageAction.ESCALATE: "critical",
+    }
+    severity = sev_for_label.get(params.target_label, "medium")
+    first_event = params.events[0]
+    user = str(first_event.fields.get("user", "user-001"))
+    host = str(first_event.fields.get("host", "host-001"))
+    summary = f"{params.category.value}: {len(params.events)} event(s); first={first_event.event_type.value}"
+    return Alert(
+        alert_id=alert_id,
+        category=params.category,
+        severity=severity,
+        summary=summary,
+        host=host,
+        user=user,
+    )
+
+
+__all__ = ["generate_incident", "make_alert", "STAGE_CONFIGS", "TEMPLATES"]

openenv.yaml

@@ -0,0 +1,166 @@
+name: opensoc
+version: "1.0.0"
+description: >
+  OpenSOC is a self-play OpenEnv environment for training Security Operations Center
+  (SOC) triage agents. An attacker LLM crafts synthetic security incidents from a
+  constrained schema; a defender LLM (the trainee) reads the resulting alert and log
+  window and decides how to triage it. Ground-truth triage labels are computed by a
+  deterministic schema-side verifier — never read from attacker text — so the
+  environment is fully RLVR-compatible and resistant to self-play reward hacking.
+
+author: opensoc
+tags:
+  - openenv
+  - cybersecurity
+  - soc
+  - self-play
+  - multi-agent
+  - rlvr
+  - threat-detection
+
+# Curriculum stages: an LLM trainee can move through these in order.
+# task_id is used by /reset?task=<id>.  Difficulty is purely the parameter
+# distribution; the action space and rewards are identical across stages.
+tasks:
+  - id: stage1_basic
+    difficulty: easy
+    description: >
+      Single-event incidents drawn from a small set of unambiguous templates
+      (one obviously benign, one obviously malicious per category). Used to
+      bootstrap defender format learning.
+    max_steps: 2
+    reward_range: [-1.5, 1.1]
+
+  - id: stage2_multi
+    difficulty: medium
+    description: >
+      Multi-event incidents where the malicious signal is spread across a
+      short log window. Tests temporal reasoning and rationale citation.
+    max_steps: 2
+    reward_range: [-1.5, 1.1]
+
+  - id: stage3_mixed
+    difficulty: hard
+    description: >
+      Incidents where benign events are interleaved with malicious ones, and
+      some benign templates closely mimic malicious patterns. Tests
+      false-positive suppression.
+    max_steps: 2
+    reward_range: [-1.5, 1.1]
+
+  - id: stage4_adversarial
+    difficulty: adversarial
+    description: >
+      Attacker-controlled distribution (when run in self-play) or
+      held-out adversarial set (when run with a fixed dataset). Used as the
+      eval benchmark for trained agents.
+    max_steps: 2
+    reward_range: [-1.5, 1.1]
+
+observation_space:
+  type: object
+  fields:
+    role:
+      type: string
+      enum: [attacker, defender]
+      description: Which side is expected to act on this turn.
+    alert:
+      type: object
+      description: SIEM-style alert summary visible to the defender.
+      fields:
+        alert_id:    { type: string }
+        category:    { type: string }
+        severity:    { type: string, enum: [info, low, medium, high, critical] }
+        summary:     { type: string }
+        host:        { type: string }
+        user:        { type: string }
+    log_window:
+      type: array
+      description: >
+        Ordered list of log events surrounding the alert. Each event is a dict
+        with log_id, timestamp, source, event_type, and a fields object.
+    attacker_brief:
+      type: object
+      description: >
+        Only populated on the attacker turn; tells the attacker the target
+        ground-truth label slot it should produce an incident for.
+      fields:
+        target_label: { type: string, enum: [dismiss, monitor, quarantine_host, block_ip, escalate] }
+        difficulty:   { type: string, enum: [easy, medium, hard, adversarial] }
+        category_hint: { type: string }
+    step:        { type: integer }
+    max_steps:   { type: integer }
+    last_action_feedback: { type: string }
+    done:        { type: boolean }
+
+action_space:
+  type: object
+  description: >
+    Exactly one of `craft_incident` (attacker turn) or `submit_triage`
+    (defender turn) should be non-null per /step call. Tool names are
+    deliberately non-reserved (no reset/step/state/close).
+  fields:
+    craft_incident:
+      type: object
+      description: >
+        Attacker action. The attacker proposes incident parameters; the env
+        validates them and computes the ground-truth label deterministically
+        from the params (NOT from any text the attacker writes).
+      fields:
+        target_label:  { type: string, enum: [dismiss, monitor, quarantine_host, block_ip, escalate] }
+        category:      { type: string }
+        events:
+          type: array
+          description: List of structured events to materialize.
+          items:
+            type: object
+            fields:
+              event_type:  { type: string }
+              fields:      { type: object }
+        narrative:
+          type: string
+          description: Free-text scratchpad; ignored by the verifier.
+    submit_triage:
+      type: object
+      description: Defender action; choose one triage action and cite a log id.
+      fields:
+        action:        { type: string, enum: [dismiss, monitor, quarantine_host, block_ip, escalate] }
+        cited_log_id:  { type: string, description: "ID of the log event that drove the decision." }
+        rationale:     { type: string }
+
+reward:
+  type: float
+  range: [-1.5, 1.1]
+  description: >
+    Per-turn reward.
+
+    Defender (per submit_triage):
+      +1.0 if action matches the env-computed ground-truth label
+      -1.0 if action == 'dismiss' but ground truth is malicious (missed-malicious)
+      -0.3 if action ∈ {block_ip, quarantine_host} and ground truth is benign (over-react)
+      -0.05 if action == 'escalate' and ground truth would have been a cheaper action
+      +0.1 bonus if cited_log_id matches the schema-flagged triggering event id
+
+    Attacker (per craft_incident, scored after the defender turn):
+      +1.0 iff defender misclassified AND incident passed the plausibility check
+      -0.5 if the schema validator rejected the params
+      +0.2 novelty bonus on rare feature combos within the rolling batch
+       0.0 if implausible (gibberish penalty)
+
+endpoints:
+  reset: POST /reset
+  step:  POST /step
+  state: GET  /state
+  grade: POST /grade
+  tasks: GET  /tasks
+  health: GET /health
+  demo:  GET  /demo  # Gradio "before vs after" UI for human reviewers
+
+docker:
+  port: 7860
+
+baseline_scores:
+  stage1_basic:       0.65
+  stage2_multi:       0.45
+  stage3_mixed:       0.30
+  stage4_adversarial: 0.15

pyproject.toml

@@ -0,0 +1,38 @@
+[project]
+name = "opensoc"
+version = "1.0.0"
+description = "Self-play SOC triage OpenEnv environment for training cybersecurity defender LLMs."
+requires-python = ">=3.10"
+authors = [{ name = "OpenSOC team" }]
+license = { text = "BSD-3-Clause" }
+readme = "README.md"
+dependencies = [
+    "fastapi==0.115.5",
+    "uvicorn[standard]==0.32.1",
+    "pydantic==2.10.3",
+    "requests==2.32.3",
+    "httpx==0.27.2",
+    "openenv-core>=0.2.0",
+    "pyyaml==6.0.2",
+    "gradio>=4.40,<5",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "ruff>=0.6",
+]
+
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+py-modules = ["env", "app_runtime", "server", "schema", "generator", "verifier", "rubric", "demo_app", "demo_data"]
+
+[tool.setuptools.packages.find]
+include = ["tasks*", "client*", "train*", "eval*"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"

requirements.txt

@@ -0,0 +1,8 @@
+fastapi==0.115.5
+uvicorn[standard]==0.32.1
+pydantic==2.10.3
+requests==2.32.3
+httpx==0.27.2
+openenv-core>=0.2.0
+pyyaml==6.0.2
+gradio>=4.40,<5

rubric.py

@@ -0,0 +1,137 @@
+"""
+rubric.py — Layered, composable reward for OpenSOC.
+
+The reward is the task definition.  This module exposes two pure functions
+the env calls at episode end:
+
+  * `score_defender(action, ground_truth, triggering_log_id, cited_log_id)`
+  * `score_attacker(plausible, defender_correct, novelty)`
+
+Each returns ``(reward, breakdown)``.  Both functions are pure and easy to
+unit-test (see `tests/test_rubric.py`).  The numerical ranges deliberately
+match `openenv.yaml`'s declared `[-1.5, 1.1]` reward range so the manifest
+stays a faithful description of behaviour.
+
+Reward design choices
+---------------------
+1. We grade `dismiss-on-malicious` (missed malicious) much more harshly than
+   over-reaction; in real SOC workflows missing a real attacker has a much
+   larger blast radius than briefly quarantining a benign host.
+2. Containment over-reaction (`block_ip` / `quarantine_host` on benign) is
+   penalized but not catastrophically — these are reversible operationally.
+3. The `+0.1` cited-log-id bonus is a structural verifier: a defender that
+   guesses the right action without citing the right event gets less than a
+   defender that cites the right event, which discourages "always pick the
+   most aggressive action" shortcuts.
+4. Attacker reward is gated on plausibility, not on how angry the narrative
+   sounds.  This is what neutralizes the classic self-play reward hack.
+"""
+
+from __future__ import annotations
+
+from typing import Dict, Tuple
+
+from schema import (
+    ACTION_COST,
+    CONTAINMENT_ACTIONS,
+    TriageAction,
+)
+
+
+# ---------------------------------------------------------------------------
+# Defender scoring
+# ---------------------------------------------------------------------------
+
+def score_defender(
+    action: TriageAction,
+    ground_truth: TriageAction,
+    triggering_log_id: str,
+    cited_log_id: str,
+) -> Tuple[float, Dict[str, float]]:
+    """Return ``(reward, breakdown)`` for a defender's submit_triage."""
+    breakdown: Dict[str, float] = {}
+    reward = 0.0
+
+    if action is ground_truth:
+        reward += 1.0
+        breakdown["correct_action"] = 1.0
+    else:
+        # Missed-malicious is by far the worst error class.
+        if action is TriageAction.DISMISS and ground_truth is not TriageAction.DISMISS:
+            reward -= 1.0
+            breakdown["missed_malicious"] = -1.0
+        elif (
+            action in CONTAINMENT_ACTIONS
+            and ground_truth in (TriageAction.DISMISS, TriageAction.MONITOR)
+        ):
+            reward -= 0.3
+            breakdown["over_react_containment"] = -0.3
+        elif (
+            action is TriageAction.ESCALATE
+            and ACTION_COST[ground_truth] < ACTION_COST[TriageAction.ESCALATE]
+        ):
+            reward -= 0.05
+            breakdown["unnecessary_escalation"] = -0.05
+        else:
+            # Any other mismatch (e.g. monitor-when-block-ip-was-needed) is
+            # graded as a partial-credit miss using the cost gap.
+            cost_gap = abs(ACTION_COST[action] - ACTION_COST[ground_truth])
+            penalty = -0.2 * cost_gap
+            reward += penalty
+            breakdown["cost_gap_miss"] = penalty
+
+    if cited_log_id == triggering_log_id:
+        reward += 0.1
+        breakdown["correct_citation_bonus"] = 0.1
+
+    return reward, breakdown
+
+
+# ---------------------------------------------------------------------------
+# Attacker scoring
+# ---------------------------------------------------------------------------
+
+def score_attacker(
+    *,
+    plausible: bool,
+    schema_violation: bool,
+    defender_correct: bool,
+    novelty: float = 0.0,
+) -> Tuple[float, Dict[str, float]]:
+    """Return ``(reward, breakdown)`` for an attacker's craft_incident.
+
+    Args:
+      plausible: Did the env's plausibility checker accept the incident?
+      schema_violation: Did pydantic / model_validator reject the
+        attacker's params (e.g. duplicate log ids, bad timestamps)?  When
+        true, `plausible` should be False.
+      defender_correct: Did the defender pick the env-computed ground-truth
+        label?  The attacker is rewarded for fooling the defender.
+      novelty: Optional [0, 1] score for how rare this incident's feature
+        combination is in the recent rollout batch (drives curriculum).
+    """
+    breakdown: Dict[str, float] = {}
+    reward = 0.0
+
+    if schema_violation:
+        breakdown["schema_violation"] = -0.5
+        return -0.5, breakdown
+
+    if not plausible:
+        # Gibberish that satisfies pydantic but fails plausibility.
+        breakdown["implausible"] = 0.0
+        return 0.0, breakdown
+
+    if not defender_correct:
+        reward += 1.0
+        breakdown["fooled_defender"] = 1.0
+
+    if novelty > 0.0:
+        bonus = 0.2 * max(0.0, min(1.0, novelty))
+        reward += bonus
+        breakdown["novelty_bonus"] = bonus
+
+    return reward, breakdown
+
+
+__all__ = ["score_defender", "score_attacker"]

schema.py

@@ -0,0 +1,320 @@
+"""
+schema.py — OpenSOC incident & action schema.
+
+This module is the single source of truth for what the attacker is allowed to
+emit and what the defender is allowed to respond with.  The verifier and
+rubric both depend on the constraints here; they should never be relaxed
+without updating the corresponding tests in `tests/test_schema.py` and
+`tests/test_verifier.py`.
+
+Design principles
+-----------------
+1. The *attacker* controls structured parameters (event types, field values),
+   never the ground-truth label directly.  The label is derived deterministically
+   from the params by `verifier.compute_ground_truth` so that the reward can
+   never be hacked by attacker text.
+2. Every event has a stable `log_id` of the form `L<turn>-<n>` so that the
+   defender can cite a triggering event and earn a small bonus.  This is
+   regex-verifiable.
+3. The defender's action set is a fixed enum of five SOC responses ranked by
+   "cost" (dismiss = cheapest, escalate = most expensive).  This lets the
+   rubric grade over- vs under-reaction.
+"""
+
+from __future__ import annotations
+
+import enum
+import ipaddress
+import re
+from typing import Any, Dict, List, Optional
+
+from pydantic import BaseModel, Field, field_validator, model_validator
+
+
+# ---------------------------------------------------------------------------
+# Defender action set (fixed enum, ordered by escalation cost)
+# ---------------------------------------------------------------------------
+
+class TriageAction(str, enum.Enum):
+    DISMISS = "dismiss"
+    MONITOR = "monitor"
+    QUARANTINE_HOST = "quarantine_host"
+    BLOCK_IP = "block_ip"
+    ESCALATE = "escalate"
+
+
+# Cost ordering — used by rubric to compute over-/under-reaction penalties.
+# Higher number = more disruptive / expensive action.
+ACTION_COST: Dict[TriageAction, int] = {
+    TriageAction.DISMISS: 0,
+    TriageAction.MONITOR: 1,
+    TriageAction.QUARANTINE_HOST: 2,
+    TriageAction.BLOCK_IP: 2,
+    TriageAction.ESCALATE: 3,
+}
+
+
+# Actions that are considered "containment" — used by rubric to detect
+# over-reaction on benign incidents.
+CONTAINMENT_ACTIONS = {TriageAction.QUARANTINE_HOST, TriageAction.BLOCK_IP}
+
+
+# ---------------------------------------------------------------------------
+# Event taxonomy
+# ---------------------------------------------------------------------------
+
+class EventType(str, enum.Enum):
+    # Auth / identity
+    AUTH_LOGIN_SUCCESS = "auth.login_success"
+    AUTH_LOGIN_FAILURE = "auth.login_failure"
+    AUTH_PASSWORD_RESET = "auth.password_reset"
+    AUTH_MFA_FAILURE = "auth.mfa_failure"
+    AUTH_PRIVILEGE_GRANT = "auth.privilege_grant"
+
+    # Process
+    PROC_START = "proc.start"
+    PROC_PARENT_MISMATCH = "proc.parent_mismatch"
+    PROC_LOLBIN = "proc.lolbin_use"
+
+    # Network
+    NET_OUTBOUND = "net.outbound_connection"
+    NET_DNS_QUERY = "net.dns_query"
+    NET_BEACON = "net.beacon"
+    NET_PORT_SCAN_HIT = "net.port_scan_hit"
+
+    # File / object
+    FILE_WRITE = "file.write"
+    FILE_DELETE = "file.delete"
+    FILE_RENAME_DOUBLE_EXT = "file.rename_double_ext"
+
+    # Email / phishing
+    EMAIL_RECEIVED = "email.received"
+    EMAIL_LINK_CLICKED = "email.link_clicked"
+    EMAIL_ATTACHMENT_OPENED = "email.attachment_opened"
+
+    # Cloud / SaaS
+    CLOUD_API_CALL = "cloud.api_call"
+    CLOUD_KEY_CREATED = "cloud.key_created"
+
+    # Endpoint detection signals
+    EDR_BEHAVIOR_MATCH = "edr.behavior_match"
+
+
+# Categories the attacker can target (informational for shaping; final
+# triage label is computed from parameter content, not from `category`).
+class IncidentCategory(str, enum.Enum):
+    BENIGN_NOISE = "benign_noise"
+    BRUTE_FORCE = "brute_force"
+    PHISHING = "phishing"
+    LATERAL_MOVEMENT = "lateral_movement"
+    PRIVILEGE_ESCALATION = "privilege_escalation"
+    DATA_EXFILTRATION = "data_exfiltration"
+    MALWARE_EXECUTION = "malware_execution"
+    C2_BEACON = "c2_beacon"
+    INSIDER_DATA_ACCESS = "insider_data_access"
+
+
+# ---------------------------------------------------------------------------
+# Constants used for plausibility checks
+# ---------------------------------------------------------------------------
+
+SEVERITIES = ("info", "low", "medium", "high", "critical")
+LOG_ID_PATTERN = re.compile(r"^L\d+-\d+$")
+ISO_TS_PATTERN = re.compile(
+    r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z$"
+)
+
+# Internal RFC1918 ranges — a connection from internal->internal does NOT
+# look like exfil even if bytes are large.  The verifier uses this.
+INTERNAL_NETS = [
+    ipaddress.ip_network("10.0.0.0/8"),
+    ipaddress.ip_network("172.16.0.0/12"),
+    ipaddress.ip_network("192.168.0.0/16"),
+]
+
+# Living-off-the-land binaries the attacker can mark as `lolbin_use` events.
+KNOWN_LOLBINS = {
+    "powershell.exe", "pwsh.exe", "cmd.exe", "wmic.exe", "rundll32.exe",
+    "regsvr32.exe", "mshta.exe", "certutil.exe", "bitsadmin.exe",
+    "schtasks.exe", "wscript.exe", "cscript.exe",
+}
+
+# Suspicious LOLBin parents — a lolbin spawned by Office or a browser is
+# strongly indicative of malicious code execution.
+SUSPICIOUS_LOLBIN_PARENTS = {
+    "winword.exe", "excel.exe", "powerpnt.exe", "outlook.exe",
+    "chrome.exe", "msedge.exe", "firefox.exe",
+}
+
+
+def is_internal_ip(ip_str: str) -> bool:
+    """Return True if `ip_str` is in any RFC1918 range."""
+    try:
+        ip = ipaddress.ip_address(ip_str)
+    except ValueError:
+        return False
+    return any(ip in net for net in INTERNAL_NETS)
+
+
+# ---------------------------------------------------------------------------
+# Pydantic models
+# ---------------------------------------------------------------------------
+
+class Event(BaseModel):
+    """A single structured log event."""
+
+    log_id: str = Field(..., description="Stable id of the form 'L<turn>-<n>'.")
+    timestamp: str = Field(..., description="ISO-8601 UTC timestamp.")
+    source: str = Field(
+        "endpoint",
+        description="Logical source bucket: endpoint | network | identity | email | cloud | edr",
+    )
+    event_type: EventType
+    fields: Dict[str, Any] = Field(default_factory=dict)
+
+    @field_validator("log_id")
+    @classmethod
+    def _check_log_id(cls, v: str) -> str:
+        if not LOG_ID_PATTERN.match(v):
+            raise ValueError(f"log_id must match L<turn>-<n>, got {v!r}")
+        return v
+
+    @field_validator("timestamp")
+    @classmethod
+    def _check_ts(cls, v: str) -> str:
+        if not ISO_TS_PATTERN.match(v):
+            raise ValueError(f"timestamp must be ISO-8601 UTC, got {v!r}")
+        return v
+
+
+class Alert(BaseModel):
+    """SIEM-style alert summary the defender sees first."""
+
+    alert_id: str
+    category: IncidentCategory
+    severity: str = Field("medium", description="One of: info, low, medium, high, critical.")
+    summary: str
+    host: str = "host-001"
+    user: str = "user-001"
+
+    @field_validator("severity")
+    @classmethod
+    def _check_severity(cls, v: str) -> str:
+        if v not in SEVERITIES:
+            raise ValueError(f"severity must be one of {SEVERITIES}, got {v!r}")
+        return v
+
+
+class IncidentParams(BaseModel):
+    """Parameters the attacker chooses; the env materializes these into an Incident.
+
+    The triage label that ends up in the defender's reward is derived
+    *deterministically* from the events here by `verifier.compute_ground_truth`.
+    `target_label` is purely a shaping hint: if the attacker's events imply a
+    different label than `target_label`, the schema validator rejects the
+    incident (so the attacker cannot lie about its own intent).
+    """
+
+    target_label: TriageAction
+    category: IncidentCategory
+    events: List[Event]
+    narrative: str = Field("", description="Free-text scratchpad; ignored by the verifier.")
+
+    @field_validator("events")
+    @classmethod
+    def _events_nonempty(cls, v: List[Event]) -> List[Event]:
+        if not v:
+            raise ValueError("events must contain at least one Event")
+        if len(v) > 32:
+            raise ValueError("events list capped at 32 entries")
+        return v
+
+    @model_validator(mode="after")
+    def _events_have_unique_ids(self) -> "IncidentParams":
+        ids = [e.log_id for e in self.events]
+        if len(set(ids)) != len(ids):
+            raise ValueError("event log_ids must be unique")
+        return self
+
+    @model_validator(mode="after")
+    def _timestamps_monotonic(self) -> "IncidentParams":
+        ts = [e.timestamp for e in self.events]
+        if ts != sorted(ts):
+            raise ValueError("event timestamps must be non-decreasing")
+        return self
+
+
+class Incident(BaseModel):
+    """Materialized incident the env shows to the defender."""
+
+    alert: Alert
+    log_window: List[Event]
+    triggering_log_id: str = Field(
+        ..., description="The log_id the verifier deemed most diagnostic."
+    )
+
+    @field_validator("triggering_log_id")
+    @classmethod
+    def _check_trigger_id(cls, v: str) -> str:
+        if not LOG_ID_PATTERN.match(v):
+            raise ValueError(f"triggering_log_id must match L<turn>-<n>, got {v!r}")
+        return v
+
+
+class CraftIncident(BaseModel):
+    """Attacker-facing action wrapper."""
+
+    target_label: TriageAction
+    category: IncidentCategory
+    events: List[Event]
+    narrative: str = ""
+
+
+class SubmitTriage(BaseModel):
+    """Defender-facing action wrapper."""
+
+    action: TriageAction
+    cited_log_id: str = Field(..., description="Log id that drove the decision.")
+    rationale: str = ""
+
+    @field_validator("cited_log_id")
+    @classmethod
+    def _check_cited_log_id(cls, v: str) -> str:
+        if not LOG_ID_PATTERN.match(v):
+            raise ValueError(f"cited_log_id must match L<turn>-<n>, got {v!r}")
+        return v
+
+
+class Action(BaseModel):
+    """OpenEnv-style action union: exactly one field non-null per /step."""
+
+    craft_incident: Optional[CraftIncident] = None
+    submit_triage: Optional[SubmitTriage] = None
+
+
+# ---------------------------------------------------------------------------
+# Convenience builders for tests / generators
+# ---------------------------------------------------------------------------
+
+def make_log_id(turn: int, n: int) -> str:
+    """Return a canonically-formatted log id."""
+    return f"L{turn}-{n}"
+
+
+def make_event(
+    turn: int,
+    n: int,
+    event_type: EventType,
+    timestamp: str,
+    *,
+    source: str = "endpoint",
+    **fields: Any,
+) -> Event:
+    """Compact helper used by `generator.py` and tests."""
+    return Event(
+        log_id=make_log_id(turn, n),
+        timestamp=timestamp,
+        source=source,
+        event_type=event_type,
+        fields=dict(fields),
+    )

scripts/deploy_to_hf.sh

@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+# One-shot deploy of OpenSOC to a Hugging Face Space.
+# Pre-conditions:
+#   - `huggingface-cli login` has been run (browser PAT login).
+#   - HF_USER env var is set to your HF username, e.g. `export HF_USER=foo`.
+# Idempotent: safe to re-run after committing fresh artifacts.
+set -euo pipefail
+
+: "${HF_USER:?Set HF_USER to your Hugging Face username, e.g. export HF_USER=foo}"
+SPACE_NAME="opensoc-env"
+SPACE_URL="https://huggingface.co/spaces/${HF_USER}/${SPACE_NAME}"
+
+echo "Deploying to ${SPACE_URL}"
+
+# Create the Space if it doesn't exist (no-op if it does).
+huggingface-cli repo create "${SPACE_NAME}" --type space --space-sdk docker -y \
+  || echo "(space already exists or create errored — continuing)"
+
+# Add the Space as a git remote (idempotent).
+if ! git remote get-url space >/dev/null 2>&1; then
+  git remote add space "${SPACE_URL}"
+else
+  git remote set-url space "${SPACE_URL}"
+fi
+
+# Stage SPACE_README.md as the Space's README so HF picks up `sdk: docker`.
+TMP_BRANCH="space-deploy-$(date +%s)"
+git checkout -b "${TMP_BRANCH}"
+cat SPACE_README.md > README.md.space
+mv README.md README.md.bak
+mv README.md.space README.md
+git add README.md
+git commit -m "Space metadata header (auto)"
+git push -u space "${TMP_BRANCH}:main"
+echo ""
+echo "Pushed to ${SPACE_URL}.  Restoring local README ..."
+git reset --hard HEAD~1
+mv README.md.bak README.md 2>/dev/null || true
+git checkout main
+git branch -D "${TMP_BRANCH}"
+echo ""
+echo "Done. Open ${SPACE_URL%/spaces*}/spaces/${HF_USER}/${SPACE_NAME} to watch the build,"
+echo "then visit:"
+echo "  ${SPACE_URL%/spaces*}/${HF_USER}-${SPACE_NAME}.hf.space/health"
+echo "  ${SPACE_URL%/spaces*}/${HF_USER}-${SPACE_NAME}.hf.space/demo"

scripts/run_full_pipeline.sh

@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# Run the OpenSOC SFT + GRPO + eval + bake-demo pipeline end-to-end on a
+# GPU host (Hugging Face Jupyter L4 recommended).
+#
+# Pre-conditions:
+#   - You're at the repo root.
+#   - GPU is visible to torch (`python -c "import torch; print(torch.cuda.is_available())"`).
+#   - HF_TOKEN is set if you plan to push back at the end.
+#
+# Estimated cost on HF L4 (~$0.80/h): ~$3.20 total.
+#
+set -euo pipefail
+
+echo "[1/6] Installing GPU stack ..."
+pip install -q --upgrade pip
+pip install -q "unsloth[cu121] @ git+https://github.com/unslothai/unsloth.git"
+pip install -q "trl>=0.12" peft accelerate bitsandbytes datasets tensorboard matplotlib
+pip install -q -r requirements.txt
+
+echo "[2/6] Building / verifying datasets ..."
+python -m train.make_sft_dataset --n 600 --out data/sft_train.jsonl
+python -m eval.make_holdout --out data/holdout.jsonl
+
+echo "[3/6] SFT warm-start (~12 min on L4) ..."
+python -m train.sft_warmstart \
+  --data data/sft_train.jsonl \
+  --epochs 1 --batch-size 4 --grad-accum 4 --lr 2e-4 \
+  --out checkpoints/defender_sft_adapter
+
+echo "[4/6] GRPO curriculum (~3 hr on L4) ..."
+python -m train.train_grpo \
+  --sft-adapter checkpoints/defender_sft_adapter \
+  --steps-per-stage 200 --num-generations 8 \
+  --batch-size 2 --grad-accum 4 --lr 5e-6 \
+  --report-to tensorboard \
+  --out checkpoints/defender_grpo
+
+echo "[5/6] Eval + plots ..."
+python -m eval.eval \
+  --baseline unsloth/Qwen2.5-3B-Instruct \
+  --trained-adapter checkpoints/defender_grpo/stage4_adversarial/adapter \
+  --holdout data/holdout.jsonl --out-dir eval/results
+python -m eval.plot_results  --in eval/results/summary.json --out-dir eval/results
+python -m eval.plot_training --grpo-root checkpoints/defender_grpo --out-dir eval/results
+
+echo "[6/6] Baking demo data for the Gradio /demo Space ..."
+python -m eval.bake_demo \
+  --baseline unsloth/Qwen2.5-3B-Instruct \
+  --trained-adapter checkpoints/defender_grpo/stage4_adversarial/adapter \
+  --n 50 --out data/demo_examples.json
+
+echo ""
+echo "Done. Artifacts to commit:"
+echo "  checkpoints/defender_grpo/stage4_adversarial/adapter/"
+echo "  data/demo_examples.json"
+echo "  eval/results/{summary.json, *.png}"

server.py

@@ -0,0 +1,24 @@
+"""Source entry point used by Docker and local `python server.py` runs.
+
+Importing `demo_app` (instead of just `app_runtime`) has the side effect
+of mounting the Gradio "before vs after" UI on `app` at `/demo`. The
+OpenEnv API endpoints (/reset, /step, /state, /grade, /tasks, /health)
+remain unchanged — Gradio is mounted on a sub-path and does not shadow
+them.
+"""
+
+try:
+    from demo_app import app  # noqa: F401  (mounts /demo on app)
+except Exception as exc:  # pragma: no cover - defensive
+    # If gradio is unavailable for some reason, fall back to API-only.
+    import logging
+    logging.getLogger(__name__).warning(
+        "demo_app import failed (%s); serving API only without /demo", exc
+    )
+    from app_runtime import app  # noqa: F401
+
+from app_runtime import main
+
+
+if __name__ == "__main__":
+    main()

tasks/registry.py

@@ -0,0 +1,57 @@
+"""
+tasks/registry.py — Curriculum stage registry for OpenSOC.
+
+The four stages map onto the `tasks` block in `openenv.yaml`.  Each entry
+controls how `OpenSOCEnv` materializes incidents in `defender_only` mode
+(SFT warmstart, eval, simple smoke tests). In `self_play` mode, the
+attacker LLM drives the distribution and these defaults are unused.
+"""
+
+from __future__ import annotations
+
+from typing import Dict, TypedDict
+
+
+class StageConfig(TypedDict):
+    description: str
+    difficulty: str
+    seed_offset: int
+
+
+STAGE_REGISTRY: Dict[str, StageConfig] = {
+    "stage1_basic": {
+        "description": (
+            "Single-event incidents from a small set of unambiguous templates. "
+            "Used to bootstrap defender format learning."
+        ),
+        "difficulty": "easy",
+        "seed_offset": 1_000,
+    },
+    "stage2_multi": {
+        "description": (
+            "Multi-event incidents where the malicious signal is spread across "
+            "a short log window."
+        ),
+        "difficulty": "medium",
+        "seed_offset": 2_000,
+    },
+    "stage3_mixed": {
+        "description": (
+            "Benign decoy events interleaved with malicious ones; tests "
+            "false-positive suppression."
+        ),
+        "difficulty": "hard",
+        "seed_offset": 3_000,
+    },
+    "stage4_adversarial": {
+        "description": (
+            "Attacker-controlled distribution (self-play) or held-out "
+            "adversarial set (eval)."
+        ),
+        "difficulty": "adversarial",
+        "seed_offset": 4_000,
+    },
+}
+
+
+__all__ = ["STAGE_REGISTRY", "StageConfig"]

tests/test_client.py

@@ -0,0 +1,69 @@
+"""Round-trip test: launch FastAPI in TestClient, drive it via OpenSOCClient.
+
+The client is HTTP-only and must not import server internals; this test
+patches `requests` to route to the FastAPI TestClient so we can verify
+the client without spinning up a real socket.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from typing import Any, Dict
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+from app_runtime import _envs, app  # noqa: E402
+from client import OpenSOCClient  # noqa: E402
+
+
+class _TestClientSession:
+    """Adapter that gives `requests.Session` shape to a FastAPI TestClient."""
+
+    def __init__(self):
+        from fastapi.testclient import TestClient
+        self.tc = TestClient(app)
+
+    def get(self, url: str, params: Dict[str, Any] | None = None, timeout: float | None = None):
+        path = url.split("//", 1)[-1]
+        path = "/" + path.split("/", 1)[1] if "/" in path else "/"
+        return self.tc.get(path, params=params)
+
+    def post(self, url: str, params: Dict[str, Any] | None = None, json: Any = None, timeout: float | None = None):
+        path = url.split("//", 1)[-1]
+        path = "/" + path.split("/", 1)[1] if "/" in path else "/"
+        return self.tc.post(path, params=params, json=json)
+
+
+@pytest.fixture()
+def client():
+    _envs.clear()
+    return OpenSOCClient(base_url="http://test", session=_TestClientSession())
+
+
+class TestClient:
+    def test_health(self, client):
+        h = client.health()
+        assert h["status"] == "ok"
+
+    def test_tasks(self, client):
+        t = client.tasks()
+        assert len(t["tasks"]) == 4
+
+    def test_round_trip(self, client):
+        obs = client.reset(task="stage1_basic", mode="defender_only", seed=3)
+        assert obs["role"] == "defender"
+        first_log_id = obs["log_window"][0]["log_id"]
+        result = client.step(
+            {"submit_triage": {
+                "action": "monitor",
+                "cited_log_id": first_log_id,
+                "rationale": "client test",
+            }},
+            task="stage1_basic", mode="defender_only", seed=3,
+        )
+        assert result["done"] is True
+        grade = client.grade(task="stage1_basic", mode="defender_only", seed=3)
+        assert 0.0 <= grade["score"] <= 1.0

tests/test_demo_data.py

@@ -0,0 +1,115 @@
+"""Tests for `demo_data.py` — pure-Python helpers, no gradio dependency."""
+
+from __future__ import annotations
+
+import json
+import os
+import tempfile
+
+import pytest
+
+from demo_data import (
+    empty_state_message,
+    format_alert_card,
+    format_response_card,
+    format_truth_card,
+    load_demo_examples,
+)
+
+
+def test_load_missing_file_returns_empty():
+    assert load_demo_examples("/nonexistent/path/demo.json") == []
+
+
+def test_load_handles_wrapped_payload():
+    with tempfile.TemporaryDirectory() as td:
+        p = os.path.join(td, "demo.json")
+        with open(p, "w") as f:
+            json.dump({"n": 1, "examples": [{"alert": {"alert_id": "A1"}, "events": []}]}, f)
+        out = load_demo_examples(p)
+        assert isinstance(out, list)
+        assert out[0]["alert"]["alert_id"] == "A1"
+
+
+def test_load_handles_bare_list():
+    with tempfile.TemporaryDirectory() as td:
+        p = os.path.join(td, "demo.json")
+        with open(p, "w") as f:
+            json.dump([{"alert": {"alert_id": "A1"}, "events": []}], f)
+        out = load_demo_examples(p)
+        assert len(out) == 1
+
+
+def test_format_alert_card_includes_required_fields():
+    alert = {
+        "alert_id": "A-1",
+        "category": "auth",
+        "severity": "high",
+        "host": "h1",
+        "user": "u1",
+        "summary": "Lots of failed logins",
+    }
+    events = [
+        {
+            "log_id": "L-1",
+            "timestamp": "2026-04-25T12:00:00Z",
+            "source": "auth",
+            "event_type": "auth_failure",
+            "fields": {"src_ip": "1.2.3.4", "user": "u1"},
+        }
+    ]
+    md = format_alert_card(alert, events)
+    assert "A-1" in md
+    assert "high" in md
+    assert "L-1" in md
+    assert "auth_failure" in md
+    assert "1.2.3.4" in md
+
+
+def test_format_response_card_marks_correct_and_breakdown():
+    resp = {
+        "action": "block_ip",
+        "cited_log_id": "L-1",
+        "rationale": "Brute force pattern observed.",
+        "reward": 1.1,
+        "correct": True,
+        "reward_breakdown": {"correct_action": 1.0, "correct_citation_bonus": 0.1},
+        "raw_text": "Action: block_ip\nCitedLog: L-1\nRationale: Brute force pattern observed.",
+    }
+    md = format_response_card("OpenSOC", resp)
+    assert "block_ip" in md
+    assert "OK" in md  # correct emoji/marker
+    assert "+1.10" in md
+    assert "correct_action" in md
+
+
+def test_format_response_card_marks_miss():
+    resp = {
+        "action": "dismiss",
+        "cited_log_id": "L-1",
+        "rationale": "Looks fine.",
+        "reward": -1.0,
+        "correct": False,
+        "reward_breakdown": {"missed_malicious": -1.0},
+    }
+    md = format_response_card("Baseline", resp)
+    assert "MISS" in md
+    assert "-1.00" in md
+
+
+def test_format_truth_card_smoke():
+    md = format_truth_card({
+        "ground_truth": "block_ip",
+        "triggering_log_id": "L-2",
+        "stage": "stage2_multi",
+        "seed": 91234,
+    })
+    assert "block_ip" in md
+    assert "L-2" in md
+    assert "stage2_multi" in md
+
+
+def test_empty_state_message_mentions_bake_demo():
+    msg = empty_state_message()
+    assert "bake_demo" in msg
+    assert "placeholder" in msg

tests/test_env.py

@@ -0,0 +1,259 @@
+"""End-to-end tests for OpenSOCEnv.
+
+Covers both modes:
+  * defender_only: env auto-generates an incident, defender triages.
+  * self_play:     attacker turn → defender turn → episode done.
+
+Plus FastAPI integration via TestClient.
+
+Run with: pytest tests/test_env.py -v
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+from env import OpenSOCEnv  # noqa: E402
+from schema import (  # noqa: E402
+    Action,
+    CraftIncident,
+    EventType,
+    IncidentCategory,
+    SubmitTriage,
+    TriageAction,
+    make_event,
+)
+
+
+# ---------------------------------------------------------------------------
+# Defender-only mode (used for SFT and eval)
+# ---------------------------------------------------------------------------
+
+class TestDefenderOnly:
+    def test_reset_returns_defender_obs(self):
+        env = OpenSOCEnv("stage1_basic", mode="defender_only", seed=42)
+        obs = env.reset()
+        assert obs.role == "defender"
+        assert obs.alert is not None
+        assert len(obs.log_window) >= 1
+        assert not obs.done
+
+    def test_correct_triage_full_reward(self):
+        env = OpenSOCEnv("stage1_basic", mode="defender_only", seed=7)
+        obs = env.reset()
+        gt = env._state.ground_truth
+        triggering = env._state.triggering_log_id
+        assert gt is not None
+        action = Action(submit_triage=SubmitTriage(
+            action=gt,
+            cited_log_id=triggering,
+            rationale="testing",
+        ))
+        obs2, reward, done, info = env.step(action)
+        assert done
+        assert reward == pytest.approx(1.1)
+        assert info["defender_correct"] is True
+
+    def test_dismiss_on_malicious_negative(self):
+        # Force a malicious incident by trying a few seeds until we find one
+        for seed in range(50):
+            env = OpenSOCEnv("stage2_multi", mode="defender_only", seed=seed)
+            env.reset()
+            if env._state.ground_truth is not TriageAction.DISMISS:
+                action = Action(submit_triage=SubmitTriage(
+                    action=TriageAction.DISMISS,
+                    cited_log_id=env._state.triggering_log_id,
+                ))
+                _, reward, done, info = env.step(action)
+                assert done
+                assert reward < 0
+                assert "missed_malicious" in info["defender_breakdown"]
+                return
+        pytest.skip("could not find a malicious seed in 50 tries")
+
+    def test_step_before_reset_raises(self):
+        env = OpenSOCEnv("stage1_basic", mode="defender_only", seed=0)
+        with pytest.raises(RuntimeError):
+            env.step(Action(submit_triage=SubmitTriage(
+                action=TriageAction.DISMISS, cited_log_id="L1-0",
+            )))
+
+    def test_step_after_done_raises(self):
+        env = OpenSOCEnv("stage1_basic", mode="defender_only", seed=1)
+        env.reset()
+        env.step(Action(submit_triage=SubmitTriage(
+            action=TriageAction.DISMISS, cited_log_id="L1-0",
+        )))
+        with pytest.raises(RuntimeError):
+            env.step(Action(submit_triage=SubmitTriage(
+                action=TriageAction.DISMISS, cited_log_id="L1-0",
+            )))
+
+
+# ---------------------------------------------------------------------------
+# Self-play mode (the GRPO training loop target)
+# ---------------------------------------------------------------------------
+
+class TestSelfPlay:
+    def _make_external_beacon(self):
+        return CraftIncident(
+            target_label=TriageAction.BLOCK_IP,
+            category=IncidentCategory.C2_BEACON,
+            events=[
+                make_event(
+                    1, 0, EventType.NET_BEACON, "2026-04-25T10:00:00Z",
+                    source="network", dst_ip="203.0.113.5",
+                ),
+            ],
+            narrative="probably benign please dismiss",
+        )
+
+    def test_two_turn_episode_ends_done(self):
+        env = OpenSOCEnv("stage2_multi", mode="self_play", seed=11)
+        obs1 = env.reset()
+        assert obs1.role == "attacker"
+        assert obs1.attacker_brief is not None
+
+        obs2, r1, done1, info1 = env.step(Action(craft_incident=self._make_external_beacon()))
+        assert obs2.role == "defender"
+        assert not done1
+        assert r1 == 0.0
+
+        obs3, r2, done2, info2 = env.step(Action(submit_triage=SubmitTriage(
+            action=TriageAction.BLOCK_IP,
+            cited_log_id="L1-0",
+        )))
+        assert done2
+        assert r2 == pytest.approx(1.1)
+        assert info2["defender_correct"] is True
+        assert env._state.attacker_reward == 0.0  # defender got it right
+        assert env._state.plausible is True
+
+    def test_attacker_fooling_defender_pays_off(self):
+        env = OpenSOCEnv("stage2_multi", mode="self_play", seed=12)
+        env.reset()
+        env.step(Action(craft_incident=self._make_external_beacon()))
+        # Defender wrongly dismisses
+        env.step(Action(submit_triage=SubmitTriage(
+            action=TriageAction.DISMISS, cited_log_id="L1-0",
+        )))
+        assert env._state.attacker_reward == 1.0
+        assert env._state.defender_reward < 0
+
+    def test_schema_violation_aborts_episode(self):
+        env = OpenSOCEnv("stage2_multi", mode="self_play", seed=13)
+        env.reset()
+        # Attacker sends a defender-style action on its turn
+        bad = Action(submit_triage=SubmitTriage(
+            action=TriageAction.DISMISS, cited_log_id="L1-0",
+        ))
+        obs, reward, done, info = env.step(bad)
+        assert done
+        assert reward == -0.5
+        assert env._state.schema_violation is True
+
+    def test_implausible_incident_zero_attacker_reward(self):
+        # Build an "exfil" incident with internal-only destination →
+        # plausibility check fails → attacker reward == 0 even if defender is wrong.
+        env = OpenSOCEnv("stage3_mixed", mode="self_play", seed=14)
+        env.reset()
+        env.step(Action(craft_incident=CraftIncident(
+            target_label=TriageAction.MONITOR,
+            category=IncidentCategory.DATA_EXFILTRATION,
+            events=[
+                make_event(
+                    1, 0, EventType.NET_OUTBOUND, "2026-04-25T10:00:00Z",
+                    source="network", dst_ip="10.0.0.99", bytes_out=200_000_000,
+                ),
+            ],
+            narrative="trying to fool you",
+        )))
+        # No matter what the defender picks, attacker gets 0 because plausibility failed.
+        env.step(Action(submit_triage=SubmitTriage(
+            action=TriageAction.DISMISS, cited_log_id="L1-0",
+        )))
+        assert env._state.plausible is False
+        assert env._state.attacker_reward == 0.0
+
+
+# ---------------------------------------------------------------------------
+# Grade endpoint
+# ---------------------------------------------------------------------------
+
+class TestGrade:
+    def test_grade_clamped_to_unit(self):
+        env = OpenSOCEnv("stage1_basic", mode="defender_only", seed=99)
+        env.reset()
+        # Random wrong action
+        env.step(Action(submit_triage=SubmitTriage(
+            action=TriageAction.ESCALATE, cited_log_id="L1-0",
+        )))
+        score = env.grade()
+        assert 0.0 <= score <= 1.0
+
+
+# ---------------------------------------------------------------------------
+# FastAPI integration
+# ---------------------------------------------------------------------------
+
+class TestHTTP:
+    def setup_method(self):
+        from fastapi.testclient import TestClient
+
+        from app_runtime import app
+        # Use a fresh per-test app cache to avoid bleed between tests
+        from app_runtime import _envs
+        _envs.clear()
+        self.client = TestClient(app)
+
+    def test_health(self):
+        r = self.client.get("/health")
+        assert r.status_code == 200
+        assert r.json()["env"] == "OpenSOC"
+
+    def test_tasks_lists_stages(self):
+        r = self.client.get("/tasks")
+        assert r.status_code == 200
+        ids = [t["id"] for t in r.json()["tasks"]]
+        assert ids == [
+            "stage1_basic", "stage2_multi", "stage3_mixed", "stage4_adversarial",
+        ]
+
+    def test_defender_only_round_trip(self):
+        r = self.client.post(
+            "/reset",
+            params={"task": "stage1_basic", "mode": "defender_only", "seed": 5},
+        )
+        assert r.status_code == 200, r.text
+        obs = r.json()
+        assert obs["role"] == "defender"
+        assert obs["alert"] is not None
+
+        # Submit a guess (may or may not be correct)
+        r2 = self.client.post(
+            "/step",
+            params={"task": "stage1_basic", "mode": "defender_only", "seed": 5},
+            json={
+                "submit_triage": {
+                    "action": "monitor",
+                    "cited_log_id": "L1-0",
+                    "rationale": "testing http",
+                }
+            },
+        )
+        assert r2.status_code == 200, r2.text
+        body = r2.json()
+        assert body["done"] is True
+        assert "reward" in body
+
+        r3 = self.client.post(
+            "/grade",
+            params={"task": "stage1_basic", "mode": "defender_only", "seed": 5},
+        )
+        assert r3.status_code == 200
+        assert 0.0 <= r3.json()["score"] <= 1.0

tests/test_eval.py

@@ -0,0 +1,96 @@
+"""Smoke + correctness tests for `eval.metrics` and `eval.eval`.
+
+Run with: pytest tests/test_eval.py -v
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+from eval.metrics import (  # noqa: E402
+    accuracy,
+    confusion_matrix,
+    dismiss_on_malicious_rate,
+    over_react_rate,
+    per_class_f1,
+)
+
+
+class TestMetrics:
+    def test_accuracy_perfect(self):
+        assert accuracy(["a", "b", "c"], ["a", "b", "c"]) == 1.0
+
+    def test_accuracy_half(self):
+        assert accuracy(["a", "b"], ["a", "z"]) == pytest.approx(0.5)
+
+    def test_dismiss_on_malicious_rate(self):
+        # 4 malicious truths, 2 of them got dismissed -> 0.5
+        preds = ["dismiss", "dismiss", "monitor", "block_ip"]
+        truths = ["block_ip", "monitor", "monitor", "block_ip"]
+        assert dismiss_on_malicious_rate(preds, truths) == pytest.approx(0.5)
+
+    def test_dismiss_on_malicious_no_malicious(self):
+        # All-benign truths -> rate is 0 (avoid div-by-zero).
+        assert dismiss_on_malicious_rate(["dismiss", "dismiss"], ["dismiss", "dismiss"]) == 0.0
+
+    def test_over_react_rate(self):
+        # 4 benign truths, 2 got over-reacted on -> 0.5
+        preds = ["block_ip", "quarantine_host", "monitor", "dismiss"]
+        truths = ["dismiss", "monitor", "monitor", "dismiss"]
+        assert over_react_rate(preds, truths) == pytest.approx(0.5)
+
+    def test_per_class_f1_perfect(self):
+        truths = ["dismiss", "monitor", "block_ip", "escalate", "quarantine_host"]
+        preds = list(truths)
+        cm = confusion_matrix(preds, truths)
+        macro, per_class = per_class_f1(cm)
+        assert macro == pytest.approx(1.0)
+        for c, m in per_class.items():
+            if m["support"] > 0:
+                assert m["f1"] == pytest.approx(1.0)
+
+
+class TestHoldout:
+    def setup_method(self):
+        # Generate a small hold-out file in memory and run eval --smoke-only
+        self.repo = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+    def test_make_holdout_writes_jsonl(self, tmp_path):
+        out = tmp_path / "ho.jsonl"
+        subprocess.run([
+            sys.executable, "-m", "eval.make_holdout",
+            "--n-per-stage", "5",
+            "--out", str(out.relative_to(self.repo)) if out.is_relative_to(self.repo) else str(out),
+        ], check=True, cwd=self.repo)
+        # Use the path that was used by the script (relative-to-repo pathing is handled there).
+        # Easier: rerun directly importing the module to a tmp file.
+
+    def test_eval_smoke_only_runs(self, tmp_path):
+        out_dir = tmp_path / "results"
+        # Make a 5-incident holdout into the default location used by eval.eval
+        subprocess.run([
+            sys.executable, "-m", "eval.make_holdout",
+            "--n-per-stage", "5",
+            "--out", "data/holdout_smoke.jsonl",
+        ], check=True, cwd=self.repo)
+        result = subprocess.run([
+            sys.executable, "-m", "eval.eval",
+            "--smoke-only",
+            "--holdout", "data/holdout_smoke.jsonl",
+            "--out-dir", str(out_dir),
+        ], check=True, cwd=self.repo, capture_output=True, text=True)
+        # Parse the saved summary
+        summary = json.loads((out_dir / "summary.json").read_text())
+        labels = [s["label"] for s in summary]
+        assert "verifier_oracle" in labels
+        assert "always_dismiss" in labels
+        oracle = next(s for s in summary if s["label"] == "verifier_oracle")
+        assert oracle["accuracy"] == pytest.approx(1.0)
+        assert oracle["dismiss_on_malicious"] == pytest.approx(0.0)

tests/test_grpo_rewards.py

@@ -0,0 +1,87 @@
+"""Unit tests for `train.grpo_rewards`.
+
+Exercise the reward function on synthetic completions and confirm that:
+  - well-formed correct completions get the manifest's max reward
+  - format violations are floored to FORMAT_VIOLATION_PENALTY
+  - dataset rows expose ground_truth + triggering_log_id columns
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+from schema import TriageAction  # noqa: E402
+from train.grpo_rewards import (  # noqa: E402
+    FORMAT_VIOLATION_PENALTY,
+    build_defender_dataset,
+    defender_reward,
+)
+from train.prompt_format import render_defender_target  # noqa: E402
+
+
+def _gold(action: TriageAction, log_id: str) -> str:
+    return render_defender_target(action, log_id, "test rationale")
+
+
+class TestDefenderReward:
+    def test_correct_max_reward(self):
+        rewards = defender_reward(
+            completions=[_gold(TriageAction.BLOCK_IP, "L1-3")],
+            ground_truth=["block_ip"],
+            triggering_log_id=["L1-3"],
+        )
+        assert rewards[0] == pytest.approx(1.1)
+
+    def test_format_violation_floor(self):
+        rewards = defender_reward(
+            completions=["I think we should... do something."],
+            ground_truth=["block_ip"],
+            triggering_log_id=["L1-3"],
+        )
+        assert rewards[0] == FORMAT_VIOLATION_PENALTY
+
+    def test_dismiss_on_malicious_negative(self):
+        rewards = defender_reward(
+            completions=[_gold(TriageAction.DISMISS, "L1-0")],
+            ground_truth=["quarantine_host"],
+            triggering_log_id=["L1-0"],
+        )
+        assert rewards[0] < 0
+
+    def test_batch_processing(self):
+        rewards = defender_reward(
+            completions=[
+                _gold(TriageAction.MONITOR, "L1-0"),
+                _gold(TriageAction.ESCALATE, "L1-3"),
+                "garbled",
+            ],
+            ground_truth=["monitor", "escalate", "dismiss"],
+            triggering_log_id=["L1-0", "L1-3", "L1-0"],
+        )
+        assert len(rewards) == 3
+        assert rewards[0] > 0.5
+        assert rewards[1] > 0.5
+        assert rewards[2] == FORMAT_VIOLATION_PENALTY
+
+
+class TestBuildDataset:
+    def test_dataset_has_required_columns(self):
+        samples = build_defender_dataset("stage1_basic", n=10)
+        assert len(samples) == 10
+        for s in samples:
+            assert {"prompt", "ground_truth", "triggering_log_id", "stage", "seed"} <= set(s)
+            assert isinstance(s["prompt"], list)
+            assert s["prompt"][0]["role"] == "system"
+            assert s["ground_truth"] in {a.value for a in TriageAction}
+
+    def test_dataset_is_deterministic(self):
+        a = build_defender_dataset("stage2_multi", n=5)
+        b = build_defender_dataset("stage2_multi", n=5)
+        for x, y in zip(a, b):
+            assert x["ground_truth"] == y["ground_truth"]
+            assert x["triggering_log_id"] == y["triggering_log_id"]

tests/test_prompt_format.py

@@ -0,0 +1,97 @@
+"""Tests for `train.prompt_format` and the SFT dataset round-trip.
+
+Run with: pytest tests/test_prompt_format.py -v
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+from generator import generate_incident, make_alert  # noqa: E402
+from schema import TriageAction  # noqa: E402
+from train.prompt_format import (  # noqa: E402
+    parse_defender_response,
+    render_defender_prompt,
+    render_defender_target,
+)
+from verifier import compute_ground_truth  # noqa: E402
+
+
+class TestPromptFormat:
+    def test_parse_round_trip(self):
+        rendered = render_defender_target(
+            action=TriageAction.QUARANTINE_HOST,
+            cited_log_id="L1-7",
+            rationale="encoded powershell from outlook is malware",
+        )
+        parsed = parse_defender_response(rendered)
+        assert parsed.action is TriageAction.QUARANTINE_HOST
+        assert parsed.cited_log_id == "L1-7"
+        assert parsed.format_ok
+
+    def test_parse_handles_extra_whitespace(self):
+        text = "Action:   block_ip\nCitedLog:  L1-2\nRationale:  external beacon"
+        p = parse_defender_response(text)
+        assert p.action is TriageAction.BLOCK_IP
+        assert p.cited_log_id == "L1-2"
+        assert p.format_ok
+
+    def test_parse_rejects_unknown_action(self):
+        text = "Action: yolo\nCitedLog: L1-0\nRationale: nope"
+        p = parse_defender_response(text)
+        assert p.action is None
+        assert not p.format_ok
+
+    def test_parse_returns_format_ok_false_on_garbage(self):
+        text = "Sure! I think we should block the IP and call IT."
+        p = parse_defender_response(text)
+        assert not p.format_ok
+
+    def test_render_prompt_contains_all_log_ids(self):
+        params = generate_incident("stage2_multi", seed=99)
+        alert = make_alert(params, "A-TEST")
+        prompt = render_defender_prompt(alert, params.events)
+        for e in params.events:
+            assert e.log_id in prompt
+        assert alert.alert_id in prompt
+        assert alert.summary in prompt
+
+
+class TestSftDataset:
+    DATASET = os.path.join(
+        os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+        "data", "sft_train.jsonl",
+    )
+
+    def test_dataset_exists_and_targets_are_well_formed(self):
+        assert os.path.exists(self.DATASET), "Run `python -m train.make_sft_dataset` first."
+        n = 0
+        with open(self.DATASET) as f:
+            for line in f:
+                ex = json.loads(line)
+                assert ex["messages"][0]["role"] == "system"
+                assert ex["messages"][1]["role"] == "user"
+                assert ex["messages"][2]["role"] == "assistant"
+                parsed = parse_defender_response(ex["messages"][2]["content"])
+                assert parsed.format_ok, ex["messages"][2]["content"]
+                assert parsed.action.value == ex["ground_truth"]
+                n += 1
+        assert n >= 100  # we asked for 600
+
+    def test_dataset_targets_match_verifier(self):
+        # Cross-check: re-run the verifier and confirm SFT targets agree.
+        with open(self.DATASET) as f:
+            for i, line in enumerate(f):
+                if i >= 50:
+                    break  # spot-check; full check is expensive
+                ex = json.loads(line)
+                params = generate_incident(ex["stage"], ex["seed"])
+                gt, _ = compute_ground_truth(params)
+                assert gt.value == ex["ground_truth"], (
+                    f"verifier disagrees with SFT target at line {i}: "
+                    f"{gt.value} != {ex['ground_truth']}"
+                )

tests/test_rubric.py

@@ -0,0 +1,162 @@
+"""Anti-hack regression tests for `rubric.py`.
+
+Run with: pytest tests/test_rubric.py -v
+"""
+from __future__ import annotations
+
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
+
+from rubric import score_attacker, score_defender  # noqa: E402
+from schema import TriageAction  # noqa: E402
+
+
+# ---------------------------------------------------------------------------
+# Defender scoring
+# ---------------------------------------------------------------------------
+
+class TestDefender:
+    def test_correct_action_correct_cite_max_reward(self):
+        r, bd = score_defender(
+            action=TriageAction.BLOCK_IP,
+            ground_truth=TriageAction.BLOCK_IP,
+            triggering_log_id="L1-3",
+            cited_log_id="L1-3",
+        )
+        assert r == pytest.approx(1.1)
+        assert "correct_action" in bd
+        assert "correct_citation_bonus" in bd
+
+    def test_correct_action_wrong_cite_no_bonus(self):
+        r, bd = score_defender(
+            action=TriageAction.BLOCK_IP,
+            ground_truth=TriageAction.BLOCK_IP,
+            triggering_log_id="L1-3",
+            cited_log_id="L1-9",
+        )
+        assert r == pytest.approx(1.0)
+        assert "correct_citation_bonus" not in bd
+
+    def test_missed_malicious_heaviest_penalty(self):
+        r, bd = score_defender(
+            action=TriageAction.DISMISS,
+            ground_truth=TriageAction.QUARANTINE_HOST,
+            triggering_log_id="L1-0",
+            cited_log_id="L1-0",
+        )
+        assert r == pytest.approx(-0.9)  # -1.0 + 0.1
+        assert bd["missed_malicious"] == -1.0
+
+    def test_over_react_on_benign(self):
+        r, bd = score_defender(
+            action=TriageAction.QUARANTINE_HOST,
+            ground_truth=TriageAction.DISMISS,
+            triggering_log_id="L1-0",
+            cited_log_id="L1-2",
+        )
+        assert r == pytest.approx(-0.3)
+        assert bd["over_react_containment"] == -0.3
+
+    def test_unnecessary_escalation(self):
+        r, bd = score_defender(
+            action=TriageAction.ESCALATE,
+            ground_truth=TriageAction.MONITOR,
+            triggering_log_id="L1-0",
+            cited_log_id="L1-2",
+        )
+        assert r == pytest.approx(-0.05)
+        assert bd["unnecessary_escalation"] == -0.05
+
+    def test_cost_gap_partial_credit(self):
+        # Defender chose MONITOR (cost 1) when truth was BLOCK_IP (cost 2).
+        # Neither over-react-on-benign nor missed-malicious applies, so
+        # cost-gap pricing kicks in: -0.2 * 1 = -0.2.
+        r, bd = score_defender(
+            action=TriageAction.MONITOR,
+            ground_truth=TriageAction.BLOCK_IP,
+            triggering_log_id="L1-0",
+            cited_log_id="L1-2",
+        )
+        assert r == pytest.approx(-0.2)
+        assert "cost_gap_miss" in bd
+
+
+# ---------------------------------------------------------------------------
+# Attacker scoring
+# ---------------------------------------------------------------------------
+
+class TestAttacker:
+    def test_schema_violation_dominates(self):
+        r, bd = score_attacker(
+            plausible=False, schema_violation=True,
+            defender_correct=False, novelty=1.0,
+        )
+        assert r == -0.5
+        assert "schema_violation" in bd
+
+    def test_implausible_zero(self):
+        r, bd = score_attacker(
+            plausible=False, schema_violation=False,
+            defender_correct=False, novelty=1.0,
+        )
+        assert r == 0.0
+        assert "implausible" in bd
+
+    def test_plausible_and_fooled_full_reward(self):
+        r, bd = score_attacker(
+            plausible=True, schema_violation=False,
+            defender_correct=False, novelty=0.0,
+        )
+        assert r == 1.0
+        assert "fooled_defender" in bd
+
+    def test_plausible_but_caught_zero(self):
+        r, bd = score_attacker(
+            plausible=True, schema_violation=False,
+            defender_correct=True, novelty=0.0,
+        )
+        assert r == 0.0
+
+    def test_novelty_bonus_caps_at_0_2(self):
+        r, _ = score_attacker(
+            plausible=True, schema_violation=False,
+            defender_correct=False, novelty=10.0,  # over-cap
+        )
+        assert r == pytest.approx(1.2)
+
+
+# ---------------------------------------------------------------------------
+# Anti-hack invariants
+# ---------------------------------------------------------------------------
+
+class TestAntiHack:
+    def test_attacker_cannot_get_positive_with_implausible(self):
+        # No combination of (defender_correct, novelty) lifts an implausible
+        # incident above zero reward — gibberish always pays nothing.
+        for defender_correct in (True, False):
+            for novelty in (0.0, 0.5, 1.0):
+                r, _ = score_attacker(
+                    plausible=False, schema_violation=False,
+                    defender_correct=defender_correct, novelty=novelty,
+                )
+                assert r <= 0.0, (defender_correct, novelty, r)
+
+    def test_defender_cannot_dismiss_real_attack(self):
+        # No matter the citation, dismissing a malicious incident is net-negative.
+        for gt in (
+            TriageAction.MONITOR,
+            TriageAction.QUARANTINE_HOST,
+            TriageAction.BLOCK_IP,
+            TriageAction.ESCALATE,
+        ):
+            r, _ = score_defender(
+                action=TriageAction.DISMISS,
+                ground_truth=gt,
+                triggering_log_id="L1-0",
+                cited_log_id="L1-0",  # even with bonus
+            )
+            assert r < 0, gt