fix: restore HF Space YAML frontmatter + update README with full content

README.md

@@ -11,44 +11,317 @@ tags:
   - reinforcement-learning
 ---
 
-# PM-Ops: Procedural Project-Management RL Environment
+# PM-Ops 🛠️
 
-A multi-app simulated environment (Ticketing, Codebase, Chat) featuring **procedural
-organizational drift**, designed for reinforcement learning training.
+> **Can a small model learn to operate inside a company it has never seen before?**
 
-## What makes it RL-shaped
+PM-Ops is a reinforcement learning benchmark and training environment where an LLM agent operates as a product manager inside a fully simulated software organization — navigating ticketing, codebases, and chat — using only the organization's own runbook as its guide.
 
-Every episode samples a fresh OrgConfig -- different label names, priority schemes,
-team-to-service mappings, and oncall channel names. Frontier LLMs fail because they
-rely on memorized conventions; a trained agent learns to read the runbook first.
+---
+
+## Links
+
+| Resource | Link |
+|---|---|
+| 🤗 **Live Environment** | [TheCrustaceans/Pm-ops — HuggingFace Space](https://huggingface.co/spaces/TheCrustaceans/Pm-ops) |
+| 📝 **Blog Post** | [BlogPost.mdx](https://huggingface.co/spaces/TheCrustaceans/Pm-ops/blob/main/BlogPost.mdx) |
+| 🎥 **Demo Video** | [Google Drive](https://drive.google.com/file/d/1JdwYukKrEaMTaOwc1W4Q8bBRjxwGZ2be/view?usp=drive_link) |
+| 📊 **Slides** | [Google Drive](https://drive.google.com/file/d/1Wb8G0WEPPvAFppBNSVMZ7EMFBPFjsP8n/view?usp=sharing) |
+
+---
+
+## Table of Contents
+
+- [Motivation](#motivation)
+- [Environment](#environment)
+  - [The Three Apps](#the-three-apps)
+  - [Action Space](#action-space)
+  - [Observation Schema](#observation-schema)
+  - [Org Generator](#org-generator)
+  - [Task Types](#task-types)
+  - [Difficulty Tiers](#difficulty-tiers)
+- [Reward Design](#reward-design)
+- [Training Pipeline](#training-pipeline)
+  - [SFT Warmup](#sft-warmup)
+  - [GRPO](#grpo)
+  - [Reward Shaping](#reward-shaping)
+- [Results](#results)
+- [Project Structure](#project-structure)
+- [Setup & Usage](#setup--usage)
+
+---
+
+## Motivation
+
+Frontier models achieve near-perfect scores on standard benchmarks. Yet when deployed inside a real organization, they routinely fail tasks that a junior employee handles on day one.
+
+The reason: **organizational context is not on the internet.**
+
+A model knows that database failures are serious. It does not know that *your* company outsources DB infrastructure to a vendor and that the correct response is an email, not an internal incident ticket. No benchmark measures this gap. PM-Ops does.
+
+**Why system prompts are not the answer:**
+
+| Problem | Detail |
+|---|---|
+| Attention decay | As conversation grows, the model attends to early context instructions with diminishing weight. Org conventions buried in a long system prompt get effectively ignored. |
+| Static snapshots | Channels get renamed. Teams get reorganized. A system prompt written last quarter is already wrong. |
+| No gradient | The model is told what to do, not trained on the consequences of ignoring it. There is no learning signal. |
+
+PM-Ops addresses this by making the agent *experience* the consequences of skipping the runbook across thousands of varied organizational configurations during training.
+
+---
+
+## Environment
+
+PM-Ops runs as a WebSocket-based environment server (OpenEnv-compatible) hosted on HuggingFace Spaces. Each episode presents the agent with a freshly generated organization — different team names, channel names, label taxonomies, and priority levels every time.
+
+### The Three Apps
+
+```
+┌─────────────────────────────────────────────────────────┐
+│                    PM-Ops Environment                    │
+│                                                         │
+│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
+│  │  Ticketing   │  │   Codebase   │  │     Chat     │  │
+│  │  (Jira-like) │  │ (GitHub-like)│  │ (Slack-like) │  │
+│  │              │  │              │  │              │  │
+│  │ tickets      │  │ repositories │  │ channels     │  │
+│  │ projects     │  │ commits      │  │ threads      │  │
+│  │ teams        │  │ pull requests│  │ DMs          │  │
+│  │ labels       │  │ file authors │  │ user profiles│  │
+│  │ priorities   │  │ changed files│  │              │  │
+│  └──────────────┘  └──────────────┘  └──────────────┘  │
+│                                                         │
+│  Ground truth verified against DB — not LLM-judged      │
+└────────────────────────────────────────��────────────────┘
+```
+
+**Ticketing App:** The source of truth for org state. Agent creates tickets, assigns teams, sets labels and priorities. Correctness is verified by checking the database, not asking another LLM.
+
+**Codebase App:** Pure detective work. No code is written. When a bug report arrives, the agent traces commit history, identifies the responsible change, and finds the author.
+
+**Chat App:** Episode-specific channels. The channel `#oncall-payments` in episode 1 may be `#urgent-billing` in episode 2. The agent must read the runbook to know which one is live — not guess from training data.
+
+### Action Space
+
+```json
+{ "action_type": "meta.read_runbook",       "args": {} }
+{ "action_type": "meta.finish",             "args": {} }
+{ "action_type": "meta.noop",               "args": {} }
+{ "action_type": "ticketing.create_ticket", "args": { "summary": "...", "label": "...", "priority": "...", "assignee": "..." } }
+{ "action_type": "ticketing.assign_ticket", "args": { "ticket_id": "...", "team": "..." } }
+{ "action_type": "chat.list_channels",      "args": {} }
+{ "action_type": "chat.post_message",       "args": { "channel": "...", "text": "..." } }
+{ "action_type": "codebase.get_commits",    "args": { "repo": "..." } }
+```
+
+All actions are emitted as chain-of-thought reasoning followed by a ` ```json ` block. Three-pass extraction handles malformed outputs: code block → raw JSON → regex fallback.
+
+### Observation Schema
+
+Each step returns:
+
+```python
+{
+    "task_brief":             str,   # the incident/task description
+    "last_action_result":     dict,  # {"ok": bool, "data": ..., "error": ...}
+    "step":                   int,
+    "steps_remaining":        int,
+    "reward":                 float, # always 0.0 until meta.finish
+    "done":                   bool,
+    "token_budget_remaining": int,
+}
+```
+
+**Delayed reward:** `reward` is always `0.0` until the agent calls `meta.finish`. This forces the agent to commit to a plan and execute it — step-by-step reward hacking is not possible.
+
+### Org Generator
+
+LLM-generated scenarios sound plausible but fail formal verification. PM-Ops uses a **deterministic, parameterized, seedable org generator** instead.
+
+Each org is generated from a seed and produces:
+
+- `label_taxonomy` — org-specific bug/feature labels
+- `priority_levels` — org-specific severity scale
+- `team_map` — service → owning team
+- `oncall_channels` — team → notification channel
+- `required_ticket_fields` — what fields must be set for a valid ticket
+
+The same seed always produces the same org. Evaluation is reproducible. Training sees a different org every episode.
+
+### Task Types
+
+| Task | Description | Key Challenge |
+|---|---|---|
+| **Triage** | Bug report arrives. Create ticket with correct label, priority, assignee. Notify correct channel. | Using org taxonomy, not generic labels. |
+| **Incident Routing** | Production incident. Identify affected services, find owning team via codebase + team map, escalate. | Cross-system reasoning: chat → codebase → ticketing. |
+| **Release Notes** | Compile and post release notes for resolved tickets in the org's specific format. | Format compliance, not just content. |
+| **Dependency Update** | Breaking library change hits multiple services. Notify each team through their own oncall channel, create per-team tickets. | Multi-target coordination without collapsing to single-service logic. |
+
+### Difficulty Tiers
+
+| Tier | Services | Labels | Priorities | Notes |
+|---|---|---|---|---|
+| **Easy** | 2 | 3 | 2 | No ambiguity. Runbook is complete. |
+| **Medium** | 3 | 4 | 4 | Partial runbook. Agent must infer across systems. Multi-owner configs. |
+| **Hard** | ≥5 | 5+ | 5+ | Noise channels designed to distract. Outdated/missing docs. Simulated panicking users in general chat — correct behavior is to ignore them. |
+
+The hard tier specifically tests resistance to NLP pressure. A model that responds to "THE SITE IS DOWN PLEASE HELP" in `#general` instead of reading the runbook and acting on verified info fails the episode.
 
-## Tasks
+---
+
+## Reward Design
+
+Rewards are **delayed** — revealed only at `meta.finish`.
+
+### Scoring Breakdown
+
+```
+Ticket created correctly     +0.25
+Correct label                +0.20
+Correct priority             +0.20
+Correct team assignment      +0.20
+Correct channel notification +0.15
+─────────────────────────────────
+Maximum per episode           1.00
+```
+
+### Penalties
+
+```
+Wrong channel post           -0.05 each
+                             (>2 wrong posts → net negative; defeats channel-spray)
+Duplicate ticket             -0.10 each
+Zero valid actions (inaction)-1.00
+```
+
+The `-0.05` per wrong channel is calibrated: posting to every channel to guarantee hitting the right one becomes net negative after 2 wrong posts. The agent must read the runbook to know the correct channel.
+
+### Combined Training Reward
+
+```python
+if valid_action_count == 0:
+    combined = -1.0                              # never output valid JSON
+elif final_score == 0.0:
+    combined = valid_json_ratio * 0.15           # tried but failed
+            + read_runbook_reward * 0.10
+            - 0.30                               # hard penalty for zero completion
+else:
+    combined = final_score         * 0.45        # task correctness
+            + no_wrong_channels    * 0.15        # anti channel-spray
+            + valid_json_ratio     * 0.15        # format discipline
+            + read_runbook_reward  * 0.15        # process compliance
+            + efficiency           * 0.10        # steps saved
+```
+
+---
+
+## Training Pipeline
+
+### SFT Warmup
+
+Before GRPO, a short supervised fine-tuning phase runs on baseline agent demonstrations. Without this, the model outputs prose instead of structured JSON actions and the GRPO gradient is zero — the model needs to learn the output format before it can learn the task.
+
+### GRPO
+
+We use **Group Relative Policy Optimization** to train a 7B parameter model directly against PM-Ops episode rewards.
+
+Each training step is a full PM-Ops episode:
+1. Model plays through up to 15 turns
+2. Environment scores the final state
+3. Gradient updates weights based on relative performance across the generation group
+
+No intermediate reward signal is given. The model must learn to plan across multiple steps.
+
+### Reward Shaping
+
+| Component | Weight | Purpose |
+|---|---|---|
+| `final_score` | 0.45 | Primary correctness from env grader |
+| `no_wrong_channels` | 0.15 | Anti-hack: penalise channel spray |
+| `valid_json_ratio` | 0.15 | Format discipline |
+| `read_runbook` | 0.15 | Process compliance |
+| `efficiency` | 0.10 | Steps saved (only when task succeeds) |
 
-| Task | Brief | Max Steps |
-|------|-------|-----------|
-| triage | Bug report -- correct label, priority, team, channel | 25 |
-| incident_routing | Alert -- identify owner via commits, page oncall | 25 |
-| release_notes | Compile + post release notes in org style | 40 |
-| dep_update | Coordinate dependency update across all owning teams | 40 |
+`PMOpsGRPOTrainer` subclasses TRL's `GRPOTrainer` and overrides `_calculate_rewards()` to inject pre-computed rewards from the rollout directly, bypassing TRL's broken kwargs flow for multi-turn rollouts.
 
-## Action space
+---
+
+## Results
+
+Evaluated on 13 matched episodes, comparing the heuristic baseline agent against the GRPO-trained model:
 
-The agent calls actions like:
+| Model | Avg Score (13 eps) |
+|---|---|
+| Heuristic Baseline | 0.269 |
+| GRPO Trained (early) | 0.362 |
+| **Improvement** | **+34.6%** |
+
+Training reward trend across 14 steps shows a positive slope of **+0.0024/step** with KL divergence remaining stable and controlled throughout.
+
+---
+
+## Project Structure
+
+```
+pm_ops/
+├── server/
+│   ├── pm_ops_environment.py   # OpenEnv Environment subclass
+│   ├── org_generator.py        # Deterministic, seedable org factory
+│   ├── grader.py               # Ground-truth reward computation
+│   └── app.py                  # FastAPI server entry point
+├── inference.py                # Heuristic baseline agent + eval runner
+└── pyproject.toml
+
+training/
+├── rollout.py                  # Multi-turn rollout: build_messages, extract_json_action
+├── rewards.py                  # Reward functions + weight constants
+├── pm_ops_trainer.py           # PMOpsGRPOTrainer (_calculate_rewards override)
+├── prompts.py                  # SYSTEM_PROMPT, format_observation
+├── dataset.py                  # Dataset loading + seed parsing
+└── train_v3.ipynb              # SFT warmup + GRPO training notebook
+```
+
+---
 
-  meta.read_runbook       -- learn this org's conventions
-  ticketing.create_ticket -- file a ticket with label + priority
-  ticketing.assign_ticket -- assign to correct team
-  chat.post_message       -- notify the right oncall channel
-  meta.finish             -- end episode, trigger scoring
+## Setup & Usage
 
-## Reward
+### Run the environment server
 
-Delayed to episode end. Deterministic Python verifier -- no LLM-as-judge.
-Partial credit for partial correctness.
+```bash
+cd pm_ops
+pip install -e .
+server  # starts FastAPI on :7860
+```
 
-## Sponsor alignment
+### Run the baseline agent
+
+```bash
+export API_BASE_URL=https://thecrustaceans-pm-ops.hf.space
+python pm_ops/inference.py
+```
+
+### Connect via OpenEnv client
+
+```python
+from openenv.core import GenericEnvClient
+
+env = GenericEnvClient(base_url="https://thecrustaceans-pm-ops.hf.space").sync()
+
+with env:
+    result = env.reset()
+    obs = result.observation
+
+    result = env.step({
+        "action_type": "meta.read_runbook",
+        "args": {}
+    })
+```
+
+### Training
+
+Open `training/train_v3.ipynb` in a Colab instance with a GPU. The notebook handles SFT warmup, GRPO setup, and checkpointing.
+
+---
 
-- AI Labs: direct PM-in-the-loop positioning
-- Patronus: schema drift -- env tests whether agents track convention changes
-- Scale AI: enterprise workflow simulation
-- Meta RFC 004: delayed trajectory reward is the native reward mode
+**Live Environment:** [https://huggingface.co/spaces/TheCrustaceans/Pm-ops](https://huggingface.co/spaces/TheCrustaceans/Pm-ops)