Spaces:
Sleeping
Sleeping
| title: PM-Ops RL Environment | |
| emoji: 📋 | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 8000 | |
| pinned: false | |
| tags: | |
| - openenv | |
| - reinforcement-learning | |
| # PM-Ops 🛠️ | |
| > **Can a small model learn to operate inside a company it has never seen before?** | |
| PM-Ops is a reinforcement learning benchmark and training environment where an LLM agent operates as a product manager inside a fully simulated software organization — navigating ticketing, codebases, and chat — using only the organization's own runbook as its guide. | |
| --- | |
| ## Links | |
| | Resource | Link | | |
| |---|---| | |
| | 🤗 **Live Environment** | [TheCrustaceans/Pm-ops — HuggingFace Space](https://huggingface.co/spaces/TheCrustaceans/Pm-ops) | | |
| | 📝 **Blog Post** | [BlogPost.mdx](https://huggingface.co/spaces/TheCrustaceans/Pm-ops/blob/main/BlogPost.md) | | |
| | 🎥 **inference image** | [Google Drive](https://drive.google.com/file/d/1JdwYukKrEaMTaOwc1W4Q8bBRjxwGZ2be/view?usp=drive_link) | | |
| | 📊 **train image** | [Google Drive](https://drive.google.com/file/d/1Wb8G0WEPPvAFppBNSVMZ7EMFBPFjsP8n/view?usp=sharing) | | |
| --- | |
| ## Table of Contents | |
| - [Motivation](#motivation) | |
| - [Environment](#environment) | |
| - [The Three Apps](#the-three-apps) | |
| - [Action Space](#action-space) | |
| - [Observation Schema](#observation-schema) | |
| - [Org Generator](#org-generator) | |
| - [Task Types](#task-types) | |
| - [Difficulty Tiers](#difficulty-tiers) | |
| - [Reward Design](#reward-design) | |
| - [Training Pipeline](#training-pipeline) | |
| - [SFT Warmup](#sft-warmup) | |
| - [GRPO](#grpo) | |
| - [Reward Shaping](#reward-shaping) | |
| - [Results](#results) | |
| - [Project Structure](#project-structure) | |
| - [Setup & Usage](#setup--usage) | |
| --- | |
| ## Motivation | |
| Frontier models achieve near-perfect scores on standard benchmarks. Yet when deployed inside a real organization, they routinely fail tasks that a junior employee handles on day one. | |
| The reason: **organizational context is not on the internet.** | |
| A model knows that database failures are serious. It does not know that *your* company outsources DB infrastructure to a vendor and that the correct response is an email, not an internal incident ticket. No benchmark measures this gap. PM-Ops does. | |
| **Why system prompts are not the answer:** | |
| | Problem | Detail | | |
| |---|---| | |
| | Attention decay | As conversation grows, the model attends to early context instructions with diminishing weight. Org conventions buried in a long system prompt get effectively ignored. | | |
| | Static snapshots | Channels get renamed. Teams get reorganized. A system prompt written last quarter is already wrong. | | |
| | No gradient | The model is told what to do, not trained on the consequences of ignoring it. There is no learning signal. | | |
| PM-Ops addresses this by making the agent *experience* the consequences of skipping the runbook across thousands of varied organizational configurations during training. | |
| --- | |
| ## Environment | |
| PM-Ops runs as a WebSocket-based environment server (OpenEnv-compatible) hosted on HuggingFace Spaces. Each episode presents the agent with a freshly generated organization — different team names, channel names, label taxonomies, and priority levels every time. | |
| ### The Three Apps | |
| ``` | |
| ┌─────────────────────────────────────────────────────────┐ | |
| │ PM-Ops Environment │ | |
| │ │ | |
| │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ | |
| │ │ Ticketing │ │ Codebase │ │ Chat │ │ | |
| │ │ (Jira-like) │ │ (GitHub-like)│ │ (Slack-like) │ │ | |
| │ │ │ │ │ │ │ │ | |
| │ │ tickets │ │ repositories │ │ channels │ │ | |
| │ │ projects │ │ commits │ │ threads │ │ | |
| │ │ teams │ │ pull requests│ │ DMs │ │ | |
| │ │ labels │ │ file authors │ │ user profiles│ │ | |
| │ │ priorities │ │ changed files│ │ │ │ | |
| │ └──────────────┘ └──────────────┘ └──────────────┘ │ | |
| │ │ | |
| │ Ground truth verified against DB — not LLM-judged │ | |
| └─────────────────────────────────────────────────────────┘ | |
| ``` | |
| **Ticketing App:** The source of truth for org state. Agent creates tickets, assigns teams, sets labels and priorities. Correctness is verified by checking the database, not asking another LLM. | |
| **Codebase App:** Pure detective work. No code is written. When a bug report arrives, the agent traces commit history, identifies the responsible change, and finds the author. | |
| **Chat App:** Episode-specific channels. The channel `#oncall-payments` in episode 1 may be `#urgent-billing` in episode 2. The agent must read the runbook to know which one is live — not guess from training data. | |
| ### Action Space | |
| ```json | |
| { "action_type": "meta.read_runbook", "args": {} } | |
| { "action_type": "meta.finish", "args": {} } | |
| { "action_type": "meta.noop", "args": {} } | |
| { "action_type": "ticketing.create_ticket", "args": { "summary": "...", "label": "...", "priority": "...", "assignee": "..." } } | |
| { "action_type": "ticketing.assign_ticket", "args": { "ticket_id": "...", "team": "..." } } | |
| { "action_type": "chat.list_channels", "args": {} } | |
| { "action_type": "chat.post_message", "args": { "channel": "...", "text": "..." } } | |
| { "action_type": "codebase.get_commits", "args": { "repo": "..." } } | |
| ``` | |
| All actions are emitted as chain-of-thought reasoning followed by a ` ```json ` block. Three-pass extraction handles malformed outputs: code block → raw JSON → regex fallback. | |
| ### Observation Schema | |
| Each step returns: | |
| ```python | |
| { | |
| "task_brief": str, # the incident/task description | |
| "last_action_result": dict, # {"ok": bool, "data": ..., "error": ...} | |
| "step": int, | |
| "steps_remaining": int, | |
| "reward": float, # always 0.0 until meta.finish | |
| "done": bool, | |
| "token_budget_remaining": int, | |
| } | |
| ``` | |
| **Delayed reward:** `reward` is always `0.0` until the agent calls `meta.finish`. This forces the agent to commit to a plan and execute it — step-by-step reward hacking is not possible. | |
| ### Org Generator | |
| LLM-generated scenarios sound plausible but fail formal verification. PM-Ops uses a **deterministic, parameterized, seedable org generator** instead. | |
| Each org is generated from a seed and produces: | |
| - `label_taxonomy` — org-specific bug/feature labels | |
| - `priority_levels` — org-specific severity scale | |
| - `team_map` — service → owning team | |
| - `oncall_channels` — team → notification channel | |
| - `required_ticket_fields` — what fields must be set for a valid ticket | |
| The same seed always produces the same org. Evaluation is reproducible. Training sees a different org every episode. | |
| ### Task Types | |
| | Task | Description | Key Challenge | | |
| |---|---|---| | |
| | **Triage** | Bug report arrives. Create ticket with correct label, priority, assignee. Notify correct channel. | Using org taxonomy, not generic labels. | | |
| | **Incident Routing** | Production incident. Identify affected services, find owning team via codebase + team map, escalate. | Cross-system reasoning: chat → codebase → ticketing. | | |
| | **Release Notes** | Compile and post release notes for resolved tickets in the org's specific format. | Format compliance, not just content. | | |
| | **Dependency Update** | Breaking library change hits multiple services. Notify each team through their own oncall channel, create per-team tickets. | Multi-target coordination without collapsing to single-service logic. | | |
| ### Difficulty Tiers | |
| | Tier | Services | Labels | Priorities | Notes | | |
| |---|---|---|---|---| | |
| | **Easy** | 2 | 3 | 2 | No ambiguity. Runbook is complete. | | |
| | **Medium** | 3 | 4 | 4 | Partial runbook. Agent must infer across systems. Multi-owner configs. | | |
| | **Hard** | ≥5 | 5+ | 5+ | Noise channels designed to distract. Outdated/missing docs. Simulated panicking users in general chat — correct behavior is to ignore them. | | |
| The hard tier specifically tests resistance to NLP pressure. A model that responds to "THE SITE IS DOWN PLEASE HELP" in `#general` instead of reading the runbook and acting on verified info fails the episode. | |
| --- | |
| ## Reward Design | |
| Rewards are **delayed** — revealed only at `meta.finish`. | |
| ### Scoring Breakdown | |
| ``` | |
| Ticket created correctly +0.25 | |
| Correct label +0.20 | |
| Correct priority +0.20 | |
| Correct team assignment +0.20 | |
| Correct channel notification +0.15 | |
| ───────────────────────────────── | |
| Maximum per episode 1.00 | |
| ``` | |
| ### Penalties | |
| ``` | |
| Wrong channel post -0.05 each | |
| (>2 wrong posts → net negative; defeats channel-spray) | |
| Duplicate ticket -0.10 each | |
| Zero valid actions (inaction)-1.00 | |
| ``` | |
| The `-0.05` per wrong channel is calibrated: posting to every channel to guarantee hitting the right one becomes net negative after 2 wrong posts. The agent must read the runbook to know the correct channel. | |
| ### Combined Training Reward | |
| ```python | |
| if valid_action_count == 0: | |
| combined = -1.0 # never output valid JSON | |
| elif final_score == 0.0: | |
| combined = valid_json_ratio * 0.15 # tried but failed | |
| + read_runbook_reward * 0.10 | |
| - 0.30 # hard penalty for zero completion | |
| else: | |
| combined = final_score * 0.45 # task correctness | |
| + no_wrong_channels * 0.15 # anti channel-spray | |
| + valid_json_ratio * 0.15 # format discipline | |
| + read_runbook_reward * 0.15 # process compliance | |
| + efficiency * 0.10 # steps saved | |
| ``` | |
| --- | |
| ## Training Pipeline | |
| ### SFT Warmup | |
| Before GRPO, a short supervised fine-tuning phase runs on baseline agent demonstrations. Without this, the model outputs prose instead of structured JSON actions and the GRPO gradient is zero — the model needs to learn the output format before it can learn the task. | |
| ### GRPO | |
| We use **Group Relative Policy Optimization** to train a 7B parameter model directly against PM-Ops episode rewards. | |
| Each training step is a full PM-Ops episode: | |
| 1. Model plays through up to 15 turns | |
| 2. Environment scores the final state | |
| 3. Gradient updates weights based on relative performance across the generation group | |
| No intermediate reward signal is given. The model must learn to plan across multiple steps. | |
| ### Reward Shaping | |
| | Component | Weight | Purpose | | |
| |---|---|---| | |
| | `final_score` | 0.45 | Primary correctness from env grader | | |
| | `no_wrong_channels` | 0.15 | Anti-hack: penalise channel spray | | |
| | `valid_json_ratio` | 0.15 | Format discipline | | |
| | `read_runbook` | 0.15 | Process compliance | | |
| | `efficiency` | 0.10 | Steps saved (only when task succeeds) | | |
| `PMOpsGRPOTrainer` subclasses TRL's `GRPOTrainer` and overrides `_calculate_rewards()` to inject pre-computed rewards from the rollout directly, bypassing TRL's broken kwargs flow for multi-turn rollouts. | |
| --- | |
| ## Results | |
| Evaluated on 13 matched episodes, comparing the heuristic baseline agent against the GRPO-trained model: | |
| | Model | Avg Score (13 eps) | | |
| |---|---| | |
| | Heuristic Baseline | 0.269 | | |
| | GRPO Trained (early) | 0.362 | | |
| | **Improvement** | **+34.6%** | | |
| Training reward trend across 14 steps shows a positive slope of **+0.0024/step** with KL divergence remaining stable and controlled throughout. | |
| --- | |
| ## Project Structure | |
| ``` | |
| pm_ops/ | |
| ├── server/ | |
| │ ├── pm_ops_environment.py # OpenEnv Environment subclass | |
| │ ├── org_generator.py # Deterministic, seedable org factory | |
| │ ├── grader.py # Ground-truth reward computation | |
| │ └── app.py # FastAPI server entry point | |
| ├── inference.py # Heuristic baseline agent + eval runner | |
| └── pyproject.toml | |
| training/ | |
| ├── rollout.py # Multi-turn rollout: build_messages, extract_json_action | |
| ├── rewards.py # Reward functions + weight constants | |
| ├── pm_ops_trainer.py # PMOpsGRPOTrainer (_calculate_rewards override) | |
| ├── prompts.py # SYSTEM_PROMPT, format_observation | |
| ├── dataset.py # Dataset loading + seed parsing | |
| └── train_v3.ipynb # SFT warmup + GRPO training notebook | |
| ``` | |
| --- | |
| ## Setup & Usage | |
| ### Run the environment server | |
| ```bash | |
| cd pm_ops | |
| pip install -e . | |
| server # starts FastAPI on :7860 | |
| ``` | |
| ### Run the baseline agent | |
| ```bash | |
| export API_BASE_URL=https://thecrustaceans-pm-ops.hf.space | |
| python pm_ops/inference.py | |
| ``` | |
| ### Connect via OpenEnv client | |
| ```python | |
| from openenv.core import GenericEnvClient | |
| env = GenericEnvClient(base_url="https://thecrustaceans-pm-ops.hf.space").sync() | |
| with env: | |
| result = env.reset() | |
| obs = result.observation | |
| result = env.step({ | |
| "action_type": "meta.read_runbook", | |
| "args": {} | |
| }) | |
| ``` | |
| ### Training | |
| Open `training/train_v3.ipynb` in a Colab instance with a GPU. The notebook handles SFT warmup, GRPO setup, and checkpointing. | |
| --- | |
| **Live Environment:** [https://huggingface.co/spaces/TheCrustaceans/Pm-ops](https://huggingface.co/spaces/TheCrustaceans/Pm-ops) | |