README.md

---
title: PM-Ops RL Environment
emoji: 📋
colorFrom: blue
colorTo: purple
sdk: docker
app_port: 8000
pinned: false
tags:
  - openenv
  - reinforcement-learning
---

# PM-Ops 🛠️

> **Can a small model learn to operate inside a company it has never seen before?**

PM-Ops is a reinforcement learning benchmark and training environment where an LLM agent operates as a product manager inside a fully simulated software organization — navigating ticketing, codebases, and chat — using only the organization's own runbook as its guide.

---

## Links

| Resource | Link |
|---|---|
| 🤗 **Live Environment** | [TheCrustaceans/Pm-ops — HuggingFace Space](https://huggingface.co/spaces/TheCrustaceans/Pm-ops) |
| 📝 **Blog Post** | [BlogPost.mdx](https://huggingface.co/spaces/TheCrustaceans/Pm-ops/blob/main/BlogPost.md) |
| 🎥 **inference image** | [Google Drive](https://drive.google.com/file/d/1JdwYukKrEaMTaOwc1W4Q8bBRjxwGZ2be/view?usp=drive_link) |
| 📊 **train image** | [Google Drive](https://drive.google.com/file/d/1Wb8G0WEPPvAFppBNSVMZ7EMFBPFjsP8n/view?usp=sharing) |

---

## Table of Contents

- [Motivation](#motivation)
- [Environment](#environment)
  - [The Three Apps](#the-three-apps)
  - [Action Space](#action-space)
  - [Observation Schema](#observation-schema)
  - [Org Generator](#org-generator)
  - [Task Types](#task-types)
  - [Difficulty Tiers](#difficulty-tiers)
- [Reward Design](#reward-design)
- [Training Pipeline](#training-pipeline)
  - [SFT Warmup](#sft-warmup)
  - [GRPO](#grpo)
  - [Reward Shaping](#reward-shaping)
- [Results](#results)
- [Project Structure](#project-structure)
- [Setup & Usage](#setup--usage)

---

## Motivation

Frontier models achieve near-perfect scores on standard benchmarks. Yet when deployed inside a real organization, they routinely fail tasks that a junior employee handles on day one.

The reason: **organizational context is not on the internet.**

A model knows that database failures are serious. It does not know that *your* company outsources DB infrastructure to a vendor and that the correct response is an email, not an internal incident ticket. No benchmark measures this gap. PM-Ops does.

**Why system prompts are not the answer:**

| Problem | Detail |
|---|---|
| Attention decay | As conversation grows, the model attends to early context instructions with diminishing weight. Org conventions buried in a long system prompt get effectively ignored. |
| Static snapshots | Channels get renamed. Teams get reorganized. A system prompt written last quarter is already wrong. |
| No gradient | The model is told what to do, not trained on the consequences of ignoring it. There is no learning signal. |

PM-Ops addresses this by making the agent *experience* the consequences of skipping the runbook across thousands of varied organizational configurations during training.

---

## Environment

PM-Ops runs as a WebSocket-based environment server (OpenEnv-compatible) hosted on HuggingFace Spaces. Each episode presents the agent with a freshly generated organization — different team names, channel names, label taxonomies, and priority levels every time.

### The Three Apps

```
┌─────────────────────────────────────────────────────────┐
│                    PM-Ops Environment                    │
│                                                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │  Ticketing   │  │   Codebase   │  │     Chat     │  │
│  │  (Jira-like) │  │ (GitHub-like)│  │ (Slack-like) │  │
│  │              │  │              │  │              │  │
│  │ tickets      │  │ repositories │  │ channels     │  │
│  │ projects     │  │ commits      │  │ threads      │  │
│  │ teams        │  │ pull requests│  │ DMs          │  │
│  │ labels       │  │ file authors │  │ user profiles│  │
│  │ priorities   │  │ changed files│  │              │  │
│  └──────────────┘  └──────────────┘  └──────────────┘  │
│                                                         │
│  Ground truth verified against DB — not LLM-judged      │
└─────────────────────────────────────────────────────────┘
```

**Ticketing App:** The source of truth for org state. Agent creates tickets, assigns teams, sets labels and priorities. Correctness is verified by checking the database, not asking another LLM.

**Codebase App:** Pure detective work. No code is written. When a bug report arrives, the agent traces commit history, identifies the responsible change, and finds the author.

**Chat App:** Episode-specific channels. The channel `#oncall-payments` in episode 1 may be `#urgent-billing` in episode 2. The agent must read the runbook to know which one is live — not guess from training data.

### Action Space

```json
{ "action_type": "meta.read_runbook",       "args": {} }
{ "action_type": "meta.finish",             "args": {} }
{ "action_type": "meta.noop",               "args": {} }
{ "action_type": "ticketing.create_ticket", "args": { "summary": "...", "label": "...", "priority": "...", "assignee": "..." } }
{ "action_type": "ticketing.assign_ticket", "args": { "ticket_id": "...", "team": "..." } }
{ "action_type": "chat.list_channels",      "args": {} }
{ "action_type": "chat.post_message",       "args": { "channel": "...", "text": "..." } }
{ "action_type": "codebase.get_commits",    "args": { "repo": "..." } }
```

All actions are emitted as chain-of-thought reasoning followed by a ` ```json ` block. Three-pass extraction handles malformed outputs: code block → raw JSON → regex fallback.

### Observation Schema

Each step returns:

```python
{
    "task_brief":             str,   # the incident/task description
    "last_action_result":     dict,  # {"ok": bool, "data": ..., "error": ...}
    "step":                   int,
    "steps_remaining":        int,
    "reward":                 float, # always 0.0 until meta.finish
    "done":                   bool,
    "token_budget_remaining": int,
}
```

**Delayed reward:** `reward` is always `0.0` until the agent calls `meta.finish`. This forces the agent to commit to a plan and execute it — step-by-step reward hacking is not possible.

### Org Generator

LLM-generated scenarios sound plausible but fail formal verification. PM-Ops uses a **deterministic, parameterized, seedable org generator** instead.

Each org is generated from a seed and produces:

- `label_taxonomy` — org-specific bug/feature labels
- `priority_levels` — org-specific severity scale
- `team_map` — service → owning team
- `oncall_channels` — team → notification channel
- `required_ticket_fields` — what fields must be set for a valid ticket

The same seed always produces the same org. Evaluation is reproducible. Training sees a different org every episode.

### Task Types

| Task | Description | Key Challenge |
|---|---|---|
| **Triage** | Bug report arrives. Create ticket with correct label, priority, assignee. Notify correct channel. | Using org taxonomy, not generic labels. |
| **Incident Routing** | Production incident. Identify affected services, find owning team via codebase + team map, escalate. | Cross-system reasoning: chat → codebase → ticketing. |
| **Release Notes** | Compile and post release notes for resolved tickets in the org's specific format. | Format compliance, not just content. |
| **Dependency Update** | Breaking library change hits multiple services. Notify each team through their own oncall channel, create per-team tickets. | Multi-target coordination without collapsing to single-service logic. |

### Difficulty Tiers

| Tier | Services | Labels | Priorities | Notes |
|---|---|---|---|---|
| **Easy** | 2 | 3 | 2 | No ambiguity. Runbook is complete. |
| **Medium** | 3 | 4 | 4 | Partial runbook. Agent must infer across systems. Multi-owner configs. |
| **Hard** | ≥5 | 5+ | 5+ | Noise channels designed to distract. Outdated/missing docs. Simulated panicking users in general chat — correct behavior is to ignore them. |

The hard tier specifically tests resistance to NLP pressure. A model that responds to "THE SITE IS DOWN PLEASE HELP" in `#general` instead of reading the runbook and acting on verified info fails the episode.

---

## Reward Design

Rewards are **delayed** — revealed only at `meta.finish`.

### Scoring Breakdown

```
Ticket created correctly     +0.25
Correct label                +0.20
Correct priority             +0.20
Correct team assignment      +0.20
Correct channel notification +0.15
─────────────────────────────────
Maximum per episode           1.00
```

### Penalties

```
Wrong channel post           -0.05 each
                             (>2 wrong posts → net negative; defeats channel-spray)
Duplicate ticket             -0.10 each
Zero valid actions (inaction)-1.00
```

The `-0.05` per wrong channel is calibrated: posting to every channel to guarantee hitting the right one becomes net negative after 2 wrong posts. The agent must read the runbook to know the correct channel.

### Combined Training Reward

```python
if valid_action_count == 0:
    combined = -1.0                              # never output valid JSON
elif final_score == 0.0:
    combined = valid_json_ratio * 0.15           # tried but failed
            + read_runbook_reward * 0.10
            - 0.30                               # hard penalty for zero completion
else:
    combined = final_score         * 0.45        # task correctness
            + no_wrong_channels    * 0.15        # anti channel-spray
            + valid_json_ratio     * 0.15        # format discipline
            + read_runbook_reward  * 0.15        # process compliance
            + efficiency           * 0.10        # steps saved
```

---

## Training Pipeline

### SFT Warmup

Before GRPO, a short supervised fine-tuning phase runs on baseline agent demonstrations. Without this, the model outputs prose instead of structured JSON actions and the GRPO gradient is zero — the model needs to learn the output format before it can learn the task.

### GRPO

We use **Group Relative Policy Optimization** to train a 7B parameter model directly against PM-Ops episode rewards.

Each training step is a full PM-Ops episode:
1. Model plays through up to 15 turns
2. Environment scores the final state
3. Gradient updates weights based on relative performance across the generation group

No intermediate reward signal is given. The model must learn to plan across multiple steps.

### Reward Shaping

| Component | Weight | Purpose |
|---|---|---|
| `final_score` | 0.45 | Primary correctness from env grader |
| `no_wrong_channels` | 0.15 | Anti-hack: penalise channel spray |
| `valid_json_ratio` | 0.15 | Format discipline |
| `read_runbook` | 0.15 | Process compliance |
| `efficiency` | 0.10 | Steps saved (only when task succeeds) |

`PMOpsGRPOTrainer` subclasses TRL's `GRPOTrainer` and overrides `_calculate_rewards()` to inject pre-computed rewards from the rollout directly, bypassing TRL's broken kwargs flow for multi-turn rollouts.

---

## Results

Evaluated on 13 matched episodes, comparing the heuristic baseline agent against the GRPO-trained model:

| Model | Avg Score (13 eps) |
|---|---|
| Heuristic Baseline | 0.269 |
| GRPO Trained (early) | 0.362 |
| **Improvement** | **+34.6%** |

Training reward trend across 14 steps shows a positive slope of **+0.0024/step** with KL divergence remaining stable and controlled throughout.

---

## Project Structure

```
pm_ops/
├── server/
│   ├── pm_ops_environment.py   # OpenEnv Environment subclass
│   ├── org_generator.py        # Deterministic, seedable org factory
│   ├── grader.py               # Ground-truth reward computation
│   └── app.py                  # FastAPI server entry point
├── inference.py                # Heuristic baseline agent + eval runner
└── pyproject.toml

training/
├── rollout.py                  # Multi-turn rollout: build_messages, extract_json_action
├── rewards.py                  # Reward functions + weight constants
├── pm_ops_trainer.py           # PMOpsGRPOTrainer (_calculate_rewards override)
├── prompts.py                  # SYSTEM_PROMPT, format_observation
├── dataset.py                  # Dataset loading + seed parsing
└── train_v3.ipynb              # SFT warmup + GRPO training notebook
```

---

## Setup & Usage

### Run the environment server

```bash
cd pm_ops
pip install -e .
server  # starts FastAPI on :7860
```

### Run the baseline agent

```bash
export API_BASE_URL=https://thecrustaceans-pm-ops.hf.space
python pm_ops/inference.py
```

### Connect via OpenEnv client

```python
from openenv.core import GenericEnvClient

env = GenericEnvClient(base_url="https://thecrustaceans-pm-ops.hf.space").sync()

with env:
    result = env.reset()
    obs = result.observation

    result = env.step({
        "action_type": "meta.read_runbook",
        "args": {}
    })
```

### Training

Open `training/train_v3.ipynb` in a Colab instance with a GPU. The notebook handles SFT warmup, GRPO setup, and checkpointing.

---

**Live Environment:** [https://huggingface.co/spaces/TheCrustaceans/Pm-ops](https://huggingface.co/spaces/TheCrustaceans/Pm-ops)