Spaces:
Sleeping
Sleeping
| ## PM-Ops: The Product Manager Benchmark | |
| ### The Problem | |
| It's 2:00AM in the morning, and our beloved frontier models with perfect benchmark scores comes across a DB failture. It does all the right things, understands that this is a serious issue and creates internal incident tickets immediately like a good boy but the twist is the company outsources its DB infrastructure!!! Thus instead of emailing to the vendor, it created an internal ticket. The model relying on general knowledge gets the organization context completely wrong while burning my credits!!! | |
| Most industries use frontier models with near-perfect benchmark scores. However, even with these high capabilities, when dropped into a real company's workflow, these models often fail basic junior-level tasks. | |
| ### Why System Prompts Don't Fix This | |
| 1. **Attention Decay:** LLMs are probabilistic systems. The next token is shaped by billions of parameters trained on averaged human text. If we fill the context window with organizational conventions, the model doesn't "memorize" them; rather, it attends to them with diminishing weight as the conversation grows. | |
| 2. **Dynamic Organizations:** A system prompt is a static snapshot. Channels get renamed, and teams get reorganized. A static prompt cannot follow these shifts, leading the model to post to dead channels or outdated aliases. | |
| --- | |
| ### The Environment | |
| PM-Ops simulates a full software organization as three interconnected applications. The agent operates as a product manager navigating all three. | |
| * **Ticketing App (Jira-like):** Tracks tickets, projects, teams, users, labels, and priorities. The agent interacts via a simulated API. The "ground truth" of the organizational state is stored in a database; rather than letting the LLM judge its own performance, we verify its actions against this database. | |
| * **Codebase App (GitHub-like):** Contains simulated repositories, commit history, pull requests, file authors, and changed files. The goal here is detective work: when a bug report arrives, the agent must trace commit history to identify changes and the responsible authors. | |
| * **Chat App (Slack-like):** Features channels, threaded conversations, direct messages, and user profiles. The agent can read history, search, and post. Channels are episode-specific; for example, a critical alert channel might be `#oncall-payments` in Episode 1, but be replaced by `#urgent-billing` in Episode 2. | |
| **Key Tool:** `meta.read_runbook`. Before taking any action, the agent can call this tool to return the organization's current conventions as a structured document. | |
| --- | |
| ### Scenarios and Difficulty Tiers | |
| LLM-generated scenarios often produce plausible-sounding outputs that fail formal verification. To solve this, we use an **org-generator** that is deterministic, parameterized, and seedable. | |
| | Tier | Complexity | Constraints | | |
| | :--- | :--- | :--- | | |
| | **Easy** | 2 services, 3 labels, 2 priorities. | No ambiguity. | | |
| | **Medium** | 4 labels, 4 priorities, 3 services. | Partial runbook; agent must infer info across systems. | | |
| | **Hard** | ≥5 services, multi-owner configs. | High noise; outdated docs; requires resisting "panic" in general chat. | | |
| ### Task Types | |
| 1. **Triage:** A bug report arrives. The agent must create a ticket with the correct label, priority, and taxonomy, then assign it to the right team and notify the correct channel. | |
| 2. **Incident Routing:** A production incident occurs. The agent must identify affected services via the codebase and team map to route the escalation correctly and quickly. | |
| 3. **Release Notes:** The agent compiles and posts release notes for resolved tickets using the organization's specific format. | |
| 4. **Dependency Update:** A library change affects multiple services. The agent must identify all affected teams, notify them through their respective on-call channels, and create tickets for each—a multi-target coordination problem. | |
| --- | |
| ### Rewards and Penalties | |
| Rewards are **delayed**; the score is revealed only when `meta.finish` is called. This prevents step-by-step optimization and forces the agent to commit to a coherent plan. | |
| * **Rewards:** | |
| * Ticket created: **+0.25** | |
| * Correct label/priority/team: **+0.20 each** | |
| * Correct channel notification: **+0.15** | |
| * **Penalties:** | |
| * Wrong channel message: **-0.05 each** (prevents "spray and pray" messaging) | |
| * Duplicate tickets: **-0.10 each** | |
| * Zero valid actions (inaction): **-1.0** | |
| ### Training with GRPO | |
| We use **GRPO (Group Relative Policy Optimization)** to train a 1.7B parameter model. | |
| * **SFT Warmup:** We begin with a short Supervised Fine-Tuning phase on baseline demonstrations to establish the JSON output format. Without this, the model outputs prose, resulting in a zero gradient for GRPO. | |
| * **Optimization:** After SFT, GRPO optimizes directly against episode rewards. Each training step is a full PM-Ops episode (up to 12 turns). Weights are updated based on relative performance across the generation group. | |
| The result is an agent that doesn't just "know" that reading conventions is good practice—it has learned through thousands of varied configurations that skipping the runbook reliably leads to penalties. The behavior becomes **instinctual, not just instructed.** | |
|  | |
| Link of image1: https://drive.google.com/uc?export=view&id=1JdwYukKrEaMTaOwc1W4Q8bBRjxwGZ2be | |
| Link of image2: https://drive.google.com/file/d/1Wb8G0WEPPvAFppBNSVMZ7EMFBPFjsP8n/view?usp=sharing | |