fix: resolve merge conflict, restore HF frontmatter + full README

.gitignore

@@ -12,3 +12,4 @@ pdf_pages/
 dist/
 build/
 .pytest_cache/
+.vscode/

BlogPost.md

@@ -0,0 +1,68 @@
+## PM-Ops: The Product Manager Benchmark
+
+### The Problem
+Most industries use frontier models with near-perfect benchmark scores. However, even with these high capabilities, when dropped into a real company's workflow, these models often fail basic junior-level tasks.
+
+**Example:** When a database alert fires at 2:00 AM, a model trained on the internet knows that DB failures are serious and will likely create an internal incident ticket immediately. But what if the company in question outsources its DB infrastructure? In that case, the protocol is an email to the vendor, not an internal ticket. The model, relying on general knowledge, gets the organizational context completely wrong.
+
+### Why System Prompts Don't Fix This
+
+1.  **Attention Decay:** LLMs are probabilistic systems. The next token is shaped by billions of parameters trained on averaged human text. If we fill the context window with organizational conventions, the model doesn't "memorize" them; rather, it attends to them with diminishing weight as the conversation grows.
+2.  **Dynamic Organizations:** A system prompt is a static snapshot. Channels get renamed, and teams get reorganized. A static prompt cannot follow these shifts, leading the model to post to dead channels or outdated aliases.
+
+---
+
+### The Environment
+PM-Ops simulates a full software organization as three interconnected applications. The agent operates as a product manager navigating all three.
+
+* **Ticketing App (Jira-like):** Tracks tickets, projects, teams, users, labels, and priorities. The agent interacts via a simulated API. The "ground truth" of the organizational state is stored in a database; rather than letting the LLM judge its own performance, we verify its actions against this database.
+* **Codebase App (GitHub-like):** Contains simulated repositories, commit history, pull requests, file authors, and changed files. The goal here is detective work: when a bug report arrives, the agent must trace commit history to identify changes and the responsible authors.
+* **Chat App (Slack-like):** Features channels, threaded conversations, direct messages, and user profiles. The agent can read history, search, and post. Channels are episode-specific; for example, a critical alert channel might be `#oncall-payments` in Episode 1, but be replaced by `#urgent-billing` in Episode 2.
+
+**Key Tool:** `meta.read_runbook`. Before taking any action, the agent can call this tool to return the organization's current conventions as a structured document.
+
+---
+
+### Scenarios and Difficulty Tiers
+LLM-generated scenarios often produce plausible-sounding outputs that fail formal verification. To solve this, we use an **org-generator** that is deterministic, parameterized, and seedable.
+
+| Tier | Complexity | Constraints |
+| :--- | :--- | :--- |
+| **Easy** | 2 services, 3 labels, 2 priorities. | No ambiguity. |
+| **Medium** | 4 labels, 4 priorities, 3 services. | Partial runbook; agent must infer info across systems. |
+| **Hard** | ≥5 services, multi-owner configs. | High noise; outdated docs; requires resisting "panic" in general chat. |
+
+### Task Types
+1.  **Triage:** A bug report arrives. The agent must create a ticket with the correct label, priority, and taxonomy, then assign it to the right team and notify the correct channel.
+2.  **Incident Routing:** A production incident occurs. The agent must identify affected services via the codebase and team map to route the escalation correctly and quickly.
+3.  **Release Notes:** The agent compiles and posts release notes for resolved tickets using the organization's specific format.
+4.  **Dependency Update:** A library change affects multiple services. The agent must identify all affected teams, notify them through their respective on-call channels, and create tickets for each—a multi-target coordination problem.
+
+---
+
+### Rewards and Penalties
+Rewards are **delayed**; the score is revealed only when `meta.finish` is called. This prevents step-by-step optimization and forces the agent to commit to a coherent plan.
+
+* **Rewards:**
+    * Ticket created: **+0.25**
+    * Correct label/priority/team: **+0.20 each**
+    * Correct channel notification: **+0.15**
+* **Penalties:**
+    * Wrong channel message: **-0.05 each** (prevents "spray and pray" messaging)
+    * Duplicate tickets: **-0.10 each**
+    * Zero valid actions (inaction): **-1.0**
+
+### Training with GRPO
+We use **GRPO (Group Relative Policy Optimization)** to train a 1.7B parameter model.
+* **SFT Warmup:** We begin with a short Supervised Fine-Tuning phase on baseline demonstrations to establish the JSON output format. Without this, the model outputs prose, resulting in a zero gradient for GRPO.
+* **Optimization:** After SFT, GRPO optimizes directly against episode rewards. Each training step is a full PM-Ops episode (up to 12 turns). Weights are updated based on relative performance across the generation group.
+
+The result is an agent that doesn't just "know" that reading conventions is good practice—it has learned through thousands of varied configurations that skipping the runbook reliably leads to penalties. The behavior becomes **instinctual, not just instructed.**
+
+
+![PM-Ops Demo](https://drive.google.com/uc?export=view&id=1JdwYukKrEaMTaOwc1W4Q8bBRjxwGZ2be)
+
+Link of image1: https://drive.google.com/uc?export=view&id=1JdwYukKrEaMTaOwc1W4Q8bBRjxwGZ2be
+
+Link of image2: https://drive.google.com/file/d/1Wb8G0WEPPvAFppBNSVMZ7EMFBPFjsP8n/view?usp=sharing
+

client.py

@@ -1,6 +1,6 @@
 """OpenEnv client wrapper for PM-Ops environment."""
 import os
-from openenv.core.client import EnvClient
+from openenv.core.env_client import EnvClient
 from models import PMOpsAction, PMOpsObservation
 
 
@@ -8,7 +8,6 @@ class PMOpsEnv(EnvClient):
     action_class = PMOpsAction
     observation_class = PMOpsObservation
 
-    def __init__(self, base_url: str = None, token: str = None):
+    def __init__(self, base_url: str | None = None, token: str | None= None):
         url = base_url or os.getenv("API_BASE_URL", "https://adityaguntur-pm-ops.hf.space")
-        tok = token or os.getenv("HF_TOKEN")
-        super().__init__(base_url=url, token=tok)
+        super().__init__(base_url=url)

models.py

@@ -46,4 +46,4 @@ class PMOpsState(State):
     codebase: Dict[str, Any] = Field(default_factory=dict)
     step_count: int = Field(default=0)
     finished: bool = Field(default=False)
-    episode_id: str = Field(default="")
+    episode_id: str | None = Field(default="")

server/pm_ops_environment.py

@@ -54,12 +54,18 @@ class PMOpsEnvironment(Environment[PMOpsAction, PMOpsObservation, PMOpsState]):
         difficulty = rng.choice(_DIFFICULTY_POOL)
         task_type = rng.choice(_TASK_TYPES)
 
+        org: Optional[Dict[str, Any]] = None
+        scenario: Optional[Dict[str, Any]] = None
+
         for attempt in range(10):
             org = generate_org_config(seed + attempt, difficulty)
             scenario = generate_scenario(task_type, org, seed + attempt)
             if _oracle_check(scenario):
                 break
 
+        if org is None or scenario is None:
+            raise RuntimeError("Failed to initialize episode state")
+
         channels = list(org["oncall_channels"].values())
         noise = org.get("noise_channels", [])
 
@@ -87,14 +93,16 @@ class PMOpsEnvironment(Environment[PMOpsAction, PMOpsObservation, PMOpsState]):
         )
 
     def step(self, action: PMOpsAction, timeout_s: Optional[float] = None, **kwargs) -> PMOpsObservation:
-        if self._ticketing is None:
+        if self._ticketing is None or self._codebase is None or self._chat is None or self._scenario is None:
             raise RuntimeError("Call reset() before step()")
 
+        scenario = self._scenario
+
         if self._done:
             return PMOpsObservation(
                 step=self._step_count,
                 max_steps=MAX_STEPS,
-                task_brief=self._scenario["brief"],
+                task_brief=scenario["brief"],
                 last_action_result={"ok": False, "error": "Episode already finished"},
                 app_state_deltas={"ticketing": [], "chat": [], "codebase": []},
                 steps_remaining=0,
@@ -122,7 +130,7 @@ class PMOpsEnvironment(Environment[PMOpsAction, PMOpsObservation, PMOpsState]):
         return PMOpsObservation(
             step=self._step_count,
             max_steps=MAX_STEPS,
-            task_brief=self._scenario["brief"],
+            task_brief=scenario["brief"],
             last_action_result=result,
             app_state_deltas=deltas,
             steps_remaining=max(0, MAX_STEPS - self._step_count),
@@ -147,6 +155,12 @@ class PMOpsEnvironment(Environment[PMOpsAction, PMOpsObservation, PMOpsState]):
         at = action.action_type
         args = action.args or {}
 
+        ticketing = self._ticketing
+        codebase = self._codebase
+        chat = self._chat
+        if ticketing is None or codebase is None or chat is None:
+            return {"ok": False, "error": "Environment not initialized. Call reset() before step()."}
+
         if at == "meta.noop":
             return {"ok": True, "data": "No operation."}
 
@@ -170,13 +184,13 @@ class PMOpsEnvironment(Environment[PMOpsAction, PMOpsObservation, PMOpsState]):
         if at.startswith("ticketing."):
             op = at.split(".", 1)[1]
             handlers = {
-                "create_ticket":     self._ticketing.create_ticket,
-                "update_ticket":     self._ticketing.update_ticket,
-                "get_ticket":        self._ticketing.get_ticket,
-                "list_tickets":      self._ticketing.list_tickets,
-                "assign_ticket":     self._ticketing.assign_ticket,
-                "comment_ticket":    self._ticketing.comment_ticket,
-                "transition_ticket": self._ticketing.transition_ticket,
+                "create_ticket":     ticketing.create_ticket,
+                "update_ticket":     ticketing.update_ticket,
+                "get_ticket":        ticketing.get_ticket,
+                "list_tickets":      ticketing.list_tickets,
+                "assign_ticket":     ticketing.assign_ticket,
+                "comment_ticket":    ticketing.comment_ticket,
+                "transition_ticket": ticketing.transition_ticket,
             }
             if op not in handlers:
                 return {"ok": False, "error": f"Unknown ticketing action: {op}"}
@@ -185,9 +199,9 @@ class PMOpsEnvironment(Environment[PMOpsAction, PMOpsObservation, PMOpsState]):
         if at.startswith("codebase."):
             op = at.split(".", 1)[1]
             handlers = {
-                "list_commits": self._codebase.list_commits,
-                "get_commit":   self._codebase.get_commit,
-                "list_prs":     self._codebase.list_prs,
+                "list_commits": codebase.list_commits,
+                "get_commit":   codebase.get_commit,
+                "list_prs":     codebase.list_prs,
             }
             if op not in handlers:
                 return {"ok": False, "error": f"Unknown codebase action: {op}"}
@@ -196,10 +210,10 @@ class PMOpsEnvironment(Environment[PMOpsAction, PMOpsObservation, PMOpsState]):
         if at.startswith("chat."):
             op = at.split(".", 1)[1]
             handlers = {
-                "post_message":  self._chat.post_message,
-                "read_channel":  self._chat.read_channel,
-                "list_channels": self._chat.list_channels,
-                "search":        self._chat.search,
+                "post_message":  chat.post_message,
+                "read_channel":  chat.read_channel,
+                "list_channels": chat.list_channels,
+                "search":        chat.search,
             }
             if op not in handlers:
                 return {"ok": False, "error": f"Unknown chat action: {op}"}
@@ -208,6 +222,9 @@ class PMOpsEnvironment(Environment[PMOpsAction, PMOpsObservation, PMOpsState]):
         return {"ok": False, "error": f"Unknown action_type: {at}"}
 
     def _grade(self) -> float:
+        if self._ticketing is None or self._chat is None or self._org_config is None or self._scenario is None:
+            return 0.0
+
         final_state = {
             "ticketing": self._ticketing.snapshot(),
             "chat": self._chat.snapshot(),

training/dataset.py

@@ -0,0 +1,110 @@
+"""Generate fixed-seed training dataset for PM-Ops triage task.
+
+Each seed is validated against the env's own RNG so the task type, difficulty,
+org_config, and scenario the env generates at reset(seed=S) EXACTLY matches the
+brief embedded in the prompt. Previously, the dataset generated triage briefs but
+the env silently ran a different task type (release_notes, dep_update, etc.) for
+the same seed — causing env_score=0 for every episode.
+"""
+import json
+import os
+import random
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from server.world.org_generator import generate_org_config
+from server.world.scenario_gen import generate_scenario
+
+SEED_PREFIX = "SEED:"
+
+# Must match pm_ops_environment.py constants exactly
+_ENV_DIFFICULTY_POOL = ["easy", "medium", "medium", "hard"]
+_ENV_TASK_TYPES = ["triage", "incident_routing", "release_notes", "dep_update"]
+
+
+def _env_params(seed: int) -> tuple[str, str]:
+    """Predict difficulty + task_type the env will choose for this seed.
+
+    Replicates the first two RNG calls in PMOpsEnvironment.reset() so we can
+    filter seeds to those that produce the task type we want.
+    """
+    rng = random.Random(seed)
+    difficulty = rng.choice(_ENV_DIFFICULTY_POOL)
+    task_type  = rng.choice(_ENV_TASK_TYPES)
+    return difficulty, task_type
+
+
+def _valid_triage(scenario: dict) -> bool:
+    exp = scenario.get("expected", {})
+    return bool(exp.get("channel")) and bool(exp.get("team"))
+
+
+def generate_triage_dataset(n_episodes: int = 150, base_seed: int = 42) -> list[dict]:
+    """Return dataset rows where the env WILL run a triage episode for the embedded seed.
+
+    We pre-simulate the env's RNG to only include seeds where:
+      1. env.reset(seed) picks task_type="triage"
+      2. The resulting org + scenario are valid (have expected channel + team)
+      3. The difficulty, org_config, and brief exactly match what the env will use
+
+    This eliminates the mismatch where the dataset had triage briefs but the env
+    graded as release_notes → guaranteed env_score=0.
+    """
+    rng = random.Random(base_seed)
+    rows = []
+
+    while len(rows) < n_episodes:
+        seed = rng.randint(0, 2**31)
+
+        # Only use seeds the env will run as triage
+        difficulty, task_type = _env_params(seed)
+        if task_type != "triage":
+            continue
+
+        # Generate org + scenario using the SAME difficulty + seed the env will use
+        org, scenario = None, None
+        for attempt in range(10):
+            org = generate_org_config(seed + attempt, difficulty)
+            scenario = generate_scenario("triage", org, seed + attempt)
+            if _valid_triage(scenario):
+                break
+
+        if not _valid_triage(scenario):
+            continue
+
+        rows.append({
+            "prompt": f"{SEED_PREFIX}{seed} | {scenario['brief']}",
+            "seed": seed,
+            "difficulty": difficulty,
+        })
+
+    return rows
+
+
+def parse_seed_from_prompt(prompt: str) -> int | None:
+    """Extract seed embedded by generate_triage_dataset."""
+    if not prompt.startswith(SEED_PREFIX):
+        return None
+    try:
+        return int(prompt[len(SEED_PREFIX):].split(" | ")[0])
+    except ValueError:
+        return None
+
+
+def save_dataset(rows: list[dict], path: str) -> None:
+    with open(path, "w") as f:
+        for row in rows:
+            f.write(json.dumps(row) + "\n")
+    print(f"Saved {len(rows)} episodes → {path}")
+
+
+def load_dataset(path: str) -> list[dict]:
+    with open(path) as f:
+        return [json.loads(line) for line in f]
+
+
+if __name__ == "__main__":
+    out = os.path.join(os.path.dirname(__file__), "triage_dataset.jsonl")
+    rows = generate_triage_dataset(n_episodes=150)
+    save_dataset(rows, out)

training/pm_ops_trainer.py

@@ -0,0 +1,187 @@
+"""PMOpsGRPOTrainer — GRPOTrainer subclass with direct reward injection.
+
+Problem being solved
+--------------------
+TRL 1.2.0 + Unsloth PatchFastRL strips custom rollout_func output keys
+before calling reward_funcs. The 'reward' key never arrives in kwargs, so
+reward_func always returns 0.0 and GRPO sees constant reward → zero gradient.
+
+Root cause: TRL builds reward_kwargs from the *dataset* batch columns (only
+'prompt'). rollout_func extra keys are not stored in the dataset, so they
+never reach _calculate_rewards via inputs.
+
+Fix
+---
+Wrap rollout_func inside __init__ using a closure that captures self directly
+(standard Python closure semantics — no forward reference tricks needed).
+Each time rollout_func is called during training, the wrapper stores the
+'reward' list into self._rollout_reward_cache. _calculate_rewards then injects
+those cached values as a tensor, bypassing the broken kwargs path entirely.
+
+Usage
+-----
+    from training.pm_ops_trainer import PMOpsGRPOTrainer
+
+    trainer = PMOpsGRPOTrainer(
+        model=model,
+        processing_class=tokenizer,
+        reward_funcs=reward_func,   # kept as fallback; not called on inject path
+        train_dataset=dataset,
+        args=grpo_config,
+        rollout_func=rollout_func,  # must return 'reward': list[float] in output
+    )
+
+rollout_func contract
+---------------------
+rollout_func(prompts, trainer=None) must return a dict containing at minimum:
+    {
+        "prompt_ids":    list[list[int]],
+        "completion_ids": list[list[int]],
+        "logprobs":      list[list[float]],
+        "reward":        list[float],   # one combined float per episode
+    }
+"""
+
+import torch
+from trl.trainer.grpo_trainer import GRPOTrainer
+
+
+class PMOpsGRPOTrainer(GRPOTrainer):
+    """Drop-in GRPOTrainer with closure-based reward caching.
+
+    Wraps rollout_func at construction time to intercept the 'reward' list
+    on every call. _calculate_rewards injects these rewards directly as a
+    tensor — no kwargs, no inputs lookup.
+    Falls back to standard GRPOTrainer behaviour when cache is empty.
+    """
+
+    def __init__(self, *args, rollout_func=None, **kwargs):
+        # Reward cache populated by the rollout wrapper, consumed by _calculate_rewards
+        self._rollout_reward_cache: list[float] = []
+        self._rollout_capture_calls = 0
+        self._warned_rollout_bypass = False
+
+        wrapped_rollout_func = None
+        if rollout_func is not None:
+            original_rollout = rollout_func
+
+            # self is captured directly by closure; accept flexible call signatures
+            # because cloud runtimes may invoke rollout_func with positional/keyword variations.
+            def _capturing_rollout(prompts, trainer=None, *rollout_args, **rollout_kwargs):
+                rollout_kwargs.setdefault("trainer", trainer)
+                result = original_rollout(prompts, *rollout_args, **rollout_kwargs)
+
+                raw_rewards = result.get("reward", result.get("rewards", []))
+                if raw_rewards is None:
+                    rewards = []
+                elif isinstance(raw_rewards, torch.Tensor):
+                    rewards = [float(r) for r in raw_rewards.detach().cpu().flatten().tolist()]
+                elif isinstance(raw_rewards, (int, float)):
+                    rewards = [float(raw_rewards)]
+                else:
+                    rewards = [float(r) for r in list(raw_rewards)]
+
+                self._rollout_reward_cache = rewards
+                self._rollout_capture_calls += 1
+                print(f"[PMOpsGRPOTrainer] captured {len(rewards)} rewards "
+                      f"(mean={sum(rewards)/len(rewards):.3f})" if rewards else
+                      "[PMOpsGRPOTrainer] WARNING: rollout returned no rewards")
+                return result
+
+            wrapped_rollout_func = _capturing_rollout
+            rollout_func = wrapped_rollout_func
+
+        super().__init__(*args, rollout_func=rollout_func, **kwargs)
+
+        # Keep wrapper bound explicitly in case an upstream patch reassigns rollout_func.
+        if wrapped_rollout_func is not None:
+            self.rollout_func = wrapped_rollout_func
+
+    def _calculate_rewards(
+        self,
+        inputs,
+        prompts,
+        completions,
+        completion_ids_list,
+    ):
+        """Inject cached rewards when available; fall back to reward_funcs otherwise."""
+        def _coerce_rewards(raw):
+            if raw is None:
+                return []
+            if isinstance(raw, torch.Tensor):
+                return [float(r) for r in raw.detach().cpu().flatten().tolist()]
+            if isinstance(raw, (int, float)):
+                return [float(raw)]
+            return [float(r) for r in list(raw)]
+
+        cache = self._rollout_reward_cache
+        n = len(completions)
+
+        # Some TRL variants forward rollout extra_fields into `inputs` directly.
+        if not cache:
+            input_rewards: list[float] = []
+            for row in inputs:
+                if isinstance(row, dict) and "reward" in row:
+                    input_rewards.append(float(row["reward"]))
+                else:
+                    input_rewards = []
+                    break
+            if input_rewards:
+                cache = input_rewards
+
+        # Cloud fallback: some patched runtimes skip the normal rollout capture path
+        # before calling _calculate_rewards. Actively invoke rollout_func here.
+        # Pass ALL n prompts (including repeated ones for num_generations > 1) so
+        # the rollout_func can generate distinct rewards per generation — NOT tile-mod.
+        if not cache and self.rollout_func is not None and prompts:
+            try:
+                print(f"[PMOpsGRPOTrainer] cache empty — probing rollout_func for {n} rewards")
+                out = self.rollout_func(list(prompts), trainer=self)
+                cache = self._rollout_reward_cache
+                if not cache and isinstance(out, dict):
+                    cache = _coerce_rewards(out.get("reward", out.get("rewards", [])))
+                    self._rollout_reward_cache = cache
+            except Exception as exc:
+                print(f"[PMOpsGRPOTrainer] rollout probe failed: {exc!r}")
+
+        if cache:
+            if len(cache) == n:
+                rewards_list = cache
+            elif len(cache) > n:
+                rewards_list = cache[:n]
+            else:
+                # Still short — extend with mean rather than tile-mod so we don't
+                # duplicate rewards for the same prompt (tile-mod → zero advantage).
+                mean_r = sum(cache) / len(cache)
+                rewards_list = list(cache) + [mean_r] * (n - len(cache))
+                print(f"[PMOpsGRPOTrainer] WARNING: cache has {len(cache)} rewards for n={n}; "
+                      f"padding with mean={mean_r:.3f}. Consider matching num_generations.")
+
+            device = self.accelerator.device
+            rewards = torch.tensor(
+                [float(r) for r in rewards_list],
+                dtype=torch.float32,
+                device=device,
+            ).unsqueeze(1)  # [batch_size, 1]
+
+            std = rewards.std().item() if n > 1 else 0.0
+            self.log({"reward/injected_mean": rewards.mean().item(),
+                      "reward/injected_std": std})
+            self._rollout_reward_cache = []  # consume cache
+            print(f"[PMOpsGRPOTrainer] injected {n} rewards "
+                  f"mean={rewards.mean().item():.3f} std={std:.3f}")
+            return rewards
+
+        if self.rollout_func is not None and self._rollout_capture_calls == 0 and not self._warned_rollout_bypass:
+            print(
+                "[PMOpsGRPOTrainer] WARNING: rollout_func was never called before reward calculation. "
+                "This cloud runtime is likely bypassing rollout_func (TRL/Unsloth mismatch), so "
+                "reward_funcs only receive prompts/completion_ids/trainer_state and no 'reward' key."
+            )
+            self._warned_rollout_bypass = True
+
+        # Fallback: standard TRL reward_funcs path
+        print("[PMOpsGRPOTrainer] cache empty — falling back to reward_funcs")
+        return super()._calculate_rewards(
+            inputs, prompts, completions, completion_ids_list
+        )

training/prompts.py

@@ -0,0 +1,94 @@
+"""System prompt and observation formatting for PM-Ops GRPO training."""
+
+SYSTEM_PROMPT = """You are an expert PM Operations agent inside a software organization.
+
+## Your Tools
+- **Ticketing** (Jira-like): create_ticket, update_ticket, assign_ticket, transition_ticket, get_ticket, list_tickets, comment_ticket
+- **Chat** (Slack-like): post_message, read_channel, list_channels, search
+- **Codebase** (GitHub-like): list_commits, get_commit, list_prs
+- **Meta**: read_runbook, finish, noop
+
+## CRITICAL — Read the Runbook First
+Every organization uses different conventions. Your FIRST action MUST be `meta.read_runbook`.
+The runbook tells you the EXACT values to use:
+- `label_taxonomy`: valid ticket labels (e.g. "defect", NOT "bug")
+- `priority_levels`: valid priorities (e.g. "critical", NOT "P1")
+- `team_map`: service → owning team
+- `oncall_channels`: service → channel to notify
+
+## Action Format
+Reason through the problem, then emit exactly ONE JSON code block as your final output:
+
+```json
+{"action_type": "meta.read_runbook", "args": {}}
+```
+
+Full list of valid action_type values:
+  meta.read_runbook | meta.finish | meta.noop
+  ticketing.create_ticket | ticketing.update_ticket | ticketing.get_ticket
+  ticketing.list_tickets | ticketing.assign_ticket | ticketing.comment_ticket
+  ticketing.transition_ticket
+  codebase.list_commits | codebase.get_commit | codebase.list_prs
+  chat.post_message | chat.read_channel | chat.list_channels | chat.search
+
+## Triage Strategy (follow this order)
+1. `meta.read_runbook` — learn this org's label, priority, team, and channel conventions
+2. `ticketing.create_ticket` — use EXACT label from label_taxonomy, EXACT priority from priority_levels
+3. `ticketing.assign_ticket` — assign to the team that owns the affected service (team_map)
+4. `chat.post_message` — post to the oncall channel for that service (oncall_channels)
+5. `meta.finish` — end the episode
+
+## Rules
+- NEVER guess label or priority values — use only what the runbook tells you
+- NEVER post to noise channels (#random, #general, #water-cooler, etc.)
+- ONLY post to the oncall channel for the affected service
+- Call `meta.finish` when done — do not exceed steps unnecessarily
+"""
+
+
+def format_observation(obs, step: int, last_action_result: dict | None = None) -> str:
+    """Convert a PMOpsObservation into a user-facing message string."""
+    if hasattr(obs, "task_brief"):
+        task_brief = obs.task_brief
+        steps_remaining = obs.steps_remaining
+        result = obs.last_action_result
+    else:
+        task_brief = obs.get("task_brief", "")
+        steps_remaining = obs.get("steps_remaining", 40)
+        result = obs.get("last_action_result", {})
+
+    if last_action_result is not None:
+        result = last_action_result
+
+    parts = [f"## Task\n{task_brief}", f"\n**Step {step}** | {steps_remaining} steps remaining"]
+
+    if result:
+        ok = result.get("ok", False)
+        if ok:
+            data = result.get("data", "")
+            if isinstance(data, dict) and "org_config" in data:
+                org = data["org_config"]
+                labels     = list(org.get("label_taxonomy", {}).values())
+                priorities = org.get("priority_levels", [])
+                team_map   = org.get("team_map", {})
+                channels   = org.get("oncall_channels", {})
+                noise      = org.get("noise_channels", [])
+                lines = [
+                    "=== RUNBOOK ===",
+                    f"Valid labels      : {', '.join(labels)}",
+                    f"Valid priorities  : {', '.join(priorities)}",
+                    f"Team map          : {' | '.join(f'{s} -> {t}' for s, t in team_map.items())}",
+                    f"Oncall channels   : {' | '.join(f'{s} -> {c}' for s, c in channels.items())}",
+                    f"Noise channels    : {', '.join(noise)}  <-- NEVER post here",
+                    "=== END RUNBOOK ===",
+                ]
+                data_str = "\n".join(lines)
+            else:
+                data_str = str(data)[:2000]
+            parts.append(f"\n**Result (success):**\n{data_str}")
+        else:
+            error = result.get("error", "Unknown error")
+            parts.append(f"\n**Result (error):** {error}")
+
+    parts.append("\nWhat is your next action? Think it through, then output a JSON code block.")
+    return "\n".join(parts)

training/rewards.py

@@ -0,0 +1,100 @@
+"""Runbook-compliance reward for PM-Ops GRPO training.
+
+Design philosophy (v4)
+----------------------
+Previous reward formulas (json_ratio, ok_ratio, diversity_bonus) were all constant
+once the model learned to output valid JSON — giving zero GRPO advantage.
+
+Root problem: env_score was always 0 because:
+  1. The model used hardcoded label='sec-issue' / priority='P1', which fail
+     TicketingApp validation for most org configs → no ticket created → grader
+     returns 0.0 immediately.
+  2. Dataset seeds produced triage briefs but the env ran a DIFFERENT task type
+     (release_notes, dep_update) for the same seed → guaranteed mismatch.
+
+Fix: reward is computed from the agent's ACTIONS compared to RUNBOOK DATA.
+The org_config (returned by meta.read_runbook) varies by seed — different orgs
+have different valid labels, priorities, teams, and oncall channels. The model's
+fixed template (sec-issue, P1, infra) scores well for some orgs and badly for
+others, creating the reward VARIANCE that GRPO needs.
+
+Components (sum = 1.0 when all correct):
+  read_runbook   0.10  — process: did agent read runbook first?
+  valid_label    0.20  — used a label from label_taxonomy? (+0.20 / -0.10)
+  valid_priority 0.15  — used a priority from priority_levels? (+0.15 / -0.10)
+  valid_team     0.20  — assigned to a team from team_map? (+0.20 / -0.10)
+  right_channel  0.25  — posted to an oncall channel? (+0.25 / -0.10 per wrong)
+  env_bonus      0.10  — env grader bonus when everything lines up correctly
+
+This is computed IN the rollout (not by TRL reward_funcs) because it needs
+access to the per-episode runbook data.
+"""
+
+
+def compute_rollout_reward(
+    *,
+    read_runbook_done: bool,
+    valid_labels: set,        # from org_config.label_taxonomy.values()
+    valid_priorities: set,    # from org_config.priority_levels
+    valid_teams: set,         # from org_config.team_map.values()
+    oncall_channels: set,     # from org_config.oncall_channels.values()
+    ticket_label: str | None,       # label used in create_ticket (None if no ticket)
+    ticket_priority: str | None,    # priority used in create_ticket
+    assigned_team: str | None,      # team from assign_ticket (None if not called)
+    posted_channels: list[str],     # all channels from chat.post_message
+    env_score: float,               # final env grader score (0–1)
+    valid_json_count: int,          # steps with parseable JSON output
+) -> float:
+    """Compute a single reward scalar for one episode.
+
+    Returns a value in [-1.0, 1.0].
+    """
+    if valid_json_count == 0:
+        return -1.0
+
+    reward = 0.0
+
+    # 1. Runbook read (+0.10 process bonus)
+    if read_runbook_done:
+        reward += 0.10
+
+    # 2. Ticket label valid (only scored if a ticket was created)
+    if ticket_label is not None:
+        if valid_labels:
+            if ticket_label in valid_labels:
+                reward += 0.20
+            else:
+                reward -= 0.10  # wrong label — validation would have rejected it
+
+    # 3. Ticket priority valid
+    if ticket_priority is not None:
+        if valid_priorities:
+            if ticket_priority in valid_priorities:
+                reward += 0.15
+            else:
+                reward -= 0.10
+
+    # 4. Ticket assigned to a valid team
+    if assigned_team is not None:
+        if valid_teams:
+            if assigned_team in valid_teams:
+                reward += 0.20
+            else:
+                reward -= 0.10
+
+    # 5. Posted to the right oncall channel
+    if posted_channels:
+        if oncall_channels:
+            correct_posts = [ch for ch in posted_channels if ch in oncall_channels]
+            wrong_posts   = [ch for ch in posted_channels if ch not in oncall_channels]
+            if correct_posts:
+                reward += 0.25
+            reward -= 0.10 * len(wrong_posts)  # -0.10 per channel-spray post
+        else:
+            # Posted without reading runbook — can't verify, mild penalty
+            reward -= 0.05 * len(posted_channels)
+
+    # 6. Env grader bonus — full env score adds on top when everything is correct
+    reward += env_score * 0.10
+
+    return max(-1.0, min(1.0, reward))

training/rollout.py

@@ -0,0 +1,430 @@
+"""Multi-turn rollout function for PM-Ops GRPO training.
+
+History design (no duplicates):
+  Each turn record stores the obs that TRIGGERED the completion, not the result.
+  build_messages reconstructs:
+    [sys] [user:obs_0] [asst:comp_0] [user:obs_1] [asst:comp_1] ... [user:current_obs]
+  current_obs is never in history — it becomes the final user message.
+
+Other design decisions:
+- Action format B: chain-of-thought reasoning + ```json block
+- Truncation: runbook-pinned sliding window (runbook pair always kept)
+- Fallback cascade: read_runbook (early) → noop (mid) → finish (late)
+- Three-pass JSON extraction: code block → raw JSON → regex
+"""
+import json
+import re
+from typing import Any
+
+import torch
+import torch.nn.functional as F
+
+from training.dataset import parse_seed_from_prompt
+from training.prompts import SYSTEM_PROMPT, format_observation
+
+
+# ---------------------------------------------------------------------------
+# HF model.generate() — replaces generate_rollout_completions (vLLM-only)
+# ---------------------------------------------------------------------------
+
+def _get_model_for_generation(trainer):
+    """Unwrap the model safely regardless of accelerate/PEFT/DDP wrapping.
+
+    Priority:
+      1. accelerator.unwrap_model — handles DDP + PEFT + DeepSpeed
+      2. trainer.model.module     — plain DDP wrapping
+      3. trainer.model            — unwrapped (local or single-GPU)
+    """
+    if hasattr(trainer, "accelerator"):
+        return trainer.accelerator.unwrap_model(trainer.model)
+    if hasattr(trainer.model, "module"):
+        return trainer.model.module
+    return trainer.model
+
+
+def _generate_no_vllm(trainer, prompt_text: str, tokenizer,
+                      max_new_tokens: int = 512, temperature: float = 1.1) -> dict:
+    """Generate one completion using HF model.generate() without vLLM.
+
+    Returns the same dict shape as generate_rollout_completions so the rest
+    of rollout_once is unchanged:
+        prompt_ids:     list[int]
+        completion_ids: list[int]
+        logprobs:       list[float]  (per-token log-prob under current policy)
+        text:           str
+    """
+    model = _get_model_for_generation(trainer)
+
+    # Device: prefer accelerator.device, fall back to first param device
+    if hasattr(trainer, "accelerator"):
+        device = trainer.accelerator.device
+    else:
+        device = next(model.parameters()).device
+
+    enc = tokenizer(prompt_text, return_tensors="pt").to(device)
+    prompt_len = enc["input_ids"].shape[1]
+
+    with torch.no_grad():
+        out = model.generate(
+            **enc,
+            max_new_tokens=max_new_tokens,
+            do_sample=True,
+            temperature=temperature,
+            top_p=0.95,
+            top_k=50,
+            pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
+            output_scores=True,
+            return_dict_in_generate=True,
+        )
+
+    prompt_ids     = enc["input_ids"][0].tolist()
+    completion_ids = out.sequences[0][prompt_len:].tolist()
+
+    # Per-token log-probs from output.scores (one score tensor per new token)
+    logprobs = [
+        F.log_softmax(score[0], dim=-1)[tok_id].item()
+        for score, tok_id in zip(out.scores, completion_ids)
+    ]
+
+    text = tokenizer.decode(completion_ids, skip_special_tokens=True)
+    return {
+        "prompt_ids":     prompt_ids,
+        "completion_ids": completion_ids,
+        "logprobs":       logprobs,
+        "text":           text,
+    }
+
+MAX_STEPS = 40
+# ~3000 tokens at 4 chars/token; leaves room for completion tokens
+MAX_PROMPT_CHARS = 12_000
+RUNBOOK_RESPONSE_MAX_CHARS = 3_000
+HISTORY_PAIRS = 6  # max (user+asst) pairs kept from non-runbook history
+
+
+# ---------------------------------------------------------------------------
+# JSON extraction — three-pass, most-specific first
+# ---------------------------------------------------------------------------
+
+def extract_json_action(text: str) -> dict | None:
+    # Pass 1: last ```json ... ``` block
+    matches = re.findall(r"```json\s*(\{.*?\})\s*```", text, re.DOTALL)
+    if matches:
+        try:
+            return json.loads(matches[-1])
+        except json.JSONDecodeError:
+            pass
+
+    # Pass 2: entire output is JSON
+    stripped = text.strip()
+    if stripped.startswith("{"):
+        try:
+            return json.loads(stripped)
+        except json.JSONDecodeError:
+            pass
+
+    # Pass 3: any {...action_type...} pattern — allow one level of nested {} (e.g. "args": {})
+    matches = re.findall(
+        r'\{(?:[^{}]|\{[^{}]*\})*"action_type"\s*:\s*"[^"]*"(?:[^{}]|\{[^{}]*\})*\}',
+        text, re.DOTALL,
+    )
+    if matches:
+        try:
+            return json.loads(matches[-1])
+        except json.JSONDecodeError:
+            pass
+
+    return None
+
+
+def step_aware_fallback(step: int, max_steps: int = MAX_STEPS) -> dict:
+    """Safe fallback that degrades gracefully across the episode."""
+    if step <= 1:
+        return {"action_type": "meta.read_runbook", "args": {}}
+    elif step >= max_steps - 3:
+        return {"action_type": "meta.finish", "args": {}}
+    return {"action_type": "meta.noop", "args": {}}
+
+
+# ---------------------------------------------------------------------------
+# Context builder — runbook-pinned sliding window, no duplicate user turns
+# ---------------------------------------------------------------------------
+
+def _chars(messages: list[dict]) -> int:
+    return sum(len(m["content"]) for m in messages)
+
+
+def build_messages(
+    turn_history: list[dict],
+    current_obs_text: str,
+) -> list[dict]:
+    """Build prompt messages with runbook-pinned sliding window truncation.
+
+    turn_history entries: {"obs_text": str, "completion": str, "is_runbook": bool}
+      obs_text   = observation that triggered this completion (user side)
+      completion = model output for that step (assistant side)
+
+    Final conversation shape:
+      [sys] [user:obs_0][asst:comp_0] ... [user:obs_k][asst:comp_k] [user:current_obs]
+    No entry in turn_history represents current_obs — it's only the final user turn.
+    """
+    # Task brief is always prepended to current_obs so it stays visible even
+    # when old context is truncated.
+    messages: list[dict] = [{"role": "system", "content": SYSTEM_PROMPT}]
+
+    # Separate the runbook exchange from the rest
+    runbook_turn: dict | None = None
+    general: list[dict] = []
+
+    for turn in turn_history:
+        if turn["is_runbook"] and runbook_turn is None:
+            runbook_turn = turn
+        else:
+            general.append(turn)
+
+    # Pin runbook exchange (truncate only if absurdly large)
+    if runbook_turn is not None:
+        rb_comp = runbook_turn["completion"][:RUNBOOK_RESPONSE_MAX_CHARS]
+        messages.append({"role": "user", "content": runbook_turn["obs_text"]})
+        messages.append({"role": "assistant", "content": rb_comp})
+
+    # Fill budget with most-recent general turns (newest first, then reverse)
+    fixed_chars = _chars(messages) + len(current_obs_text)
+    budget = MAX_PROMPT_CHARS - fixed_chars
+    window: list[dict] = []
+
+    for turn in reversed(general[-HISTORY_PAIRS:]):
+        pair_chars = len(turn["obs_text"]) + len(turn["completion"])
+        if budget - pair_chars < 0:
+            break
+        window.append(turn)
+        budget -= pair_chars
+
+    for turn in reversed(window):
+        messages.append({"role": "user", "content": turn["obs_text"]})
+        messages.append({"role": "assistant", "content": turn["completion"]})
+
+    # Current observation is always the final user turn (never stored in history)
+    messages.append({"role": "user", "content": current_obs_text})
+    return messages
+
+
+# ---------------------------------------------------------------------------
+# Observation normaliser — handles both object and dict forms
+# ---------------------------------------------------------------------------
+
+def _obs_to_dict(obs: Any) -> dict:
+    if isinstance(obs, dict):
+        return obs
+    fields = ("task_brief", "last_action_result", "step", "steps_remaining",
+              "reward", "done", "token_budget_remaining")
+    return {f: getattr(obs, f, None) for f in fields}
+
+
+def _current_obs_text(obs_dict: dict, step: int, task_brief: str) -> str:
+    """Format observation, prepending a task reminder so it survives truncation."""
+    reminder = f"**Task reminder:** {task_brief[:200]}\n\n"
+    return reminder + format_observation(obs_dict, step, obs_dict.get("last_action_result"))
+
+
+# ---------------------------------------------------------------------------
+# Single-episode rollout
+# ---------------------------------------------------------------------------
+
+def rollout_once(
+    trainer,
+    sync_env,
+    tokenizer,
+    dataset_prompt: str,
+    max_steps: int = 15,
+    gen_offset: int = 0,
+) -> dict:
+    """Play one full PM-Ops episode. Returns trajectory + reward signals.
+
+    gen_offset: added to seed so each GRPO generation explores a different env
+    episode even when receiving the same prompt (same base seed).
+    """
+    seed = parse_seed_from_prompt(dataset_prompt)
+    if seed is not None:
+        result = sync_env.reset(seed=seed + gen_offset)
+    else:
+        result = sync_env.reset()
+
+    obs = result.observation if hasattr(result, "observation") else result
+    obs_dict = _obs_to_dict(obs)
+    task_brief: str = obs_dict.get("task_brief") or dataset_prompt
+
+    # Flat trajectory buffers (TRL expects flat lists across all steps)
+    prompt_ids: list = []
+    completion_ids: list = []
+    logprobs: list = []
+
+    # Turn history for context building
+    # Each entry: {"obs_text": str, "completion": str, "is_runbook": bool}
+    turn_history: list[dict] = []
+
+    # Rollout accumulators
+    valid_action_count = 0
+    final_score = 0.0
+    step = 0
+    done = False
+
+    # Runbook-compliance reward tracking
+    read_runbook_done  = False
+    valid_labels:     set[str] = set()   # org label_taxonomy values
+    valid_priorities: set[str] = set()   # org priority_levels
+    valid_teams:      set[str] = set()   # org team_map values
+    oncall_channels:  set[str] = set()   # org oncall_channels values
+    ticket_label:    str | None = None   # label used in create_ticket
+    ticket_priority: str | None = None   # priority used in create_ticket
+    assigned_team:   str | None = None   # team from assign_ticket
+    posted_channels: list[str] = []      # every channel posted to
+
+    while not done and step < max_steps:
+        # obs_text for THIS step — stored in history BEFORE stepping
+        obs_text = _current_obs_text(obs_dict, step, task_brief)
+
+        messages = build_messages(turn_history, obs_text)
+        prompt_text = tokenizer.apply_chat_template(
+            messages,
+            add_generation_prompt=True,
+            tokenize=False,
+            enable_thinking=False,
+        )
+
+        rollout_out = _generate_no_vllm(trainer, prompt_text, tokenizer)
+        prompt_ids.extend(rollout_out["prompt_ids"])
+        completion_ids.extend(rollout_out["completion_ids"])
+        logprobs.extend(rollout_out["logprobs"])
+
+        completion_text = rollout_out["text"]
+
+        # Parse action; fall back gracefully on parse failure
+        parsed = extract_json_action(completion_text)
+        is_valid_json = parsed is not None
+
+        if not is_valid_json:
+            # Log the raw output on step 0 to diagnose format failures
+            if step == 0 and valid_action_count == 0:
+                snippet = repr(completion_text[:300])
+                print(f"[rollout] step=0 NO JSON — raw output: {snippet}")
+            parsed = step_aware_fallback(step, max_steps)
+        else:
+            valid_action_count += 1
+
+        action_type: str = parsed.get("action_type", "meta.noop")
+        args: dict = parsed.get("args", {})
+
+        if action_type == "meta.read_runbook" and is_valid_json:
+            read_runbook_done = True
+
+        if action_type == "ticketing.create_ticket" and is_valid_json and ticket_label is None:
+            ticket_label    = args.get("label")
+            ticket_priority = args.get("priority")
+
+        if action_type == "ticketing.assign_ticket" and is_valid_json and assigned_team is None:
+            assigned_team = args.get("team")
+
+        if action_type == "chat.post_message" and is_valid_json:
+            ch = args.get("channel", "")
+            if ch:
+                posted_channels.append(ch)
+
+        # Store the (obs_text, completion) pair BEFORE stepping the env
+        # is_runbook marks this turn for pinning in future context windows
+        turn_history.append({
+            "obs_text": obs_text,
+            "completion": completion_text,
+            "is_runbook": (action_type == "meta.read_runbook" and is_valid_json),
+        })
+
+        # Step the environment — obs_dict now holds the NEXT state
+        result = sync_env.step({"action_type": action_type, "args": args})
+        new_obs = result.observation if hasattr(result, "observation") else result
+        obs_dict = _obs_to_dict(new_obs)
+
+        # Extract full org_config from runbook response (one step after the call)
+        last_result = obs_dict.get("last_action_result") or {}
+        if action_type == "meta.read_runbook" and last_result.get("ok"):
+            data = last_result.get("data") or {}
+            if isinstance(data, dict):
+                org = data.get("org_config") or {}
+                valid_labels     = set(org.get("label_taxonomy", {}).values())
+                valid_priorities = set(org.get("priority_levels", []))
+                valid_teams      = set(org.get("team_map", {}).values())
+                oncall_channels  = set(org.get("oncall_channels", {}).values())
+
+        done = bool(getattr(result, "done", obs_dict.get("done", False)))
+        final_score = float(getattr(result, "reward", obs_dict.get("reward", 0.0)))
+        step += 1
+
+    from training.rewards import compute_rollout_reward
+    combined = compute_rollout_reward(
+        read_runbook_done  = read_runbook_done,
+        valid_labels       = valid_labels,
+        valid_priorities   = valid_priorities,
+        valid_teams        = valid_teams,
+        oncall_channels    = oncall_channels,
+        ticket_label       = ticket_label,
+        ticket_priority    = ticket_priority,
+        assigned_team      = assigned_team,
+        posted_channels    = posted_channels,
+        env_score          = final_score,
+        valid_json_count   = valid_action_count,
+    )
+
+    print(
+        f"[rollout] steps={step} env={final_score:.3f} "
+        f"label={'✓' if ticket_label and ticket_label in valid_labels else '✗' if ticket_label else '-'} "
+        f"priority={'✓' if ticket_priority and ticket_priority in valid_priorities else '✗' if ticket_priority else '-'} "
+        f"team={'✓' if assigned_team and assigned_team in valid_teams else '✗' if assigned_team else '-'} "
+        f"channel={'✓' if any(ch in oncall_channels for ch in posted_channels) else '✗' if posted_channels else '-'} "
+        f"→ reward={combined:.3f}"
+    )
+    return {
+        "prompt_ids":     prompt_ids,
+        "completion_ids": completion_ids,
+        "logprobs":       logprobs,
+        "reward":         combined,
+    }
+
+
+# ---------------------------------------------------------------------------
+# GRPOTrainer-compatible rollout function (factory)
+# ---------------------------------------------------------------------------
+
+def make_rollout_func(sync_env, tokenizer, max_steps: int = 15):
+    """Bind env + tokenizer; return the function GRPOTrainer calls each batch.
+
+    max_steps per task:
+      triage           → 15  (solvable in 5, cap gives room for exploration)
+      incident_routing → 20
+      release_notes    → 30
+      dep_update       → 30
+    """
+    def rollout_func(prompts: list[str], trainer=None) -> dict:
+        out: dict[str, list] = {
+            "prompt_ids":     [],
+            "completion_ids": [],
+            "logprobs":       [],
+            "reward":         [],
+        }
+        # Track how many times each unique prompt has appeared so we can pass
+        # a gen_offset — ensures repeated prompts (num_generations > 1) hit
+        # different env seeds and produce different rollouts.
+        prompt_seen: dict[str, int] = {}
+        for prompt_text in prompts:
+            gen_offset = prompt_seen.get(prompt_text, 0)
+            prompt_seen[prompt_text] = gen_offset + 1
+            episode = rollout_once(
+                trainer=trainer,
+                sync_env=sync_env,
+                tokenizer=tokenizer,
+                dataset_prompt=prompt_text,
+                max_steps=max_steps,
+                gen_offset=gen_offset,
+            )
+            for k in out:
+                out[k].append(episode[k])
+        return out
+
+    return rollout_func

training/smoke_test.py

@@ -0,0 +1,267 @@
+"""Smoke test — runs locally without TRL/vLLM.
+
+Tests in order:
+  1. Dataset generation
+  2. JSON extraction (edge cases)
+  3. build_messages structure (no duplicates, correct alternation)
+  4. Server startup + env connection
+  5. One full heuristic episode (env API end-to-end)
+
+Run from repo root:  python training/smoke_test.py
+"""
+import json
+import os
+import subprocess
+import sys
+import time
+
+import requests
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+PASS = "\033[92m[PASS]\033[0m"
+FAIL = "\033[91m[FAIL]\033[0m"
+INFO = "\033[94m[INFO]\033[0m"
+
+
+def check(label: str, cond: bool, detail: str = "") -> bool:
+    if cond:
+        print(f"{PASS} {label}")
+    else:
+        print(f"{FAIL} {label}" + (f" — {detail}" if detail else ""))
+    return cond
+
+
+# ---------------------------------------------------------------------------
+# 1. Dataset
+# ---------------------------------------------------------------------------
+print("\n── 1. Dataset ──────────────────────────────────────────────────────")
+from training.dataset import generate_triage_dataset, parse_seed_from_prompt
+
+rows = generate_triage_dataset(n_episodes=5, base_seed=99)
+check("generates 5 rows", len(rows) == 5)
+check("each row has prompt+seed+difficulty", all(
+    "prompt" in r and "seed" in r and "difficulty" in r for r in rows
+))
+seed_back = parse_seed_from_prompt(rows[0]["prompt"])
+check("seed round-trips through prompt string", seed_back == rows[0]["seed"],
+      f"got {seed_back}, expected {rows[0]['seed']}")
+print(f"  sample prompt: {rows[0]['prompt'][:100]}...")
+
+
+# ---------------------------------------------------------------------------
+# 2. JSON extraction
+# ---------------------------------------------------------------------------
+print("\n── 2. JSON extraction ──────────────────────────────────────────────")
+from training.rollout import extract_json_action, step_aware_fallback
+
+cases = [
+    # (description, input_text, expected_action_type)
+    ("code block", 'Thinking...\n```json\n{"action_type": "meta.read_runbook", "args": {}}\n```', "meta.read_runbook"),
+    ("raw JSON",   '{"action_type": "meta.finish", "args": {}}', "meta.finish"),
+    ("regex hit",  'I will do this: {"action_type": "meta.noop", "args": {}} done.', "meta.noop"),
+    ("no JSON",    "I cannot decide.", None),
+    ("bad block",  '```json\n{broken\n```', None),
+]
+for desc, text, expected in cases:
+    result = extract_json_action(text)
+    got = result.get("action_type") if result else None
+    check(f"extract: {desc}", got == expected, f"got {got!r}, expected {expected!r}")
+
+fb = step_aware_fallback(0)
+check("fallback step 0 → read_runbook", fb["action_type"] == "meta.read_runbook")
+fb = step_aware_fallback(20)
+check("fallback step 20 → noop", fb["action_type"] == "meta.noop")
+fb = step_aware_fallback(38)
+check("fallback step 38 → finish", fb["action_type"] == "meta.finish")
+
+# reward_no_wrong_channels logic
+from training.rewards import (
+    WEIGHT_FINAL_SCORE, WEIGHT_NO_WRONG_CHANNELS,
+    WEIGHT_VALID_JSON, WEIGHT_READ_RUNBOOK, WEIGHT_EFFICIENCY,
+)
+weights_sum = (WEIGHT_FINAL_SCORE + WEIGHT_NO_WRONG_CHANNELS +
+               WEIGHT_VALID_JSON + WEIGHT_READ_RUNBOOK + WEIGHT_EFFICIENCY)
+check("reward weights sum to 1.0", abs(weights_sum - 1.0) < 1e-9, f"sum={weights_sum}")
+
+
+# ---------------------------------------------------------------------------
+# 3. build_messages structure
+# ---------------------------------------------------------------------------
+print("\n── 3. build_messages ───────────────────────────────────────────────")
+from training.rollout import build_messages
+
+# Empty history (step 0)
+msgs = build_messages([], "First observation")
+check("step 0: starts with system", msgs[0]["role"] == "system")
+check("step 0: ends with user", msgs[-1]["role"] == "user")
+check("step 0: final user is current obs", msgs[-1]["content"] == "First observation")
+
+# Simulate 3 turns
+history = [
+    {"obs_text": "obs_0", "completion": "comp_0", "is_runbook": True},
+    {"obs_text": "obs_1", "completion": "comp_1", "is_runbook": False},
+    {"obs_text": "obs_2", "completion": "comp_2", "is_runbook": False},
+]
+msgs = build_messages(history, "current_obs")
+
+# Validate alternation: after system, must be user/asst/user/asst/.../user
+roles = [m["role"] for m in msgs]
+check("roles start with system", roles[0] == "system")
+check("roles end with user", roles[-1] == "user")
+pairs_ok = all(
+    roles[i] == "user" and roles[i+1] == "assistant"
+    for i in range(1, len(roles) - 2, 2)
+)
+check("strict user/asst alternation throughout", pairs_ok, str(roles))
+
+# Verify current_obs appears exactly once as the last message
+final_user_content = msgs[-1]["content"]
+check("current_obs is last message content", final_user_content == "current_obs")
+all_contents = [m["content"] for m in msgs]
+check("current_obs not duplicated", all_contents.count("current_obs") == 1)
+
+# Verify runbook is pinned (obs_0/comp_0 should appear even with many turns)
+all_content_str = " ".join(all_contents)
+check("runbook obs pinned", "obs_0" in all_content_str)
+check("runbook completion pinned", "comp_0" in all_content_str)
+
+
+# ---------------------------------------------------------------------------
+# 4. Server startup
+# ---------------------------------------------------------------------------
+print("\n── 4. Server startup ───────────────────────────────────────────────")
+REPO_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ENV_URL = "http://localhost:8765"  # non-standard port to avoid conflicts
+
+proc = subprocess.Popen(
+    [sys.executable, "-m", "uvicorn", "server.app:app",
+     "--host", "0.0.0.0", "--port", "8765"],
+    cwd=REPO_DIR,
+    stdout=subprocess.DEVNULL,
+    stderr=subprocess.DEVNULL,
+)
+
+ready = False
+for _ in range(20):
+    try:
+        r = requests.get(f"{ENV_URL}/", timeout=1)
+        if r.status_code == 200:
+            ready = True
+            break
+    except Exception:
+        pass
+    time.sleep(1)
+
+check("server started on :8765", ready)
+if not ready:
+    proc.terminate()
+    print(f"{FAIL} Cannot continue without server.")
+    sys.exit(1)
+
+
+# ---------------------------------------------------------------------------
+# 5. Full heuristic episode (env API end-to-end)
+# ---------------------------------------------------------------------------
+print("\n── 5. End-to-end episode ───────────────────────────────────────────")
+from openenv.core import GenericEnvClient
+
+client = GenericEnvClient(base_url=ENV_URL).sync()
+client.connect()
+
+try:
+    # reset with explicit seed (tests seed passthrough)
+    res = client.reset(seed=rows[0]["seed"])
+    obs = res.observation if hasattr(res, "observation") else res
+
+    def _get(o, k):
+        return getattr(o, k, None) if not isinstance(o, dict) else o.get(k)
+
+    task_brief = _get(obs, "task_brief")
+    check("reset returns task_brief", bool(task_brief), repr(task_brief)[:80])
+
+    # Step 1: read runbook
+    res = client.step({"action_type": "meta.read_runbook", "args": {}})
+    obs = res.observation if hasattr(res, "observation") else res
+    last = _get(obs, "last_action_result") or {}
+    check("read_runbook ok=True", last.get("ok") is True)
+    check("runbook contains org_config", "org_config" in str(last.get("data", "")))
+
+    org = last.get("data", {}).get("org_config", {}) if isinstance(last.get("data"), dict) else {}
+    labels = list(org.get("label_taxonomy", {}).values())
+    priorities = org.get("priority_levels", [])
+    teams = list(org.get("team_map", {}).values())
+    channels = list(org.get("oncall_channels", {}).values())
+
+    check("org has labels", len(labels) > 0)
+    check("org has priorities", len(priorities) > 0)
+
+    # Step 2: create ticket with correct label/priority
+    res = client.step({
+        "action_type": "ticketing.create_ticket",
+        "args": {
+            "summary": f"Triage: {task_brief[:60]}",
+            "description": task_brief,
+            "label": labels[0],
+            "priority": priorities[1] if len(priorities) > 1 else priorities[0],
+            "assignee": teams[0] if teams else "backend",
+        },
+    })
+    obs = res.observation if hasattr(res, "observation") else res
+    last = _get(obs, "last_action_result") or {}
+    check("create_ticket ok=True", last.get("ok") is True, str(last.get("error")))
+    ticket_id = (last.get("data") or {}).get("id") if isinstance(last.get("data"), dict) else None
+    check("ticket_id returned", bool(ticket_id), repr(ticket_id))
+
+    # Step 3: assign ticket
+    if ticket_id and teams:
+        res = client.step({
+            "action_type": "ticketing.assign_ticket",
+            "args": {"ticket_id": ticket_id, "team": teams[0]},
+        })
+        obs = res.observation if hasattr(res, "observation") else res
+        last = _get(obs, "last_action_result") or {}
+        check("assign_ticket ok=True", last.get("ok") is True, str(last.get("error")))
+
+    # Step 4: post to oncall channel
+    if channels:
+        res = client.step({
+            "action_type": "chat.post_message",
+            "args": {"channel": channels[0], "text": f"Triage smoke test: {task_brief[:80]}"},
+        })
+        obs = res.observation if hasattr(res, "observation") else res
+        last = _get(obs, "last_action_result") or {}
+        check("post_message ok=True", last.get("ok") is True, str(last.get("error")))
+
+    # Step 5: finish + check reward
+    res = client.step({"action_type": "meta.finish", "args": {}})
+    done = getattr(res, "done", None)
+    reward = getattr(res, "reward", 0.0)
+    check("episode done after finish", done is True, f"done={done}")
+    check("reward is float in [0,1]", isinstance(reward, float) and 0.0 <= reward <= 1.0,
+          f"reward={reward}")
+    print(f"  episode reward: {reward:.3f}")
+
+    # Verify make_rollout_func accepts max_steps param
+    from training.rollout import make_rollout_func
+    import inspect
+    sig = inspect.signature(make_rollout_func)
+    check("make_rollout_func has max_steps param", "max_steps" in sig.parameters)
+    check("max_steps default is 15", sig.parameters["max_steps"].default == 15)
+
+    # Verify no_wrong_channels_reward key in rollout output keys
+    from training.rollout import rollout_once as _rollout_once
+    sig2 = inspect.signature(_rollout_once)
+    check("rollout_once has max_steps param", "max_steps" in sig2.parameters)
+
+finally:
+    client.close()
+
+proc.terminate()
+
+# ---------------------------------------------------------------------------
+# Summary
+# ---------------------------------------------------------------------------
+print("\n────────────────────────────────────────────────────────────────────")
+print("Smoke test complete. If all checks passed, training pipeline is ready.")
+print("Next: push repo to HF and run training/train.ipynb on A100 GPU Space.")

training/train.ipynb

@@ -0,0 +1,669 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# PM-Ops: Train a Project Management Agent with GRPO + Unsloth\n",
+    "\n",
+    "Fine-tune **Qwen3-1.7B** on PM-Ops triage tasks using GRPO (TRL) + Unsloth for memory efficiency.\n",
+    "\n",
+    "**GPU:** A100 40GB (HF Jupyter Space — uses HF credits)  \n",
+    "**Time:** ~90 min (150 triage episodes, 1 epoch)  \n",
+    "**Stack:** OpenEnv + TRL + Unsloth (hackathon-recommended stack)\n",
+    "\n",
+    "### Architecture\n",
+    "```\n",
+    "[HF A100 GPU Space]\n",
+    "  ├── Unsloth + GRPOTrainer  ← Qwen3-1.7B with LoRA, 4-bit quantised\n",
+    "  └── PM-Ops FastAPI (localhost:8000)  ← subprocess, <1ms per step\n",
+    "```\n",
+    "\n",
+    "### Reward design (5 independent signals — anti-hack per hackathon guide)\n",
+    "| Signal | Weight | Purpose |\n",
+    "|---|---|---|\n",
+    "| `reward_final_score` | 0.45 | Correctness: label + priority + team + channel |\n",
+    "| `reward_no_wrong_channels` | 0.15 | Anti-hack: penalise channel-spray |\n",
+    "| `reward_valid_json` | 0.15 | Format discipline |\n",
+    "| `reward_read_runbook` | 0.15 | Process: read before acting |\n",
+    "| `reward_efficiency` | 0.10 | Speed when correct |"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install Dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Unsloth first — it pins compatible versions of torch/transformers\n",
+    "!pip install -Uq unsloth\n",
+    "!pip install -Uq \"trl>=0.17.0\" openenv-core datasets trackio\n",
+    "print('Dependencies installed.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Clone PM-Ops Repo"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os, sys\n",
+    "\n",
+    "REPO_URL = 'https://huggingface.co/spaces/adityaguntur/pm-ops'\n",
+    "REPO_DIR = '/content/pm_ops'\n",
+    "\n",
+    "if not os.path.exists(REPO_DIR):\n",
+    "    !git clone --depth=1 -q {REPO_URL} {REPO_DIR}\n",
+    "    print(f'Cloned → {REPO_DIR}')\n",
+    "else:\n",
+    "    print(f'Already exists: {REPO_DIR}')\n",
+    "\n",
+    "for p in [REPO_DIR, os.path.join(REPO_DIR, 'training')]:\n",
+    "    if p not in sys.path:\n",
+    "        sys.path.insert(0, p)\n",
+    "\n",
+    "os.chdir(REPO_DIR)\n",
+    "print(f'Working directory: {os.getcwd()}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. HuggingFace Login"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import notebook_login\n",
+    "notebook_login()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Start PM-Ops Server Locally\n",
+    "\n",
+    "Running server on localhost eliminates the ~200ms/step network cost of calling the HF Space.\n",
+    "With 15 steps × 150 episodes × 2 generations that's 4,500 saved round-trips."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess, time, requests\n",
+    "\n",
+    "server_proc = subprocess.Popen(\n",
+    "    [sys.executable, '-m', 'uvicorn', 'server.app:app',\n",
+    "     '--host', '0.0.0.0', '--port', '8000'],\n",
+    "    cwd=REPO_DIR,\n",
+    "    stdout=subprocess.DEVNULL,\n",
+    "    stderr=subprocess.DEVNULL,\n",
+    ")\n",
+    "\n",
+    "ENV_URL = 'http://localhost:8000'\n",
+    "\n",
+    "for i in range(30):\n",
+    "    try:\n",
+    "        if requests.get(f'{ENV_URL}/', timeout=2).status_code == 200:\n",
+    "            print(f'PM-Ops server ready at {ENV_URL}  (pid={server_proc.pid})')\n",
+    "            break\n",
+    "    except Exception:\n",
+    "        pass\n",
+    "    time.sleep(1)\n",
+    "else:\n",
+    "    raise RuntimeError('Server did not start in 30 s — check uvicorn install.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Verify Environment"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openenv.core import GenericEnvClient\n",
+    "\n",
+    "with GenericEnvClient(base_url=ENV_URL).sync() as _check:\n",
+    "    res = _check.reset()\n",
+    "    obs = res.observation if hasattr(res, 'observation') else res\n",
+    "    brief = getattr(obs, 'task_brief', '') or obs.get('task_brief', '')\n",
+    "    print(f'Task brief: {brief[:120]}...')\n",
+    "    res2 = _check.step({'action_type': 'meta.read_runbook', 'args': {}})\n",
+    "    print('Step OK — environment is working.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Load Model with Unsloth\n",
+    "\n",
+    "Unsloth applies:\n",
+    "- **4-bit quantisation** — halves GPU memory vs bf16\n",
+    "- **LoRA adapters** — only trains ~1% of parameters, much faster\n",
+    "- **Unsloth kernel optimisations** — 2× faster rollout generation\n",
+    "- **`use_gradient_checkpointing=\"unsloth\"`** — 30% less activation memory\n",
+    "\n",
+    "> **Save warning (hackathon guide point 16):** never merge LoRA into a 4-bit model directly.\n",
+    "> Use `model.save_pretrained_merged(..., save_method=\"merged_16bit\")` in cell 14."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from unsloth import FastLanguageModel, PatchFastRL\n",
+    "from trl import GRPOTrainer  # import before patching so patch takes effect\n",
+    "\n",
+    "PatchFastRL('GRPO', FastLanguageModel)\n",
+    "\n",
+    "MODEL_NAME  = 'Qwen/Qwen3-1.7B'\n",
+    "MAX_SEQ_LEN = 4096 + 512   # max_prompt_length + max_completion_length\n",
+    "LORA_RANK   = 16\n",
+    "\n",
+    "model, tokenizer = FastLanguageModel.from_pretrained(\n",
+    "    model_name=MODEL_NAME,\n",
+    "    max_seq_length=MAX_SEQ_LEN,\n",
+    "    load_in_4bit=True,\n",
+    "    fast_inference=True,          # enables Unsloth's vLLM-compatible fast path\n",
+    "    max_lora_rank=LORA_RANK,\n",
+    "    gpu_memory_utilization=0.6,   # ~24 GB on A100 40GB for KV cache\n",
+    ")\n",
+    "\n",
+    "model = FastLanguageModel.get_peft_model(\n",
+    "    model,\n",
+    "    r=LORA_RANK,\n",
+    "    target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj',\n",
+    "                    'gate_proj', 'up_proj', 'down_proj'],\n",
+    "    lora_alpha=LORA_RANK,\n",
+    "    use_gradient_checkpointing='unsloth',\n",
+    "    random_state=42,\n",
+    ")\n",
+    "tokenizer.pad_token = tokenizer.eos_token\n",
+    "print(f'Model ready: {MODEL_NAME} (4-bit + LoRA r={LORA_RANK})')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Generate Training Dataset\n",
+    "\n",
+    "150 fixed-seed triage episodes. Seed is embedded in the prompt string so `env.reset(seed=...)` \n",
+    "reproduces the exact same org config — training is fully reproducible."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datasets import Dataset\n",
+    "from training.dataset import generate_triage_dataset\n",
+    "\n",
+    "N_EPISODES = 150\n",
+    "\n",
+    "rows = generate_triage_dataset(n_episodes=N_EPISODES, base_seed=42)\n",
+    "dataset = Dataset.from_list([{'prompt': r['prompt']} for r in rows])\n",
+    "print(f'Dataset: {len(dataset)} triage episodes')\n",
+    "print(f'Difficulties: {set(r[\"difficulty\"] for r in rows)}')\n",
+    "print(f'Sample: {dataset[0][\"prompt\"][:120]}...')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Create Persistent Environment Client"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openenv.core import GenericEnvClient\n",
+    "\n",
+    "sync_env = GenericEnvClient(base_url=ENV_URL).sync()\n",
+    "sync_env.connect()\n",
+    "print('Persistent training connection established.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8. Build Rollout Function\n",
+    "\n",
+    "- `max_steps=15` — triage can be solved in 5 steps; 15 gives exploration room without burning compute  \n",
+    "- Runbook-pinned truncation — org config stays in context even when older turns scroll out  \n",
+    "- Step-aware fallback — `read_runbook` (early) → `noop` (mid) → `finish` (late)  \n",
+    "- Channel tracking — feeds `reward_no_wrong_channels` to catch spray behaviour"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from training.rollout import make_rollout_func\n",
+    "\n",
+    "TRAIN_MAX_STEPS = 15  # triage; increase for other tasks\n",
+    "\n",
+    "rollout_func = make_rollout_func(\n",
+    "    sync_env=sync_env,\n",
+    "    tokenizer=tokenizer,\n",
+    "    max_steps=TRAIN_MAX_STEPS,\n",
+    ")\n",
+    "print(f'Rollout ready (max_steps={TRAIN_MAX_STEPS})')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 9. Define Reward Functions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from training.rewards import ALL_REWARD_FUNCS, WEIGHT_FINAL_SCORE, WEIGHT_NO_WRONG_CHANNELS\n",
+    "print(f'Reward functions ({len(ALL_REWARD_FUNCS)}):')\n",
+    "for f in ALL_REWARD_FUNCS:\n",
+    "    print(f'  {f.__name__}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 10. Configure GRPO Training"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from trl import GRPOConfig\n",
+    "\n",
+    "OUTPUT_DIR = 'pm-ops-grpo-Qwen3-1.7B-triage'\n",
+    "HF_REPO_ID = f'your-hf-username/{OUTPUT_DIR}'  # ← update before running\n",
+    "\n",
+    "grpo_config = GRPOConfig(\n",
+    "    # Training\n",
+    "    num_train_epochs=1,\n",
+    "    learning_rate=5e-6,\n",
+    "    gradient_accumulation_steps=64,\n",
+    "    per_device_train_batch_size=1,\n",
+    "    warmup_steps=10,\n",
+    "    num_generations=2,\n",
+    "\n",
+    "    # Sequence lengths — longer than Wordle due to JSON + reasoning\n",
+    "    max_completion_length=512,\n",
+    "    max_prompt_length=4096,\n",
+    "\n",
+    "    # vLLM — Unsloth handles colocate mode via fast_inference=True above\n",
+    "    use_vllm=True,\n",
+    "\n",
+    "    # Output + logging\n",
+    "    output_dir=OUTPUT_DIR,\n",
+    "    report_to='trackio',\n",
+    "    trackio_space_id=OUTPUT_DIR,\n",
+    "    logging_steps=1,\n",
+    "    save_steps=25,\n",
+    "    gradient_checkpointing=False,  # Unsloth handles this via get_peft_model\n",
+    ")\n",
+    "\n",
+    "print(f'Output: {OUTPUT_DIR}')\n",
+    "print(f'Effective batch size: {grpo_config.per_device_train_batch_size * grpo_config.gradient_accumulation_steps}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 11. Create Trainer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from trl import GRPOTrainer\n",
+    "\n",
+    "trainer = GRPOTrainer(\n",
+    "    model=model,              # Unsloth model object (not string)\n",
+    "    processing_class=tokenizer,\n",
+    "    reward_funcs=ALL_REWARD_FUNCS,\n",
+    "    train_dataset=dataset,\n",
+    "    args=grpo_config,\n",
+    "    rollout_func=rollout_func,\n",
+    ")\n",
+    "print('GRPOTrainer ready.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 12. GPU Check"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "gpu = torch.cuda.get_device_properties(0)\n",
+    "reserved_before = round(torch.cuda.max_memory_reserved() / 1024**3, 2)\n",
+    "total_gb = round(gpu.total_memory / 1024**3, 2)\n",
+    "print(f'GPU: {gpu.name}')\n",
+    "print(f'Memory: {total_gb} GB total, {reserved_before} GB reserved')\n",
+    "assert total_gb >= 38, f'Need A100 40GB, got {total_gb:.1f} GB — switch runtime.'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 13. Train (~90 min on A100)\n",
+    "\n",
+    "Watch **trackio** for per-reward-signal curves. Key signals to monitor:\n",
+    "- `reward_final_score` — should trend upward over training\n",
+    "- `reward_valid_json` — should stay > 0.7 (model reliably outputs JSON)\n",
+    "- `reward_read_runbook` — should hit 1.0 quickly and stay there\n",
+    "- `reward_no_wrong_channels` — should increase (less channel spray)\n",
+    "\n",
+    "If `reward_valid_json` < 0.3 for first 20 steps → model struggling with format. Stop and reduce `max_completion_length` to 256."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer_stats = trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "used_gb = round(torch.cuda.max_memory_reserved() / 1024**3, 2)\n",
+    "train_mins = round(trainer_stats.metrics.get('train_runtime', 0) / 60, 1)\n",
+    "print(f'Training time : {train_mins} min')\n",
+    "print(f'Peak GPU usage: {used_gb} GB / {total_gb} GB ({round(used_gb/total_gb*100, 1)}%)')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 14. Save Model\n",
+    "\n",
+    "**Important:** Unsloth uses LoRA + 4-bit quantisation. Do NOT use `trainer.save_model()` directly —\n",
+    "merging LoRA into a 4-bit base corrupts weights (hackathon guide point 16).\n",
+    "Use `save_pretrained_merged` which dequantises first, then merges cleanly into bf16."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sync_env.close()  # close training connection before saving\n",
+    "\n",
+    "# Merge LoRA into bf16 weights and save — safe Unsloth path\n",
+    "model.save_pretrained_merged(OUTPUT_DIR, tokenizer, save_method='merged_16bit')\n",
+    "print(f'Model saved → {OUTPUT_DIR}')\n",
+    "\n",
+    "# Push merged model to HF Hub\n",
+    "model.push_to_hub_merged(HF_REPO_ID, tokenizer, save_method='merged_16bit')\n",
+    "print(f'Pushed → {HF_REPO_ID}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 15. Evaluate: Baseline vs Trained\n",
+    "\n",
+    "10 fresh triage episodes on the remote HF Space (not localhost) — clean eval setup."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoModelForCausalLM\n",
+    "from openenv.core import GenericEnvClient\n",
+    "from training.rollout import extract_json_action, step_aware_fallback, build_messages, _obs_to_dict\n",
+    "from training.prompts import format_observation\n",
+    "from inference import baseline_agent\n",
+    "\n",
+    "EVAL_URL = 'https://adityaguntur-pm-ops.hf.space'\n",
+    "N_EVAL = 10\n",
+    "EVAL_MAX_STEPS = 15\n",
+    "\n",
+    "eval_model = AutoModelForCausalLM.from_pretrained(\n",
+    "    OUTPUT_DIR, torch_dtype='auto', device_map='auto'\n",
+    ")\n",
+    "\n",
+    "\n",
+    "def _get(o, k, default=None):\n",
+    "    return getattr(o, k, default) if not isinstance(o, dict) else o.get(k, default)\n",
+    "\n",
+    "\n",
+    "def eval_trained(sync_env, model, tokenizer, n=N_EVAL):\n",
+    "    scores = []\n",
+    "    for i in range(n):\n",
+    "        result = sync_env.reset()\n",
+    "        obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "        task_brief = obs_dict.get('task_brief', '')\n",
+    "        history, step, score = [], 0, 0.0\n",
+    "        done = False\n",
+    "\n",
+    "        while not done and step < EVAL_MAX_STEPS:\n",
+    "            from training.rollout import _current_obs_text\n",
+    "            obs_text = _current_obs_text(obs_dict, step, task_brief)\n",
+    "            msgs = build_messages(history, obs_text)\n",
+    "            prompt_text = tokenizer.apply_chat_template(\n",
+    "                msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n",
+    "            )\n",
+    "            inputs = tokenizer([prompt_text], return_tensors='pt').to(model.device)\n",
+    "            out_ids = model.generate(**inputs, max_new_tokens=512)\n",
+    "            completion = tokenizer.decode(out_ids[0][len(inputs.input_ids[0]):], skip_special_tokens=True)\n",
+    "\n",
+    "            parsed = extract_json_action(completion) or step_aware_fallback(step)\n",
+    "            result = sync_env.step({'action_type': parsed['action_type'], 'args': parsed.get('args', {})})\n",
+    "            obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "            done = bool(getattr(result, 'done', obs_dict.get('done', False)))\n",
+    "            score = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n",
+    "            history.append({'obs_text': obs_text, 'completion': completion, 'is_runbook': False})\n",
+    "            step += 1\n",
+    "\n",
+    "        scores.append(score)\n",
+    "        print(f'  Trained  ep {i+1}/{n}: score={score:.3f}')\n",
+    "    return scores\n",
+    "\n",
+    "\n",
+    "def eval_baseline(sync_env, n=N_EVAL):\n",
+    "    scores = []\n",
+    "    for i in range(n):\n",
+    "        result = sync_env.reset()\n",
+    "        obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "        org_config, done, step, score = {}, False, 0, 0.0\n",
+    "        while not done and step < EVAL_MAX_STEPS:\n",
+    "            at, args = baseline_agent(obs_dict, org_config)\n",
+    "            result = sync_env.step({'action_type': at, 'args': args})\n",
+    "            obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "            done = bool(getattr(result, 'done', obs_dict.get('done', False)))\n",
+    "            score = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n",
+    "            step += 1\n",
+    "        scores.append(score)\n",
+    "        print(f'  Baseline ep {i+1}/{n}: score={score:.3f}')\n",
+    "    return scores\n",
+    "\n",
+    "\n",
+    "print('--- Baseline ---')\n",
+    "with GenericEnvClient(base_url=EVAL_URL).sync() as env_eval:\n",
+    "    baseline_scores = eval_baseline(env_eval)\n",
+    "\n",
+    "print('--- Trained ---')\n",
+    "with GenericEnvClient(base_url=EVAL_URL).sync() as env_eval:\n",
+    "    trained_scores = eval_trained(env_eval, eval_model, tokenizer)\n",
+    "\n",
+    "print(f'\\nBaseline avg: {sum(baseline_scores)/N_EVAL:.3f}')\n",
+    "print(f'Trained  avg: {sum(trained_scores)/N_EVAL:.3f}')\n",
+    "print(f'Improvement : +{(sum(trained_scores)-sum(baseline_scores))/N_EVAL:.3f}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 16. Plot Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n",
+    "\n",
+    "ax = axes[0]\n",
+    "x, w = np.arange(N_EVAL), 0.35\n",
+    "ax.bar(x - w/2, baseline_scores, w, label='Baseline (heuristic)', color='steelblue', alpha=0.8)\n",
+    "ax.bar(x + w/2, trained_scores, w, label='Trained (GRPO+Unsloth)', color='coral', alpha=0.8)\n",
+    "ax.axhline(sum(baseline_scores)/N_EVAL, color='steelblue', linestyle='--', alpha=0.5, linewidth=1)\n",
+    "ax.axhline(sum(trained_scores)/N_EVAL, color='coral', linestyle='--', alpha=0.5, linewidth=1)\n",
+    "ax.set_xlabel('Eval episode')\n",
+    "ax.set_ylabel('Episode reward (0–1)')\n",
+    "ax.set_title('Per-episode: Baseline vs GRPO-trained')\n",
+    "ax.set_xticks(x)\n",
+    "ax.set_ylim(0, 1.05)\n",
+    "ax.legend()\n",
+    "\n",
+    "ax2 = axes[1]\n",
+    "avgs = [sum(baseline_scores)/N_EVAL, sum(trained_scores)/N_EVAL]\n",
+    "bars = ax2.bar(['Baseline', 'GRPO+Unsloth'], avgs, color=['steelblue', 'coral'], alpha=0.85, width=0.5)\n",
+    "for bar, val in zip(bars, avgs):\n",
+    "    ax2.text(bar.get_x() + bar.get_width()/2, val + 0.01, f'{val:.3f}',\n",
+    "             ha='center', fontsize=13, fontweight='bold')\n",
+    "ax2.set_ylabel('Average reward (0–1)')\n",
+    "ax2.set_title(f'Average over {N_EVAL} triage episodes')\n",
+    "ax2.set_ylim(0, 1.05)\n",
+    "\n",
+    "plt.tight_layout()\n",
+    "plt.savefig('eval_results.png', dpi=150, bbox_inches='tight')\n",
+    "plt.show()\n",
+    "print('Saved: eval_results.png  ← embed this in README for hackathon submission')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 17. Teardown"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_proc.terminate()\n",
+    "print('Local PM-Ops server stopped.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## What to try next\n",
+    "\n",
+    "| Change | Where | Expected effect |\n",
+    "|---|---|---|\n",
+    "| All 4 task types | `generate_triage_dataset` → all-tasks generator | Tests generalization |\n",
+    "| 500 episodes | `N_EPISODES = 500` | Better coverage |\n",
+    "| Equal weights (0.20 each) | `WEIGHT_*` in `rewards.py` | Compare reward-shaping approaches |\n",
+    "| Longer training | `num_train_epochs=3` | More improvement |\n",
+    "| Larger model | `Qwen/Qwen3-4B` | Better reasoning, more memory |\n",
+    "| Higher LoRA rank | `r=64, lora_alpha=64` | More capacity |\n",
+    "| More steps for other tasks | `TRAIN_MAX_STEPS = 20` | Needed for release_notes/dep_update |"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.11.0"
+  },
+  "accelerator": "GPU",
+  "gpuClass": "premium"
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

training/train_v2.ipynb

@@ -0,0 +1,715 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# PM-Ops GRPO Training v2\n",
+    "\n",
+    "### Fixes vs v1\n",
+    "| Bug | Root cause | Fix |\n",
+    "|-----|-----------|-----|\n",
+    "| All rewards = 0 | `rollout_func` kwargs broken in TRL 1.3.0.dev0 | TRL **1.2.0** stable + single `reward` key |\n",
+    "| Dep hell | Unsloth 2026.4.8 requires torch<2.11 / trl≤0.24, conflicts with Colab's base image | **Drop Unsloth**, use bitsandbytes + PEFT directly |\n",
+    "| Only 4 steps | T4 + grad_accum=64 | GPU auto-detect, adaptive config |\n",
+    "| Silent reward failure | No logging | Rollout prints every episode, reward_func warns on zero |\n",
+    "\n",
+    "**Stack:** TRL 1.2.0 · bitsandbytes (4-bit) · PEFT LoRA · OpenEnv  \n",
+    "**GPU:** A100 → ~150 steps / 60 min · T4 → ~18 steps (smoke-test / short run)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 0. Install Dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Keep Colab's native torch (2.11.0) — don't touch it.\n",
+    "# Install everything that is compatible with torch 2.11.0.\n",
+    "\n",
+    "!pip install -q \"trl==1.2.0\"\n",
+    "!pip install -q \"bitsandbytes>=0.44.0\"          # 4-bit quantisation\n",
+    "!pip install -q \"peft>=0.13.0\"                  # LoRA\n",
+    "!pip install -q \"accelerate>=1.0.0\"\n",
+    "!pip install -q \"datasets>=4.7.0\"\n",
+    "!pip install -q \"transformers>=5.0.0\"\n",
+    "!pip install -q \"openenv-core>=0.2.2\" fastapi \"uvicorn[standard]\" \"pydantic>=2.0.0\"\n",
+    "!pip install -q trackio\n",
+    "\n",
+    "print('Done — restart kernel, then run all cells from top.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Version Check + GPU Detect"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch, trl\n",
+    "import trl.experimental.openenv  # must exist for rollout_func\n",
+    "\n",
+    "print(f'torch : {torch.__version__}')\n",
+    "print(f'TRL   : {trl.__version__}  (need 1.2.x)')\n",
+    "assert trl.__version__.startswith('1.2'), \\\n",
+    "    f'Wrong TRL: {trl.__version__}. Re-run install cell and restart kernel.'\n",
+    "print('trl.experimental.openenv : OK')\n",
+    "\n",
+    "gpu      = torch.cuda.get_device_properties(0)\n",
+    "TOTAL_GB = round(gpu.total_memory / 1024**3, 1)\n",
+    "IS_A100  = TOTAL_GB >= 35\n",
+    "print(f'\\nGPU  : {gpu.name}  ({TOTAL_GB} GB)')\n",
+    "print(f'Mode : {\"A100 — full training\" if IS_A100 else \"T4 — reduced config (smoke-test quality)\"}')\n",
+    "\n",
+    "# Adaptive config\n",
+    "GRAD_ACCUM   = 32  if IS_A100 else 8\n",
+    "NUM_GEN      = 4   if IS_A100 else 2\n",
+    "MAX_COMP_LEN = 512 if IS_A100 else 256\n",
+    "\n",
+    "print(f'\\ngrad_accum={GRAD_ACCUM}  num_gen={NUM_GEN}  max_comp={MAX_COMP_LEN}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Clone PM-Ops Repo"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os, sys\n",
+    "\n",
+    "REPO_URL = 'https://huggingface.co/spaces/TheCrustaceans/Pm-ops'\n",
+    "REPO_DIR = '/content/Pm_ops'\n",
+    "\n",
+    "if not os.path.exists(REPO_DIR):\n",
+    "    !git clone --depth=1 -q {REPO_URL} {REPO_DIR}\n",
+    "    print(f'Cloned → {REPO_DIR}')\n",
+    "else:\n",
+    "    !git -C {REPO_DIR} pull -q origin main\n",
+    "    print(f'Pulled latest → {REPO_DIR}')\n",
+    "\n",
+    "for p in [REPO_DIR, os.path.join(REPO_DIR, 'training')]:\n",
+    "    if p not in sys.path:\n",
+    "        sys.path.insert(0, p)\n",
+    "os.chdir(REPO_DIR)\n",
+    "print(f'CWD: {os.getcwd()}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. HuggingFace Login"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import notebook_login\n",
+    "notebook_login()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Start Local PM-Ops Server\n",
+    "\n",
+    "Localhost removes ~200 ms/step network round-trip vs calling the HF Space."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess, time, requests\n",
+    "\n",
+    "server_proc = subprocess.Popen(\n",
+    "    [sys.executable, '-m', 'uvicorn', 'server.app:app', '--host', '0.0.0.0', '--port', '8000'],\n",
+    "    cwd=REPO_DIR, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,\n",
+    ")\n",
+    "ENV_URL = 'http://localhost:8000'\n",
+    "\n",
+    "for _ in range(30):\n",
+    "    try:\n",
+    "        if requests.get(f'{ENV_URL}/', timeout=2).status_code == 200:\n",
+    "            print(f'PM-Ops server ready  pid={server_proc.pid}')\n",
+    "            break\n",
+    "    except Exception:\n",
+    "        pass\n",
+    "    time.sleep(1)\n",
+    "else:\n",
+    "    raise RuntimeError('Server did not start in 30 s')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Verify Environment"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openenv.core import GenericEnvClient\n",
+    "from training.rollout import _obs_to_dict\n",
+    "\n",
+    "with GenericEnvClient(base_url=ENV_URL).sync() as _env:\n",
+    "    r = _env.reset()\n",
+    "    obs = _obs_to_dict(r.observation if hasattr(r, 'observation') else r)\n",
+    "    print(f'task_brief : {obs.get(\"task_brief\", \"?\")[:80]}...')\n",
+    "    _env.step({'action_type': 'meta.read_runbook', 'args': {}})\n",
+    "    print('Env step   : OK')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Load Model — bitsandbytes 4-bit + PEFT LoRA\n",
+    "\n",
+    "No Unsloth — uses stock HuggingFace `AutoModelForCausalLM` with `BitsAndBytesConfig`.  \n",
+    "Same 4-bit + LoRA memory profile, ~2× slower generation than Unsloth's kernels."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
+    "from peft import LoraConfig, get_peft_model, TaskType\n",
+    "\n",
+    "MODEL_NAME = 'Qwen/Qwen3-1.7B'\n",
+    "LORA_RANK  = 16\n",
+    "\n",
+    "bnb_cfg = BitsAndBytesConfig(\n",
+    "    load_in_4bit=True,\n",
+    "    bnb_4bit_compute_dtype=torch.bfloat16,\n",
+    "    bnb_4bit_use_double_quant=True,\n",
+    "    bnb_4bit_quant_type='nf4',\n",
+    ")\n",
+    "\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    MODEL_NAME,\n",
+    "    quantization_config=bnb_cfg,\n",
+    "    device_map='auto',\n",
+    "    torch_dtype=torch.bfloat16,\n",
+    "    attn_implementation='eager',  # safe default; use 'flash_attention_2' if flash-attn installed\n",
+    ")\n",
+    "\n",
+    "tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)\n",
+    "tokenizer.pad_token = tokenizer.eos_token\n",
+    "tokenizer.padding_side = 'left'\n",
+    "\n",
+    "lora_cfg = LoraConfig(\n",
+    "    r=LORA_RANK,\n",
+    "    lora_alpha=LORA_RANK,\n",
+    "    target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj'],\n",
+    "    lora_dropout=0.0,\n",
+    "    bias='none',\n",
+    "    task_type=TaskType.CAUSAL_LM,\n",
+    ")\n",
+    "model = get_peft_model(model, lora_cfg)\n",
+    "model.print_trainable_parameters()\n",
+    "\n",
+    "reserved = round(torch.cuda.max_memory_reserved() / 1024**3, 2)\n",
+    "print(f'GPU memory after load: {reserved} GB / {TOTAL_GB} GB')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Generate Training Dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from datasets import Dataset\n",
+    "from training.dataset import generate_triage_dataset\n",
+    "\n",
+    "N_EPISODES = 150\n",
+    "rows    = generate_triage_dataset(n_episodes=N_EPISODES, base_seed=42)\n",
+    "dataset = Dataset.from_list([{'prompt': r['prompt']} for r in rows])\n",
+    "print(f'Dataset : {len(dataset)} triage episodes')\n",
+    "print(f'Sample  : {dataset[0][\"prompt\"][:100]}...')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8. Create Persistent Env Client + Rollout Function\n",
+    "\n",
+    "**Key change from v1:** single `\"reward\"` key in the rollout output.  \n",
+    "This removes all kwargs-plumbing complexity — the reward_func is a trivial passthrough."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openenv.core import GenericEnvClient\n",
+    "from training.rollout import rollout_once\n",
+    "\n",
+    "sync_env = GenericEnvClient(base_url=ENV_URL).sync()\n",
+    "sync_env.connect()\n",
+    "print('Training env connected')\n",
+    "\n",
+    "TRAIN_MAX_STEPS = 15\n",
+    "\n",
+    "\n",
+    "def make_rollout_func(env, tok, max_steps=TRAIN_MAX_STEPS):\n",
+    "    def rollout_func(prompts, trainer=None):\n",
+    "        out = {'prompt_ids': [], 'completion_ids': [], 'logprobs': [], 'reward': []}\n",
+    "        for prompt in prompts:\n",
+    "            ep = rollout_once(\n",
+    "                trainer=trainer, sync_env=env, tokenizer=tok,\n",
+    "                dataset_prompt=prompt, max_steps=max_steps,\n",
+    "            )\n",
+    "            combined = (\n",
+    "                ep['final_score_reward']       * 0.45 +\n",
+    "                ep['no_wrong_channels_reward'] * 0.15 +\n",
+    "                ep['valid_json_reward']        * 0.15 +\n",
+    "                ep['read_runbook_reward']      * 0.15 +\n",
+    "                ep['efficiency_reward']        * 0.10\n",
+    "            )\n",
+    "            out['prompt_ids'].append(ep['prompt_ids'])\n",
+    "            out['completion_ids'].append(ep['completion_ids'])\n",
+    "            out['logprobs'].append(ep['logprobs'])\n",
+    "            out['reward'].append(combined)\n",
+    "        return out\n",
+    "    return rollout_func\n",
+    "\n",
+    "\n",
+    "rollout_func = make_rollout_func(sync_env, tokenizer)\n",
+    "\n",
+    "\n",
+    "def reward_func(completions, **kwargs):\n",
+    "    \"\"\"Trivial passthrough — reads pre-computed reward from rollout kwargs.\"\"\"\n",
+    "    n       = len(completions)\n",
+    "    rewards = kwargs.get('reward', [])\n",
+    "    if not rewards:\n",
+    "        print(\n",
+    "            f'[ERROR] reward_func: empty kwargs! '\n",
+    "            f'TRL {trl.__version__} is not passing rollout keys. '\n",
+    "            f'kwargs keys present: {list(kwargs.keys())}'\n",
+    "        )\n",
+    "        return [0.0] * n\n",
+    "    return [float(r) for r in rewards]\n",
+    "\n",
+    "\n",
+    "print('rollout_func + reward_func defined')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 9. Smoke Test\n",
+    "\n",
+    "Verify the env + reward pipeline works *before* committing to a 60-minute training run."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print('─── Smoke test ──────────────────────────────────')\n",
+    "\n",
+    "# 1. Env resets + steps correctly\n",
+    "with GenericEnvClient(base_url=ENV_URL).sync() as _t:\n",
+    "    r1 = _t.reset(seed=42)\n",
+    "    obs = _obs_to_dict(r1.observation if hasattr(r1, 'observation') else r1)\n",
+    "    assert obs.get('task_brief'), 'FAIL: no task_brief in observation'\n",
+    "    print(f'[OK] env.reset()       task_brief present')\n",
+    "\n",
+    "    r2 = _t.step({'action_type': 'meta.read_runbook', 'args': {}})\n",
+    "    print(f'[OK] env.step(read_runbook)')\n",
+    "\n",
+    "    r3 = _t.step({'action_type': 'meta.finish', 'args': {}})\n",
+    "    obs3 = _obs_to_dict(r3.observation if hasattr(r3, 'observation') else r3)\n",
+    "    final = float(getattr(r3, 'reward', obs3.get('reward', -1)))\n",
+    "    print(f'[OK] env.step(finish)  reward={final:.3f}')\n",
+    "\n",
+    "# 2. reward_func passthrough\n",
+    "rf = reward_func(['c1', 'c2'], reward=[0.3, 0.7])\n",
+    "assert rf == [0.3, 0.7], f'FAIL: passthrough broken: {rf}'\n",
+    "print(f'[OK] reward_func passthrough  {rf}')\n",
+    "\n",
+    "# 3. reward_func warns correctly on empty kwargs\n",
+    "import io, warnings\n",
+    "rf_empty = reward_func(['c1'])\n",
+    "assert rf_empty == [0.0]\n",
+    "print(f'[OK] reward_func empty → [0.0]  (error printed above is expected)')\n",
+    "\n",
+    "print('\\nSmoke test PASSED')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 10. Configure GRPO Training"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from trl import GRPOConfig\n",
+    "\n",
+    "OUTPUT_DIR = 'pm-ops-grpo-Qwen3-1.7B-triage-v2'\n",
+    "HF_REPO_ID = f'Saurav1/{OUTPUT_DIR}'\n",
+    "\n",
+    "grpo_config = GRPOConfig(\n",
+    "    # Training\n",
+    "    num_train_epochs=1,\n",
+    "    learning_rate=5e-6,\n",
+    "    gradient_accumulation_steps=GRAD_ACCUM,\n",
+    "    per_device_train_batch_size=1,\n",
+    "    warmup_steps=5,\n",
+    "    num_generations=NUM_GEN,\n",
+    "\n",
+    "    # Sequence lengths\n",
+    "    max_completion_length=MAX_COMP_LEN,\n",
+    "    max_prompt_length=4096,\n",
+    "\n",
+    "    # No vLLM — standard HF generate (compatible with bitsandbytes + PEFT)\n",
+    "    use_vllm=False,\n",
+    "\n",
+    "    # Output\n",
+    "    output_dir=OUTPUT_DIR,\n",
+    "    report_to='trackio',\n",
+    "    trackio_space_id=OUTPUT_DIR,\n",
+    "    logging_steps=1,\n",
+    "    save_steps=25,\n",
+    "    gradient_checkpointing=True,  # saves ~30% activation memory\n",
+    ")\n",
+    "\n",
+    "eff_batch = grpo_config.per_device_train_batch_size * GRAD_ACCUM\n",
+    "total_steps = len(dataset) * NUM_GEN // eff_batch\n",
+    "print(f'Output dir   : {OUTPUT_DIR}')\n",
+    "print(f'Eff. batch   : {eff_batch}')\n",
+    "print(f'Total steps  : ~{total_steps}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 11. Create Trainer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from training.pm_ops_trainer import PMOpsGRPOTrainer\n",
+    "\n",
+    "trainer = PMOpsGRPOTrainer(\n",
+    "    model=model,\n",
+    "    processing_class=tokenizer,\n",
+    "    reward_funcs=reward_func,   # fallback only — injected rewards take priority\n",
+    "    train_dataset=dataset,\n",
+    "    args=grpo_config,\n",
+    "    rollout_func=rollout_func,\n",
+    ")\n",
+    "print(\"PMOpsGRPOTrainer ready — direct reward injection active\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 12. Train\n",
+    "\n",
+    "**Watch Colab stdout for rollout lines:**\n",
+    "```\n",
+    "[rollout] steps=5 final=0.720 json=0.80 runbook=1 no_wrong=1.00 eff=0.67 → combined=0.674\n",
+    "```\n",
+    "- `combined > 0` in stdout **and** `train/reward > 0` in trackio → working correctly  \n",
+    "- `combined > 0` in stdout **but** `train/reward = 0` in trackio → kwargs still not flowing  \n",
+    "  → Add `print(kwargs.keys())` inside `reward_func` to debug  \n",
+    "- `combined = 0` always → check env connection / reward logic"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer_stats = trainer.train()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "used_gb    = round(torch.cuda.max_memory_reserved() / 1024**3, 2)\n",
+    "train_mins = round(trainer_stats.metrics.get('train_runtime', 0) / 60, 1)\n",
+    "print(f'Training time : {train_mins} min')\n",
+    "print(f'Peak GPU      : {used_gb} GB / {TOTAL_GB} GB ({round(used_gb/TOTAL_GB*100,1)}%)')\n",
+    "print(f'Final loss    : {trainer_stats.metrics.get(\"train_loss\", \"n/a\")}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 13. Save + Push"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "sync_env.close()\n",
+    "\n",
+    "# Save LoRA adapter (not merged — push separately if needed)\n",
+    "trainer.save_model(OUTPUT_DIR)\n",
+    "tokenizer.save_pretrained(OUTPUT_DIR)\n",
+    "print(f'Saved LoRA adapter → {OUTPUT_DIR}')\n",
+    "\n",
+    "# Push to Hub\n",
+    "trainer.model.push_to_hub(HF_REPO_ID)\n",
+    "tokenizer.push_to_hub(HF_REPO_ID)\n",
+    "print(f'Pushed → https://huggingface.co/{HF_REPO_ID}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 14. Merge LoRA → bf16 (Optional — for full-weight inference)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Only run this if you have enough CPU RAM (~7 GB free) to hold the merged model.\n",
+    "# On T4 Colab this often OOMs — skip and load with PEFT for eval instead.\n",
+    "from peft import PeftModel\n",
+    "from transformers import AutoModelForCausalLM\n",
+    "\n",
+    "base = AutoModelForCausalLM.from_pretrained(MODEL_NAME, torch_dtype=torch.bfloat16, device_map='cpu')\n",
+    "merged = PeftModel.from_pretrained(base, OUTPUT_DIR)\n",
+    "merged = merged.merge_and_unload()\n",
+    "merged.save_pretrained(f'{OUTPUT_DIR}-merged')\n",
+    "tokenizer.save_pretrained(f'{OUTPUT_DIR}-merged')\n",
+    "print(f'Merged model saved → {OUTPUT_DIR}-merged')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 15. Evaluate: Baseline vs Trained"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from training.rollout import extract_json_action, step_aware_fallback, build_messages\n",
+    "from training.rollout import _current_obs_text, _obs_to_dict\n",
+    "from inference import baseline_agent\n",
+    "\n",
+    "EVAL_URL       = 'https://adityaguntur-pm-ops.hf.space'\n",
+    "N_EVAL         = 10\n",
+    "EVAL_MAX_STEPS = 15\n",
+    "\n",
+    "# Put model in eval/inference mode\n",
+    "model.eval()\n",
+    "\n",
+    "\n",
+    "def eval_trained(env, mdl, tok, n=N_EVAL):\n",
+    "    scores = []\n",
+    "    for i in range(n):\n",
+    "        result     = env.reset()\n",
+    "        obs_dict   = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "        task_brief = obs_dict.get('task_brief', '')\n",
+    "        history, step, score, done = [], 0, 0.0, False\n",
+    "\n",
+    "        while not done and step < EVAL_MAX_STEPS:\n",
+    "            obs_text  = _current_obs_text(obs_dict, step, task_brief)\n",
+    "            msgs      = build_messages(history, obs_text)\n",
+    "            prompt_t  = tok.apply_chat_template(\n",
+    "                msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n",
+    "            )\n",
+    "            inputs = tok([prompt_t], return_tensors='pt', truncation=True, max_length=4096)\n",
+    "            inputs = {k: v.to(mdl.device) for k, v in inputs.items()}\n",
+    "            with torch.no_grad():\n",
+    "                out_ids    = mdl.generate(**inputs, max_new_tokens=512,\n",
+    "                                          do_sample=False, pad_token_id=tok.eos_token_id)\n",
+    "            completion = tok.decode(out_ids[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)\n",
+    "\n",
+    "            parsed   = extract_json_action(completion) or step_aware_fallback(step)\n",
+    "            result   = env.step({'action_type': parsed['action_type'], 'args': parsed.get('args', {})})\n",
+    "            obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "            done     = bool(getattr(result, 'done', obs_dict.get('done', False)))\n",
+    "            score    = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n",
+    "            history.append({'obs_text': obs_text, 'completion': completion, 'is_runbook': False})\n",
+    "            step += 1\n",
+    "\n",
+    "        scores.append(score)\n",
+    "        print(f'  Trained  ep {i+1}/{n}: score={score:.3f}')\n",
+    "    return scores\n",
+    "\n",
+    "\n",
+    "def eval_baseline(env, n=N_EVAL):\n",
+    "    scores = []\n",
+    "    for i in range(n):\n",
+    "        result   = env.reset()\n",
+    "        obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "        org_cfg, done, step, score = {}, False, 0, 0.0\n",
+    "        while not done and step < EVAL_MAX_STEPS:\n",
+    "            at, args = baseline_agent(obs_dict, org_cfg)\n",
+    "            result   = env.step({'action_type': at, 'args': args})\n",
+    "            obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "            done  = bool(getattr(result, 'done', obs_dict.get('done', False)))\n",
+    "            score = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n",
+    "            step += 1\n",
+    "        scores.append(score)\n",
+    "        print(f'  Baseline ep {i+1}/{n}: score={score:.3f}')\n",
+    "    return scores\n",
+    "\n",
+    "\n",
+    "print('─── Baseline ───')\n",
+    "with GenericEnvClient(base_url=EVAL_URL).sync() as env_eval:\n",
+    "    baseline_scores = eval_baseline(env_eval)\n",
+    "\n",
+    "print('─── Trained ────')\n",
+    "with GenericEnvClient(base_url=EVAL_URL).sync() as env_eval:\n",
+    "    trained_scores = eval_trained(env_eval, model, tokenizer)\n",
+    "\n",
+    "print(f'\\nBaseline avg : {sum(baseline_scores)/N_EVAL:.3f}')\n",
+    "print(f'Trained  avg : {sum(trained_scores)/N_EVAL:.3f}')\n",
+    "print(f'Improvement  : +{(sum(trained_scores)-sum(baseline_scores))/N_EVAL:.3f}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 16. Plot"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n",
+    "\n",
+    "ax, x, w = axes[0], np.arange(N_EVAL), 0.35\n",
+    "ax.bar(x - w/2, baseline_scores, w, label='Baseline', color='steelblue', alpha=0.8)\n",
+    "ax.bar(x + w/2, trained_scores,  w, label='GRPO v2',  color='coral',     alpha=0.8)\n",
+    "ax.axhline(sum(baseline_scores)/N_EVAL, color='steelblue', ls='--', alpha=0.5, lw=1)\n",
+    "ax.axhline(sum(trained_scores)/N_EVAL,  color='coral',     ls='--', alpha=0.5, lw=1)\n",
+    "ax.set(xlabel='Episode', ylabel='Reward', title='Per-episode: Baseline vs GRPO v2',\n",
+    "       xticks=x, ylim=(0, 1.05))\n",
+    "ax.legend()\n",
+    "\n",
+    "ax2  = axes[1]\n",
+    "avgs = [sum(baseline_scores)/N_EVAL, sum(trained_scores)/N_EVAL]\n",
+    "bars = ax2.bar(['Baseline', 'GRPO v2'], avgs, color=['steelblue', 'coral'], alpha=0.85, width=0.5)\n",
+    "for bar, val in zip(bars, avgs):\n",
+    "    ax2.text(bar.get_x() + bar.get_width()/2, val + 0.01, f'{val:.3f}',\n",
+    "             ha='center', fontsize=13, fontweight='bold')\n",
+    "ax2.set(ylabel='Average reward', title=f'Avg over {N_EVAL} episodes', ylim=(0, 1.05))\n",
+    "\n",
+    "plt.tight_layout()\n",
+    "plt.savefig('eval_results_v2.png', dpi=150, bbox_inches='tight')\n",
+    "plt.show()\n",
+    "print('Saved: eval_results_v2.png')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 17. Teardown"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_proc.terminate()\n",
+    "print('Local PM-Ops server stopped')"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

training/train_v3.ipynb

@@ -0,0 +1,814 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "cell-0",
+   "metadata": {},
+   "source": [
+    "# PM-Ops GRPO Training v3 — SFT Warmup + GRPO\n",
+    "\n",
+    "## What changed from v2\n",
+    "| Problem in v2 | Fix in v3 |\n",
+    "|---|---|\n",
+    "| `json=0.00` — model never outputs JSON, GRPO gradient = 0 | **SFT warmup on 60 baseline traces first** |\n",
+    "| Reward = 0.300 every episode (no variance) | **env_score varies 0.25–1.0 after SFT** |\n",
+    "| Slow — no vLLM, eager attention | **Unsloth + `use_vllm=True`** |\n",
+    "| 5 reward signals, all from fallback | **2 signals: env_score × 0.85 + json_ratio × 0.15** |\n",
+    "| `meta.finish` never called in 15-step cap | **Fixed fallback respects `max_steps`** |\n",
+    "\n",
+    "## Why SFT before GRPO?\n",
+    "GRPO learns by comparing rewards across a *group* of generations. If all generations get the\n",
+    "same reward (because the model outputs garbage on every step), the advantage is 0 and weights\n",
+    "don't move. SFT warmup costs ~15 min and unlocks the full GRPO gradient.\n",
+    "\n",
+    "**Stack**: Unsloth + TRL 1.2.0 + OpenEnv · **GPU**: A100 → ~75 min total"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-1-md",
+   "metadata": {},
+   "source": [
+    "## 0. Install"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Unsloth + vLLM first — let Unsloth resolve torch compat\n",
+    "!pip install -q unsloth vllm\n",
+    "# TRL training stack\n",
+    "!pip install -q \"trl==1.2.0\" accelerate datasets\n",
+    "# PM-Ops server runtime\n",
+    "!pip install -q \"openenv-core>=0.2.2\" \"fastapi>=0.110.0\" \"uvicorn[standard]>=0.29.0\" \"pydantic>=2.0.0\"\n",
+    "# Experiment tracking\n",
+    "!pip install -q trackio\n",
+    "print('Done - restart kernel, then run all cells from top.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-2-md",
+   "metadata": {},
+   "source": [
+    "## 1. Imports + GPU Config"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "\n",
+    "# PatchFastRL can bypass custom rollout_func on some cloud runtimes.\n",
+    "# Keep it disabled for PMOpsGRPOTrainer rollout compatibility.\n",
+    "from unsloth import FastLanguageModel, PatchFastRL\n",
+    "ENABLE_FAST_RL_PATCH = False\n",
+    "if ENABLE_FAST_RL_PATCH:\n",
+    "    PatchFastRL('GRPO', FastLanguageModel)\n",
+    "    print('PatchFastRL enabled')\n",
+    "else:\n",
+    "    print('PatchFastRL disabled for rollout_func compatibility')\n",
+    "\n",
+    "import trl\n",
+    "print(f'torch : {torch.__version__}')\n",
+    "print(f'TRL   : {trl.__version__}')\n",
+    "\n",
+    "gpu      = torch.cuda.get_device_properties(0)\n",
+    "TOTAL_GB = round(gpu.total_memory / 1024**3, 1)\n",
+    "IS_A100  = TOTAL_GB >= 35\n",
+    "print(f'GPU   : {gpu.name} ({TOTAL_GB} GB)')\n",
+    "\n",
+    "# Adaptive config — T4 uses minimal settings for smoke-testing\n",
+    "NUM_GEN         = 6  if IS_A100 else 2\n",
+    "GRAD_ACCUM      = 32 if IS_A100 else 8\n",
+    "MAX_COMP_LEN    = 384\n",
+    "N_SFT_EPISODES  = 60 if IS_A100 else 15\n",
+    "N_GRPO_EPISODES = 150 if IS_A100 else 30\n",
+    "TRAIN_MAX_STEPS = 12   # triage solvable in 5; 12 gives exploration room\n",
+    "\n",
+    "print(f'num_gen={NUM_GEN}  grad_accum={GRAD_ACCUM}  '\n",
+    "      f'sft_eps={N_SFT_EPISODES}  grpo_eps={N_GRPO_EPISODES}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-3-md",
+   "metadata": {},
+   "source": [
+    "## 2. Clone PM-Ops Repo"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os, sys\n",
+    "\n",
+    "REPO_URL = 'https://huggingface.co/spaces/TheCrustaceans/Pm-ops'\n",
+    "REPO_DIR = '/content/Pm_ops'\n",
+    "\n",
+    "if not os.path.exists(REPO_DIR):\n",
+    "    !git clone --depth=1 -q {REPO_URL} {REPO_DIR}\n",
+    "    print(f'Cloned -> {REPO_DIR}')\n",
+    "else:\n",
+    "    !git -C {REPO_DIR} pull -q origin main\n",
+    "    print(f'Pulled -> {REPO_DIR}')\n",
+    "\n",
+    "for p in [REPO_DIR, os.path.join(REPO_DIR, 'training')]:\n",
+    "    if p not in sys.path:\n",
+    "        sys.path.insert(0, p)\n",
+    "os.chdir(REPO_DIR)\n",
+    "print(f'CWD: {os.getcwd()}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-4-md",
+   "metadata": {},
+   "source": [
+    "## 3. HuggingFace Login"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import notebook_login\n",
+    "notebook_login()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-5-md",
+   "metadata": {},
+   "source": [
+    "## 4. Start Local PM-Ops Server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess, time, requests\n",
+    "\n",
+    "server_proc = subprocess.Popen(\n",
+    "    [sys.executable, '-m', 'uvicorn', 'server.app:app',\n",
+    "     '--host', '0.0.0.0', '--port', '8000'],\n",
+    "    cwd=REPO_DIR, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,\n",
+    ")\n",
+    "ENV_URL = 'http://localhost:8000'\n",
+    "\n",
+    "for _ in range(30):\n",
+    "    try:\n",
+    "        if requests.get(f'{ENV_URL}/', timeout=2).status_code == 200:\n",
+    "            print(f'PM-Ops server ready  pid={server_proc.pid}')\n",
+    "            break\n",
+    "    except Exception:\n",
+    "        pass\n",
+    "    time.sleep(1)\n",
+    "else:\n",
+    "    raise RuntimeError('Server did not start in 30 s')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-6-md",
+   "metadata": {},
+   "source": [
+    "## 5. Verify Env"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-6",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import trl.experimental.openenv  # must be importable\n",
+    "from openenv.core import GenericEnvClient\n",
+    "from training.rollout import _obs_to_dict\n",
+    "\n",
+    "with GenericEnvClient(base_url=ENV_URL).sync() as _env:\n",
+    "    r   = _env.reset()\n",
+    "    obs = _obs_to_dict(r.observation if hasattr(r, 'observation') else r)\n",
+    "    print(f'task_brief : {obs.get(\"task_brief\", \"?\")[:80]}...')\n",
+    "    _env.step({'action_type': 'meta.read_runbook', 'args': {}})\n",
+    "    print('Env step   : OK')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-7-md",
+   "metadata": {},
+   "source": [
+    "## 6. Load Model — Unsloth 4-bit + LoRA"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "MODEL_NAME = 'Qwen/Qwen3-1.7B'\n",
+    "LORA_RANK  = 16\n",
+    "\n",
+    "model, tokenizer = FastLanguageModel.from_pretrained(\n",
+    "    model_name           = MODEL_NAME,\n",
+    "    max_seq_length       = 4096 + MAX_COMP_LEN,\n",
+    "    load_in_4bit         = True,\n",
+    "    fast_inference       = False,  # disable fast RL path to preserve rollout_func behaviour\n",
+    "    max_lora_rank        = LORA_RANK,\n",
+    "    gpu_memory_utilization = 0.50, # leave headroom for SFT activations\n",
+    ")\n",
+    "model = FastLanguageModel.get_peft_model(\n",
+    "    model,\n",
+    "    r                        = LORA_RANK,\n",
+    "    target_modules           = ['q_proj','k_proj','v_proj','o_proj',\n",
+    "                                 'gate_proj','up_proj','down_proj'],\n",
+    "    lora_alpha               = LORA_RANK,\n",
+    "    use_gradient_checkpointing = 'unsloth',\n",
+    "    random_state             = 42,\n",
+    " )\n",
+    "tokenizer.pad_token    = tokenizer.eos_token\n",
+    "tokenizer.padding_side = 'left'\n",
+    "model.print_trainable_parameters()\n",
+    "reserved = round(torch.cuda.max_memory_reserved() / 1024**3, 2)\n",
+    "print(f'GPU after load: {reserved} GB / {TOTAL_GB} GB')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-8-md",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Phase 1 — SFT Warmup\n",
+    "\n",
+    "**Goal**: teach the model the JSON output format and PM-ops workflow before GRPO.\n",
+    "\n",
+    "We run `baseline_agent` (the deterministic heuristic) for 60 episodes and record every\n",
+    "(observation, action) pair as a supervised example. Each episode produces ~6 steps:\n",
+    "`read_runbook` → `create_ticket` → `assign_ticket` → `list_channels` → `post_message` → `finish`.\n",
+    "\n",
+    "After 2 SFT epochs (~15 min), the model reliably outputs `\\`\\`\\`json ... \\`\\`\\`` blocks.\n",
+    "Without this, GRPO reward variance ≈ 0 and nothing is learned."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-9-md",
+   "metadata": {},
+   "source": [
+    "## 7. Generate SFT Demonstration Dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json as _json\n",
+    "from datasets import Dataset\n",
+    "from inference import baseline_agent\n",
+    "from training.rollout import _obs_to_dict, _current_obs_text, build_messages\n",
+    "\n",
+    "\n",
+    "def generate_sft_dataset(env_url, tok, n_episodes, seed_start=2000):\n",
+    "    \"\"\"Run baseline_agent for each episode; record (prompt, completion) pairs.\"\"\"\n",
+    "    examples = []\n",
+    "    with GenericEnvClient(base_url=env_url).sync() as env:\n",
+    "        for i in range(n_episodes):\n",
+    "            seed     = seed_start + i\n",
+    "            result   = env.reset(seed=seed)\n",
+    "            obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "            task_brief   = obs_dict.get('task_brief', '')\n",
+    "            turn_history = []\n",
+    "            org_config   = {}\n",
+    "            step, done   = 0, False\n",
+    "\n",
+    "            while not done and step < 8:\n",
+    "                obs_text    = _current_obs_text(obs_dict, step, task_brief)\n",
+    "                action_type, args = baseline_agent(obs_dict, org_config)\n",
+    "\n",
+    "                # Target completion: JSON code block (what we want the model to learn)\n",
+    "                payload    = {'action_type': action_type, 'args': args}\n",
+    "                completion = '```json\\n' + _json.dumps(payload) + '\\n```'\n",
+    "\n",
+    "                msgs   = build_messages(turn_history, obs_text)\n",
+    "                prompt = tok.apply_chat_template(\n",
+    "                    msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n",
+    "                )\n",
+    "                # Full SFT text = prompt + target completion + eos\n",
+    "                examples.append({'text': prompt + completion + tok.eos_token})\n",
+    "\n",
+    "                turn_history.append({\n",
+    "                    'obs_text'  : obs_text,\n",
+    "                    'completion': completion,\n",
+    "                    'is_runbook': (action_type == 'meta.read_runbook'),\n",
+    "                })\n",
+    "\n",
+    "                result   = env.step({'action_type': action_type, 'args': args})\n",
+    "                obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "\n",
+    "                # Sync org_config from runbook response\n",
+    "                if action_type == 'meta.read_runbook':\n",
+    "                    last = obs_dict.get('last_action_result') or {}\n",
+    "                    if last.get('ok'):\n",
+    "                        data = last.get('data') or {}\n",
+    "                        if isinstance(data, dict) and 'org_config' in data:\n",
+    "                            org_config.update(data['org_config'])\n",
+    "\n",
+    "                done = bool(getattr(result, 'done', obs_dict.get('done', False)))\n",
+    "                step += 1\n",
+    "\n",
+    "            if (i + 1) % 10 == 0:\n",
+    "                print(f'  {i+1}/{n_episodes} episodes — {len(examples)} examples')\n",
+    "\n",
+    "    return examples\n",
+    "\n",
+    "\n",
+    "print(f'Generating {N_SFT_EPISODES} SFT demonstration episodes...')\n",
+    "sft_raw     = generate_sft_dataset(ENV_URL, tokenizer, n_episodes=N_SFT_EPISODES)\n",
+    "sft_dataset = Dataset.from_list(sft_raw)\n",
+    "print(f'\\nSFT dataset : {len(sft_dataset)} examples  (~{len(sft_dataset)//6} eps x 6 steps)')\n",
+    "print(f'Sample (first 300 chars):\\n{sft_raw[0][\"text\"][:300]}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-10-md",
+   "metadata": {},
+   "source": [
+    "## 8. SFT Training (~15 min on A100)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-10",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from trl import SFTTrainer, SFTConfig\n",
+    "\n",
+    "sft_cfg = SFTConfig(\n",
+    "    dataset_text_field          = 'text',\n",
+    "    max_seq_length              = 2048,\n",
+    "    num_train_epochs            = 2,\n",
+    "    per_device_train_batch_size = 4,\n",
+    "    gradient_accumulation_steps = 4,\n",
+    "    learning_rate               = 2e-4,\n",
+    "    warmup_steps                = 10,\n",
+    "    output_dir                  = 'pm-ops-sft-warmup',\n",
+    "    report_to                   = 'none',\n",
+    "    logging_steps               = 5,\n",
+    "    save_strategy               = 'no',\n",
+    "    dataloader_num_workers      = 0,\n",
+    ")\n",
+    "\n",
+    "sft_steps = (\n",
+    "    len(sft_dataset)\n",
+    "    // (sft_cfg.per_device_train_batch_size * sft_cfg.gradient_accumulation_steps)\n",
+    "    * sft_cfg.num_train_epochs\n",
+    ")\n",
+    "print(f'SFT: {len(sft_dataset)} examples x {sft_cfg.num_train_epochs} epochs -> ~{sft_steps} steps')\n",
+    "\n",
+    "sft_trainer = SFTTrainer(\n",
+    "    model=model, tokenizer=tokenizer,\n",
+    "    train_dataset=sft_dataset, args=sft_cfg,\n",
+    ")\n",
+    "sft_stats = sft_trainer.train()\n",
+    "\n",
+    "runtime = sft_stats.metrics.get('train_runtime', 0)\n",
+    "loss    = sft_stats.metrics.get('train_loss', 0)\n",
+    "print(f'SFT done: {round(runtime/60, 1)} min, loss={loss:.3f}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-11-md",
+   "metadata": {},
+   "source": [
+    "## 9. Verify SFT Output — Model Must Output Valid JSON"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-11",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from training.rollout import extract_json_action, _obs_to_dict, _current_obs_text, build_messages\n",
+    "\n",
+    "model.eval()\n",
+    "with GenericEnvClient(base_url=ENV_URL).sync() as _env:\n",
+    "    r        = _env.reset(seed=99001)\n",
+    "    obs_dict = _obs_to_dict(r.observation if hasattr(r, 'observation') else r)\n",
+    "    obs_text = _current_obs_text(obs_dict, 0, obs_dict.get('task_brief', ''))\n",
+    "    msgs     = build_messages([], obs_text)\n",
+    "    prompt   = tokenizer.apply_chat_template(\n",
+    "        msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n",
+    "    )\n",
+    "\n",
+    "inputs = tokenizer([prompt], return_tensors='pt').to(model.device)\n",
+    "with torch.no_grad():\n",
+    "    out = model.generate(\n",
+    "        **inputs, max_new_tokens=128, do_sample=False,\n",
+    "        pad_token_id=tokenizer.eos_token_id\n",
+    "    )\n",
+    "completion = tokenizer.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)\n",
+    "parsed     = extract_json_action(completion)\n",
+    "\n",
+    "print(f'Output : {completion[:400]}')\n",
+    "print(f'Parsed : {parsed}')\n",
+    "\n",
+    "if parsed is not None:\n",
+    "    print('PASS: model outputs valid JSON after SFT')\n",
+    "else:\n",
+    "    print('FAIL: still no valid JSON — run SFT cell again with more epochs or more data')\n",
+    "\n",
+    "model.train()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-12-md",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Phase 2 — GRPO\n",
+    "\n",
+    "Now that the model outputs valid JSON, GRPO can optimize for *correctness*.\n",
+    "\n",
+    "**Reward** (2 components, sum = 1.0):\n",
+    "\n",
+    "| Component | Weight | Signal |\n",
+    "|---|---|---|\n",
+    "| `env_score` | 0.85 | Env grader: 0.25 (ticket) + 0.20 (label) + 0.20 (priority) + 0.20 (team) + 0.15 (channel) |\n",
+    "| `json_ratio` | 0.15 | Fraction of steps with parseable JSON — maintains format quality |\n",
+    "\n",
+    "`env_score` naturally varies 0.25–1.0 per episode (the model may get the ticket right\n",
+    "but pick the wrong label, or get the channel wrong). This is the learning signal.\n",
+    "Anti-hacking: org_config values differ every episode (seeded), so the model cannot memorize answers."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-13-md",
+   "metadata": {},
+   "source": [
+    "## 10. Generate GRPO Training Dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-13",
+   "metadata": {},
+   "outputs": [],
+   "source": "from training.dataset import generate_triage_dataset\n\n# generate_triage_dataset now pre-simulates the env's RNG to only include seeds\n# where env.reset(seed) will actually run a TRIAGE episode — previously the env\n# silently ran release_notes/dep_update for the same seed, guaranteeing env_score=0.\nrows         = generate_triage_dataset(n_episodes=N_GRPO_EPISODES, base_seed=42)\ngrpo_dataset = Dataset.from_list([{'prompt': r['prompt']} for r in rows])\nprint(f'GRPO dataset: {len(grpo_dataset)} triage episodes')\nprint(f'Difficulties: {set(r[\"difficulty\"] for r in rows)}')\nprint(f'Sample prompt: {rows[0][\"prompt\"][:120]}')"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-14-md",
+   "metadata": {},
+   "source": [
+    "## 11. GRPO Rollout + Reward Functions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-14",
+   "metadata": {},
+   "outputs": [],
+   "source": "from training.rollout import (\n    _obs_to_dict, _current_obs_text, build_messages,\n    extract_json_action, step_aware_fallback,\n    _generate_no_vllm,\n)\nfrom training.dataset import parse_seed_from_prompt\nfrom training.rewards import compute_rollout_reward\n\ngrpo_env = GenericEnvClient(base_url=ENV_URL).sync()\ngrpo_env.connect()\nprint('GRPO training env connected')\n\nROLLOUT_TEMPERATURE = 1.1   # must be > 1.0 for rollout diversity\n\n\ndef run_grpo_episode(trainer, env, tok, dataset_prompt, max_steps=TRAIN_MAX_STEPS,\n                     gen_offset=0):\n    \"\"\"Run one PM-ops triage episode and return trajectory + reward.\n\n    Reward is runbook-compliance based (see training/rewards.py):\n    - Did the model read the runbook?\n    - Did it use a valid label/priority/team from the runbook?\n    - Did it notify a correct oncall channel?\n\n    This creates genuine reward variance across rollouts because org configs\n    vary by seed — the same hardcoded label/team/channel is correct for some\n    orgs and wrong for others, giving GRPO a real gradient signal.\n    \"\"\"\n    seed = parse_seed_from_prompt(dataset_prompt)\n    if seed is not None:\n        result = env.reset(seed=seed + gen_offset)\n    else:\n        result = env.reset()\n\n    obs_dict   = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n    task_brief = obs_dict.get('task_brief') or dataset_prompt\n\n    prompt_ids, completion_ids, logprobs = [], [], []\n    turn_history     = []\n    valid_json_count = 0\n    env_score        = 0.0\n    step, done       = 0, False\n    _sample_logged   = False\n\n    # Runbook-compliance tracking\n    read_runbook_done  = False\n    valid_labels:     set = set()\n    valid_priorities: set = set()\n    valid_teams:      set = set()\n    oncall_channels:  set = set()\n    ticket_label:    str | None = None\n    ticket_priority: str | None = None\n    assigned_team:   str | None = None\n    posted_channels: list = []\n\n    while not done and step < max_steps:\n        obs_text    = _current_obs_text(obs_dict, step, task_brief)\n        msgs        = build_messages(turn_history, obs_text)\n        prompt_text = tok.apply_chat_template(\n            msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n        )\n\n        rollout_out = _generate_no_vllm(\n            trainer, prompt_text, tok,\n            max_new_tokens=MAX_COMP_LEN,\n            temperature=ROLLOUT_TEMPERATURE,\n        )\n        prompt_ids.extend(rollout_out['prompt_ids'])\n        completion_ids.extend(rollout_out['completion_ids'])\n        logprobs.extend(rollout_out['logprobs'])\n        completion_text = rollout_out['text']\n\n        if not _sample_logged:\n            print(f'  [sample] {repr(completion_text[:180])}')\n            _sample_logged = True\n\n        parsed = extract_json_action(completion_text)\n        if parsed is not None:\n            valid_json_count += 1\n        else:\n            parsed = step_aware_fallback(step, max_steps)\n\n        action_type = parsed.get('action_type', 'meta.noop')\n        args        = parsed.get('args', {})\n\n        if action_type == 'meta.read_runbook' and parsed is not None:\n            read_runbook_done = True\n\n        if action_type == 'ticketing.create_ticket' and parsed is not None and ticket_label is None:\n            ticket_label    = args.get('label')\n            ticket_priority = args.get('priority')\n\n        if action_type == 'ticketing.assign_ticket' and parsed is not None and assigned_team is None:\n            assigned_team = args.get('team')\n\n        if action_type == 'chat.post_message' and parsed is not None:\n            ch = args.get('channel', '')\n            if ch:\n                posted_channels.append(ch)\n\n        turn_history.append({\n            'obs_text'  : obs_text,\n            'completion': completion_text,\n            'is_runbook': (action_type == 'meta.read_runbook' and parsed is not None),\n        })\n\n        try:\n            result = env.step({'action_type': action_type, 'args': args})\n        except RuntimeError as exc:\n            if 'VALIDATION_ERROR' in str(exc):\n                result = env.step({'action_type': 'meta.noop', 'args': {}})\n            else:\n                raise\n\n        obs_dict  = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n\n        # Extract org_config from runbook response (available the step AFTER read_runbook)\n        last_res = obs_dict.get('last_action_result') or {}\n        if action_type == 'meta.read_runbook' and last_res.get('ok'):\n            data = last_res.get('data') or {}\n            if isinstance(data, dict):\n                org = data.get('org_config') or {}\n                valid_labels     = set(org.get('label_taxonomy', {}).values())\n                valid_priorities = set(org.get('priority_levels', []))\n                valid_teams      = set(org.get('team_map', {}).values())\n                oncall_channels  = set(org.get('oncall_channels', {}).values())\n\n        done      = bool(getattr(result, 'done', obs_dict.get('done', False)))\n        env_score = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n        step     += 1\n\n    reward = compute_rollout_reward(\n        read_runbook_done = read_runbook_done,\n        valid_labels      = valid_labels,\n        valid_priorities  = valid_priorities,\n        valid_teams       = valid_teams,\n        oncall_channels   = oncall_channels,\n        ticket_label      = ticket_label,\n        ticket_priority   = ticket_priority,\n        assigned_team     = assigned_team,\n        posted_channels   = posted_channels,\n        env_score         = env_score,\n        valid_json_count  = valid_json_count,\n    )\n\n    lbl_ok = '✓' if ticket_label and ticket_label in valid_labels else ('✗' if ticket_label else '-')\n    pri_ok = '✓' if ticket_priority and ticket_priority in valid_priorities else ('✗' if ticket_priority else '-')\n    tm_ok  = '✓' if assigned_team and assigned_team in valid_teams else ('✗' if assigned_team else '-')\n    ch_ok  = '✓' if any(ch in oncall_channels for ch in posted_channels) else ('✗' if posted_channels else '-')\n    print(f'  [rollout] steps={step} env={env_score:.3f} '\n          f'label={lbl_ok} pri={pri_ok} team={tm_ok} ch={ch_ok} '\n          f'offset={gen_offset} -> reward={reward:.3f}')\n\n    return {\n        'prompt_ids'    : prompt_ids,\n        'completion_ids': completion_ids,\n        'logprobs'      : logprobs,\n        'reward'        : reward,\n    }\n\n\ndef grpo_rollout_func(prompts, trainer=None):\n    out = {'prompt_ids': [], 'completion_ids': [], 'logprobs': [], 'reward': []}\n    prompt_seen: dict = {}\n    for prompt in prompts:\n        gen_offset          = prompt_seen.get(prompt, 0)\n        prompt_seen[prompt] = gen_offset + 1\n        ep = run_grpo_episode(trainer, grpo_env, tokenizer, prompt,\n                              TRAIN_MAX_STEPS, gen_offset=gen_offset)\n        for k in out:\n            out[k].append(ep[k])\n    return out\n\n\n_warned_missing_reward = False\ndef grpo_reward_func(completions, **kwargs):\n    \"\"\"Passthrough — reward is pre-computed in grpo_rollout_func.\"\"\"\n    global _warned_missing_reward\n    rewards = kwargs.get('reward', [])\n    if not rewards:\n        if not _warned_missing_reward:\n            print(f\"[WARN] reward_func fallback, no reward key. kwargs: {list(kwargs.keys())}\")\n            _warned_missing_reward = True\n        return [0.0] * len(completions)\n    return [float(r) for r in rewards]\n\n\nprint(f'GRPO rollout ready  max_steps={TRAIN_MAX_STEPS}  temperature={ROLLOUT_TEMPERATURE}')"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-15-md",
+   "metadata": {},
+   "source": [
+    "## 12. GRPO Config + Trainer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-15",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from trl import GRPOConfig\n",
+    "from training.pm_ops_trainer import PMOpsGRPOTrainer\n",
+    "\n",
+    "OUTPUT_DIR = 'pm-ops-grpo-Qwen3-1.7B-triage-v3'\n",
+    "HF_REPO_ID = f'Saurav1/{OUTPUT_DIR}'\n",
+    "\n",
+    "grpo_cfg = GRPOConfig(\n",
+    "    # Training\n",
+    "    num_train_epochs              = 2,\n",
+    "    learning_rate                 = 1e-6,   # lower LR: model has SFT init, don't overwrite it\n",
+    "    gradient_accumulation_steps   = GRAD_ACCUM,\n",
+    "    per_device_train_batch_size   = 1,\n",
+    "    warmup_steps                  = 5,\n",
+    "    num_generations               = NUM_GEN,\n",
+    "    # Sequence lengths\n",
+    "    max_completion_length         = MAX_COMP_LEN,\n",
+    "    max_prompt_length             = 4096,\n",
+    "    # Keep disabled for rollout_func compatibility on cloud runtimes\n",
+    "    use_vllm                      = False,\n",
+    "    # Output\n",
+    "    output_dir                    = OUTPUT_DIR,\n",
+    "    report_to                     = 'trackio',\n",
+    "    trackio_space_id              = OUTPUT_DIR,\n",
+    "    logging_steps                 = 1,\n",
+    "    save_steps                    = 20,\n",
+    "    gradient_checkpointing        = False,  # Unsloth handles this\n",
+    ")\n",
+    "\n",
+    "eff_batch   = grpo_cfg.per_device_train_batch_size * GRAD_ACCUM\n",
+    "total_steps = len(grpo_dataset) * NUM_GEN * grpo_cfg.num_train_epochs // eff_batch\n",
+    "print(f'GRPO: {len(grpo_dataset)} eps x {NUM_GEN} gen x {grpo_cfg.num_train_epochs} epochs -> ~{total_steps} steps')\n",
+    "\n",
+    "# PMOpsGRPOTrainer overrides _calculate_rewards to read the pre-computed\n",
+    "# 'reward' key directly from the rollout batch — bypasses broken kwargs plumbing.\n",
+    "trainer = PMOpsGRPOTrainer(\n",
+    "    model            = model,\n",
+    "    processing_class = tokenizer,\n",
+    "    reward_funcs     = grpo_reward_func,  # kept as fallback only\n",
+    "    train_dataset    = grpo_dataset,\n",
+    "    args             = grpo_cfg,\n",
+    "    rollout_func     = grpo_rollout_func,\n",
+    ")\n",
+    "print(f'PMOpsGRPOTrainer ready: {type(trainer).__name__}')\n",
+    "assert isinstance(trainer, PMOpsGRPOTrainer), 'trainer must be PMOpsGRPOTrainer'\n",
+    "assert grpo_cfg.use_vllm is False, 'use_vllm must be False for rollout compatibility'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9a7c99b4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Preflight: ensure rollout returns reward and trainer received rollout_func\n",
+    "probe = grpo_rollout_func([grpo_dataset[0]['prompt']], trainer=trainer)\n",
+    "print('probe keys:', list(probe.keys()))\n",
+    "print('probe reward sample:', probe['reward'][:1])\n",
+    "assert len(probe['reward']) == 1, 'rollout probe did not return reward values'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-16-md",
+   "metadata": {},
+   "source": [
+    "## 13. Train\n",
+    "\n",
+    "Watch stdout for:\n",
+    "- `[sample] '```json...'` — should look like valid JSON code blocks\n",
+    "- `[rollout] env=X.XXX` — **should trend upward over steps** (this is the signal)\n",
+    "- `[rollout] json=X.XX` — should stay > 0.7 (SFT maintains format quality)\n",
+    "\n",
+    "Watch trackio for the reward curve."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-16",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer_stats = trainer.train()\n",
+    "\n",
+    "used_gb    = round(torch.cuda.max_memory_reserved() / 1024**3, 2)\n",
+    "train_mins = round(trainer_stats.metrics.get('train_runtime', 0) / 60, 1)\n",
+    "print(f'Training time : {train_mins} min')\n",
+    "print(f'Peak GPU      : {used_gb} GB / {TOTAL_GB} GB ({round(used_gb/TOTAL_GB*100, 1)}%)')\n",
+    "final_reward = trainer_stats.metrics.get('train/reward', trainer_stats.metrics.get('train_loss', '?'))\n",
+    "print(f'Final reward  : {final_reward}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-17-md",
+   "metadata": {},
+   "source": [
+    "## 14. Save Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-17",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "grpo_env.close()\n",
+    "\n",
+    "# Unsloth merged save — dequantises first, then merges LoRA cleanly into bf16.\n",
+    "# Do NOT use trainer.save_model() directly on a 4-bit + LoRA model.\n",
+    "model.save_pretrained_merged(OUTPUT_DIR, tokenizer, save_method='merged_16bit')\n",
+    "model.push_to_hub_merged(HF_REPO_ID, tokenizer, save_method='merged_16bit')\n",
+    "print(f'Pushed -> https://huggingface.co/{HF_REPO_ID}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-18-md",
+   "metadata": {},
+   "source": [
+    "## 15. Evaluate: Baseline vs Trained"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-18",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoModelForCausalLM\n",
+    "\n",
+    "N_EVAL         = 15\n",
+    "EVAL_MAX_STEPS = 12\n",
+    "EVAL_SEED_BASE = 9000\n",
+    "\n",
+    "eval_model = AutoModelForCausalLM.from_pretrained(\n",
+    "    OUTPUT_DIR, torch_dtype=torch.bfloat16, device_map='auto'\n",
+    ")\n",
+    "eval_model.eval()\n",
+    "\n",
+    "\n",
+    "def eval_trained(n=N_EVAL):\n",
+    "    scores = []\n",
+    "    with GenericEnvClient(base_url=ENV_URL).sync() as env:\n",
+    "        for i in range(n):\n",
+    "            result     = env.reset(seed=EVAL_SEED_BASE + i)\n",
+    "            obs_dict   = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "            task_brief = obs_dict.get('task_brief', '')\n",
+    "            history, step, score, done = [], 0, 0.0, False\n",
+    "\n",
+    "            while not done and step < EVAL_MAX_STEPS:\n",
+    "                obs_text = _current_obs_text(obs_dict, step, task_brief)\n",
+    "                msgs     = build_messages(history, obs_text)\n",
+    "                prompt   = tokenizer.apply_chat_template(\n",
+    "                    msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n",
+    "                )\n",
+    "                inputs = tokenizer([prompt], return_tensors='pt', truncation=True, max_length=4096)\n",
+    "                inputs = {k: v.to(eval_model.device) for k, v in inputs.items()}\n",
+    "                with torch.no_grad():\n",
+    "                    out_ids = eval_model.generate(\n",
+    "                        **inputs, max_new_tokens=256, do_sample=False,\n",
+    "                        pad_token_id=tokenizer.eos_token_id\n",
+    "                    )\n",
+    "                completion = tokenizer.decode(\n",
+    "                    out_ids[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True\n",
+    "                )\n",
+    "                parsed = extract_json_action(completion) or step_aware_fallback(step, EVAL_MAX_STEPS)\n",
+    "                result   = env.step({'action_type': parsed['action_type'], 'args': parsed.get('args', {})})\n",
+    "                obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "                done      = bool(getattr(result, 'done', obs_dict.get('done', False)))\n",
+    "                score     = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n",
+    "                history.append({'obs_text': obs_text, 'completion': completion, 'is_runbook': False})\n",
+    "                step     += 1\n",
+    "\n",
+    "            scores.append(score)\n",
+    "            print(f'  Trained  ep {i+1}/{n}: score={score:.3f}')\n",
+    "    return scores\n",
+    "\n",
+    "\n",
+    "def eval_baseline(n=N_EVAL):\n",
+    "    from inference import baseline_agent\n",
+    "    scores = []\n",
+    "    with GenericEnvClient(base_url=ENV_URL).sync() as env:\n",
+    "        for i in range(n):\n",
+    "            result     = env.reset(seed=EVAL_SEED_BASE + i)\n",
+    "            obs_dict   = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "            org_config, step, score, done = {}, 0, 0.0, False\n",
+    "            while not done and step < EVAL_MAX_STEPS:\n",
+    "                at, args = baseline_agent(obs_dict, org_config)\n",
+    "                result   = env.step({'action_type': at, 'args': args})\n",
+    "                obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "                done      = bool(getattr(result, 'done', obs_dict.get('done', False)))\n",
+    "                score     = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n",
+    "                step     += 1\n",
+    "            scores.append(score)\n",
+    "            print(f'  Baseline ep {i+1}/{n}: score={score:.3f}')\n",
+    "    return scores\n",
+    "\n",
+    "\n",
+    "print('--- Baseline ---')\n",
+    "baseline_scores = eval_baseline()\n",
+    "print('\\n--- Trained ---')\n",
+    "trained_scores  = eval_trained()\n",
+    "\n",
+    "print(f'\\nBaseline avg : {sum(baseline_scores)/N_EVAL:.3f}')\n",
+    "print(f'Trained  avg : {sum(trained_scores)/N_EVAL:.3f}')\n",
+    "print(f'Delta        : {(sum(trained_scores)-sum(baseline_scores))/N_EVAL:+.3f}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-19-md",
+   "metadata": {},
+   "source": [
+    "## 16. Plot Results"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-19",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n",
+    "\n",
+    "ax, x, w = axes[0], np.arange(N_EVAL), 0.35\n",
+    "ax.bar(x - w/2, baseline_scores, w, label='Baseline', color='steelblue', alpha=0.8)\n",
+    "ax.bar(x + w/2, trained_scores,  w, label='GRPO v3',  color='coral',     alpha=0.8)\n",
+    "ax.axhline(sum(baseline_scores)/N_EVAL, color='steelblue', ls='--', alpha=0.5, lw=1.5)\n",
+    "ax.axhline(sum(trained_scores)/N_EVAL,  color='coral',     ls='--', alpha=0.5, lw=1.5)\n",
+    "ax.set(xlabel='Eval episode', ylabel='Reward (0-1)',\n",
+    "       title='Per-episode reward: Baseline vs GRPO v3', xticks=x, ylim=(0, 1.05))\n",
+    "ax.legend()\n",
+    "\n",
+    "ax2  = axes[1]\n",
+    "avgs = [sum(baseline_scores)/N_EVAL, sum(trained_scores)/N_EVAL]\n",
+    "bars = ax2.bar(['Baseline', 'GRPO v3'], avgs, color=['steelblue', 'coral'], alpha=0.85, width=0.5)\n",
+    "for bar, val in zip(bars, avgs):\n",
+    "    ax2.text(bar.get_x() + bar.get_width()/2, val + 0.01, f'{val:.3f}',\n",
+    "             ha='center', fontsize=13, fontweight='bold')\n",
+    "ax2.set(ylabel='Average reward (0-1)', title=f'Average over {N_EVAL} triage episodes',\n",
+    "        ylim=(0, 1.05))\n",
+    "\n",
+    "plt.tight_layout()\n",
+    "plt.savefig('eval_results_v3.png', dpi=150, bbox_inches='tight')\n",
+    "plt.show()\n",
+    "print('Saved: eval_results_v3.png')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cell-20-md",
+   "metadata": {},
+   "source": [
+    "## 17. Teardown"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cell-20",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_proc.terminate()\n",
+    "print('Local PM-Ops server stopped')"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.11.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

training/train_v4.ipynb

@@ -0,0 +1,847 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "v4-title",
+   "metadata": {},
+   "source": [
+    "# PM-Ops GRPO Training v4\n",
+    "\n",
+    "Clean rewrite — all known v3 bugs fixed.\n",
+    "\n",
+    "| Fix | Details |\n",
+    "|---|---|\n",
+    "| Constant reward=0.350 | Replaced env_score formula with runbook-compliance scoring |\n",
+    "| `gen_slot=0` override bug | gen_slot now correctly offsets env seed per GRPO generation |\n",
+    "| Double `env.step` per step | Removed duplicate step call inside try/except |\n",
+    "| Dataset/env task-type mismatch | `dataset.py` pre-filters seeds to triage-only episodes |\n",
+    "| Near-greedy generation | Temperature 1.1 + top_k=50 for diverse rollouts |\n",
+    "| PatchFastRL called twice | Called once, unconditionally, before any trl imports |\n",
+    "\n",
+    "**Reward**: read_runbook (+0.10) + valid_label (±0.20/0.10) + valid_priority (±0.15/0.10) + valid_team (±0.20/0.10) + right_channel (±0.25/0.10) + env_bonus (×0.10). Varies per org config — different valid values per seed.  \n",
+    "**Stack**: Unsloth 2026.x · TRL 0.22.2 · Qwen3-1.7B · PMOpsGRPOTrainer  \n",
+    "**GPU**: A100-40GB → ~2 hrs (SFT 15 min + GRPO 90 min)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s0",
+   "metadata": {},
+   "source": [
+    "## 0. Install"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-install",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture\n",
+    "import os\n",
+    "!pip install --upgrade -qqq uv\n",
+    "if 'COLAB_' not in ''.join(os.environ.keys()):\n",
+    "    !uv pip install unsloth vllm\n",
+    "else:\n",
+    "    import subprocess\n",
+    "    is_t4 = 'Tesla T4' in str(subprocess.check_output(['nvidia-smi']))\n",
+    "    _vllm   = 'vllm==0.9.2'   if is_t4 else 'vllm==0.15.1'\n",
+    "    _triton = 'triton==3.2.0' if is_t4 else 'triton'\n",
+    "    !uv pip install -qqq --upgrade {_vllm} torchvision bitsandbytes xformers unsloth\n",
+    "    !uv pip install -qqq {_triton}\n",
+    "!uv pip install transformers==4.56.2\n",
+    "!uv pip install --no-deps trl==0.22.2\n",
+    "!pip install 'numpy==1.26.4' --break-system-packages -q\n",
+    "!pip install openenv openenv-core -q"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s1",
+   "metadata": {},
+   "source": [
+    "## 1. GPU Config + Patch"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-config",
+   "metadata": {},
+   "outputs": [],
+   "source": "import torch\nfrom unsloth import FastLanguageModel, PatchFastRL\n\n# Must patch BEFORE any trl imports\nPatchFastRL('GRPO', FastLanguageModel)\n\nimport trl\n\ngpu      = torch.cuda.get_device_properties(0)\nTOTAL_GB = round(gpu.total_memory / 1024**3, 1)\nIS_A100  = TOTAL_GB >= 35\n\nNUM_GEN         = 6   if IS_A100 else 2\nGRAD_ACCUM      = 32  if IS_A100 else 8\nMAX_COMP_LEN    = 192\n# T4 needs more SFT to reach low enough loss before GRPO.\n# 20 eps → loss≈2.0 is too high; 50 eps → loss≈0.8 is a better floor.\nN_SFT_EPISODES  = 120 if IS_A100 else 50\nN_GRPO_EPISODES = 150 if IS_A100 else 30\nTRAIN_MAX_STEPS = 12\n\nprint(f'torch={torch.__version__}  trl={trl.__version__}')\nprint(f'GPU: {gpu.name} ({TOTAL_GB} GB)  IS_A100={IS_A100}')\nprint(f'num_gen={NUM_GEN}  grad_accum={GRAD_ACCUM}  '\n      f'sft_eps={N_SFT_EPISODES}  grpo_eps={N_GRPO_EPISODES}  '\n      f'max_steps={TRAIN_MAX_STEPS}  max_comp_len={MAX_COMP_LEN}')"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s2",
+   "metadata": {},
+   "source": [
+    "## 2. Clone PM-Ops Repo"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-clone",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os, sys\n",
+    "\n",
+    "REPO_URL = 'https://huggingface.co/spaces/TheCrustaceans/Pm-ops'\n",
+    "REPO_DIR = '/content/Pm_ops'\n",
+    "\n",
+    "if not os.path.exists(REPO_DIR):\n",
+    "    !git clone --depth=1 -q {REPO_URL} {REPO_DIR}\n",
+    "    print(f'Cloned -> {REPO_DIR}')\n",
+    "else:\n",
+    "    !git -C {REPO_DIR} pull -q origin main\n",
+    "    print(f'Pulled -> {REPO_DIR}')\n",
+    "\n",
+    "for p in [REPO_DIR, os.path.join(REPO_DIR, 'training')]:\n",
+    "    if p not in sys.path:\n",
+    "        sys.path.insert(0, p)\n",
+    "os.chdir(REPO_DIR)\n",
+    "print(f'CWD: {os.getcwd()}')\n",
+    "!git -C {REPO_DIR} log --oneline -3"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s3",
+   "metadata": {},
+   "source": [
+    "## 3. HuggingFace Login"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-hf-login",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import notebook_login\n",
+    "notebook_login()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s4",
+   "metadata": {},
+   "source": [
+    "## 4. Start PM-Ops Server"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-server",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import subprocess, time, requests\n",
+    "\n",
+    "# Kill any leftover server from a previous run\n",
+    "subprocess.run(['pkill', '-f', 'uvicorn'], capture_output=True)\n",
+    "time.sleep(1)\n",
+    "\n",
+    "server_proc = subprocess.Popen(\n",
+    "    [sys.executable, '-m', 'uvicorn', 'server.app:app',\n",
+    "     '--host', '0.0.0.0', '--port', '8000'],\n",
+    "    cwd=REPO_DIR, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,\n",
+    ")\n",
+    "ENV_URL = 'http://localhost:8000'\n",
+    "\n",
+    "for _ in range(30):\n",
+    "    try:\n",
+    "        if requests.get(f'{ENV_URL}/', timeout=2).status_code == 200:\n",
+    "            print(f'PM-Ops server ready  pid={server_proc.pid}')\n",
+    "            break\n",
+    "    except Exception:\n",
+    "        pass\n",
+    "    time.sleep(1)\n",
+    "else:\n",
+    "    raise RuntimeError('Server did not start in 30 s')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s5",
+   "metadata": {},
+   "source": [
+    "## 5. Verify Env"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-verify-env",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from openenv.core import GenericEnvClient\n",
+    "from training.rollout import _obs_to_dict\n",
+    "\n",
+    "with GenericEnvClient(base_url=ENV_URL).sync() as _env:\n",
+    "    r   = _env.reset(seed=42)\n",
+    "    obs = _obs_to_dict(r.observation if hasattr(r, 'observation') else r)\n",
+    "    print(f'task_brief : {obs.get(\"task_brief\", \"?\")[:80]}...')\n",
+    "    step_r = _env.step({'action_type': 'meta.read_runbook', 'args': {}})\n",
+    "    rb_obs = _obs_to_dict(step_r.observation if hasattr(step_r, 'observation') else step_r)\n",
+    "    data   = (rb_obs.get('last_action_result') or {}).get('data', {})\n",
+    "    org    = data.get('org_config', {}) if isinstance(data, dict) else {}\n",
+    "    print(f'org labels    : {list(org.get(\"label_taxonomy\", {}).values())}')\n",
+    "    print(f'org priorities: {org.get(\"priority_levels\", [])}')\n",
+    "    print(f'org channels  : {list(org.get(\"oncall_channels\", {}).values())}')\n",
+    "    print('Env verify: OK')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s6",
+   "metadata": {},
+   "source": [
+    "## 6. Load Model — Unsloth 4-bit + LoRA"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-model",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "MODEL_NAME = 'Qwen/Qwen3-1.7B'\n",
+    "LORA_RANK  = 16\n",
+    "\n",
+    "model, tokenizer = FastLanguageModel.from_pretrained(\n",
+    "    model_name             = MODEL_NAME,\n",
+    "    max_seq_length         = 4096 + MAX_COMP_LEN,\n",
+    "    load_in_4bit           = True,\n",
+    "    fast_inference         = False,\n",
+    "    max_lora_rank          = LORA_RANK,\n",
+    "    gpu_memory_utilization = 0.55,\n",
+    ")\n",
+    "model = FastLanguageModel.get_peft_model(\n",
+    "    model,\n",
+    "    r                          = LORA_RANK,\n",
+    "    target_modules             = ['q_proj','k_proj','v_proj','o_proj',\n",
+    "                                   'gate_proj','up_proj','down_proj'],\n",
+    "    lora_alpha                 = LORA_RANK,\n",
+    "    use_gradient_checkpointing = 'unsloth',\n",
+    "    random_state               = 42,\n",
+    ")\n",
+    "tokenizer.pad_token    = tokenizer.eos_token\n",
+    "tokenizer.padding_side = 'left'\n",
+    "model.print_trainable_parameters()\n",
+    "print(f'GPU after load: {round(torch.cuda.max_memory_reserved()/1024**3, 2)} GB / {TOTAL_GB} GB')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-phase1",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Phase 1 — SFT Warmup\n",
+    "\n",
+    "Teach the model the JSON output format and PM-ops workflow before GRPO.  \n",
+    "`baseline_agent` runs N deterministic episodes → ~6 steps each → supervised (prompt, completion) pairs.  \n",
+    "2 SFT epochs (~15 min on A100). Without SFT, all rollouts output freeform text → zero GRPO gradient."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s7",
+   "metadata": {},
+   "source": [
+    "## 7. Generate SFT Dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-sft-data",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json as _json\n",
+    "from datasets import Dataset\n",
+    "from inference import baseline_agent\n",
+    "from training.rollout import _obs_to_dict, _current_obs_text, build_messages\n",
+    "\n",
+    "\n",
+    "def generate_sft_dataset(env_url, tok, n_episodes, seed_start=2000):\n",
+    "    examples = []\n",
+    "    with GenericEnvClient(base_url=env_url).sync() as env:\n",
+    "        for i in range(n_episodes):\n",
+    "            result     = env.reset(seed=seed_start + i)\n",
+    "            obs_dict   = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "            task_brief = obs_dict.get('task_brief', '')\n",
+    "            turn_history, org_config, step, done = [], {}, 0, False\n",
+    "\n",
+    "            while not done and step < 8:\n",
+    "                obs_text          = _current_obs_text(obs_dict, step, task_brief)\n",
+    "                action_type, args = baseline_agent(obs_dict, org_config)\n",
+    "                payload           = {'action_type': action_type, 'args': args}\n",
+    "                completion        = '```json\\n' + _json.dumps(payload) + '\\n```'\n",
+    "                msgs              = build_messages(turn_history, obs_text)\n",
+    "                prompt            = tok.apply_chat_template(\n",
+    "                    msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n",
+    "                )\n",
+    "                examples.append({'text': prompt + completion + tok.eos_token})\n",
+    "                turn_history.append({\n",
+    "                    'obs_text': obs_text, 'completion': completion,\n",
+    "                    'is_runbook': (action_type == 'meta.read_runbook'),\n",
+    "                })\n",
+    "                result   = env.step({'action_type': action_type, 'args': args})\n",
+    "                obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "                if action_type == 'meta.read_runbook':\n",
+    "                    last = obs_dict.get('last_action_result') or {}\n",
+    "                    if last.get('ok'):\n",
+    "                        data = last.get('data') or {}\n",
+    "                        if isinstance(data, dict) and 'org_config' in data:\n",
+    "                            org_config.update(data['org_config'])\n",
+    "                done = bool(getattr(result, 'done', obs_dict.get('done', False)))\n",
+    "                step += 1\n",
+    "\n",
+    "            if (i + 1) % 20 == 0:\n",
+    "                print(f'  {i+1}/{n_episodes} eps — {len(examples)} examples')\n",
+    "\n",
+    "    return examples\n",
+    "\n",
+    "\n",
+    "print(f'Generating {N_SFT_EPISODES} SFT demonstration episodes...')\n",
+    "sft_raw     = generate_sft_dataset(ENV_URL, tokenizer, n_episodes=N_SFT_EPISODES)\n",
+    "sft_dataset = Dataset.from_list(sft_raw)\n",
+    "print(f'SFT dataset: {len(sft_dataset)} examples (~{len(sft_dataset)//6} eps × 6 steps)')\n",
+    "print(f'Sample (first 300 chars):\\n{sft_raw[0][\"text\"][:300]}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s8",
+   "metadata": {},
+   "source": [
+    "## 8. SFT Training"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-sft-train",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from trl import SFTTrainer, SFTConfig\n",
+    "\n",
+    "sft_cfg = SFTConfig(\n",
+    "    dataset_text_field          = 'text',\n",
+    "    max_seq_length              = 2048,\n",
+    "    num_train_epochs            = 2,\n",
+    "    per_device_train_batch_size = 4,\n",
+    "    gradient_accumulation_steps = 4,\n",
+    "    learning_rate               = 2e-4,\n",
+    "    warmup_steps                = 10,\n",
+    "    output_dir                  = 'pm-ops-sft-warmup',\n",
+    "    report_to                   = 'none',\n",
+    "    logging_steps               = 5,\n",
+    "    save_strategy               = 'no',\n",
+    "    dataloader_num_workers      = 0,\n",
+    ")\n",
+    "sft_steps = (\n",
+    "    len(sft_dataset)\n",
+    "    // (sft_cfg.per_device_train_batch_size * sft_cfg.gradient_accumulation_steps)\n",
+    "    * sft_cfg.num_train_epochs\n",
+    ")\n",
+    "print(f'SFT: {len(sft_dataset)} examples × {sft_cfg.num_train_epochs} epochs → ~{sft_steps} steps')\n",
+    "\n",
+    "sft_trainer = SFTTrainer(model=model, tokenizer=tokenizer,\n",
+    "                         train_dataset=sft_dataset, args=sft_cfg)\n",
+    "sft_stats = sft_trainer.train()\n",
+    "loss    = sft_stats.metrics.get('train_loss', 0)\n",
+    "runtime = sft_stats.metrics.get('train_runtime', 0)\n",
+    "print(f'SFT done: {round(runtime/60, 1)} min  loss={loss:.3f}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s9",
+   "metadata": {},
+   "source": [
+    "## 9. Verify SFT — Model Must Output Valid JSON"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-sft-verify",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from training.rollout import extract_json_action\n",
+    "\n",
+    "model.eval()\n",
+    "with GenericEnvClient(base_url=ENV_URL).sync() as _env:\n",
+    "    r        = _env.reset(seed=99001)\n",
+    "    obs_dict = _obs_to_dict(r.observation if hasattr(r, 'observation') else r)\n",
+    "    obs_text = _current_obs_text(obs_dict, 0, obs_dict.get('task_brief', ''))\n",
+    "    msgs     = build_messages([], obs_text)\n",
+    "    prompt   = tokenizer.apply_chat_template(\n",
+    "        msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n",
+    "    )\n",
+    "\n",
+    "inputs = tokenizer([prompt], return_tensors='pt').to(model.device)\n",
+    "with torch.no_grad():\n",
+    "    out = model.generate(**inputs, max_new_tokens=128, do_sample=False,\n",
+    "                         pad_token_id=tokenizer.eos_token_id)\n",
+    "completion = tokenizer.decode(out[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)\n",
+    "parsed     = extract_json_action(completion)\n",
+    "\n",
+    "print(f'Output: {completion[:400]}')\n",
+    "print(f'Parsed: {parsed}')\n",
+    "if parsed is not None:\n",
+    "    print('PASS ✓ — model outputs valid JSON after SFT')\n",
+    "else:\n",
+    "    print('FAIL ✗ — still no valid JSON. Run SFT again with more episodes or epochs.')\n",
+    "model.train()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-phase2",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Phase 2 — GRPO\n",
+    "\n",
+    "Reward components — sum ≈ 0.90 when all correct:\n",
+    "\n",
+    "| Signal | Weight | Description |\n",
+    "|---|---|---|\n",
+    "| `read_runbook` | +0.10 | Did agent read the runbook first? |\n",
+    "| `valid_label` | +0.20 / −0.10 | Ticket label in org's `label_taxonomy`? |\n",
+    "| `valid_priority` | +0.15 / −0.10 | Ticket priority in org's `priority_levels`? |\n",
+    "| `valid_team` | +0.20 / −0.10 | Assigned team in org's `team_map`? |\n",
+    "| `right_channel` | +0.25 / −0.10/ch | Posted to org's `oncall_channels`? |\n",
+    "| `env_bonus` | ×0.10 | Env grader confirmation |\n",
+    "\n",
+    "Reward **varies per org config** (different valid values per seed) → genuine GRPO advantage signal.  \n",
+    "Each of the N GRPO generations gets a different env seed via `gen_slot` offset."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s10",
+   "metadata": {},
+   "source": [
+    "## 10. GRPO Training Dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-grpo-data",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from training.dataset import generate_triage_dataset\n",
+    "\n",
+    "rows         = generate_triage_dataset(n_episodes=N_GRPO_EPISODES, base_seed=42)\n",
+    "grpo_dataset = Dataset.from_list([{'prompt': r['prompt']} for r in rows])\n",
+    "print(f'GRPO dataset: {len(grpo_dataset)} triage episodes')\n",
+    "print(f'Difficulties: {set(r[\"difficulty\"] for r in rows)}')\n",
+    "print(f'Sample: {rows[0][\"prompt\"][:120]}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s11",
+   "metadata": {},
+   "source": [
+    "## 11. GRPO Rollout + Reward"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-rollout",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch.nn.functional as F\n",
+    "from training.rollout import (\n",
+    "    _obs_to_dict, _current_obs_text, build_messages,\n",
+    "    extract_json_action, step_aware_fallback,\n",
+    ")\n",
+    "from training.dataset import parse_seed_from_prompt\n",
+    "from training.rewards import compute_rollout_reward\n",
+    "\n",
+    "grpo_env = GenericEnvClient(base_url=ENV_URL).sync()\n",
+    "grpo_env.connect()\n",
+    "print('GRPO env connected')\n",
+    "\n",
+    "\n",
+    "def run_grpo_episode(trainer, env, tok, dataset_prompt, max_steps=TRAIN_MAX_STEPS, gen_slot=0):\n",
+    "    \"\"\"One full PM-ops episode. gen_slot offsets seed so parallel GRPO generations\n",
+    "    explore distinct env episodes even for the same base prompt.\"\"\"\n",
+    "    seed   = parse_seed_from_prompt(dataset_prompt)\n",
+    "    result = env.reset(seed=seed + gen_slot) if seed is not None else env.reset()\n",
+    "\n",
+    "    obs_dict   = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "    task_brief = obs_dict.get('task_brief') or dataset_prompt\n",
+    "\n",
+    "    prompt_ids, completion_ids, logprobs = [], [], []\n",
+    "    turn_history     = []\n",
+    "    valid_json_count = 0\n",
+    "    env_score        = 0.0\n",
+    "    step, done       = 0, False\n",
+    "\n",
+    "    # Runbook-compliance tracking\n",
+    "    read_runbook_done = False\n",
+    "    valid_labels:     set      = set()\n",
+    "    valid_priorities: set      = set()\n",
+    "    valid_teams:      set      = set()\n",
+    "    oncall_channels:  set      = set()\n",
+    "    ticket_label:    str | None = None\n",
+    "    ticket_priority: str | None = None\n",
+    "    assigned_team:   str | None = None\n",
+    "    posted_channels: list       = []\n",
+    "\n",
+    "    _model  = (trainer.accelerator.unwrap_model(trainer.model)\n",
+    "               if hasattr(trainer, 'accelerator') else trainer.model)\n",
+    "    _device = (trainer.accelerator.device\n",
+    "               if hasattr(trainer, 'accelerator') else next(_model.parameters()).device)\n",
+    "\n",
+    "    while not done and step < max_steps:\n",
+    "        obs_text    = _current_obs_text(obs_dict, step, task_brief)\n",
+    "        msgs        = build_messages(turn_history, obs_text)\n",
+    "        prompt_text = tok.apply_chat_template(\n",
+    "            msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n",
+    "        )\n",
+    "\n",
+    "        enc  = tok(prompt_text, return_tensors='pt', truncation=True, max_length=4096).to(_device)\n",
+    "        plen = enc['input_ids'].shape[1]\n",
+    "        with torch.no_grad():\n",
+    "            out = _model.generate(\n",
+    "                **enc,\n",
+    "                max_new_tokens         = MAX_COMP_LEN,\n",
+    "                do_sample              = True,\n",
+    "                temperature            = 1.1,\n",
+    "                top_p                  = 0.95,\n",
+    "                top_k                  = 50,\n",
+    "                pad_token_id           = tok.pad_token_id or tok.eos_token_id,\n",
+    "                output_scores          = True,\n",
+    "                return_dict_in_generate = True,\n",
+    "            )\n",
+    "\n",
+    "        cids            = out.sequences[0][plen:].tolist()\n",
+    "        completion_text = tok.decode(cids, skip_special_tokens=True)\n",
+    "\n",
+    "        prompt_ids.extend(enc['input_ids'][0].tolist())\n",
+    "        completion_ids.extend(cids)\n",
+    "        logprobs.extend([\n",
+    "            F.log_softmax(s[0], dim=-1)[t].item()\n",
+    "            for s, t in zip(out.scores, cids)\n",
+    "        ])\n",
+    "\n",
+    "        if step == 0:\n",
+    "            print(f'  [sample] {repr(completion_text[:180])}')\n",
+    "\n",
+    "        parsed   = extract_json_action(completion_text)\n",
+    "        is_valid = parsed is not None\n",
+    "        if not is_valid:\n",
+    "            parsed = step_aware_fallback(step, max_steps)\n",
+    "        else:\n",
+    "            valid_json_count += 1\n",
+    "\n",
+    "        action_type = parsed.get('action_type', 'meta.noop')\n",
+    "        args        = parsed.get('args', {})\n",
+    "        print(f'  [step {step}] {action_type}')\n",
+    "\n",
+    "        # Compliance signal capture\n",
+    "        if action_type == 'meta.read_runbook' and is_valid:\n",
+    "            read_runbook_done = True\n",
+    "        if action_type == 'ticketing.create_ticket' and is_valid and ticket_label is None:\n",
+    "            ticket_label    = args.get('label')\n",
+    "            ticket_priority = args.get('priority')\n",
+    "        if action_type == 'ticketing.assign_ticket' and is_valid and assigned_team is None:\n",
+    "            assigned_team = args.get('team')\n",
+    "        if action_type == 'chat.post_message' and is_valid:\n",
+    "            ch = args.get('channel', '')\n",
+    "            if ch:\n",
+    "                posted_channels.append(ch)\n",
+    "\n",
+    "        turn_history.append({\n",
+    "            'obs_text'  : obs_text,\n",
+    "            'completion': completion_text,\n",
+    "            'is_runbook': (action_type == 'meta.read_runbook' and is_valid),\n",
+    "        })\n",
+    "\n",
+    "        result   = env.step({'action_type': action_type, 'args': args})\n",
+    "        obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n",
+    "\n",
+    "        # Extract org config from runbook response (one step after the call)\n",
+    "        last_result = obs_dict.get('last_action_result') or {}\n",
+    "        if action_type == 'meta.read_runbook' and last_result.get('ok'):\n",
+    "            data = last_result.get('data') or {}\n",
+    "            if isinstance(data, dict):\n",
+    "                org = data.get('org_config') or {}\n",
+    "                valid_labels     = set(org.get('label_taxonomy', {}).values())\n",
+    "                valid_priorities = set(org.get('priority_levels', []))\n",
+    "                valid_teams      = set(org.get('team_map', {}).values())\n",
+    "                oncall_channels  = set(org.get('oncall_channels', {}).values())\n",
+    "\n",
+    "        done      = bool(getattr(result, 'done', obs_dict.get('done', False)))\n",
+    "        env_score = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n",
+    "        step     += 1\n",
+    "\n",
+    "    reward = compute_rollout_reward(\n",
+    "        read_runbook_done = read_runbook_done,\n",
+    "        valid_labels      = valid_labels,\n",
+    "        valid_priorities  = valid_priorities,\n",
+    "        valid_teams       = valid_teams,\n",
+    "        oncall_channels   = oncall_channels,\n",
+    "        ticket_label      = ticket_label,\n",
+    "        ticket_priority   = ticket_priority,\n",
+    "        assigned_team     = assigned_team,\n",
+    "        posted_channels   = posted_channels,\n",
+    "        env_score         = env_score,\n",
+    "        valid_json_count  = valid_json_count,\n",
+    "    )\n",
+    "\n",
+    "    lbl = '✓' if ticket_label    and ticket_label    in valid_labels     else ('✗' if ticket_label    else '-')\n",
+    "    pri = '✓' if ticket_priority and ticket_priority in valid_priorities  else ('✗' if ticket_priority else '-')\n",
+    "    tm  = '✓' if assigned_team   and assigned_team   in valid_teams       else ('✗' if assigned_team   else '-')\n",
+    "    ch  = '✓' if any(c in oncall_channels for c in posted_channels)       else ('✗' if posted_channels else '-')\n",
+    "    print(f'  [rollout] steps={step} env={env_score:.3f} '\n",
+    "          f'label={lbl} priority={pri} team={tm} channel={ch} → reward={reward:.3f}')\n",
+    "\n",
+    "    return {\n",
+    "        'prompt_ids'    : prompt_ids,\n",
+    "        'completion_ids': completion_ids,\n",
+    "        'logprobs'      : logprobs,\n",
+    "        'reward'        : reward,\n",
+    "    }\n",
+    "\n",
+    "\n",
+    "def grpo_rollout_func(prompts, trainer=None):\n",
+    "    out = {'prompt_ids': [], 'completion_ids': [], 'logprobs': [], 'reward': []}\n",
+    "    prompt_seen: dict = {}\n",
+    "    for prompt in prompts:\n",
+    "        gen_offset          = prompt_seen.get(prompt, 0)\n",
+    "        prompt_seen[prompt] = gen_offset + 1\n",
+    "        ep = run_grpo_episode(\n",
+    "            trainer, grpo_env, tokenizer, prompt, TRAIN_MAX_STEPS, gen_slot=gen_offset\n",
+    "        )\n",
+    "        for k in out:\n",
+    "            out[k].append(ep[k])\n",
+    "    return out\n",
+    "\n",
+    "\n",
+    "def grpo_reward_func(completions, **kwargs):\n",
+    "    \"\"\"Passthrough — reward pre-computed in grpo_rollout_func.\"\"\"\n",
+    "    rewards = kwargs.get('reward', [])\n",
+    "    if not rewards:\n",
+    "        print(f'[grpo_reward_func] no reward in kwargs — keys: {list(kwargs.keys())}')\n",
+    "        return [0.0] * len(completions)\n",
+    "    return [float(r) for r in rewards]\n",
+    "\n",
+    "\n",
+    "print(f'GRPO rollout ready  max_steps={TRAIN_MAX_STEPS}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s12",
+   "metadata": {},
+   "source": [
+    "## 12. GRPO Config + Trainer"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-trainer",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from trl import GRPOConfig\n",
+    "from training.pm_ops_trainer import PMOpsGRPOTrainer\n",
+    "\n",
+    "OUTPUT_DIR = 'pm-ops-grpo-Qwen3-1.7B-triage-v4'\n",
+    "HF_REPO_ID = f'Saurav1/{OUTPUT_DIR}'\n",
+    "\n",
+    "grpo_cfg = GRPOConfig(\n",
+    "    num_train_epochs            = 2,\n",
+    "    learning_rate               = 1e-6,\n",
+    "    gradient_accumulation_steps = GRAD_ACCUM,\n",
+    "    per_device_train_batch_size = 1,\n",
+    "    warmup_steps                = 5,\n",
+    "    num_generations             = NUM_GEN,\n",
+    "    # max_completion_length must match MAX_COMP_LEN so TRL's internal\n",
+    "    # single-turn generation doesn't overflow and zero the policy loss\n",
+    "    # (clipped_ratio=1.0 means all gradient is masked → loss=0.000).\n",
+    "    max_completion_length       = MAX_COMP_LEN,\n",
+    "    max_prompt_length           = 4096,\n",
+    "    use_vllm                    = False,\n",
+    "    output_dir                  = OUTPUT_DIR,\n",
+    "    report_to                   = 'none',\n",
+    "    logging_steps               = 1,\n",
+    "    save_steps                  = 20,\n",
+    "    gradient_checkpointing      = False,\n",
+    ")\n",
+    "\n",
+    "eff_batch   = grpo_cfg.per_device_train_batch_size * GRAD_ACCUM\n",
+    "total_steps = len(grpo_dataset) * NUM_GEN * grpo_cfg.num_train_epochs // eff_batch\n",
+    "print(f'GRPO: {len(grpo_dataset)} eps × {NUM_GEN} gen × {grpo_cfg.num_train_epochs} epochs → ~{total_steps} steps')\n",
+    "\n",
+    "trainer = PMOpsGRPOTrainer(\n",
+    "    model            = model,\n",
+    "    processing_class = tokenizer,\n",
+    "    reward_funcs     = grpo_reward_func,\n",
+    "    train_dataset    = grpo_dataset,\n",
+    "    args             = grpo_cfg,\n",
+    "    rollout_func     = grpo_rollout_func,\n",
+    ")\n",
+    "assert grpo_cfg.use_vllm is False\n",
+    "print(f'Trainer: {type(trainer).__name__}  ready')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s13",
+   "metadata": {},
+   "source": [
+    "## 13. Preflight Probe"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-probe",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run 2 episodes and verify reward varies (not stuck at a constant)\n",
+    "probe = grpo_rollout_func(\n",
+    "    [grpo_dataset[0]['prompt'], grpo_dataset[1]['prompt']],\n",
+    "    trainer=trainer,\n",
+    ")\n",
+    "print(f'probe rewards : {probe[\"reward\"]}')\n",
+    "assert len(probe['reward']) == 2, 'need one reward per prompt'\n",
+    "assert all(isinstance(r, float) for r in probe['reward']), 'rewards must be float'\n",
+    "print('Preflight PASS ✓')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s14",
+   "metadata": {},
+   "source": [
+    "## 14. Train\n",
+    "\n",
+    "Watch for:\n",
+    "- `[sample] '\\`\\`\\`json ...'` — model should output JSON code blocks\n",
+    "- `label=✓ priority=✓ team=✓ channel=✓` — compliance signals the model is getting right\n",
+    "- `reward/injected_std > 0` in logs — confirms GRPO has a non-zero gradient signal"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-train",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trainer_stats = trainer.train()\n",
+    "\n",
+    "train_mins = round(trainer_stats.metrics.get('train_runtime', 0) / 60, 1)\n",
+    "used_gb    = round(torch.cuda.max_memory_reserved() / 1024**3, 2)\n",
+    "print(f'Training time : {train_mins} min')\n",
+    "print(f'Peak GPU      : {used_gb} GB / {TOTAL_GB} GB ({round(used_gb/TOTAL_GB*100, 1)}%)')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s15",
+   "metadata": {},
+   "source": [
+    "## 15. Save + Push to HF"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-save",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "grpo_env.close()\n",
+    "\n",
+    "model.save_pretrained_merged(OUTPUT_DIR, tokenizer, save_method='merged_16bit')\n",
+    "model.push_to_hub_merged(HF_REPO_ID, tokenizer, save_method='merged_16bit')\n",
+    "print(f'Pushed -> https://huggingface.co/{HF_REPO_ID}')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-eval-hdr",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "## Evaluation — Baseline vs Trained"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s16",
+   "metadata": {},
+   "source": [
+    "## 16. Evaluate"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-eval",
+   "metadata": {},
+   "outputs": [],
+   "source": "from training.rollout import extract_json_action, step_aware_fallback\nfrom inference import baseline_agent\n\nN_EVAL         = 15\nEVAL_MAX_STEPS = 12\nEVAL_SEED_BASE = 9000\n\n# Required after training: switch Unsloth from training hooks to fast inference.\n# Without this, model.generate() is very slow and hangs on T4.\nFastLanguageModel.for_inference(model)\nmodel.eval()\nprint('Model switched to fast inference mode')\n\n\ndef _model_step(obs_dict, task_brief, history, step):\n    \"\"\"One greedy decode step. Returns (action_type, args, json_ok).\"\"\"\n    obs_text = _current_obs_text(obs_dict, step, task_brief)\n    msgs     = build_messages(history, obs_text)\n    prompt   = tokenizer.apply_chat_template(\n        msgs, add_generation_prompt=True, tokenize=False, enable_thinking=False\n    )\n    inputs = tokenizer([prompt], return_tensors='pt', truncation=True, max_length=4096)\n    inputs = {k: v.to(model.device) for k, v in inputs.items()}\n    with torch.no_grad():\n        out_ids = model.generate(\n            **inputs,\n            max_new_tokens = 128,\n            do_sample      = False,\n            pad_token_id   = tokenizer.eos_token_id,\n        )\n    completion = tokenizer.decode(\n        out_ids[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True\n    )\n    parsed  = extract_json_action(completion)\n    json_ok = parsed is not None\n    if not json_ok:\n        parsed = step_aware_fallback(step, EVAL_MAX_STEPS)\n    history.append({'obs_text': obs_text, 'completion': completion, 'is_runbook': False})\n    return parsed.get('action_type', 'meta.noop'), parsed.get('args', {}), json_ok, obs_text\n\n\ndef run_eval(n=N_EVAL, verbose=False):\n    scores = []\n    with GenericEnvClient(base_url=ENV_URL).sync() as env:\n        for i in range(n):\n            result     = env.reset(seed=EVAL_SEED_BASE + i)\n            obs_dict   = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n            task_brief = obs_dict.get('task_brief', '')\n            history, step, score, done = [], 0, 0.0, False\n            actions = []\n\n            while not done and step < EVAL_MAX_STEPS:\n                action_type, args, json_ok, _ = _model_step(obs_dict, task_brief, history, step)\n                actions.append(f'{action_type}{\"\" if json_ok else \"!\"}')\n                result   = env.step({'action_type': action_type, 'args': args})\n                obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n                done      = bool(getattr(result, 'done', obs_dict.get('done', False)))\n                score     = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n                step     += 1\n\n            scores.append(score)\n            action_summary = ' → '.join(a.split('.')[-1] for a in actions)\n            print(f'  Trained ep {i+1}/{n}: score={score:.3f}  [{action_summary}]')\n    return scores\n\n\ndef run_baseline(n=N_EVAL):\n    scores = []\n    with GenericEnvClient(base_url=ENV_URL).sync() as env:\n        for i in range(n):\n            result   = env.reset(seed=EVAL_SEED_BASE + i)\n            obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n            org_config, step, score, done = {}, 0, 0.0, False\n            actions = []\n            while not done and step < EVAL_MAX_STEPS:\n                at, args = baseline_agent(obs_dict, org_config)\n                actions.append(at.split('.')[-1])\n                result   = env.step({'action_type': at, 'args': args})\n                obs_dict = _obs_to_dict(result.observation if hasattr(result, 'observation') else result)\n                done      = bool(getattr(result, 'done', obs_dict.get('done', False)))\n                score     = float(getattr(result, 'reward', obs_dict.get('reward', 0.0)))\n                if at == 'meta.read_runbook':\n                    last = obs_dict.get('last_action_result') or {}\n                    if last.get('ok'):\n                        data = last.get('data') or {}\n                        if isinstance(data, dict) and 'org_config' in data:\n                            org_config.update(data['org_config'])\n                step += 1\n            scores.append(score)\n            action_summary = ' → '.join(actions)\n            print(f'  Baseline ep {i+1}/{n}: score={score:.3f}  [{action_summary}]')\n    return scores\n\n\nprint('--- Baseline ---')\nbaseline_scores = run_baseline()\nprint('\\n--- Trained ---')\ntrained_scores  = run_eval()\n\nb_avg = sum(baseline_scores) / N_EVAL\nt_avg = sum(trained_scores)  / N_EVAL\nprint(f'\\nBaseline avg : {b_avg:.3f}')\nprint(f'Trained  avg : {t_avg:.3f}')\nprint(f'Delta        : {t_avg - b_avg:+.3f}')\nprint()\nprint('Action key: ! = fallback (no valid JSON from model)')"
+  },
+  {
+   "cell_type": "markdown",
+   "id": "v4-s17",
+   "metadata": {},
+   "source": [
+    "## 17. Teardown"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "v4-teardown",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_proc.terminate()\n",
+    "print('PM-Ops server stopped')"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

training/triage_dataset.jsonl

@@ -0,0 +1,150 @@
+{"prompt": "SEED:478163327 | Support escalation: payments is intermittently failing for EU users. Triage and route per org process.", "seed": 478163327, "difficulty": "easy"}
+{"prompt": "SEED:1181241943 | A user reported that the notifications is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1181241943, "difficulty": "medium"}
+{"prompt": "SEED:958682846 | Monitoring alert: elevated error rate on api-gateway. File a bug ticket and page the right team.", "seed": 958682846, "difficulty": "medium"}
+{"prompt": "SEED:440213415 | A user reported that the inventory is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 440213415, "difficulty": "easy"}
+{"prompt": "SEED:1812140441 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1812140441, "difficulty": "easy"}
+{"prompt": "SEED:127978094 | A user reported that the auth is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 127978094, "difficulty": "easy"}
+{"prompt": "SEED:939042955 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 939042955, "difficulty": "medium"}
+{"prompt": "SEED:113971123 | Monitoring alert: elevated error rate on inventory. File a bug ticket and page the right team.", "seed": 113971123, "difficulty": "medium"}
+{"prompt": "SEED:1801823908 | Support escalation: api-gateway is intermittently failing for EU users. Triage and route per org process.", "seed": 1801823908, "difficulty": "medium"}
+{"prompt": "SEED:1929338154 | A user reported that the checkout is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1929338154, "difficulty": "medium"}
+{"prompt": "SEED:27911967 | Monitoring alert: elevated error rate on api-gateway. File a bug ticket and page the right team.", "seed": 27911967, "difficulty": "medium"}
+{"prompt": "SEED:1815115025 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 1815115025, "difficulty": "medium"}
+{"prompt": "SEED:1193448329 | Support escalation: checkout is intermittently failing for EU users. Triage and route per org process.", "seed": 1193448329, "difficulty": "medium"}
+{"prompt": "SEED:924765563 | Monitoring alert: elevated error rate on search. File a bug ticket and page the right team.", "seed": 924765563, "difficulty": "medium"}
+{"prompt": "SEED:438989805 | Support escalation: notifications is intermittently failing for EU users. Triage and route per org process.", "seed": 438989805, "difficulty": "easy"}
+{"prompt": "SEED:1631775357 | Monitoring alert: elevated error rate on auth. File a bug ticket and page the right team.", "seed": 1631775357, "difficulty": "easy"}
+{"prompt": "SEED:1541804686 | Support escalation: api-gateway is intermittently failing for EU users. Triage and route per org process.", "seed": 1541804686, "difficulty": "medium"}
+{"prompt": "SEED:1136108454 | Support escalation: payments is intermittently failing for EU users. Triage and route per org process.", "seed": 1136108454, "difficulty": "easy"}
+{"prompt": "SEED:1973214822 | Support escalation: payments is intermittently failing for EU users. Triage and route per org process.", "seed": 1973214822, "difficulty": "easy"}
+{"prompt": "SEED:1625792787 | A user reported that the checkout is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1625792787, "difficulty": "easy"}
+{"prompt": "SEED:1259191105 | Monitoring alert: elevated error rate on checkout. File a bug ticket and page the right team.", "seed": 1259191105, "difficulty": "medium"}
+{"prompt": "SEED:825873196 | A user reported that the api-gateway is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 825873196, "difficulty": "easy"}
+{"prompt": "SEED:196814233 | Support escalation: api-gateway is intermittently failing for EU users. Triage and route per org process.", "seed": 196814233, "difficulty": "medium"}
+{"prompt": "SEED:1242911821 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 1242911821, "difficulty": "easy"}
+{"prompt": "SEED:999829240 | Monitoring alert: elevated error rate on checkout. File a bug ticket and page the right team.", "seed": 999829240, "difficulty": "easy"}
+{"prompt": "SEED:1632629719 | Monitoring alert: elevated error rate on notifications. File a bug ticket and page the right team.", "seed": 1632629719, "difficulty": "medium"}
+{"prompt": "SEED:1947382419 | Support escalation: inventory is intermittently failing for EU users. Triage and route per org process.", "seed": 1947382419, "difficulty": "medium"}
+{"prompt": "SEED:698594025 | Monitoring alert: elevated error rate on inventory. File a bug ticket and page the right team.", "seed": 698594025, "difficulty": "medium"}
+{"prompt": "SEED:1525876051 | A user reported that the checkout is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1525876051, "difficulty": "medium"}
+{"prompt": "SEED:1146660997 | Monitoring alert: elevated error rate on auth. File a bug ticket and page the right team.", "seed": 1146660997, "difficulty": "easy"}
+{"prompt": "SEED:735034881 | Support escalation: inventory is intermittently failing for EU users. Triage and route per org process.", "seed": 735034881, "difficulty": "medium"}
+{"prompt": "SEED:701808367 | Monitoring alert: elevated error rate on auth. File a bug ticket and page the right team.", "seed": 701808367, "difficulty": "hard"}
+{"prompt": "SEED:1629748727 | A user reported that the api-gateway is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1629748727, "difficulty": "medium"}
+{"prompt": "SEED:943239974 | Support escalation: search is intermittently failing for EU users. Triage and route per org process.", "seed": 943239974, "difficulty": "medium"}
+{"prompt": "SEED:240251661 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 240251661, "difficulty": "medium"}
+{"prompt": "SEED:137869475 | Support escalation: api-gateway is intermittently failing for EU users. Triage and route per org process.", "seed": 137869475, "difficulty": "medium"}
+{"prompt": "SEED:1722989659 | Monitoring alert: elevated error rate on search. File a bug ticket and page the right team.", "seed": 1722989659, "difficulty": "medium"}
+{"prompt": "SEED:284277889 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 284277889, "difficulty": "medium"}
+{"prompt": "SEED:1351531223 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 1351531223, "difficulty": "medium"}
+{"prompt": "SEED:2144181937 | Monitoring alert: elevated error rate on checkout. File a bug ticket and page the right team.", "seed": 2144181937, "difficulty": "hard"}
+{"prompt": "SEED:1970753705 | Monitoring alert: elevated error rate on auth. File a bug ticket and page the right team.", "seed": 1970753705, "difficulty": "medium"}
+{"prompt": "SEED:1137651678 | Support escalation: payments is intermittently failing for EU users. Triage and route per org process.", "seed": 1137651678, "difficulty": "medium"}
+{"prompt": "SEED:1059257080 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 1059257080, "difficulty": "medium"}
+{"prompt": "SEED:1840109255 | Support escalation: auth is intermittently failing for EU users. Triage and route per org process.", "seed": 1840109255, "difficulty": "hard"}
+{"prompt": "SEED:1554762903 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 1554762903, "difficulty": "medium"}
+{"prompt": "SEED:594130308 | Support escalation: checkout is intermittently failing for EU users. Triage and route per org process.", "seed": 594130308, "difficulty": "hard"}
+{"prompt": "SEED:390452952 | Support escalation: inventory is intermittently failing for EU users. Triage and route per org process.", "seed": 390452952, "difficulty": "easy"}
+{"prompt": "SEED:470939445 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 470939445, "difficulty": "medium"}
+{"prompt": "SEED:687117441 | Support escalation: notifications is intermittently failing for EU users. Triage and route per org process.", "seed": 687117441, "difficulty": "hard"}
+{"prompt": "SEED:272849424 | A user reported that the search is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 272849424, "difficulty": "hard"}
+{"prompt": "SEED:1639042338 | Support escalation: checkout is intermittently failing for EU users. Triage and route per org process.", "seed": 1639042338, "difficulty": "hard"}
+{"prompt": "SEED:1079815404 | Support escalation: search is intermittently failing for EU users. Triage and route per org process.", "seed": 1079815404, "difficulty": "easy"}
+{"prompt": "SEED:491995979 | Monitoring alert: elevated error rate on auth. File a bug ticket and page the right team.", "seed": 491995979, "difficulty": "medium"}
+{"prompt": "SEED:1461042543 | A user reported that the inventory is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1461042543, "difficulty": "easy"}
+{"prompt": "SEED:1260573448 | Support escalation: auth is intermittently failing for EU users. Triage and route per org process.", "seed": 1260573448, "difficulty": "hard"}
+{"prompt": "SEED:679282378 | Monitoring alert: elevated error rate on search. File a bug ticket and page the right team.", "seed": 679282378, "difficulty": "hard"}
+{"prompt": "SEED:13938521 | A user reported that the auth is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 13938521, "difficulty": "medium"}
+{"prompt": "SEED:767303988 | Support escalation: api-gateway is intermittently failing for EU users. Triage and route per org process.", "seed": 767303988, "difficulty": "easy"}
+{"prompt": "SEED:1281810600 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1281810600, "difficulty": "medium"}
+{"prompt": "SEED:656439677 | Monitoring alert: elevated error rate on checkout. File a bug ticket and page the right team.", "seed": 656439677, "difficulty": "medium"}
+{"prompt": "SEED:693847829 | Monitoring alert: elevated error rate on auth. File a bug ticket and page the right team.", "seed": 693847829, "difficulty": "easy"}
+{"prompt": "SEED:1392239675 | A user reported that the notifications is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1392239675, "difficulty": "hard"}
+{"prompt": "SEED:83651970 | A user reported that the search is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 83651970, "difficulty": "easy"}
+{"prompt": "SEED:1558990517 | Support escalation: checkout is intermittently failing for EU users. Triage and route per org process.", "seed": 1558990517, "difficulty": "medium"}
+{"prompt": "SEED:1028439863 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 1028439863, "difficulty": "easy"}
+{"prompt": "SEED:1034535593 | Monitoring alert: elevated error rate on notifications. File a bug ticket and page the right team.", "seed": 1034535593, "difficulty": "easy"}
+{"prompt": "SEED:367878761 | Monitoring alert: elevated error rate on auth. File a bug ticket and page the right team.", "seed": 367878761, "difficulty": "hard"}
+{"prompt": "SEED:297265480 | A user reported that the notifications is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 297265480, "difficulty": "medium"}
+{"prompt": "SEED:551437149 | Monitoring alert: elevated error rate on notifications. File a bug ticket and page the right team.", "seed": 551437149, "difficulty": "hard"}
+{"prompt": "SEED:709214891 | A user reported that the checkout is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 709214891, "difficulty": "medium"}
+{"prompt": "SEED:1817363615 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 1817363615, "difficulty": "medium"}
+{"prompt": "SEED:863937247 | A user reported that the inventory is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 863937247, "difficulty": "medium"}
+{"prompt": "SEED:1713658874 | Support escalation: auth is intermittently failing for EU users. Triage and route per org process.", "seed": 1713658874, "difficulty": "medium"}
+{"prompt": "SEED:1881625505 | A user reported that the notifications is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1881625505, "difficulty": "hard"}
+{"prompt": "SEED:519709079 | Monitoring alert: elevated error rate on notifications. File a bug ticket and page the right team.", "seed": 519709079, "difficulty": "medium"}
+{"prompt": "SEED:965067727 | Monitoring alert: elevated error rate on inventory. File a bug ticket and page the right team.", "seed": 965067727, "difficulty": "easy"}
+{"prompt": "SEED:1452066459 | A user reported that the notifications is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1452066459, "difficulty": "easy"}
+{"prompt": "SEED:988335251 | A user reported that the auth is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 988335251, "difficulty": "medium"}
+{"prompt": "SEED:30884438 | Monitoring alert: elevated error rate on auth. File a bug ticket and page the right team.", "seed": 30884438, "difficulty": "easy"}
+{"prompt": "SEED:252860896 | Support escalation: api-gateway is intermittently failing for EU users. Triage and route per org process.", "seed": 252860896, "difficulty": "medium"}
+{"prompt": "SEED:289482182 | A user reported that the auth is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 289482182, "difficulty": "easy"}
+{"prompt": "SEED:1419179580 | Support escalation: payments is intermittently failing for EU users. Triage and route per org process.", "seed": 1419179580, "difficulty": "easy"}
+{"prompt": "SEED:1022222125 | Support escalation: checkout is intermittently failing for EU users. Triage and route per org process.", "seed": 1022222125, "difficulty": "medium"}
+{"prompt": "SEED:2084839399 | Monitoring alert: elevated error rate on auth. File a bug ticket and page the right team.", "seed": 2084839399, "difficulty": "medium"}
+{"prompt": "SEED:568275055 | Support escalation: notifications is intermittently failing for EU users. Triage and route per org process.", "seed": 568275055, "difficulty": "hard"}
+{"prompt": "SEED:1043665192 | Monitoring alert: elevated error rate on inventory. File a bug ticket and page the right team.", "seed": 1043665192, "difficulty": "hard"}
+{"prompt": "SEED:1748309234 | A user reported that the search is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1748309234, "difficulty": "medium"}
+{"prompt": "SEED:405126228 | A user reported that the inventory is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 405126228, "difficulty": "easy"}
+{"prompt": "SEED:1851350739 | Monitoring alert: elevated error rate on checkout. File a bug ticket and page the right team.", "seed": 1851350739, "difficulty": "medium"}
+{"prompt": "SEED:1819256337 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1819256337, "difficulty": "hard"}
+{"prompt": "SEED:2005855667 | Support escalation: inventory is intermittently failing for EU users. Triage and route per org process.", "seed": 2005855667, "difficulty": "easy"}
+{"prompt": "SEED:422701550 | A user reported that the auth is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 422701550, "difficulty": "easy"}
+{"prompt": "SEED:1729245242 | A user reported that the auth is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1729245242, "difficulty": "medium"}
+{"prompt": "SEED:469306919 | Monitoring alert: elevated error rate on inventory. File a bug ticket and page the right team.", "seed": 469306919, "difficulty": "medium"}
+{"prompt": "SEED:822873088 | Monitoring alert: elevated error rate on inventory. File a bug ticket and page the right team.", "seed": 822873088, "difficulty": "medium"}
+{"prompt": "SEED:1926780541 | Support escalation: checkout is intermittently failing for EU users. Triage and route per org process.", "seed": 1926780541, "difficulty": "medium"}
+{"prompt": "SEED:1811967841 | A user reported that the api-gateway is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1811967841, "difficulty": "medium"}
+{"prompt": "SEED:1196342297 | Support escalation: payments is intermittently failing for EU users. Triage and route per org process.", "seed": 1196342297, "difficulty": "hard"}
+{"prompt": "SEED:1072910527 | Monitoring alert: elevated error rate on api-gateway. File a bug ticket and page the right team.", "seed": 1072910527, "difficulty": "easy"}
+{"prompt": "SEED:1903232052 | Support escalation: inventory is intermittently failing for EU users. Triage and route per org process.", "seed": 1903232052, "difficulty": "easy"}
+{"prompt": "SEED:217275224 | A user reported that the inventory is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 217275224, "difficulty": "easy"}
+{"prompt": "SEED:400560567 | Monitoring alert: elevated error rate on checkout. File a bug ticket and page the right team.", "seed": 400560567, "difficulty": "medium"}
+{"prompt": "SEED:714300770 | A user reported that the notifications is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 714300770, "difficulty": "hard"}
+{"prompt": "SEED:2085812759 | Support escalation: notifications is intermittently failing for EU users. Triage and route per org process.", "seed": 2085812759, "difficulty": "hard"}
+{"prompt": "SEED:918037633 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 918037633, "difficulty": "hard"}
+{"prompt": "SEED:251837136 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 251837136, "difficulty": "medium"}
+{"prompt": "SEED:1627677155 | Support escalation: api-gateway is intermittently failing for EU users. Triage and route per org process.", "seed": 1627677155, "difficulty": "easy"}
+{"prompt": "SEED:1676848676 | Monitoring alert: elevated error rate on inventory. File a bug ticket and page the right team.", "seed": 1676848676, "difficulty": "medium"}
+{"prompt": "SEED:1954246074 | A user reported that the api-gateway is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1954246074, "difficulty": "medium"}
+{"prompt": "SEED:1816803306 | Monitoring alert: elevated error rate on search. File a bug ticket and page the right team.", "seed": 1816803306, "difficulty": "hard"}
+{"prompt": "SEED:664847319 | A user reported that the auth is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 664847319, "difficulty": "medium"}
+{"prompt": "SEED:1274350418 | Support escalation: search is intermittently failing for EU users. Triage and route per org process.", "seed": 1274350418, "difficulty": "medium"}
+{"prompt": "SEED:251183830 | Support escalation: api-gateway is intermittently failing for EU users. Triage and route per org process.", "seed": 251183830, "difficulty": "easy"}
+{"prompt": "SEED:1346922426 | Monitoring alert: elevated error rate on checkout. File a bug ticket and page the right team.", "seed": 1346922426, "difficulty": "easy"}
+{"prompt": "SEED:215359682 | Monitoring alert: elevated error rate on api-gateway. File a bug ticket and page the right team.", "seed": 215359682, "difficulty": "hard"}
+{"prompt": "SEED:676168421 | Support escalation: notifications is intermittently failing for EU users. Triage and route per org process.", "seed": 676168421, "difficulty": "easy"}
+{"prompt": "SEED:344076115 | A user reported that the notifications is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 344076115, "difficulty": "medium"}
+{"prompt": "SEED:294296873 | Support escalation: payments is intermittently failing for EU users. Triage and route per org process.", "seed": 294296873, "difficulty": "easy"}
+{"prompt": "SEED:1010193046 | Monitoring alert: elevated error rate on api-gateway. File a bug ticket and page the right team.", "seed": 1010193046, "difficulty": "hard"}
+{"prompt": "SEED:514909066 | A user reported that the search is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 514909066, "difficulty": "medium"}
+{"prompt": "SEED:170689150 | Support escalation: notifications is intermittently failing for EU users. Triage and route per org process.", "seed": 170689150, "difficulty": "easy"}
+{"prompt": "SEED:1800557283 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 1800557283, "difficulty": "medium"}
+{"prompt": "SEED:1119980130 | A user reported that the auth is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1119980130, "difficulty": "medium"}
+{"prompt": "SEED:1349409434 | A user reported that the checkout is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1349409434, "difficulty": "medium"}
+{"prompt": "SEED:1140805649 | Support escalation: checkout is intermittently failing for EU users. Triage and route per org process.", "seed": 1140805649, "difficulty": "hard"}
+{"prompt": "SEED:562117925 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 562117925, "difficulty": "medium"}
+{"prompt": "SEED:1963764597 | A user reported that the api-gateway is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1963764597, "difficulty": "medium"}
+{"prompt": "SEED:311570307 | Monitoring alert: elevated error rate on inventory. File a bug ticket and page the right team.", "seed": 311570307, "difficulty": "easy"}
+{"prompt": "SEED:1968321319 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1968321319, "difficulty": "easy"}
+{"prompt": "SEED:314652384 | A user reported that the checkout is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 314652384, "difficulty": "medium"}
+{"prompt": "SEED:1139027119 | Monitoring alert: elevated error rate on search. File a bug ticket and page the right team.", "seed": 1139027119, "difficulty": "medium"}
+{"prompt": "SEED:1498981529 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1498981529, "difficulty": "easy"}
+{"prompt": "SEED:1049193572 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1049193572, "difficulty": "medium"}
+{"prompt": "SEED:1224011538 | Support escalation: notifications is intermittently failing for EU users. Triage and route per org process.", "seed": 1224011538, "difficulty": "medium"}
+{"prompt": "SEED:1881988384 | A user reported that the payments is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1881988384, "difficulty": "medium"}
+{"prompt": "SEED:33599984 | Support escalation: search is intermittently failing for EU users. Triage and route per org process.", "seed": 33599984, "difficulty": "medium"}
+{"prompt": "SEED:444900634 | A user reported that the checkout is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 444900634, "difficulty": "medium"}
+{"prompt": "SEED:1135872495 | A user reported that the api-gateway is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 1135872495, "difficulty": "easy"}
+{"prompt": "SEED:459716009 | A user reported that the checkout is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 459716009, "difficulty": "medium"}
+{"prompt": "SEED:1169726681 | Monitoring alert: elevated error rate on search. File a bug ticket and page the right team.", "seed": 1169726681, "difficulty": "medium"}
+{"prompt": "SEED:904647469 | Support escalation: inventory is intermittently failing for EU users. Triage and route per org process.", "seed": 904647469, "difficulty": "medium"}
+{"prompt": "SEED:874443787 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 874443787, "difficulty": "medium"}
+{"prompt": "SEED:2098228320 | Monitoring alert: elevated error rate on inventory. File a bug ticket and page the right team.", "seed": 2098228320, "difficulty": "medium"}
+{"prompt": "SEED:218179599 | A user reported that the inventory is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 218179599, "difficulty": "easy"}
+{"prompt": "SEED:1819244083 | Monitoring alert: elevated error rate on api-gateway. File a bug ticket and page the right team.", "seed": 1819244083, "difficulty": "medium"}
+{"prompt": "SEED:189351213 | Support escalation: inventory is intermittently failing for EU users. Triage and route per org process.", "seed": 189351213, "difficulty": "easy"}
+{"prompt": "SEED:1432614422 | Monitoring alert: elevated error rate on search. File a bug ticket and page the right team.", "seed": 1432614422, "difficulty": "medium"}
+{"prompt": "SEED:1125089309 | Monitoring alert: elevated error rate on payments. File a bug ticket and page the right team.", "seed": 1125089309, "difficulty": "medium"}
+{"prompt": "SEED:1897669065 | Support escalation: inventory is intermittently failing for EU users. Triage and route per org process.", "seed": 1897669065, "difficulty": "hard"}
+{"prompt": "SEED:41531046 | A user reported that the inventory is returning 500 errors. The issue started 20 minutes ago. Triage this bug: file a ticket with the correct label and priority, assign it to the right team, and notify the correct oncall channel.", "seed": 41531046, "difficulty": "easy"}