docs: rewrite README with full system documentation

Covers: core RL insight (procedural org drift), all 4 tasks, scoring
breakdown, action space, org world generation, SFT→GRPO pipeline,
repo layout, quick start, and key design decisions (why each fix was
needed). Keeps HF Space frontmatter.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

README.md

@@ -11,44 +11,249 @@ tags:
   - reinforcement-learning
 ---
 
-# PM-Ops: Procedural Project-Management RL Environment
+# PM-Ops: Procedural PM Operations RL Environment
 
-A multi-app simulated environment (Ticketing, Codebase, Chat) featuring **procedural
-organizational drift**, designed for reinforcement learning training.
+A simulated software-org environment — Ticketing, Chat, Codebase — designed to train LLM agents that learn *organizational conventions* rather than memorize fixed answers.
 
-## What makes it RL-shaped
+**Hosted env:** `https://huggingface.co/spaces/TheCrustaceans/Pm-ops`  
+**Trained checkpoint:** `Saurav1/pm-ops-grpo-Qwen3-1.7B-triage-v4`  
+**Stack:** Qwen3-1.7B · Unsloth · TRL 0.22.2 · SFT → GRPO
 
-Every episode samples a fresh OrgConfig -- different label names, priority schemes,
-team-to-service mappings, and oncall channel names. Frontier LLMs fail because they
-rely on memorized conventions; a trained agent learns to read the runbook first.
+---
+
+## The Core Idea
+
+Every episode samples a fresh `OrgConfig` — different label names, priority schemes, team-to-service mappings, and oncall channels. An agent that hardcodes `label="bug"` or `priority="P1"` will fail most episodes. A trained agent learns to **read the runbook first**, then use exactly what it says.
+
+```
+Episode A                        Episode B
+──────────────────────────────   ──────────────────────────────
+labels:    issue, perf, docs     labels:    defect, vuln, guide
+priority:  critical/high/low     priority:  P0/P1/P2/P3
+team:      payments → data        team:      payments → backend
+channel:   #data-alerts           channel:   #backend-oncall
+```
+
+The same hardcoded action scores 1.0 in Episode A and 0.0 in Episode B. This is the learning signal.
+
+---
 
 ## Tasks
 
-| Task | Brief | Max Steps |
-|------|-------|-----------|
-| triage | Bug report -- correct label, priority, team, channel | 25 |
-| incident_routing | Alert -- identify owner via commits, page oncall | 25 |
-| release_notes | Compile + post release notes in org style | 40 |
-| dep_update | Coordinate dependency update across all owning teams | 40 |
+| Task | What the agent must do | Max Steps |
+|---|---|---|
+| **triage** | File a bug ticket with the correct org label + priority, assign to the owning team, page the oncall channel | 25 |
+| **incident_routing** | Identify the commit that broke a service, page the right oncall channel, optionally ticket it | 25 |
+| **release_notes** | Compile resolved tickets into release notes in the org's style (terse / verbose / bulleted) and post them | 40 |
+| **dep_update** | Coordinate a dependency update across all teams that own affected services | 40 |
+
+---
+
+## Scoring
+
+Reward is **delayed** to episode end via a deterministic Python verifier — no LLM-as-judge.
+
+### Triage (example)
+```
++0.25  ticket created
++0.20  correct label (must match label_taxonomy)
++0.20  correct priority (must match priority_levels)
++0.20  assigned to correct team (via team_map)
++0.10  message posted to correct oncall channel
++0.05  message body ≥ 20 chars
+─0.10  per duplicate ticket
+─0.05  per message posted to a noise channel
+```
+
+### GRPO Training Reward (runbook-compliance)
+
+During RL training we add a **dense shaped reward** on top of the env score:
+
+| Signal | Reward |
+|---|---|
+| Called `meta.read_runbook` | +0.10 |
+| Ticket label in `label_taxonomy` | +0.20 / −0.10 |
+| Ticket priority in `priority_levels` | +0.15 / −0.10 |
+| Team in `team_map` | +0.20 / −0.10 |
+| Posted to `oncall_channels` | +0.25 / −0.10 per wrong channel |
+| Env grader score | ×0.10 bonus |
+
+Reward naturally varies per seed because every org has different valid values — the model can't memorize answers.
+
+---
+
+## Action Space
+
+The agent emits one JSON block per step:
+
+```json
+{"action_type": "ticketing.create_ticket", "args": {"summary": "...", "label": "issue", "priority": "high"}}
+```
+
+| Namespace | Actions |
+|---|---|
+| `meta` | `read_runbook` · `finish` · `noop` |
+| `ticketing` | `create_ticket` · `update_ticket` · `get_ticket` · `list_tickets` · `assign_ticket` · `comment_ticket` · `transition_ticket` |
+| `chat` | `post_message` · `read_channel` · `list_channels` · `search` |
+| `codebase` | `list_commits` · `get_commit` · `list_prs` |
+
+The optimal triage sequence takes **5 steps**:
+```
+meta.read_runbook → ticketing.create_ticket → ticketing.assign_ticket → chat.post_message → meta.finish
+```
+
+---
+
+## Procedural World Generation
+
+Each episode is fully seeded — same seed always produces the same org + scenario.
+
+```python
+# org_generator.py
+rng = random.Random(seed)
+difficulty = rng.choice(["easy", "medium", "medium", "hard"])
+label_taxonomy  = {k: rng.choice(variants[k]) for k in sampled_keys}
+priority_levels = rng.choice([["P0","P1","P2","P3"], ["critical","high","medium","low"], ...])
+team_map        = {service: team for service, team in zip(services, teams)}
+oncall_channels = {svc: f"#{team_map[svc]}{rng.choice(suffixes)}" for svc in services}
+```
+
+**Difficulty** controls the number of services (2 / 3 / 5), labels (3 / 4 / 5), noise channels (1 / 2 / 3), and required ticket fields.
+
+---
+
+## Training Pipeline
 
-## Action space
+### Phase 1 — SFT Warmup (~15 min on A100)
 
-The agent calls actions like:
+Run the deterministic `baseline_agent` for N episodes and collect every `(observation, action)` pair as a supervised example. Without this, the model outputs freeform text → all GRPO advantages are zero.
 
-  meta.read_runbook       -- learn this org's conventions
-  ticketing.create_ticket -- file a ticket with label + priority
-  ticketing.assign_ticket -- assign to correct team
-  chat.post_message       -- notify the right oncall channel
-  meta.finish             -- end episode, trigger scoring
+```
+baseline_agent: read_runbook → create_ticket → assign_ticket → list_channels → post_message → finish
+SFT target: prompt + ```json\n{"action_type": ..., "args": {...}}\n``` + EOS
+```
 
-## Reward
+After 2 epochs, the model reliably emits `\`\`\`json ... \`\`\`` blocks.
 
-Delayed to episode end. Deterministic Python verifier -- no LLM-as-judge.
-Partial credit for partial correctness.
+### Phase 2 — GRPO (~90 min on A100)
+
+GRPO learns by comparing rewards across N generations of the same prompt. Each generation gets a different env seed (`gen_slot` offset), so they explore distinct org configs and produce varied rewards — giving GRPO a real gradient signal.
+
+```
+prompt (same seed) → gen_slot=0 → org_A → reward=0.45
+                  → gen_slot=1 → org_B → reward=0.80
+                  → gen_slot=2 → org_C → reward=0.10
+advantage = reward - mean(group)  →  non-zero gradient
+```
+
+**Critical: the dataset is pre-filtered.** `dataset.py` simulates the env's RNG to only include seeds where `env.reset(seed)` will run a *triage* task — eliminating the 75% of seeds that would produce release_notes or dep_update episodes and always score 0.
+
+---
+
+## Repository Layout
+
+```
+├── server/
+│   ├── pm_ops_environment.py    # OpenEnv Environment — reset/step/grade
+│   ├── apps/
+│   │   ├── ticketing.py         # Jira-like app (create, assign, transition tickets)
+│   │   ├── chat.py              # Slack-like app (post, read, search channels)
+│   │   └── codebase.py          # GitHub-like app (commits, PRs)
+│   ├── tasks/
+│   │   ├── triage_task.py       # Grader: label + priority + team + channel
+│   │   ├── incident_routing_task.py
+│   │   ├── release_notes_task.py
+│   │   └── dep_update_task.py
+│   └── world/
+│       ├── org_generator.py     # Procedural OrgConfig from seed
+│       └── scenario_gen.py      # Task brief + expected solution from seed
+│
+├── training/
+│   ├── train_v4.ipynb           # ← run this (current clean notebook)
+│   ├── dataset.py               # Pre-filtered triage dataset (seed-matched to env)
+│   ├── rollout.py               # Multi-turn episode runner + JSON extractor
+│   ├── rewards.py               # compute_rollout_reward (runbook-compliance)
+│   ├── prompts.py               # System prompt + observation formatter
+│   └── pm_ops_trainer.py        # PMOpsGRPOTrainer — reward cache injection
+│
+├── inference.py                 # Deterministic baseline_agent (heuristic)
+├── models.py                    # Pydantic types: PMOpsAction, PMOpsObservation
+├── client.py                    # OpenEnv client helpers
+└── Dockerfile                   # HF Space deployment
+```
+
+---
+
+## Quick Start
+
+### Run the env locally
+
+```bash
+pip install -e .
+uvicorn server.app:app --host 0.0.0.0 --port 8000
+```
+
+### Run the baseline agent
+
+```bash
+python inference.py
+# [END] success=True steps=6 score=0.75 ...
+```
+
+### Connect a custom agent
+
+```python
+from openenv.core import GenericEnvClient
+
+with GenericEnvClient(base_url="http://localhost:8000").sync() as env:
+    result   = env.reset(seed=42)
+    obs      = result.observation
+    result   = env.step({"action_type": "meta.read_runbook", "args": {}})
+    # ... agent loop ...
+    result   = env.step({"action_type": "meta.finish", "args": {}})
+    print(f"score = {result.reward:.3f}")
+```
+
+### Train (cloud — A100 or T4)
+
+Open `training/train_v4.ipynb` on your cloud runtime, run top to bottom. The notebook is self-contained: installs deps, clones the HF Space, runs SFT then GRPO, saves merged 16-bit weights to HF Hub.
+
+Key hyperparameters auto-adapt to GPU:
+
+| | A100 (40 GB) | T4 (16 GB) |
+|---|---|---|
+| `NUM_GEN` | 6 | 2 |
+| `GRAD_ACCUM` | 32 | 8 |
+| `N_SFT_EPISODES` | 120 | 50 |
+| `N_GRPO_EPISODES` | 150 | 30 |
+| `MAX_COMP_LEN` | 192 | 192 |
+
+---
+
+## Key Design Decisions
+
+**Why SFT before GRPO?**  
+GRPO computes `advantage = reward �� mean(group)`. If the model outputs garbage on every step, all rewards are ~0 and all advantages are 0 → zero gradient. SFT teaches the JSON format first so GRPO has something to improve.
+
+**Why runbook-compliance reward instead of env score?**  
+The env grader requires a precise 5-step sequence with *exact* field values (label, priority, team, channel). Early in training the model rarely completes the full sequence, so env score is 0 for nearly every episode — zero variance, zero GRPO signal. The compliance reward fires on individual correct actions and creates variance even from partial completions.
+
+**Why filter training seeds to triage-only?**  
+`PMOpsEnvironment.reset(seed)` draws task type uniformly from 4 options. Without filtering, 75% of training prompts describe a triage task but the env runs release_notes or dep_update → guaranteed env score = 0 regardless of what the model does. The dataset pre-simulates the env's RNG to only include seeds where `task_type == "triage"`.
+
+**Why `gen_slot` offsets?**  
+GRPO needs reward *variance across generations of the same prompt*. Without the offset, all N generations of the same prompt reset the env with the same seed → same org config → same correct answers → potentially same reward → zero advantage. `gen_slot=k` adds `k` to the seed so generation k explores a slightly different org.
+
+**Why `PMOpsGRPOTrainer`?**  
+TRL's GRPOTrainer builds `reward_kwargs` from dataset columns only. Our multi-turn rollout reward (computed inside `rollout_func`) never reaches `_calculate_rewards` through the normal kwargs path. `PMOpsGRPOTrainer` wraps `rollout_func` to cache rewards and injects them directly as a tensor, bypassing TRL's broken plumbing.
+
+---
 
-## Sponsor alignment
+## Sponsor Alignment
 
-- AI Labs: direct PM-in-the-loop positioning
-- Patronus: schema drift -- env tests whether agents track convention changes
-- Scale AI: enterprise workflow simulation
-- Meta RFC 004: delayed trajectory reward is the native reward mode
+| Sponsor | Relevance |
+|---|---|
+| **Meta RFC 004** | Delayed trajectory reward is the native reward mode |
+| **Patronus** | Schema drift — tests whether agents track per-org convention changes |
+| **Scale AI** | Enterprise workflow simulation with procedurally generated organizations |
+| **AI Labs** | Direct PM-in-the-loop positioning for agentic systems |