plan.md

THE DEFINITIVE FINAL DESIGN

Core Mechanic: Language-Sensitive Scripted Opponent

This is the one thing that makes everything work. The opponent's behavior is deterministic given a seed AND sensitive to the agent's language quality.

# Deterministic keyword detection — pure Python
COLLABORATIVE_SIGNALS = [
    "understand", "partnership", "mutual", "together", "value",
    "appreciate", "flexible", "work with", "long-term", "relationship"
]

AGGRESSIVE_SIGNALS = [
    "demand", "require", "final offer", "unacceptable", "must",
    "non-negotiable", "take it or leave", "bottom line", "ultimatum"
]

def update_rapport(current_rapport: float, agent_message: str) -> float:
    msg_lower = agent_message.lower()
    delta = 0.0
    delta += sum(0.08 for w in COLLABORATIVE_SIGNALS if w in msg_lower)
    delta -= sum(0.08 for w in AGGRESSIVE_SIGNALS if w in msg_lower)
    delta = max(-0.20, min(0.20, delta))  # cap per-round change
    return max(0.0, min(1.0, current_rapport + delta))

The rapport score directly modifies the opponent's concession rate:

• Rapport 0.8: opponent concedes 12% per round
• Rapport 0.5: opponent concedes 7% per round (neutral)
• Rapport 0.2: opponent concedes 3% per round (hardened)

A heuristic agent that outputs nothing or outputs aggressive language gets neutral/hostile opponent. An LLM that learns collaborative framing gets cooperative opponent. This is the LLM advantage.

The Three Tasks — Final, Locked

Task 1: single_issue (Easy)

Scenario: Renew software license. Price only.

Buyer target: $38,000
Seller opens: $52,000
Seller floor: $44,000
Pareto optimal: $43,000
Max rounds: 6
Persona: Cooperative (concedes 10% baseline, rapport-sensitive)

Calibration: A base LLM that simply offers reasonable prices without collaborative language scores ~0.38. A base LLM that naturally uses professional language scores ~0.52. Scores above 0.75 require learning to time concessions correctly.

Grader:

def grade_single_issue(final_price, deal_reached, rounds_taken):
    if not deal_reached:
        return 0.0

    # Value: how close to buyer target
    value = (44000 - final_price) / (44000 - 38000)
    value = max(0.0, min(1.0, value))

    # Efficiency: penalty grows sharply in late rounds
    efficiency = 1.0 - (rounds_taken / 6) ** 1.5 * 0.4
    efficiency = max(0.0, efficiency)

    return round(value * efficiency, 4)

Task 2: multi_issue (Medium)

Scenario: Enterprise software. Price + payment terms.

Issues: price ($40K-$58K) + payment_days (30-90)
Seller persona: Cash Flow Stressed
  → price_weight: 0.35 (somewhat cares)
  → payment_weight: 0.65 (cares much more)
Buyer weights: price 0.70, payment 0.30
Pareto insight: buyer should offer Net-30 to get lower price
Max rounds: 8

Why medium: Base LLM treats both issues equally, misses the trade opportunity. Score ~0.25. LLM that discovers seller cares about payment can bundle correctly. Score ~0.50.

Grader:

def grade_multi_issue(final_terms, deal_reached, rounds_taken):
    if not deal_reached:
        return 0.0

    # Buyer utility function
    price_score = (58000 - final_terms['price']) / (58000 - 40000)
    payment_score = (90 - final_terms['payment_days']) / (90 - 30)

    price_score = max(0.0, min(1.0, price_score))
    payment_score = max(0.0, min(1.0, payment_score))

    value = 0.70 * price_score + 0.30 * payment_score
    efficiency = 1.0 - (rounds_taken / 8) * 0.30

    return round(value * efficiency, 4)

Task 3: adversarial (Hard)

Scenario: Large contract. Price + payment + support hours.

Issues: price + payment_days + support_hours
Seller persona: Aggressive Anchor
  → Opens at ceiling on all issues
  → Hardens position if agent makes consecutive concessions
  → Rapport-sensitive but requires consistent collaborative framing
Adaptation: if agent concedes 2+ rounds in a row, seller increases floor by 3%
Max rounds: 10
Survival floor: deal at any terms scores minimum 0.15

Why hard: Agent must resist anchoring, break consecutive concession patterns, maintain collaborative tone under pressure. Base LLM score ~0.15. Strong LLM ~0.40.

Grader:

def grade_adversarial(final_terms, deal_reached, rounds_taken, consecutive_flag):
    if not deal_reached:
        return 0.0

    # Survival floor — completing deal always scores at least 0.15
    floor = 0.15

    price_score = (120000 - final_terms['price']) / (120000 - 80000)
    payment_score = (90 - final_terms['payment_days']) / (90 - 30)
    support_score = (final_terms['support_hours'] - 80) / (200 - 80)

    for s in [price_score, payment_score, support_score]:
        s = max(0.0, min(1.0, s))

    value = 0.40 * price_score + 0.35 * payment_score + 0.25 * support_score
    efficiency = 1.0 - (rounds_taken / 10) * 0.25

    # Penalty for consecutive concession pattern
    pattern_penalty = 0.1 if consecutive_flag else 0.0

    raw = (value * efficiency) - pattern_penalty
    return round(max(floor, raw), 4)

Score Calibration Table

Agent Type	single_issue	multi_issue	adversarial
Random/heuristic	0.15–0.25	0.08–0.15	0.03–0.10
Base LLM (no language)	0.35–0.45	0.20–0.30	0.12–0.20
Base LLM (natural language)	0.48–0.58	0.28–0.38	0.18–0.28
GRPO-trained LLM (goal)	0.68–0.78	0.55–0.65	0.45–0.55

This gives clear score spread at every level. Phase 2 will show meaningful differentiation.

---

THE CLAUDE CODE PROMPT

Paste this entire block into Claude Code:

---

Build ProcureRL: A Procurement Negotiation RL Environment

This is a complete OpenEnv-compliant environment. Build everything exactly as specified. No additions, no changes to the design.

---

Project Structure:

procure-rl/
├── procure_rl/
│   ├── __init__.py
│   ├── environment.py
│   ├── models.py
│   ├── opponent.py
│   ├── graders.py
│   └── scenarios.py
├── server/
│   └── app.py
├── inference.py
├── openenv.yaml
├── Dockerfile
├── requirements.txt
└── README.md

---

models.py — exact dataclasses:

from dataclasses import dataclass, field
from typing import Optional, List, Dict, Any

try:
    from openenv.core.env_server import Action, Observation, State
except ImportError:
    Action = object
    Observation = object
    State = object

@dataclass
class NegotiationAction(Action):
    move_type: str  # make_offer | accept | reject | bundle
    terms: Dict[str, Any]  # {price: 44000, payment_days: 45, support_hours: 120}
    message: str = ""  # natural language — affects opponent rapport

@dataclass
class NegotiationObservation(Observation):
    task_id: str
    round_number: int
    max_rounds: int
    supplier_message: str
    current_offer: Dict[str, Any]
    last_4_exchanges: List[Dict]  # capped at 4 for token efficiency
    buyer_constraints: Dict[str, Any]  # buyer's targets and limits
    rapport_hint: str  # "positive" | "neutral" | "negative" — visible to agent
    done: bool

@dataclass
class NegotiationState(State):
    task_id: str = ""
    episode_id: str = ""
    round_number: int = 0
    rapport_score: float = 0.5
    consecutive_concessions: int = 0
    deal_reached: bool = False
    final_terms: Optional[Dict] = None
    cumulative_reward: float = 0.0

---

opponent.py — ScriptedPersonaOpponent:

import random
from dataclasses import dataclass, field
from typing import Dict, Tuple

COLLABORATIVE_SIGNALS = [
    "understand", "partnership", "mutual", "together", "value",
    "appreciate", "flexible", "work with", "long-term", "relationship",
    "reasonable", "fair", "both", "solution"
]

AGGRESSIVE_SIGNALS = [
    "demand", "require", "final offer", "unacceptable", "must",
    "non-negotiable", "take it or leave", "bottom line", "ultimatum",
    "insist", "refuse", "absolutely not"
]

PERSONA_TEMPLATES = {
    "cooperative": {
        "opening": [
            "Thanks for reaching out. Our standard pricing for this package is ${target}. Happy to discuss.",
            "We value your interest. We're pricing this at ${target} based on current market rates.",
        ],
        "counter": [
            "I appreciate you working with us. Based on our costs, ${counter} is where we can be.",
            "Thank you for your offer. We can move to ${counter} given our margin requirements.",
        ],
        "near_close": [
            "I think we're close. If you can do ${close}, I can get this approved today.",
            "We're almost there. ${close} works for our team. Shall we finalize?"
        ],
        "accept": "That works for us. Let's move forward at those terms.",
        "reject": "That's below what we can accept, but we want to make this work."
    },
    "cash_flow_stressed": {
        "opening": [
            "Our pricing is ${target}. I should mention — payment timing is particularly important to us this quarter.",
            "We're at ${target}. Between us, our finance team has specific requirements around cash flow timing.",
        ],
        "counter": [
            "We can move on price if payment terms work for you. ${counter} with your payment preference?",
            "Price flexibility depends on receivables timing for us. ${counter} if we can discuss payment terms.",
        ],
        "near_close": [
            "If you can do Net-30 on payment, we can get to ${close} on price.",
            "Payment timing is our real constraint. ${close} with faster payment terms?"
        ],
        "accept": "Agreed. The payment structure works for our cash flow needs.",
        "reject": "The price is tight but we could explore it if payment terms align."
    },
    "aggressive_anchor": {
        "opening": [
            "Our price is ${target}. This reflects our full service quality and market position.",
            "We're firm at ${target}. This is based on our cost structure and service level.",
        ],
        "counter": [
            "We can go to ${counter}. That's already a significant concession from our position.",
            "${counter} is our revised position. We're not in a position to move much further.",
        ],
        "hardening": [
            "We've already moved considerably. ${floor} is our absolute position.",
            "I need to be direct — we're at ${floor} and that's where we'll stay.",
        ],
        "near_close": [
            "Final position: ${close}. We need a decision today.",
            "${close} is where we are. This is our best and final offer."
        ],
        "accept": "Accepted.",
        "reject": "That doesn't work. Come back with a serious offer."
    }
}

class ScriptedPersonaOpponent:
    def __init__(self, task_id: str, seed: int, persona: str):
        self.rng = random.Random(seed)
        self.task_id = task_id
        self.persona = persona
        self.templates = PERSONA_TEMPLATES[persona]

        # Sampled reservation values — never revealed to agent
        if task_id == "single_issue":
            self.price_floor = self.rng.uniform(42000, 46000)
            self.price_target = self.price_floor * self.rng.uniform(1.28, 1.38)
        elif task_id == "multi_issue":
            self.price_floor = self.rng.uniform(40000, 46000)
            self.price_target = self.price_floor * self.rng.uniform(1.25, 1.35)
            self.payment_preference = self.rng.choice([30, 45, 60])  # preferred days
        elif task_id == "adversarial":
            self.price_floor = self.rng.uniform(85000, 95000)
            self.price_target = self.price_floor * self.rng.uniform(1.30, 1.40)

        self.rapport = 0.5
        self.concession_count = 0
        self.current_position = self.price_target

    def update_rapport(self, agent_message: str) -> None:
        msg_lower = agent_message.lower()
        delta = 0.0
        delta += sum(0.08 for w in COLLABORATIVE_SIGNALS if w in msg_lower)
        delta -= sum(0.08 for w in AGGRESSIVE_SIGNALS if w in msg_lower)
        delta = max(-0.20, min(0.20, delta))
        self.rapport = max(0.0, min(1.0, self.rapport + delta))

    def get_concession_rate(self) -> float:
        # Base rate by persona
        base_rates = {
            "cooperative": 0.10,
            "cash_flow_stressed": 0.07,
            "aggressive_anchor": 0.04
        }
        base = base_rates[self.persona]
        # Rapport modifier: +/- 50% of base rate
        modifier = (self.rapport - 0.5) * base
        return max(0.01, base + modifier)

    def respond(self, agent_message: str, agent_terms: Dict,
                round_number: int, consecutive_concessions: int) -> Tuple[str, Dict]:

        self.update_rapport(agent_message)
        self.concession_count += 1

        agent_price = agent_terms.get('price', 0)

        # Check if we should accept
        if agent_price >= self.price_floor and self._acceptance_condition(agent_terms):
            return self.templates["accept"], {**agent_terms, "_accepted": True}

        # Compute counter position
        concession = self.get_concession_rate()

        # Aggressive anchor hardens if consecutive concessions detected
        if self.persona == "aggressive_anchor" and consecutive_concessions >= 2:
            concession = concession * 0.4  # barely moves
            template_key = "hardening"
        elif round_number >= self._max_rounds() * 0.7:
            template_key = "near_close"
        else:
            template_key = "counter"

        new_position = self.current_position * (1 - concession)
        new_position = max(self.price_floor, new_position)
        self.current_position = new_position

        # Select template
        templates_for_key = self.templates.get(template_key, self.templates["counter"])
        template = self.rng.choice(templates_for_key)
        message = template.replace("${counter}", f"${new_position:,.0f}")
        message = message.replace("${floor}", f"${self.price_floor:,.0f}")
        message = message.replace("${close}", f"${new_position:,.0f}")

        counter_terms = dict(agent_terms)
        counter_terms['price'] = round(new_position, 2)

        # Cash flow stressed adds payment commentary
        if self.persona == "cash_flow_stressed" and 'payment_days' in agent_terms:
            if agent_terms['payment_days'] > 60:
                message += " Though I'll need to flag the payment timing to our finance team."

        return message, counter_terms

    def _acceptance_condition(self, terms: Dict) -> bool:
        if self.persona == "cash_flow_stressed":
            payment_ok = terms.get('payment_days', 60) <= 45
            return payment_ok
        return True

    def _max_rounds(self) -> int:
        return {"single_issue": 6, "multi_issue": 8, "adversarial": 10}[self.task_id]

    def get_opening_message(self) -> Tuple[str, Dict]:
        template = self.rng.choice(self.templates["opening"])
        message = template.replace("${target}", f"${self.price_target:,.0f}")
        terms = {"price": round(self.price_target, 2)}
        if self.task_id in ["multi_issue", "adversarial"]:
            terms["payment_days"] = 90
        if self.task_id == "adversarial":
            terms["support_hours"] = 80
        return message, terms

---

graders.py — pure Python, zero LLM calls:

from typing import Dict, Optional

def grade_single_issue(
    final_terms: Dict,
    deal_reached: bool,
    rounds_taken: int,
    max_rounds: int = 6
) -> float:
    if not deal_reached:
        return 0.0

    final_price = final_terms.get('price', 99999)

    # Buyer target: $38K, seller floor: ~$44K
    BUYER_TARGET = 38000
    SELLER_FLOOR = 44000

    value = (SELLER_FLOOR - final_price) / (SELLER_FLOOR - BUYER_TARGET)
    value = max(0.0, min(1.0, value))

    # Efficiency penalty grows sharply in late rounds
    efficiency = 1.0 - (rounds_taken / max_rounds) ** 1.5 * 0.4
    efficiency = max(0.1, efficiency)

    return round(value * efficiency, 4)


def grade_multi_issue(
    final_terms: Dict,
    deal_reached: bool,
    rounds_taken: int,
    max_rounds: int = 8
) -> float:
    if not deal_reached:
        return 0.0

    final_price = final_terms.get('price', 99999)
    payment_days = final_terms.get('payment_days', 90)

    # Price component (buyer cares 70%)
    PRICE_WORST = 58000
    PRICE_TARGET = 40000
    price_score = (PRICE_WORST - final_price) / (PRICE_WORST - PRICE_TARGET)
    price_score = max(0.0, min(1.0, price_score))

    # Payment component (buyer cares 30%)
    PAYMENT_WORST = 90
    PAYMENT_TARGET = 30
    payment_score = (PAYMENT_WORST - payment_days) / (PAYMENT_WORST - PAYMENT_TARGET)
    payment_score = max(0.0, min(1.0, payment_score))

    value = 0.70 * price_score + 0.30 * payment_score
    efficiency = 1.0 - (rounds_taken / max_rounds) * 0.30
    efficiency = max(0.1, efficiency)

    return round(value * efficiency, 4)


def grade_adversarial(
    final_terms: Dict,
    deal_reached: bool,
    rounds_taken: int,
    consecutive_concessions_flag: bool,
    max_rounds: int = 10
) -> float:
    if not deal_reached:
        return 0.0

    SURVIVAL_FLOOR = 0.15

    final_price = final_terms.get('price', 999999)
    payment_days = final_terms.get('payment_days', 90)
    support_hours = final_terms.get('support_hours', 80)

    # Price (buyer weight 40%)
    PRICE_WORST = 120000
    PRICE_TARGET = 80000
    price_score = (PRICE_WORST - final_price) / (PRICE_WORST - PRICE_TARGET)
    price_score = max(0.0, min(1.0, price_score))

    # Payment (buyer weight 35%)
    payment_score = (90 - payment_days) / (90 - 30)
    payment_score = max(0.0, min(1.0, payment_score))

    # Support hours (buyer weight 25%)
    support_score = (support_hours - 80) / (200 - 80)
    support_score = max(0.0, min(1.0, support_score))

    value = 0.40 * price_score + 0.35 * payment_score + 0.25 * support_score
    efficiency = 1.0 - (rounds_taken / max_rounds) * 0.25
    efficiency = max(0.1, efficiency)

    # Penalty for being exploited by consecutive concession pattern
    pattern_penalty = 0.10 if consecutive_concessions_flag else 0.0

    raw = (value * efficiency) - pattern_penalty
    return round(max(SURVIVAL_FLOOR, raw), 4)


def grade(task_id: str, final_terms: Dict, deal_reached: bool,
          rounds_taken: int, **kwargs) -> float:
    if task_id == "single_issue":
        return grade_single_issue(final_terms, deal_reached, rounds_taken)
    elif task_id == "multi_issue":
        return grade_multi_issue(final_terms, deal_reached, rounds_taken)
    elif task_id == "adversarial":
        return grade_adversarial(
            final_terms, deal_reached, rounds_taken,
            kwargs.get("consecutive_concessions_flag", False)
        )
    raise ValueError(f"Unknown task: {task_id}")

---

environment.py:

import uuid
from typing import Optional
from procure_rl.models import NegotiationAction, NegotiationObservation, NegotiationState
from procure_rl.opponent import ScriptedPersonaOpponent
from procure_rl.graders import grade

TASK_CONFIG = {
    "single_issue": {
        "persona": "cooperative",
        "max_rounds": 6,
        "buyer_constraints": {
            "price": {"target": 38000, "worst": 52000, "budget": 50000}
        }
    },
    "multi_issue": {
        "persona": "cash_flow_stressed",
        "max_rounds": 8,
        "buyer_constraints": {
            "price": {"target": 40000, "worst": 58000, "budget": 55000},
            "payment_days": {"target": 60, "worst": 30, "preference": 60}
        }
    },
    "adversarial": {
        "persona": "aggressive_anchor",
        "max_rounds": 10,
        "buyer_constraints": {
            "price": {"target": 80000, "worst": 120000, "budget": 115000},
            "payment_days": {"target": 60, "worst": 30, "preference": 60},
            "support_hours": {"target": 150, "worst": 80, "preference": 150}
        }
    }
}

try:
    from openenv.core.env_server import Environment
except ImportError:
    class Environment:
        pass

class ProcureRLEnvironment(Environment):

    def __init__(self):
        self._state = NegotiationState()
        self._opponent = None
        self._task_config = None
        self._done = False
        self._last_offer = {}
        self._consecutive_concessions = 0
        self._prev_agent_price = None

    def reset(self, task_id: str = "single_issue", seed: int = 42) -> NegotiationObservation:
        if task_id not in TASK_CONFIG:
            raise ValueError(f"Unknown task: {task_id}. Valid: {list(TASK_CONFIG.keys())}")

        config = TASK_CONFIG[task_id]
        self._task_config = config
        self._done = False
        self._consecutive_concessions = 0
        self._prev_agent_price = None

        self._opponent = ScriptedPersonaOpponent(
            task_id=task_id,
            seed=seed,
            persona=config["persona"]
        )

        opening_msg, opening_terms = self._opponent.get_opening_message()
        self._last_offer = opening_terms

        self._state = NegotiationState(
            task_id=task_id,
            episode_id=str(uuid.uuid4())[:8],
            round_number=0,
            rapport_score=0.5,
            consecutive_concessions=0,
            deal_reached=False,
            final_terms=None,
            cumulative_reward=0.0
        )

        return NegotiationObservation(
            task_id=task_id,
            round_number=0,
            max_rounds=config["max_rounds"],
            supplier_message=opening_msg,
            current_offer=opening_terms,
            last_4_exchanges=[{"role": "supplier", "message": opening_msg, "terms": opening_terms}],
            buyer_constraints=config["buyer_constraints"],
            rapport_hint="neutral",
            done=False
        )

    def step(self, action: NegotiationAction):
        if self._done:
            obs = self._make_obs("Episode finished. Call reset().")
            return obs, 0.0, True, {"error": "episode_done"}

        self._state.round_number += 1
        round_num = self._state.round_number
        config = self._task_config
        max_rounds = config["max_rounds"]

        reward = 0.0
        error = None

        # Track consecutive concessions
        if self._prev_agent_price is not None:
            current_price = action.terms.get('price', self._prev_agent_price)
            if current_price > self._prev_agent_price:  # agent conceded (price went up toward seller)
                self._consecutive_concessions += 1
            else:
                self._consecutive_concessions = 0
        self._prev_agent_price = action.terms.get('price')
        self._state.consecutive_concessions = self._consecutive_concessions

        # Handle accept
        if action.move_type == "accept":
            self._done = True
            self._state.deal_reached = True
            self._state.final_terms = self._last_offer
            reward = grade(
                self._state.task_id,
                self._last_offer,
                True,
                round_num,
                consecutive_concessions_flag=(self._consecutive_concessions >= 2)
            )
            self._state.cumulative_reward = reward
            obs = self._make_obs()
            obs.done = True
            return obs, reward, True, {"deal_price": self._last_offer.get('price')}

        # Handle reject
        if action.move_type == "reject":
            if round_num >= max_rounds:
                self._done = True
                reward = 0.0
                obs = self._make_obs()
                obs.done = True
                return obs, reward, True, {"error": "rejected_at_limit"}
            obs = self._make_obs()
            return obs, 0.0, False, {}

        # Handle make_offer or bundle
        opponent_msg, opponent_terms = self._opponent.respond(
            agent_message=action.message,
            agent_terms=action.terms,
            round_number=round_num,
            consecutive_concessions=self._consecutive_concessions
        )

        # Check if opponent accepted
        if opponent_terms.get("_accepted"):
            self._done = True
            self._state.deal_reached = True
            self._state.final_terms = action.terms
            reward = grade(
                self._state.task_id,
                action.terms,
                True,
                round_num,
                consecutive_concessions_flag=(self._consecutive_concessions >= 2)
            )
            self._state.cumulative_reward = reward
            obs = self._make_obs(supplier_message=opponent_msg)
            obs.done = True
            return obs, reward, True, {"deal_price": action.terms.get('price')}

        self._last_offer = opponent_terms
        self._state.rapport_score = self._opponent.rapport

        # Episode limit
        if round_num >= max_rounds:
            self._done = True
            reward = 0.0
            obs = self._make_obs(supplier_message=opponent_msg)
            obs.done = True
            return obs, reward, True, {"error": "max_rounds_reached"}

        obs = self._make_obs(supplier_message=opponent_msg)
        return obs, 0.0, False, {}

    def state(self) -> NegotiationState:
        return self._state

    def _make_obs(self, supplier_message: str = None) -> NegotiationObservation:
        rapport = self._state.rapport_score
        if rapport >= 0.65:
            hint = "positive"
        elif rapport <= 0.35:
            hint = "negative"
        else:
            hint = "neutral"

        return NegotiationObservation(
            task_id=self._state.task_id,
            round_number=self._state.round_number,
            max_rounds=self._task_config["max_rounds"],
            supplier_message=supplier_message or "",
            current_offer=self._last_offer,
            last_4_exchanges=[],
            buyer_constraints=self._task_config["buyer_constraints"],
            rapport_hint=hint,
            done=self._done
        )

---

server/app.py:

import sys, os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from dataclasses import asdict
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import Optional, Dict, Any

from procure_rl.environment import ProcureRLEnvironment
from procure_rl.models import NegotiationAction

app = FastAPI(title="ProcureRL", version="1.0.0")
app.add_middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"])

_env = ProcureRLEnvironment()

class ResetRequest(BaseModel):
    task_id: Optional[str] = "single_issue"
    seed: Optional[int] = 42

class StepRequest(BaseModel):
    move_type: str = "make_offer"
    terms: Dict[str, Any] = {}
    message: str = ""

@app.get("/health")
async def health():
    return {"status": "ok", "service": "procure-rl"}

@app.get("/metadata")
async def metadata():
    return {
        "name": "procure-rl",
        "tasks": ["single_issue", "multi_issue", "adversarial"],
        "description": "LLM agent learns procurement negotiation"
    }

@app.post("/reset")
async def reset(req: ResetRequest = ResetRequest()):
    try:
        obs = _env.reset(task_id=req.task_id, seed=req.seed)
        return asdict(obs)
    except ValueError as e:
        raise HTTPException(400, str(e))
    except Exception as e:
        raise HTTPException(500, f"Reset failed: {e}")

@app.post("/step")
async def step(req: StepRequest):
    action = NegotiationAction(
        move_type=req.move_type,
        terms=req.terms,
        message=req.message
    )
    try:
        obs, reward, done, info = _env.step(action)
        return {"observation": asdict(obs), "reward": reward, "done": done, "info": info}
    except Exception as e:
        raise HTTPException(500, f"Step failed: {e}")

@app.get("/state")
async def state():
    return asdict(_env.state())

if __name__ == "__main__":
    import uvicorn
    port = int(os.getenv("PORT", 7860))
    uvicorn.run("server.app:app", host="0.0.0.0", port=port)

---

inference.py — exact stdout format, no deviation:

import os
import json
from openai import OpenAI

API_KEY = os.getenv("HF_TOKEN") or os.getenv("API_KEY")
API_BASE_URL = os.getenv("API_BASE_URL", "https://router.huggingface.co/v1")
MODEL_NAME = os.getenv("MODEL_NAME", "Qwen/Qwen2.5-72B-Instruct")
BENCHMARK = "procure-rl"
MAX_STEPS = 10

client = OpenAI(api_key=API_KEY, base_url=API_BASE_URL)

# Import environment directly (not via HTTP for baseline)
import sys
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
from procure_rl.environment import ProcureRLEnvironment
from procure_rl.models import NegotiationAction

TASKS = ["single_issue", "multi_issue", "adversarial"]

SYSTEM_PROMPT = """You are a professional procurement negotiator. Your goal is to negotiate the best possible deal for your company.

You will receive a supplier's message and current offer terms. You must respond with a JSON action in this exact format:
{
  "move_type": "make_offer",
  "terms": {"price": 42000, "payment_days": 45},
  "message": "Your natural language response to the supplier"
}

move_type must be one of: make_offer, accept, reject, bundle
terms must include price and any other issues being negotiated.
message should be professional and collaborative when possible.

Your buyer constraints will be provided. Do not exceed your budget. Try to reach the target price."""

def get_agent_action(obs_dict: dict) -> dict:
    task_id = obs_dict.get("task_id", "single_issue")
    supplier_msg = obs_dict.get("supplier_message", "")
    current_offer = obs_dict.get("current_offer", {})
    constraints = obs_dict.get("buyer_constraints", {})
    rapport_hint = obs_dict.get("rapport_hint", "neutral")
    round_num = obs_dict.get("round_number", 0)
    max_rounds = obs_dict.get("max_rounds", 10)

    user_content = f"""Task: {task_id}
Round: {round_num}/{max_rounds}
Supplier says: "{supplier_msg}"
Current offer on table: {json.dumps(current_offer)}
Your constraints: {json.dumps(constraints)}
Relationship rapport: {rapport_hint}

Respond with your negotiation action as JSON."""

    response = client.chat.completions.create(
        model=MODEL_NAME,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_content}
        ],
        max_tokens=300,
        temperature=0.3
    )

    content = response.choices[0].message.content.strip()

    # Parse JSON from response
    try:
        # Find JSON in response
        start = content.find('{')
        end = content.rfind('}') + 1
        if start >= 0 and end > start:
            action_dict = json.loads(content[start:end])
        else:
            # Fallback
            action_dict = {
                "move_type": "make_offer",
                "terms": current_offer,
                "message": content[:200]
            }
    except:
        action_dict = {
            "move_type": "make_offer",
            "terms": current_offer,
            "message": "I'd like to continue our discussion."
        }

    return action_dict

def run_task(task_id: str) -> dict:
    env = ProcureRLEnvironment()
    obs = env.reset(task_id=task_id, seed=42)
    obs_dict = {
        "task_id": obs.task_id,
        "round_number": obs.round_number,
        "max_rounds": obs.max_rounds,
        "supplier_message": obs.supplier_message,
        "current_offer": obs.current_offer,
        "buyer_constraints": obs.buyer_constraints,
        "rapport_hint": obs.rapport_hint,
        "done": obs.done
    }

    print(f"[START] task={task_id} env={BENCHMARK} model={MODEL_NAME}")

    rewards = []
    step = 0
    done = False
    final_score = 0.0

    while not done and step < MAX_STEPS:
        step += 1

        action_dict = get_agent_action(obs_dict)
        action = NegotiationAction(
            move_type=action_dict.get("move_type", "make_offer"),
            terms=action_dict.get("terms", {}),
            message=action_dict.get("message", "")
        )

        obs, reward, done, info = env.step(action)
        rewards.append(reward)

        action_str = f"{action.move_type}({json.dumps(action.terms)})"
        error = info.get("error", None)

        print(f"[STEP] step={step} action={action_str} reward={reward:.2f} done={str(done).lower()} error={error if error else 'null'}")

        if done:
            final_score = reward if reward > 0 else (max(rewards) if rewards else 0.0)
            break

        obs_dict = {
            "task_id": obs.task_id,
            "round_number": obs.round_number,
            "max_rounds": obs.max_rounds,
            "supplier_message": obs.supplier_message,
            "current_offer": obs.current_offer,
            "buyer_constraints": obs.buyer_constraints,
            "rapport_hint": obs.rapport_hint,
            "done": obs.done
        }

    rewards_str = ",".join(f"{r:.2f}" for r in rewards)
    success = final_score > 0.1

    print(f"[END] success={str(success).lower()} steps={step} score={final_score:.2f} rewards={rewards_str}")

    return {"task": task_id, "score": final_score, "steps": step}

if __name__ == "__main__":
    results = []
    for task in TASKS:
        result = run_task(task)
        results.append(result)

    print(f"\nBaseline Results:")
    for r in results:
        print(f"  {r['task']}: {r['score']:.3f}")

---

openenv.yaml:

name: procure-rl
version: "1.0.0"
description: "LLM agent learns procurement negotiation strategy against scripted supplier opponents with hidden utility functions"
author: "your-hf-username"
tags:
  - openenv
  - negotiation
  - procurement
  - real-world
  - rl
tasks:
  - id: single_issue
    description: "Negotiate software license price with cooperative supplier"
    difficulty: easy
    max_steps: 6
    reward_range: [0.0, 1.0]
  - id: multi_issue
    description: "Negotiate price and payment terms with cash-flow-sensitive supplier"
    difficulty: medium
    max_steps: 8
    reward_range: [0.0, 1.0]
  - id: adversarial
    description: "Negotiate multiple issues against aggressive anchoring supplier"
    difficulty: hard
    max_steps: 10
    reward_range: [0.0, 1.0]
reward_range: [0.0, 1.0]
observation_space:
  type: object
  description: "Natural language supplier message with structured negotiation state and rapport signal"
action_space:
  type: object
  description: "Negotiation move type, structured terms, and natural language message"

---

Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
ENV PORT=7860
EXPOSE 7860
CMD ["python", "-m", "uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "7860"]

---

requirements.txt:

fastapi==0.109.0
uvicorn==0.27.0
pydantic>=2.0.0
openai>=1.0.0
openenv-core>=0.1.0

---

Build all files exactly as specified. Run locally with:

docker build -t procure-rl .
docker run -p 7860:7860 procure-rl

Test with:

curl -X POST http://localhost:7860/reset -H "Content-Type: application/json" -d '{"task_id": "single_issue"}'

Then run inference.py locally with HF_TOKEN set to verify [START][STEP][END] format.

---

PLAN.MD

# ProcureRL — Implementation Plan

## What We Are Building

An OpenEnv-compliant RL environment where an LLM agent learns
procurement negotiation strategy against scripted supplier opponents.

The key innovation: language-sensitive opponent behavior. The agent's
natural language quality affects opponent concession rates, making LLM
genuinely required — not just for parsing but for output quality.

## Why This Wins

- Zero negotiation environments in OpenEnv hub — confirmed
- Documented LLM weakness in buyer negotiation (ACL 2024)
- Walmart/Pactum market validation — real enterprise deployment exists
- Nash-inspired grader with language mechanism — novel and memorable
- Deterministic, reproducible, pure Python graders

## Implementation Order (strict)

### Phase 1: Core Logic (Day 1, first 4 hours)

- [ ] procure_rl/models.py — dataclasses only
- [ ] procure_rl/opponent.py — ScriptedPersonaOpponent
- [ ] procure_rl/graders.py — three grader functions
- [ ] procure_rl/environment.py — ProcureRLEnvironment
- [ ] Test: import and run reset() + step() in Python shell

### Phase 2: Server (Day 1, next 2 hours)

- [ ] server/app.py — FastAPI with /health /reset /step /state
- [ ] requirements.txt
- [ ] Test: uvicorn server.app:app, curl /health

### Phase 3: Spec Compliance (Day 1, final 2 hours)

- [ ] openenv.yaml — exact schema
- [ ] Run: openenv validate
- [ ] Fix any validation errors

### Phase 4: Dockerfile + HF Spaces (Day 2, first 3 hours)

- [ ] Dockerfile
- [ ] docker build -t procure-rl .
- [ ] docker run -p 7860:7860 procure-rl
- [ ] curl http://localhost:7860/health
- [ ] Push to HF Spaces

### Phase 5: Inference Script (Day 2, next 2 hours)

- [ ] inference.py
- [ ] Run locally: HF_TOKEN=xxx python inference.py
- [ ] Verify [START][STEP][END] format exactly
- [ ] Verify runtime < 20 minutes

### Phase 6: README + Calibration (Day 2, final 2 hours)

- [ ] README.md with all required sections
- [ ] Run inference.py with weak model (7B) and strong model (72B)
- [ ] Verify score spread exists
- [ ] Submit

## Critical Checks Before Submission

```bash
# 1. Spec compliance
openenv validate

# 2. Docker build
docker build -t procure-rl .

# 3. Docker run
docker run -p 7860:7860 procure-rl &
curl -X POST http://localhost:7860/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "single_issue"}'

# 4. Inference script
HF_TOKEN=your_token python inference.py

# 5. Score verification
# single_issue: should be 0.30-0.55
# multi_issue: should be 0.15-0.35
# adversarial: should be 0.10-0.25
```

Score Calibration Targets

Task	Random	Base LLM	Goal
single_issue	0.15-0.25	0.35-0.50	0.65-0.78
multi_issue	0.08-0.15	0.20-0.32	0.52-0.65
adversarial	0.03-0.10	0.12-0.22	0.42-0.55

If base LLM scores above 0.55 on single_issue → opponent too easy, reduce cooperative concession rate. If base LLM scores below 0.15 on single_issue → opponent too hard, increase cooperative concession rate.

README Required Sections

1. Environment description and motivation (Walmart/Pactum reference)
2. The Language-Sensitive Opponent (this is your wow factor)
3. Action space definition with examples
4. Observation space definition
5. Task descriptions with expected scores
6. Setup instructions (pip install + docker)
7. Baseline scores (from inference.py run)

What NOT To Add

• Nash bargaining (too complex, edge cases)
• Step reward shaping (shaping bias risk)
• LLM inside environment (reproducibility)
• More than 3 tasks (scope creep)
• Preference shift mechanics (complexity risk)

The One Sentence For Every Judge Question

"Why RL?" → Sequential decisions, delayed reward, hidden opponent utility — policy only emerges through thousands of negotiation episodes.

"Why LLM?" → Language quality directly affects opponent rapport score and concession rate. A heuristic agent gets neutral rapport. An LLM that learns collaborative framing gets cooperative responses. The language IS the policy.

"Is this real?" → Walmart deployed Pactum for exactly this. 90% of CPOs adopting AI negotiation in 2025. The gap between rule-based current tools and trained LLM policy is the research contribution.

"Is this novel?" → Zero negotiation environments in OpenEnv hub. Confirmed.

---

**This is the final version. Build it exactly as specified.**