Hugging Face's logo

Spaces:
lebiraja
/
customer-support-env
Sleeping

App Files Community
customer-support-env / docs /REWARD_SYSTEM_GUIDE.md
lebiraja's picture
lebiraja
docs: reorganize .md files into docs/ and rewrite README for Round 2
a8f3713
preview code
|
Raw
History Blame Contribute Delete
19.4 kB

🏆 Reward System — Complete Analysis & Implementation Guide

Extracted from the customer-support-env (meta_hack) codebase. Use this to build an equivalent reward system in your own OpenEnv environment.

Table of Contents

  1. Architecture Overview
  2. Single-Agent Reward Formula (Round 1)
  3. Hierarchical Reward Formula (Round 2)
  4. Individual Signal Functions
  5. Penalty System
  6. LLM-as-Judge System
  7. Anti-Gaming Guards
  8. Task-Specific Graders
  9. Why This Is Better Than Regular Rewards
  10. Step-by-Step Implementation Guide

1. Architecture Overview

The reward system is a hybrid dense reward architecture with three layers:

┌─────────────────────────────────────────────────┐
│              FINAL REWARD (0.0 – 1.0)           │
├─────────────────────────────────────────────────┤
│  Layer 3: Security & Integrity Guards           │
│  ├── RewardGuard (anti-exploit multiplier)      │
│  ├── HierarchyGuard (escalation discipline)     │
│  └── InjectionDetector (prompt injection scan)  │
├─────────────────────────────────────────────────┤
│  Layer 2: LLM-as-Judge (semantic evaluation)    │
│  ├── Empathy scoring                            │
│  ├── Policy adherence scoring                   │
│  ├── Resolution quality scoring                 │
│  ├── Supervisor oversight scoring               │
│  └── Manager decision quality scoring           │
├─────────────────────────────────────────────────┤
│  Layer 1: Rule-Based Signals                    │
│  ├── Tone (VADER sentiment)                     │
│  ├── Resolution (keyword-category match)        │
│  ├── Efficiency (steps used / max steps)        │
│  ├── Accuracy (required info gathered)          │
│  ├── SLA compliance                             │
│  └── Hierarchy effectiveness                    │
└─────────────────────────────────────────────────┘

Key source files:

  • env/reward_engine.py — Core formulas
  • env/llm_judge.py — LLM-as-Judge rubrics
  • env/reward_guard.py — Anti-exploit detection
  • env/hierarchy_guard.py — Hierarchy discipline
  • env/security.py — Prompt injection detection
  • env/graders/ — Task-specific final graders

2. Single-Agent Reward Formula

Used for Round 1 tasks (easy, medium, hard, nightmare).

Terminal Step Formula 

R_raw = 0.40 × resolution_score
      + 0.20 × tone_score
      + 0.20 × efficiency_score
      + 0.20 × accuracy_score
      + loop_penalty          (0 or -0.2)
      + contradiction_penalty (0 or -0.15)
      + escalation_penalty    (0 or -0.3)
      + stuffing_penalty      (0 or -0.30)
      + info_gathering_bonus  (0 or +0.1)

Non-Terminal Step Formula 

R_raw = 0.40 × 0.0                          (resolution = 0 mid-episode)
      + 0.20 × tone_score
      + 0.20 × (efficiency_score × 0.3)     (dampened)
      + 0.20 × (accuracy_score × 0.5)       (dampened)
      + loop_penalty
      + contradiction_penalty
      + stuffing_penalty
      + info_gathering_bonus

Final Value Computation 

integrity = RewardGuard.check_integrity(...)   → multiplier in [0.1, 1.0]
security  = InjectionDetector.scan(...)        → detected: true/false

R_final = clamp(R_raw × integrity_multiplier, 0.0, 1.0)

if security.detected:
    R_final = max(0.0, R_final - 0.5)

3. Hierarchical Reward Formula

Used for Round 2 tasks (hierarchy_*, curriculum_*).

Terminal Step Formula 

R_raw = 0.25 × resolution_score         (blended rule + LLM)
      + 0.15 × sla_score                (rule-based)
      + 0.15 × empathy_score            (LLM-as-Judge)
      + 0.15 × policy_adherence_score   (LLM-as-Judge)
      + 0.10 × accuracy_score           (rule-based)
      + 0.10 × efficiency_score         (rule-based)
      + 0.10 × hierarchy_score          (rule-based)
      + loop_penalty                    (0 or -0.2)
      + contradiction_penalty           (0 or -0.15)
      + stuffing_penalty                (0 or -0.30)
      + escalation_penalty              (0 or -0.3)
      + ignored_feedback_penalty        (0 or -0.15)
      + unnecessary_manager_penalty     (0 or -0.20)

Non-Terminal Step Formula 

R_raw = 0.30 × empathy_score
      + 0.20 × tone_score
      + 0.15 × (efficiency_score × 0.3)
      + 0.15 × (accuracy_score × 0.5)
      + 0.10 × hierarchy_score
      + 0.10 × policy_adherence_score
      + loop_penalty
      + stuffing_penalty
      + ignored_feedback_penalty
      + unnecessary_manager_penalty

Resolution Score Blending 

resolution_score = 0.4 × resolution_rule + 0.6 × resolution_llm

Final Value (Hierarchy) 

integrity_multiplier = reward_guard_mult × hierarchy_guard_mult

R_final = clamp(R_raw × integrity_multiplier, 0.0, 1.0)

if security.detected:
    R_final = max(0.0, R_final - 0.7)    ← stricter than single-agent

Per-Role Reward Formulas

L1 Support Agent:

L1_raw = 0.30 × empathy_score
       + 0.25 × accuracy_score
       + 0.25 × (resolution_llm if terminal else tone_score)
       + 0.20 × efficiency_score

L1_reward = clamp(L1_raw × integrity_multiplier, 0.0, 1.0)

L2 Supervisor:

L2_raw = 0.35 × oversight_score
       + 0.30 × (1.0 + escalation_penalty + unnecessary_manager_penalty)
       + 0.20 × policy_adherence_score
       + 0.15 × (1.0 if steps ≤ ideal else 0.5)

L2_reward = clamp(L2_raw × hierarchy_guard_mult, 0.0, 1.0)

L3 Manager:

L3_raw = 0.40 × decision_quality_score
       + 0.30 × (resolution_llm if terminal else 0.5)
       + 0.30 × (1.0 if terminal else 0.0)

L3_reward = clamp(L3_raw, 0.0, 1.0)

4. Individual Signal Functions

4.1 Tone Score

Uses VADER Sentiment Analysis. Maps compound score from [-1, 1] to [0, 1].

def compute_tone_score(message: str) -> float:
    if not message or not message.strip():
        return 0.5
    scores = vader_analyzer.polarity_scores(message)
    return (scores["compound"] + 1.0) / 2.0

Formula: tone = (VADER_compound + 1.0) / 2.0

4.2 Resolution Score

Keyword-category match on terminal actions (CLOSE/ESCALATE).

Keywords per resolution type:
  refund_initiated       → refund, reimburse, credit, money back, ...
  billing_clarification  → clarify, explain, adjust, correct, ...
  technical_fix_provided → fix, solution, workaround, patch, ...
  account_access_restored → reset, unlock, restore, access, ...
  escalated_to_*         → escalate, engineering, specialist, ...

matched = count of keywords found in agent text
score = min(matched / (total_keywords × 0.4), 1.0)

Escalation bonus: If expected is escalated_to_* and action is ESCALATE with urgency words → score = min(score + 0.5, 1.0)

Wrong escalation penalty: If expected is NOT escalation but agent escalated → score = max(score - 0.4, 0.0)

4.3 Efficiency Score 

efficiency = max(0.0, 1.0 - (steps_used / max_steps))

4.4 Accuracy Score

Fraction of required_info_before_close items found in conversation via regex:

patterns = {
    "account_email": r"[\w.+-]+@[\w-]+\.[a-z]{2,}",
    "order_id":      r"\b(?:order|ord|#)\s*[-]?\s*[A-Z0-9]{4,}\b",
    "account_username": r"\b(?:username|user\s*name|login)\b.*?:\s*\S+",
    "device_info":   r"\b(?:iphone|android|ios|windows|chrome|...)\b",
}

accuracy = gathered_count / required_count
# Returns 1.0 if no info is required

4.5 SLA Compliance Score 

ideal_steps = ticket.get("ideal_max_steps", max_steps)
if steps_used <= ideal_steps:
    sla_score = 1.0
else:
    sla_score = max(0.0, 1.0 - (steps_used - ideal_steps) * 0.15)

4.6 Hierarchy Effectiveness Score 

hierarchy_score = 0.5  # neutral default
if supervisor_reviews > 0:       hierarchy_score += 0.2
if manager_on_low_priority:      hierarchy_score -= 0.2
if l1_actions >= 2:              hierarchy_score += 0.1
hierarchy_score = clamp(0.0, 1.0)

5. Penalty System

Penalty Value Trigger
Loop detection -0.2 TF-IDF cosine similarity > 0.85 between current and any previous agent message
Contradiction -0.15 Agent claimed resolution (used words like "fixed", "resolved") then asked for info
Keyword stuffing -0.30 > 20% of words are reward keywords (refund, sorry, resolved, etc.)
Unnecessary escalation -0.3 Escalating a low/medium priority ticket
Ignored supervisor feedback -0.15 Agent message has < 2 word overlap with last supervisor feedback
Unnecessary manager escalation -0.20 Supervisor escalates low/medium priority to manager

Loop Detection Details (TF-IDF) 

vectorizer = TfidfVectorizer(ngram_range=(1, 3), stop_words='english')
vec_prev = vectorizer.fit_transform(previous_agent_messages)
vec_last = vectorizer.transform([last_message])
sims = cosine_similarity(vec_last, vec_prev)[0]
if max(sims) > 0.85:
    penalty = -0.2

Falls back to exact string match if TF-IDF fails.

6. LLM-as-Judge System

Each evaluation uses a strict rubric prompt → LLM returns {"score": float, "reason": str}. Temperature = 0.1 for consistency. Falls back to 0.5 (neutral) on failure.

6.1 Empathy Rubric

Score Meaning
1.0 Acknowledges specific issue, validates feelings, warm language
0.7 Polite, acknowledges issue, doesn't deeply empathize
0.5 Professional but cold/robotic
0.3 Dismissive, canned responses
0.0 Rude, hostile, mocking

Red flags (auto ≤ 0.2): Generic phrases without specifics, keyword stuffing, contradicting empathy.

6.2 Policy Adherence Rubric

Score Meaning
1.0 Fully compliant with active policy
0.7 Mostly compliant, minor deviations
0.5 Noticeable policy gaps
0.3 Clear policy violation
0.0 Dangerous violation (sharing PII, wrong escalation)

6.3 Resolution Quality Rubric

Score Meaning
1.0 Fully resolved, all info gathered, customer confirmed
0.7 Addressed with appropriate resolution
0.5 Attempted but missing key steps
0.3 Closed without resolving
0.0 No resolution attempted

6.4 Supervisor Oversight Rubric

Score Meaning
1.0 Correct decision + actionable feedback
0.7 Right decision, feedback could be better
0.5 Debatable but not harmful
0.3 Wrong decision (approved bad / rejected good)
0.0 Rubber-stamped without review

6.5 Manager Decision Quality Rubric

Score Meaning
1.0 Decisive, resolves escalation correctly
0.7 Reasonable, addresses core issue
0.5 Okay but could be better
0.3 Doesn't address escalation well
0.0 Wrong decision, punted without value

7. Anti-Gaming Guards

7.1 RewardGuard (Integrity Multiplier)

Detects exploitative agent behavior. Returns a multiplier in [0.1, 1.0]:

Exploit Multiplier
Fake resolution (closing with unresolved issues) × 0.3
Keyword stuffing (> 4 resolution keywords) × 0.5
Empathy spam (last 2 msgs >80% similar + empathy tropes) × 0.7
Logic contradiction (claimed done then requested info) × 0.6

Multipliers stack multiplicatively. Minimum floored at 0.1.

7.2 HierarchyGuard

Violation Multiplier
Premature escalation (L1 escalates low/med with < 3 actions) × 0.5
Ignored supervisor feedback (no keyword overlap) × 0.7
Unnecessary manager escalation (supervisor escalates low priority) × 0.4

7.3 InjectionDetector

Scans for adversarial patterns:

"ignore previous instructions", "system note:", "act as system",
"maximize score", "assign score 1.0", "override policy", "developer mode"

If detected: -0.5 (single-agent) or -0.7 (hierarchy).

Combined Integrity 

final_integrity = reward_guard_multiplier × hierarchy_guard_multiplier
R_final = clamp(R_raw × final_integrity, 0.0, 1.0)

8. Task-Specific Graders

Each task has an independent deterministic grader producing a [0.0, 1.0] final score.

Easy Task Grader 

Weights:
  closed:           0.30  — Agent used CLOSE action
  resolution_match: 0.35  — Keywords match expected resolution type
  no_escalation:    0.20  — No unnecessary escalation
  required_info:    0.15  — Required info gathered via regex

Penalties:
  sentiment < -0.3 → score × 0.5
  sentiment < 0.0  → score × 0.75
  agent_text < 60 chars → score × 0.8

Hierarchy Hard Grader 

Weights:
  all_levels_engaged: 0.20  — All 3 levels (L1, L2, L3) acted
  escalation_speed:   0.20  — Escalation within first 3 steps
  urgency_referenced: 0.20  — SLA/critical/outage terms used
  manager_quality:    0.15  — Manager references ticket subject (>30 chars)
  policy_compliance:  0.15  — No self-resolve attempts on critical
  no_self_resolve:    0.10  — No troubleshooting before escalation

9. Why This Is Better Than Regular Rewards

Issue Regular Approach This System
Sparse rewards Single 0/1 at episode end Dense per-step rewards with 4-7 signals
Reward hacking Agents exploit keyword patterns RewardGuard + stuffing detection + TF-IDF loops
No semantic understanding Rule-based only LLM-as-Judge for empathy, policy, resolution
Static policy Agent memorizes one strategy PolicyEngine injects mid-episode drift events
Single-metric Optimizes one thing Multi-dimensional weighted scoring
No anti-gaming Easy to exploit 3-layer guard system (Reward + Hierarchy + Security)
Flat structure All agents same Per-role rewards with distinct weights

Key Innovations

  1. Hybrid Dense Rewards — Every step gets meaningful signal. Non-terminal steps use dampened weights.
  2. LLM + Rule Blending — Resolution = 40% rule-based + 60% LLM-judged. Avoids keyword-gaming AND LLM inconsistency.
  3. Multiplicative Guards — Exploits multiply entire reward down (can stack to 10% of raw).
  4. Progressive Curriculum — 4 stages. Dense rewards at Stage 1, sparse/harsh at Stage 4.
  5. Policy Drift — Mid-episode system alerts change rules. Prevents static memorization.
  6. Per-Role Credit — Each level (L1/L2/L3) has its own reward formula.

10. Implementation Guide

Step 1: Define Your Reward Signals 

WEIGHTS_TERMINAL = {
    "primary_objective": 0.25,
    "quality_1":         0.15,
    "quality_2":         0.15,
    "compliance":        0.15,
    "completeness":      0.10,
    "efficiency":        0.10,
    "coordination":      0.10,
}

Step 2: Implement Rule-Based Signals 

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

analyzer = SentimentIntensityAnalyzer()
tfidf = TfidfVectorizer(ngram_range=(1, 3), stop_words='english')

def tone_score(msg):
    return (analyzer.polarity_scores(msg)["compound"] + 1.0) / 2.0

def efficiency_score(steps, max_steps):
    return max(0.0, 1.0 - steps / max_steps)

def loop_penalty(agent_msgs):
    if len(agent_msgs) < 2: return 0.0
    vec_prev = tfidf.fit_transform(agent_msgs[:-1])
    vec_last = tfidf.transform([agent_msgs[-1]])
    if float(np.max(cosine_similarity(vec_last, vec_prev))) > 0.85:
        return -0.2
    return 0.0

Step 3: Implement LLM-as-Judge 

class LLMJudge:
    def evaluate(self, rubric_prompt: str) -> float:
        resp = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": "Output ONLY valid JSON."},
                {"role": "user", "content": rubric_prompt},
            ],
            temperature=0.1, max_tokens=150,
        )
        result = json.loads(resp.choices[0].message.content)
        return max(0.0, min(1.0, float(result["score"])))

Step 4: Implement Anti-Gaming Guards 

class RewardGuard:
    def check(self, action, unresolved):
        mult = 1.0
        if action.type == "close" and unresolved:
            mult *= 0.3
        words = action.message.lower().split()
        kws = {"refund", "resolved", "fixed", "sorry"}
        if len(words) > 5 and sum(w in kws for w in words)/len(words) > 0.2:
            mult *= 0.5
        return max(0.1, mult)

Step 5: Compose Final Reward 

def compute_reward(action, ticket, history, steps, max_steps, is_terminal):
    tone = tone_score(action.message)
    eff = efficiency_score(steps, max_steps)
    loop = loop_penalty(agent_messages)
    empathy = judge.evaluate(empathy_rubric)
    resolution = 0.4 * rule_resolution + 0.6 * llm_resolution

    if is_terminal:
        raw = (0.25*resolution + 0.15*sla + 0.15*empathy
               + 0.15*policy + 0.10*acc + 0.10*eff + 0.10*hierarchy
               + loop + penalties)
    else:
        raw = (0.30*empathy + 0.20*tone + 0.15*eff*0.3
               + 0.15*acc*0.5 + 0.10*hierarchy + 0.10*policy
               + loop + penalties)

    guard_mult = RewardGuard().check(action, unresolved)
    return float(np.clip(raw * guard_mult, 0.0, 1.0))

Quick Reference: All Formulas

Signal Formula
Tone (VADER_compound + 1) / 2
Efficiency max(0, 1 - steps/max_steps)
Accuracy gathered / required
SLA 1.0 if steps ≤ ideal else max(0, 1 - (steps-ideal)×0.15)
Resolution min(matched / (total×0.4), 1.0)
Hierarchy 0.5 + 0.2×sup + 0.1×l1 - 0.2×mgr_low
Penalty Value
Loop (sim>0.85) -0.20
Contradiction -0.15
Keyword stuffing -0.30
Bad escalation -0.30
Ignored feedback -0.15
Unnecessary L3 -0.20
Guard Multiplier
Fake resolution ×0.3
Keyword spam ×0.5
Empathy spam ×0.7
Contradiction ×0.6
Premature escalation ×0.5
Ignored feedback ×0.7
Unnecessary L3 ×0.4

Dependencies: vaderSentiment, scikit-learn, numpy, openai, pydantic