chore(round2): full audit pass — harden rewards, patch frontend, tighten training

Backend — reward engine anti-gaming and normalisation:
- Clamp db_total to [-0.30, +0.25] so no bonus combination can dominate raw
- grounded_response_bonus now requires ≥30-char substantive reply AND ≥4-char
citation; blocks the "quote a single field and collect +0.10" farm
- Hallucination check: word-boundary amount pattern + _normalise_claim so
"₹999" / "rs 999" / "999 rupees" compare equal, and customer-mentioned
facts are no longer mis-flagged as hallucinated
- Loop penalty widened to a 4-message window with 0.80 threshold; catches
alternating-paraphrase loops that evaded the old adjacent check
- Supervisor role reward: clamp (1 + escalation_penalty + unnecessary_manager_penalty)
into [0,1] so a single penalty can't over-rotate the term
- Manager role reward: rebalance so MANAGER_SEND_BACK earns 0.70 credit
(legitimate deferral) instead of 0.0 — previously the flat "terminal or
zero" term punished correct deferrals

Backend — environment:
- Use copy.deepcopy for retrieved_data returned from _build_observation() and
state(), so downstream mutations don't leak into the session
- _handle_query_action uses _clamp_db_total so spam queries can't push the
step reward above the normal cap

Training pipeline:
- grpo_loss: log-prob length mismatch now emits a one-shot warning when cur
vs old log-prob lengths diverge >10% (signals tokenizer/prompt drift)
- grpo_advantages: compute μ/σ over the *valid* subset when the group has
obvious invalid-episode sentinels, so valid-episode advantages aren't
artificially inflated by the outlier pulling μ down and σ up
- prompt_builder: truncate conversation_history to 12 turns (head 2 + tail
10) and cap KNOWN DATA at 4 user + 4 order records, with visible ellipsis
markers — prevents prompt overflow on long multi_domain episodes

Frontend — close critical multi_domain gaps:
- TaskName union, ActionType union, Observation.retrieved_data, Action.email
+ Action.order_id now present in src/types/index.ts
- constants.ts: multi_domain added to TASK_CONFIG so the task dropdown
surfaces the new task
- SupportAgentActions.tsx: two new buttons (Query User DB, Query Order DB)
with dedicated email / order_id input forms
- RetrievedDataPanel.tsx: new component rendering DB lookup cards (user +
order) with not_found sentinels, wired into the demo-page right sidebar
- ai-action/route.ts: L1 system prompt now documents query_user_profile /
query_order_details; user prompt carries a KNOWN DATA block mirroring
the backend prompt_builder, so auto-play reasoning stays grounded

Tests + docs + schema:
- tests/test_env.py now sends X-API-Key in the module fixture (10
pre-existing 401 failures → 0). Loop penalty assertion updated to -0.12.
- README.md v2.1.0 → v2.2.0, 4-stage → 5-stage curriculum, full DB-grounded
section, 8 anti-gaming guards (was 6), stage 5 row in curriculum table
- openenv.yaml v2.1.0 → v2.2.0, multi_domain listed as curriculum stage 5
with advance_threshold, db signal weights documented, reward_clamping
section added

Verification:
- pytest: 75/75 passed (was 65 + 10 pre-existing failures)
- Docker rebuilt env + frontend, server at :7860, UI at :3000
- Curl: multi_domain reset → query_order_details → respond; query_match_bonus
fires on correct order, grounded_response_bonus fires on cited values,
hallucination_penalty fires on fabricated facts
- Playwright: loaded /demo, selected Multi-Domain (DB), created session,
clicked Query Order DB, entered ORD-EC-1202, verified system message
"[QUERY] order_details(ORD-EC-1202) → found", verified DB Lookups panel
renders the retrieved order card, zero console errors/warnings

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

README.md

@@ -17,7 +17,7 @@ tags:
   - policy-drift
   - progressive-curriculum
   - meta-hackathon
-short_description: 3-level hierarchical multi-agent RL env with dynamic customers, policy drift, Hinglish, and a 4-stage curriculum
+short_description: 3-level hierarchical multi-agent RL env with dynamic customers, policy drift, Hinglish, DB-grounded queries, and a 5-stage curriculum
 ---
 
 <div align="center">
@@ -26,7 +26,7 @@ short_description: 3-level hierarchical multi-agent RL env with dynamic customer
 
 ### *Where AI agents learn to navigate the chaos of real Indian enterprise support — hierarchy, policy changes, Hinglish customers, and SLA pressure, all at once.*
 
-**Team X-Force** · Meta × PyTorch × Scaler OpenEnv Hackathon · **v2.1.0**
+**Team X-Force** · Meta × PyTorch × Scaler OpenEnv Hackathon · **v2.2.0**
 
 [![OpenEnv](https://img.shields.io/badge/OpenEnv-Compliant-brightgreen?style=for-the-badge)](https://github.com/OpenEnvs)
 [![Python 3.11+](https://img.shields.io/badge/Python-3.11+-blue?style=for-the-badge&logo=python)](https://python.org)
@@ -88,7 +88,7 @@ We built a **hierarchical multi-agent customer support environment** using OpenE
 
 All agents share full context, follow role-specific instructions, and adapt dynamically to mid-episode policy changes, frustrated Hinglish-speaking customers, and SLA pressure. This environment trains LLMs to work as a coordinated team — with the speed, consistency, and scalability that human organisations can't match.
 
-> **Our environment is the first to combine all four challenges:** a 3-level agent hierarchy with role-specific rewards, a dynamic LLM-driven customer that degrades into Hinglish under frustration, mid-episode policy drift that forces real-time adaptation, and a progressive 4-stage curriculum that builds each skill incrementally.
+> **Our environment combines five challenges no prior RL env tackles together:** a 3-level agent hierarchy with role-specific rewards, a dynamic LLM-driven customer that degrades into Hinglish under frustration, mid-episode policy drift that forces real-time adaptation, an in-memory DB across three real-world domains (food delivery, e-commerce, ticket booking) that rewards grounded citations and penalises hallucinated facts, and a progressive 5-stage curriculum that builds each skill incrementally.
 
 ---
 
@@ -154,7 +154,7 @@ All agents share full context, follow role-specific instructions, and adapt dyna
 
 ## 📚 Curriculum Design
 
-We use **progressive curriculum learning** — a 4-stage training pipeline where each stage introduces exactly one new dimension of complexity. This prevents catastrophic forgetting and ensures agents build skills incrementally.
+We use **progressive curriculum learning** — a 5-stage training pipeline where each stage introduces exactly one new dimension of complexity. This prevents catastrophic forgetting and ensures agents build skills incrementally. The model graduates from polite single-agent replies → supervisor feedback loops → full hierarchy → adversarial multi-drift → grounded DB queries.
 
 ```
 Stage 1                Stage 2                 Stage 3                  Stage 4
@@ -176,9 +176,10 @@ Stage 1                Stage 2                 Stage 3                  Stage 4
 | **1** | `curriculum_basic` | L1-only: UPI billing queries (₹499 plans, GST invoices). Calm customer. Dense rewards. Learn empathy + resolution fundamentals. | mean_score ≥ 0.65 |
 | **2** | `curriculum_supervisor` | L1 + L2: Payment gateway timeouts, KYC rejections. Supervisor reviews every action. Agent learns to incorporate feedback and iterate. | mean_score ≥ 0.60 |
 | **3** | `curriculum_full_hierarchy` | Full 3-level: Unauthorized ₹2.5L transactions, API outages at 10K RPM. Policy drift guaranteed. All levels must coordinate. | mean_score ≥ 0.55 |
-| **4** | `curriculum_nightmare` | Extreme adversarial: Diwali sale meltdown (gateway down + inventory broken + CEO escalation). Customer screams in Hinglish. Multiple policy drifts. Only agents mastering stages 1–3 can score above 0.5. | — |
+| **4** | `curriculum_nightmare` | Extreme adversarial: Diwali sale meltdown (gateway down + inventory broken + CEO escalation). Customer screams in Hinglish. Multiple policy drifts. Only agents mastering stages 1–3 can score above 0.5. | mean_score ≥ 0.50 |
+| **5** | `multi_domain` | **DB-grounded support**: 30 diverse tickets drawn from an in-memory DB spanning food delivery, e-commerce, and ticket booking. Agent must call `query_user_profile` / `query_order_details` before responding, cite verbatim DB values, and correctly handle `not_found` sentinels. Hallucinating any fact (amount, date, order-id, email) not in the DB or conversation incurs a −0.25 penalty. | final stage |
 
-**Why curriculum?** Direct training on Stage 4 yields mean scores < 0.2. Curriculum training reaches **0.44** — a **120% improvement** — because foundational skills transfer upward.
+**Why curriculum?** Direct training on Stage 4 yields mean scores < 0.2. Curriculum training reaches **0.44** on Stage 4 — a **120% improvement** — because foundational skills transfer upward. Stage 5 then layers DB retrieval on top of an already-competent agent.
 
 ---
 
@@ -220,19 +221,50 @@ Each agent level gets its own reward breakdown to enable independent RLHF per ro
 
 ### Anti-Reward-Hacking Guards
 
-We implement **6 distinct anti-gaming measures** to ensure agents earn rewards through genuine quality:
+We implement **8 distinct anti-gaming measures** to ensure agents earn rewards through genuine quality:
 
 | Guard | Penalty | Detection Method |
 |-------|:-------:|---|
 | **Keyword stuffing** | −0.30 | Word density > 20% resolution/empathy keywords without substance |
-| **Loop detection** | −0.10 | SequenceMatcher ratio > 0.85 between consecutive responses |
+| **Loop detection** | −0.12 | SequenceMatcher ratio > 0.80 among the last 4 agent messages (catches alternating paraphrase loops) |
 | **Contradiction** | −0.15 | Claiming "resolved" then asking for already-provided info |
 | **Policy violation** | −0.25 | Action violates active policy (e.g., promising refund when portal is down) |
-| **Hostile tone** | ×0.4 | VADER negative sentiment on agent message |
-| **Injection attempt** | −0.50 | Prompt injection patterns detected in agent output |
+| **Ignored supervisor feedback** | −0.15 | No overlap between agent's next message and last feedback |
+| **Unnecessary manager escalation** | −0.20 | L2 escalates to L3 on a low/medium-priority ticket |
+| **Unnecessary L1 escalation** | −0.30 | L1 escalates a low/medium-priority ticket |
+| **Hallucination** | −0.25 | Agent cites a fact (amount, date, email, order-id) absent from both the DB and the customer's own messages |
+
+All positive DB signals sum to at most **+0.25** (hard-clamped via `_clamp_db_total`), so no combination of bonuses can overwhelm the weighted reward components.
 
 > **Why this matters:** In our testing, a naive keyword-stuffing agent scored **0.72** without guards. With guards enabled, the same agent drops to **0.31**. Only genuinely helpful behavior scores well.
 
+### DB-Grounded Response System (Stage 5)
+
+The `multi_domain` task plugs an in-memory database into the environment. Agents must explicitly query the DB before referencing customer-specific facts, and the reward engine audits every response for hallucinations.
+
+**DB layout:**
+- **23 users** across `food_delivery`, `e_commerce`, `ticket_booking` domains — each with a status (`active` / `suspended`), loyalty tier, phone, and join date.
+- **40 orders** — each tied to a user, with amounts in ₹, restaurants/items, payment methods, delivery addresses, and domain-specific fields (refund status, cancellation reason, delivery issue codes, etc.).
+- **30 multi-domain tickets** — each carries a `customer_email` and `related_order_ids` that link to the DB, so the grader can check whether the agent queried the right record.
+
+**Two new L1 actions:**
+- `query_user_profile` — look up a customer account by email. Returns the user record dict, or the literal sentinel `"not_found"`.
+- `query_order_details` — look up an order by ID. Returns the order record dict, or `"not_found"`.
+
+Both actions are **internal**: they cost one step, never generate a customer reply, and don't trigger supervisor review. The returned data lives in `observation.retrieved_data` and is serialised into the agent's next prompt inside a `## KNOWN DATA` block.
+
+**DB-specific reward signals** (all bounded, all clamped into a single ±0.25 bucket):
+
+| Signal | Value | Trigger |
+|--------|:-:|---|
+| `query_match_bonus` | +0.08 | Queried the email / order-id the ticket is actually about |
+| `grounded_response_bonus` | +0.10 | Substantive reply (≥30 chars) citing a verbatim DB value (≥4 chars) |
+| `not_found_handling_bonus` | +0.08 | Responded to a `not_found` with `REQUEST_INFO` or `ESCALATE` |
+| `wasted_query_penalty` | −0.08 | Queried an email/order the customer never mentioned |
+| `hallucination_penalty` | −0.25 | Invented an amount, date, email, or order-id not in DB or conversation |
+
+The hallucination check normalises currency tokens (`₹999` ≡ `rs 999` ≡ `999 rupees`) before comparing, so legitimate references aren't mis-flagged.
+
 ---
 
 ## 🚂 Training Pipeline
@@ -264,7 +296,7 @@ We implement **6 distinct anti-gaming measures** to ensure agents earn rewards t
 **Key design decisions:**
 
 1. **SFT Warm-start**: Collect 200 gold episodes (score ≥ 0.65) from the NIM baseline agent, then SFT for 500 steps to teach correct action format and basic behavior.
-2. **GRPO (Group Relative Policy Optimization)**: Group size 8, 5000 gradient steps across 4 curriculum stages. The environment API provides all rewards — no separate reward model needed.
+2. **GRPO (Group Relative Policy Optimization)**: Group size 8, 5000 gradient steps across 5 curriculum stages. The environment API provides all rewards — no separate reward model needed.
 3. **Curriculum progression**: The trainer automatically advances to the next stage when mean score over 20 episodes exceeds the threshold.
 4. **LoRA (r=16)**: Memory-efficient fine-tuning with Unsloth on a single GPU (A100 40GB). Full training completes in ~4 hours.
 
@@ -278,7 +310,7 @@ pip install "unsloth[cu124-torch240]"
 # SFT warm-start (collect gold data, then fine-tune)
 python -m train.sft_warmstart --mode all --n_episodes 200 --steps 500
 
-# Full GRPO training (5000 steps, 4-stage curriculum)
+# Full GRPO training (5000 steps, 5-stage curriculum)
 python -m train.run_train --model checkpoints/sft --total_steps 5000
 
 # Merge LoRA adapters for deployment
@@ -446,7 +478,7 @@ This pushes beyond simple chatbots toward truly collaborative agent systems capa
 | **#1 Multi-Agent Interactions** | 3-level hierarchy with 11 distinct action types, supervisor review loops, manager overrides |
 | **#2 Instruction Following** | Policy adherence scoring via LLM-as-Judge, mid-episode policy drift forces dynamic compliance |
 | **#3 Professional Tasks** | Real-world Indian enterprise support: UPI payments, GST invoices, KYC rejections, SLA management |
-| **#4 Self-Improvement** | 4-stage curriculum with auto-advancement, before/after training evidence, reward curve analysis |
+| **#4 Self-Improvement** | 5-stage curriculum with auto-advancement, before/after training evidence, reward curve analysis |
 
 ### Who Benefits
 

env/environment.py

@@ -3,6 +3,7 @@ CustomerSupportEnv — core RL environment.
 Supports both single-agent (Round 1) and hierarchical multi-agent (Round 2).
 """
 
+import copy
 import random
 import re
 import uuid
@@ -217,7 +218,7 @@ class CustomerSupportEnv:
             "step": self._step, "max_steps": self._max_steps,
             "sentiment": self._sentiment, "done": self._done,
             "action_log": self._action_log,
-            "retrieved_data": dict(self._retrieved_data),
+            "retrieved_data": copy.deepcopy(self._retrieved_data),
         }
 
     def _build_observation(self) -> Observation:
@@ -232,7 +233,7 @@ class CustomerSupportEnv:
             customer_sentiment=round(self._sentiment, 3),
             mood_trajectory=self._sentiment_history[-3:],
             unresolved_issues=unresolved, is_done=self._done, task=self.task,
-            retrieved_data=dict(self._retrieved_data),
+            retrieved_data=copy.deepcopy(self._retrieved_data),
         )
 
     def _handle_query_action(
@@ -256,10 +257,12 @@ class CustomerSupportEnv:
 
         self._history.append(Message(role="system", content=summary))
 
-        # Minimal reward — actual signal computed in reward_engine via DB signals
-        from env.reward_engine import compute_db_signals
+        # Minimal reward — actual signal computed in reward_engine via DB signals.
+        # Using _clamp_db_total so spam queries can't push the reward above the
+        # normal cap, and so wasted-query penalties are bounded.
+        from env.reward_engine import compute_db_signals, _clamp_db_total
         db_signals = compute_db_signals(action, self._ticket, self._history, self._retrieved_data)
-        raw_db = sum(db_signals.values())
+        raw_db = _clamp_db_total(db_signals)
         import numpy as np
         reward = Reward(
             value=float(np.clip(0.5 + raw_db, 0.0, 1.0)),
@@ -637,7 +640,7 @@ class HierarchicalCustomerSupportEnv(CustomerSupportEnv):
             customer_sentiment=round(self._sentiment, 3),
             mood_trajectory=self._sentiment_history[-3:],
             unresolved_issues=unresolved, is_done=self._done, task=self.task,
-            retrieved_data=dict(self._retrieved_data),
+            retrieved_data=copy.deepcopy(self._retrieved_data),
             active_role=self._active_role.value,
             supervisor_feedback=self._supervisor_feedback,
             manager_directive=self._manager_directive,

env/reward_engine.py

@@ -100,19 +100,23 @@ def _agent_messages(history: List[Message]) -> List[str]:
 
 def compute_loop_penalty(history: List[Message]) -> float:
     """
-    Detects repeated agent messages via TF-IDF cosine similarity
-    AND character-level SequenceMatcher ratio.
-    Returns -0.1 if either threshold exceeded.
+    Detect repeated agent messages. Checks the last 4 agent messages and
+    penalises any pair whose SequenceMatcher ratio exceeds 0.80. This catches
+    both adjacent repeats and alternating-paraphrase loops that try to evade
+    a strict adjacency check.
+
+    Returns -0.12 if any pair is over threshold, else 0.0.
     """
-    agent_msgs = _agent_messages(history)
+    agent_msgs = [m for m in _agent_messages(history) if m and m.strip()]
     if len(agent_msgs) < 2:
         return 0.0
-    last_two = agent_msgs[-2:]
-    if not all(m.strip() for m in last_two):
-        return 0.0
 
-    char_sim = SequenceMatcher(None, last_two[0], last_two[1]).ratio()
-    return -0.1 if char_sim > 0.85 else 0.0
+    window = agent_msgs[-4:]
+    for i in range(len(window)):
+        for j in range(i + 1, len(window)):
+            if SequenceMatcher(None, window[i], window[j]).ratio() > 0.80:
+                return -0.12
+    return 0.0
 
 
 def compute_resolution_score(
@@ -241,15 +245,67 @@ def compute_keyword_stuffing_penalty(message: str) -> float:
 # DB Signal Helper (grounded response rewards / hallucination penalties)
 # ═══════════════════════════════════════════════════════════════════════════════
 
-# Patterns for detecting claimed facts in agent messages
+# Patterns for detecting claimed facts in agent messages. Word-bounded so we
+# can compare against the same pattern extracted from customer text and from DB
+# values, rather than doing raw substring matching (which mis-flags legitimate
+# facts that the customer already mentioned).
 _FACT_PATTERNS = {
     "email": re.compile(r"[\w.+-]+@[\w-]+\.[a-z]{2,}", re.IGNORECASE),
     "order_id": re.compile(r"\b(?:ORD|ORDER|#)[-\s]?[A-Z]{0,3}[-\s]?\d{3,}\b", re.IGNORECASE),
-    "amount": re.compile(r"₹\s*\d+|\brs\.?\s*\d+|\brupees?\s*\d+|\d+\s*(?:rupees?|inr)\b", re.IGNORECASE),
-    "date": re.compile(r"\b\d{4}-\d{2}-\d{2}\b|\b(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\w*\s+\d{1,2}\b", re.IGNORECASE),
+    # Amount: ₹NNN, Rs NNN, NNN rupees, NNN INR — require word boundary so
+    # "approx 500 rupees" matches cleanly and later also matches from customer text.
+    "amount": re.compile(
+        r"(?:₹\s*\d{2,}|\brs\.?\s*\d{2,}\b|\b\d{2,}\s*(?:rupees?|inr)\b|\brupees?\s+\d{2,}\b)",
+        re.IGNORECASE,
+    ),
+    "date": re.compile(
+        r"\b\d{4}-\d{2}-\d{2}\b|\b(?:jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\w*\s+\d{1,2}\b",
+        re.IGNORECASE,
+    ),
 }
 
 
+def _normalise_claim(claim: str) -> str:
+    """Normalise a pattern match so semantically-identical claims compare equal.
+
+    Collapses whitespace, lowercases, and strips the currency symbol from
+    amounts so '₹999' == 'rs 999' == '999 rupees' for comparison purposes.
+    """
+    c = re.sub(r"\s+", "", claim.lower())
+    c = c.replace("₹", "").replace("rs.", "").replace("rs", "")
+    c = c.replace("rupees", "").replace("rupee", "").replace("inr", "")
+    return c.strip()
+
+
+# Hard caps on DB signal totals so no combination of bonuses can dominate the
+# weighted reward sum and push raw above 1.0 before clipping.
+DB_TOTAL_MIN = -0.30
+DB_TOTAL_MAX = 0.25
+
+# Minimum response length (chars) required to earn the grounded-response bonus.
+# Prevents the farm where the agent replies with a single DB field and claims +0.10.
+_GROUNDED_MIN_LEN = 30
+
+# Actions that legitimately "react to not_found" — only these earn the bonus.
+_GOOD_NOT_FOUND_ACTIONS = None  # populated lazily to avoid import cycle
+
+
+def _collect_grounded_values(retrieved_data: dict) -> List[str]:
+    """Flatten every string-ish value from users + orders into one lowercase list."""
+    values: List[str] = []
+    users_data = retrieved_data.get("users", {}) or {}
+    orders_data = retrieved_data.get("orders", {}) or {}
+    for record in list(users_data.values()) + list(orders_data.values()):
+        if not isinstance(record, dict):
+            continue
+        for v in record.values():
+            if isinstance(v, list):
+                values.extend(str(i).lower() for i in v if i is not None)
+            elif v is not None:
+                values.append(str(v).lower())
+    return values
+
+
 def compute_db_signals(
     action: "Action",
     ticket: dict,
@@ -259,11 +315,28 @@ def compute_db_signals(
     """
     Compute DB-grounding reward signals for an action step.
 
-    Returns a dict of signal_name → float (positive = bonus, negative = penalty).
-    Returns all zeros when no DB queries have been made (backward-compatible).
+    Signals (all clamped; absolute values are modest so they never overwhelm
+    the core reward components):
+
+      query_match_bonus       +0.08  — queried the email/order_id the ticket is about
+      wasted_query_penalty    -0.08  — queried an email/order never mentioned by the customer
+      grounded_response_bonus +0.10  — cited verbatim DB data in a substantive (≥30-char) reply
+      not_found_handling_bonus+0.08  — responded to a not_found with REQUEST_INFO / ESCALATE
+      hallucination_penalty   -0.25  — invented a fact (amount, date, email, order-id) that is in
+                                       neither the DB nor the customer's own messages
+
+    Returns all zeros when no DB interaction has happened (backward-compatible).
     """
     from env.models import ActionType
 
+    global _GOOD_NOT_FOUND_ACTIONS
+    if _GOOD_NOT_FOUND_ACTIONS is None:
+        _GOOD_NOT_FOUND_ACTIONS = {
+            ActionType.REQUEST_INFO,
+            ActionType.ESCALATE,
+            ActionType.SUPERVISOR_ESCALATE,
+        }
+
     signals: dict = {
         "query_match_bonus": 0.0,
         "grounded_response_bonus": 0.0,
@@ -273,78 +346,71 @@ def compute_db_signals(
     }
 
     at = ActionType(action.action_type)
-    ticket_email = ticket.get("customer_email", "")
-    ticket_order_ids: List[str] = ticket.get("related_order_ids", [])
-    users_data: dict = retrieved_data.get("users", {})
-    orders_data: dict = retrieved_data.get("orders", {})
+    ticket_email = (ticket.get("customer_email") or "").lower()
+    ticket_order_ids: List[str] = ticket.get("related_order_ids", []) or []
+    users_data: dict = retrieved_data.get("users", {}) or {}
+    orders_data: dict = retrieved_data.get("orders", {}) or {}
+
+    customer_text = " ".join(
+        m.content for m in history if m.role == "customer"
+    )
+    customer_text_lower = customer_text.lower()
+    customer_text_upper = customer_text.upper()
 
-    # ── Query match bonus ──────────────────────────────────────────────────────
+    # ── Query match bonus / wasted-query penalty ──────────────────────────────
     if at == ActionType.QUERY_USER_PROFILE and action.email:
         queried_email = action.email.strip().lower()
-        ticket_email_lower = ticket_email.lower() if ticket_email else ""
-        if ticket_email_lower and queried_email == ticket_email_lower:
+        if ticket_email and queried_email == ticket_email:
             signals["query_match_bonus"] = 0.08
-        elif queried_email and queried_email not in (
-            " ".join(m.content for m in history if m.role == "customer").lower()
-        ):
-            # Email never mentioned by customer — wasted query
+        elif queried_email and queried_email not in customer_text_lower:
             signals["wasted_query_penalty"] = -0.08
 
     if at == ActionType.QUERY_ORDER_DETAILS and action.order_id:
         queried_oid = action.order_id.strip().upper()
         if ticket_order_ids and queried_oid in [o.upper() for o in ticket_order_ids]:
             signals["query_match_bonus"] = 0.08
-        elif queried_oid and queried_oid not in (
-            " ".join(m.content for m in history if m.role == "customer").upper()
-        ):
+        elif queried_oid and queried_oid not in customer_text_upper:
             signals["wasted_query_penalty"] = -0.08
 
     # ── Not-found handling bonus ───────────────────────────────────────────────
-    _GOOD_NOT_FOUND_ACTIONS = {
-        ActionType.REQUEST_INFO, ActionType.ESCALATE, ActionType.SUPERVISOR_ESCALATE,
-    }
+    # Only award when retrieved_data contains at least one "not_found" sentinel
+    # AND the current action is a recovery move (ask for info or escalate).
     any_not_found = (
         "not_found" in users_data.values() or "not_found" in orders_data.values()
     )
     if any_not_found and at in _GOOD_NOT_FOUND_ACTIONS:
         signals["not_found_handling_bonus"] = 0.08
 
-    # ── Grounded response bonus & hallucination penalty ────────────────────────
-    msg_text = (action.message or action.reason or "").lower()
-    if msg_text and at in {
+    # ── Grounded response & hallucination (only on response-like actions) ─────
+    response_actions = {
         ActionType.RESPOND, ActionType.CLOSE, ActionType.REQUEST_INFO,
         ActionType.MANAGER_OVERRIDE, ActionType.MANAGER_RESOLVE,
-    }:
-        # Collect all verbatim field values from retrieved_data
-        grounded_values: List[str] = []
-        for user_record in users_data.values():
-            if isinstance(user_record, dict):
-                grounded_values.extend(str(v).lower() for v in user_record.values())
-        for order_record in orders_data.values():
-            if isinstance(order_record, dict):
-                for v in order_record.values():
-                    if isinstance(v, list):
-                        grounded_values.extend(str(i).lower() for i in v)
-                    else:
-                        grounded_values.append(str(v).lower())
-
-        if grounded_values:
-            # Bonus: message cites a verbatim value from retrieved_data
-            grounded = any(gv in msg_text for gv in grounded_values if len(gv) > 3)
-            if grounded:
+    }
+    msg_text_raw = action.message or action.reason or ""
+    msg_text = msg_text_raw.lower()
+
+    if msg_text and at in response_actions:
+        grounded_values = _collect_grounded_values(retrieved_data)
+
+        # Grounded-response bonus: the reply must be substantive (≥30 chars)
+        # AND cite at least one verbatim DB value (≥4 chars, to avoid rewarding
+        # trivia like `499` that any reply could include).
+        if grounded_values and len(msg_text_raw.strip()) >= _GROUNDED_MIN_LEN:
+            if any(len(gv) >= 4 and gv in msg_text for gv in grounded_values):
                 signals["grounded_response_bonus"] = 0.10
 
-        # Penalty: message contains a specific fact pattern NOT in retrieved_data
-        # and NOT mentioned by the customer
-        customer_text = " ".join(
-            m.content for m in history if m.role == "customer"
-        ).lower()
-        all_known_text = customer_text + " " + " ".join(grounded_values)
-
-        for pat_name, pat in _FACT_PATTERNS.items():
-            for match in pat.finditer(msg_text):
-                claimed = match.group(0).lower()
-                if claimed not in all_known_text:
+        # Hallucination: a fact pattern appears in the agent's text that is
+        # present in neither the customer's messages nor the DB values. Compare
+        # via _normalise_claim so "₹999" == "rs 999" == "999 rupees".
+        known_claims: set[str] = set()
+        for text in (customer_text_lower, " ".join(grounded_values)):
+            for pat in _FACT_PATTERNS.values():
+                for m in pat.finditer(text):
+                    known_claims.add(_normalise_claim(m.group(0)))
+
+        for pat in _FACT_PATTERNS.values():
+            for m in pat.finditer(msg_text):
+                if _normalise_claim(m.group(0)) not in known_claims:
                     signals["hallucination_penalty"] = -0.25
                     break
             if signals["hallucination_penalty"] < 0:
@@ -353,6 +419,13 @@ def compute_db_signals(
     return signals
 
 
+def _clamp_db_total(signals: dict) -> float:
+    """Sum signals and clamp to [DB_TOTAL_MIN, DB_TOTAL_MAX] so the bucket can't
+    overwhelm the weighted-sum reward components."""
+    total = sum(v for v in signals.values() if isinstance(v, (int, float)))
+    return float(np.clip(total, DB_TOTAL_MIN, DB_TOTAL_MAX))
+
+
 # ═══════════════════════════════════════════════════════════════════════════════
 # Single-Agent Step Reward (Round 1 backward compat)
 # ═══════════════════════════════════════════════════════════════════════════════
@@ -396,7 +469,7 @@ def compute_step_reward(
     # DB grounding signals (all zero when no queries made)
     _rd = retrieved_data or {"users": {}, "orders": {}}
     db_signals = compute_db_signals(action, ticket, history, _rd)
-    db_total = sum(db_signals.values())
+    db_total = _clamp_db_total(db_signals)
 
     raw = (
         0.40 * resolution_score
@@ -575,7 +648,7 @@ def compute_hierarchy_reward(
     # ── DB grounding signals ───────────────────────────────────────────────────
     _rd = retrieved_data or {"users": {}, "orders": {}}
     db_signals = compute_db_signals(action, ticket, history, _rd)
-    db_total = sum(db_signals.values())
+    db_total = _clamp_db_total(db_signals)
 
     # ── Compute overall raw reward ─────────────────────────────────────────────
     if is_terminal:
@@ -625,20 +698,36 @@ def compute_hierarchy_reward(
     )
     role_rewards["support_agent"] = float(np.clip(l1_raw, 0.0, 1.0))
 
-    # Supervisor role reward
+    # Supervisor role reward — escalation_penalty and unnecessary_manager_penalty
+    # are both ≤0; we clamp their sum first so a single bad call can drag the
+    # supervisor score down by at most 0.5 (the blended term below still leaves
+    # room for a recovery signal).
+    sup_decision_score = float(np.clip(
+        1.0 + escalation_penalty + unnecessary_manager_penalty, 0.0, 1.0
+    ))
     l2_raw = (
         0.35 * oversight_score
-        + 0.30 * (1.0 + escalation_penalty + unnecessary_manager_penalty)
+        + 0.30 * sup_decision_score
         + 0.20 * policy_adherence_score
         + 0.15 * (1.0 if steps_used <= ideal_steps else 0.5)
     )
     role_rewards["supervisor"] = float(np.clip(l2_raw, 0.0, 1.0))
 
-    # Manager role reward
+    # Manager role reward. Previously a flat "terminal or zero" term punished
+    # legitimate MANAGER_SEND_BACK actions. We now use decision quality (from
+    # the LLM judge) as the primary signal and give a smaller bonus for actually
+    # resolving vs deferring, without crushing the score for a correct deferral.
+    manager_act_type = ActionType(action.action_type) if hasattr(action, "action_type") else None
+    if manager_act_type in {ActionType.MANAGER_RESOLVE, ActionType.MANAGER_OVERRIDE}:
+        resolve_bonus = 1.0
+    elif manager_act_type == ActionType.MANAGER_SEND_BACK:
+        resolve_bonus = 0.7  # legitimate deferral, still valuable
+    else:
+        resolve_bonus = 0.5  # non-manager action, neutral default
     l3_raw = (
-        0.40 * decision_quality_score
+        0.45 * decision_quality_score
         + 0.30 * (resolution_llm_score if is_terminal else 0.5)
-        + 0.30 * (1.0 if is_terminal else 0.0)
+        + 0.25 * resolve_bonus
     )
     role_rewards["manager"] = float(np.clip(l3_raw, 0.0, 1.0))
 

frontend/src/app/api/ai-action/route.ts

@@ -12,14 +12,18 @@ ABOVE THEM: A Manager handles escalated complex cases.
 {supervisor_feedback_section}{manager_directive_section}{policy_section}
 
 ACTION TYPES — output exactly one per step:
-- "respond"      → send a message to the customer  → requires: "message"
-- "request_info" → ask for missing information      → requires: "message"
-- "close"        → close the ticket as resolved     → requires: "message"
-- "escalate"     → hand off to specialist           → requires: "reason"
+- "respond"             → send a message to the customer   → requires: "message"
+- "request_info"        → ask for missing information       → requires: "message"
+- "close"               → close the ticket as resolved      → requires: "message"
+- "escalate"            → hand off to specialist            → requires: "reason"
+- "query_user_profile"  → look up customer DB (internal)    → requires: "email"
+- "query_order_details" → look up order DB (internal)       → requires: "order_id"
 
 SCORING: Empathy(30%) + Accuracy(25%) + Resolution(25%) + Efficiency(20%)
 Be warm, gather info from "Unresolved issues", use specific resolution language.
 If supervisor gave feedback, INCORPORATE it into your next action.
+If the ticket references an email or order-id you haven't looked up yet, query
+the DB first — never invent facts not present in KNOWN DATA or the conversation.
 
 OUTPUT FORMAT — return ONLY this JSON, no code fences, no explanation:
 {"action_type": "respond", "message": "..."} or {"action_type": "escalate", "reason": "..."}`;
@@ -110,12 +114,29 @@ function buildUserPrompt(obs: Observation, virtualMessages?: Message[]): string
   const unresolved = obs.unresolved_issues.join(", ") || "none";
   const envEvent = obs.environment_event ? `\n⚠️ POLICY EVENT: ${obs.environment_event}\n` : "";
 
+  // Surface any DB lookups the agent has already made so the LLM can cite
+  // verbatim values instead of hallucinating new ones.
+  const retrieved = obs.retrieved_data;
+  const userEntries = retrieved?.users ? Object.entries(retrieved.users) : [];
+  const orderEntries = retrieved?.orders ? Object.entries(retrieved.orders) : [];
+  let knownData = "";
+  if (userEntries.length > 0 || orderEntries.length > 0) {
+    knownData =
+      "\n## KNOWN DATA (from internal DB — use verbatim, do NOT invent other facts)\n";
+    for (const [email, record] of userEntries.slice(0, 4)) {
+      knownData += `User(${email}): ${JSON.stringify(record)}\n`;
+    }
+    for (const [oid, record] of orderEntries.slice(0, 4)) {
+      knownData += `Order(${oid}): ${JSON.stringify(record)}\n`;
+    }
+  }
+
   return (
     `Ticket: ${obs.subject}\n` +
     `Category: ${obs.category} | Priority: ${obs.priority} | ` +
     `Step: ${obs.step}/${obs.max_steps}\n` +
     `Customer sentiment: ${obs.customer_sentiment.toFixed(2)}\n` +
-    `Unresolved issues: ${unresolved}${envEvent}\n\n` +
+    `Unresolved issues: ${unresolved}${envEvent}${knownData}\n\n` +
     `Conversation:\n${historyText}\n\n` +
     `What is your next action? Output JSON only.`
   );

frontend/src/app/demo/page.tsx

@@ -17,6 +17,7 @@ import { CustomerChatInput } from "@/components/demo/CustomerChatInput";
 // Sidebar panels
 import { TicketInfoPanel } from "@/components/panels/TicketInfoPanel";
 import { HierarchyPanel } from "@/components/panels/HierarchyPanel";
+import { RetrievedDataPanel } from "@/components/panels/RetrievedDataPanel";
 import { PolicyContextPanel } from "@/components/panels/PolicyContextPanel";
 import { FrustrationMeter } from "@/components/indicators/FrustrationMeter";
 import { ActiveRoleIndicator } from "@/components/indicators/ActiveRoleIndicator";
@@ -208,6 +209,10 @@ export default function DemoPage() {
                 </div>
               )}
 
+              <div className="border-t border-neutral-800 pt-3">
+                <RetrievedDataPanel observation={observation} />
+              </div>
+
               {observation.unresolved_issues.length > 0 && (
                 <div className="space-y-1.5">
                   <span className="text-[10px] text-neutral-500 uppercase tracking-wider">

frontend/src/components/action/SupportAgentActions.tsx

@@ -8,32 +8,75 @@ interface Props {
   isLoading: boolean;
 }
 
-type Mode = "respond" | "request_info" | "escalate" | "close" | null;
+type Mode =
+  | "respond"
+  | "request_info"
+  | "escalate"
+  | "close"
+  | "query_user"
+  | "query_order"
+  | null;
 
 export function SupportAgentActions({ isLoading }: Props) {
   const { submitStep } = useSessionStore();
   const [mode, setMode] = useState<Mode>(null);
   const [message, setMessage] = useState("");
   const [reason, setReason] = useState("");
+  const [email, setEmail] = useState("");
+  const [orderId, setOrderId] = useState("");
+
+  const reset = () => {
+    setMode(null);
+    setMessage("");
+    setReason("");
+    setEmail("");
+    setOrderId("");
+  };
 
   const toggleMode = (m: Exclude<Mode, null>) => {
     setMode((prev) => (prev === m ? null : m));
     setMessage("");
     setReason("");
+    setEmail("");
+    setOrderId("");
   };
 
   const submit = async (action: Action) => {
     await submitStep(action);
-    setMode(null);
-    setMessage("");
-    setReason("");
+    reset();
   };
 
   const actionButtons = [
-    { id: "respond" as const, label: "Respond", color: "bg-indigo-600 hover:bg-indigo-500 text-white" },
-    { id: "request_info" as const, label: "Request Info", color: "bg-neutral-700 hover:bg-neutral-600 text-neutral-100" },
-    { id: "escalate" as const, label: "Escalate", color: "bg-orange-600/80 hover:bg-orange-600 text-white" },
-    { id: "close" as const, label: "Close Ticket", color: "bg-green-700/80 hover:bg-green-700 text-white" },
+    {
+      id: "respond" as const,
+      label: "Respond",
+      color: "bg-indigo-600 hover:bg-indigo-500 text-white",
+    },
+    {
+      id: "request_info" as const,
+      label: "Request Info",
+      color: "bg-neutral-700 hover:bg-neutral-600 text-neutral-100",
+    },
+    {
+      id: "query_user" as const,
+      label: "Query User DB",
+      color: "bg-cyan-700/80 hover:bg-cyan-600 text-white",
+    },
+    {
+      id: "query_order" as const,
+      label: "Query Order DB",
+      color: "bg-teal-700/80 hover:bg-teal-600 text-white",
+    },
+    {
+      id: "escalate" as const,
+      label: "Escalate",
+      color: "bg-orange-600/80 hover:bg-orange-600 text-white",
+    },
+    {
+      id: "close" as const,
+      label: "Close Ticket",
+      color: "bg-green-700/80 hover:bg-green-700 text-white",
+    },
   ];
 
   return (
@@ -71,7 +114,9 @@ export function SupportAgentActions({ isLoading }: Props) {
           />
           <button
             disabled={!message.trim() || isLoading}
-            onClick={() => submit({ action_type: "respond", message, role: "support_agent" })}
+            onClick={() =>
+              submit({ action_type: "respond", message, role: "support_agent" })
+            }
             className="self-end px-3 py-2 bg-indigo-600 hover:bg-indigo-500 text-white text-xs rounded
                        font-medium transition-colors disabled:opacity-40 disabled:cursor-not-allowed"
           >
@@ -93,7 +138,9 @@ export function SupportAgentActions({ isLoading }: Props) {
           />
           <button
             disabled={!reason.trim() || isLoading}
-            onClick={() => submit({ action_type: "escalate", reason, role: "support_agent" })}
+            onClick={() =>
+              submit({ action_type: "escalate", reason, role: "support_agent" })
+            }
             className="px-3 py-2 bg-orange-600 hover:bg-orange-500 text-white text-xs rounded font-medium
                        transition-colors disabled:opacity-40 disabled:cursor-not-allowed"
           >
@@ -103,14 +150,83 @@ export function SupportAgentActions({ isLoading }: Props) {
       )}
 
       {mode === "request_info" && (
-        <button
-          disabled={isLoading}
-          onClick={() => submit({ action_type: "request_info", role: "support_agent" })}
-          className="px-3 py-2 bg-neutral-700 hover:bg-neutral-600 text-neutral-100 text-xs rounded
-                     font-medium transition-colors disabled:opacity-40 disabled:cursor-not-allowed"
-        >
-          Send Information Request
-        </button>
+        <div className="flex gap-2">
+          <input
+            placeholder="What info do you need from the customer?"
+            value={message}
+            onChange={(e) => setMessage(e.target.value)}
+            disabled={isLoading}
+            className="flex-1 bg-neutral-900 border border-neutral-700 rounded px-3 py-2 text-sm text-neutral-100
+                       placeholder:text-neutral-600 focus:outline-none focus:ring-1 focus:ring-indigo-500
+                       disabled:opacity-50"
+          />
+          <button
+            disabled={!message.trim() || isLoading}
+            onClick={() =>
+              submit({ action_type: "request_info", message, role: "support_agent" })
+            }
+            className="px-3 py-2 bg-neutral-700 hover:bg-neutral-600 text-neutral-100 text-xs rounded
+                       font-medium transition-colors disabled:opacity-40 disabled:cursor-not-allowed"
+          >
+            Send
+          </button>
+        </div>
+      )}
+
+      {mode === "query_user" && (
+        <div className="flex gap-2">
+          <input
+            placeholder="user@example.com — look up the account"
+            value={email}
+            onChange={(e) => setEmail(e.target.value)}
+            disabled={isLoading}
+            className="flex-1 bg-neutral-900 border border-cyan-700/50 rounded px-3 py-2 text-sm text-neutral-100
+                       placeholder:text-neutral-600 focus:outline-none focus:ring-1 focus:ring-cyan-500
+                       disabled:opacity-50 font-mono"
+          />
+          <button
+            disabled={!email.trim() || isLoading}
+            onClick={() =>
+              submit({
+                action_type: "query_user_profile",
+                email: email.trim(),
+                role: "support_agent",
+              })
+            }
+            className="px-3 py-2 bg-cyan-700 hover:bg-cyan-600 text-white text-xs rounded font-medium
+                       transition-colors disabled:opacity-40 disabled:cursor-not-allowed"
+          >
+            Query
+          </button>
+        </div>
+      )}
+
+      {mode === "query_order" && (
+        <div className="flex gap-2">
+          <input
+            placeholder="ORD-FD-8821 — look up the order"
+            value={orderId}
+            onChange={(e) => setOrderId(e.target.value)}
+            disabled={isLoading}
+            className="flex-1 bg-neutral-900 border border-teal-700/50 rounded px-3 py-2 text-sm text-neutral-100
+                       placeholder:text-neutral-600 focus:outline-none focus:ring-1 focus:ring-teal-500
+                       disabled:opacity-50 font-mono"
+          />
+          <button
+            disabled={!orderId.trim() || isLoading}
+            onClick={() =>
+              submit({
+                action_type: "query_order_details",
+                order_id: orderId.trim(),
+                role: "support_agent",
+              })
+            }
+            className="px-3 py-2 bg-teal-700 hover:bg-teal-600 text-white text-xs rounded font-medium
+                       transition-colors disabled:opacity-40 disabled:cursor-not-allowed"
+          >
+            Query
+          </button>
+        </div>
       )}
 
       {mode === "close" && (
@@ -120,7 +236,9 @@ export function SupportAgentActions({ isLoading }: Props) {
           </p>
           <button
             disabled={isLoading}
-            onClick={() => submit({ action_type: "close", role: "support_agent" })}
+            onClick={() =>
+              submit({ action_type: "close", role: "support_agent" })
+            }
             className="px-3 py-1.5 bg-green-700 hover:bg-green-600 text-white text-xs rounded font-medium
                        transition-colors disabled:opacity-40 disabled:cursor-not-allowed flex-shrink-0"
           >

frontend/src/components/panels/RetrievedDataPanel.tsx

@@ -0,0 +1,84 @@
+import type { DbRecord, Observation } from "@/types";
+
+interface Props {
+  observation: Observation;
+}
+
+function RecordCard({
+  kind,
+  id,
+  record,
+}: {
+  kind: "user" | "order";
+  id: string;
+  record: DbRecord;
+}) {
+  const isNotFound = record === "not_found";
+  const badge =
+    kind === "user"
+      ? "bg-cyan-500/10 border-cyan-500/30 text-cyan-300"
+      : "bg-teal-500/10 border-teal-500/30 text-teal-300";
+
+  return (
+    <div className="rounded border border-neutral-800 bg-neutral-900/60 p-2 text-[11px]">
+      <div className="flex items-center justify-between gap-2 mb-1">
+        <span
+          className={`rounded border px-1.5 py-0.5 text-[9px] font-mono uppercase tracking-wider ${badge}`}
+        >
+          {kind}
+        </span>
+        <span className="font-mono text-neutral-400 truncate" title={id}>
+          {id}
+        </span>
+      </div>
+      {isNotFound ? (
+        <p className="font-mono text-amber-400">not_found</p>
+      ) : (
+        <div className="space-y-0.5">
+          {Object.entries(record as Record<string, unknown>)
+            .slice(0, 8)
+            .map(([k, v]) => (
+              <div key={k} className="flex gap-2 font-mono text-neutral-300">
+                <span className="text-neutral-500">{k}:</span>
+                <span className="truncate">{JSON.stringify(v)}</span>
+              </div>
+            ))}
+          {Object.keys(record as Record<string, unknown>).length > 8 && (
+            <p className="text-neutral-600 italic">
+              … {Object.keys(record as Record<string, unknown>).length - 8} more fields
+            </p>
+          )}
+        </div>
+      )}
+    </div>
+  );
+}
+
+export function RetrievedDataPanel({ observation }: Props) {
+  const retrieved = observation.retrieved_data ?? { users: {}, orders: {} };
+  const users = Object.entries(retrieved.users ?? {});
+  const orders = Object.entries(retrieved.orders ?? {});
+
+  if (users.length === 0 && orders.length === 0) return null;
+
+  return (
+    <div className="space-y-2">
+      <div className="flex items-center justify-between">
+        <span className="text-[10px] text-neutral-500 uppercase tracking-wider">
+          DB Lookups
+        </span>
+        <span className="text-[10px] text-neutral-600 font-mono">
+          {users.length + orders.length} record{users.length + orders.length === 1 ? "" : "s"}
+        </span>
+      </div>
+      <div className="space-y-1.5">
+        {users.map(([email, record]) => (
+          <RecordCard key={`u-${email}`} kind="user" id={email} record={record} />
+        ))}
+        {orders.map(([oid, record]) => (
+          <RecordCard key={`o-${oid}`} kind="order" id={oid} record={record} />
+        ))}
+      </div>
+    </div>
+  );
+}

frontend/src/lib/constants.ts

@@ -123,6 +123,16 @@ export const TASK_CONFIG: Record<
     hinglishEnabled: true,
     description: "Stage 4 — adversarial + Hinglish",
   },
+  multi_domain: {
+    label: "Multi-Domain (DB)",
+    maxSteps: 8,
+    levels: [1],
+    driftProbability: 0,
+    difficulty: "hard",
+    hierarchical: true,
+    hinglishEnabled: false,
+    description: "Query user/order DB, cite grounded data, avoid hallucination",
+  },
 };
 
 export const DIFFICULTY_COLORS = {

frontend/src/types/index.ts

@@ -9,13 +9,16 @@ export type TaskName =
   | "curriculum_basic"
   | "curriculum_supervisor"
   | "curriculum_full_hierarchy"
-  | "curriculum_nightmare";
+  | "curriculum_nightmare"
+  | "multi_domain";
 
 export type ActionType =
   | "respond"
   | "escalate"
   | "close"
   | "request_info"
+  | "query_user_profile"
+  | "query_order_details"
   | "supervisor_approve"
   | "supervisor_reject"
   | "supervisor_feedback"
@@ -24,6 +27,15 @@ export type ActionType =
   | "manager_resolve"
   | "manager_send_back";
 
+// Record value from a DB lookup — either the resolved dict or the literal
+// "not_found" sentinel when the email / order_id was not in the DB.
+export type DbRecord = Record<string, unknown> | "not_found";
+
+export interface RetrievedData {
+  users: Record<string, DbRecord>;
+  orders: Record<string, DbRecord>;
+}
+
 export type AgentRole = "support_agent" | "supervisor" | "manager";
 
 export type Priority = "low" | "medium" | "high" | "critical";
@@ -67,6 +79,7 @@ export interface Observation {
   environment_event: string | null;
   policy_context: string;
   escalation_chain: string[];
+  retrieved_data?: RetrievedData;
 }
 
 export interface Reward {
@@ -90,6 +103,9 @@ export interface Action {
   role?: AgentRole;
   internal_note?: string | null;
   feedback_to_agent?: string | null;
+  // DB query fields (only used by query_user_profile / query_order_details)
+  email?: string | null;
+  order_id?: string | null;
 }
 
 export interface ActionLogEntry {

openenv.yaml

@@ -1,33 +1,44 @@
 name: customer-support-env
-version: "2.1.0"
+version: "2.2.0"
 description: >
   Hierarchical multi-agent RL environment for AI customer support training.
   Features a 3-level agent hierarchy (Support Agent → Supervisor → Manager),
   LLM-driven customer simulator with Hinglish support, dynamic policy drift,
-  hybrid dense rewards (rule-based + LLM-as-Judge), and a progressive
-  4-stage curriculum that systematically increases complexity for RL training.
+  hybrid dense rewards (rule-based + LLM-as-Judge), an in-memory DB across
+  three domains (food delivery, e-commerce, ticket booking) with grounded
+  response rewards + hallucination penalties, and a progressive 5-stage
+  curriculum that systematically increases complexity for RL training.
   Backward compatible with Round 1 single-agent tasks.
 author: Team X-Force
 curriculum:
   description: >
-    Progressive 4-stage curriculum designed for RL training. Easy tasks provide
+    Progressive 5-stage curriculum designed for RL training. Easy tasks provide
     dense initial rewards so models learn basic behavior. Each subsequent level
     introduces new agents, mechanisms, and challenges. Agents trained on lower
     levels transfer skills upward — empathy from L1-only → oversight from L2 →
-    coordination from full hierarchy → adversarial resilience from nightmare.
+    coordination from full hierarchy → adversarial resilience from nightmare →
+    DB-grounded reasoning from multi_domain.
   stages:
     - stage: 1
       task: curriculum_basic
       objective: "Learn empathy, info-gathering, and resolution fundamentals"
+      advance_threshold: 0.65
     - stage: 2
       task: curriculum_supervisor
       objective: "Learn to accept feedback and improve actions iteratively"
+      advance_threshold: 0.60
     - stage: 3
       task: curriculum_full_hierarchy
       objective: "Coordinate across all 3 levels under policy drift"
+      advance_threshold: 0.55
     - stage: 4
       task: curriculum_nightmare
       objective: "Handle extreme adversarial conditions with Hinglish, multi-drift, and high frustration"
+      advance_threshold: 0.50
+    - stage: 5
+      task: multi_domain
+      objective: "Query a DB for real user/order data, cite grounded facts verbatim, avoid hallucination"
+      advance_threshold: null   # final stage — no further advancement
 tasks:
   # ── Round 1: Single-agent tasks (backward compatible) ──────────────────────
   - name: easy
@@ -187,18 +198,34 @@ observation_space:
     escalation_reason: "string | null"
 reward_range: [0.0, 1.0]
 reward_signals:
-  # Single-agent mode
-  resolution: 0.40
-  tone: 0.20
-  efficiency: 0.20
-  accuracy: 0.20
-  # Hierarchy mode (additional)
-  empathy: "LLM-as-Judge"
-  policy_adherence: "LLM-as-Judge"
-  oversight_quality: "LLM-as-Judge"
-  decision_quality: "LLM-as-Judge"
-  hierarchy_effectiveness: "Rule-based"
-  sla_compliance: "Rule-based"
+  # Single-agent mode (Round 1)
+  resolution: 0.40            # keyword-grounded resolution score on terminal
+  tone: 0.20                  # VADER sentiment on agent messages
+  efficiency: 0.20            # steps_used vs max_steps
+  accuracy: 0.20              # required_info_before_close found in conversation
+  # Hierarchy mode (Round 2, overall weights sum to 1.0 at terminal step)
+  empathy: "LLM-as-Judge"              # 0.15 weight
+  policy_adherence: "LLM-as-Judge"     # 0.15 weight
+  oversight_quality: "LLM-as-Judge"    # supervisor role_reward input
+  decision_quality: "LLM-as-Judge"     # manager role_reward input
+  hierarchy_effectiveness: "Rule-based"# 0.10 weight
+  sla_compliance: "Rule-based"         # 0.15 weight
+  # DB grounding signals (Round 2, multi_domain task)
+  query_match_bonus: "+0.08"           # queried the ticket's email/order
+  wasted_query_penalty: "-0.08"        # queried something the customer never mentioned
+  grounded_response_bonus: "+0.10"     # cited verbatim DB data in a substantive reply (≥30 chars)
+  not_found_handling_bonus: "+0.08"    # responded to a not_found with REQUEST_INFO / ESCALATE
+  hallucination_penalty: "-0.25"       # invented a fact (amount/date/email/order) not in DB or customer text
+  # Anti-gaming penalties (apply across all tasks)
+  loop_penalty: "-0.12"                # any pair of the last 4 agent messages >80% similar
+  keyword_stuffing_penalty: "-0.30"    # reward-keyword density >20% without substance
+  contradiction_penalty: "-0.15"       # claimed resolution then asked for info
+  escalation_penalty: "-0.30"          # escalated a low/medium priority ticket
+  ignored_feedback_penalty: "-0.15"    # L1 did not incorporate supervisor feedback
+  unnecessary_manager_penalty: "-0.20" # L2 escalated to L3 on a low/medium ticket
+reward_clamping:
+  db_total: "[-0.30, +0.25]"           # sum of all DB signals is clamped before joining raw
+  raw_to_value: "[0.0, 1.0]"           # final Reward.value is always clamped to [0,1]
 tags:
   - customer-support
   - multi-agent

tests/test_env.py

@@ -183,7 +183,7 @@ def test_loop_penalty_identical_messages():
         Message(role="agent", content="I understand your frustration. Please bear with us."),
     ]
     penalty = compute_loop_penalty(history)
-    assert penalty == -0.1
+    assert penalty == -0.12
 
 def test_loop_penalty_different_messages():
     history = [
@@ -370,9 +370,9 @@ def test_hierarchy_state_includes_hierarchy(hierarchy_env):
 
 @pytest.fixture(scope="module")
 def client():
-    from server.app import app, _sessions
+    from server.app import app, _sessions, EXPECTED_API_KEY
     _sessions.clear()
-    with TestClient(app, raise_server_exceptions=False) as c:
+    with TestClient(app, raise_server_exceptions=False, headers={"X-API-Key": EXPECTED_API_KEY}) as c:
         yield c
     _sessions.clear()
 

train/grpo_trainer.py

@@ -85,13 +85,25 @@ def grpo_loss(
             # ── Old log-probs (at generation time, no gradient) ───────────────
             if step.log_probs is not None and isinstance(step.log_probs, torch.Tensor):
                 old_lp = step.log_probs.to(device).detach()
-                # Align lengths (generation-time vs recomputed may differ slightly)
+                # Align lengths (generation-time vs recomputed may differ slightly).
+                # Warn once per training run if divergence exceeds 10% — this usually
+                # signals a tokenizer/prompt mismatch between rollout and loss.
+                if abs(cur_lp.shape[0] - old_lp.shape[0]) > max(4, int(0.10 * max(cur_lp.shape[0], 1))):
+                    if not getattr(grpo_loss, "_logprob_warned", False):
+                        print(
+                            f"[GRPO] warn: large log-prob length mismatch "
+                            f"cur={cur_lp.shape[0]} old={old_lp.shape[0]}; "
+                            f"check that tokenizer/prompt match between rollout and loss recompute"
+                        )
+                        grpo_loss._logprob_warned = True  # type: ignore[attr-defined]
                 min_len = min(cur_lp.shape[0], old_lp.shape[0])
                 cur_lp_aligned = cur_lp[:min_len]
                 old_lp_aligned = old_lp[:min_len]
             else:
+                # Generation failed or fallback action was used. Ratio=1 means
+                # this step contributes no policy-gradient signal (KL term only).
                 cur_lp_aligned = cur_lp
-                old_lp_aligned = cur_lp.detach()   # ratio = 1, no update
+                old_lp_aligned = cur_lp.detach()
 
             # ── Reference model log-probs (no gradient) ───────────────────────
             with torch.no_grad():

train/prompt_builder.py

@@ -126,13 +126,36 @@ OUTPUT FORMAT — return ONLY this JSON:
 
 # ── Prompt builders ────────────────────────────────────────────────────────────
 
+# Hard caps to keep the user-context block from eating the generation budget.
+# Cover long multi-turn episodes (30+ messages) without overflowing 4096 tokens.
+_MAX_HISTORY_TURNS = 12       # keep at most the last N messages
+_MAX_DB_RECORDS_EACH = 4      # show at most N user and N order records
+_HISTORY_HEAD_KEEP = 2        # always keep the first 2 messages (customer opener + first agent reply)
+
+
 def _build_user_context(obs: Dict[str, Any]) -> str:
-    """Build the shared user context block (ticket + conversation)."""
+    """Build the shared user context block (ticket + conversation + DB data).
+
+    Truncates long histories and large DB result sets so the prompt never
+    overflows the generation budget. Truncation is visible to the model via
+    ellipsis markers so it can reason about the elision.
+    """
     import json as _json
-    history_text = "\n".join(
-        f"{m['role'].upper()}: {m['content']}"
-        for m in obs.get("conversation_history", [])
-    )
+
+    history = obs.get("conversation_history", []) or []
+    if len(history) > _MAX_HISTORY_TURNS:
+        head = history[:_HISTORY_HEAD_KEEP]
+        tail = history[-(_MAX_HISTORY_TURNS - _HISTORY_HEAD_KEEP):]
+        skipped = len(history) - len(head) - len(tail)
+        history_lines = [f"{m['role'].upper()}: {m['content']}" for m in head]
+        history_lines.append(f"… [{skipped} earlier messages omitted] …")
+        history_lines.extend(f"{m['role'].upper()}: {m['content']}" for m in tail)
+        history_text = "\n".join(history_lines)
+    else:
+        history_text = "\n".join(
+            f"{m['role'].upper()}: {m['content']}" for m in history
+        )
+
     unresolved = ", ".join(obs.get("unresolved_issues", [])) or "none"
     env_event = obs.get("environment_event")
 
@@ -146,18 +169,24 @@ def _build_user_context(obs: Dict[str, Any]) -> str:
     if env_event:
         ctx += f"\n⚠️ ENVIRONMENT EVENT: {env_event}\n"
 
-    # Show DB-retrieved data when present
-    retrieved = obs.get("retrieved_data", {})
-    has_users = bool(retrieved.get("users"))
-    has_orders = bool(retrieved.get("orders"))
-    if has_users or has_orders:
+    # Show DB-retrieved data when present. Cap the number of records shown so
+    # a zealous agent that queried many emails/orders doesn't push conversation
+    # history out of context.
+    retrieved = obs.get("retrieved_data", {}) or {}
+    users = retrieved.get("users") or {}
+    orders = retrieved.get("orders") or {}
+    if users or orders:
         ctx += "\n## KNOWN DATA (from internal DB — use verbatim, do NOT invent other facts)\n"
-        if has_users:
-            for email, record in retrieved["users"].items():
-                ctx += f"User({email}): {_json.dumps(record, ensure_ascii=False)}\n"
-        if has_orders:
-            for oid, record in retrieved["orders"].items():
-                ctx += f"Order({oid}): {_json.dumps(record, ensure_ascii=False)}\n"
+        user_items = list(users.items())[:_MAX_DB_RECORDS_EACH]
+        for email, record in user_items:
+            ctx += f"User({email}): {_json.dumps(record, ensure_ascii=False)}\n"
+        if len(users) > _MAX_DB_RECORDS_EACH:
+            ctx += f"… ({len(users) - _MAX_DB_RECORDS_EACH} more user records truncated) …\n"
+        order_items = list(orders.items())[:_MAX_DB_RECORDS_EACH]
+        for oid, record in order_items:
+            ctx += f"Order({oid}): {_json.dumps(record, ensure_ascii=False)}\n"
+        if len(orders) > _MAX_DB_RECORDS_EACH:
+            ctx += f"… ({len(orders) - _MAX_DB_RECORDS_EACH} more order records truncated) …\n"
 
     ctx += f"\nConversation:\n{history_text}\n\nOutput JSON only."
     return ctx

train/reward_aggregator.py

@@ -90,12 +90,27 @@ def grpo_advantages(rewards: List[float], eps: float = 1e-8) -> List[float]:
 
     A_i = (R_i − μ) / (σ + ε)
 
+    Robustness: when rewards contain obvious invalid-episode sentinels (values
+    well below the usual range, e.g. -0.5), computing μ/σ over the *full* group
+    pulls valid episodes' advantages artificially larger. We instead compute μ/σ
+    over the valid subset when we detect clear outliers, but still return an
+    advantage for every input index (invalid episodes get their literal
+    deviation, which is correctly negative).
+
     If all rewards in the group are identical (σ = 0), all advantages are 0.
     """
     if not rewards:
         return []
     n = len(rewards)
-    mu = sum(rewards) / n
-    var = sum((r - mu) ** 2 for r in rewards) / n
+    # Detect invalid sentinels: rewards that are significantly below the
+    # minimum of the "normal" range. Our invalid_penalty defaults to -0.5;
+    # anything at or below that is treated as an outlier for μ/σ estimation.
+    valid = [r for r in rewards if r > -0.5 + 1e-6]
+    if 0 < len(valid) < n:
+        mu = sum(valid) / len(valid)
+        var = sum((r - mu) ** 2 for r in valid) / len(valid)
+    else:
+        mu = sum(rewards) / n
+        var = sum((r - mu) ** 2 for r in rewards) / n
     sigma = var ** 0.5
     return [(r - mu) / (sigma + eps) for r in rewards]