docs/walkthrough.md

Meta OpenEnv Hackathon — Round 2 Upgrade Documentation

This document serves as the comprehensive completion report and technical walkthrough for the Round 2 upgrade of the customer-support-env.

1. Architectural Overhaul: Hierarchical Multi-Agent System

We transitioned the environment from a single-agent RL loop to a robust 3-Level Hierarchical Multi-Agent System, designed to simulate complex, real-world enterprise customer support workflows.

The environment now dynamically manages states and role transitions between three distinct agent levels:

1. Level 1: Support Agent
   • Role: Front-line interaction, gathers information, provides initial responses.
   • Actions: respond, request_info, close, escalate (to L2).
2. Level 2: Supervisor
   • Role: Oversight, QA, policy enforcement, and coaching. Reviews every L1 action before it reaches the customer.
   • Actions: supervisor_approve, supervisor_reject, supervisor_feedback (sends back to L1), supervisor_escalate (to L3).
3. Level 3: Manager
   • Role: Executive escalation, conflict resolution, policy overriding.
   • Actions: manager_override, manager_resolve, manager_send_back.

This is implemented via the new HierarchicalCustomerSupportEnv class, utilizing a phase-based state machine (HierarchyState) that routes actions and transitions seamlessly. Crucially, the environment maintains 100% backward compatibility with Round 1 single-agent tasks.

2. Core Feature Additions

LLM-Driven Customer Simulator & Hinglish Degradation

• File: env/customer_simulator.py
• Replaced deterministic customer logic with a dynamic LLM-driven simulator.
• Tracks frustration_level based on agent interactions.
• Hinglish Degradation: If frustration exceeds 0.6, the customer simulator begins degrading its language into Hinglish, simulating real-world agitated Indian enterprise customers.

Hybrid Reward Engine & LLM-as-a-Judge

• Files: env/reward_engine.py, env/llm_judge.py
• Eliminated gamable, purely deterministic rewards.
• Implemented an asynchronous LLM-as-a-Judge module evaluating 5 specific rubrics: empathy, resolution, policy_adherence, oversight_quality, and decision_quality.
• Role-Specific Rewards: The engine now issues isolated scores for support_agent, supervisor, and manager, enabling targeted RL updates (e.g., GRPO) per role.
• Anti-Gaming Penalties: Strict deductions for hallucinated policies, ignored supervisor feedback, and repetitive loops.

Dynamic Policy Drift Engine

• File: env/policy_engine.py
• Introduces mid-episode "schema drift" to test agent adaptability.
• Example: The system might inject an ENVIRONMENT EVENT: Refund portal is down at step 3, forcing the Supervisor to reject a Support Agent's attempt to issue a refund and demanding an alternative solution.

Indian Enterprise Dataset

• File: env/ticket_store.py
• Added 6 high-complexity, hierarchical tickets focused on the Indian Enterprise context (UPI failures, Big Billion Days timeouts, KYC rejections, SLA breaches).

Hierarchy Graders

• Files: env/graders/task_hierarchy_easy.py, medium.py, hard.py
• Developed 3 new graders explicitly designed to evaluate the multi-agent interaction flow, ensuring L2s actually review and L3s actually resolve critical paths.

3. Workflow & Integration

Server Routing

• File: server/app.py
• The FastAPI server was upgraded to auto-detect task requests. Requests starting with hierarchy_* are routed to the HierarchicalCustomerSupportEnv, while standard tasks use CustomerSupportEnv.

Inference Loop

• File: inference.py
• The inference script was entirely rewritten to support dynamic role switching.
• It parses the active_role from the observation state and injects role-specific system prompts (Support Agent vs. Supervisor vs. Manager) containing contextual data like supervisor_feedback and manager_directives.
• Retains API key failover logic using NVIDIA NIM endpoints.

API Specification

• File: openenv.yaml
• Bumped version to 2.0.0.
• Added 3 new hierarchy tasks and expanded the action/observation space to include fields like active_role, supervisor_feedback, and policy_context.

4. Testing & Results

The entire system is successfully dockerized and validated.

Test Suite Execution

• Wrote extensive pytest test cases covering session isolation, LLM-as-a-judge boundaries, hierarchy phase transitions, and end-to-end API flows.
• Result: 50/50 tests passed in 3.71s within the Docker container.

Live Endpoint Verification

• Started the meta_hack-env Docker container and ran a live test_live.py validation script.
• Single-Agent Backward Compatibility: Verified easy and hard tasks complete properly.
• Hierarchy Easy Flow: Verified L1 respond → L2 approve → L1 close → L2 approve.
• Hierarchy Hard Flow: Verified L1 respond → L2 escalate → L3 resolve.
• Role Rewards Verification: Confirmed JSON output parses correct dense role-rewards (e.g., {'support_agent': 0.425, 'supervisor': 0.725, 'manager': 0.65}).

5. Next Steps

With the environment, endpoints, and inference architecture fully deployed and validated on the feat/round2 branch, the final step for Hackathon readiness is:

1. Training Pipeline: Develop the train_grpo.py script utilizing Unsloth / TRL to consume the new role_rewards dictionaries and fine-tune models to operate efficiently within this hierarchical structure.