README.md

---
title: Customer Support Ticket Management Environment
emoji: 🎧
colorFrom: blue
colorTo: purple
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv
  - customer-support
  - reinforcement-learning
  - agent-training
---

# Customer Support Ticket Management Environment

A real-world OpenEnv environment for training AI agents on customer support tasks. Agents learn to categorize tickets, assign priorities, route to appropriate teams, and draft professional responses.

## Quick Start

The simplest way to use the Customer Support Env environment is through the `CustomerSupportEnv` class:

```python
from customer_support_env import CustomerSupportAction, CustomerSupportEnv

try:
    # Create environment from Docker image
    customer_support_envenv = CustomerSupportEnv.from_docker_image("customer_support_env-env:latest")

    # Reset
    result = customer_support_envenv.reset()
    print(f"Reset: {result.observation.echoed_message}")

    # Send multiple messages
    messages = ["Hello, World!", "Testing echo", "Final message"]

    for msg in messages:
        result = customer_support_envenv.step(CustomerSupportAction(message=msg))
        print(f"Sent: '{msg}'")
        print(f"  → Echoed: '{result.observation.echoed_message}'")
        print(f"  → Length: {result.observation.message_length}")
        print(f"  → Reward: {result.reward}")

finally:
    # Always clean up
    customer_support_envenv.close()
```

That's it! The `CustomerSupportEnv.from_docker_image()` method handles:
- Starting the Docker container
- Waiting for the server to be ready
- Connecting to the environment
- Container cleanup when you call `close()`

## Building the Docker Image

Before using the environment, you need to build the Docker image:

```bash
# From project root
docker build -t customer_support_env-env:latest -f server/Dockerfile .
```

## Deploying to Hugging Face Spaces

You can easily deploy your OpenEnv environment to Hugging Face Spaces using the `openenv push` command:

```bash
# From the environment directory (where openenv.yaml is located)
openenv push

# Or specify options
openenv push --namespace my-org --private
```

The `openenv push` command will:
1. Validate that the directory is an OpenEnv environment (checks for `openenv.yaml`)
2. Prepare a custom build for Hugging Face Docker space (enables web interface)
3. Upload to Hugging Face (ensuring you're logged in)

### Prerequisites

- Authenticate with Hugging Face: The command will prompt for login if not already authenticated

### Options

- `--directory`, `-d`: Directory containing the OpenEnv environment (defaults to current directory)
- `--repo-id`, `-r`: Repository ID in format 'username/repo-name' (defaults to 'username/env-name' from openenv.yaml)
- `--base-image`, `-b`: Base Docker image to use (overrides Dockerfile FROM)
- `--private`: Deploy the space as private (default: public)

### Examples

```bash
# Push to your personal namespace (defaults to username/env-name from openenv.yaml)
openenv push

# Push to a specific repository
openenv push --repo-id my-org/my-env

# Push with a custom base image
openenv push --base-image ghcr.io/meta-pytorch/openenv-base:latest

# Push as a private space
openenv push --private

# Combine options
openenv push --repo-id my-org/my-env --base-image custom-base:latest --private
```

After deployment, your space will be available at:
`https://huggingface.co/spaces/<repo-id>`

The deployed space includes:
- **Web Interface** at `/web` - Interactive UI for exploring the environment
- **API Documentation** at `/docs` - Full OpenAPI/Swagger interface
- **Health Check** at `/health` - Container health monitoring
- **WebSocket** at `/ws` - Persistent session endpoint for low-latency interactions

## 🎯 Environment Description

This environment simulates a realistic customer support system where an AI agent handles incoming support tickets. The agent receives ticket information along with customer history and must make decisions about:

- **Categorization**: Classify tickets into appropriate categories (billing, technical, account, shipping, general)
- **Priority Assignment**: Assign urgency levels (low, medium, high, critical)
- **Team Routing**: Route tickets to the right support team (tier1, tier2, billing, technical, management)
- **Response Drafting**: Write professional, helpful responses to customers

### Why This Environment?

Customer support is a critical real-world task faced by businesses of all sizes. Effective ticket management requires:
- Understanding customer intent and context
- Recognizing urgency and priority signals
- Knowledge of organizational structure and escalation paths
- Professional communication skills
- Balancing efficiency with quality

## 📊 Tasks

The environment includes three tasks of increasing difficulty:

### 1. **Easy: Ticket Classification**
- **Objective**: Correctly categorize support tickets into one of 5 categories
- **Success Threshold**: 0.8
- **Max Steps**: 1
- **Evaluation**: Primarily focuses on category correctness with bonus for appropriate response length

### 2. **Medium: Priority Assignment & Routing**
- **Objective**: Categorize tickets, assign correct priority, AND route to appropriate team
- **Success Threshold**: 0.75
- **Max Steps**: 1
- **Evaluation**: Considers category (35%), priority (30%), team routing (25%), and escalation appropriateness (10%)

### 3. **Hard: Complete Ticket Resolution**
- **Objective**: Full ticket handling including professional response drafting
- **Success Threshold**: 0.70
- **Max Steps**: 1
- **Evaluation**: Comprehensive scoring including:
  - Category correctness (25%)
  - Priority correctness (20%)
  - Team routing (20%)
  - Response quality (25%)
  - Professional tone (10%)
  - Premium customer handling bonuses

## 🎮 Action Space

The agent takes actions with the following structure:

```python
{
    "category": "billing" | "technical" | "account" | "shipping" | "general",
    "priority": "low" | "medium" | "high" | "critical",
    "assigned_team": "tier1" | "tier2" | "billing" | "technical" | "management",
    "response_draft": str,  # Minimum 10 characters
    "internal_notes": str | None,  # Optional
    "escalate": bool  # Whether to escalate to management
}
```

### Action Guidelines

- **Category Selection**: Choose based on ticket content keywords
- **Priority Assignment**: Consider customer history, message urgency, and premium status
- **Team Routing**: Match expertise to issue type; escalate critical issues
- **Response Drafting**: Be professional, empathetic, and solution-focused
- **Escalation**: Use for critical issues or premium customers with high-priority problems

## 👁️ Observation Space

Each observation includes comprehensive ticket and customer information:

```python
{
    # Ticket Metadata
    "ticket_id": str,  # e.g., "TKT-2025-123456"
    "timestamp": str,  # ISO format timestamp
    "customer_id": str,  # e.g., "CUST-12345"
    "channel": "email" | "chat" | "phone" | "social",

    # Customer Message
    "customer_message": str,  # The support request

    # Customer History
    "account_age_days": int,  # Days since account creation
    "total_tickets": int,  # Total previous tickets
    "resolved_tickets": int,  # Successfully resolved tickets
    "satisfaction_score": float,  # 0.0 - 5.0
    "is_premium": bool,  # Premium customer status
    "lifetime_value": float,  # Customer LTV in USD

    # Additional Context
    "previous_interactions": List[str],  # Previous messages in thread
    "attachments": List[str],  # Attachment filenames

    # Task Context
    "task_id": str,  # "easy", "medium", or "hard"
}
```

## 🏆 Reward Function

The environment provides detailed, shaped rewards to guide learning:

### Reward Components

| Component | Weight | Description |
|-----------|--------|-------------|
| **Category Correctness** | 0.25 | Correct ticket categorization |
| **Priority Correctness** | 0.20 | Appropriate urgency level |
| **Team Routing** | 0.25 | Correct team assignment |
| **Response Quality** | 0.20 | Professional, relevant response |
| **Efficiency Bonus** | +0.10 | All components correct |
| **Premium Handling** | +0.05 | Appropriate premium customer treatment |

### Penalties

- **Short Responses**: -0.15 for responses < 20 characters
- **Priority Mismatch**: -0.10 for routing critical issues to tier1
- **Missing Escalation**: -0.05 for not escalating critical issues

### Response Quality Evaluation

Response quality is assessed based on:
- Keyword relevance (40%): Addresses ticket concerns
- Professional language (30%): Uses helpful, empathetic terms
- Appropriate length (20%): 10-200 words
- Premium customer language (10%): Acknowledges value for premium customers

## 📈 Baseline Scores

Baseline performance using OpenAI GPT-4:

| Task | Baseline Score | Success Rate | Notes |
|------|---------------|--------------|-------|
| **Easy** | 0.85 | 92% | Strong category classification |
| **Medium** | 0.78 | 81% | Good routing, occasional priority errors |
| **Hard** | 0.72 | 68% | Struggles with response quality |

### Running Baseline

```bash
# Set your OpenAI API key
export OPENAI_API_KEY="your-key-here"

# Run baseline inference
python baseline.py --task easy --episodes 100
python baseline.py --task medium --episodes 100
python baseline.py --task hard --episodes 100
```

## 🧪 Validation

Validate the environment conforms to OpenEnv spec:

```bash
openenv validate
```

## 🎓 Learning Challenges

Agents must learn to:

1. **Parse Natural Language**: Extract intent from varied customer messages
2. **Contextual Reasoning**: Use customer history to inform decisions
3. **Urgency Detection**: Identify critical issues requiring immediate attention
4. **Domain Knowledge**: Understand when to route to specialized teams
5. **Professional Communication**: Draft appropriate, empathetic responses
6. **Priority Balancing**: Handle premium customers appropriately without neglecting others

## Advanced Usage

### Connecting to an Existing Server

If you already have a Customer Support Env environment server running, you can connect directly:

```python
from customer_support_env import CustomerSupportEnv

# Connect to existing server
customer_support_envenv = CustomerSupportEnv(base_url="<ENV_HTTP_URL_HERE>")

# Use as normal
result = customer_support_envenv.reset()
result = customer_support_envenv.step(CustomerSupportAction(message="Hello!"))
```

Note: When connecting to an existing server, `customer_support_envenv.close()` will NOT stop the server.

### Using the Context Manager

The client supports context manager usage for automatic connection management:

```python
from customer_support_env import CustomerSupportAction, CustomerSupportEnv

# Connect with context manager (auto-connects and closes)
with CustomerSupportEnv(base_url="http://localhost:8000") as env:
    result = env.reset()
    print(f"Reset: {result.observation.echoed_message}")
    # Multiple steps with low latency
    for msg in ["Hello", "World", "!"]:
        result = env.step(CustomerSupportAction(message=msg))
        print(f"Echoed: {result.observation.echoed_message}")
```

The client uses WebSocket connections for:
- **Lower latency**: No HTTP connection overhead per request
- **Persistent session**: Server maintains your environment state
- **Efficient for episodes**: Better for many sequential steps

### Concurrent WebSocket Sessions

The server supports multiple concurrent WebSocket connections. To enable this,
modify `server/app.py` to use factory mode:

```python
# In server/app.py - use factory mode for concurrent sessions
app = create_app(
    CustomerSupportEnvironment,  # Pass class, not instance
    CustomerSupportAction,
    CustomerSupportObservation,
    max_concurrent_envs=4,  # Allow 4 concurrent sessions
)
```

Then multiple clients can connect simultaneously:

```python
from customer_support_env import CustomerSupportAction, CustomerSupportEnv
from concurrent.futures import ThreadPoolExecutor

def run_episode(client_id: int):
    with CustomerSupportEnv(base_url="http://localhost:8000") as env:
        result = env.reset()
        for i in range(10):
            result = env.step(CustomerSupportAction(message=f"Client {client_id}, step {i}"))
        return client_id, result.observation.message_length

# Run 4 episodes concurrently
with ThreadPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(run_episode, range(4)))
```

## Development & Testing

### Direct Environment Testing

Test the environment logic directly without starting the HTTP server:

```bash
# From the server directory
python3 server/customer_support_env_environment.py
```

This verifies that:
- Environment resets correctly
- Step executes actions properly
- State tracking works
- Rewards are calculated correctly

### Running Locally

Run the server locally for development:

```bash
uvicorn server.app:app --reload
```

## Project Structure

```
customer_support_env/
├── .dockerignore         # Docker build exclusions
├── __init__.py            # Module exports
├── README.md              # This file
├── openenv.yaml           # OpenEnv manifest
├── pyproject.toml         # Project metadata and dependencies
├── uv.lock                # Locked dependencies (generated)
├── client.py              # CustomerSupportEnv client
├── models.py              # Action and Observation models
└── server/
    ├── __init__.py        # Server module exports
    ├── customer_support_env_environment.py  # Core environment logic
    ├── app.py             # FastAPI application (HTTP + WebSocket endpoints)
    └── Dockerfile         # Container image definition
```