README.md

---
title: B2B Support Triage OpenEnv
emoji: "🧭"
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv
---

# B2B Support Triage OpenEnv Benchmark

A deterministic, real-world OpenEnv environment that simulates B2B SaaS customer support triage.

## Why this environment

Support teams repeatedly perform structured triage decisions under policy constraints:
- classify ticket domain
- assign urgency
- route to the right queue
- set SLA targets
- draft policy-compliant replies

This environment is designed for agent evaluation with deterministic graders and shaped rewards, not toy gameplay.

## OpenEnv API

The server exposes standard OpenEnv simulation endpoints:
- `POST /reset`
- `POST /step`
- `GET /state`

`reset()` supports `task_id` and `seed`.

## Action Space

Typed action model: `B2BSupportTriageAction`

- `action_type`: `classify | set_priority | route | draft_reply | submit`
- `ticket_id`: required for all actions except `submit`
- `payload`:
  - `category`
  - `priority`
  - `route_queue`
  - `sla_minutes`
  - `escalate`
  - `reply_text`

Validation is deterministic and action-specific.

## Observation Space

Typed observation model: `B2BSupportTriageObservation`

- `task_id`, `step_index`, `max_steps`
- `visible_ticket`
- `current_plan`
- `applied_decisions`
- `last_action_error`
- `progress_score` in `[0, 1]`
- `reward_breakdown`:
  - `correctness_delta`
  - `policy_bonus`
  - `repeat_penalty`
  - `invalid_penalty`
  - `terminal_bonus`

## State Space

Typed state model: `B2BSupportTriageState`

- `episode_id`, `step_count`
- `task_id`, `seed`, `max_steps`
- `cumulative_reward`
- `applied_decisions`
- `action_history`
- `completion_flags`

## Tasks and Difficulty

Three deterministic fixtures are bundled in `fixtures/tasks.json`.

1. `easy`
- Goal: classify + prioritize one billing ticket.
- Grader: category + priority.

2. `medium`
- Goal: classify + prioritize + route + SLA for enterprise billing anomaly.
- Grader: category + priority + route + SLA.

3. `hard`
- Goal: security incident triage with escalation and compliant customer response.
- Grader: category + priority + route + SLA + escalate + reply policy phrase coverage.

Each grader returns a normalized deterministic score in `[0.0, 1.0]`.

## Reward Design

Step reward uses shaped components:
- positive reward for incremental correctness gains (no double counting)
- penalties for invalid actions
- penalties for repeated/no-progress loops and contradictory edits
- terminal bonus tied to final task score

## Baseline Inference

`inference.py` is at repository root (hackathon requirement).

It:
- uses OpenAI client calls (`OpenAI(base_url=API_BASE_URL, api_key=HF_TOKEN)`)
- runs all tasks in fixed order with fixed seeds
- emits required structured logs:
  - `[START] ...`
  - `[STEP] ...`
  - `[END] ...`

Environment variables:
- `API_BASE_URL` (default: `https://router.huggingface.co/v1`)
- `MODEL_NAME` (default: `Qwen/Qwen2.5-72B-Instruct`)
- `HF_TOKEN` (required)
- `LOCAL_IMAGE_NAME` (default: `b2b_support_triage_env-env:latest`)

Reference deterministic policy target (with valid execution):
- easy: `1.000`
- medium: `1.000`
- hard: `1.000`
- aggregate: `1.000`

## Setup

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install pytest
```

## Local Run

```bash
uvicorn server.app:app --host 0.0.0.0 --port 8000
```

## Validation

```bash
pytest -q
openenv validate -v
```

## Docker

```bash
docker build -t b2b_support_triage_env-env:latest -f server/Dockerfile .
docker run --rm -p 8000:8000 b2b_support_triage_env-env:latest
```

## Hugging Face Space Deployment

```bash
openenv push
```

Then run validator script provided by the hackathon:

```bash
./validate-submission.sh <your-space-url> .
```

## Inlined Documentation

### local_setup.md

# Local Setup and Run Guide

This guide explains how to set up, run, test, and execute inference for this project end-to-end on your local machine.

## 1) Prerequisites

Install and verify:

- Python 3.10+
- Docker Desktop (running)
- `openenv` CLI
- `curl`

Quick checks:

```bash
python3 --version
docker --version
openenv --help >/dev/null && echo "openenv OK"
curl --version | head -n 1
```

## 2) Project Directory

```bash
cd /Users/aksudhak/Documents/Akhil/POC/Scaler/OpenENV
```

## 3) Optional: Local Python test dependencies

If `pytest` is not installed:

```bash
python3 -m pip install pytest
```

## 4) Environment Variables for Inference

Set these before running `inference.py`:

```bash
export HF_TOKEN="<YOUR_NEW_HF_TOKEN>"
export API_BASE_URL="https://router.huggingface.co/v1"
export MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
export LOCAL_IMAGE_NAME="b2b_support_triage_env-env:latest"
```

Notes:
- `HF_TOKEN` is required.
- `API_BASE_URL`, `MODEL_NAME`, and `LOCAL_IMAGE_NAME` have defaults in code, but export them explicitly for submission clarity.

## 5) Build Docker Image

```bash
docker build -t b2b_support_triage_env-env:latest -f server/Dockerfile .
```

## 6) Run Application Locally

```bash
docker run --rm -p 8000:8000 b2b_support_triage_env-env:latest
```

Keep this terminal running. Open a second terminal for API checks.

## 7) API Smoke Test

### Health

```bash
curl -s http://127.0.0.1:8000/health
```

### Reset

```bash
curl -s -X POST http://127.0.0.1:8000/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id":"easy","seed":1}'
```

### Step

```bash
curl -s -X POST http://127.0.0.1:8000/step \
  -H "Content-Type: application/json" \
  -d '{"action":{"action_type":"classify","ticket_id":"T-EASY-1001","payload":{"category":"billing"}}}'
```

### State

```bash
curl -s http://127.0.0.1:8000/state
```

## 8) Run Unit + Spec Validation

```bash
pytest -q
openenv validate -v
```

## 9) Run Inference Directly

```bash
python3 inference.py
```

Expected log pattern in stdout:
- `[START] ...`
- multiple `[STEP] ...`
- `[END] ...`
- final aggregate score line

## 10) Recommended Helper Scripts

### A) Full local checks (tests + validate + docker + endpoint checks + optional inference)

```bash
./run_all_checks.sh
```

Options:

```bash
RUN_INFERENCE=no ./run_all_checks.sh
RUN_INFERENCE=yes ./run_all_checks.sh
```

### B) Inference helper with token prompt

If `HF_TOKEN` is missing, this script prompts securely for it.

```bash
./run_inference.sh
```

It also sets defaults for:
- `API_BASE_URL`
- `MODEL_NAME`
- `LOCAL_IMAGE_NAME`

## 11) Common Issues

### Docker daemon not running
Start Docker Desktop and retry.

### Token error
Use a fresh valid Hugging Face token and re-export `HF_TOKEN`.

### Port 8000 already in use
Run container on a different host port:

```bash
docker run --rm -p 8001:8000 b2b_support_triage_env-env:latest
```

Then use `http://127.0.0.1:8001` in curl commands.

### project.md

# B2B Support Triage OpenEnv Benchmark - Complete Project Overview

## 1) What this project is

This project is a complete OpenEnv simulation environment designed for **real-world B2B SaaS support triage**.

Instead of a toy game, it models a workflow support engineers actually perform:
- classify an incoming ticket
- set urgency/priority
- route to the correct team queue
- define SLA expectation
- (hard task) provide policy-compliant customer communication

It exposes the standard OpenEnv contract:
- `reset(...)`
- `step(action)`
- `state()`

with typed Pydantic models and deterministic graders.

---

## 2) Why this environment is useful

### Practical utility
This environment can evaluate agent quality on an enterprise operation where correctness, consistency, and policy compliance matter.

### Deterministic benchmarking
All tasks are fixture-driven and deterministic, so model comparisons are reproducible.

### Multi-step reasoning
Agents must make a sequence of structured decisions, not just one-shot classification.

---

## 3) High-level architecture

### Main components

1. **Environment runtime**
- File: `server/support_triage_environment.py`
- Implements environment state machine and reward shaping.

2. **Typed schemas**
- File: `models.py`
- Defines action, observation, state, and reward-breakdown models.

3. **Task fixtures**
- File: `fixtures/tasks.json`
- Defines easy/medium/hard scenarios, allowed values, answer keys, and policy hints.

4. **Grader logic**
- File: `graders.py`
- Computes normalized score in `[0,1]` using weighted criteria per task.

5. **OpenEnv API server wiring**
- File: `server/app.py`
- Binds environment + typed models into OpenEnv HTTP/WebSocket endpoints.

6. **Inference baseline**
- File: `inference.py`
- Runs all tasks using OpenAI client and emits required structured logs.

7. **Client adapter**
- File: `client.py`
- Converts typed Python actions/observations to/from wire payloads.

---

## 4) OpenEnv contract and endpoint behavior

### Manifest
- File: `openenv.yaml`
- Declares:
  - `name: b2b_support_triage_env`
  - `runtime: fastapi`
  - `app: server.app:app`
  - `port: 8000`

### Runtime endpoints
- `POST /reset`
- `POST /step`
- `GET /state`
- `GET /health`
- `GET /schema`

### Environment lifecycle

1. `reset(task_id=..., seed=...)`
- loads selected task fixture
- clears previous episode state
- returns initial observation with visible ticket + dynamic plan hints

2. `step(action)`
- validates action semantics and allowed values
- updates decision state
- computes score delta and shaped reward
- checks done conditions (`submit` or max steps)

3. `state()`
- returns current internal state object (episode info, step count, cumulative reward, history)

---

## 5) Data model design (typed)

## 5.1 Action model
File: `models.py`

`B2BSupportTriageAction` fields:
- `action_type`: enum
  - `classify`
  - `set_priority`
  - `route`
  - `draft_reply`
  - `submit`
- `ticket_id`: required for non-`submit`
- `payload`: action-specific content

`payload` supports:
- `category`
- `priority`
- `route_queue`
- `sla_minutes`
- `escalate`
- `reply_text`

### Validation rules
- non-submit requires `ticket_id`
- required fields per action:
  - `classify -> category`
  - `set_priority -> priority`
  - `route -> route_queue + sla_minutes`
  - `draft_reply -> reply_text`
  - `submit -> none`

This enforces structured behavior and prevents ambiguous free-text action control.

## 5.2 Observation model
`B2BSupportTriageObservation` includes:
- `task_id`, `step_index`, `max_steps`
- `visible_ticket`
- `current_plan` (next actions + policy hints)
- `applied_decisions`
- `last_action_error`
- `progress_score` (grader score in `[0,1]`)
- `reward_breakdown`:
  - `correctness_delta`
  - `policy_bonus`
  - `repeat_penalty`
  - `invalid_penalty`
  - `terminal_bonus`
- inherited OpenEnv fields `reward`, `done`, `metadata`

## 5.3 State model
`B2BSupportTriageState` tracks:
- episode metadata (`episode_id`, `step_count`, `task_id`, `seed`)
- `max_steps`
- `cumulative_reward`
- `applied_decisions`
- `action_history`
- `completion_flags`

---

## 6) Task system (easy -> medium -> hard)

Defined in `fixtures/tasks.json`.

## 6.1 Easy
Goal:
- correct category + priority for one billing ticket

Hidden answer key includes:
- `category=billing`
- `priority=medium`
- plus routing defaults

## 6.2 Medium
Goal:
- category + priority + queue routing + SLA

Hidden answer key includes:
- `category=billing`
- `priority=high`
- `route_queue=billing-l2`
- `sla_minutes=120`

## 6.3 Hard
Goal:
- security incident handling with escalation + compliant customer response

Hidden answer key includes:
- `category=security`
- `priority=urgent`
- `route_queue=security-incident-response`
- `sla_minutes=120`
- `escalate=true`
- required reply phrases

---

## 7) Grading mechanics

File: `graders.py`

Each task has deterministic weighted scoring.

## 7.1 Easy weighting
- category: 0.6
- priority: 0.4

## 7.2 Medium weighting
- category: 0.35
- priority: 0.25
- route_queue: 0.25
- sla_minutes: 0.15

## 7.3 Hard weighting
- category: 0.15
- priority: 0.15
- route_queue: 0.15
- sla_minutes: 0.10
- escalate: 0.20
- reply_policy phrase coverage: 0.25

The result is clamped and normalized to `[0,1]`.

### Determinism
The grader is pure:
- no network calls
- no random operations
- same input decisions always yield same score

---

## 8) Reward shaping logic

File: `server/support_triage_environment.py`

Reward is step-wise, dense enough for learning signals, and punishes bad behavior.

### Positive components
- `correctness_delta`: only when progress score increases
- `policy_bonus`: bonus for substantive policy-style draft replies
- `terminal_bonus`: `0.2 * final_score` when episode ends

### Penalty components
- `invalid_penalty`: invalid ticket/value/action
- `repeat_penalty`:
  - repeated identical action
  - contradiction penalties (changing already-set value)
  - no-progress streak penalties

### Episode boundaries
- done on `submit`
- done on max step limit
- steps after done receive penalty (`episode_already_done`)

This discourages looping and rewards incremental correctness.

---

## 9) Inference baseline design

File: `inference.py`

## 9.1 Runtime setup
Reads:
- `HF_TOKEN` (required)
- `API_BASE_URL` (default set)
- `MODEL_NAME` (default set)
- `LOCAL_IMAGE_NAME` (default set)

## 9.2 Task execution
- runs tasks in fixed order: easy, medium, hard
- fixed seeds per task for reproducibility

## 9.3 Action selection strategy
- requests JSON action from model (OpenAI client call)
- validates/coerces action
- falls back to deterministic policy if output invalid or misaligned

This keeps baseline robust and reproducible while still exercising LLM calls.

## 9.4 Required stdout format
Per task episode:
- `[START] ...`
- multiple `[STEP] ...`
- `[END] ...`

Then aggregate score summary line.

---

## 10) Deployment and packaging

## 10.1 Containerization
- Dockerfile: `server/Dockerfile`
- builds runnable image exposing uvicorn on port 8000
- includes healthcheck on `/health`

## 10.2 Python packaging
- `pyproject.toml` defines project metadata and dependencies
- installs environment package editable in image

## 10.3 OpenEnv compatibility
- `openenv validate -v` passes
- simulation API surface is compliant

---

## 11) Testing strategy

Files under `tests/` verify behavior:

- `test_environment.py`
  - reset clean state
  - invalid ticket penalty
  - hard task can reach full score
  - max-step termination behavior

- `test_graders.py`
  - perfect answer gives 1.0
  - weighted partial score check
  - deterministic hard-task reply scoring

- `test_inference_logging.py`
  - `[START]/[STEP]/[END]` formatting checks

This coverage ensures both task semantics and evaluation fidelity.

---

## 12) How to use the project locally

## 12.1 End-to-end helper
```bash
./run_all_checks.sh
```
Runs tests, validation, docker build/run smoke checks, and optional inference.

## 12.2 Inference helper (prompts for token if missing)
```bash
./run_inference.sh
```

## 12.3 Manual core commands
```bash
pytest -q
openenv validate -v
docker build -t b2b_support_triage_env-env:latest -f server/Dockerfile .
docker run --rm -p 8000:8000 b2b_support_triage_env-env:latest
```

Detailed runbook: `local_setup.md`.

---

## 13) Usability summary for your hackathon goal

What this project gives you:
- clear real-world domain
- structured and typed agent action space
- deterministic multi-level tasks
- strong reward shaping (partial progress + anti-loop penalties)
- reproducible baseline script with required logs
- Docker + OpenEnv validation support

This aligns well with your round requirements and helps reduce submission-time surprises.

---

## 14) Current limitations and next improvements

### Current limitations
- single-ticket episode scope (per task) rather than multi-ticket queue backlog
- deterministic phrase matching for hard reply policy (simple lexical strategy)
- baseline includes deterministic fallback policy (useful for reliability, less pure model-only behavior)

### Next upgrades (optional)
- multi-ticket routing episodes with queue-level capacity constraints
- richer policy engine with explicit rule graph and violation tags
- adversarial or noisy customer phrasing variants per seed
- benchmark report exporter with per-criterion analytics

---

## 15) Security and publication posture

- fixtures are synthetic
- no hardcoded credentials
- token usage is environment-variable based only
- suitable for external publication, assuming you do not commit local secret files (`.env`, shell history, etc.)