project.md

B2B Support Triage OpenEnv Benchmark - Complete Project Overview

1) What this project is

This project is a complete OpenEnv simulation environment designed for real-world B2B SaaS support triage.

Instead of a toy game, it models a workflow support engineers actually perform:

• classify an incoming ticket
• set urgency/priority
• route to the correct team queue
• define SLA expectation
• (hard task) provide policy-compliant customer communication

It exposes the standard OpenEnv contract:

• reset(...)
• step(action)
• state()

with typed Pydantic models and deterministic graders.

---

2) Why this environment is useful

Practical utility

This environment can evaluate agent quality on an enterprise operation where correctness, consistency, and policy compliance matter.

Deterministic benchmarking

All tasks are fixture-driven and deterministic, so model comparisons are reproducible.

Multi-step reasoning

Agents must make a sequence of structured decisions, not just one-shot classification.

---

3) High-level architecture

Main components

1. Environment runtime

• File: server/support_triage_environment.py
• Implements environment state machine and reward shaping.

2. Typed schemas

• File: models.py
• Defines action, observation, state, and reward-breakdown models.

3. Task fixtures

• File: fixtures/tasks.json
• Defines easy/medium/hard scenarios, allowed values, answer keys, and policy hints.

4. Grader logic

• File: graders.py
• Computes normalized score in [0,1] using weighted criteria per task.

5. OpenEnv API server wiring

• File: server/app.py
• Binds environment + typed models into OpenEnv HTTP/WebSocket endpoints.

6. Inference baseline

• File: inference.py
• Runs all tasks using OpenAI client and emits required structured logs.

7. Client adapter

• File: client.py
• Converts typed Python actions/observations to/from wire payloads.

---

4) OpenEnv contract and endpoint behavior

Manifest

• File: openenv.yaml
• Declares:
  • name: b2b_support_triage_env
  • runtime: fastapi
  • app: server.app:app
  • port: 8000

Runtime endpoints

• POST /reset
• POST /step
• GET /state
• GET /health
• GET /schema

Environment lifecycle

1. reset(task_id=..., seed=...)

• loads selected task fixture
• clears previous episode state
• returns initial observation with visible ticket + dynamic plan hints

2. step(action)

• validates action semantics and allowed values
• updates decision state
• computes score delta and shaped reward
• checks done conditions (submit or max steps)

3. state()

• returns current internal state object (episode info, step count, cumulative reward, history)

---

5) Data model design (typed)

5.1 Action model

File: models.py

B2BSupportTriageAction fields:

• action_type: enum
  • classify
  • set_priority
  • route
  • draft_reply
  • submit
• ticket_id: required for non-submit
• payload: action-specific content

payload supports:

• category
• priority
• route_queue
• sla_minutes
• escalate
• reply_text

Validation rules

• non-submit requires ticket_id
• required fields per action:
  • classify -> category
  • set_priority -> priority
  • route -> route_queue + sla_minutes
  • draft_reply -> reply_text
  • submit -> none

This enforces structured behavior and prevents ambiguous free-text action control.

5.2 Observation model

B2BSupportTriageObservation includes:

• task_id, step_index, max_steps
• visible_ticket
• current_plan (next actions + policy hints)
• applied_decisions
• last_action_error
• progress_score (grader score in [0,1])
• reward_breakdown:
  • correctness_delta
  • policy_bonus
  • repeat_penalty
  • invalid_penalty
  • terminal_bonus
• inherited OpenEnv fields reward, done, metadata

5.3 State model

B2BSupportTriageState tracks:

• episode metadata (episode_id, step_count, task_id, seed)
• max_steps
• cumulative_reward
• applied_decisions
• action_history
• completion_flags

---

6) Task system (easy -> medium -> hard)

Defined in fixtures/tasks.json.

6.1 Easy

Goal:

• correct category + priority for one billing ticket

Hidden answer key includes:

• category=billing
• priority=medium
• plus routing defaults

6.2 Medium

Goal:

• category + priority + queue routing + SLA

Hidden answer key includes:

• category=billing
• priority=high
• route_queue=billing-l2
• sla_minutes=120

6.3 Hard

Goal:

• security incident handling with escalation + compliant customer response

Hidden answer key includes:

• category=security
• priority=urgent
• route_queue=security-incident-response
• sla_minutes=120
• escalate=true
• required reply phrases

---

7) Grading mechanics

File: graders.py

Each task has deterministic weighted scoring.

7.1 Easy weighting

• category: 0.6
• priority: 0.4

7.2 Medium weighting

• category: 0.35
• priority: 0.25
• route_queue: 0.25
• sla_minutes: 0.15

7.3 Hard weighting

• category: 0.15
• priority: 0.15
• route_queue: 0.15
• sla_minutes: 0.10
• escalate: 0.20
• reply_policy phrase coverage: 0.25

The result is clamped and normalized to [0,1].

Determinism

The grader is pure:

• no network calls
• no random operations
• same input decisions always yield same score

---

8) Reward shaping logic

File: server/support_triage_environment.py

Reward is step-wise, dense enough for learning signals, and punishes bad behavior.

Positive components

• correctness_delta: only when progress score increases
• policy_bonus: bonus for substantive policy-style draft replies
• terminal_bonus: 0.2 * final_score when episode ends

Penalty components

• invalid_penalty: invalid ticket/value/action
• repeat_penalty:
  • repeated identical action
  • contradiction penalties (changing already-set value)
  • no-progress streak penalties

Episode boundaries

• done on submit
• done on max step limit
• steps after done receive penalty (episode_already_done)

This discourages looping and rewards incremental correctness.

---

9) Inference baseline design

File: inference.py

9.1 Runtime setup

Reads:

• HF_TOKEN (required)
• API_BASE_URL (default set)
• MODEL_NAME (default set)
• LOCAL_IMAGE_NAME (default set)

9.2 Task execution

• runs tasks in fixed order: easy, medium, hard
• fixed seeds per task for reproducibility

9.3 Action selection strategy

• requests JSON action from model (OpenAI client call)
• validates/coerces action
• falls back to deterministic policy if output invalid or misaligned

This keeps baseline robust and reproducible while still exercising LLM calls.

9.4 Required stdout format

Per task episode:

• [START] ...
• multiple [STEP] ...
• [END] ...

Then aggregate score summary line.

---

10) Deployment and packaging

10.1 Containerization

• Dockerfile: server/Dockerfile
• builds runnable image exposing uvicorn on port 8000
• includes healthcheck on /health

10.2 Python packaging

• pyproject.toml defines project metadata and dependencies
• installs environment package editable in image

10.3 OpenEnv compatibility

• openenv validate -v passes
• simulation API surface is compliant

---

11) Testing strategy

Files under tests/ verify behavior:

• test_environment.py

  • reset clean state
  • invalid ticket penalty
  • hard task can reach full score
  • max-step termination behavior

• test_graders.py

  • perfect answer gives 1.0
  • weighted partial score check
  • deterministic hard-task reply scoring

• test_inference_logging.py

  • [START]/[STEP]/[END] formatting checks

This coverage ensures both task semantics and evaluation fidelity.

---

12) How to use the project locally

12.1 End-to-end helper

./run_all_checks.sh

Runs tests, validation, docker build/run smoke checks, and optional inference.

12.2 Inference helper (prompts for token if missing)

./run_inference.sh

12.3 Manual core commands

pytest -q
openenv validate -v
docker build -t b2b_support_triage_env-env:latest -f server/Dockerfile .
docker run --rm -p 8000:8000 b2b_support_triage_env-env:latest

Detailed runbook: local_setup.md.

---

13) Usability summary for your hackathon goal

What this project gives you:

• clear real-world domain
• structured and typed agent action space
• deterministic multi-level tasks
• strong reward shaping (partial progress + anti-loop penalties)
• reproducible baseline script with required logs
• Docker + OpenEnv validation support

This aligns well with your round requirements and helps reduce submission-time surprises.

---

14) Current limitations and next improvements

Current limitations

• single-ticket episode scope (per task) rather than multi-ticket queue backlog
• deterministic phrase matching for hard reply policy (simple lexical strategy)
• baseline includes deterministic fallback policy (useful for reliability, less pure model-only behavior)

Next upgrades (optional)

• multi-ticket routing episodes with queue-level capacity constraints
• richer policy engine with explicit rule graph and violation tags
• adversarial or noisy customer phrasing variants per seed
• benchmark report exporter with per-criterion analytics

---

15) Security and publication posture

• fixtures are synthetic
• no hardcoded credentials
• token usage is environment-variable based only
• suitable for external publication, assuming you do not commit local secret files (.env, shell history, etc.)