Spaces:
Sleeping
Sleeping
Sync README
Browse files
README.md
CHANGED
|
@@ -1,17 +1,17 @@
|
|
| 1 |
-
---
|
| 2 |
-
title: Customer Support OpenEnv
|
| 3 |
-
emoji: "🎫"
|
| 4 |
-
colorFrom: blue
|
| 5 |
-
colorTo: green
|
| 6 |
-
sdk: docker
|
| 7 |
-
app_port: 8000
|
| 8 |
-
short_description: Deterministic B2B SaaS support benchmark.
|
| 9 |
-
pinned: false
|
| 10 |
-
---
|
| 11 |
-
|
| 12 |
-
# AcmeCloud Customer Support Ticket Handler
|
| 13 |
-
|
| 14 |
-
A deterministic OpenEnv-style environment for training and evaluating agents on realistic B2B SaaS support workflows.
|
| 15 |
|
| 16 |
## What It Simulates
|
| 17 |
|
|
@@ -128,37 +128,56 @@ uvicorn server.app:app --host 0.0.0.0 --port 8000 --reload
|
|
| 128 |
|
| 129 |
Open the docs at [http://localhost:8000/docs](http://localhost:8000/docs) or the simple UI at [http://localhost:8000/web](http://localhost:8000/web).
|
| 130 |
|
| 131 |
-
## Docker
|
| 132 |
-
|
| 133 |
-
```bash
|
| 134 |
-
docker build -t customer-support-openenv
|
| 135 |
-
docker run -p 8000:8000 customer-support-openenv
|
| 136 |
-
```
|
| 137 |
-
|
| 138 |
-
## Baseline Inference
|
| 139 |
-
|
| 140 |
-
The baseline script uses the OpenAI client interface and supports any OpenAI-compatible endpoint.
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
- `HF_TOKEN`
|
| 145 |
-
- `API_BASE_URL`
|
| 146 |
-
- `MODEL_NAME`
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
```
|
| 152 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 153 |
```
|
| 154 |
|
| 155 |
The script emits strict stdout lines in the required format:
|
| 156 |
|
| 157 |
-
- `[START]`
|
| 158 |
-
- `[STEP]`
|
| 159 |
-
- `[END]`
|
| 160 |
-
|
| 161 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 162 |
|
| 163 |
## Example Gold Scores
|
| 164 |
|
|
@@ -168,20 +187,65 @@ Using the included scripted policy:
|
|
| 168 |
- `duplicate_charge_refund`: `1.0`
|
| 169 |
- `enterprise_data_loss_escalation`: `1.0`
|
| 170 |
|
| 171 |
-
## Deployment Notes
|
| 172 |
-
|
| 173 |
-
-
|
| 174 |
-
-
|
| 175 |
-
-
|
| 176 |
-
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
``
|
| 183 |
-
|
| 184 |
-
```
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: Customer Support OpenEnv
|
| 3 |
+
emoji: "🎫"
|
| 4 |
+
colorFrom: blue
|
| 5 |
+
colorTo: green
|
| 6 |
+
sdk: docker
|
| 7 |
+
app_port: 8000
|
| 8 |
+
short_description: Deterministic B2B SaaS support benchmark.
|
| 9 |
+
pinned: false
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# AcmeCloud Customer Support Ticket Handler
|
| 13 |
+
|
| 14 |
+
A deterministic OpenEnv-style environment for training and evaluating agents on realistic B2B SaaS support workflows.
|
| 15 |
|
| 16 |
## What It Simulates
|
| 17 |
|
|
|
|
| 128 |
|
| 129 |
Open the docs at [http://localhost:8000/docs](http://localhost:8000/docs) or the simple UI at [http://localhost:8000/web](http://localhost:8000/web).
|
| 130 |
|
| 131 |
+
## Docker
|
| 132 |
+
|
| 133 |
+
```bash
|
| 134 |
+
docker build -t customer-support-openenv .
|
| 135 |
+
docker run -p 8000:8000 customer-support-openenv
|
| 136 |
+
```
|
| 137 |
+
|
| 138 |
+
## Baseline Inference
|
| 139 |
+
|
| 140 |
+
The baseline script uses the OpenAI client interface and supports any OpenAI-compatible endpoint.
|
| 141 |
+
|
| 142 |
+
Mandatory environment variables for hosted model inference:
|
| 143 |
+
|
| 144 |
+
- `HF_TOKEN`
|
| 145 |
+
- `API_BASE_URL`
|
| 146 |
+
- `MODEL_NAME`
|
| 147 |
+
|
| 148 |
+
Optional environment variables:
|
| 149 |
+
|
| 150 |
+
- `ENV_BASE_URL` to target a running local server or deployed HF Space
|
| 151 |
+
- `LOCAL_IMAGE_NAME` if you want the script to instantiate the environment via `from_docker_image(...)`
|
| 152 |
+
|
| 153 |
+
Inference environment selection:
|
| 154 |
+
|
| 155 |
+
1. `LOCAL_IMAGE_NAME` set: use `from_docker_image(...)`
|
| 156 |
+
2. otherwise `ENV_BASE_URL` set: use the running HTTP environment
|
| 157 |
+
3. otherwise: use the in-process local environment for offline reproducibility
|
| 158 |
+
|
| 159 |
+
Run:
|
| 160 |
+
|
| 161 |
+
```bash
|
| 162 |
+
python inference.py
|
| 163 |
```
|
| 164 |
|
| 165 |
The script emits strict stdout lines in the required format:
|
| 166 |
|
| 167 |
+
- `[START]`
|
| 168 |
+
- `[STEP]`
|
| 169 |
+
- `[END]`
|
| 170 |
+
|
| 171 |
+
Output contract:
|
| 172 |
+
|
| 173 |
+
- one `[START]` line per task
|
| 174 |
+
- one `[STEP]` line immediately after each `env.step()`
|
| 175 |
+
- one `[END]` line per task, even on exception
|
| 176 |
+
- `reward`, `rewards`, and `score` formatted to 2 decimal places
|
| 177 |
+
- `done` and `success` emitted as lowercase booleans
|
| 178 |
+
- `error` emitted as the raw `last_action_error` string or `null`
|
| 179 |
+
|
| 180 |
+
If the model call fails or credentials are missing, the script falls back to a deterministic scripted policy so the benchmark still runs reproducibly.
|
| 181 |
|
| 182 |
## Example Gold Scores
|
| 183 |
|
|
|
|
| 187 |
- `duplicate_charge_refund`: `1.0`
|
| 188 |
- `enterprise_data_loss_escalation`: `1.0`
|
| 189 |
|
| 190 |
+
## Deployment Notes
|
| 191 |
+
|
| 192 |
+
- HF Space page: `https://huggingface.co/spaces/Dar3devil/customer-support-openenv`
|
| 193 |
+
- HF app URL: `https://dar3devil-customer-support-openenv.hf.space`
|
| 194 |
+
- The app exposes `/health`, `/reset`, `/step`, `/state`, `/docs`, `/web`, and `/ws`
|
| 195 |
+
- Sessions are managed in-memory
|
| 196 |
+
- No external services are required to run the environment server itself
|
| 197 |
+
- The benchmark is designed to fit comfortably in the hackathon resource limits
|
| 198 |
+
|
| 199 |
+
## Validation
|
| 200 |
+
|
| 201 |
+
If `openenv` is installed locally, run:
|
| 202 |
+
|
| 203 |
+
```bash
|
| 204 |
+
openenv validate
|
| 205 |
+
```
|
| 206 |
+
|
| 207 |
+
## Pre-Submission Commands
|
| 208 |
+
|
| 209 |
+
Local checks:
|
| 210 |
+
|
| 211 |
+
```powershell
|
| 212 |
+
cd "C:\Users\aarya\.codex\worktrees\e74f\Task Scheduler\customer_support_openenv"
|
| 213 |
+
openenv validate
|
| 214 |
+
pytest -q
|
| 215 |
+
```
|
| 216 |
+
|
| 217 |
+
Baseline run:
|
| 218 |
+
|
| 219 |
+
```powershell
|
| 220 |
+
$env:HF_TOKEN="<your-hf-token>"
|
| 221 |
+
$env:API_BASE_URL="https://router.huggingface.co/v1"
|
| 222 |
+
$env:MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
|
| 223 |
+
python inference.py
|
| 224 |
+
```
|
| 225 |
+
|
| 226 |
+
Local Docker smoke test:
|
| 227 |
+
|
| 228 |
+
```powershell
|
| 229 |
+
docker build -t customer-support-openenv .
|
| 230 |
+
docker run --rm -p 8000:8000 customer-support-openenv
|
| 231 |
+
curl.exe -sS http://localhost:8000/health
|
| 232 |
+
curl.exe -sS -X POST http://localhost:8000/reset -H "Content-Type: application/json" -d "{}"
|
| 233 |
+
```
|
| 234 |
+
|
| 235 |
+
Live Space smoke test:
|
| 236 |
+
|
| 237 |
+
```powershell
|
| 238 |
+
curl.exe -sS https://dar3devil-customer-support-openenv.hf.space/health
|
| 239 |
+
curl.exe -sS -X POST https://dar3devil-customer-support-openenv.hf.space/reset -H "Content-Type: application/json" -d "{}"
|
| 240 |
+
```
|
| 241 |
+
|
| 242 |
+
Submission validator:
|
| 243 |
+
|
| 244 |
+
```powershell
|
| 245 |
+
wsl bash -lc "cd '/mnt/c/Users/aarya/.codex/worktrees/e74f/Task Scheduler/customer_support_openenv' && chmod +x scripts/validate-submission.sh && scripts/validate-submission.sh https://dar3devil-customer-support-openenv.hf.space ."
|
| 246 |
+
```
|
| 247 |
+
|
| 248 |
+
Windows users should run the validator script through WSL or Git Bash.
|
| 249 |
+
|
| 250 |
+
This repository does not depend on an LLM judge for grading.
|
| 251 |
+
All graders are deterministic and implemented directly in the environment scorer.
|