Dar3devil commited on
Commit
92cb19d
·
verified ·
1 Parent(s): dfdec70

Sync README

Browse files
Files changed (1) hide show
  1. README.md +122 -58
README.md CHANGED
@@ -1,17 +1,17 @@
1
- ---
2
- title: Customer Support OpenEnv
3
- emoji: "🎫"
4
- colorFrom: blue
5
- colorTo: green
6
- sdk: docker
7
- app_port: 8000
8
- short_description: Deterministic B2B SaaS support benchmark.
9
- pinned: false
10
- ---
11
-
12
- # AcmeCloud Customer Support Ticket Handler
13
-
14
- A deterministic OpenEnv-style environment for training and evaluating agents on realistic B2B SaaS support workflows.
15
 
16
  ## What It Simulates
17
 
@@ -128,37 +128,56 @@ uvicorn server.app:app --host 0.0.0.0 --port 8000 --reload
128
 
129
  Open the docs at [http://localhost:8000/docs](http://localhost:8000/docs) or the simple UI at [http://localhost:8000/web](http://localhost:8000/web).
130
 
131
- ## Docker
132
-
133
- ```bash
134
- docker build -t customer-support-openenv -f server/Dockerfile .
135
- docker run -p 8000:8000 customer-support-openenv
136
- ```
137
-
138
- ## Baseline Inference
139
-
140
- The baseline script uses the OpenAI client interface and supports any OpenAI-compatible endpoint.
141
-
142
- Environment variables:
143
-
144
- - `HF_TOKEN` or `OPENAI_API_KEY`
145
- - `API_BASE_URL`
146
- - `MODEL_NAME`
147
- - optional `ENV_BASE_URL` if you want the script to hit a running server instead of the in-process environment
148
-
149
- Run:
150
-
151
- ```bash
152
- python inference.py
 
 
 
 
 
 
 
 
 
 
153
  ```
154
 
155
  The script emits strict stdout lines in the required format:
156
 
157
- - `[START]`
158
- - `[STEP]`
159
- - `[END]`
160
-
161
- If the model call fails or credentials are missing, the script falls back to a deterministic scripted policy so the benchmark still runs reproducibly.
 
 
 
 
 
 
 
 
 
162
 
163
  ## Example Gold Scores
164
 
@@ -168,20 +187,65 @@ Using the included scripted policy:
168
  - `duplicate_charge_refund`: `1.0`
169
  - `enterprise_data_loss_escalation`: `1.0`
170
 
171
- ## Deployment Notes
172
-
173
- - The app exposes `/health`, `/reset`, `/step`, `/state`, `/docs`, `/web`, and `/ws`
174
- - Sessions are managed in-memory
175
- - No external services are required to run the environment server itself
176
- - The benchmark is designed to fit comfortably in the hackathon resource limits
177
-
178
- ## Validation
179
-
180
- If `openenv` is installed locally, run:
181
-
182
- ```bash
183
- openenv validate
184
- ```
185
-
186
- This repository does not depend on an LLM judge for grading.
187
- All graders are deterministic and implemented directly in the environment scorer.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: Customer Support OpenEnv
3
+ emoji: "🎫"
4
+ colorFrom: blue
5
+ colorTo: green
6
+ sdk: docker
7
+ app_port: 8000
8
+ short_description: Deterministic B2B SaaS support benchmark.
9
+ pinned: false
10
+ ---
11
+
12
+ # AcmeCloud Customer Support Ticket Handler
13
+
14
+ A deterministic OpenEnv-style environment for training and evaluating agents on realistic B2B SaaS support workflows.
15
 
16
  ## What It Simulates
17
 
 
128
 
129
  Open the docs at [http://localhost:8000/docs](http://localhost:8000/docs) or the simple UI at [http://localhost:8000/web](http://localhost:8000/web).
130
 
131
+ ## Docker
132
+
133
+ ```bash
134
+ docker build -t customer-support-openenv .
135
+ docker run -p 8000:8000 customer-support-openenv
136
+ ```
137
+
138
+ ## Baseline Inference
139
+
140
+ The baseline script uses the OpenAI client interface and supports any OpenAI-compatible endpoint.
141
+
142
+ Mandatory environment variables for hosted model inference:
143
+
144
+ - `HF_TOKEN`
145
+ - `API_BASE_URL`
146
+ - `MODEL_NAME`
147
+
148
+ Optional environment variables:
149
+
150
+ - `ENV_BASE_URL` to target a running local server or deployed HF Space
151
+ - `LOCAL_IMAGE_NAME` if you want the script to instantiate the environment via `from_docker_image(...)`
152
+
153
+ Inference environment selection:
154
+
155
+ 1. `LOCAL_IMAGE_NAME` set: use `from_docker_image(...)`
156
+ 2. otherwise `ENV_BASE_URL` set: use the running HTTP environment
157
+ 3. otherwise: use the in-process local environment for offline reproducibility
158
+
159
+ Run:
160
+
161
+ ```bash
162
+ python inference.py
163
  ```
164
 
165
  The script emits strict stdout lines in the required format:
166
 
167
+ - `[START]`
168
+ - `[STEP]`
169
+ - `[END]`
170
+
171
+ Output contract:
172
+
173
+ - one `[START]` line per task
174
+ - one `[STEP]` line immediately after each `env.step()`
175
+ - one `[END]` line per task, even on exception
176
+ - `reward`, `rewards`, and `score` formatted to 2 decimal places
177
+ - `done` and `success` emitted as lowercase booleans
178
+ - `error` emitted as the raw `last_action_error` string or `null`
179
+
180
+ If the model call fails or credentials are missing, the script falls back to a deterministic scripted policy so the benchmark still runs reproducibly.
181
 
182
  ## Example Gold Scores
183
 
 
187
  - `duplicate_charge_refund`: `1.0`
188
  - `enterprise_data_loss_escalation`: `1.0`
189
 
190
+ ## Deployment Notes
191
+
192
+ - HF Space page: `https://huggingface.co/spaces/Dar3devil/customer-support-openenv`
193
+ - HF app URL: `https://dar3devil-customer-support-openenv.hf.space`
194
+ - The app exposes `/health`, `/reset`, `/step`, `/state`, `/docs`, `/web`, and `/ws`
195
+ - Sessions are managed in-memory
196
+ - No external services are required to run the environment server itself
197
+ - The benchmark is designed to fit comfortably in the hackathon resource limits
198
+
199
+ ## Validation
200
+
201
+ If `openenv` is installed locally, run:
202
+
203
+ ```bash
204
+ openenv validate
205
+ ```
206
+
207
+ ## Pre-Submission Commands
208
+
209
+ Local checks:
210
+
211
+ ```powershell
212
+ cd "C:\Users\aarya\.codex\worktrees\e74f\Task Scheduler\customer_support_openenv"
213
+ openenv validate
214
+ pytest -q
215
+ ```
216
+
217
+ Baseline run:
218
+
219
+ ```powershell
220
+ $env:HF_TOKEN="<your-hf-token>"
221
+ $env:API_BASE_URL="https://router.huggingface.co/v1"
222
+ $env:MODEL_NAME="Qwen/Qwen2.5-72B-Instruct"
223
+ python inference.py
224
+ ```
225
+
226
+ Local Docker smoke test:
227
+
228
+ ```powershell
229
+ docker build -t customer-support-openenv .
230
+ docker run --rm -p 8000:8000 customer-support-openenv
231
+ curl.exe -sS http://localhost:8000/health
232
+ curl.exe -sS -X POST http://localhost:8000/reset -H "Content-Type: application/json" -d "{}"
233
+ ```
234
+
235
+ Live Space smoke test:
236
+
237
+ ```powershell
238
+ curl.exe -sS https://dar3devil-customer-support-openenv.hf.space/health
239
+ curl.exe -sS -X POST https://dar3devil-customer-support-openenv.hf.space/reset -H "Content-Type: application/json" -d "{}"
240
+ ```
241
+
242
+ Submission validator:
243
+
244
+ ```powershell
245
+ wsl bash -lc "cd '/mnt/c/Users/aarya/.codex/worktrees/e74f/Task Scheduler/customer_support_openenv' && chmod +x scripts/validate-submission.sh && scripts/validate-submission.sh https://dar3devil-customer-support-openenv.hf.space ."
246
+ ```
247
+
248
+ Windows users should run the validator script through WSL or Git Bash.
249
+
250
+ This repository does not depend on an LLM judge for grading.
251
+ All graders are deterministic and implemented directly in the environment scorer.