docs/test_usage_report_1.md

# CustomerSupportEnv Usage Report

Date: 2026-04-08
Environment: Live API at `http://localhost:7860`
Method: Manual customer-style API testing with `curl`

## 1. Access and Base URL Checks

- `GET http://0.0.0.0:7860/` -> `200 OK` (service reachable, root metadata returned)
- `GET http://localhost:7860/` -> `200 OK` (recommended local URL)
- `GET http://localhot:7860/` -> failed (`Could not resolve host`)

Note: `localhot` is a typo. Use `localhost`.

## 2. Health Check

- `GET /health` response:
  - `{"status":"ok","active_sessions":4}`
- Result: API is healthy and serving requests.

## 3. End-to-End Scenario Results (All Tasks)

### Easy Scenario (`task=easy`)
Ticket observed: `TKT-001` (billing, medium)

Flow executed:
1. `respond`
2. `request_info`
3. `close`

Observed outcome:
- Episode completed: `done=true`
- Terminal reward value: `0.5229`
- Final score: `1.0`
- Steps used: `3`
- Behavior: Correctly gathered required info and closed with refund resolution.

### Medium Scenario (`task=medium`)
Ticket observed: `TKT-015` (technical, medium)

Flow executed:
1. `respond` (empathetic)
2. `request_info`
3. `respond` (workaround/solution)
4. `close`

Observed outcome:
- Episode completed: `done=true`
- Terminal reward value: `0.7511`
- Final score: `1.0`
- Steps used: `4`
- Behavior: Strong handling of multi-turn support with information gathering and practical fix.

### Hard Scenario (`task=hard`)
Ticket observed: `TKT-022` (technical, critical)

Flow executed:
1. `respond` (acknowledge urgency)
2. `escalate` (SLA/critical urgency in reason)

Observed outcome:
- Episode completed: `done=true`
- Terminal reward value: `0.6282`
- Final score: `0.955`
- Steps used: `2`
- Behavior: Correct critical-incident triage (early escalation with urgency language).

## 4. API Usage Summary

Primary endpoints validated in real usage:
- `POST /reset?task=easy|medium|hard`
- `POST /step?session_id=...`
- `GET /health`
- `GET /`

Request contract validated:
- `step` accepts action payloads:
  - `{"action_type":"respond","message":"..."}`
  - `{"action_type":"request_info","message":"..."}`
  - `{"action_type":"close","message":"..."}`
  - `{"action_type":"escalate","reason":"..."}`

Response contract observed:
- `observation` object updates each step
- `reward` object includes `value`, component scores, and breakdown
- `done` flips to `true` on terminal actions
- `final_score` appears on terminal responses

## 5. Operational Notes

- The application is suitable for demo and integration testing of support-agent action policies.
- Reward shaping is clearly exposed and useful for debugging policy behavior.
- `active_sessions` increases during testing; completed sessions are cleaned up when done, and stale sessions are managed by TTL logic.

## 6. Recommended Next Test Pass (Optional)

1. Negative tests:
   - invalid `session_id`
   - malformed action payloads
   - `step` after terminal state
2. Load/concurrency tests:
   - multiple concurrent sessions across all tasks
3. Regression automation:
   - convert these `curl` flows into a repeatable shell script or pytest API tests

## 7. Single Scenario Report (Latest Live Run)

Scenario ID: `medium-live-2026-04-08`
Execution type: Manual `curl` flow against running server
Start endpoint: `POST /reset?task=medium`

Initial context:
- Session ID: `1c715e2f-1af0-474c-bd81-d1362d48d690`
- Ticket ID: `TKT-019`
- Category: `account`
- Priority: `medium`
- Subject: `Email notifications stopped arriving`

Action sequence executed:
1. `respond` -> "I understand this is frustrating and I am here to help."
2. `request_info` -> "Please share your account email and device details so I can investigate this properly."
3. `respond` -> "Please try signing out, clearing app cache, and updating to the latest version."
4. `close` -> "Issue appears resolved with the workaround and verification. Closing this ticket."

Observed metrics by step:
- Step 1: `done=false`, reward `0.1473`
- Step 2: `done=false`, reward `0.2992`
- Step 3: `done=false`, reward `0.2693`
- Step 4: `done=true`, terminal reward `0.6678`, `final_score=1.0`

Final state summary:
- Episode completed successfully in `4` steps
- Final customer sentiment: `0.203`
- Unresolved issues: `[]`
- Info/action log returned by API without errors (`"error": null`)

Conclusion:
- This medium-difficulty customer scenario passed end-to-end and validates expected behavior:
   - empathy -> info gathering -> actionable resolution -> closure.