README.md

---
title: AdaptiveWorld Env
emoji: 🌐
colorFrom: indigo
colorTo: purple
sdk: docker
app_file: server/app.py
app_port: 7860
pinned: false
---

# AdaptiveWorld Env

**Theme 3.1** (World Modeling / Professional Tasks) + **Theme 5** (Wild Card)  
**Sub-theme:** Patronus AI — Consumer Workflows with Schema Drift  
**Built on:** [ProthamD/api-debug-env](https://huggingface.co/spaces/ProthamD/api-debug-env) (Round 1)

---

## What This Is

An OpenEnv environment where an AI agent completes multi-step professional tasks
(booking, ordering, filing claims) while the world **mutates mid-episode without announcement**.

API schemas change, field names shift, policies flip, and sometimes the API returns
`200 OK` but the meaning of the response has silently changed.

### Five novel contributions (stacked)

1. **Non-stationary world** — environment mutates mid-episode (drift mechanic)
2. **Dual metrics** — `task_reward` + `belief_accuracy` tracked separately
3. **Adaptive difficulty** — environment auto-escalates when agent masters easy drift types
4. **Drift provenance hidden** — agent must infer drift from evidence, not from being told
5. **Cross-episode persistence** — agent's world model carries across episodes

No existing OpenEnv environment has any of these five properties simultaneously.

---

## Quick Start

```bash
# Install
pip install -e .

# Run server
uvicorn server.app:app --host 0.0.0.0 --port 7860

# Health check
curl http://localhost:7860/health

# Run inference
python inference.py --difficulty easy --episodes 3
```

---

## Project Structure

```
adaptive-world-env/
├── server/
│   ├── app.py                        ← FastAPI entry point
│   ├── adaptive_world_environment.py ← Core environment logic
│   ├── mock_api.py                   ← Dynamic mock API (4 domains)
│   ├── drift_injector.py             ← World state mutation manager
│   └── difficulty_controller.py     ← Adaptive difficulty escalation
├── scenarios/
│   ├── easy.py    (E1: field rename, E2: endpoint version, E3: auth scheme)
│   ├── medium.py  (M1: double drift, M2: policy flip, M3: rate limit)
│   ├── hard.py    (H1: silent status, H2: response structure, H3: cascading)
│   └── expert.py  (EX1: transient vs real, EX2: cross-service, EX3: red herring)
├── graders/grader.py                 ← grade_task + grade_belief + infer_belief
├── models.py                         ← AdaptiveAction, AdaptiveObservation, AdaptiveState
├── inference.py                      ← LLM inference loop (OpenAI-compatible)
├── training_notebook.ipynb          ← Colab GRPO training notebook
└── openenv.yaml                      ← OpenEnv spec
```

---

## Action Space

| `action_type` | What it does |
|---|---|
| `call_api` | Standard HTTP request to mock API |
| `probe_schema` | GET current API schema — reveals post-drift contract |
| `query_history` | Review last N step logs — compare pre vs post drift responses |
| `declare_belief` | Agent states its current world model (scored later) |
| `submit_result` | End episode, agent declares final `belief_state` |

---

## Observation Space

All fields the agent sees. **Note:** `drift_occurred` and `drift_type` are intentionally absent.
The agent must infer drift from evidence.

| Field | Type | Description |
|---|---|---|
| `task_description` | str | What to accomplish |
| `domain` | str | e-commerce / hotel / flight / insurance |
| `prior_world_model` | dict | Agent's beliefs from previous episodes |
| `current_step` | int | Step number |
| `max_steps` | int | Episode step limit |
| `last_status_code` | int | HTTP status of last call |
| `last_response_body` | str | Response body |
| `step_feedback` | str | Contextual hint |
| `episode_history` | list | Returned by `query_history` |
| `task_reward` | float | Task completion score (on done) |
| `belief_accuracy` | float | Belief accuracy score (on done) |
| `difficulty_level` | int | Current escalation level (0-3) |
| `reward` | float | Combined: 0.7×task + 0.3×belief |

---

## Drift Types

| Type | Error Signal | Example |
|---|---|---|
| `field_rename` | 422 | `qty` → `quantity` |
| `endpoint_version` | 404 | `/book` → `/v2/book` |
| `policy_change` | 403 or 429 | Discount requires gold membership |
| `silent_semantic` | None (200 OK) | `"confirmed"` → `"approved"` |

---

## Training Results

| Metric | Early window (dashboard) | Late window (dashboard) |
|---|---|---|
| Task reward (avg) | 0.35 | 0.49 |
| Belief accuracy (avg) | 0.16 | 0.23 |
| Combined reward (avg) | 0.30 | 0.41 |
| Correlation r (task vs belief) | 0.957 | **0.977** |

These aggregates come from the same 200-step GRPO run (Qwen2.5-3B 4-bit, 4 completions/step)
after SFT (30 steps) and before/with DPO (~50 steps), matching the Training Runs dashboard export.

The correlation chart is the headline: early-window correlation is already high (~0.957) on smoothed traces;
late-window correlation reaches ~**0.977** — task success and belief understanding stay aligned as difficulty ramps.

---

## Combating Reward Hacking

During development, we discovered that highly capable models (like Llama-3.3-70B) would exhibit **reward hacking**. The models were so efficient they would complete the task within 2 steps—before the environment's original `drift_trigger_step` even had a chance to mutate the API!

Furthermore, when the API returned a `200 OK` (even with a silent semantic drift inside the body), the environment was auto-terminating the episode. This short-circuited the agent, preventing it from investigating the response and resulting in a `0.0` belief accuracy score despite a high task reward.

**How we solved it:**
1. **Immediate Drift:** We lowered `drift_trigger_step = 1` for most scenarios. The world mutates immediately on the very first API call, forcing the agent to confront the non-stationary world.
2. **Preventing Auto-Termination:** We removed the environment's premature auto-termination on task completion. Instead of ending the episode immediately upon a `200 OK`, the environment returns a feedback hint: *"Task completed successfully. Please review the response for any silent changes, and use 'submit_result' to end the episode."* This forces the agent to explicitly declare its understanding of the new API contract before scoring occurs.

---

## HF Space Deployment

```bash
# Create new HF Space (Docker)
huggingface-cli repo create prothamd-adaptive-world-env --type space --space-sdk docker

# Push
git remote add hf https://huggingface.co/spaces/ProthamD/prothamd-adaptive-world-env
git push hf main
```

Space URL: `https://huggingface.co/spaces/ProthamD/prothamd-adaptive-world-env`  
API URL: `https://prothamd-adaptive-world-env.hf.space`

---

## Run Tests

```bash
pytest tests/ -v
```

---

*Built on: `ProthamD/api-debug-env` (Round 1 submission)*  
*Version: 2.3 — Adaptive Difficulty + Drift Provenance Hidden + Cross-Episode Persistence*


Licensed under CC BY-NC 4.0 — https://creativecommons.org/licenses/by-nc/4.0/