README.md

metadata

title: AdaptiveWorld Env
emoji: 🌐
colorFrom: indigo
colorTo: purple
sdk: docker
app_file: server/app.py
app_port: 7860
pinned: false

AdaptiveWorld Env

Theme 3.1 (World Modeling / Professional Tasks) + Theme 5 (Wild Card)
Sub-theme: Patronus AI — Consumer Workflows with Schema Drift
Built on: ProthamD/api-debug-env (Round 1)

---

What This Is

An OpenEnv environment where an AI agent completes multi-step professional tasks (booking, ordering, filing claims) while the world mutates mid-episode without announcement.

API schemas change, field names shift, policies flip, and sometimes the API returns 200 OK but the meaning of the response has silently changed.

Five novel contributions (stacked)

1. Non-stationary world — environment mutates mid-episode (drift mechanic)
2. Dual metrics — task_reward + belief_accuracy tracked separately
3. Adaptive difficulty — environment auto-escalates when agent masters easy drift types
4. Drift provenance hidden — agent must infer drift from evidence, not from being told
5. Cross-episode persistence — agent's world model carries across episodes

No existing OpenEnv environment has any of these five properties simultaneously.

---

Quick Start

# Install
pip install -e .

# Run server
uvicorn server.app:app --host 0.0.0.0 --port 7860

# Health check
curl http://localhost:7860/health

# Run inference
python inference.py --difficulty easy --episodes 3

---

Project Structure

adaptive-world-env/
├── server/
│   ├── app.py                        ← FastAPI entry point
│   ├── adaptive_world_environment.py ← Core environment logic
│   ├── mock_api.py                   ← Dynamic mock API (4 domains)
│   ├── drift_injector.py             ← World state mutation manager
│   └── difficulty_controller.py     ← Adaptive difficulty escalation
├── scenarios/
│   ├── easy.py    (E1: field rename, E2: endpoint version, E3: auth scheme)
│   ├── medium.py  (M1: double drift, M2: policy flip, M3: rate limit)
│   ├── hard.py    (H1: silent status, H2: response structure, H3: cascading)
│   └── expert.py  (EX1: transient vs real, EX2: cross-service, EX3: red herring)
├── graders/grader.py                 ← grade_task + grade_belief + infer_belief
├── models.py                         ← AdaptiveAction, AdaptiveObservation, AdaptiveState
├── inference.py                      ← LLM inference loop (OpenAI-compatible)
├── training_notebook.ipynb          ← Colab GRPO training notebook
└── openenv.yaml                      ← OpenEnv spec

---

Action Space

action_type	What it does
call_api	Standard HTTP request to mock API
probe_schema	GET current API schema — reveals post-drift contract
query_history	Review last N step logs — compare pre vs post drift responses
declare_belief	Agent states its current world model (scored later)
submit_result	End episode, agent declares final belief_state

---

Observation Space

All fields the agent sees. Note: drift_occurred and drift_type are intentionally absent. The agent must infer drift from evidence.

Field	Type	Description
task_description	str	What to accomplish
domain	str	e-commerce / hotel / flight / insurance
prior_world_model	dict	Agent's beliefs from previous episodes
current_step	int	Step number
max_steps	int	Episode step limit
last_status_code	int	HTTP status of last call
last_response_body	str	Response body
step_feedback	str	Contextual hint
episode_history	list	Returned by query_history
task_reward	float	Task completion score (on done)
belief_accuracy	float	Belief accuracy score (on done)
difficulty_level	int	Current escalation level (0-3)
reward	float	Combined: 0.7×task + 0.3×belief

---

Drift Types

Type	Error Signal	Example
field_rename	422	qty → quantity
endpoint_version	404	/book → /v2/book
policy_change	403 or 429	Discount requires gold membership
silent_semantic	None (200 OK)	"confirmed" → "approved"

---

Training Results

Metric	Early window (dashboard)	Late window (dashboard)
Task reward (avg)	0.35	0.49
Belief accuracy (avg)	0.16	0.23
Combined reward (avg)	0.30	0.41
Correlation r (task vs belief)	0.957	0.977

These aggregates come from the same 200-step GRPO run (Qwen2.5-3B 4-bit, 4 completions/step) after SFT (30 steps) and before/with DPO (~50 steps), matching the Training Runs dashboard export.

The correlation chart is the headline: early-window correlation is already high (~0.957) on smoothed traces; late-window correlation reaches ~0.977 — task success and belief understanding stay aligned as difficulty ramps.

---

Combating Reward Hacking

During development, we discovered that highly capable models (like Llama-3.3-70B) would exhibit reward hacking. The models were so efficient they would complete the task within 2 steps—before the environment's original drift_trigger_step even had a chance to mutate the API!

Furthermore, when the API returned a 200 OK (even with a silent semantic drift inside the body), the environment was auto-terminating the episode. This short-circuited the agent, preventing it from investigating the response and resulting in a 0.0 belief accuracy score despite a high task reward.

How we solved it:

1. Immediate Drift: We lowered drift_trigger_step = 1 for most scenarios. The world mutates immediately on the very first API call, forcing the agent to confront the non-stationary world.
2. Preventing Auto-Termination: We removed the environment's premature auto-termination on task completion. Instead of ending the episode immediately upon a 200 OK, the environment returns a feedback hint: "Task completed successfully. Please review the response for any silent changes, and use 'submit_result' to end the episode." This forces the agent to explicitly declare its understanding of the new API contract before scoring occurs.

---

HF Space Deployment

# Create new HF Space (Docker)
huggingface-cli repo create prothamd-adaptive-world-env --type space --space-sdk docker

# Push
git remote add hf https://huggingface.co/spaces/ProthamD/prothamd-adaptive-world-env
git push hf main

Space URL: https://huggingface.co/spaces/ProthamD/prothamd-adaptive-world-env
API URL: https://prothamd-adaptive-world-env.hf.space

---

Run Tests

pytest tests/ -v

---

Built on: ProthamD/api-debug-env (Round 1 submission)
Version: 2.3 — Adaptive Difficulty + Drift Provenance Hidden + Cross-Episode Persistence

Licensed under CC BY-NC 4.0 — https://creativecommons.org/licenses/by-nc/4.0/