✦ Hackathon Write-up

Teaching Agents to
Understand Change

What happens when the world your agent is operating in mutates mid-episode — no warning, no changelog? This is the problem AdaptiveWorldEnv was built to solve.

Author ProthamD

Theme 3.1 + 5 (Wild Card)

Model Qwen2.5-3B

Pipeline SFT → GRPO → DPO

Pratham

3rd year B.Tech (IT), IIEST Shibpur · OpenEnv Hackathon

The Problem Nobody Talks About

Almost all of us have seen a Google Assistant demo — it schedules appointments, makes calls, books things. But when you say "book a flight, also a hotel, and buy some products" — it falls apart. There are several reasons. One of the biggest: the APIs agents depend on change, silently, mid-operation. No warning. No changelog. The agent has no idea.

Suppose our agent is halfway through booking a flight and the API it was just successfully talking to decides to rename a field, or version its endpoint, or silently change what a response value means. No error — sometimes not even a 4xx. Just a 200 OK with a different meaning. And the agent keeps going, completely wrong, with full confidence.

This is what AdaptiveWorldEnv is about.

The Solution: A Picture of the World

The solution I came up with is from a real-world reference — the way all of us work. When we do something wrong we figure out how we could have done it better. When playing a game, we don't know what the next level will bring. We go in, take actions, win or lose, and then figure out what it actually needed. We build a picture of the world in our head as we go.

So that is exactly what I gave the agent — a picture of the world. And to measure how accurate that picture is, I introduced a metric other than just REWARD:

combined_reward = 0.7 × task_reward + 0.3 × belief_accuracy

task_reward tells us if the agent completed the task. belief_accuracy tells us if the agent actually understood what was happening. Those two things are not always the same — an agent can get lucky and complete a task with a completely wrong understanding of the world, and you'd never know from reward alone.

How It Actually Works

The environment gives the agent a task — place an order, book a flight, file a claim. The agent starts calling APIs, getting responses. Simple enough. But at some point mid-episode, without telling the agent, the world mutates.

A field gets renamed. An endpoint gets versioned. A policy flips. Or the scariest one — the API returns 200 OK, same endpoint, same call, but the meaning of a response value has silently changed. No error. Nothing. Just wrong.

The agent has to figure it out — not because somebody told it something changed, but because it starts seeing things that don't match what it knew before. So it calls query_history, compares responses, calls probe_schema to check what the API actually expects today, updates its picture of the world, and retries.

Two Types of Agents

Suppose the agent is doing an e-commerce order. Step 1 and 2 go fine. Step 3 — the world drifts. qty becomes quantity, a new required field customer_id appears. Agent gets a 422.

Agent A just retries slightly differently and somehow gets through. Task reward goes up. But belief accuracy is low — it never figured out what changed.

Agent B calls query_history, compares step 2 response to step 4 response, calls probe_schema, finds the new field names, updates its belief, retries correctly. Task reward goes up. Belief accuracy also goes up. That's the agent we're training.

Adaptive Difficulty

The environment doesn't stay at one difficulty. When the agent gets good at handling a single drift — say a field rename — the environment automatically starts combining drift types in the same episode. Field rename plus a policy change at the same time. Then it starts hiding error signals more. Then it forces silent semantic drift where there's no 4xx anywhere. Difficulty escalates based on how well the agent is actually understanding the world, not just completing tasks.

The agent's world model does not reset between episodes. Whatever the agent learned about how this API behaves — it carries that forward. Just like how when you go back to a game after a break you still remember the patterns. Except here that memory can also hurt you, because the world might have changed.

The Training Pipeline: SFT → GRPO → DPO

The training pipeline has three stages and understanding all of them is important because most people just throw a model into RL and wonder why it's not learning.

Stage 1: SFT (30 steps, lr=5e-5)

I downloaded a dataset of correct behavior examples — broken API call, agent calling probe_schema, comparing history, corrected call with right field name, declared belief state. Basically showing the model what good looks like before asking it to figure out good on its own. Think of it like watching someone else play the game before you pick up the controller.

Stage 2: GRPO (200 steps, lr=2e-6)

Once the model had that baseline I switched to GRPO — group relative policy optimization. The model starts generating its own actions, those actions get sent to the environment, and the reward function scores on both task_reward and belief_accuracy.

Combating Reward Hacking

The model could just always output probe_schema because that always returns a small positive reward (0.05) without doing anything meaningful. Or it could guess the endpoint correctly and get high task reward with a completely wrong belief state.

So: the combined reward weights belief accuracy at 0.3 — you can't fully hack one without the other. Plus the reward function has a local scorer checking actual content — did the field_name in the belief state match the ground truth, did the model actually declare drift_detected: true. And max_grad_norm=0.2 is tight — prevents collapse into one repetitive strategy.

Stage 3: DPO (~50 steps)

During GRPO, every generation saves the prompt, completion, and reward. Those rollouts become preference pairs for DPO — high-reward completions as chosen, low-reward as rejected. About 50 steps of DPO on top of GRPO, then pushed the final model to HuggingFace.

The Results: Honest Numbers

The most important chart is the correlation. Early training still looks messy if we only watch raw step-by-step samples — four GRPO completions per step means some draws look lucky even when belief is wrong. On smoothed rolling windows, an early segment already sits around r≈0.957, and late training pushes to about r≈0.977 in this run. The agent is succeeding because it understood what changed, not despite misunderstanding it. That shift is the whole point of this environment.

Combined reward averaged about 0.41 late vs ~0.30 early, task reward about 0.49 vs ~0.35, belief accuracy about 0.23 vs ~0.16 — measured from the same 200-step GRPO run (4 completions/step). Raw traces stay noisy; smoothed curves and drift-detection rate show stabilization toward the end.

Most environments just show task reward going up and call it done. Here we can see that task reward going up means nothing if belief accuracy is not following. The model needs to earn both. r≈0.977 is not the ceiling — it is the headline measured correlation from this submission run.

Post Hackathon — Honest Assessment

I tried to go beyond traditional RL and ran into several failures during the hackathon window. I retrained many times, burned through most compute credits, and went through a lot of iteration before landing on a training script that held together.

My honest estimate: we are probably looking at needing 1000+ GRPO steps, and only after that would DPO on top start producing really significant improvement. The environment is hard, the reward signal is sparse in early steps, and the model needs a lot more exploration before it can start exploiting what it has learned.

The dual reward signal, belief tracking, three stage pipeline — all of that is sound. It just needs more runway than a hackathon gives. If the judges think this is worth going deeper into — I'm genuinely willing to take that opportunity.

AdaptiveWorld Env

Theme 3.1 (World Modeling / Professional Tasks) + Theme 5 (Wild Card)
Sub-theme: Patronus AI — Consumer Workflows with Schema Drift
Built on: ProthamD/api-debug-env (Round 1)

An OpenEnv environment where an AI agent completes multi-step professional tasks (booking, ordering, filing claims) while the world mutates mid-episode without announcement. API schemas change, field names shift, policies flip, and sometimes the API returns 200 OK but the meaning of the response has silently changed.

Five Novel Contributions

Non-stationary world

Environment mutates mid-episode (drift mechanic)

Dual metrics

task_reward + belief_accuracy tracked separately

Adaptive difficulty

Auto-escalates when agent masters easy drift types

Drift provenance hidden

Agent must infer drift from evidence, not be told

Cross-episode persistence

Agent's world model carries across episodes

Quick Start

# Install
pip install -e .

# Run server
uvicorn server.app:app --host 0.0.0.0 --port 7860

# Health check
curl http://localhost:7860/health

# Run inference
python inference.py --difficulty easy --episodes 3

Project Structure

adaptive-world-env/
├── server/
│   ├── app.py                        ← FastAPI entry point
│   ├── adaptive_world_environment.py ← Core environment logic
│   ├── mock_api.py                   ← Dynamic mock API (4 domains)
│   ├── drift_injector.py             ← World state mutation manager
│   └── difficulty_controller.py     ← Adaptive difficulty escalation
├── scenarios/
│   ├── easy.py    (E1: field rename, E2: endpoint version, E3: auth scheme)
│   ├── medium.py  (M1: double drift, M2: policy flip, M3: rate limit)
│   ├── hard.py    (H1: silent status, H2: response structure, H3: cascading)
│   └── expert.py  (EX1: transient vs real, EX2: cross-service, EX3: red herring)
├── graders/grader.py                 ← grade_task + grade_belief + infer_belief
├── models.py                         ← AdaptiveAction, AdaptiveObservation, AdaptiveState
├── inference.py                      ← LLM inference loop (OpenAI-compatible)
└── openenv.yaml                      ← OpenEnv spec

Action Space

action_type	What it does
`call_api`	Standard HTTP request to mock API
`probe_schema`	GET current API schema — reveals post-drift contract
`query_history`	Review last N step logs — compare pre vs post drift responses
`declare_belief`	Agent states its current world model (scored later)
`submit_result`	End episode, agent declares final belief_state

Drift Types

Type	Error Signal	Example
`field_rename`	422	`qty` → `quantity`
`endpoint_version`	404	`/book` → `/v2/book`
`policy_change`	403 or 429	Discount requires gold membership
`silent_semantic`	None (200 OK)	`"confirmed"` → `"approved"`

Training Results

Metric	Early training (first 20 steps)	Late training (last 20 steps)
Task reward (avg)	0.35	0.49
Belief accuracy (avg)	0.16	0.23
Combined reward (avg)	0.30	0.41
Correlation r (task vs belief)	0.957	0.977

Task and belief averages are roughly stable — expected, as the environment escalated difficulty mid-run (easy → medium at step ~50). The correlation is the headline metric.

Combating Reward Hacking

During development, highly capable models would exhibit reward hacking — completing tasks within 2 steps before the drift trigger even fired. When the API returned 200 OK (even with silent semantic drift inside), the environment was auto-terminating, resulting in 0.0 belief accuracy.

Immediate Drift: Lowered drift_trigger_step = 1 for most scenarios. World mutates on the very first API call.
Preventing Auto-Termination: Removed premature auto-termination on task completion. On 200 OK, environment returns: "Task completed. Please review the response for any silent changes, and use submit_result to end the episode."

HF Space

Space URL: https://huggingface.co/spaces/ProthamD/prothamd-adaptive-world-env
API URL:   https://prothamd-adaptive-world-env.hf.space

The world changes underneath you

Teaching Agents to
Understand Change

The Problem Nobody Talks About

The Solution: A Picture of the World

How It Actually Works

Two Types of Agents

Adaptive Difficulty

The Training Pipeline: SFT → GRPO → DPO

Stage 1: SFT (30 steps, lr=5e-5)

Stage 2: GRPO (200 steps, lr=2e-6)

Combating Reward Hacking

Stage 3: DPO (~50 steps)

The Results: Honest Numbers

Post Hackathon — Honest Assessment

Training Runs

Training Pipeline

Raw Training Signals — Task vs Belief (per step)

Smoothed Metrics + Difficulty Transition

Task vs Belief Correlation — Early (pink) vs Late (green)

Before vs After Training (first 20 / last 20 steps)

Combined Reward over Training

Reward Distribution — Early vs Late

Actual Training Output — GRPO Step-by-Step (200 steps)

AdaptiveWorld Env

Five Novel Contributions

Quick Start

Project Structure

Action Space

Drift Types

Training Results

Combating Reward Hacking

HF Space

Space & API

Teaching Agents toUnderstand Change

The Problem Nobody Talks About

The Solution: A Picture of the World

How It Actually Works

Two Types of Agents

Adaptive Difficulty

The Training Pipeline: SFT → GRPO → DPO

Stage 1: SFT (30 steps, lr=5e-5)

Stage 2: GRPO (200 steps, lr=2e-6)

Combating Reward Hacking

Stage 3: DPO (~50 steps)

The Results: Honest Numbers

Post Hackathon — Honest Assessment

Training Runs

Training Pipeline

Raw Training Signals — Task vs Belief (per step)

Smoothed Metrics + Difficulty Transition

Task vs Belief Correlation — Early (pink) vs Late (green)

Before vs After Training (first 20 / last 20 steps)

Combined Reward over Training

Reward Distribution — Early vs Late

Actual Training Output — GRPO Step-by-Step (200 steps)

AdaptiveWorld Env

Five Novel Contributions

Quick Start

Project Structure

Action Space

Drift Types

Training Results

Combating Reward Hacking

HF Space

Space & API

Teaching Agents to
Understand Change