Blog.MD

# Building an RL Environment for the Benefit of the Environment

When I first sat down to think about what RL environment to build, I got the usual suggestions everyone gets. Supply and demand simulators, coding agents, inventory management, trading bots, the kind of stuff that shows up on every "RL project ideas" list. None of it felt right, because they were all the same well-trodden ground, useful but unambitious. If I was going to spend weeks on this, I wanted the thing the agent learns to do to actually matter.

So I flipped the question. Instead of asking what makes a clean RL problem, I asked what is a real problem worth solving where RL might help. That gave me a tagline I kept coming back to, an RL environment for the benefit of the environment. Once I had that framing, wildfires were the obvious pick. They are getting worse every year, the dispatch problem is genuinely hard, and the underlying physics is rich enough to be interesting without being so chaotic that learning is hopeless.

The first version I asked Claude to build was quick, and maybe too quick. What came back was a grid where cells caught fire if their neighbors were on fire, with a wind variable you could nudge to change the spread probability. It worked, and it even looked like fire if you squinted at the renders, but it was the cartoon version of a wildfire, the kind of thing where you draw flames as orange triangles. The problem was not that it was wrong, the problem was that it was flat. There were no underlying dynamics for an agent to discover, and no reason for the model to learn that a fire on this slope at this time of day was different from one a few cells over. Every fire was basically the same fire, and that is not an environment, it is a screensaver.

The example that kept rattling around in my head was a feedback loop. A forest catches fire, the air above it heats up, hot air rises, that creates a low pressure pocket, surrounding wind rushes in toward the fire, the fire gets more oxygen, it grows bigger, which heats more air, and the cycle repeats. That is a firestorm, and the important part is that nobody has to code it as a rule. It just falls out when you model the base variables honestly. That was the kind of texture I wanted in my environment, things that were not scripted but that emerged on their own when the underlying physics was right.

Modelling every such phenomenon explicitly is impossible because fire science is its own field with its own PhDs, so I made a call. Instead of trying to model the phenomena, I would model the base variables honestly and let the phenomena emerge.

I sat down with the codebase and started layering things in. Wind became a vector with its own daily cycle and random drift, with direction stored in the convention real fire models use, where it points to where the wind is coming from. Humidity got tied inversely to the daily cycle, so things dry out during peak heat the way they actually do. Temperature got a base value plus a daily swing that peaks around 70 percent through the episode. Fuel moisture is tracked per cell, drifting toward equilibrium at fuel-class-specific time constants of one hour for grass and ten hours for brush and forest, which is the time-lag model from the foundational 1972 fire physics paper by Rothermel.

I added elevation and slope, so cells uphill from a fire receive two to three times more heat than downhill neighbors, which is why uphill runs are so dangerous in real life. I added aspect, so north-facing slopes get different temperatures and different drying rates than south-facing ones, meaning south faces stay dry all day while north faces recover overnight. I added fuel types for grass, brush, and forest, each with different burn durations and different responses to moisture.

Then I asked AI to dig up the actual research on how these variables interact, and I wired the formulas in with real coefficients. The Alexandridis 2008 model gives wind-directional spread, which is what creates the elliptical fire perimeters pointing downwind that you see in real fire maps. The Rothermel moisture damping curve is non-linear, so a small moisture change near the burning point flips a fire from uncontrollable to stoppable. Spotting, where embers fly ahead of the main fire and start new ignitions, only kicks in when wind is above 28.8 km per hour and intensity is above a threshold.

The piece I am most proud of is what I call the airflow potential field. It is a hidden field that burning intensity feeds into, that diffuses and decays each step, and whose gradient perturbs the local wind by up to plus or minus 4 km per hour. I never coded a firestorm rule, but because intense fires now generate their own pressure gradient that pulls wind toward them, the firestorm loop emerges from the physics. Big fires literally accelerate their own spread by inducing a draft. The exact phenomenon I had been imagining at the start fell out of the model on its own, and that was the moment the environment stopped being a screensaver.

Once the fire side was honest, the firefighting side had to match. A high-fidelity fire model paired with a cartoon dispatch model would be useless, because the agent would just learn to abuse the cheap abstractions. So I went back through the National Wildfire Coordinating Group doctrine and modelled six resource classes, each with realistic preparation times, travel speeds, and return-to-base overheads.

Hand crews are 20-person ground teams with hand tools that take ten minutes to prep and fifteen minutes to return. Wildland engines are pumpers with foam, faster to deploy and faster to return. Helicopters carry water in a Bambi bucket of three to seven hundred gallons, with smart routing that scoops from the nearest lake instead of base if the lake is closer. Air tankers are large fixed-wing planes carrying two to four thousand gallons of retardant, with a long thirty minute return so each drop is precious. Dozers are slow to roll out at fifteen minutes prep, but their lines are permanent firebreaks, and their production rate depends on the fuel they are cutting through. Smokejumpers are the fastest to deploy at two minutes prep with a quick jump, but they need an extraction helicopter to come pick them up, so they pay an overhead on the way back.

The point of all this granularity is not realism for its own sake, it is that the agent has to learn trade-offs that mirror real incident command decisions. Smokejumpers are fastest in but slowest out. Air tankers are devastating but each drop is expensive and slow to reload. Dozers are slow to start but their lines pay off if you commit early. Helicopters near water get free reloads, so positioning matters.

Each suppression action also has a different mechanical effect. Wet line saturates fuel moisture for a few steps until it drifts back toward equilibrium. Retardant deposits a persistent ignition resistance bonus that decays faster when adjacent cells are actively burning, calibrated against the published USDA effectiveness window of four to five hours for the actual chemical retardants used in real operations. That is the level of grounding I wanted, where every constant in the file should be defensible against a published source.

The reward function took more iterations than the physics. The physics has the courtesy of being grounded in equations, while the reward function is just you, alone, trying to anticipate every way your agent will cheat you. And it will cheat you every single time.

The first version was naive, plus one for every cell saved and minus one for every cell burned. Sounds reasonable, but the agent immediately learned it could do nothing on small fires that would burn out on their own and still rack up positive reward from the unburned cells around them, which is passive reward farming. So I added an active fire pressure penalty for every burning cell on the map. Now the agent had urgency, but on the hard 25 by 25 grid that uniform negative signal drowned out everything else, and the agent learned that the optimal policy was to give up early and minimize regret. So I added a grid-size normalization factor that scales penalties down on bigger maps, so the positive learning signal does not get buried.

Then I caught the agent doing the reverse, dispatching a single hand crew to nibble at the edge of a fire that was about to self-extinguish anyway and claiming credit for containment. So I gated the containment bonus to only fire if the agent actually suppressed something on that tick. You do not get credit for the fire going out unless you helped put it out.

What I converged on after all of this was a small set of principles that hold the reward function together. Reward transitions, not states, so the big penalty fires the moment a structure becomes fully burned rather than continuously while it is on fire. This makes the gradient cleaner because the agent learns what action prevented the bad outcome rather than just that some state is bad. Treat weather as information, not penalty, because penalizing the agent for high winds would just teach it learned helplessness. Use one-time bonuses for one-time events, with tracking sets so the fire-extinguished bonus and structure-lost penalty cannot fire twice. Cap penalties so they keep the gradient signal alive, because without caps one term dominates and the others become invisible. And gate positive signals on agent contribution, so the containment bonus only triggers if the agent actually suppressed something this tick and never gives credit for nature taking its course. The actual numbers are in the code, and they were tuned more by watching the agent break things than by any clean theory.

Building this was less about reinforcement learning and more about honest modelling of a complex system. The RL part is almost incidental, because if you give a competent algorithm an honest environment it will find policies, and if you give it a dishonest one it will find exploits.

The biggest shift in how I think now is this. Do not try to script the interesting behavior, model the substrate honestly and let the interesting behavior emerge. I never wrote a firestorm rule, but I wrote a buoyancy field with realistic decay, and firestorms appeared. I never wrote a rule that fires spread faster uphill, but I wrote a slope coefficient and an elevation grid, and uphill runs emerged. That is the bar I want to hold future environments to, where if the only way an interesting phenomenon shows up is because I hard-coded it, the model is not deep enough yet. Find the underlying variables, wire them up honestly, and let the emergence do the work. And if the agent ends up learning to position fires near lakes for free helicopter reloads, that is not a bug, that is the environment doing its job.

This environment fits most naturally under Theme 3.1, World Modeling for Professional Tasks. The theme description talks about environments that require real interaction with dynamic systems, where the model is expected to do real hard work instead of exploiting shortcuts, and where the agent has to maintain consistent internal state, update beliefs based on outcomes, and orchestrate multi-step workflows. That maps almost word for word to what this environment does. The reward function is explicitly designed to prevent shortcut exploitation. The agent has to track fuel moisture drift, fleet status across the prep and travel and operate and return cycle, and forecasted weather changes. The causal reasoning piece is exactly the firestorm story, where the agent has to learn that wind and slope and fuel and buoyancy compose causally rather than as independent features. And wildfire incident command is a professional task in the most literal sense, because there are certified incident commanders whose entire job is exactly this dispatch problem.

There is also a strong secondary fit to Theme 2, Long-Horizon Planning. Rewards are sparse and delayed, with the fire-extinguished terminal bonus only firing once at the end of the episode. Multi-step reasoning is forced by the resource model, because dozers take fifteen minutes of prep plus travel time before they cut their first foot of line, so commitments must be made many steps before the payoff arrives. And the recover-from-early-mistakes property holds, because if you misallocate the air tanker early it is gone for thirty minutes and you have to adapt the rest of your plan around that gap.

Built for the benefit of the environment, one Rothermel coefficient at a time.