OutfitterBench: Evaluating LLM Agents on Long-Horizon Resource Allocation and Commitment Coordination
Abstract
OutfitterBench evaluates LLM agents managing a 200-day outdoor-equipment rental business under fixed capital and uncertain demand, revealing limitations in sustaining feasible commitments and reallocating resources.
In many long-horizon settings, an agent's own choices create future constraints: committing finite capacity establishes future obligations and forecloses later opportunities. Sustaining performance therefore requires both keeping evolving plans feasible and comparing the value of resource allocations with the alternatives they displace. Recent benchmarks increasingly evaluate LLM agents in continuing operations, but give less attention to how agents keep their own commitments feasible while reassessing resource allocations as opportunities change. We introduce OutfitterBench, where agents manage an outdoor-equipment rental business for 200 days, purchasing from a 46-product catalog and handling roughly 1,100 reservation requests. Under a fixed capital ceiling, reservations and uncertain walk-in demand compete for reusable equipment whose availability changes over time. Experiments with seven frontier LLMs show that several agents underperform a simple rule-based baseline, while only the leading agent consistently surpasses it. Trajectory analysis shows that some agents apply known constraints inconsistently and make new decisions that conflict with existing commitments. Agents with more reliable fulfillment can still secure low returns from comparable commitments or delay resource reallocation. The leading agent better connects preparation with economic choices but also misses opportunities for more profitable asset use. These findings highlight limitations in sustaining coherent resource plans and reassessing their value over time. The benchmark code is publicly available at https://github.com/adminblackjack/OutfitterBench.
Community
Hi everyone! Author here — happy to share OutfitterBench and answer any questions.
Most agent benchmarks reset constraints between tasks or model consumable inventory. We built OutfitterBench to test what happens over 200 operating days when an agent's own choices create binding future commitments over reusable resources (an outdoor-equipment rental shop with a 46-product catalog, 43 tools, ~1,100 reservation requests, and a fixed capital ceiling).
A few highlights from our study:
- Deterministic & judge-free: Customer choices, turnaround cleaning queues, and our four process-diagnostic families are all seeded and computed via state replay (zero LLM-as-a-judge noise).
- High headroom: Across 7 reasoning LLMs, 6 fail to consistently beat a simple public-information rule baseline (
Calendar), and onlyGPT 6.1 Solsurpasses it on both world seeds. Serving more bookings often destroys net business value when agents drift on turnaround rules, underprice comparable commitments, or delay replacing low-yield gear. - Easy to run: Fully open-sourced (Apache-2.0) and works out of the box with any OpenAI-compatible tool-calling endpoint, with
--stop-after-daysmoke tests and checkpoint resume.
Code & quickstart: https://github.com/adminblackjack/OutfitterBench
We'd love to hear your feedback or see how other models and agent scaffolds perform!
Get this paper in your agent:
hf papers read 2611.17933 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper