AgentBeats Phase 2 — OpenEnv Challenge Submission | Author: MateFin
--- ## Agents should be judged by whether they finish the job Large language models are often evaluated on what they can **say**. Real agents, however, are judged by whether they can **finish the job** when tools fail. In practical API workflows, failure rarely comes from language alone. Pages drift. Duplicate rows appear across requests. Rate limits interrupt execution. Transient server errors force retries. Summary rows contaminate aggregates. Budgets make brute-force strategies impossible. These are not unusual edge cases. **They are normal operating conditions for production systems.** ComtradeBench is an OpenEnv benchmark designed to measure exactly this problem: can an LLM agent execute a multi-step API workflow reliably under realistic failure modes? --- ## Why this benchmark matters Many current evaluations still focus on final answers, clean tool calls, or static environments. But deployed agents fail for more operational reasons: | Failure | What goes wrong | |---------|----------------| | Miss pages | Incomplete data submitted as complete | | Retry incorrectly | Page skipped after error — silent data gap | | Double-count duplicates | Overcounted rows, inflated aggregates | | Leak summary rows | Contaminated totals corrupt downstream analysis | | Waste budget | Redundant fetches exhaust request limit | | Recover silently | No auditable trace — failure invisible in production | These are **execution failures**, not just reasoning failures. If we want useful agents, we need benchmarks that measure reliable task completion under imperfect conditions — not only answer quality in idealized settings. --- ## What ComtradeBench is > ComtradeBench is an OpenEnv-native benchmark and training environment for reliable tool-use. The domain is trade-data retrieval; the problem is broader: robust multi-step API execution under shifting, imperfect, and partially adversarial conditions. The environment asks an agent to retrieve, clean, and submit records from a paginated API while handling: - **Pagination drift** — page ordering randomized between calls - **Duplicate records** — within-page (8%) and cross-page (3%) overlap - **Transient errors** — HTTP 429 rate-limits and HTTP 500 server faults - **Totals trap** — synthetic summary rows mixed into real data - **Mixed faults** — rate-limit retry + dedup simultaneously - **Constrained budget** — halved request limit, no room for waste The goal is not to test whether the agent can *describe* the workflow. The goal is to test whether it can *execute* it — correctly, completely, efficiently, and robustly. --- ## Environment design Each episode gives the agent a parameterized retrieval task and a limited request budget. The agent interacts through **three MCP tools only**: ``` get_task_info() → task parameters + request budget fetch_page(page, size) → {rows, has_more} or {status: 429|500, retry: true} submit_results(...) → {reward, score, breakdown} ``` The benchmark is structured as a **curriculum of ten tasks**: | # | Task | Core challenge | |---|------|----------------| | T1 | Single page | Baseline correctness | | T2 | Multi-page pagination | Merge 2,345+ rows across pages | | T3 | Duplicates | Primary-key deduplication | | T4 | HTTP 429 | Backoff + retry without data loss | | T5 | HTTP 500 | Transient error recovery | | T6 | Page drift | Canonicalize under non-deterministic ordering | | T7 | Totals trap | Filter `is_total=true` rows | | T8 | Mixed faults | Retry AND dedup simultaneously | | **T9** | **Adaptive adversary** | **Fault intensity escalates mid-episode** | | **T10** | **Constrained budget** | **50 requests instead of 100** | T9 is, to our knowledge, among the earliest OpenEnv-style tasks to model **within-episode fault escalation** — where the environment becomes harder as the agent makes progress. --- ## Why OpenEnv We built ComtradeBench on OpenEnv because this benchmark is meant to be more than a one-off simulator. OpenEnv gives us a standard environment interface, reproducible execution, and clean integration with evaluation and post-training workflows. The same environment code runs both in-process during GRPO training and as a deployed Docker service during evaluation — with no divergence. Our goal is not only to score agents, but to provide a **reusable environment where robustness can be studied and trained systematically**. --- ## Scoring what actually matters ComtradeBench uses structured evaluation across **six dimensions** — not a binary pass/fail: | Dimension | Weight | What it measures | |-----------|:------:|-----------------| | Correctness | **30%** | All expected rows present with correct field values | | Completeness | 15% | Zero missing records | | Robustness | 15% | Correct fault handling with logged evidence | | Efficiency | 15% | Request count vs. task-optimal minimum | | Data Quality | 15% | No duplicates or leaked totals rows | | Observability | 10% | Structured execution trace in the run log | **Why multi-dimensional scoring matters:** An agent that retrieves correct data but skips retry logging loses 15 points on Robustness. An agent that skips pages to save budget loses Completeness and all Efficiency credit. These behaviors are not equivalent — the benchmark does not treat them as equivalent. The **Observability** dimension deserves special note: requiring structured log entries incentivizes the agent to maintain explicit execution state. This is not artificial — structured logs are how production ETL pipelines are monitored and debugged. --- ## Baselines and results ### Rule-based baseline (no LLM) A deterministic rule-based agent achieves **96.8 / 100** average across all ten tasks, confirming the environment is well-calibrated and solvable. | Task | Score | Reward | |------|------:|-------:| | T1 Single page | 98.0 | 0.980 | | T2 Multi-page | 98.0 | 0.980 | | T3 Duplicates | 98.0 | 0.980 | | T4 Rate limit (429) | 95.0 | 0.950 | | T5 Server error (500) | 95.7 | 0.957 | | T6 Page drift | 94.0 | 0.940 | | T7 Totals trap | 98.0 | 0.980 | | T8 Mixed faults | 96.4 | 0.964 | | T9 Adaptive adversary | 96.9 | 0.969 | | T10 Constrained budget | 98.0 | 0.980 | | **Average** | **96.8** | **0.968** | ### LLM agent — Moonshot V1-8K (Kimi API) | Task | Score | Reward | |------|------:|-------:| | T1 Single page | 98.7 | 0.987 | | T2 Multi-page | 98.7 | 0.987 | | T3 Duplicates | 98.7 | 0.987 | | T4 Rate limit | 83.7 | 0.837 | | T5 Server error | 84.3 | 0.843 | | T6 Page drift | 94.7 | 0.947 | | T7 Totals trap | 98.7 | 0.987 | | T8 Mixed faults | 97.3 | 0.973 | | **Average** | **94.4** | **0.944** | The LLM outperforms the rule-based baseline on Observability — natural language models generate more informative execution traces. The gap on T4/T5 reflects that the Robustness dimension requires **explicit logged evidence** of retry behavior, not just correct output. ### GRPO training curve We ran 8 iterations of GRPO-style rollouts with group-relative advantage normalization. Training signal is reward-only — no human labels, no reward model. Mean reward exceeded the rule-based baseline in **6 of 8 iterations**.
--- ### 💬 *Can an agent still finish the job when the API fights back?* ---
That question matters far beyond trade data. It applies to any agent expected to operate against real interfaces with pagination, retries, noisy outputs, and resource limits. If we want more reliable agents, we need environments that reward reliability directly. That is the role ComtradeBench is designed to play. ---