README.md

---
sdk: static
---

# ToolGym

**ToolGym** is an **open-world tool-using environment** for *scalable agent testing and data curation*.

> Large tool pools • long-horizon workflows • wild constraints • unreliable tool states  

---

## Quick links

- 🏆 **[Leaderboard](https://huggingface.co/spaces/ToolGym/leaderboard)**
- 📦 **Dataset(s)**: `/datasets/ToolGym/ToolGym`
- 📄 **[Paper]**(https://arxiv.org/abs/2601.06328)
- 💻 **[Code](https://github.com/Ziqiao-git/ToolGym)**

---

## Key highlights

- **5,571** validated tools (unified in **MCP format**)  
- **204** real-world apps covered, from **276** MCP servers  
- Long-horizon, constraint-dense tasks  
  - Avg. **28.5** tool-use rounds per task (**averaged across evaluated models**)  
- A **State Controller** that injects realistic failures & drift  
  (timeouts, rate limits, transient unavailability, etc.)  
- **Planner–Actor** agent framework  
  - ToolGym supports and releases data signals for **both**:  
    - **Planner**: deliberate reasoning, reflection, progress tracking, self-correction  
    - **Actor**: step-wise tool retrieval, invocation, and execution  
- **Data-efficient training (experiment)**: we show strong gains using only **1,170** curated training samples  
  (this number refers to the *training subset used in our experiments*, not the full scale/upper bound of ToolGym as a data engine)

---

## What is ToolGym?

ToolGym is designed to close the gap between “clean” function-calling benchmarks and **messy real-world tool ecosystems**. It supports both:

- **Benchmarking**: stress-test agents on long, multi-tool workflows under constraints and failures  
- **Data curation**: collect high-quality trajectories for training tool-using agents  

---

## Core components

### 1) Tool universe (MCP)

We curate and validate a large library of production-like tools, then standardize them under a unified **Model Context Protocol (MCP)** interface so agents can call tools consistently across apps and servers.

### 2) Tool retrieval index

Because open-world tool selection is the real challenge, ToolGym includes a retrieval layer so agents can search tools using natural language queries and load relevant tools on demand.

### 3) Task creation engine

ToolGym synthesizes **long-horizon, multi-tool workflows** that resemble real user requests:
- multi-step dependencies
- cross-app orchestration
- dense constraints (format, ordering, trade-offs, verification requirements, etc.)

### 4) State Controller (robustness testing)

To go beyond “happy-path” evaluation, ToolGym introduces a controllable middleware that can inject:
- tool-level failures (timeouts, temporary unavailability)
- state-level drift (corrupted/delayed results, expired sessions)
- constraint changes mid-execution (updated preferences, shifting deadlines)

### 5) Evaluation protocol

ToolGym evaluates agents on multiple axes, including:
- **Answer quality** (completeness, grounding)
- **Robustness** (schema compliance, recovery, flexibility)
- **Constraint following** (format + other constraints)
- **Planning** (goal decomposition, progress tracking, efficiency)

### 6) Planner–Actor decomposition

To better handle long-horizon objectives and error-prone tool ecosystems, we separate agent behavior into:
- **Planner**: global reasoning & self-correction (keeps the agent aligned over long trajectories)  
- **Actor**: efficient step-by-step execution (retrieval → tool call → observe → iterate)  

---

## Leaderboard

We maintain a public leaderboard for ToolGym.  
➡️ **[Leaderboard link](https://huggingface.co/spaces/ToolGym/leaderboard)**

---

## License

- This organization and its public repos are released under the **MIT** license unless otherwise specified in each repo.

---

## Contributing

Community contributions are welcome:
- Open a discussion: `/datasets/ToolGym/ToolGym/discussions`
- Submit PRs to the relevant repo (dataset / code / leaderboard Space)

---

## Contact

For questions, collaborations, or leaderboard submissions, please open an issue/discussion or contact the maintainers via the links above.