Buckets:
| # ThinkingBox | |
| ## Introduction | |
| [ThinkingBox](https://github.com/microsoft/thinkingbox) is an agentic harness for | |
| defining isolated MCP tool environments, creating stateful scenarios and test | |
| cases, running an LLM agent with simulated-user interaction, and evaluating | |
| the resulting backend state, side effects, and response requirements. It can | |
| support offline evaluation or conversation generation and agentic training | |
| workflows. | |
| This OpenEnv adapter exposes the public | |
| [ThinkingBox-Bench](https://arxiv.org/abs/2608.19741) through the standard | |
| WebSocket reset/step interface. The adapter is currently designed for | |
| evaluation only. Canonical coverage uses the pinned public benchmark release; | |
| user-supplied ThinkingBox scenarios can also be evaluated when their executable | |
| data and external services are provided, but those runs are not canonical | |
| ThinkingBox-Bench results. | |
| ## Quick Start | |
| Start the externally managed ThinkingBox services described under | |
| [Deployment Boundary](#deployment-boundary), then run the OpenEnv server: | |
| ```bash | |
| OPENENV_TB_CONFIG=/path/to/thinkingbox.yaml \ | |
| uv run --project envs/thinkingbox_env --frozen server | |
| ``` | |
| Connect from a trusted harness: | |
| ```python | |
| from thinkingbox_env import ThinkingBoxEnv | |
| async with ThinkingBoxEnv(base_url="http://127.0.0.1:8000") as env: | |
| result = await env.reset("file.py:test_name") | |
| tools = await env.list_tools() | |
| result = await env.call_tool("tool_name", {"argument": "value"}) | |
| result = await env.submit_message("Final response") | |
| ``` | |
| ## Examples | |
| - [Public client example](https://github.com/huggingface/OpenEnv/blob/main/examples/thinkingbox/example_usage.py): | |
| reset a server-visible task and inspect its public task and tool observations | |
| without loading benchmark implementation data on the client. | |
| - [Evaluation wrapper](https://github.com/huggingface/OpenEnv/blob/main/examples/thinkingbox/eval_testlist.py): | |
| invoke the packaged `thinkingbox-eval` CLI for native model orchestration, | |
| repetition IDs, privacy-reviewed canonical JSONL, and strict aggregation. | |
| Run either script from the repository root: | |
| ```bash | |
| uv run --project envs/thinkingbox_env \ | |
| python examples/thinkingbox/example_usage.py file.py:test_name | |
| uv run --project envs/thinkingbox_env \ | |
| python examples/thinkingbox/eval_testlist.py \ | |
| --config /path/to/thinkingbox.yaml \ | |
| --output results.jsonl | |
| ``` | |
| The evaluator's `--config` and `--dataset` paths are local to the trusted | |
| evaluator. For a remote OpenEnv server, pass the corresponding server-visible | |
| paths with `--env-config` and `--env-dataset`. Strict `canary` and `full` | |
| coverage profiles accept only the pinned public benchmark; custom scenario | |
| bundles use ordinary non-profiled evaluation. | |
| ## Environment Details | |
| The adapter preserves native ThinkingBox hydration, history replay, tool | |
| overrides, parallel tool batches, direct responses, simulated users, effects, | |
| fixtures, `TestContext`, `TestScript`, `Judge`, error latching, and teardown. | |
| Final rewards are binary pass/fail; infrastructure failures are reported | |
| separately and never converted into benchmark failures. | |
| The model-facing action surface contains tool discovery, tool calls, and | |
| assistant messages. A trusted harness may call `finish()`, but finish is not | |
| advertised in the public action schema. Private user context, initialization | |
| state, assertion source, credentials, proxy session identifiers, effects, and | |
| grading internals remain inside the environment. | |
| `seed` is accepted for OpenEnv API compatibility but intentionally unused | |
| because tasks are selected deterministically by UID. Concurrent sessions are | |
| stacked independent environment instances, one trajectory per instance; they | |
| are not multiplexed trajectories. | |
| Executable assets come from the pinned `thinkingbox-bench-v1.0` release in | |
| [`microsoft/thinkingbox-data`](https://github.com/microsoft/thinkingbox-data). | |
| The [Hugging Face `microsoft/ThinkingBox-Bench` | |
| dataset](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench) is a | |
| viewer-friendly representation, while GitHub remains the executable source. | |
| ## Configuration | |
| The server accepts an ordinary native ThinkingBox YAML configuration through | |
| `OPENENV_TB_CONFIG` or the reset `config=` argument. The file supplies | |
| `mcp_proxy`, `orchestrator.agent_model`, `user_model`, `judge_model`, | |
| `judge_type`, and user-completion behavior; see the | |
| [ThinkingBox LLM configuration | |
| guide](https://github.com/microsoft/thinkingbox/blob/main/docs/llm_endpoint_config.md). | |
| | Variable | Default | Purpose | | |
| |---|---|---| | |
| | `OPENENV_TB_PROXY_URL` | `http://127.0.0.1:7111` | Session Proxy fallback | | |
| | `OPENENV_TB_DATASET` | pinned release | Local executable-data override | | |
| | `OPENENV_TB_DATA_CACHE` | `~/.cache/openenv/thinkingbox_bench` | Verified release cache | | |
| | `OPENENV_TB_AGENT` | `think` | Native agent definition | | |
| | `OPENENV_TB_CONFIG` | unset | Native server-side model/proxy config | | |
| | `OPENENV_TB_PROXY_TIMEOUT` | `120` | Proxy timeout without config override | | |
| | `OPENENV_TB_DATA_TIMEOUT` | `120` | Release download timeout | | |
| | `OPENENV_TB_MAX_CONCURRENT_ENVS` | `8` | Independent environment instances | | |
| The client message timeout remains longer than server-side model and episode | |
| operations so a valid long-running turn is not abandoned prematurely. | |
| ## Deployment Boundary | |
| The image starts only the OpenEnv API. It currently needs an externally | |
| managed ThinkingBox Session Proxy, benchmark MCP servers, Typesense, and agent, | |
| user-simulator, and judge model endpoints. `GET /health` reports process | |
| liveness; `GET /ready` reports the observable external dependencies and marks | |
| scenario-specific Typesense readiness as unobservable from this process. | |
| ## Citation | |
| Please cite the paper if ThinkingBox is useful in your experiments. | |
| - Paper: [One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark | |
| for Agents in Stateful Business Workflows](https://arxiv.org/abs/2608.19741) | |
| - Framework: [`microsoft/thinkingbox`](https://github.com/microsoft/thinkingbox) | |
| - Executable benchmark: | |
| [`microsoft/thinkingbox-data`](https://github.com/microsoft/thinkingbox-data) | |
| - Viewer dataset: | |
| [`microsoft/ThinkingBox-Bench`](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench) | |
| ```bibtex | |
| @misc{li2026successisntreliabilitythinkingbox, | |
| title={One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows}, | |
| author={Zhuochun Li and Youngmin Ko and Ali Keramati and Nicola Ferri and Susana Palmaz Lopez Pelaez and Liang-Chun Tsai and Calvin Wang and Mirco Milletari and Tuhin Kundu and Vadim Smolyakov and Kjartan Olafsson and Tommy Guy}, | |
| year={2026}, | |
| eprint={2608.19741}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CL}, | |
| url={https://arxiv.org/abs/2608.19741}, | |
| } | |
| ``` | |
Xet Storage Details
- Size:
- 6.76 kB
- Xet hash:
- 0640c61e9ee5ea71aa0490e6ef70fcd2667f26b78755ca408a8b0a7f847ceec2
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.