README.md

metadata

title: ToolForge Environment Server
colorFrom: gray
colorTo: blue
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv
  - agents
  - devops
  - tool-use
  - reinforcement-learning

ToolForge Environment

An OpenEnv environment designed to test and train LLM agents in pattern recognition and macro-tool creation.

In ToolForge, agents are tasked with completing sequential DevOps workflows (e.g., deployments, rollbacks, scaling). As agents encounter repetitive sequences of atomic tool calls, they are incentivized to recognize these patterns and compose them into reusable macro tools. By using macros, agents save on token consumption (used for thinking, planning, and executing), instantly accessing complex workflows and optimizing their efficiency across the episode.

What Makes This Different

ToolForge goes beyond simple tool execution benchmarks. It evaluates an agent's ability to abstract and optimize its own capabilities over time:

• Macro Creation: Agents can submit a propose_plan_with_macro action to permanently add a new, cheaper master tool to their environment state.
• Dual-Objective Scoring: Plans are evaluated on both semantic accuracy (did it solve the task?) and token efficiency (did it use macros to save tokens?).
• Progressive Difficulty: Tasks range from highly repetitive deployment sprints (easy) to complex, multi-phase infrastructure migrations (hard).

Quick Start

from toolforge_env.toolforge_env_environment import ToolforgeEnvironment
from toolforge_env.models import ToolforgeAction, ToolCall, Tool

env = ToolforgeEnvironment()
result = env.reset(mode="eval", difficulty="easy")

print(f"Task: {result.current_task.prompt}")

action = ToolforgeAction(
    action_type="propose_plan",
    plan=[
        ToolCall(tool_name="deploy"),
        ToolCall(tool_name="healthcheck"),
        ToolCall(tool_name="notify")
    ]
)
result = env.step(action)
print(f"Reward: {result.reward}")

macro_action = ToolforgeAction(
    action_type="propose_plan_with_macro",
    plan=[
        ToolCall(tool_name="deploy"),
        ToolCall(tool_name="healthcheck"),
        ToolCall(tool_name="notify")
    ],
    macro_proposal=Tool(
        name="deploy_verify_notify",
        description="Standard deployment pipeline",
        is_macro=True,
        token_cost=1, 
        steps=[
            ToolCall(tool_name="deploy"),
            ToolCall(tool_name="healthcheck"),
            ToolCall(tool_name="notify")
        ]
    )
)
result = env.step(macro_action)

Server Setup & Deployment

Local Docker (Recommended)

docker build -t toolforge-env:latest -f server/Dockerfile .
docker run --rm -p 8000:8000 toolforge-env:latest
curl http://localhost:8000/health

Deploy to Hugging Face Spaces

openenv push --repo-id your-org/toolforge-env

Core API Surface

Actions

ToolforgeAction

• action_type: "propose_plan" or "propose_plan_with_macro"
• plan: List of ToolCall
• macro_proposal: Optional Tool

Observations

ToolforgeObservation

• current_task
• available_tools
• reward
• done

The Toolbox

1. deploy
2. healthcheck
3. notify
4. rollback
5. scale
6. restart

Evaluation & Rewards

Reward Bounds

• Step rewards: -0.2 to 1.0
• Final score: 0.0 to 1.0

5-Stage Evaluation Pipeline

1. Sanity Validation
2. Semantic Slot Judgment
3. Plan Accuracy Score
4. Token Efficiency
5. Reward Calculation

Task Curriculums

• Easy: Repetitive workflows
• Medium: Incident response
• Hard: Complex infrastructure workflows