Biomni MCP Tool Scaling Experiment

This experiment studies how Biomni behaves when the MCP tool background grows from about 100 tools to about 500, 1000, 1500, and 2000 tools on bioagent-bench.

Generated Configs

The configs are generated from /225040511/project/Biomni/mcp_generated and can be registered directly with Biomni:

from biomni.agent import A1

agent = A1(use_tool_retriever=True)
agent.add_mcp(
    "/225040511/project/Biomni/experiments/bioagent_bench/configs/mcp_scale_100.yaml"
)

Generated files:

configs/mcp_scale_100.yaml: 10 MCP servers, 107 tools
configs/mcp_scale_500.yaml: 96 MCP servers, 504 tools
configs/mcp_scale_1000.yaml: 276 MCP servers, 1003 tools
configs/mcp_scale_1500.yaml: 378 MCP servers, 1527 tools
configs/mcp_scale_2000.yaml: 526 MCP servers, 2000 tools
configs/mcp_scale_manifest.json: exact server/tool counts and selected server order
configs/mcp_server_inventory.json: parseable MCP inventory

The generator writes explicit tools metadata into YAML, so A1.add_mcp() does not have to start every server just to discover schemas. The runtime command for each tool still points to the corresponding generated MCP server file.

Regenerate

cd /225040511/project/Biomni
python experiments/bioagent_bench/scripts/generate_mcp_scale_configs.py

The generator prioritizes benchmark-relevant tools first, then fills each tier by whole MCP server units until the target tool count is reached.

Benchmark Design

Use the tasks in /225040511/project/bioagent-bench/src/task_metadata.json. For each task, run Biomni once per tool tier:

Create a fresh run directory named like: runs/<tier>/<task_id>_<timestamp>/
Initialize Biomni with use_tool_retriever=True.
Register exactly one tier config with agent.add_mcp(config_path=...).
Run the bioagent-bench task prompt with the same data policy used in your existing runs.
Save retrieval_plan.json, execution_log.json, execution_log.txt, run_metadata.json, run_summary.json, output_validation.json, and final deliverables in the run directory.

Use identical model, decoding settings, timeout, conda environment, and benchmark data policy across tiers. Repeat each task at least 3 times if you want confidence intervals, because LLM tool retrieval and planning can vary.

Metrics

Metric definitions used here:

Retrieval Recall@k: fraction of task gold tools/servers present in the first k retrieved candidates from retrieval_plan.json. The table export uses a primary k value, defaulting to 10.
Workflow Validity: whether observed tool/log evidence covers the task's gold workflow stages in order.
Execution Success Rate: whether benchmark output validation or required output paths indicate the pipeline completed.
Context Tokens: rough planning-context token count from retrieval_plan.json/planning_context_text.
Planning Latency: retrieval/planning latency recorded by the agent during tool selection.
Hallucinated Tool Rate: fraction of executed MCP tools mentioned in the execution trace that were not present in the retrieved candidate tool set persisted in retrieval_plan.json.
Data-Type Mismatch Rate: heuristic rate of execution attempts that emitted argument/schema/type mismatch errors, estimated from execution-log patterns.
Biological Constraint Error Rate: heuristic rate of execution attempts that violated benchmark biological/data constraints, such as sibling-task data access, truth/results access, or external installs/downloads.

Gold tool and workflow definitions are in gold_tools.json. They are deliberately pattern based, because MCP wrapper names differ across generated servers.

Aggregate Existing Runs

python experiments/bioagent_bench/scripts/evaluate_mcp_scaling_runs.py \
  --runs-root /path/to/runs/scale_100 \
  --out experiments/bioagent_bench/results/scale_100_metrics.json

Run this once per tier, then compare the aggregate blocks or the exported table:

results/scale_<N>_mcp_metrics.json
results/experiment1_scaling_table.json
results/experiment1_scaling_table.csv