experiments/bioagent_bench/README.md

# Biomni MCP Tool Scaling Experiment

This experiment studies how Biomni behaves when the MCP tool background grows from
about 100 tools to about 500, 1000, 1500, and 2000 tools on bioagent-bench.

## Generated Configs

The configs are generated from `/225040511/project/Biomni/mcp_generated` and can be
registered directly with Biomni:

```python
from biomni.agent import A1

agent = A1(use_tool_retriever=True)
agent.add_mcp(
    "/225040511/project/Biomni/experiments/bioagent_bench/configs/mcp_scale_100.yaml"
)
```

Generated files:

- `configs/mcp_scale_100.yaml`: 10 MCP servers, 107 tools
- `configs/mcp_scale_500.yaml`: 96 MCP servers, 504 tools
- `configs/mcp_scale_1000.yaml`: 276 MCP servers, 1003 tools
- `configs/mcp_scale_1500.yaml`: 378 MCP servers, 1527 tools
- `configs/mcp_scale_2000.yaml`: 526 MCP servers, 2000 tools
- `configs/mcp_scale_manifest.json`: exact server/tool counts and selected server order
- `configs/mcp_server_inventory.json`: parseable MCP inventory

The generator writes explicit `tools` metadata into YAML, so `A1.add_mcp()` does
not have to start every server just to discover schemas. The runtime command for
each tool still points to the corresponding generated MCP server file.

## Regenerate

```bash
cd /225040511/project/Biomni
python experiments/bioagent_bench/scripts/generate_mcp_scale_configs.py
```

The generator prioritizes benchmark-relevant tools first, then fills each tier by
whole MCP server units until the target tool count is reached.

## Benchmark Design

Use the tasks in `/225040511/project/bioagent-bench/src/task_metadata.json`.
For each task, run Biomni once per tool tier:

1. Create a fresh run directory named like:
   `runs/<tier>/<task_id>_<timestamp>/`
2. Initialize Biomni with `use_tool_retriever=True`.
3. Register exactly one tier config with `agent.add_mcp(config_path=...)`.
4. Run the bioagent-bench task prompt with the same data policy used in your
   existing runs.
5. Save `retrieval_plan.json`, `execution_log.json`, `execution_log.txt`,
   `run_metadata.json`, `run_summary.json`, `output_validation.json`, and final
   deliverables in the run directory.

Use identical model, decoding settings, timeout, conda environment, and benchmark
data policy across tiers. Repeat each task at least 3 times if you want confidence
intervals, because LLM tool retrieval and planning can vary.

## Metrics

Metric definitions used here:

- `Retrieval Recall@k`: fraction of task gold tools/servers present in the first
  `k` retrieved candidates from `retrieval_plan.json`. The table export uses a
  primary `k` value, defaulting to `10`.
- `Workflow Validity`: whether observed tool/log evidence covers the task's gold
  workflow stages in order.
- `Execution Success Rate`: whether benchmark output validation or required
  output paths indicate the pipeline completed.
- `Context Tokens`: rough planning-context token count from
  `retrieval_plan.json/planning_context_text`.
- `Planning Latency`: retrieval/planning latency recorded by the agent during
  tool selection.
- `Hallucinated Tool Rate`: fraction of executed MCP tools mentioned in the
  execution trace that were not present in the retrieved candidate tool set
  persisted in `retrieval_plan.json`.
- `Data-Type Mismatch Rate`: heuristic rate of execution attempts that emitted
  argument/schema/type mismatch errors, estimated from execution-log patterns.
- `Biological Constraint Error Rate`: heuristic rate of execution attempts that
  violated benchmark biological/data constraints, such as sibling-task data
  access, truth/results access, or external installs/downloads.

Gold tool and workflow definitions are in `gold_tools.json`. They are deliberately
pattern based, because MCP wrapper names differ across generated servers.

## Aggregate Existing Runs

```bash
python experiments/bioagent_bench/scripts/evaluate_mcp_scaling_runs.py \
  --runs-root /path/to/runs/scale_100 \
  --out experiments/bioagent_bench/results/scale_100_metrics.json
```

Run this once per tier, then compare the aggregate blocks or the exported table:

- `results/scale_<N>_mcp_metrics.json`
- `results/experiment1_scaling_table.json`
- `results/experiment1_scaling_table.csv`