docs: custom MCP guide + minimal FastAPI stub

- Delete obsolete example_server.py (old HTTP-endpoint
submission model, superseded by contamination-safe flow).
- Add example_mcp_server.py: ~150-line FastAPI stub showing
the MCP contract (POST {name, arguments} -> JSON),
bearer-token auth, 17 tool name list, and placeholder
handlers for fork-and-replace.
- README.md: new 'Bringing your own MCP tools' section with
the contract, hosting option matrix (Modal / cloud VM /
Runpod / k8s / ngrok+local GPU), link to the reference
implementation at RomeroLab/protein-design-mcp.
- app.py About tab: rewrite 'How to submit' for the new
we-host-the-agent architecture; link to custom-MCP docs
and example stub.

README.md

@@ -39,6 +39,72 @@ Novelty, and Diversity. See the *About* tab for the full methodology and the
 - **Depth Gap** — Forced-depth and low-diversity intervention results
 - **About** — Methodology, submission guide, and citation info
 
+## Bringing your own MCP tools
+
+BioDesignBench is "bring your own LLM, optionally bring your own tools."
+The **Custom MCP** submission mode lets you evaluate any 17-tool
+implementation against the identical 76 tasks, the identical agent
+harness, and the identical scoring rubric used by the paper's reference
+runs. Our [`protein-design-mcp`](https://github.com/RomeroLab/protein-design-mcp)
+is just one reference implementation.
+
+### The contract
+
+Your MCP server is a public HTTPS endpoint that accepts POST requests:
+
+```
+POST https://your-mcp.example.com/
+Authorization: Bearer <optional shared token>
+Content-Type: application/json
+
+{
+  "name": "predict_structure",
+  "arguments": {"sequence": "MKKL..."}
+}
+```
+
+The response must be a JSON object. Report errors in a top-level
+`error` field rather than via HTTP status codes so the agent loop can
+see the reason.
+
+### Tool schemas
+
+The 17 reference tool names plus full JSON Schema for each argument
+set live in [`mcp_tool_schemas.json`](./mcp_tool_schemas.json). Your
+implementation must accept the same tool names and argument shapes —
+the leaderboard's agent loop picks tools by name from this list.
+
+### Reference implementation
+
+See [`RomeroLab/protein-design-mcp`](https://github.com/RomeroLab/protein-design-mcp)
+for the lab's reference implementation. It is published as a Docker
+image and a Modal app; you can fork it, replace individual tool
+handlers, and redeploy.
+
+### Hosting options
+
+Any public HTTPS endpoint works — we only check that the contract is
+satisfied:
+
+| Option | Pros | Cons |
+|---|---|---|
+| **Modal** (serverless GPU) | Cheap pay-per-use, auto-scale | Modal account needed |
+| **AWS / GCP / Azure VM** | Full control, reuse existing cloud | 24/7 billing or manual shutdown |
+| **Runpod / Lambda Labs / Vast** | Cheap GPU rentals | Manual spin-up per submission |
+| **Kubernetes / HPC** | Reuse on-prem GPU | Ops overhead |
+| **ngrok + local GPU** | $0, fastest iteration | URL is ephemeral |
+
+The lab's reference MCP is hosted on Modal for serverless
+pay-per-use cost control; your submission does not have to match.
+
+### Minimal stub
+
+A ~150-line FastAPI template you can fork is at
+[`example_mcp_server.py`](./example_mcp_server.py). Replace the
+`handle_*` stubs with your implementations, deploy, and paste the URL
+into the submission form's **Advanced: Custom MCP** section.
+
+
 ## Backend pipeline phases
 
 Submission processing runs in 4 admin-controlled phases:

app.py

@@ -816,70 +816,67 @@ def build_about() -> str:
 
       <div {card}>
         <h2 {h2}>How to submit</h2>
-        <h3 {h3}>1. Build your agent</h3>
         <p {p}>
-          Create a protein design agent that runs the full plan &rarr;
-          sample &rarr; evaluate &rarr; iterate loop on each task. Pick one
-          of two MCP options:</p>
-        <ul style="color:#475569;padding-left:1.5rem;margin-bottom:0.8rem;
+          Unlike most agent benchmarks, <strong>you do not host an HTTP
+          endpoint</strong>. The 76 task descriptions never leave Romero
+          Lab infrastructure. Instead you provide:</p>
+        <ol style="color:#475569;padding-left:1.5rem;margin-bottom:0.8rem;
                    line-height:1.7">
-          <li><strong>Reference MCP</strong> &mdash; connect to our published
+          <li>an <strong>LLM provider + API key</strong>
+            (Anthropic / OpenAI / Google / DeepSeek).
+            We run the BioDesignBench agent loop against your chosen
+            model inside the leaderboard backend. Your key is
+            <em>scrubbed</em> from our records immediately after the
+            dispatch phase completes.</li>
+          <li>optionally, a <strong>custom MCP URL</strong> if you want
+            to evaluate your own tool implementations. Otherwise, the
+            agent calls our reference
             <a href="https://github.com/RomeroLab/protein-design-mcp"
                style="color:#2563eb;font-weight:600">protein-design-mcp</a>
-            server (Docker image / Modal endpoint, in progress). Eligible for
-            the reference ranking.</li>
-          <li><strong>Custom MCP</strong> &mdash; bring your own tool
-            implementations. Tagged with a <code>custom</code> badge on the
-            leaderboard, excluded from the reference ranking.</li>
-        </ul>
-        <h3 {h3}>2. Host an API endpoint</h3>
-        <p {p}>
-          Your agent must be accessible as a POST endpoint that accepts
-          task payloads and returns designed sequences plus a tool-call
-          trace. See <code>biodesignbench-leaderboard/example_server.py</code>
-          for a 200-line reference.</p>
-        <h3 {h3}>API specification</h3>
-        <pre style="background:#0f172a;color:#e2e8f0;padding:1.2rem;
-                    border-radius:10px;font-size:0.8rem;overflow-x:auto;
-                    line-height:1.6">POST /api/run
-
-Request:
-{{
-  "task_id": "dnb_ab_001",
-  "task_description": "Design a de novo binder for...",
-  "available_tools": [...],
-  "input_files": {{ "<pdb-name>": "<base64>" }},
-  "design_constraints": {{ ... }},
-  "max_steps": 50,
-  "timeout_sec": 300
-}}
-
-Response:
-{{
-  "sequences": ["MKKL..."],
-  "run_log": [{{ "step": 1, "tool": "...", "success": true }}],
-  "total_steps": 12,
-  "total_time_sec": 142.5,
-  "metrics": {{}}
-}}</pre>
-        <h3 {h3}>3. Submit and wait</h3>
+            endpoint (in progress).</li>
+        </ol>
+
+        <h3 {h3}>Data flow</h3>
         <p {p}>
-          We dispatch 73 hidden tasks to your endpoint, run Boltz-2
-          structure verification on each design, and score against the
-          100-point hybrid rubric (algorithmic + 3-judge LLM panel).
-          Maximum <strong>1 submission per month</strong> per
-          organization &mdash; LLM-judge API costs are paid by Romero
-          Lab.</p>
+          Each task prompt is sent to your chosen LLM provider via
+          their standard API (Anthropic, OpenAI, Google, DeepSeek) &mdash;
+          that single channel is the only path by which task data leaves
+          Romero Lab. The MCP server (reference or custom) only ever
+          sees operational tool arguments (sequences, PDB paths, hotspot
+          residues); it never sees the raw task prompt or evaluation
+          criteria. Every task prompt also carries a unique 16-character
+          canary token as an HTML comment, for retrospective leakage
+          detection.</p>
+
+        <h3 {h3}>Bring your own tools (Custom MCP)</h3>
         <p {p}>
-          3 example tasks are publicly available for development and
-          testing your endpoint before submission.</p>
+          If you want to benchmark a new tool implementation (a faster
+          structure predictor, a different diffusion backbone, your own
+          stability model) against the same 76 tasks and rubric, stand
+          up an HTTPS endpoint that satisfies the MCP contract and paste
+          the URL into the submission form's
+          <em>Advanced: Custom MCP</em> section:</p>
+        <ul style="color:#475569;padding-left:1.5rem;margin-bottom:0.8rem;
+                   line-height:1.7">
+          <li><strong>Contract + hosting options</strong>:
+            <a href="https://github.com/RomeroLab/BioDesignBench/blob/main/biodesignbench-leaderboard/README.md#bringing-your-own-mcp-tools"
+               style="color:#2563eb;font-weight:600">leaderboard README</a></li>
+          <li><strong>Minimal FastAPI stub (~150 lines)</strong>:
+            <a href="https://github.com/RomeroLab/BioDesignBench/blob/main/biodesignbench-leaderboard/example_mcp_server.py"
+               style="color:#2563eb;font-weight:600"><code>example_mcp_server.py</code></a></li>
+          <li><strong>Reference implementation to fork</strong>:
+            <a href="https://github.com/RomeroLab/protein-design-mcp"
+               style="color:#2563eb;font-weight:600">RomeroLab/protein-design-mcp</a></li>
+        </ul>
 
         <h3 {h3}>Limits</h3>
         <ul style="color:#475569;padding-left:1.5rem;margin-bottom:0.8rem;
                    line-height:1.7">
           <li>Maximum 1 submission per calendar month per organization</li>
-          <li>73 hidden tasks are used for ranking</li>
-          <li>3 public example tasks are available for development</li>
+          <li>73 hidden tasks are used for ranking; 3 public example
+            tasks are available for development</li>
+          <li>LLM-judge API costs are paid by Romero Lab; your own
+            agent LLM calls are billed to your provider</li>
         </ul>
       </div>
 

example_mcp_server.py

@@ -0,0 +1,147 @@
+"""Minimal reference implementation of the BioDesignBench MCP contract.
+
+Submitters who want to evaluate their own tool implementations against
+the BioDesignBench leaderboard can fork this file, plug in their own
+logic for each tool, deploy it behind a public HTTPS URL, and paste
+that URL into the submission form under "Advanced: Custom MCP".
+
+The leaderboard's agent loop will POST each tool invocation to your
+endpoint and treat the JSON response as the tool result. The MCP never
+sees the raw task description or evaluation criteria — only the
+operational arguments the agent chooses to pass (a protein sequence,
+a PDB path, a set of hotspot residues, etc.).
+
+Contract:
+
+    POST <your-url>/
+    Authorization: Bearer <optional shared token>
+    Content-Type: application/json
+
+    {
+      "name": "<tool_name>",          # one of the 17 tool names
+      "arguments": { ... }            # per-tool JSON object
+    }
+
+    Response: arbitrary JSON object describing the tool output.
+    Errors should be reported in a top-level "error" field rather
+    than via HTTP status codes.
+
+Install:
+    pip install fastapi uvicorn
+
+Run (locally, with ngrok for a public URL):
+    uvicorn example_mcp_server:app --host 0.0.0.0 --port 8000
+    ngrok http 8000
+
+Deploy (Modal example):
+    modal deploy example_mcp_server.py  # see bdb-boltz as a template
+
+Tool name list (17 total) — the full JSON Schema for each lives in
+`mcp_tool_schemas.json` in this repo:
+
+    design_binder, analyze_interface, validate_design, optimize_sequence,
+    suggest_hotspots, get_design_status, predict_complex, predict_structure,
+    score_stability, energy_minimize, generate_backbone, rosetta_score,
+    rosetta_relax, rosetta_interface_score, rosetta_design,
+    predict_structure_boltz, predict_affinity_boltz
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Any
+
+from fastapi import FastAPI, Header, HTTPException
+from pydantic import BaseModel
+
+app = FastAPI(title="BioDesignBench MCP (example stub)")
+
+SHARED_TOKEN = os.environ.get("BDB_MCP_TOKEN", "")
+
+
+class MCPRequest(BaseModel):
+    name: str
+    arguments: dict[str, Any] = {}
+
+
+# ---------------------------------------------------------------------------
+#  Tool handlers (REPLACE THESE STUBS WITH YOUR ACTUAL IMPLEMENTATIONS)
+# ---------------------------------------------------------------------------
+
+
+def handle_predict_structure(args: dict) -> dict:
+    """Predict the structure of a single protein sequence.
+
+    Real implementations would call AlphaFold2, ESMFold, Boltz, etc.
+    """
+    sequence = args.get("sequence") or ""
+    if not sequence:
+        return {"error": "predict_structure requires a 'sequence' argument"}
+    # TODO: replace this stub with your structure predictor.
+    return {
+        "pdb": ">dummy\nATOM ...",
+        "pLDDT": 0.0,
+        "pTM": 0.0,
+        "note": "stub implementation -- replace with your predictor",
+    }
+
+
+def handle_design_binder(args: dict) -> dict:
+    """Design a protein binder against a target. Real implementations
+    would call RFdiffusion followed by ProteinMPNN and AlphaFold2."""
+    return {"error": "design_binder not implemented in this stub"}
+
+
+def handle_score_stability(args: dict) -> dict:
+    """Score single-point stability. Real implementations might call
+    Rosetta, DDG_predictor, or a learned model."""
+    return {"error": "score_stability not implemented in this stub"}
+
+
+TOOL_HANDLERS = {
+    "predict_structure": handle_predict_structure,
+    "design_binder": handle_design_binder,
+    "score_stability": handle_score_stability,
+    # ... add handlers for the other 14 tools here
+}
+
+
+# ---------------------------------------------------------------------------
+#  Dispatcher
+# ---------------------------------------------------------------------------
+
+
+@app.post("/")
+def call_tool(
+    req: MCPRequest,
+    authorization: str | None = Header(default=None),
+) -> dict:
+    if SHARED_TOKEN:
+        bearer = (authorization or "").removeprefix("Bearer ").strip()
+        if bearer != SHARED_TOKEN:
+            raise HTTPException(status_code=401, detail="Unauthorized")
+
+    handler = TOOL_HANDLERS.get(req.name)
+    if handler is None:
+        return {
+            "error": f"Unknown tool: {req.name}",
+            "available": sorted(TOOL_HANDLERS.keys()),
+        }
+
+    try:
+        return handler(req.arguments)
+    except Exception as e:
+        return {"error": f"{type(e).__name__}: {e}", "tool": req.name}
+
+
+@app.get("/health")
+def health() -> dict:
+    return {
+        "ok": True,
+        "implemented_tools": sorted(TOOL_HANDLERS.keys()),
+        "note": (
+            "This is the reference stub. Replace the handle_* functions "
+            "with your actual tool implementations before submitting to "
+            "the BioDesignBench leaderboard."
+        ),
+    }

example_server.py

@@ -1,205 +0,0 @@
-"""Reference FastAPI server for BioDesignBench submitters.
-
-This example shows how to implement the API endpoint that BioDesignBench
-will call during benchmarking. Replace the mock agent logic with your
-actual LLM agent + MCP tool pipeline.
-
-Usage:
-    pip install fastapi uvicorn
-    python example_server.py
-
-    # Or with uvicorn directly:
-    uvicorn example_server:app --host 0.0.0.0 --port 8000
-
-Your endpoint will receive POST requests at /api/run with the task payload.
-
-Task Payload Format:
-    {
-        "task_id": "dnb_sig_001",
-        "task_description": "Design a de novo binder for...",
-        "available_tools": [... 17 tool schemas ...],
-        "input_files": {"7n1j.pdb": "<base64>"},
-        "design_constraints": {"length_range": [80, 150], "max_designs": 10},
-        "max_steps": 50,
-        "timeout_sec": 300
-    }
-
-Expected Response Format:
-    {
-        "sequences": ["MKKL...", "MFQR..."],
-        "run_log": [{"step": 1, "tool": "suggest_hotspots", "success": true}, ...],
-        "total_steps": 12,
-        "total_time_sec": 142.5,
-        "metrics": {}
-    }
-"""
-
-from __future__ import annotations
-
-import base64
-import random
-import time
-from pathlib import Path
-from typing import Any
-
-from fastapi import FastAPI
-from pydantic import BaseModel
-
-app = FastAPI(
-    title="BioDesignBench Example Agent",
-    description="Reference implementation for benchmark submission",
-    version="0.1.0",
-)
-
-
-# ---------------------------------------------------------------------------
-#  Request/Response models
-# ---------------------------------------------------------------------------
-
-
-class TaskPayload(BaseModel):
-    task_id: str
-    task_description: str
-    available_tools: list[dict[str, Any]] = []
-    input_files: dict[str, str] = {}  # filename -> base64 data
-    design_constraints: dict[str, Any] = {}
-    max_steps: int = 50
-    timeout_sec: int = 300
-
-
-class AgentResponse(BaseModel):
-    sequences: list[str]
-    run_log: list[dict[str, Any]]
-    total_steps: int
-    total_time_sec: float
-    metrics: dict[str, Any] = {}
-
-
-# ---------------------------------------------------------------------------
-#  Mock agent (replace with your real agent)
-# ---------------------------------------------------------------------------
-
-# Standard amino acids for mock sequence generation
-_AAS = "ACDEFGHIKLMNPQRSTVWY"
-
-
-def _generate_mock_sequence(length: int) -> str:
-    """Generate a random protein sequence with reasonable composition."""
-    # Weight towards common amino acids
-    weights = [
-        7, 2, 5, 6, 4, 7, 2, 5, 6, 9,  # A C D E F G H I K L
-        2, 4, 5, 4, 5, 7, 6, 7, 1, 3,   # M N P Q R S T V W Y
-    ]
-    return "".join(random.choices(_AAS, weights=weights, k=length))
-
-
-def mock_agent(payload: TaskPayload) -> AgentResponse:
-    """Mock agent that generates random but valid designs.
-
-    Replace this with your actual LLM agent + MCP tool pipeline.
-    This mock demonstrates the expected response format.
-    """
-    start = time.monotonic()
-
-    # Determine design parameters
-    constraints = payload.design_constraints
-    length_range = constraints.get("length_range", [80, 150])
-    max_designs = constraints.get("max_designs", 10)
-    num_designs = min(max_designs, 5)  # Generate 5 for this mock
-
-    # "Decode" input PDB files (in a real agent, you'd use these)
-    for filename, b64_data in payload.input_files.items():
-        pdb_bytes = base64.b64decode(b64_data)
-        # In a real agent: save to temp file and pass to MCP tools
-
-    # Simulate a multi-step design pipeline
-    run_log = [
-        {
-            "step": 1,
-            "tool": "suggest_hotspots",
-            "success": True,
-            "args_summary": {"target": "from_pdb"},
-            "output_summary": "Found 5 hotspot residues",
-        },
-        {
-            "step": 2,
-            "tool": "generate_backbone",
-            "success": True,
-            "args_summary": {"length": length_range[0]},
-            "output_summary": f"Generated {num_designs} backbones",
-        },
-        {
-            "step": 3,
-            "tool": "optimize_sequence",
-            "success": True,
-            "args_summary": {"optimization_target": "both"},
-            "output_summary": f"Optimized {num_designs} sequences",
-        },
-        {
-            "step": 4,
-            "tool": "predict_structure",
-            "success": True,
-            "args_summary": {"predictor": "esmfold"},
-            "output_summary": "Predicted structures for all designs",
-        },
-        {
-            "step": 5,
-            "tool": "validate_design",
-            "success": True,
-            "args_summary": {},
-            "output_summary": "Validated all designs",
-        },
-    ]
-
-    # Generate mock sequences
-    min_len, max_len = length_range
-    sequences = [
-        _generate_mock_sequence(random.randint(min_len, max_len))
-        for _ in range(num_designs)
-    ]
-
-    elapsed = time.monotonic() - start
-
-    return AgentResponse(
-        sequences=sequences,
-        run_log=run_log,
-        total_steps=len(run_log),
-        total_time_sec=round(elapsed, 2),
-        metrics={},  # Agent-reported metrics (optional)
-    )
-
-
-# ---------------------------------------------------------------------------
-#  API endpoint
-# ---------------------------------------------------------------------------
-
-
-@app.post("/api/run", response_model=AgentResponse)
-async def run_task(payload: TaskPayload) -> AgentResponse:
-    """Run a single benchmark task.
-
-    This is the endpoint that BioDesignBench will POST to during benchmarking.
-    Replace mock_agent() with your actual agent logic.
-    """
-    return mock_agent(payload)
-
-
-@app.get("/health")
-async def health():
-    """Health check endpoint."""
-    return {"status": "ok", "agent": "example-mock"}
-
-
-# ---------------------------------------------------------------------------
-#  Entry point
-# ---------------------------------------------------------------------------
-
-if __name__ == "__main__":
-    import uvicorn
-
-    print("Starting BioDesignBench example server...")
-    print("POST endpoint: http://localhost:8000/api/run")
-    print("Health check:  http://localhost:8000/health")
-    print()
-    print("Replace mock_agent() with your real agent logic.")
-    uvicorn.run(app, host="0.0.0.0", port=8000)