inputs/AgentSelect/README.md

---

AgentSelectBench 🌟

AgentSelectBench (AGENTSELECT) is a unified-supervision benchmark for narrative query-to-agent recommendation: given a free-form natural-language request, rank deployable agent configurations represented as capability profiles (backbone LLM, toolkit). It systematically converts heterogeneous evaluation artifacts (LLM leaderboards, tool-use benchmarks, etc.) into query-conditioned, positive-only interactions for training and evaluating agent recommenders at scale.

🔗 Getting Started / Documentation: https://v0-agent-recommendation-website.vercel.app/#getting-started
🧪 Online Demo (Agent Recommendation API, WIP): https://api.achieva-ai.com/OneAgent/

Status: this repository is under active refinement. We are progressively cleaning code, adding missing scripts/docs, and improving reproducibility. If, during review, you notice incomplete parts or rough edges, please treat them as ongoing engineering work—we are actively consolidating everything.

---

News

• We have updated the camera-ready paper: icml2026_AgentSelection_cr.pdf.

• We have further cleaned Part III dataset.

• Additional experimental details and data-processing workflows are provided in the Other/ folder.

Why AgentSelectBench ✨

Modern agent ecosystems offer an exploding space of configurations, but existing benchmarks evaluate components in isolation (models or tools). AgentSelectBench instead supports the end task:

• Input: a narrative query (no persistent user ID; intent is fully expressed in the query)
• Output: a ranked list of deployable agents as capability profiles (M, T)
• Supervision: positive-only query–agent interactions unified across sources

---

Capability Profile Format 🧾

Each agent is represented as a capability profile:

• Backbone LLM: M
• Toolkit: T (a set of tools with name + description)
• Stored as a YAML configuration to keep agents deployable (while we benchmark the stable capability core (M, T)). When deployed with agent framework, some additional configurations C may also required.

---

Benchmark Overview 📦

AgentSelectBench comprises three complementary dataset parts:

🧩 Part I — LLM-only Agents

Query-conditioned supervision derived from LLM evaluations/leaderboards (tools absent). Positives are typically constructed as top-k preferred backbones per query.

🧰 Part II — Toolkit-only Agents

Tool-use benchmarks provide the required/reference toolkit for each query; we treat each query’s toolkit as the positive target (backbone fixed to a placeholder).

🔗 Part III — Compositional Agents

We synthesize realistic (M, T) configurations by retrieving relevant components and composing them into candidate agents, yielding pseudo-positive interactions designed to reflect capability-consistent supervision.

Scale (current release): 111,179 queries, 107,721 agents, 251,103 interactions aggregated from 40+ sources.

---

Project Code Structure 🏗️

A typical structure (may evolve as we refactor):

AgentSelectBench/
├── agent_rec/                      # Research scaffold for agent recommendation
│   ├── data/                        # Dataset loaders / parsing
│   ├── features/                    # Unified feature interfaces (text + IDs)
│   ├── models/                      # Baselines (MF/LightFM/TwoTower/etc.)
│   ├── eval/                        # Metrics + evaluation harness
│   └── utils.py                     # Shared utilities (metrics printing, etc.)
├── scripts/                         # Helper scripts (training / eval wrappers)
├── run_bpr_mf_knn.py                # MF baseline with KNN query-vector surrogate
├── run_lightfm_handwritten.py       # LightFM baseline
├── run_generative.py                # Inference-only structured/generative baseline
├── run_generative_train.py          # Optional: seq2seq finetuning from exported pairs
├── requirements.txt
└── README.md

---

Getting Started 🚀

1) Clone the repository

git clone https://github.com/<your-org-or-anon-link>/AgentSelectBench.git
cd AgentSelectBench

2) Install dependencies

python -m venv .venv
source .venv/bin/activate  # Linux/Mac
# .venv\Scripts\activate   # Windows

pip install -r requirements.txt

3) Prepare dataset

AgentSelectBench is constructed from publicly available leaderboards / benchmarks. Depending on upstream redistribution constraints, we provide derived annotations/statistics and scripts to reconstruct raw sources when required.

---

Evaluation Protocol 📊

• Positives: Part I (top-10), Part II (top-1), Part III (top-5)
• Ranking cutoff: fixed Top-10 evaluation
• Reporting: metrics are reported for Part I / Part II / Part III / Overall

---

Quick Runs 🛠️

Run (BPR-MF + KNN q-vector)

python run_bpr_mf_knn.py \
  --data_root /path/to/dataset_root \
  --device cuda:0 \
  --epochs 5 --batch_size 4096 --factors 128 --neg_per_pos 1 \
  --knn_N 3 --eval_cand_size 100 --score_mode dot

Run (LightFM)

python run_lightfm_handwritten.py \
  --data_root /path/to/dataset_root \
  --device cuda:0 \
  --epochs 5 --batch_size 4096 --factors 128 --neg_per_pos 1 \
  --alpha_id 1.0 --alpha_feat 1.0 --max_features 5000 \
  --knn_N 3 --eval_cand_size 100 \
  --use_tool_id_emb 1

Note: this scaffold assumes you have utils.py in the same folder as the entry scripts, providing print_metrics_table(...) (consistent with the current research scaffold).

---

Generative / Structured Baseline (Inference-Only) ✍️

This entrypoint is inference-only (no training). It fits a TF-IDF retriever each run, then formats structured token outputs for a query. You can also export supervised pairs for training an external seq2seq model.

Generate structured token outputs (LLM + tools) for a query

python run_generative.py \
  --data_root /path/to/dataset_root \
  --query "How do I write a web scraper?" \
  --top_k 3 \
  --with_metadata 1

Export supervised pairs for seq2seq finetuning (JSONL)

python run_generative.py \
  --data_root /path/to/dataset_root \
  --export_pairs /tmp/generative_pairs.jsonl \
  --max_examples 5000

Fine-tune a seq2seq model (e.g., T5) on exported pairs

python run_generative_train.py \
  --data_root /path/to/dataset_root \
  --output_dir /tmp/generative_t5_ckpt \
  --model_name t5-small \
  --epochs 3 \
  --batch_size 16

Shell helper (env-style)

DATA_ROOT=/path/to/dataset_root \
QUERY="How do I write a web scraper?" \
TOP_K=3 \
WITH_METADATA=1 \
./scripts/run_generative.sh

---

Reproducibility Notes 🧪

• We will keep adding: dataset build scripts, caching, and deterministic evaluation harnesses.
• If you are reviewing this work and find a missing script or unclear step, it likely reflects ongoing repository cleanup rather than missing methodology; please feel free to flag it—we are actively addressing gaps.

---

Citation 📚

@inproceedings{ anonymous2026agentselect, title={AgentSelect: Benchmark for Narrative Query-to-Agent Recommendation}, author={Yunxiao Shi, Wujiang Xu, Tingwei Chen, Haoning Shang, Ling Yang, Yunfeng Wan, Zhuo Cao, Xing Zi, Dimitris N. Metaxas, Min Xu}, booktitle={Forty-third International Conference on Machine Learning}, year={2026}, url={https://openreview.net/forum?id=4M5Kj2UqaM} }