Note SciBench is a novel benchmark for college-level scientific problems sourced from instructional textbooks

Note 824 challenging multi-hop questions requiring information from 2-15 Wikipedia articles. If we adapt this to a search agent with no RAG component it's super interesting.

Note can be nicely adapted to an agentic setting with code

Note A simple and challenging benchmark that measures the ability of AI agents to locate hard-to-find information online

Note The ComplexFuncBench dataset encompass 1,000 complex function calling samples from five aspects: (1) Function calling with multi-step in single turn; (2) Function calling with user-provided constraints; (3) Function calling that requires parameter value reasoning from implicit information; (4) Function calling with long parameter values that exceed 500 tokens; and (5) Function calling with 128k long-context length.

Note Comprehensive evaluation across multiple domains and interaction types by leveraging diverse datasets: BFCL: Mathematics, Entertainment, Education, and Academic Domains τ-bench: Retail and Airline Industry Scenarios xLAM: Cross-domain Data Generation (21 Domains) ToolACE: API Interactions across 390 Domains

Note Training traces that helped a 32B model achieve gpt-4o level performance on SWE-bench. OpenHands framework