Observation-grounded benchmarks for scientific modeling and design agents.
GitHub · PyPI · Documentation · Dataset Hub
SciModelingBench separates versioned scientific observations, Agent-visible inputs, trusted scientific evaluation, and Task metrics into explicit `Dataset`, `Protocol`, `Objective`, and `Task` interfaces. It is designed for studying how agents use domain representations, experimental context, raw measurements, and iterative modeling rather than treating every scientific problem as an unstructured prompt or a fixed learned surrogate. ## What We Publish - **Python framework:** typed interfaces, validators, reproducible builders, Protocols, Objectives, Tasks, and common candidate-ranking metrics. - **Canonical data:** revision-pinned scientific observations with manifests, source checksums, licenses, and raw-to-release provenance. - **Benchmark settings:** exact, measured, posterior-derived, and simulator-grounded evaluation settings for scientific optimization and finite-pool ranking. ## Current Scientific Settings | Area | Settings | Evaluation basis | |---|---|---| | DNA binding | TFBind8, TFBind10 Pho4 | Exact landscape and raw replicate-count posterior | | Biological sequence design | UTR MRL, GFP | Measured sequence and protein-level outcomes | | Materials | Superconductor | Measured composition-group critical temperature | | Molecular toxicology | DrugMatrix | Individual-animal measurements and matched controls | | Algorithm and controller design | CellDAG-NAS, Hopper Controller | Official NAS records and repeated simulator rollouts | ## Design Principles 1. **Observation grounded:** preserve raw repeats and experimental context when they materially affect the scientific mapping. 2. **Explicit trust boundary:** distinguish measured, exact, simulated, posterior-derived, and learned evaluators. 3. **Reproducible identity:** pin source artifacts, canonical data, manifests, provenance, package versions, and Hub revisions. 4. **Agent-facing separation:** Protocols define visible information; hidden evaluator state and interaction policy remain outside the Dataset. 5. **Continuous feedback:** candidate Tasks report interpretable score, enrichment, regret, and ranking metrics rather than pass/fail alone. ## Start Here ```bash python -m pip install sci-modeling-bench ``` ```python from sci_modeling_bench.suites.design_bench import TFBind8BlackBoxOptimizationTask task = TFBind8BlackBoxOptimizationTask.from_hub() agent_input = task.build_input() ``` The project is under active development. Reproducible experiments should pin both the Python package version and the immutable Hugging Face dataset revision. Public artifacts do not by themselves prevent answer lookup; strict agent evaluations still require an isolated external harness.