README.md

metadata

title: README
emoji: 🔬
colorFrom: green
colorTo: gray
sdk: static
app_file: index.html
fullWidth: true
header: mini
short_description: Benchmarks for scientific modeling and design agents.
pinned: false

SciModelingBench

Observation-grounded benchmarks for scientific modeling and design agents.

GitHub  ·  PyPI  ·  Documentation  ·  Dataset Hub

SciModelingBench separates versioned scientific observations, Agent-visible inputs, trusted scientific evaluation, and Task metrics into explicit Dataset, Protocol, Objective, and Task interfaces. It is designed for studying how agents use domain representations, experimental context, raw measurements, and iterative modeling rather than treating every scientific problem as an unstructured prompt or a fixed learned surrogate.

What We Publish

• Python framework: typed interfaces, validators, reproducible builders, Protocols, Objectives, Tasks, and common candidate-ranking metrics.
• Canonical data: revision-pinned scientific observations with manifests, source checksums, licenses, and raw-to-release provenance.
• Benchmark settings: exact, measured, posterior-derived, and simulator-grounded evaluation settings for scientific optimization and finite-pool ranking.

Current Scientific Settings

Area	Settings	Evaluation basis
DNA binding	TFBind8, TFBind10 Pho4	Exact landscape and raw replicate-count posterior
Biological sequence design	UTR MRL, GFP	Measured sequence and protein-level outcomes
Materials	Superconductor	Measured composition-group critical temperature
Molecular toxicology	DrugMatrix	Individual-animal measurements and matched controls
Algorithm and controller design	CellDAG-NAS, Hopper Controller	Official NAS records and repeated simulator rollouts

Design Principles

1. Observation grounded: preserve raw repeats and experimental context when they materially affect the scientific mapping.
2. Explicit trust boundary: distinguish measured, exact, simulated, posterior-derived, and learned evaluators.
3. Reproducible identity: pin source artifacts, canonical data, manifests, provenance, package versions, and Hub revisions.
4. Agent-facing separation: Protocols define visible information; hidden evaluator state and interaction policy remain outside the Dataset.
5. Continuous feedback: candidate Tasks report interpretable score, enrichment, regret, and ranking metrics rather than pass/fail alone.

Start Here

python -m pip install sci-modeling-bench

from sci_modeling_bench.suites.design_bench import TFBind8BlackBoxOptimizationTask

task = TFBind8BlackBoxOptimizationTask.from_hub()
agent_input = task.build_input()

The project is under active development. Reproducible experiments should pin both the Python package version and the immutable Hugging Face dataset revision. Public artifacts do not by themselves prevent answer lookup; strict agent evaluations still require an isolated external harness.