README.md

---
title: README
emoji: 🔬
colorFrom: green
colorTo: gray
sdk: static
app_file: index.html
fullWidth: true
header: mini
short_description: Benchmarks for scientific modeling and design agents.
pinned: false
---

<p align="center">
  <img
    src="https://huggingface.co/spaces/sci-modeling-bench/README/resolve/main/logo.png"
    alt="SciModelingBench logo"
    width="150"
  >
</p>

<h1 align="center">SciModelingBench</h1>

<p align="center">
  <strong>Observation-grounded benchmarks for scientific modeling and design agents.</strong>
</p>

<p align="center">
  <a href="https://github.com/xukp20/sci-modeling-bench">GitHub</a>
  &nbsp;&middot;&nbsp;
  <a href="https://pypi.org/project/sci-modeling-bench/">PyPI</a>
  &nbsp;&middot;&nbsp;
  <a href="https://github.com/xukp20/sci-modeling-bench/tree/main/docs">Documentation</a>
  &nbsp;&middot;&nbsp;
  <a href="https://huggingface.co/datasets/sci-modeling-bench/design-bench">Dataset Hub</a>
</p>

SciModelingBench separates versioned scientific observations, Agent-visible
inputs, trusted scientific evaluation, and Task metrics into explicit
`Dataset`, `Protocol`, `Objective`, and `Task` interfaces. It is designed for
studying how agents use domain representations, experimental context, raw
measurements, and iterative modeling rather than treating every scientific
problem as an unstructured prompt or a fixed learned surrogate.

## What We Publish

- **Python framework:** typed interfaces, validators, reproducible builders,
  Protocols, Objectives, Tasks, and common candidate-ranking metrics.
- **Canonical data:** revision-pinned scientific observations with manifests,
  source checksums, licenses, and raw-to-release provenance.
- **Benchmark settings:** exact, measured, posterior-derived, and
  simulator-grounded evaluation settings for scientific optimization and
  finite-pool ranking.

## Current Scientific Settings

| Area | Settings | Evaluation basis |
|---|---|---|
| DNA binding | TFBind8, TFBind10 Pho4 | Exact landscape and raw replicate-count posterior |
| Biological sequence design | UTR MRL, GFP | Measured sequence and protein-level outcomes |
| Materials | Superconductor | Measured composition-group critical temperature |
| Molecular toxicology | DrugMatrix | Individual-animal measurements and matched controls |
| Algorithm and controller design | CellDAG-NAS, Hopper Controller | Official NAS records and repeated simulator rollouts |

## Design Principles

1. **Observation grounded:** preserve raw repeats and experimental context when
   they materially affect the scientific mapping.
2. **Explicit trust boundary:** distinguish measured, exact, simulated,
   posterior-derived, and learned evaluators.
3. **Reproducible identity:** pin source artifacts, canonical data, manifests,
   provenance, package versions, and Hub revisions.
4. **Agent-facing separation:** Protocols define visible information; hidden
   evaluator state and interaction policy remain outside the Dataset.
5. **Continuous feedback:** candidate Tasks report interpretable score,
   enrichment, regret, and ranking metrics rather than pass/fail alone.

## Start Here

```bash
python -m pip install sci-modeling-bench
```

```python
from sci_modeling_bench.suites.design_bench import TFBind8BlackBoxOptimizationTask

task = TFBind8BlackBoxOptimizationTask.from_hub()
agent_input = task.build_input()
```

The project is under active development. Reproducible experiments should pin
both the Python package version and the immutable Hugging Face dataset
revision. Public artifacts do not by themselves prevent answer lookup; strict
agent evaluations still require an isolated external harness.