Spaces:
Running
Running
Update organization card with Paper2Arm-5135 and E2EBench-118
Browse files
README.md
CHANGED
|
@@ -11,16 +11,18 @@ pinned: false
|
|
| 11 |
|
| 12 |
**Open benchmarks and evaluation resources for materials-science Agents.**
|
| 13 |
|
| 14 |
-
MatMaster-DP is maintained by [DP Technology](https://www.dp.tech/en). We develop benchmarks for evaluating how AI Agents reason about materials, construct atomistic models, prepare and analyze scientific calculations, and plan laboratory operations.
|
| 15 |
|
| 16 |
## Public benchmarks
|
| 17 |
|
| 18 |
-
| Benchmark |
|
| 19 |
|---|---|---|
|
| 20 |
-
| [MATTER-604](https://huggingface.co/datasets/MatMaster-DP/MATTER-604) | 604 materials-science Agent tasks covering structure construction and retrieval, scientific analysis, simulation input generation, workflow execution, data diagnosis, and
|
| 21 |
| [Materials AutoLab 144](https://huggingface.co/datasets/MatMaster-DP/Materials-AutoLab-144) | 144 tasks for materials-experiment planning and laboratory operation through a public OPC interface contract | Task packages, action and capability catalogs, schemas, and evaluation guide |
|
|
|
|
|
|
|
| 22 |
|
| 23 |
-
|
| 24 |
|
| 25 |
## Evaluation access
|
| 26 |
|
|
@@ -28,6 +30,8 @@ Reference answers and evaluation materials are separated from the public questio
|
|
| 28 |
|
| 29 |
- [MATTER-604 Evaluator Access](https://huggingface.co/datasets/MatMaster-DP/MATTER-604-evaluator-access)
|
| 30 |
- [Materials AutoLab 144 Evaluator Access](https://huggingface.co/datasets/MatMaster-DP/Materials-AutoLab-144-evaluator-access)
|
|
|
|
|
|
|
| 31 |
|
| 32 |
Please disclose benchmark exposure, Agent and model versions, prompts, tools, runtime configuration, and any task-specific failures when reporting results.
|
| 33 |
|
|
|
|
| 11 |
|
| 12 |
**Open benchmarks and evaluation resources for materials-science Agents.**
|
| 13 |
|
| 14 |
+
MatMaster-DP is maintained by [DP Technology](https://www.dp.tech/en). We develop benchmarks for evaluating how AI Agents reason about materials, construct atomistic models, prepare and analyze scientific calculations, reproduce paper-grounded research workflows, and plan laboratory operations.
|
| 15 |
|
| 16 |
## Public benchmarks
|
| 17 |
|
| 18 |
+
| Benchmark | What is evaluated | Public package |
|
| 19 |
|---|---|---|
|
| 20 |
+
| [MATTER-604](https://huggingface.co/datasets/MatMaster-DP/MATTER-604) | 604 materials-science Agent tasks covering structure construction and retrieval, scientific analysis, simulation input generation, workflow execution, data diagnosis, and research constraints | Task prompts, redistributable input fixtures, submission schema, and evaluation guide |
|
| 21 |
| [Materials AutoLab 144](https://huggingface.co/datasets/MatMaster-DP/Materials-AutoLab-144) | 144 tasks for materials-experiment planning and laboratory operation through a public OPC interface contract | Task packages, action and capability catalogs, schemas, and evaluation guide |
|
| 22 |
+
| [Paper2Arm-5135](https://huggingface.co/datasets/MatMaster-DP/Paper2Arm-5135) | 5,135 paper-grounded task instances across 975 workflow families, evaluating whether an Agent can turn a reproduction objective into code, calculation inputs, analysis procedures, and structured research outputs | Instructions, runtime definitions, paper titles and DOI metadata, submission schema, and evaluation guide |
|
| 23 |
+
| [E2EBench-118](https://huggingface.co/datasets/MatMaster-DP/E2EBench-118) | 118 curated end-to-end computational materials-science tasks covering atomistic simulation, electronic structure, thermodynamics, kinetics, mechanics, magnetic and optical models, scientific data analysis, and machine-learning interatomic potentials | Instructions, runtime definitions, solver-visible resources, bibliographic metadata, and evaluation guide |
|
| 24 |
|
| 25 |
+
Each dataset card documents the scientific scope, package structure, and evaluation workflow. Machine-readable task records and public inputs are provided by the corresponding release.
|
| 26 |
|
| 27 |
## Evaluation access
|
| 28 |
|
|
|
|
| 30 |
|
| 31 |
- [MATTER-604 Evaluator Access](https://huggingface.co/datasets/MatMaster-DP/MATTER-604-evaluator-access)
|
| 32 |
- [Materials AutoLab 144 Evaluator Access](https://huggingface.co/datasets/MatMaster-DP/Materials-AutoLab-144-evaluator-access)
|
| 33 |
+
- [Paper2Arm-5135 Evaluator Access](https://huggingface.co/datasets/MatMaster-DP/Paper2Arm-5135-evaluator-access)
|
| 34 |
+
- [E2EBench-118 Evaluator Access](https://huggingface.co/datasets/MatMaster-DP/E2EBench-118-evaluator-access)
|
| 35 |
|
| 36 |
Please disclose benchmark exposure, Agent and model versions, prompts, tools, runtime configuration, and any task-specific failures when reporting results.
|
| 37 |
|