eval/datasets/README.md

MathSolver Benchmark Datasets

Standardized evaluation benchmarks for regression testing, metric tracking, and ablation studies.

Dataset Structure

Each JSON file in eval/datasets/ contains problem samples adhering to the BenchmarkSample schema:

{
  "id": "geo_01_square_pyramid",
  "category": "3d_pyramid",
  "problem_text": "Cho hình chóp S.ABCD...",
  "expected_type": "pyramid",
  "expected_entities": ["S", "A", "B", "C", "D"],
  "expected_dsl": "PYRAMID(S_ABCD)\nSQUARE(ABCD)...",
  "expected_answer": "32"
}

Running Evaluation

To evaluate deterministic DSL solvability & geometry validator pass rates:

from eval.benchmark import BenchmarkDataset
from eval.runner import EvalRunner

dataset = BenchmarkDataset.load_all_standard()
runner = EvalRunner()
metrics = runner.evaluate_dsl_deterministic(dataset)
print(metrics.to_dict())