benchmarks/README.md

ControlAI benchmark

v0.jsonl is the 12-item qualitative seed. v1_dev.jsonl is the 300-item, machine-reference development benchmark. Its six families are absent from both SFT train and validation.

Rules

• Never copy benchmark prompts, references, close paraphrases, or numerical variants into SFT, continued-pretraining, preference, or synthetic datasets.
• Split by family, not by individual question. A parameter change does not create an independent test example.
• Keep benchmark answers in English.
• Score the saved model response against every rubric item. Do not let the model see reference or rubric while generating its answer.
• MATLAB and Simulink code may be statically reviewed now. Mark execution status separately until a licensed MATLAB runner is connected.

Growth path

1. v0 seed: at least one checked item in every domain.
2. v0.1: 5 families per domain and a mix of concept, derivation, numerical, code, critique, design, and underspecified tasks.
3. v1 development: 300 family-separated items with machine-readable ground truth. The later release gate remains a private 1,000-item/200-family test.

Run the structural and numerical checks with:

python scripts/validate_benchmark.py benchmarks/v0.jsonl
python scripts/validate_benchmark_v1.py benchmarks/v1_dev.jsonl

Run a one-question MLX smoke test from an activated project environment:

python scripts/run_benchmark_mlx.py --limit 1

If that succeeds, resume and finish the remaining questions by omitting --limit. The runner appends after every answer and skips completed IDs:

python scripts/run_benchmark_mlx.py

For v1, always name the output explicitly:

python scripts/run_benchmark_mlx.py \
  --benchmark benchmarks/v1_dev.jsonl \
  --output benchmarks/responses/qwen3_4b_base_v1_dev.jsonl

The benchmark runner sends the system prompt, question, language, and word limit to the model. Rubrics and references remain hidden. Use a different --output path for each model/configuration so runs cannot be mixed. The original qwen3_4b_instruct_v0.jsonl run is retained as a prompt-wiring baseline; the corrected default output is qwen3_4b_instruct_v0_1.jsonl.

For a fast, deterministic regression signal on v1, run:

python scripts/score_benchmark_v1.py \
  benchmarks/responses/qwen3_4b_base_v1_dev.jsonl

This checks derived numbers, methods/APIs, conclusions, and word limits. It is explicitly provisional; final model selection still requires rubric review of the saved answers, especially Kharitonov and generated MATLAB code.