type: code

prompt: Write a Python helper that loads the latest evaluation summary JSON, compares it with the previous run, and returns a structured regression report with pass-rate delta and impacted suites.

task: eval-regression-report

generated_by: Maris AI

project_area: core-python

audience: ml-ops

suite: regression

source: maris-eval-benchmark

suite: regression

difficulty: medium

evaluation_mode: review-and-static-check

risk_level: medium

expected_behavior: Return a structured regression summary instead of free-form text.

expected_behavior: Compare current and previous pass-rate and identify impacted suites.

scoring_hints: Reward typed or clearly structured output fields.

scoring_hints: Fail if no delta or impacted suite reporting is present.

reference_answer: Implement a Python helper that accepts current and previous evaluation summaries, computes pass-rate delta, lists impacted suites, and returns a structured regression report dictionary.

acceptance_criteria: Returns structured output suitable for automation.

branch: coder


type: code

prompt: Produce a focused patch plan for rejecting invalid eval benchmark records that are missing task IDs, scoring hints, or reference answers in conversation/code splits.

task: eval-schema-guard

audience: maintainer

suite: sanity

source: maris-eval-benchmark

suite: sanity

difficulty: easy

evaluation_mode: review

risk_level: low

expected_behavior: Propose validation-focused changes only.

expected_behavior: Cover task IDs, scoring hints, and reference answers explicitly.

scoring_hints:
Prefer minimal, test-backed changes.
Avoid unrelated refactors.

reference_answer: Describe a minimal validator update that adds required eval fields and tests for missing task IDs, scoring hints, and conversation/code reference answers.

acceptance_criteria:
Mentions validator changes.
Mentions tests or regression coverage.

branch: coder