BENCHMARK_SYSTEM_CARD.md

RESMP Benchmark Instrument

Release: Rev 01
Type: Zero-dependency Web Component for grouped evaluation charts
Status: Experimental, usable, and versioned with the RESMP-DEV/README Space

What it does

The RESMP Benchmark Instrument renders a matrix of model or system evaluations as grouped small-multiple bar charts. It is designed for reports where readers need to compare the same ordered series across many tasks without relearning colors, scale, or layout in each panel.

It accepts local JSON, performs no runtime data fetch, and produces:

• a responsive browser view;
• a print-clean report;
• direct numeric labels on every available observation;
• explicit N/A marks for missing observations;
• a browser SVG export of the full composition.

Intended uses

• model, quantization, runtime, and hardware comparisons;
• grouped benchmark families such as STEM, instruction following, tool use, and agentic tasks;
• matched-control reports where fixed series identity matters;
• static Hugging Face, GitHub Pages, or local evaluation reports.

Non-goals

This component does not calculate metrics, retrieve benchmark results, establish statistical significance, or decide whether comparisons are methodologically valid. It visualizes values supplied by the publisher. A polished chart is not evidence that the underlying experiment was controlled.

Input contract

Inputs must conform to benchmark.schema.json. The core structure is:

{
  "title": "MODEL SYSTEM EVALUATIONS",
  "subtitle": "PINNED RUN / MATCHED CONTROL",
  "higherIsBetter": true,
  "series": [
    {"id": "candidate", "label": "CANDIDATE", "color": "#6f36d9"}
  ],
  "groups": [
    {
      "name": "STEM",
      "panels": [
        {"name": "MATH EXACT", "values": {"candidate": 62.4}}
      ]
    }
  ]
}

Series IDs must be unique. Values must be finite numbers or null. Missing keys and null render as N/A; they are never silently converted to zero.

Rendering semantics

• Series ordering is global and follows the series array.
• The numeric scale is shared across the complete chart, not recomputed per panel.
• Zero is always included in the domain.
• Negative and positive values share a visible zero baseline.
• higherIsBetter controls the displayed interpretation label only; it does not transform values.
• Values are displayed to two decimal places in Rev 01.

Reproducibility guidance

The chart JSON should be stored beside the evaluation receipt. For publishable results, record:

• model and dataset revisions;
• runtime, hardware, precision, and kernel versions;
• prompts, sampling settings, and evaluator versions;
• seeds or nondeterminism boundaries;
• raw result artifact hashes;
• aggregation code and any excluded samples.

The demonstration values in benchmark.example.json are synthetic layout data, not claimed benchmark results.

Accessibility

Every panel is emitted as an SVG with an accessible benchmark name. Direct values reduce dependence on color alone, and series order remains fixed. Rev 01 does not yet emit a full hidden data table, keyboard chart navigation, configurable patterns, or automatic contrast validation. Those are known gaps.

Export behavior

The Export SVG action creates a self-contained SVG containing the browser-rendered HTML/CSS inside <foreignObject>. It is suitable for browsers and Chromium-based capture workflows. Some vector editors and non-browser SVG pipelines do not fully support <foreignObject>; use browser print-to-PDF or a screenshot pipeline for those consumers.

Limitations

• One global scale can compress small differences when metric ranges vary widely.
• Rev 01 has no confidence intervals, error bars, sample counts, or significance annotations.
• Long series or panel labels may require custom CSS.
• Color selection is publisher-controlled and not automatically checked for accessibility.
• The component trusts already-sanitized text after escaping display fields; do not inject arbitrary HTML through config values.
• It does not validate experimental methodology.

Files

• benchmark-chart.js — Web Component and export logic
• benchmark-component.css — portable chart styling
• benchmark.schema.json — machine-readable input schema
• benchmark.example.json — canonical example input
• benchmark.html — full reference presentation

License and reuse

The component is published for reuse with RESMP.DEV releases. Preserve result provenance and do not use its visual consistency to imply experimental comparability where none exists.