reports/summary.md

Evaluation Summary (v7.21)

• Run ID: 20260629_184555
• Model ID: /mnt/d/Development/Libraries/Mistral-Nemo-Instruct-2407-AWQ_20260529_152601
• Quant types requested: []
• Artifacts recorded: 0

Battery Coverage

• Fixtures version: v7.21
• json_multistep_cases: 5 cases
• json_shelf_cases: 4 cases
• mixed_brief_json_cases: 2 cases
• stateful_followup_cases: 2 cases
• toolcall_only_cases: 2 cases
• Total: 15 cases

Pass-Rate Signals

Pass rates are computed per family as the fraction of rows where all listed signals equal 1. (If a signal column is missing for a row, it is ignored for that row.)

• json_multistep: pass if schema_ok, checks_consistent_ok, stop_semantics_ok, oracle_equiv_ok
• stateful_followup: pass if turn1_parse_ok, turn2_parse_ok, turn1_exact_match, turn2_exact_match
• toolcall_only: pass if tool_name_ok, args_ok
• mixed_brief_json: pass if answer_line_ok, json_parse_ok, schema_ok
• json: pass if schema_ok, constraints_ok
• fuzz: pass if schema_ok, constraints_ok
• toolcall: pass if stage1_tool_parse_ok, stage1_tool_schema_ok
• mcq: pass if choice_ok

Bucket Score Scale

Some families report an auxiliary bucket_score used for ranking/triage. The scale is family-specific:

• json_multistep: Range 0–10. Score 10 means plan_exact_match=1 AND checks_ok=1 AND final_match_reported=1. Otherwise the score is plan_prefix_match_len (how many leading steps match the oracle plan). Tier-1 uses oracle_equiv_ok (computed final matches oracle) as outcome correctness; final_match_reported/final_consistent_ok are tracked as Tier-2 reporting discipline.
• stateful_followup: Range 0–10. Higher is better; 10 indicates both turns parse and match expected state updates.
• toolcall_only: Range 0–10. Higher is better; 10 indicates correct tool name AND valid args under the tool schema.
• mixed_brief_json: Range 0–10. Higher is better; 10 indicates correct ANSWER line AND valid JSON block meeting schema.
• json: Range 0–10. Higher is better; 10 indicates exact/valid JSON shelf placement meeting constraints. (See CSV for exact signal columns.)
• fuzz: Range 0–10. Property-based regression. Score mirrors the JSON shelf scoring signals when present; pass rate uses schema_ok AND constraints_ok on rows where those signals exist.
• toolcall: Range 0–10. Higher is better; 10 indicates tool-call JSON parses and validates against schema.
• mcq: Range 0–1. 1 indicates the correct choice was selected.

Results by Runner and Family

Runner: quantized_vllm

Family	N	Pass rate	Averages
json_multistep	5	0.400	bucket_score=1.800
stateful_followup	2	1.000	bucket_score=2.000
toolcall_only	2	1.000	bucket_score=2.000
mixed_brief_json	2	1.000	bucket_score=2.000
json	4	n/a	bucket_score=10.000
fuzz	20	n/a	bucket_score=10.000
toolcall	2	1.000	bucket_score=11.000
mcq	5	n/a	bucket_score=0.600

Fuzz Regression Notes

fuzz is a property-based regression suite and is summarized as a family for stability tracking; it is not a direct agent-readiness proxy.

Signal Breakdown (Diagnostics)

For each family, the pass rate is a conjunction of signals. The tables below show per-signal success rates, which helps isolate which signal is blocking a family from passing.

Runner: quantized_vllm

Family: json_multistep

Signal	Rate	Notes
schema_ok	1.000	tier-1 (gating)
checks_consistent_ok	0.400	tier-1 (gating)
stop_semantics_ok	0.800	tier-1 (gating)
oracle_equiv_ok	0.600	tier-1 (gating)
final_consistent_ok	0.000	tier-2 (tracked; non-gating)
final_match_reported	0.000	tier-2 (tracked; non-gating)

Family: stateful_followup

Signal	Rate	Notes
turn1_parse_ok	1.000	tier-1 (gating)
turn2_parse_ok	1.000	tier-1 (gating)
turn1_exact_match	1.000	tier-1 (gating)
turn2_exact_match	1.000	tier-1 (gating)

Family: toolcall_only

Signal	Rate	Notes
tool_name_ok	1.000	tier-1 (gating)
args_ok	1.000	tier-1 (gating)

Family: mixed_brief_json

Signal	Rate	Notes
answer_line_ok	1.000	tier-1 (gating)
json_parse_ok	1.000	tier-1 (gating)
schema_ok	1.000	tier-1 (gating)

Readiness Snapshot

The snapshot below provides a coarse, decision-oriented view derived from pass rates. Thresholds are intentionally conservative: PASS ≥ 0.90, WARN ≥ 0.70, else FAIL.

Runner: quantized_vllm

• json_multistep: FAIL (provisional; n<10) (pass_rate=0.400)
• stateful_followup: PASS (provisional; n<10) (pass_rate=1.000)
• toolcall_only: PASS (provisional; n<10) (pass_rate=1.000)
• mixed_brief_json: PASS (provisional; n<10) (pass_rate=1.000)

Recorded Artifacts

Artifacts are recorded in the run manifest with file sizes and SHA256 hashes for verification.

• (No artifacts listed in manifest.)

Top Failure Reasons (when available)

quantized_vllm / json_multistep

• 2× checks_consistent_ok+oracle_equiv_ok
• 1× checks_consistent_ok+stop_semantics_ok