# Evaluation Summary (v7.21)

- Run ID: `20260629_184555`
- Model ID: `/mnt/d/Development/Libraries/Mistral-Nemo-Instruct-2407-AWQ_20260529_152601`
- Quant types requested: `[]`
- Artifacts recorded: `0`

## Battery Coverage

- Fixtures version: `v7.21`
- json_multistep_cases: 5 cases
- json_shelf_cases: 4 cases
- mixed_brief_json_cases: 2 cases
- stateful_followup_cases: 2 cases
- toolcall_only_cases: 2 cases
- Total: 15 cases

## Pass-Rate Signals

Pass rates are computed per family as the fraction of rows where **all** listed signals equal 1.
(If a signal column is missing for a row, it is ignored for that row.)

- `json_multistep`: pass if schema_ok, checks_consistent_ok, stop_semantics_ok, oracle_equiv_ok
- `stateful_followup`: pass if turn1_parse_ok, turn2_parse_ok, turn1_exact_match, turn2_exact_match
- `toolcall_only`: pass if tool_name_ok, args_ok
- `mixed_brief_json`: pass if answer_line_ok, json_parse_ok, schema_ok
- `json`: pass if schema_ok, constraints_ok
- `fuzz`: pass if schema_ok, constraints_ok
- `toolcall`: pass if stage1_tool_parse_ok, stage1_tool_schema_ok
- `mcq`: pass if choice_ok

## Bucket Score Scale

Some families report an auxiliary `bucket_score` used for ranking/triage. The scale is family-specific:

- `json_multistep`: Range 0–10. Score 10 means plan_exact_match=1 AND checks_ok=1 AND final_match_reported=1. Otherwise the score is plan_prefix_match_len (how many leading steps match the oracle plan). Tier-1 uses oracle_equiv_ok (computed final matches oracle) as outcome correctness; final_match_reported/final_consistent_ok are tracked as Tier-2 reporting discipline.
- `stateful_followup`: Range 0–10. Higher is better; 10 indicates both turns parse and match expected state updates.
- `toolcall_only`: Range 0–10. Higher is better; 10 indicates correct tool name AND valid args under the tool schema.
- `mixed_brief_json`: Range 0–10. Higher is better; 10 indicates correct ANSWER line AND valid JSON block meeting schema.
- `json`: Range 0–10. Higher is better; 10 indicates exact/valid JSON shelf placement meeting constraints. (See CSV for exact signal columns.)
- `fuzz`: Range 0–10. Property-based regression. Score mirrors the JSON shelf scoring signals when present; pass rate uses schema_ok AND constraints_ok on rows where those signals exist.
- `toolcall`: Range 0–10. Higher is better; 10 indicates tool-call JSON parses and validates against schema.
- `mcq`: Range 0–1. 1 indicates the correct choice was selected.

## Results by Runner and Family

### Runner: `quantized_vllm`

| Family | N | Pass rate | Averages |
|---|---:|---:|---|
| json_multistep | 5 | 0.400 | bucket_score=1.800 |
| stateful_followup | 2 | 1.000 | bucket_score=2.000 |
| toolcall_only | 2 | 1.000 | bucket_score=2.000 |
| mixed_brief_json | 2 | 1.000 | bucket_score=2.000 |
| json | 4 | n/a | bucket_score=10.000 |
| fuzz | 20 | n/a | bucket_score=10.000 |
| toolcall | 2 | 1.000 | bucket_score=11.000 |
| mcq | 5 | n/a | bucket_score=0.600 |

## Fuzz Regression Notes

`fuzz` is a property-based regression suite and is summarized as a family for stability tracking; it is not a direct agent-readiness proxy.

## Signal Breakdown (Diagnostics)

For each family, the pass rate is a conjunction of signals. The tables below show per-signal success rates, which helps isolate which signal is blocking a family from passing.

### Runner: `quantized_vllm`

#### Family: `json_multistep`

| Signal | Rate | Notes |
|---|---:|---|
| schema_ok | 1.000 | tier-1 (gating) |
| checks_consistent_ok | 0.400 | tier-1 (gating) |
| stop_semantics_ok | 0.800 | tier-1 (gating) |
| oracle_equiv_ok | 0.600 | tier-1 (gating) |
| final_consistent_ok | 0.000 | tier-2 (tracked; non-gating) |
| final_match_reported | 0.000 | tier-2 (tracked; non-gating) |

#### Family: `stateful_followup`

| Signal | Rate | Notes |
|---|---:|---|
| turn1_parse_ok | 1.000 | tier-1 (gating) |
| turn2_parse_ok | 1.000 | tier-1 (gating) |
| turn1_exact_match | 1.000 | tier-1 (gating) |
| turn2_exact_match | 1.000 | tier-1 (gating) |

#### Family: `toolcall_only`

| Signal | Rate | Notes |
|---|---:|---|
| tool_name_ok | 1.000 | tier-1 (gating) |
| args_ok | 1.000 | tier-1 (gating) |

#### Family: `mixed_brief_json`

| Signal | Rate | Notes |
|---|---:|---|
| answer_line_ok | 1.000 | tier-1 (gating) |
| json_parse_ok | 1.000 | tier-1 (gating) |
| schema_ok | 1.000 | tier-1 (gating) |

## Readiness Snapshot

The snapshot below provides a coarse, decision-oriented view derived from pass rates.
Thresholds are intentionally conservative: PASS ≥ 0.90, WARN ≥ 0.70, else FAIL.

### Runner: `quantized_vllm`

- json_multistep: **FAIL (provisional; n<10)** (pass_rate=0.400)
- stateful_followup: **PASS (provisional; n<10)** (pass_rate=1.000)
- toolcall_only: **PASS (provisional; n<10)** (pass_rate=1.000)
- mixed_brief_json: **PASS (provisional; n<10)** (pass_rate=1.000)

## Recorded Artifacts

Artifacts are recorded in the run manifest with file sizes and SHA256 hashes for verification.

- (No artifacts listed in manifest.)

## Top Failure Reasons (when available)

### `quantized_vllm` / `json_multistep`

- 2× checks_consistent_ok+oracle_equiv_ok
- 1× checks_consistent_ok+stop_semantics_ok