# SAM Benchmark v1 — Full Report

## Methodology

This benchmark evaluates structured action generation: given a natural language
task and a JSON schema, produce a syntactically valid JSON sequence of operations
that matches a reference answer.

**Test suite**: 200 prompts, deterministically generated from
SEED=42. The exact prompts are saved to `test_suite.json` for
reproducibility and audit.

**Tier distribution** (current suite):
- Tier 1 — Atomic — single op: **40 cases** (20.0%)
- Tier 2 — Compound — 2-3 ops same domain: **40 cases** (20.0%)
- Tier 3 — Noisy — typos / casing / fillers: **40 cases** (20.0%)
- Tier 4 — Long chain — 4+ ops: **40 cases** (20.0%)
- Tier 5 — Cross-domain workflow: **40 cases** (20.0%)

**Domain distribution** (current suite):
- `WORKFLOW`: 49 cases (24.5%)
- `ROS`: 46 cases (23.0%)
- `DB`: 27 cases (13.5%)
- `HTTP`: 25 cases (12.5%)
- `VEHICLE`: 17 cases (8.5%)
- `MQTT`: 15 cases (7.5%)
- `ECOMMERCE`: 8 cases (4.0%)
- `HOME`: 7 cases (3.5%)
- `FILE`: 4 cases (2.0%)
- `CAL`: 2 cases (1.0%)

**Metrics**:
- `JSON Valid`: output parses as valid JSON ∈ [0,100]%
- `Op Correct`: output is a list of dicts AND every op name matches the expected set ∈ [0,100]%
- `Exact Match`: full structural match with reference, with float tolerance 0.02 for numerics ∈ [0,100]%
- `Latency`: mean wall-clock time per prompt in milliseconds (single-thread, no batching)
- `Size`: model file size on disk (MB)

**Important disclosures**:
1. SAM is **trained on the same domain distribution as this benchmark** (multi-domain JSON ops).
   General-purpose LLMs are evaluated zero-shot. This is a deliberate comparison of
   "specialist 35M model" vs "generalist N-B model" on a structured task, NOT a claim
   of general superiority.
2. **Tier distribution is currently imbalanced** toward Tier 1 (atomic). Aggregate
   exact-match numbers are dominated by easy cases. Always read the per-tier
   breakdown below for fair comparison.
3. **Domain distribution is currently imbalanced** toward ROS (robotics). This
   reflects SAM's lineage from Foros (ROS-only). Per-domain breakdowns reveal
   each model's actual strength.
4. All LLMs are loaded from publicly available HuggingFace weights with no
   fine-tuning. Prompt template is identical across all LLMs.

## Headline results (aggregate across all 200 cases)

| Rank | Model | Params | Size (MB) | JSON Valid | Op Correct | Exact Match | Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | **[SAM]** SAM | 33.4M | 127.4 | 88.0% | 63.5% | 18.0% | 715.4 |
| 2 |  Qwen2.5-3B | 3085.9M | 5886.0 | 100.0% | 82.5% | 16.0% | 2061.1 |
| 3 |  DeepSeek-R1-Qwen3-8B-GGUF | 8000.0M | 4794.9 | 98.0% | 79.0% | 11.5% | 31737.9 |
| 4 |  TinyLlama-1.1B | 1100.0M | 2098.2 | 92.0% | 55.5% | 9.0% | 4199.3 |
| 5 |  Qwen2.5-1.5B | 1543.7M | 2944.4 | 100.0% | 78.5% | 9.0% | 1852.4 |
| 6 |  Qwen2.5-0.5B | 494.0M | 942.3 | 70.5% | 30.0% | 2.5% | 1937.2 |
| 7 |  SmolLM2-1.7B | 1711.4M | 3264.2 | 89.0% | 38.5% | 2.5% | 1694.9 |
| 8 |  SmolLM2-360M | 361.8M | 690.1 | 89.0% | 0.0% | 0.0% | 2042.9 |
| 9 |  Llama-3.2-1B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 10 |  Gemma-2-2B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 11 |  Llama-3.2-3B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 12 |  Phi-3.5-mini-3.8B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 13 |  Llama-3.2-3B-uns | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 14 |  Gemma-3-270M-GGUF | 270.0M | 241.4 | 92.0% | 2.0% | 0.0% | 6322.5 |
| 15 |  Ministral-3-8B-GGUF | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |

## Per-tier exact-match breakdown (all models)

_This is the breakdown that matters for fair comparison. Each model's exact-match score is shown for each difficulty tier._

| Model | T1 atomic | T2 compound | T3 noisy | T4 long | T5 cross |
|---|---|---|---|---|---|
| **[SAM]** SAM | 37.5% (n=40) | 27.5% (n=40) | 25.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| Qwen2.5-3B | 40.0% (n=40) | 12.5% (n=40) | 27.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| DeepSeek-R1-Qwen3-8B-GGUF | 27.5% (n=40) | 7.5% (n=40) | 22.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| TinyLlama-1.1B | 27.5% (n=40) | 12.5% (n=40) | 5.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| Qwen2.5-1.5B | 30.0% (n=40) | 12.5% (n=40) | 2.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| Qwen2.5-0.5B | 12.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| SmolLM2-1.7B | 12.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| SmolLM2-360M | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| Gemma-3-270M-GGUF | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |

## Per-tier JSON-valid breakdown (all models)

| Model | T1 atomic | T2 compound | T3 noisy | T4 long | T5 cross |
|---|---|---|---|---|---|
| **[SAM]** SAM | 100.0% | 100.0% | 100.0% | 75.0% | 65.0% |
| Qwen2.5-3B | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| DeepSeek-R1-Qwen3-8B-GGUF | 100.0% | 95.0% | 100.0% | 95.0% | 100.0% |
| TinyLlama-1.1B | 100.0% | 90.0% | 100.0% | 70.0% | 100.0% |
| Qwen2.5-1.5B | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Qwen2.5-0.5B | 60.0% | 65.0% | 67.5% | 95.0% | 65.0% |
| SmolLM2-1.7B | 92.5% | 80.0% | 100.0% | 72.5% | 100.0% |
| SmolLM2-360M | 100.0% | 100.0% | 100.0% | 45.0% | 100.0% |
| Gemma-3-270M-GGUF | 85.0% | 97.5% | 90.0% | 87.5% | 100.0% |

## Per-domain exact-match breakdown (all models)

| Model | CAL (n=2) | DB (n=27) | ECOMMERCE (n=8) | FILE (n=4) | HOME (n=7) | HTTP (n=25) | MQTT (n=15) | ROS (n=46) | VEHICLE (n=17) | WORKFLOW (n=49) |
|---|---|---|---|---|---|---|---|---|---|---|
| **[SAM]** SAM | 100.0% | 11.1% | 0.0% | 75.0% | 0.0% | 44.0% | 6.7% | 10.9% | 17.6% | 16.3% |
| Qwen2.5-3B | 50.0% | 18.5% | 0.0% | 0.0% | 0.0% | 8.0% | 73.3% | 21.7% | 17.6% | 0.0% |
| DeepSeek-R1-Qwen3-8B-GGUF | 50.0% | 0.0% | 0.0% | 25.0% | 0.0% | 8.0% | 73.3% | 10.9% | 17.6% | 0.0% |
| TinyLlama-1.1B | 0.0% | 18.5% | 0.0% | 25.0% | 0.0% | 0.0% | 0.0% | 17.4% | 23.5% | 0.0% |
| Qwen2.5-1.5B | 0.0% | 18.5% | 0.0% | 0.0% | 0.0% | 4.0% | 0.0% | 19.6% | 17.6% | 0.0% |
| Qwen2.5-0.5B | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 4.3% | 17.6% | 0.0% |
| SmolLM2-1.7B | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 4.3% | 17.6% | 0.0% |
| SmolLM2-360M | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Gemma-3-270M-GGUF | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |

## How to read this report

- The **headline aggregate** can be misleading because tier 1 is over-represented.
- The **per-tier breakdown** is the honest comparison. Look at T1-T5 separately.
- The **per-domain breakdown** shows which models are strong in which areas.
- SAM is expected to win on the structured-output dimension (exact match on
  domains it was trained on) and to be **much smaller and faster**. Where
  general LLMs win is usually noisy/cross-domain (their broader training helps).
- **No metric in isolation is the full story.** A reviewer should look at all
  three (exact match, latency, size) together to evaluate the trade-off.

## Reproducibility

```bash
python sam_benchmark.py
```

Test suite regenerated deterministically from SEED=42. LLM weights from
public HuggingFace repos. SAM weights from `AMFORGE/sam-v1-checkpoints/best.pt`
(auto-downloaded if not present locally).

Raw per-case results are in `results_full.json` (one entry per (model, case)
including model output and pass/fail flags). Per-suite aggregates are in
`results.json`. The test suite itself is in `test_suite.json`.