File size: 7,518 Bytes
d2ec8c9 b866d36 d2ec8c9 b866d36 d2ec8c9 b866d36 d2ec8c9 b866d36 d2ec8c9 b866d36 d2ec8c9 b866d36 d2ec8c9 b866d36 d2ec8c9 b866d36 d2ec8c9 b866d36 d2ec8c9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | # SAM Benchmark v1 — Full Report
## Methodology
This benchmark evaluates structured action generation: given a natural language
task and a JSON schema, produce a syntactically valid JSON sequence of operations
that matches a reference answer.
**Test suite**: 200 prompts, deterministically generated from
SEED=42. The exact prompts are saved to `test_suite.json` for
reproducibility and audit.
**Tier distribution** (current suite):
- Tier 1 — Atomic — single op: **40 cases** (20.0%)
- Tier 2 — Compound — 2-3 ops same domain: **40 cases** (20.0%)
- Tier 3 — Noisy — typos / casing / fillers: **40 cases** (20.0%)
- Tier 4 — Long chain — 4+ ops: **40 cases** (20.0%)
- Tier 5 — Cross-domain workflow: **40 cases** (20.0%)
**Domain distribution** (current suite):
- `WORKFLOW`: 49 cases (24.5%)
- `ROS`: 46 cases (23.0%)
- `DB`: 27 cases (13.5%)
- `HTTP`: 25 cases (12.5%)
- `VEHICLE`: 17 cases (8.5%)
- `MQTT`: 15 cases (7.5%)
- `ECOMMERCE`: 8 cases (4.0%)
- `HOME`: 7 cases (3.5%)
- `FILE`: 4 cases (2.0%)
- `CAL`: 2 cases (1.0%)
**Metrics**:
- `JSON Valid`: output parses as valid JSON ∈ [0,100]%
- `Op Correct`: output is a list of dicts AND every op name matches the expected set ∈ [0,100]%
- `Exact Match`: full structural match with reference, with float tolerance 0.02 for numerics ∈ [0,100]%
- `Latency`: mean wall-clock time per prompt in milliseconds (single-thread, no batching)
- `Size`: model file size on disk (MB)
**Important disclosures**:
1. SAM is **trained on the same domain distribution as this benchmark** (multi-domain JSON ops).
General-purpose LLMs are evaluated zero-shot. This is a deliberate comparison of
"specialist 35M model" vs "generalist N-B model" on a structured task, NOT a claim
of general superiority.
2. **Tier distribution is currently imbalanced** toward Tier 1 (atomic). Aggregate
exact-match numbers are dominated by easy cases. Always read the per-tier
breakdown below for fair comparison.
3. **Domain distribution is currently imbalanced** toward ROS (robotics). This
reflects SAM's lineage from Foros (ROS-only). Per-domain breakdowns reveal
each model's actual strength.
4. All LLMs are loaded from publicly available HuggingFace weights with no
fine-tuning. Prompt template is identical across all LLMs.
## Headline results (aggregate across all 200 cases)
| Rank | Model | Params | Size (MB) | JSON Valid | Op Correct | Exact Match | Latency (ms) |
|---|---|---|---|---|---|---|---|
| 1 | **[SAM]** SAM | 33.4M | 127.4 | 88.0% | 63.5% | 18.0% | 715.4 |
| 2 | Qwen2.5-3B | 3085.9M | 5886.0 | 100.0% | 82.5% | 16.0% | 2061.1 |
| 3 | DeepSeek-R1-Qwen3-8B-GGUF | 8000.0M | 4794.9 | 98.0% | 79.0% | 11.5% | 31737.9 |
| 4 | TinyLlama-1.1B | 1100.0M | 2098.2 | 92.0% | 55.5% | 9.0% | 4199.3 |
| 5 | Qwen2.5-1.5B | 1543.7M | 2944.4 | 100.0% | 78.5% | 9.0% | 1852.4 |
| 6 | Qwen2.5-0.5B | 494.0M | 942.3 | 70.5% | 30.0% | 2.5% | 1937.2 |
| 7 | SmolLM2-1.7B | 1711.4M | 3264.2 | 89.0% | 38.5% | 2.5% | 1694.9 |
| 8 | SmolLM2-360M | 361.8M | 690.1 | 89.0% | 0.0% | 0.0% | 2042.9 |
| 9 | Llama-3.2-1B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 10 | Gemma-2-2B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 11 | Llama-3.2-3B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 12 | Phi-3.5-mini-3.8B | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 13 | Llama-3.2-3B-uns | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
| 14 | Gemma-3-270M-GGUF | 270.0M | 241.4 | 92.0% | 2.0% | 0.0% | 6322.5 |
| 15 | Ministral-3-8B-GGUF | ? | 0.0 | 0.0% | 0.0% | 0.0% | 0.0 |
## Per-tier exact-match breakdown (all models)
_This is the breakdown that matters for fair comparison. Each model's exact-match score is shown for each difficulty tier._
| Model | T1 atomic | T2 compound | T3 noisy | T4 long | T5 cross |
|---|---|---|---|---|---|
| **[SAM]** SAM | 37.5% (n=40) | 27.5% (n=40) | 25.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| Qwen2.5-3B | 40.0% (n=40) | 12.5% (n=40) | 27.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| DeepSeek-R1-Qwen3-8B-GGUF | 27.5% (n=40) | 7.5% (n=40) | 22.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| TinyLlama-1.1B | 27.5% (n=40) | 12.5% (n=40) | 5.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| Qwen2.5-1.5B | 30.0% (n=40) | 12.5% (n=40) | 2.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| Qwen2.5-0.5B | 12.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| SmolLM2-1.7B | 12.5% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| SmolLM2-360M | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
| Gemma-3-270M-GGUF | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) | 0.0% (n=40) |
## Per-tier JSON-valid breakdown (all models)
| Model | T1 atomic | T2 compound | T3 noisy | T4 long | T5 cross |
|---|---|---|---|---|---|
| **[SAM]** SAM | 100.0% | 100.0% | 100.0% | 75.0% | 65.0% |
| Qwen2.5-3B | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| DeepSeek-R1-Qwen3-8B-GGUF | 100.0% | 95.0% | 100.0% | 95.0% | 100.0% |
| TinyLlama-1.1B | 100.0% | 90.0% | 100.0% | 70.0% | 100.0% |
| Qwen2.5-1.5B | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Qwen2.5-0.5B | 60.0% | 65.0% | 67.5% | 95.0% | 65.0% |
| SmolLM2-1.7B | 92.5% | 80.0% | 100.0% | 72.5% | 100.0% |
| SmolLM2-360M | 100.0% | 100.0% | 100.0% | 45.0% | 100.0% |
| Gemma-3-270M-GGUF | 85.0% | 97.5% | 90.0% | 87.5% | 100.0% |
## Per-domain exact-match breakdown (all models)
| Model | CAL (n=2) | DB (n=27) | ECOMMERCE (n=8) | FILE (n=4) | HOME (n=7) | HTTP (n=25) | MQTT (n=15) | ROS (n=46) | VEHICLE (n=17) | WORKFLOW (n=49) |
|---|---|---|---|---|---|---|---|---|---|---|
| **[SAM]** SAM | 100.0% | 11.1% | 0.0% | 75.0% | 0.0% | 44.0% | 6.7% | 10.9% | 17.6% | 16.3% |
| Qwen2.5-3B | 50.0% | 18.5% | 0.0% | 0.0% | 0.0% | 8.0% | 73.3% | 21.7% | 17.6% | 0.0% |
| DeepSeek-R1-Qwen3-8B-GGUF | 50.0% | 0.0% | 0.0% | 25.0% | 0.0% | 8.0% | 73.3% | 10.9% | 17.6% | 0.0% |
| TinyLlama-1.1B | 0.0% | 18.5% | 0.0% | 25.0% | 0.0% | 0.0% | 0.0% | 17.4% | 23.5% | 0.0% |
| Qwen2.5-1.5B | 0.0% | 18.5% | 0.0% | 0.0% | 0.0% | 4.0% | 0.0% | 19.6% | 17.6% | 0.0% |
| Qwen2.5-0.5B | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 4.3% | 17.6% | 0.0% |
| SmolLM2-1.7B | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 4.3% | 17.6% | 0.0% |
| SmolLM2-360M | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Gemma-3-270M-GGUF | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
## How to read this report
- The **headline aggregate** can be misleading because tier 1 is over-represented.
- The **per-tier breakdown** is the honest comparison. Look at T1-T5 separately.
- The **per-domain breakdown** shows which models are strong in which areas.
- SAM is expected to win on the structured-output dimension (exact match on
domains it was trained on) and to be **much smaller and faster**. Where
general LLMs win is usually noisy/cross-domain (their broader training helps).
- **No metric in isolation is the full story.** A reviewer should look at all
three (exact match, latency, size) together to evaluate the trade-off.
## Reproducibility
```bash
python sam_benchmark.py
```
Test suite regenerated deterministically from SEED=42. LLM weights from
public HuggingFace repos. SAM weights from `AMFORGE/sam-v1-checkpoints/best.pt`
(auto-downloaded if not present locally).
Raw per-case results are in `results_full.json` (one entry per (model, case)
including model output and pass/fail flags). Per-suite aggregates are in
`results.json`. The test suite itself is in `test_suite.json`.
|