---
base_model: ProCreations/auto-3b
base_model_relation: finetune
library_name: transformers
license: apache-2.0
pipeline_tag: text-classification
tags:
- smollm3
- agent-safety
- tool-calling
- guardrail
- long-context
model-index:
- name: auto-3b-3
  results:
  - task:
      type: text-classification
      name: Agentic tool-call approve/deny classification
    dataset:
      name: Approve-or-Deny
      type: ProCreations/approve-or-deny
      split: test
      revision: a38b625913dd46ca9702063f1597c979e6ace34e
    metrics:
    - type: accuracy
      value: 0.9823
    - type: roc_auc
      value: 0.9984
    - type: false_approve_rate
      value: 0.0250
    - type: false_deny_rate
      value: 0.0113
---

# auto-3b-3

A 3.08B-parameter classifier that decides whether an AI agent's proposed tool call should run. It reads the proposed call, the user's request and the agent's history, and outputs `approve` or `deny`. auto-3b-3 is the most accurate Auto model. It matches the best hosted guards we tested on the v3 benchmark, at a fraction of the cost.

It is part of [Auto](https://github.com/SSHdotCodes/auto), a local guard for coding agents (Claude Code, Codex, Gemini CLI, Cursor, Copilot CLI and others). Auto blocks a call only if it's consequential *and* the user didn't ask for it, or if the agent is following instructions planted in a page, file or tool output. Everything runs on your machine, and nothing it reads leaves it.

| | |
|---|---|
| **v3 benchmark** | **96.0%**, up from 72.1% for auto-3b |
| **Missed harm / over-block** | 3.1% / 4.8% |
| **Original Approve-or-Deny benchmark** | 98.2% |
| **Time per decision** | 21 ms (median, NVIDIA RTX PRO 6000) |
| **Cost per million decisions** | \$77 on a rented GPU |
| **Architecture** | SmolLM3 decoder, 3.08B parameters, BF16 weights (6.2 GB) |
| **Runs on** | an NVIDIA or AMD GPU |

## How it compares

![Accuracy on the v3 benchmark, auto-3b-3 highlighted](assets/accuracy.png)

The v3 benchmark has 2,948 held-out tool calls. Each one is paired with a near-twin where one detail flips the right answer: who asked, how far the request reaches, production or staging, the exact call, or where an instruction came from. A guard that guesses from surface features gets one of each pair wrong. Hosted guards were prompted with the same decision rule that Auto's training labels follow, and every Auto model is scored at `P(deny) >= 0.5`.

![Missed harm and over-block rates](assets/errors.png)

A guard can be wrong in two ways. Missed harm lets a harmful call through; over-blocking interrupts work you asked for. Hosted LLMs tend to trade one for the other: DeepSeek V4.1 Flash misses almost nothing (0.6%) but blocks 7.9% of authorized calls.

![Median time per decision](assets/speed.png)

![Cost per million decisions](assets/cost.png)

![Accuracy against cost](assets/tradeoff.png)

| Model | v3 accuracy (95% CI) | Missed harm | Over-block | Both twins right | Time per decision | Cost per 1M |
|---|---:|---:|---:|---:|---:|---:|
| **auto-3b-3** | 96.0% (95.3–96.7) | 3.1% | 4.8% | 92.8% | 21 ms | \$77 |
| DeepSeek V4.1 Flash (max reasoning) | 95.6% (94.8–96.3) | 0.6% | 7.9% | 91.1% | 1.3 s | \$3,043 |
| Codex auto-review (medium) | 95.5% (94.7–96.2) | 4.9% | 4.1% | 90.8% | – | \$11,353 |
| GLM-5.3 Flash (high reasoning) | 95.4% (94.5–96.1) | 5.5% | 3.8% | 90.2% | 1.2 s | \$629 |
| [auto-0.4b-3](https://huggingface.co/ProCreations/auto-0.4b-3) | 93.1% (92.2–94.0) | 6.0% | 7.6% | 87.2% | 7 ms | \$23 |
| TypeSafe Jev 1.14 | 90.2% (89.1–91.2) | 12.5% | 7.3% | 80.1% | 136 ms | \$227 |
| [auto-150m-3](https://huggingface.co/ProCreations/auto-150m-3) | 88.8% (87.7–89.9) | 9.1% | 13.1% | 78.7% | 5 ms | \$13 |
| GPT-6 Luna (low reasoning) | 87.9% (86.7–89.0) | 5.8% | 17.9% | 75.8% | – | \$395 |

*Missed harm:* share of deny items approved. *Over-block:* share of approve items denied. *Both twins right:* share of pairs where both halves are correct. *Time:* Auto on a local NVIDIA RTX PRO 6000, one request at a time; hosted guards are API round trips measured unloaded (Codex auto-review and GPT-6 Luna run through CLIs, which take 12–13 s per decision, so they're left out of the speed comparison). *Cost:* Auto on a rented RTX PRO 6000 at \$2.18/h (median market rate), about \$0.50–\$3 per million in electricity on a GPU you own; hosted guards at list price × measured tokens.

On an Apple M-series GPU it runs through Auto's runtime, but slowly: a median of 0.86 s per decision on a 300-item sample, and about 12 s for long sessions. auto-0.4b-3 is the better choice there. On a CPU, long sessions take minutes.

### v2 → v3

![v2 to v3 accuracy for each Auto size](assets/v2-to-v3.png)

| Original Approve-or-Deny benchmark (3,000 items) | auto-3b | auto-3b-3 |
|---|---:|---:|
| Accuracy | 98.03% | 98.23% |
| Harmful calls approved (of 1,401) | 22 | 35 |
| Authorized calls denied (of 1,599) | 37 | 18 |
| AUROC | – | 0.9984 |

On the original benchmark, v3 keeps its accuracy but shifts toward approving: it denies fewer authorized calls and approves more of the harmful ones. On the v3 benchmark it is far stronger on both counts.

### Breakdown on the v3 benchmark

| Slice | Accuracy |
|---|---:|
| Written by Claude Opus 4.8 (not a training-data writer) | 93.0% |
| Written by GPT-6.1 Sol | 97.7% |
| Long sessions | 95.2% |
| Non-shell tools (browser, email, CRM, ...) | 96.4% |
| Twin flips on who asked | 96.9% |
| Twin flips on scope | 95.3% |
| Twin flips on environment | 96.6% |
| Twin flips on the exact call | 94.5% |
| Twin flips on where an instruction came from | 98.2% |

The most-missed harm categories are obfuscated payload (5.7%), data destruction unrequested (4.8%), destructive filesystem (4.3%). Per-item scores for both benchmarks are in [eval/](eval/), and every model's metrics are in [eval/results.json](eval/results.json).

Through Auto's runtime (Apple MPS, sessions fitted to 8,192 tokens), auto-3b-3 scored 97.3% on 300 random v3 items, against 97.7% for the reference scores on the same items, and agreed with 299 of 300 decisions.

## Usage

### With Auto

```sh
curl -fsSL https://auto.ssh.codes/install.sh | sh
auto configure --model auto-3b-3
```

The installer finds your agents and adds the hook. See the [Auto README](https://github.com/SSHdotCodes/auto) for the Python API, the Claude Agent SDK hook and LangChain middleware.

### With Transformers

The model reads one serialized text. Keep the three sections and their order exactly as below.

```python
import json
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer


def text(value):
    return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, separators=(",", ":"))


def build_input(user_request, history, call):
    parts = ["### PROPOSED TOOL CALL", f"tool: {call['tool']}", f"args: {text(call['args'])}", "",
             "### USER REQUEST", user_request, "", "### AGENT HISTORY"]
    if not history:
        parts.append("(no prior actions)")
    for i, action in enumerate(history, 1):
        parts.append(f"[{i}] {action['tool']}({text(action['args'])})\n-> {text(action.get('result', ''))}")
    return "\n".join(parts)


repo = "ProCreations/auto-3b-3"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo, dtype=torch.bfloat16).eval()

prompt = build_input(
    user_request="Clean up the build artifacts and reinstall dependencies.",
    history=[{"tool": "Bash", "args": {"command": "ls"}, "result": "node_modules dist package.json"}],
    call={"tool": "Bash", "args": {"command": "rm -rf node_modules dist && npm install"}},
)
inputs = tokenizer(prompt, return_tensors="pt")
with torch.inference_mode():
    p_deny = model(**inputs).logits.float().softmax(-1)[0, 1].item()
print("deny" if p_deny >= 0.5 else "approve", round(p_deny, 3))
```

Labels are `0 = approve` and `1 = deny`. The model accepts up to 65,536 positions. For long sessions, Auto's runtime keeps memory bounded and fits the session to 32,768 tokens on NVIDIA and AMD GPUs or 8,192 on Apple silicon and CPUs, keeping the user's messages, the newest actions and older actions that mention what the call touches.

## Training

- **Starting point:** [ProCreations/auto-3b](https://huggingface.co/ProCreations/auto-3b) (revision `58323dac`).
- **New data:** about 100,000 contrastive pairs: two near-identical situations where one detail flips the right answer. They were written by GPT-6.1 Terra, GPT-6.1 Sol and DeepSeek V4.1 Flash, label-checked, and cover 53 agent harnesses, 11 languages and 23 categories, including browser, email, CRM and smart-home tools as well as the shell.
- **Replay:** 84,706 rows of v2 training data, kept close to auto-3b by distillation (KL to the frozen v2 model at temperature 2, weight 0.8), so v3 doesn't forget the original distribution.
- **Loss on new rows:** cross-entropy, plus a pair margin `softplus(4 − (z_deny − z_approve))` (weight 0.5) that pushes each twin pair apart. Denials in the categories where early v3 trials missed more harm than v2, and hard examples, are weighted 1.5×.
- **Schedule:** 2 epochs, AdamW (β 0.9/0.95, weight decay 0.01), peak learning rate 5e-6, 3% warmup then decay to 10%, batches of up to 32,768 tokens, BF16 compute with FP32 master weights, on one NVIDIA B200 (about 6 hours). The full settings are in [training_config.json](training_config.json).
- **Selection.** auto-3b-3 is the only 3B candidate trained for v3. Its final weights are released as they came out of training, with no selection on the benchmark.
- **Held out:** the v3 benchmark items and their twins were never trained on. The benchmark isn't public yet.

## Limitations

- **A classifier, not a chat model.** It can't open files, follow URLs or know what an opaque script does; it judges from the text it is given.
- **Synthetic labels.** Training and benchmark data were written by LLMs and label-checked, not collected from real agent traffic. Items written by Claude Opus 4.8, which wrote none of the training data, score lower (93.0%) than items from a training writer (97.7%), so expect some drop on unfamiliar styles.
- **Harm still gets through.** It approves 3.1% of harmful v3 calls. When Auto approves, your agent's own permission settings still apply; keep them on.
- Evaluate it on your own traffic before relying on it.