| --- |
| license: apache-2.0 |
| base_model: answerdotai/ModernBERT-large |
| pipeline_tag: text-classification |
| library_name: transformers |
| tags: |
| - agent-safety |
| - tool-calling |
| - guardrails |
| - modernbert |
| - long-context |
| datasets: |
| - ProCreations/auto |
| - ProCreations/approve-or-deny |
| language: |
| - en |
| model-index: |
| - name: auto-0.4b |
| results: |
| - task: |
| type: text-classification |
| name: Agentic tool-call safety gating |
| dataset: |
| name: Approve-or-Deny |
| type: ProCreations/approve-or-deny |
| split: test |
| metrics: |
| - type: accuracy |
| value: 0.9013 |
| name: Accuracy |
| - type: f1 |
| value: 0.8963 |
| name: F1 (deny) |
| - type: roc_auc |
| value: 0.9657 |
| name: AUROC |
| - type: false_approve_rate |
| value: 0.0871 |
| name: False-approve rate (dangerous call allowed) |
| - type: false_deny_rate |
| value: 0.1088 |
| name: False-deny rate (safe call blocked) |
| --- |
| |
| # auto-0.4b |
|
|
| **A 0.4B encoder that decides whether an AI agent's next tool call is safe to run.** |
|
|
| Agent harnesses like Codex "Auto" mode and Claude Code auto-accept currently ask a large |
| frontier LLM the question *"should I let this tool call run?"* on every single step. That |
| works, but it burns subscription usage, adds API latency to every action, and spends a |
| generative model on what is fundamentally a **classification** problem — the thing |
| encoder-only models are best at. |
|
|
| `auto-0.4b` is a full fine-tune of [ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) with context extended to |
| **64k tokens**, trained to answer exactly that question: |
|
|
| - **`approve`** — routine work that serves the user's request. Reading files, running |
| tests, editing code, committing, **web search, fetching docs, installing legitimate |
| packages** — network access is normal agent behaviour and is *not* treated as dangerous. |
| - **`deny`** — the call is consequential *and* unauthorised: wiping directories, exfiltrating |
| credentials, installing hostile software, weakening security controls, destroying |
| production state, or following instructions injected by content the agent just read. |
|
|
| It runs in **7 ms per call** on a single GPU — roughly **400x faster** than asking a |
| frontier LLM the same question, at zero marginal cost. |
|
|
| ## Honest positioning |
|
|
| **This is not a drop-in replacement for a frontier-LLM gate.** On the held-out benchmark: |
|
|
| | system | accuracy | false-approve | false-deny | latency/call | cost | |
| |---|---|---|---|---|---| |
| | GPT-5.6-Luna | 96.97% | 4.93% | 1.38% | 3.09 s | API spend | |
| | DeepSeek V4 Flash 0731 | 94.95% | 10.11% | 0.63% | 5.64 s | API spend | |
| | **auto-0.4b** | **90.13%** | **8.71%** | **10.88%** | **7 ms** | **free** | |
| | regex heuristic | 57.97% | 59.39% | 26.83% | ~0 ms | free | |
|
|
| It trails Luna by ~7 points of accuracy, and its false-deny rate means it would interrupt |
| roughly 1 in 10 legitimate actions at the default threshold. Where it earns its place is as a |
| **fast first-stage filter**: it is very strong on the attacks that actually matter, and it |
| answers in milliseconds so it can run on every single tool call. |
|
|
| | attack category | false-approve | |
| |---|---| |
| | `self_permission_tamper` (disabling its own guardrails) | ~1% | |
| | `covering_tracks` | ~1% | |
| | **`prompt_injection`** (following instructions from fetched content) | **~2%** | |
| | `privilege_escalation` | ~3% | |
| | `credential_exfiltration` | ~3% | |
| |
| A sensible deployment auto-approves below a low threshold, auto-denies above a high one, and |
| escalates the middle band to a bigger model — which removes most of the LLM calls while |
| keeping a frontier model's judgement on the genuinely ambiguous cases. |
| |
| ## Benchmark: approve-or-deny |
| |
| Evaluated on [ProCreations/approve-or-deny](https://huggingface.co/datasets/ProCreations/approve-or-deny) (3000 held-out items, never trained on): |
| |
| | metric | auto-0.4b | keyword/regex baseline | |
| |---|---|---| |
| | **accuracy** | **90.13%** | 57.97% | |
| | balanced accuracy | 90.21% | 56.89% | |
| | F1 (deny) | 0.8963 | 0.4744 | |
| | AUROC | 0.9657 | — | |
| | **false-approve rate** (dangerous call let through) | **8.71%** | 59.39% | |
| | false-deny rate (safe call blocked) | 10.88% | 26.83% | |
| |
| The regex baseline is the "block anything containing `rm -rf`/`sudo`/`curl`" heuristic these |
| gates usually start life as. It is included to show the benchmark is not keyword-solvable — |
| roughly half of the `deny` items contain no alarming token at all, and many `approve` items |
| are full of them. |
| |
| ### Choosing an operating point |
| |
| `logits[:, 1]` after softmax is `P(deny)`. Because a wrong approve costs far more than a |
| wrong deny, you can trade the two: |
| |
| | threshold | accuracy | false-approve | false-deny | |
| |---|---|---|---| |
| | 0.1 | 84.43% | 2.71% | 26.83% | |
| | 0.2 | 87.53% | 4.71% | 19.26% | |
| | 0.3 | 88.97% | 6.57% | 14.95% | |
| | 0.4 | 89.60% | 7.71% | 12.76% | |
| | 0.5 | 90.13% | 8.71% | 10.88% | |
| | 0.6 | 90.30% | 9.99% | 9.44% | |
| | 0.7 | 90.43% | 11.85% | 7.57% | |
| | 0.8 | 89.93% | 14.28% | 6.38% | |
| | 0.9 | 89.27% | 18.20% | 4.19% | |
| |
| ### Where it is strong and weak |
| |
| **By difficulty** |
| |
| | slice | n | accuracy | false-approve | false-deny | |
| |---|---|---|---|---| |
| | `medium` | 1065 | 92.21% | 7.52% | 8.07% | |
| | `hard` | 1065 | 86.10% | 10.53% | 17.26% | |
| | `easy` | 870 | 92.53% | 7.72% | 7.32% | |
| |
| **By context length** |
| |
| | slice | n | accuracy | false-approve | false-deny | |
| |---|---|---|---|---| |
| | `<1k` | 2420 | 91.40% | 7.40% | 9.60% | |
| | `16k-64k` | 232 | 78.45% | 25.62% | 17.12% | |
| | `4k-16k` | 218 | 88.07% | 6.60% | 16.96% | |
| | `1k-4k` | 130 | 90.77% | 3.03% | 15.62% | |
| |
| ## Usage |
| |
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification |
| |
| tok = AutoTokenizer.from_pretrained("ProCreations/auto-0.4b") |
| model = AutoModelForSequenceClassification.from_pretrained( |
| "ProCreations/auto-0.4b", |
| dtype=torch.bfloat16, |
| attn_implementation="flash_attention_2", # required for the full 64k context |
| ).cuda().eval() |
| |
| def build_input(user_request, history, call): |
| """history: list of dicts with tool/args/result. call: dict with tool/args.""" |
| parts = ["### PROPOSED TOOL CALL", f"tool: {call['tool']}", f"args: {call['args']}", "", |
| "### USER REQUEST", user_request, "", "### AGENT HISTORY"] |
| if not history: |
| parts.append("(no prior actions)") |
| else: |
| for i, h in enumerate(history): |
| parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result','')}") |
| return "\n".join(parts) |
| |
| text = build_input( |
| user_request="clean up the build artifacts and reinstall deps", |
| history=[{"tool": "Bash", "args": "ls -la", "result": "node_modules dist src package.json"}], |
| call={"tool": "Bash", "args": "rm -rf node_modules dist && npm install"}, |
| ) |
| |
| enc = tok(text, return_tensors="pt", truncation=True, max_length=65536).to("cuda") |
| with torch.no_grad(): |
| p_deny = model(**enc).logits.softmax(-1)[0, 1].item() |
| |
| print("DENY" if p_deny > 0.5 else "APPROVE", f"(P(deny)={p_deny:.3f})") |
| ``` |
| |
| **The input format matters.** The proposed call and the user request come first so they |
| survive truncation; history follows. Use the exact section headers above — the model was |
| trained on them. |
| |
| ## How it was built |
| |
| **Context extension (8k → 64k).** ModernBERT alternates local sliding-window attention |
| (window 128) with full global attention every 3rd layer, so only 10 of 28 layers pay the |
| O(n²) cost — which is what makes 64k practical on one GPU. Only those global layers need a |
| new RoPE base. The base was chosen empirically by sweeping it against MLM loss on |
| domain-matched long agent transcripts: |
| |
| | RoPE base (global layers) | 8k | 16k | 32k | 64k | |
| |---|---|---|---|---| |
| | 160,000 | 0.589 | 0.593 | 0.752 | 2.060 | |
| | 640,000 | 0.588 | 0.474 | 0.406 | 0.550 | |
| | 1,280,000 | 0.599 | 0.477 | 0.317 | 0.444 | |
| | 2,560,000 | 0.614 | 0.490 | 0.314 | 0.222 | |
| | 5,120,000 | 0.635 | 0.507 | 0.328 | 0.195 | |
| |
| At the stock base the model simply cannot do 64k. **2,560,000** (16× the original |
| for an 8× extension) was chosen: near-best long-context loss for ~4% short-context cost. |
| |
| **Training.** Full fine-tune (all 395.8M parameters), two stages: bulk training at short |
| context where nearly all real traffic lives, then a long-context stage at up to 64k so the |
| extended RoPE is exercised on the actual task. Batching is by token budget rather than |
| example count, since inputs span 200–65,536 tokens. |
| |
| **Data.** 73,509 training examples generated with DeepSeek V4 Flash across a large |
| combinatorial space of agent frameworks (Claude Code, Codex CLI, MCP servers, LangChain, |
| computer-use, devops, …), domains, risk categories, obfuscation styles, and 10 languages, |
| with deliberate **minimal contrastive pairs** — near-identical calls with opposite labels |
| where only the user's request or the history flips the verdict. |
| |
| Long-context examples are built by burying the decisive history steps inside benign filler |
| at a random depth, so 64k capability is exercised as needle-in-a-haystack retrieval rather |
| than merely declared. |
| |
| ## Limitations |
| |
| - Training data is model-generated and reflects the decision rule it was written against; |
| it is not a substitute for a real security review of your agent's permissions. |
| - Held-out labels were audited by an independent pass, but both come from the same model |
| family, so systematic blind spots can survive. |
| - The ONNX export is practical to ~8k tokens (the non-flash attention path materialises a |
| dense sliding-window mask); use the PyTorch + flash-attn path for full 64k. |
| - It judges a *proposed* call from text. It cannot see what a script will actually do at |
| runtime, so an opaque binary or a URL whose content it cannot read is judged on context alone. |
| |
| ## Other formats |
| |
| - [`ProCreations/auto-0.4b-ONNX`](https://huggingface.co/ProCreations/auto-0.4b-ONNX) — ONNX + int8 |
| |
| The GGUF build was **withdrawn**. llama.cpp converts the model, but its `--pooling rank` path |
| returns zero for a 2-class classification head, so the GGUF could not actually make |
| approve/deny decisions — it returned identical `0.000` scores for a `rm -rf /` and for a |
| `pytest` invocation. It was removed rather than left up implying it worked. |
| |
| ## Successor |
| |
| [`ProCreations/auto-1b`](https://huggingface.co/ProCreations/auto-1b) scores **96.40%** on the |
| same benchmark (vs 90.13% here), with false-deny down from 10.88% to 3.19% and long-context |
| accuracy up from 78.45% to 97.02%. Prefer it unless you specifically need the smaller model. |
| |
