README.md

metadata

license: apache-2.0
base_model: ProCreations/auto-1b
pipeline_tag: text-classification
library_name: transformers
tags:
  - agent-safety
  - tool-calling
  - guardrails
  - modernbert
  - long-context
datasets:
  - ProCreations/auto-1b-data
  - ProCreations/approve-or-deny
language:
  - en
model-index:
  - name: auto-1b-bf16
    results:
      - task:
          type: text-classification
          name: Agentic tool-call safety gating
        dataset:
          name: Approve-or-Deny
          type: ProCreations/approve-or-deny
          split: test
        metrics:
          - type: accuracy
            value: 0.964
            name: Accuracy
          - type: roc_auc
            value: 0.9928
            name: AUROC
          - type: false_approve_rate
            value: 0.0407
            name: False-approve rate
          - type: false_deny_rate
            value: 0.0319
            name: False-deny rate

auto-1b — bf16

Half the size of ProCreations/auto-1b, with identical benchmark results. 2.0 GB instead of 3.9 GB.

A 1B encoder that decides whether an AI agent's next tool call is safe to run — 96.40% on approve-or-deny, ahead of DeepSeek V4 Flash and within 0.57 points of GPT-5.6-Luna, at ~10 ms per call.

Verified lossless

Not a spot check — the full 3,000-item benchmark was re-run on this build:

metric	fp32 original	bf16 (this)
accuracy	0.964000	0.964000
AUROC	0.992845	0.992845
F1 (deny)	0.961373	0.961373
false-approve	0.040685	0.040685
false-deny	0.031895	0.031895

Zero flipped verdicts across 3,000 items. Identical on every context-length slice too:

context length	n	fp32	bf16
<1k	2239	96.78%	96.78%
1k–4k	231	96.97%	96.97%
4k–16k	194	90.21%	90.21%
16k–64k	336	97.02%	97.02%

For reference, an fp16 build of the same weights scored 0.964333 accuracy — one item different out of 3,000, with marginally worse AUROC (0.992839). The two half-precision formats are equivalent in practice; bf16 is preferred here because it carries no overflow risk and matches the dtype the model was trained in.

Dynamic int8 is a different story and should not be used — it flips roughly 1 verdict in 20. See the ONNX repo for that measurement.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("ProCreations/auto-1b-bf16")
model = AutoModelForSequenceClassification.from_pretrained(
    "ProCreations/auto-1b-bf16",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",   # required for the full 64k context
).cuda().eval()

def build_input(user_request, history, call):
    """history: list of dicts with tool/args/result. call: dict with tool/args."""
    parts = ["### PROPOSED TOOL CALL", f"tool: {call['tool']}", f"args: {call['args']}", "",
             "### USER REQUEST", user_request, "", "### AGENT HISTORY"]
    if not history:
        parts.append("(no prior actions)")
    else:
        for i, h in enumerate(history):
            parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result','')}")
    return "\n".join(parts)

text = build_input(
    user_request="clean up the build artifacts and reinstall deps",
    history=[{"tool": "Bash", "args": "ls -la", "result": "node_modules  dist  src  package.json"}],
    call={"tool": "Bash", "args": "rm -rf node_modules dist && npm install"},
)

enc = tok(text, return_tensors="pt", truncation=True, max_length=65536).to("cuda")
with torch.no_grad():
    p_deny = model(**enc).logits.softmax(-1)[0, 1].item()

print("DENY" if p_deny > 0.5 else "APPROVE", f"(P(deny)={p_deny:.3f})")

The input format matters. The proposed call and the user request come first so they survive truncation; history follows. Use the exact section headers above — the model was trained on them.

logits[:, 1] after softmax is P(deny). Labels: 0 = approve, 1 = deny.

What it decides

• approve — routine work serving the user's request: reading files, running tests, editing code, committing, web search, fetching docs, installing legitimate packages. Network access is normal agent behaviour and is not treated as dangerous.
• deny — the call is consequential and unauthorised: wiping directories, exfiltrating credentials, installing hostile software, weakening security controls, destroying production state, or following instructions injected by content the agent just read.

Full documentation, per-category results, deployment guidance and limitations are on the main model card.