README.md

---
license: apache-2.0
base_model: ProCreations/auto-1b
pipeline_tag: text-classification
library_name: transformers
tags:
  - agent-safety
  - tool-calling
  - guardrails
  - modernbert
  - long-context
datasets:
  - ProCreations/auto-1b-data
  - ProCreations/approve-or-deny
language:
  - en
model-index:
  - name: auto-1b-bf16
    results:
      - task:
          type: text-classification
          name: Agentic tool-call safety gating
        dataset:
          name: Approve-or-Deny
          type: ProCreations/approve-or-deny
          split: test
        metrics:
          - type: accuracy
            value: 0.9640
            name: Accuracy
          - type: roc_auc
            value: 0.9928
            name: AUROC
          - type: false_approve_rate
            value: 0.0407
            name: False-approve rate
          - type: false_deny_rate
            value: 0.0319
            name: False-deny rate
---

# auto-1b — bf16

**Half the size of [`ProCreations/auto-1b`](https://huggingface.co/ProCreations/auto-1b), with
identical benchmark results.** 2.0 GB instead of 3.9 GB.

A 1B encoder that decides whether an AI agent's next tool call is safe to run — **96.40%** on
[approve-or-deny](https://huggingface.co/datasets/ProCreations/approve-or-deny), ahead of
DeepSeek V4 Flash and within 0.57 points of GPT-5.6-Luna, at ~10 ms per call.

## Verified lossless

Not a spot check — the **full 3,000-item benchmark** was re-run on this build:

| metric | fp32 original | **bf16 (this)** |
|---|---|---|
| accuracy | 0.964000 | **0.964000** |
| AUROC | 0.992845 | **0.992845** |
| F1 (deny) | 0.961373 | **0.961373** |
| false-approve | 0.040685 | **0.040685** |
| false-deny | 0.031895 | **0.031895** |

**Zero flipped verdicts across 3,000 items.** Identical on every context-length slice too:

| context length | n | fp32 | bf16 |
|---|---|---|---|
| <1k | 2239 | 96.78% | 96.78% |
| 1k–4k | 231 | 96.97% | 96.97% |
| 4k–16k | 194 | 90.21% | 90.21% |
| 16k–64k | 336 | 97.02% | 97.02% |

For reference, an fp16 build of the same weights scored 0.964333 accuracy — one item different
out of 3,000, with marginally *worse* AUROC (0.992839). The two half-precision formats are
equivalent in practice; bf16 is preferred here because it carries no overflow risk and matches
the dtype the model was trained in.

**Dynamic int8 is a different story and should not be used** — it flips roughly 1 verdict in 20.
See the [ONNX repo](https://huggingface.co/ProCreations/auto-1b-ONNX) for that measurement.

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("ProCreations/auto-1b-bf16")
model = AutoModelForSequenceClassification.from_pretrained(
    "ProCreations/auto-1b-bf16",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",   # required for the full 64k context
).cuda().eval()

def build_input(user_request, history, call):
    """history: list of dicts with tool/args/result. call: dict with tool/args."""
    parts = ["### PROPOSED TOOL CALL", f"tool: {call['tool']}", f"args: {call['args']}", "",
             "### USER REQUEST", user_request, "", "### AGENT HISTORY"]
    if not history:
        parts.append("(no prior actions)")
    else:
        for i, h in enumerate(history):
            parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result','')}")
    return "\n".join(parts)

text = build_input(
    user_request="clean up the build artifacts and reinstall deps",
    history=[{"tool": "Bash", "args": "ls -la", "result": "node_modules  dist  src  package.json"}],
    call={"tool": "Bash", "args": "rm -rf node_modules dist && npm install"},
)

enc = tok(text, return_tensors="pt", truncation=True, max_length=65536).to("cuda")
with torch.no_grad():
    p_deny = model(**enc).logits.softmax(-1)[0, 1].item()

print("DENY" if p_deny > 0.5 else "APPROVE", f"(P(deny)={p_deny:.3f})")
```

**The input format matters.** The proposed call and the user request come first so they survive
truncation; history follows. Use the exact section headers above — the model was trained on them.

`logits[:, 1]` after softmax is `P(deny)`. Labels: `0 = approve`, `1 = deny`.

## What it decides

- **`approve`** — routine work serving the user's request: reading files, running tests, editing
  code, committing, **web search, fetching docs, installing legitimate packages**. Network access
  is normal agent behaviour and is *not* treated as dangerous.
- **`deny`** — the call is consequential *and* unauthorised: wiping directories, exfiltrating
  credentials, installing hostile software, weakening security controls, destroying production
  state, or following instructions injected by content the agent just read.

Full documentation, per-category results, deployment guidance and limitations are on the
**[main model card](https://huggingface.co/ProCreations/auto-1b)**.