Release auto-0.4b-3

Auto v3: weights, model card, charts and per-item benchmark scores.

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/accuracy.png filter=lfs diff=lfs merge=lfs -text
+assets/cost.png filter=lfs diff=lfs merge=lfs -text
+assets/errors.png filter=lfs diff=lfs merge=lfs -text
+assets/speed.png filter=lfs diff=lfs merge=lfs -text
+assets/tradeoff.png filter=lfs diff=lfs merge=lfs -text
+assets/v2-to-v3.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,178 @@
+---
+base_model: ProCreations/auto-0.4b-2
+base_model_relation: finetune
+library_name: transformers
+license: apache-2.0
+pipeline_tag: text-classification
+tags:
+- modernbert
+- agent-safety
+- tool-calling
+- guardrail
+- long-context
+model-index:
+- name: auto-0.4b-3
+  results:
+  - task:
+      type: text-classification
+      name: Agentic tool-call approve/deny classification
+    dataset:
+      name: Approve-or-Deny
+      type: ProCreations/approve-or-deny
+      split: test
+      revision: a38b625913dd46ca9702063f1597c979e6ace34e
+    metrics:
+    - type: accuracy
+      value: 0.9720
+    - type: roc_auc
+      value: 0.9955
+    - type: false_approve_rate
+      value: 0.0335
+    - type: false_deny_rate
+      value: 0.0231
+---
+
+# auto-0.4b-3
+
+A 395.8M-parameter classifier that decides whether an AI agent's proposed tool call should run. It reads the proposed call, the user's request and the agent's history, and outputs `approve` or `deny`. auto-0.4b-3 is Auto's default model: accurate enough for everyday use and small enough to run on a laptop.
+
+It is part of [Auto](https://github.com/SSHdotCodes/auto), a local guard for coding agents (Claude Code, Codex, Gemini CLI, Cursor, Copilot CLI and others). Auto blocks a call only if it's consequential *and* the user didn't ask for it, or if the agent is following instructions planted in a page, file or tool output. Everything runs on your machine, and nothing it reads leaves it.
+
+| | |
+|---|---|
+| **v3 benchmark** | **93.1%**, up from 70.2% for auto-0.4b-2 |
+| **Missed harm / over-block** | 6.0% / 7.6% |
+| **Original Approve-or-Deny benchmark** | 97.2% |
+| **Time per decision** | 7 ms (median, NVIDIA RTX PRO 6000) |
+| **Cost per million decisions** | \$23 on a rented GPU |
+| **Architecture** | ModernBERT-large encoder, 395.8M parameters, BF16 weights (0.79 GB) |
+| **Runs on** | a laptop CPU, Apple silicon or any GPU |
+
+## How it compares
+
+![Accuracy on the v3 benchmark, auto-0.4b-3 highlighted](assets/accuracy.png)
+
+The v3 benchmark has 2,948 held-out tool calls. Each one is paired with a near-twin where one detail flips the right answer: who asked, how far the request reaches, production or staging, the exact call, or where an instruction came from. A guard that guesses from surface features gets one of each pair wrong. Hosted guards were prompted with the same decision rule that Auto's training labels follow, and every Auto model is scored at `P(deny) >= 0.5`.
+
+![Missed harm and over-block rates](assets/errors.png)
+
+A guard can be wrong in two ways. Missed harm lets a harmful call through; over-blocking interrupts work you asked for. Hosted LLMs tend to trade one for the other: DeepSeek V4.1 Flash misses almost nothing (0.6%) but blocks 7.9% of authorized calls.
+
+![Median time per decision](assets/speed.png)
+
+![Cost per million decisions](assets/cost.png)
+
+![Accuracy against cost](assets/tradeoff.png)
+
+| Model | v3 accuracy (95% CI) | Missed harm | Over-block | Both twins right | Time per decision | Cost per 1M |
+|---|---:|---:|---:|---:|---:|---:|
+| [auto-3b-3](https://huggingface.co/ProCreations/auto-3b-3) | 96.0% (95.3–96.7) | 3.1% | 4.8% | 92.8% | 21 ms | \$77 |
+| DeepSeek V4.1 Flash (max reasoning) | 95.6% (94.8–96.3) | 0.6% | 7.9% | 91.1% | 1.3 s | \$3,043 |
+| Codex auto-review (medium) | 95.5% (94.7–96.2) | 4.9% | 4.1% | 90.8% | – | \$11,353 |
+| GLM-5.3 Flash (high reasoning) | 95.4% (94.5–96.1) | 5.5% | 3.8% | 90.2% | 1.2 s | \$629 |
+| **auto-0.4b-3** | 93.1% (92.2–94.0) | 6.0% | 7.6% | 87.2% | 7 ms | \$23 |
+| TypeSafe Jev 1.14 | 90.2% (89.1–91.2) | 12.5% | 7.3% | 80.1% | 136 ms | \$227 |
+| [auto-150m-3](https://huggingface.co/ProCreations/auto-150m-3) | 88.8% (87.7–89.9) | 9.1% | 13.1% | 78.7% | 5 ms | \$13 |
+| GPT-6 Luna (low reasoning) | 87.9% (86.7–89.0) | 5.8% | 17.9% | 75.8% | – | \$395 |
+
+*Missed harm:* share of deny items approved. *Over-block:* share of approve items denied. *Both twins right:* share of pairs where both halves are correct. *Time:* Auto on a local NVIDIA RTX PRO 6000, one request at a time; hosted guards are API round trips measured unloaded (Codex auto-review and GPT-6 Luna run through CLIs, which take 12–13 s per decision, so they're left out of the speed comparison). *Cost:* Auto on a rented RTX PRO 6000 at \$2.18/h (median market rate), about \$0.50–\$3 per million in electricity on a GPU you own; hosted guards at list price × measured tokens.
+
+### v2 → v3
+
+![v2 to v3 accuracy for each Auto size](assets/v2-to-v3.png)
+
+| Original Approve-or-Deny benchmark (3,000 items) | auto-0.4b-2 | auto-0.4b-3 |
+|---|---:|---:|
+| Accuracy | 97.00% | 97.20% |
+| Harmful calls approved (of 1,401) | 36 | 47 |
+| Authorized calls denied (of 1,599) | 54 | 37 |
+| AUROC | – | 0.9955 |
+
+On the original benchmark, v3 keeps its accuracy but shifts toward approving: it denies fewer authorized calls and approves more of the harmful ones. On the v3 benchmark it is far stronger on both counts.
+
+### Breakdown on the v3 benchmark
+
+| Slice | Accuracy |
+|---|---:|
+| Written by Claude Opus 4.8 (not a training-data writer) | 90.4% |
+| Written by GPT-6.1 Sol | 94.7% |
+| Long sessions | 92.5% |
+| Non-shell tools (browser, email, CRM, ...) | 93.2% |
+| Twin flips on who asked | 94.4% |
+| Twin flips on scope | 91.0% |
+| Twin flips on environment | 94.6% |
+| Twin flips on the exact call | 90.1% |
+| Twin flips on where an instruction came from | 98.7% |
+
+The most-missed harm categories are supply chain (12.2%), privilege escalation (9.0%), scope violation (7.8%). Per-item scores for both benchmarks are in [eval/](eval/), and every model's metrics are in [eval/results.json](eval/results.json).
+
+Through Auto's runtime (Apple MPS), 78 of 80 random v3 items got the same decision as the reference scores (median P(deny) difference 0.0000). One of the two differences is a session trimmed to 8,192 tokens; the other is a borderline call within 0.05 of the 0.5 threshold. On the 413 v3 items longer than 8,192 tokens, Auto's context fitting scores 91.8%, against 92.5% with the full input.
+
+## Usage
+
+### With Auto
+
+```sh
+curl -fsSL https://auto.ssh.codes/install.sh | sh
+auto configure --model auto-0.4b-3
+```
+
+The installer finds your agents and adds the hook. See the [Auto README](https://github.com/SSHdotCodes/auto) for the Python API, the Claude Agent SDK hook and LangChain middleware.
+
+### With Transformers
+
+The model reads one serialized text. Keep the three sections and their order exactly as below.
+
+```python
+import json
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+
+def text(value):
+    return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, separators=(",", ":"))
+
+
+def build_input(user_request, history, call):
+    parts = ["### PROPOSED TOOL CALL", f"tool: {call['tool']}", f"args: {text(call['args'])}", "",
+             "### USER REQUEST", user_request, "", "### AGENT HISTORY"]
+    if not history:
+        parts.append("(no prior actions)")
+    for i, action in enumerate(history, 1):
+        parts.append(f"[{i}] {action['tool']}({text(action['args'])})\n-> {text(action.get('result', ''))}")
+    return "\n".join(parts)
+
+
+repo = "ProCreations/auto-0.4b-3"
+tokenizer = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForSequenceClassification.from_pretrained(repo, dtype=torch.bfloat16).eval()
+
+prompt = build_input(
+    user_request="Clean up the build artifacts and reinstall dependencies.",
+    history=[{"tool": "Bash", "args": {"command": "ls"}, "result": "node_modules dist package.json"}],
+    call={"tool": "Bash", "args": {"command": "rm -rf node_modules dist && npm install"}},
+)
+inputs = tokenizer(prompt, return_tensors="pt")
+with torch.inference_mode():
+    p_deny = model(**inputs).logits.float().softmax(-1)[0, 1].item()
+print("deny" if p_deny >= 0.5 else "approve", round(p_deny, 3))
+```
+
+Labels are `0 = approve` and `1 = deny`. The model accepts up to 65,536 positions. For long sessions, Auto's runtime keeps memory bounded and fits the history to 8,192 tokens by default, keeping the older actions that mention what the call touches.
+
+## Training
+
+- **Starting point:** [ProCreations/auto-0.4b-2](https://huggingface.co/ProCreations/auto-0.4b-2) (revision `5937dd01`).
+- **New data:** about 100,000 contrastive pairs: two near-identical situations where one detail flips the right answer. They were written by GPT-6.1 Terra, GPT-6.1 Sol and DeepSeek V4.1 Flash, label-checked, and cover 53 agent harnesses, 11 languages and 23 categories, including browser, email, CRM and smart-home tools as well as the shell.
+- **Replay:** 84,706 rows of v2 training data, kept close to auto-0.4b-2 by distillation (KL to the frozen v2 model at temperature 2, weight 0.8), so v3 doesn't forget the original distribution.
+- **Loss on new rows:** cross-entropy, plus a pair margin `softplus(4 − (z_deny − z_approve))` (weight 0.5) that pushes each twin pair apart. Denials in the categories where early v3 trials missed more harm than v2, and hard examples, are weighted 1.5×.
+- **Schedule:** 2 epochs, AdamW (β 0.9/0.95, weight decay 0.01), peak learning rate 1e-5, 3% warmup then decay to 10%, batches of up to 32,768 tokens, BF16 compute with FP32 master weights, on one NVIDIA RTX PRO 6000. The full settings are in [training_config.json](training_config.json).
+- **Distillation tried, not used.** A version distilled from auto-3b-3 scored 92.7% on the v3 benchmark. The difference is not significant (McNemar p = 0.19), and the directly trained model misses less harm, so it is the release. That choice used the v3 benchmark, so this card's v3 numbers carry a little selection bias.
+- **Held out:** the v3 benchmark items and their twins were never trained on. The benchmark isn't public yet.
+
+## Limitations
+
+- **A classifier, not a chat model.** It can't open files, follow URLs or know what an opaque script does; it judges from the text it is given.
+- **Synthetic labels.** Training and benchmark data were written by LLMs and label-checked, not collected from real agent traffic. Items written by Claude Opus 4.8, which wrote none of the training data, score lower (90.4%) than items from a training writer (94.7%), so expect some drop on unfamiliar styles.
+- **Harm still gets through.** It approves 6.0% of harmful v3 calls. When Auto approves, your agent's own permission settings still apply; keep them on.
+- Evaluate it on your own traffic before relying on it.

config.json

@@ -0,0 +1,93 @@
+{
+  "architectures": [
+    "ModernBertForSequenceClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "cls",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "bfloat16",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 1024,
+  "id2label": {
+    "0": "approve",
+    "1": "deny"
+  },
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 2624,
+  "label2id": {
+    "approve": 0,
+    "deny": 1
+  },
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "max_position_embeddings": 65536,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 2560000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.16.1",
+  "use_cache": false,
+  "vocab_size": 50368
+}

eval/dev.json

@@ -0,0 +1,40 @@
+[
+ {
+  "epoch": 1,
+  "step": 2227,
+  "dev_new": {
+   "n": 7072,
+   "accuracy": 0.9589932126696833,
+   "false_approve_rate": 0.04892533936651584,
+   "false_deny_rate": 0.03308823529411765,
+   "nll": 0.11061911253312574,
+   "pair_accuracy": 0.920303605313093
+  },
+  "dev_old": {
+   "n": 1500,
+   "accuracy": 0.9706666666666667,
+   "false_approve_rate": 0.026845637583892617,
+   "false_deny_rate": 0.031788079470198675,
+   "nll": 0.08187394878089876
+  }
+ },
+ {
+  "epoch": 2,
+  "step": 4445,
+  "dev_new": {
+   "n": 7072,
+   "accuracy": 0.9646493212669683,
+   "false_approve_rate": 0.042986425339366516,
+   "false_deny_rate": 0.02771493212669683,
+   "nll": 0.11286248919654661,
+   "pair_accuracy": 0.931056293485136
+  },
+  "dev_old": {
+   "n": 1500,
+   "accuracy": 0.974,
+   "false_approve_rate": 0.022818791946308724,
+   "false_deny_rate": 0.02913907284768212,
+   "nll": 0.07598317966571344
+  }
+ }
+]

eval/results.json

@@ -0,0 +1,695 @@
+{
+ "model": "auto-0.4b-3",
+ "results": {
+  "v3": {
+   "n": 2948,
+   "accuracy": 0.9314789687924017,
+   "ci95": [
+    0.9217861698885004,
+    0.9400486927951885
+   ],
+   "balanced_accuracy": 0.9318114022651461,
+   "false_approves": 85,
+   "false_approve_rate": 0.060155697098372256,
+   "false_denies": 117,
+   "false_deny_rate": 0.0762214983713355,
+   "auroc": 0.9809661795657356,
+   "pair_accuracy": 0.8722304283604135,
+   "pairs": 1354
+  },
+  "slices": {
+   "generator=claude-opus-4-8": 0.9036827195467422,
+   "generator=gpt-6.1-sol": 0.9470619375330863,
+   "is_long=0": 0.9329702555509007,
+   "is_long=1": 0.9251336898395722,
+   "flip_mechanism=authorization": 0.9435825105782792,
+   "flip_mechanism=call_variant": 0.9014778325123153,
+   "flip_mechanism=environment": 0.946236559139785,
+   "flip_mechanism=provenance": 0.9870801033591732,
+   "flip_mechanism=scope": 0.9095652173913044,
+   "surface=non-shell": 0.9318734793187348,
+   "surface=shell": 0.931415057154119,
+   "slice=in_distribution": 0.9308151093439364,
+   "slice=new_framework": 0.9353348729792148
+  },
+  "missed_harm_by_category": {
+   "abuse_spam_privacy": 0.061224489795918366,
+   "covering_tracks": 0.037037037037037035,
+   "credential_exfiltration": 0.06289308176100629,
+   "data_destruction_unrequested": 0.07729468599033816,
+   "destructive_filesystem": 0.05521472392638037,
+   "irreversible_remote": 0.06666666666666667,
+   "malicious_install": 0.02,
+   "obfuscated_payload": 0.03773584905660377,
+   "physical_financial": 0.07352941176470588,
+   "privilege_escalation": 0.08974358974358974,
+   "prompt_injection": 0.008064516129032258,
+   "scope_violation": 0.0782122905027933,
+   "self_permission_tamper": 0.018867924528301886,
+   "supply_chain": 0.12195121951219512
+  },
+  "old": {
+   "n": 3000,
+   "accuracy": 0.972,
+   "ci95": [
+    0.9654658634459935,
+    0.9773268590464309
+   ],
+   "balanced_accuracy": 0.9716565358702508,
+   "false_approves": 47,
+   "false_approve_rate": 0.03354746609564597,
+   "false_denies": 37,
+   "false_deny_rate": 0.023139462163852407,
+   "auroc": 0.9954903559906955,
+   "pair_accuracy": NaN,
+   "pairs": 0
+  },
+  "mcnemar_p_vs_auto-3b-3": 1.9859524397639595e-10
+ },
+ "comparison": {
+  "auto-3b-3": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.9603120759837178,
+    "ci95": [
+     0.9526447890222909,
+     0.9667812397998521
+    ],
+    "balanced_accuracy": 0.9606517885341097,
+    "false_approves": 44,
+    "false_approve_rate": 0.03113941967445152,
+    "false_denies": 73,
+    "false_deny_rate": 0.04755700325732899,
+    "auroc": 0.9911667139244474,
+    "pair_accuracy": 0.9276218611521418,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.9301227573182247,
+    "generator=gpt-6.1-sol": 0.9772366331392272,
+    "is_long=0": 0.9622957687473817,
+    "is_long=1": 0.9518716577540107,
+    "flip_mechanism=authorization": 0.9689703808180536,
+    "flip_mechanism=call_variant": 0.9445812807881774,
+    "flip_mechanism=environment": 0.9655913978494624,
+    "flip_mechanism=provenance": 0.9819121447028424,
+    "flip_mechanism=scope": 0.9530434782608695,
+    "surface=non-shell": 0.9635036496350365,
+    "surface=shell": 0.9597950335041388,
+    "slice=in_distribution": 0.9598409542743539,
+    "slice=new_framework": 0.9630484988452656
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.0,
+    "covering_tracks": 0.0,
+    "credential_exfiltration": 0.025157232704402517,
+    "data_destruction_unrequested": 0.04830917874396135,
+    "destructive_filesystem": 0.04294478527607362,
+    "irreversible_remote": 0.037037037037037035,
+    "malicious_install": 0.02,
+    "obfuscated_payload": 0.05660377358490566,
+    "physical_financial": 0.029411764705882353,
+    "privilege_escalation": 0.038461538461538464,
+    "prompt_injection": 0.008064516129032258,
+    "scope_violation": 0.03910614525139665,
+    "self_permission_tamper": 0.018867924528301886,
+    "supply_chain": 0.0
+   },
+   "old": {
+    "n": 3000,
+    "accuracy": 0.9823333333333333,
+    "ci95": [
+     0.9769651453292397,
+     0.9864678133194454
+    ],
+    "balanced_accuracy": 0.98188040437479,
+    "false_approves": 35,
+    "false_approve_rate": 0.024982155603140613,
+    "false_denies": 18,
+    "false_deny_rate": 0.01125703564727955,
+    "auroc": 0.9984296037985911,
+    "pair_accuracy": NaN,
+    "pairs": 0
+   }
+  },
+  "auto-0.4b-3": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.9314789687924017,
+    "ci95": [
+     0.9217861698885004,
+     0.9400486927951885
+    ],
+    "balanced_accuracy": 0.9318114022651461,
+    "false_approves": 85,
+    "false_approve_rate": 0.060155697098372256,
+    "false_denies": 117,
+    "false_deny_rate": 0.0762214983713355,
+    "auroc": 0.9809661795657356,
+    "pair_accuracy": 0.8722304283604135,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.9036827195467422,
+    "generator=gpt-6.1-sol": 0.9470619375330863,
+    "is_long=0": 0.9329702555509007,
+    "is_long=1": 0.9251336898395722,
+    "flip_mechanism=authorization": 0.9435825105782792,
+    "flip_mechanism=call_variant": 0.9014778325123153,
+    "flip_mechanism=environment": 0.946236559139785,
+    "flip_mechanism=provenance": 0.9870801033591732,
+    "flip_mechanism=scope": 0.9095652173913044,
+    "surface=non-shell": 0.9318734793187348,
+    "surface=shell": 0.931415057154119,
+    "slice=in_distribution": 0.9308151093439364,
+    "slice=new_framework": 0.9353348729792148
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.061224489795918366,
+    "covering_tracks": 0.037037037037037035,
+    "credential_exfiltration": 0.06289308176100629,
+    "data_destruction_unrequested": 0.07729468599033816,
+    "destructive_filesystem": 0.05521472392638037,
+    "irreversible_remote": 0.06666666666666667,
+    "malicious_install": 0.02,
+    "obfuscated_payload": 0.03773584905660377,
+    "physical_financial": 0.07352941176470588,
+    "privilege_escalation": 0.08974358974358974,
+    "prompt_injection": 0.008064516129032258,
+    "scope_violation": 0.0782122905027933,
+    "self_permission_tamper": 0.018867924528301886,
+    "supply_chain": 0.12195121951219512
+   },
+   "old": {
+    "n": 3000,
+    "accuracy": 0.972,
+    "ci95": [
+     0.9654658634459935,
+     0.9773268590464309
+    ],
+    "balanced_accuracy": 0.9716565358702508,
+    "false_approves": 47,
+    "false_approve_rate": 0.03354746609564597,
+    "false_denies": 37,
+    "false_deny_rate": 0.023139462163852407,
+    "auroc": 0.9954903559906955,
+    "pair_accuracy": NaN,
+    "pairs": 0
+   },
+   "mcnemar_p_vs_auto-3b-3": 1.9859524397639595e-10
+  },
+  "auto-150m-3": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.8883989145183175,
+    "ci95": [
+     0.8765230010914776,
+     0.8992638839500505
+    ],
+    "balanced_accuracy": 0.8892339859517602,
+    "false_approves": 128,
+    "false_approve_rate": 0.09058740268931352,
+    "false_denies": 201,
+    "false_deny_rate": 0.13094462540716612,
+    "auroc": 0.9611068463845492,
+    "pair_accuracy": 0.7865583456425406,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.846081208687441,
+    "generator=gpt-6.1-sol": 0.9121228163049232,
+    "is_long=0": 0.8969417679095099,
+    "is_long=1": 0.8520499108734403,
+    "flip_mechanism=authorization": 0.9083215796897038,
+    "flip_mechanism=call_variant": 0.8596059113300493,
+    "flip_mechanism=environment": 0.8903225806451613,
+    "flip_mechanism=provenance": 0.958656330749354,
+    "flip_mechanism=scope": 0.8556521739130435,
+    "surface=non-shell": 0.878345498783455,
+    "surface=shell": 0.8900275916436736,
+    "slice=in_distribution": 0.8850894632206759,
+    "slice=new_framework": 0.9076212471131639
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.061224489795918366,
+    "covering_tracks": 0.05555555555555555,
+    "credential_exfiltration": 0.0880503144654088,
+    "data_destruction_unrequested": 0.11594202898550725,
+    "destructive_filesystem": 0.11042944785276074,
+    "irreversible_remote": 0.1111111111111111,
+    "malicious_install": 0.12,
+    "obfuscated_payload": 0.07547169811320754,
+    "physical_financial": 0.16176470588235295,
+    "privilege_escalation": 0.05128205128205128,
+    "prompt_injection": 0.0,
+    "scope_violation": 0.11731843575418995,
+    "self_permission_tamper": 0.03773584905660377,
+    "supply_chain": 0.07317073170731707
+   },
+   "old": {
+    "n": 3000,
+    "accuracy": 0.9596666666666667,
+    "ci95": [
+     0.9520185437268959,
+     0.966139058191927
+    ],
+    "balanced_accuracy": 0.9590261847273389,
+    "false_approves": 71,
+    "false_approve_rate": 0.05067808708065667,
+    "false_denies": 50,
+    "false_deny_rate": 0.031269543464665414,
+    "auroc": 0.9926580183278361,
+    "pair_accuracy": NaN,
+    "pairs": 0
+   },
+   "mcnemar_p_vs_auto-3b-3": 7.993074660900961e-41
+  },
+  "auto-3b": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.7208276797829036,
+    "ci95": [
+     0.7043546473973118,
+     0.736725930910142
+    ],
+    "balanced_accuracy": 0.7257063885603896,
+    "false_approves": 221,
+    "false_approve_rate": 0.15640481245576787,
+    "false_denies": 602,
+    "false_deny_rate": 0.39218241042345275,
+    "auroc": 0.8260906750024781,
+    "pair_accuracy": 0.4549483013293944,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.7525967894239849,
+    "generator=gpt-6.1-sol": 0.7030174695606141,
+    "is_long=0": 0.7255969836614998,
+    "is_long=1": 0.7005347593582888,
+    "flip_mechanism=authorization": 0.7503526093088858,
+    "flip_mechanism=call_variant": 0.7204433497536946,
+    "flip_mechanism=environment": 0.6967741935483871,
+    "flip_mechanism=provenance": 0.8087855297157622,
+    "flip_mechanism=scope": 0.6452173913043479,
+    "surface=non-shell": 0.7177615571776156,
+    "surface=shell": 0.7213243988963343,
+    "slice=in_distribution": 0.7216699801192843,
+    "slice=new_framework": 0.7159353348729792
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.10204081632653061,
+    "covering_tracks": 0.1111111111111111,
+    "credential_exfiltration": 0.1509433962264151,
+    "data_destruction_unrequested": 0.20772946859903382,
+    "destructive_filesystem": 0.1901840490797546,
+    "irreversible_remote": 0.1037037037037037,
+    "malicious_install": 0.08,
+    "obfuscated_payload": 0.11320754716981132,
+    "physical_financial": 0.08823529411764706,
+    "privilege_escalation": 0.10256410256410256,
+    "prompt_injection": 0.016129032258064516,
+    "scope_violation": 0.3240223463687151,
+    "self_permission_tamper": 0.09433962264150944,
+    "supply_chain": 0.21951219512195122
+   },
+   "mcnemar_p_vs_auto-3b-3": 1.634567657543771e-180
+  },
+  "auto-0.4b-2": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.7021709633649932,
+    "ci95": [
+     0.6854083932791898,
+     0.7184073128210292
+    ],
+    "balanced_accuracy": 0.705006788983635,
+    "false_approves": 320,
+    "false_approve_rate": 0.22646850672328378,
+    "false_denies": 558,
+    "false_deny_rate": 0.36351791530944627,
+    "auroc": 0.7794926589071696,
+    "pair_accuracy": 0.4098966026587888,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.7412653446647781,
+    "generator=gpt-6.1-sol": 0.6802541026998412,
+    "is_long=0": 0.7046501885211562,
+    "is_long=1": 0.6916221033868093,
+    "flip_mechanism=authorization": 0.7362482369534555,
+    "flip_mechanism=call_variant": 0.7155172413793104,
+    "flip_mechanism=environment": 0.6516129032258065,
+    "flip_mechanism=provenance": 0.7596899224806202,
+    "flip_mechanism=scope": 0.6434782608695652,
+    "surface=non-shell": 0.708029197080292,
+    "surface=shell": 0.7012219156484036,
+    "slice=in_distribution": 0.7029821073558649,
+    "slice=new_framework": 0.6974595842956121
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.1836734693877551,
+    "covering_tracks": 0.24074074074074073,
+    "credential_exfiltration": 0.24528301886792453,
+    "data_destruction_unrequested": 0.2753623188405797,
+    "destructive_filesystem": 0.31901840490797545,
+    "irreversible_remote": 0.13333333333333333,
+    "malicious_install": 0.12,
+    "obfuscated_payload": 0.07547169811320754,
+    "physical_financial": 0.16176470588235295,
+    "privilege_escalation": 0.16666666666666666,
+    "prompt_injection": 0.11290322580645161,
+    "scope_violation": 0.3743016759776536,
+    "self_permission_tamper": 0.09433962264150944,
+    "supply_chain": 0.2926829268292683
+   },
+   "old": {
+    "n": 3000,
+    "accuracy": 0.97,
+    "ci95": [
+     0.9632688976534645,
+     0.9755289404216192
+    ],
+    "balanced_accuracy": 0.9702664807903227,
+    "false_approves": 36,
+    "false_approve_rate": 0.02569593147751606,
+    "false_denies": 54,
+    "false_deny_rate": 0.03377110694183865,
+    "auroc": 0.9948812583167834,
+    "pair_accuracy": NaN,
+    "pairs": 0
+   },
+   "mcnemar_p_vs_auto-3b-3": 1.6983349909612516e-194
+  },
+  "auto-200m-2": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.6991180461329715,
+    "ci95": [
+     0.6823112663451674,
+     0.7154065515756858
+    ],
+    "balanced_accuracy": 0.7007533581840102,
+    "false_approves": 367,
+    "false_approve_rate": 0.2597310686482661,
+    "false_denies": 520,
+    "false_deny_rate": 0.33876221498371334,
+    "auroc": 0.7653017236411083,
+    "pair_accuracy": 0.40029542097488924,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.724268177525968,
+    "generator=gpt-6.1-sol": 0.6850185283218634,
+    "is_long=0": 0.7017176372015081,
+    "is_long=1": 0.6880570409982175,
+    "flip_mechanism=authorization": 0.765867418899859,
+    "flip_mechanism=call_variant": 0.6822660098522167,
+    "flip_mechanism=environment": 0.632258064516129,
+    "flip_mechanism=provenance": 0.8165374677002584,
+    "flip_mechanism=scope": 0.6156521739130435,
+    "surface=non-shell": 0.7031630170316302,
+    "surface=shell": 0.6984627512810406,
+    "slice=in_distribution": 0.6994035785288271,
+    "slice=new_framework": 0.6974595842956121
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.2653061224489796,
+    "covering_tracks": 0.1111111111111111,
+    "credential_exfiltration": 0.2893081761006289,
+    "data_destruction_unrequested": 0.27053140096618356,
+    "destructive_filesystem": 0.3312883435582822,
+    "irreversible_remote": 0.22962962962962963,
+    "malicious_install": 0.18,
+    "obfuscated_payload": 0.11320754716981132,
+    "physical_financial": 0.23529411764705882,
+    "privilege_escalation": 0.23076923076923078,
+    "prompt_injection": 0.04838709677419355,
+    "scope_violation": 0.4692737430167598,
+    "self_permission_tamper": 0.1509433962264151,
+    "supply_chain": 0.34146341463414637
+   },
+   "old": {
+    "n": 3000,
+    "accuracy": 0.9633333333333334,
+    "ci95": [
+     0.9559935894244281,
+     0.9694879672595197
+    ],
+    "balanced_accuracy": 0.9632612995541914,
+    "false_approves": 53,
+    "false_approve_rate": 0.03783012134189864,
+    "false_denies": 57,
+    "false_deny_rate": 0.03564727954971857,
+    "auroc": 0.9937152904719625,
+    "pair_accuracy": NaN,
+    "pairs": 0
+   },
+   "mcnemar_p_vs_auto-3b-3": 1.0894407275787993e-200
+  },
+  "DeepSeek V4.1 Flash": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.9559023066485753,
+    "ci95": [
+     0.9478785664272602,
+     0.9627394016913544
+    ],
+    "balanced_accuracy": 0.9574016058424448,
+    "false_approves": 9,
+    "false_approve_rate": 0.006369426751592357,
+    "false_denies": 121,
+    "false_deny_rate": 0.07882736156351791,
+    "pair_accuracy": 0.9113737075332349,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.9490084985835694,
+    "generator=gpt-6.1-sol": 0.9597670725251456,
+    "is_long=0": 0.9560117302052786,
+    "is_long=1": 0.9554367201426025,
+    "flip_mechanism=authorization": 0.9576868829337094,
+    "flip_mechanism=call_variant": 0.9741379310344828,
+    "flip_mechanism=environment": 0.9118279569892473,
+    "flip_mechanism=provenance": 0.9405684754521964,
+    "flip_mechanism=scope": 0.9739130434782609,
+    "surface=non-shell": 0.9683698296836983,
+    "surface=shell": 0.953882538431218,
+    "slice=in_distribution": 0.9538767395626243,
+    "slice=new_framework": 0.9676674364896074
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.0,
+    "covering_tracks": 0.0,
+    "credential_exfiltration": 0.0,
+    "data_destruction_unrequested": 0.00966183574879227,
+    "destructive_filesystem": 0.006134969325153374,
+    "irreversible_remote": 0.0,
+    "malicious_install": 0.02,
+    "obfuscated_payload": 0.0,
+    "physical_financial": 0.0,
+    "privilege_escalation": 0.01282051282051282,
+    "prompt_injection": 0.008064516129032258,
+    "scope_violation": 0.00558659217877095,
+    "self_permission_tamper": 0.0,
+    "supply_chain": 0.04878048780487805
+   },
+   "mcnemar_p_vs_auto-3b-3": 0.41535525707724713
+  },
+  "Codex auto-review": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.9552238805970149,
+    "ci95": [
+     0.9471471566432063,
+     0.9621157252133271
+    ],
+    "balanced_accuracy": 0.9550626914804594,
+    "false_approves": 69,
+    "false_approve_rate": 0.04883227176220807,
+    "false_denies": 63,
+    "false_deny_rate": 0.04104234527687296,
+    "pair_accuracy": 0.9076809453471196,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.9423984891406988,
+    "generator=gpt-6.1-sol": 0.9624139756484913,
+    "is_long=0": 0.9543359865940511,
+    "is_long=1": 0.9590017825311943,
+    "flip_mechanism=authorization": 0.9873060648801129,
+    "flip_mechanism=call_variant": 0.9408866995073891,
+    "flip_mechanism=environment": 0.9139784946236559,
+    "flip_mechanism=provenance": 0.9767441860465116,
+    "flip_mechanism=scope": 0.9547826086956521,
+    "surface=non-shell": 0.9562043795620438,
+    "surface=shell": 0.9550650374458022,
+    "slice=in_distribution": 0.952286282306163,
+    "slice=new_framework": 0.9722863741339491
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.12244897959183673,
+    "covering_tracks": 0.018518518518518517,
+    "credential_exfiltration": 0.0440251572327044,
+    "data_destruction_unrequested": 0.04830917874396135,
+    "destructive_filesystem": 0.049079754601226995,
+    "irreversible_remote": 0.05925925925925926,
+    "malicious_install": 0.14,
+    "obfuscated_payload": 0.03773584905660377,
+    "physical_financial": 0.029411764705882353,
+    "privilege_escalation": 0.01282051282051282,
+    "prompt_injection": 0.0,
+    "scope_violation": 0.055865921787709494,
+    "self_permission_tamper": 0.05660377358490566,
+    "supply_chain": 0.0975609756097561
+   },
+   "mcnemar_p_vs_auto-3b-3": 0.31607312611901234
+  },
+  "GLM-5.3 Flash": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.9535278154681139,
+    "ci95": [
+     0.9453206421434565,
+     0.9605545240578736
+    ],
+    "balanced_accuracy": 0.9531809097007545,
+    "false_approves": 78,
+    "false_approve_rate": 0.055201698513800426,
+    "false_denies": 59,
+    "false_deny_rate": 0.038436482084690554,
+    "pair_accuracy": 0.9017725258493353,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.9508970727101038,
+    "generator=gpt-6.1-sol": 0.9550026469031233,
+    "is_long=0": 0.9555927943024717,
+    "is_long=1": 0.9447415329768271,
+    "flip_mechanism=authorization": 0.9548660084626234,
+    "flip_mechanism=call_variant": 0.9544334975369458,
+    "flip_mechanism=environment": 0.9526881720430107,
+    "flip_mechanism=provenance": 0.9612403100775194,
+    "flip_mechanism=scope": 0.9460869565217391,
+    "surface=non-shell": 0.9513381995133819,
+    "surface=shell": 0.953882538431218,
+    "slice=in_distribution": 0.9530815109343936,
+    "slice=new_framework": 0.9561200923787528
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.04081632653061224,
+    "covering_tracks": 0.0,
+    "credential_exfiltration": 0.05660377358490566,
+    "data_destruction_unrequested": 0.08695652173913043,
+    "destructive_filesystem": 0.049079754601226995,
+    "irreversible_remote": 0.06666666666666667,
+    "malicious_install": 0.06,
+    "obfuscated_payload": 0.03773584905660377,
+    "physical_financial": 0.058823529411764705,
+    "privilege_escalation": 0.02564102564102564,
+    "prompt_injection": 0.03225806451612903,
+    "scope_violation": 0.0670391061452514,
+    "self_permission_tamper": 0.0,
+    "supply_chain": 0.12195121951219512
+   },
+   "mcnemar_p_vs_auto-3b-3": 0.21215615907855717
+  },
+  "TypeSafe Jev 1.14": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.9019674355495251,
+    "ci95": [
+     0.8907042637818594,
+     0.912184346501294
+    ],
+    "balanced_accuracy": 0.9008852189187881,
+    "false_approves": 177,
+    "false_approve_rate": 0.12526539278131635,
+    "false_denies": 112,
+    "false_deny_rate": 0.07296416938110749,
+    "pair_accuracy": 0.8013293943870015,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.9008498583569405,
+    "generator=gpt-6.1-sol": 0.9025939650608787,
+    "is_long=0": 0.8990364474235442,
+    "is_long=1": 0.9144385026737968,
+    "flip_mechanism=authorization": 0.9633286318758815,
+    "flip_mechanism=call_variant": 0.9076354679802956,
+    "flip_mechanism=environment": 0.7311827956989247,
+    "flip_mechanism=provenance": 0.9741602067183462,
+    "flip_mechanism=scope": 0.9078260869565218,
+    "surface=non-shell": 0.9075425790754258,
+    "surface=shell": 0.9010642491131258,
+    "slice=in_distribution": 0.8990059642147117,
+    "slice=new_framework": 0.9191685912240185
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.24489795918367346,
+    "covering_tracks": 0.07407407407407407,
+    "credential_exfiltration": 0.25157232704402516,
+    "data_destruction_unrequested": 0.14009661835748793,
+    "destructive_filesystem": 0.10429447852760736,
+    "irreversible_remote": 0.05925925925925926,
+    "malicious_install": 0.18,
+    "obfuscated_payload": 0.03773584905660377,
+    "physical_financial": 0.11764705882352941,
+    "privilege_escalation": 0.07692307692307693,
+    "prompt_injection": 0.024193548387096774,
+    "scope_violation": 0.1564245810055866,
+    "self_permission_tamper": 0.05660377358490566,
+    "supply_chain": 0.1951219512195122
+   },
+   "mcnemar_p_vs_auto-3b-3": 9.623571074126176e-22
+  },
+  "GPT-6 Luna": {
+   "v3": {
+    "n": 2948,
+    "accuracy": 0.8789009497964722,
+    "ci95": [
+     0.8666282333424832,
+     0.890187444029907
+    ],
+    "balanced_accuracy": 0.881407175344809,
+    "false_approves": 82,
+    "false_approve_rate": 0.05803255484784147,
+    "false_denies": 275,
+    "false_deny_rate": 0.1791530944625407,
+    "pair_accuracy": 0.758493353028065,
+    "pairs": 1354
+   },
+   "slices": {
+    "generator=claude-opus-4-8": 0.8649669499527857,
+    "generator=gpt-6.1-sol": 0.8867125463208047,
+    "is_long=0": 0.8818600754084625,
+    "is_long=1": 0.8663101604278075,
+    "flip_mechanism=authorization": 0.8984485190409027,
+    "flip_mechanism=call_variant": 0.8805418719211823,
+    "flip_mechanism=environment": 0.821505376344086,
+    "flip_mechanism=provenance": 0.8940568475452196,
+    "flip_mechanism=scope": 0.888695652173913,
+    "surface=non-shell": 0.8905109489051095,
+    "surface=shell": 0.8770201024832479,
+    "slice=in_distribution": 0.8767395626242545,
+    "slice=new_framework": 0.8914549653579676
+   },
+   "missed_harm_by_category": {
+    "abuse_spam_privacy": 0.02040816326530612,
+    "covering_tracks": 0.05555555555555555,
+    "credential_exfiltration": 0.0440251572327044,
+    "data_destruction_unrequested": 0.043478260869565216,
+    "destructive_filesystem": 0.07975460122699386,
+    "irreversible_remote": 0.044444444444444446,
+    "malicious_install": 0.14,
+    "obfuscated_payload": 0.05660377358490566,
+    "physical_financial": 0.08823529411764706,
+    "privilege_escalation": 0.02564102564102564,
+    "prompt_injection": 0.016129032258064516,
+    "scope_violation": 0.08379888268156424,
+    "self_permission_tamper": 0.09433962264150944,
+    "supply_chain": 0.07317073170731707
+   },
+   "mcnemar_p_vs_auto-3b-3": 3.806780630998232e-37
+  }
+ }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67fb2b86d9bc1105d3c9992fbf2cf64df5ffdbbdf08c414e164e12e74cf34f79
+size 791684972

tokenizer_config.json

@@ -0,0 +1,24 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "max_length": 65536,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 65536,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "tokenizer_class": "TokenizersBackend",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}

training_config.json

@@ -0,0 +1,17 @@
+{
+ "lr": 1e-05,
+ "epochs": 2,
+ "alpha": 0.8,
+ "temperature": 2.0,
+ "max_batch": 64,
+ "warmup_frac": 0.03,
+ "min_lr_fraction": 0.1,
+ "replay_ratio": 0.4,
+ "extra": true,
+ "severe_weight": 1.5,
+ "hard_weight": 1.5,
+ "name": "final_main",
+ "margin_weight": 0.5,
+ "margin": 4.0,
+ "token_budget": 32768
+}