README.md

---
license: apache-2.0
language:
- de
- en
metrics:
- f1
- precision
- recall
base_model:
- jhu-clsp/mmBERT-small
pipeline_tag: text-classification
tags:
- tool-use
- ai-agents
- function-calling
- mcp
- tool-security
- security
- llm-security
- ai-safety
- ai-agent-security
- patronus
- multilingual
- modernbert
- onnx
---

# Model Card for Husky Sight Tool Type Classifier

**Multilingual Tool-Type Classifier for Real-World AI Agent Security**

Husky Sight is a multilingual ModernBERT-based ([mmBERT](https://huggingface.co/blog/mmbert)) classifier that identifies *which kind of tool* a request, tool call, or agent step involves. It is part of the Patronus Protect security stack and is a member of the Husky tool-analysis family, alongside [Husky Paw](https://huggingface.co/patronus-studio/husky-paw-tool-action-classifier) (operation) and [Husky Nose](https://huggingface.co/patronus-studio/husky-nose-tool-security-properties-classifier) (security properties).

## Intended Uses

The model maps an input text to exactly one class:

| id | label | description |
|---:|---|---|
| 0 | `file` | Tool operating on the local file system. |
| 1 | `database` | Tool operating on a database. |
| 2 | `vcs` | Tool operating on version control. |
| 3 | `api` | Tool operating on an API. |
| 4 | `memory` | Tool operating on persistent memory. |
| 5 | `messaging` | Tool operating on a messaging service. |
| 6 | `web` | Tool operating on the web. |
| 7 | `browser` | Tool operating on a browser. |
| 8 | `shell` | Tool operating on a shell / OS command. |
| 9 | `code` | Tool operating on code execution. |
| 10 | `system` | Tool operating on the operating system. |
| 11 | `secrets` | Tool operating on secrets / credentials. |
| 12 | `infra` | Tool operating on infrastructure (k8s, cloud). |
| 13 | `unknown` | Tool operating on an unidentified tool. |

Examples:

| Input | Expected class |
|---|---|
| Read the config file at /etc/app.conf | `file` |
| SELECT email FROM users WHERE id = 1 | `database` |
| Create a branch and open a pull request | `vcs` |
| Send a POST request to the payments API | `api` |
| Run `kubectl scale deployment web --replicas=3` | `infra` |
| How are you today? | `unknown` |

Typical downstream uses:

- tool-risk routing,
- AI agent policy enforcement,
- approval workflows,
- runtime monitoring.

## Limitations

- A positive prediction describes an apparent property of the input, not proof that an action was executed.
- The model does not track information flow across multiple agent steps.
- German and English are the primary evaluated languages; other languages run through the multilingual backbone but were not actively validated.
- False positives and negatives are possible. High-impact enforcement should combine the model with deterministic policy and calibrated thresholds.

## Model Variants

- **Husky Sight Tool Type Classifier** – full ModernBERT model in FP32 (`model.safetensors`).
- **Husky Sight Tool Type Classifier ONNX (FP16)** – `onnx/onnx_fp16/model_fp16.onnx` in this repository.
- **[Husky Sight Tool Type Classifier Edge](https://huggingface.co/patronus-studio/husky-sight-tool-type-classifier-edge)** – quantized ONNX builds (`int8`, `int8_int4_embeddings`, `fp16`) in a separate edge repository.
- **Husky Sight Tool Type Classifier NTDB L2** – lightweight multilingual cascade components under `l2/` for efficient local runtime classification.

## Training Data

Trained on Patronus' in-house multilingual dataset for this task, built from cleaned
real-world sources plus internally generated examples. Real-world sources were judge-cleaned
by content (no keyword heuristics) and contaminated rows removed.

### Augmentations

To improve robustness the dataset includes modern obfuscation techniques:

- Unicode variants
- Homoglyph attacks
- Encodings (e.g. base64)
- Tag wrappers (User:, System:)
- HTML tags
- Code comments
- Spacing noise
- Leetspeak
- Case noise
- Combination of N augmentation techniques

### Regularization

- Natural-language wrappers around the payload
- Counterfactual samples
- Trigger-word / spurious-correlation corpora
- ~90% similarity deduplication with a train/(val ∪ test) leakage guard

### Reducing bias

All augmentations and regularizers are applied to positive and negative examples alike so
the model keys on content rather than surface form.

## Benchmark

Held-out test set (n = 2,914), single-label:

| Metric | Score |
|---|---|
| **Accuracy** | **0.957** |
| **F1 (macro)** | **0.957** |
| Precision (macro) | 0.957 |
| Recall (macro) | 0.957 |

Per-class F1:

| Class | F1 |
|---|---|
| database | 0.992 |
| secrets | 0.990 |
| messaging | 0.984 |
| infra | 0.981 |
| code | 0.981 |
| memory | 0.980 |
| browser | 0.977 |
| file | 0.975 |
| vcs | 0.974 |
| unknown | 0.938 |
| web | 0.936 |
| system | 0.904 |
| shell | 0.891 |
| api | 0.890 |

## Usage

```python
from transformers import pipeline

clf = pipeline("text-classification", model="patronus-studio/husky-sight-tool-type-classifier")
clf("Run kubectl scale deployment web --replicas=3")
# -> [{"label": "infra", "score": 0.98}]
```

## ONNX

The FP16 ONNX export lives under `onnx/onnx_fp16`; the quantized builds (`int8`, `int8_int4_embeddings`) live in the separate [Husky Sight Tool Type Classifier Edge](https://huggingface.co/patronus-studio/husky-sight-tool-type-classifier-edge) repository. Apply a
softmax over the logits and take the argmax:

```python
from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer

model_id = "patronus-studio/husky-sight-tool-type-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = ORTModelForSequenceClassification.from_pretrained(model_id, subfolder="onnx/onnx_fp16", file_name="model_fp16.onnx")

inputs = tokenizer("Run kubectl scale deployment web --replicas=3", return_tensors="pt")
logits = model(**inputs).logits.detach().cpu().numpy()[0]
print(model.config.id2label[int(logits.argmax())])
```

## Citation

```bibtex
@misc{huskysight2026,
  title={Husky Sight Tool Type Classifier: Multilingual Classification for Real-World AI Agent Security},
  author={Patronus Protect},
  year={2026},
  howpublished={\url{https://huggingface.co/patronus-studio/husky-sight-tool-type-classifier}}
}
```

## License

This model is released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).
A copy of the license is included as `LICENSE` in this repository.

The model is derived from [jhu-clsp/mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small), which is distributed
under the **MIT License**. The upstream copyright and permission notice are retained; the
MIT terms continue to apply to the portions originating from that work.

## Patronus Ark

This model is built to run inside **Patronus Ark**, Patronus' open-source on-device
AI-security scanning library (L1 native rules → L2 NTDB cascade → L3 transformer).
Ark is not publicly released yet — a repository link will be added here at launch.

---

## 🛡️ Patronus Protect

Brought to you by [Patronus Protect](https://patronus.studio) — a local AI firewall that
secures every AI interaction, including prompts, tools and documents, before it reaches
your models.

Try it for free at [patronus.studio](https://patronus.studio).