| --- |
| license: apache-2.0 |
| language: |
| - de |
| - en |
| metrics: |
| - f1 |
| - accuracy |
| base_model: |
| - jhu-clsp/mmBERT-small |
| pipeline_tag: text-classification |
| tags: |
| - document-classification |
| - topic-classification |
| - security |
| - dlp |
| - patronus |
| - multilingual |
| - modernbert |
| - text-classification |
| - safetensors |
| - german |
| - english |
| - llm-guard |
| - protect-ai |
|
|
| --- |
| |
| # Model Card for Orca-Sonar |
|
|
| **Multilingual Document Topic Classifier for Real-World AI Security & DLP** |
|
|
| Orca-Sonar is a Multilingual ModernBERT-based ([mmBERT](https://huggingface.co/blog/mmbert)) classifier that |
| assigns a document/text to one of **7 topic classes**. It is part of the Patronus Protect security stack and |
| is designed for topic-/risk-routing of incoming texts (e.g. before they reach an LLM, a DLP gate, or a storage tier). |
|
|
| It classifies German and English text and is robust to **user-to-AI wrappers** (e.g. *"Summarize this contract: …"*), |
| i.e. the *topic* of the content determines the class, not the surface format of the request. |
|
|
| ## Intended Uses |
|
|
| The model maps an input text to one of: |
|
|
| | id | label | description | |
| |---|---|---| |
| | 0 | `finance` | invoices, balance sheets, quarterly/annual reports, cash-flow, SEC filings, forecasts | |
| | 1 | `hr` | CVs, job ads, employment contracts, terminations, HR policies, performance reviews, recruiting | |
| | 2 | `internal_and_tech` | ADRs, RFCs, postmortems, specs, READMEs, wikis, architecture & strategy memos, runbooks | |
| | 3 | `legal` | contracts, NDAs, ToS/AGB, privacy policies, statutes/judgments, compliance, legal correspondence | |
| | 4 | `marketing` | press releases, newsletters, landing-page/sales copy, outbound pitches, case studies | |
| | 5 | `other` | conversational / non-business: smalltalk, recipes, travel, hobby, learning, creative | |
| | 6 | `source_code` | raw program code & configs (Python/Go/Rust/JS/TS/SQL/Bash/Dockerfile/k8s/Terraform …) | |
|
|
| **Disambiguation:** on a tie, the more sensitive class wins — |
| `legal > hr > finance > internal_and_tech > source_code > marketing > other`. |
|
|
| ## Limitations |
|
|
| - Highly accurate on German and English; other languages were not actively tested. |
| - The model can produce false positives; for high-stakes routing combine it with a confidence/abstention gate. |
| - Robustness against adversarial / out-of-distribution / pure-PII / pathological-length inputs is partial; pair the |
| model with a deterministic pre-gate (length + PII) for production DLP use. |
|
|
| ## Model Variants |
|
|
| - **orca-sonar** – full model (`model.safetensors`, fp32). |
| - **orca-sonar-fp16 (ONNX)** – FP16 ONNX export under `onnx/onnx_fp16/` — half the size, argmax-faithful to the full model. |
|
|
| # Training Data |
|
|
| Trained on our own in-house dataset (German + English, 7 topic classes), purpose-built for this model. |
| **The dataset will be published soon.** |
|
|
| # Benchmark |
|
|
| Held-out test set (**100 % real data**), per-class F1: |
|
|
| | Metric | Score | |
| |---|---| |
| | **Accuracy** | **0.978** | |
| | **F1 (macro)** | **0.978** | |
| | F1 legal | 0.995 | |
| | F1 source_code | 0.985 | |
| | F1 marketing | 0.980 | |
| | F1 internal_and_tech | 0.977 | |
| | F1 hr | 0.971 | |
| | F1 finance | 0.970 | |
| | F1 other | 0.970 | |
| |
| # Usage |
| |
| ```python |
| from transformers import pipeline |
| |
| clf = pipeline("text-classification", model="patronus-studio/orca-sonar-document-classifier") |
| clf("Fasse mir diesen Dienstleistungsvertrag zusammen: Laufzeit 24 Monate, Gerichtsstand München …") |
| # -> [{'label': 'legal', 'score': 0.99}] |
| ``` |
| |
| ## ONNX |
| |
| An FP16 ONNX version is available under `onnx/onnx_fp16/`: |
|
|
| ```python |
| import torch |
| from optimum.onnxruntime import ORTModelForSequenceClassification |
| from transformers import AutoTokenizer |
| |
| model_id = "patronus-studio/orca-sonar-document-classifier" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = ORTModelForSequenceClassification.from_pretrained(model_id, subfolder="onnx/onnx_fp16") |
| |
| inputs = tokenizer("def add(a, b):\n return a + b", return_tensors="pt") |
| logits = model(**inputs).logits |
| print(model.config.id2label[int(torch.argmax(logits, dim=-1))]) |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{orcasonar2026, |
| title={Orca-Sonar: Multilingual Document Topic Classification for Real-World AI Security}, |
| author={Patronus Protect}, |
| year={2026}, |
| howpublished={\url{https://huggingface.co/patronus-studio/orca-sonar-document-classifier}} |
| } |
| ``` |
|
|
