| --- |
| license: apache-2.0 |
| language: |
| - zh |
| - en |
| tags: |
| - safety |
| - guardrail |
| - prompt-injection |
| - security |
| - text-classification |
| - mlx |
| metrics: |
| - f1 |
| - accuracy |
| --- |
| # TinyGuard-Injection (base + fine-tuned) |
|
|
| **Bilingual prompt-injection detector (incl. indirect injection) trained completely from scratch** · 9.1M parameters · MLX/Metal · Apache-2.0 (code) · fully synthetic training data |
|
|
| [中文说明](MODEL_CARD-injections.zh.md) |
|
|
| ## Model description |
|
|
| A 3-way guardrail deployed at LLM-app input gates and RAG retrieval gates: `benign / direct injection / indirect injection`. Complements Meta Llama Prompt Guard 2 (22M/86M, **no Chinese support**): this model is trained bilingual with equal weight. |
|
|
| - Backbone: shared TinyGuard-Encoder (same pretrained base as TinyGuard-Command) |
| - Fine-tune: 12,330 bilingual labeled examples across six carriers (code & collaboration platforms, emails & web pages, long RAG chunks, tickets & customer-service dialogs, benign hard negatives, hidden-character & encoded payloads), with "minimal-difference pairs" and anti-overrefusal design |
|
|
| ## Results (held-out data only, never seen in training) |
|
|
| | Benchmark | Metric | Value | |
| |---|---|---| |
| | Validation set, 616 rows | acc / macro-F1 | **99.0% / 0.990** | |
| | Adversarial set, 40 rows (zero-width chars / JSON rules / Japanese carriers / quoted disclaimers) | intervention recall / precision / F1 | 75% / 88% / 0.81 | |
|
|
| Reference: Llama Prompt Guard 2-86M reports EN injection F1 ≈ 0.98 with no Chinese. This model provides first-tier Chinese capability; its English adversarial numbers are below Meta's home turf (expected at 9.1M vs 86M). |
|
|
| ## Usage |
|
|
| Weights: [`QuanxinLi/TinyGuard-Injection`](https://huggingface.co/QuanxinLi/TinyGuard-Injection) · |
| Code & training pipeline: [`Quanxin-Lee/TinyGuard`](https://github.com/Quanxin-Lee/TinyGuard) |
|
|
| ```python |
| # quickstart (transformers/MLX dual format) — see the GitHub repo for the full walkthrough |
| ``` |
|
|
| ## Limitations |
|
|
| - 5/40 misses on the adversarial set: JSON-encoded rules, URL-less "prioritize this" phrasing, direct asks inside authorized framing (red-team wording) — all weak-signal forms. |
| - English training volume is lower than Chinese; English adversarial generalization needs strengthening. |
| - Recommended to stack with a regex layer: this model handles semantics, regex handles known fingerprints. |
|
|
| ## Training details & data statement |
|
|
| All data is synthetic/hand-written — no real user data. The full training pipeline, four audit scripts (three deep-scan rounds) and every dataset ship with the repository. |
|
|
