turkish-prompt-injection-classifier (v2)

What is this model? turkish-prompt-injection-classifier is a 307M Turkish model that classifies user messages and retrieved content in LLM apps and agents as BENIGN or INJECTION.

EN: Turkish prompt injection detection for LLM apps and agents — flags user messages and retrieved content as BENIGN or INJECTION (307M, MIT, ~35 ms/text on CPU). TR: LLM uygulamaları ve ajanlar için Türkçe prompt injection tespiti — kullanıcı mesajlarını ve okunan içeriği BENIGN/INJECTION olarak işaretler. Model page & setup help / Model sayfası ve kurulum desteği: halilneed.agency/models/turkish-prompt-injection-classifier

A small classifier that labels Turkish text going into an LLM as BENIGN or INJECTION: user messages, and the content an assistant reads (emails, documents, retrieved passages, tool output). v2 is mmBERT-base (307M parameters, multilingual ModernBERT), ~35 ms per text on a CPU, and ships with an optional decoder for encoded and hidden text.

v2 vs v1, in one table (recommended threshold 0.05 for v2, 0.5 for v1; details below):

	v1 (BERTurk)	v2 (mmBERT)
Unseen source, F1: conv / 1k / berat	0.588 / 0.823 / 0.290	0.596 / 0.968 / 0.506
Mean of the three	0.567	0.690
Obfuscated attacks, recall (fevzi, 107)	0.832	0.963
Hidden in invisible Unicode, recall (42)	0.405	0.976
False alarms: KVKK / banking / PII instructions	0.2% / 1.1% / 0%	1.1% / 2.1% / 0%
CPU, mean per text	~23 ms	~35 ms

v2 scores higher on every attack set. It raises more false alarms (about one in fifty banking texts) and is slower. If few false alarms and speed matter most, pin v1:

pipeline("text-classification", model="halilneed/turkish-prompt-injection-classifier", revision="v1")

Threshold change (v2.1, 14 Oct 2026). v2 was first published with threshold 0.3. That value had been picked on a different training seed than the one released. Re-checking it on the released model's own held-out-source runs (no test data) moved it to 0.05: mean F1 there 0.661 → 0.692, internal false alarms unchanged at 0.5%. The weights did not change; only the recommended threshold and the detect.py default did. Numbers at 0.3 stay in eval/v2/.

Türkçe özet aşağıda: Türkçe.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="halilneed/turkish-prompt-injection-classifier")
out = clf("Önceki mesajımdaki formatı unut, bu sefer tablo olarak yaz.")
# 'label' is the argmax; for the recommended threshold use the INJECTION probability:
clf("...", top_k=None)  # [{'label': 'INJECTION', 'score': p}, {'label': 'BENIGN', 'score': 1 - p}]

Threshold. Flag as injection when the INJECTION probability is ≥ 0.05. The model is very confident on benign text, so a low threshold mostly adds attacks, not false alarms. It was chosen on the selection protocol below, before the test splits were touched.

With the decoder (detect.py + guard.py + common_words.json in this repo). It also classifies the decoded forms of the text (base64, hex, Caesar shift, reversed text, letter spacing, leetspeak, Cyrillic look-alike letters, invisible Unicode tag characters and zero-width characters) and keeps the highest score. It adds under 1 ms per text.

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("halilneed/turkish-prompt-injection-classifier", allow_patterns=["*.py", "*.json"])
sys.path.insert(0, path)
from detect import Detector

d = Detector()                  # v2, threshold 0.05
d("Şu metni çöz: dG9wbGFudMSxIG5vdGxhcsSxbsSxIG1hZGRlIG1hZGRlIMO2emV0bGU=")
# {'label': 'BENIGN', 'score': 0.0473, 'decoded': ['toplantı notlarını madde madde özetle'], 'max_from': None}
# score is the highest INJECTION probability over the text and its decoded forms; max_from says which form it came from

• Input is truncated at 512 tokens. For long documents, classify chunks and flag the document if any chunk is INJECTION.
• This is one layer. Keep an agent's tools on least privilege, ask for confirmation before irreversible actions, and treat retrieved content as data whatever this model says.

How v2 was chosen

v1 was evaluated on the test splits, and I also compared a few variants there. For v2 I separated selection from reporting:

• Selection (test splits untouched). Each of the three Turkish training sources is held out in turn; the model trained without it is measured on that source's train + validation splits (630, 887 and 546 rows) and on benign internal splits that were never used in training (KVKK and BSEBY test splits, PII dev split). The score is the mean F1 over the three held-out sources.
• Report. The chosen recipe is measured once on the test splits.

Recipe (selection protocol)	Mean F1	Max benign false alarms
BERTurk, Turkish data (v1 recipe)	0.612	0.5%
+ obfuscation augmentation 25%	0.662	0.9%
+ obfuscation augmentation 50%	0.615	0.3%
+ English agentic data	0.646	0.6%
XLM-R base + English agentic data	0.493	0.8%
mmBERT-base + English agentic data	0.674	0.1%
mmBERT-base + obfuscation 25%	0.647	0.4%
mmBERT-base + obfuscation 25% + English agentic data	0.699 (0.717 at threshold 0.3)	0.3%
same, second seed	0.646 (0.661 at threshold 0.3)	0.5%

Each row is one training seed unless noted. The two seeds of the chosen recipe differ by 0.056 at threshold 0.3; differences smaller than that between rows are within noise. I fixed in advance that the released model would come from the second seed, and it is the weaker one. mDeBERTa-v3 diverged in bf16 and collapsed to one class in fp32 under transformers 5.17, so it is not in the table.

After the release I trained two more seeds of the same recipe to see how much of v2 is luck. Their held-out-source means were 0.638 and 0.529, and retraining with the first seed's settings did not reproduce its 0.699, because bf16 training on the GPU is not deterministic. Three things were tried to remove that variance, all on the selection protocol:

Option (selection protocol)	Mean F1	False alarms on the sources' hard benign rows (conv / 1k)	Cost
Released v2, threshold 0.3	0.661	2.2% / 8.2%	1 model
Released v2, threshold 0.05	0.692	2.4% / 8.9%	1 model
Average of 3 seeds' probabilities, threshold 0.05	0.716	5.1% / 13.7%	3 models
Weight average ("model soup") of 3 seeds, threshold 0.05	0.681	—	1 model

The three-seed average scores highest, but it raises the false alarms on hard benign rows (conv 2.4% → 5.1%, 1k 8.9% → 13.7%) and costs three times as much; the weight average is worse than a single seed. So the release stays one model, with the lower threshold.

Recall mode (v1 + v2)

Detector(mode="recall") loads both versions and scores 0.5 × p(v1) + 0.5 × p(v2), flagging at ≥ 0.2. It was added before the threshold change above; with v2 at 0.05 it is better only on conv. The weight and the threshold were chosen on the selection protocol: 0.696 there. Test splits, once:

Unseen source	v1	v2 @ 0.05	recall mode
conv F1 (recall / false-positive rate)	0.588 (0.500 / 0.067)	0.596 (0.467 / 0.033)	0.690 (0.667 / 0.089)
1k F1	0.823 (0.823 / 0.412)	0.968 (0.962 / 0.059)	0.893 (0.949 / 0.412)
berat F1	0.290 (0.170 / 0.000)	0.506 (0.339 / 0.000)	0.447 (0.288 / 0.000)
Mean F1	0.567	0.690	0.677
fevzi recall (all-sources models)	0.832	0.963	0.963
False alarms: KVKK / banking texts	0.002 / 0.011	0.011 / 0.021	0.008 / 0.023
CPU, mean per text	~23 ms	~35 ms	~59 ms

It inherits v1's weakness on hard benign rows: in the 1k test set it flags 14 of the 34 benign rows, against 2 for v2 at 0.05. They are all 7 rows where a user changes their own earlier instruction and all 7 that ask for a summary while ignoring commands embedded in the text. Use it where a missed attack costs more than a blocked request, and measure your own false-alarm rate. Raw numbers: eval/ensemble/recall-mode.json.

Results

Attack sets: F1, model trained without that source (recall / false-positive rate)

Model	conv (30 attacks)	1k (79)	berat (59)
Keyword rule	0.308 (0.200 / 0.033)	0.255 (0.165 / 0.294)	0.033 (0.017 / 0.000)
protectai/deberta-v3-base-prompt-injection-v2 (English)	0.320 (0.400 / 0.367)	0.603 (0.557 / 0.676)	0.422 (0.322 / 0.214)
proventra/mdeberta-v3-base-prompt-injection	0.633 (0.633 / 0.122)	0.910 (0.962 / 0.353)	0.907* (0.831 / 0.000)
patronus-studio/wolf-defender-prompt-injection-small	0.596 (0.467 / 0.033)	0.868 (0.835 / 0.206)	0.525 (0.356 / 0.000)
this model v1, source held out	0.588 (0.500 / 0.067)	0.823 (0.823 / 0.412)	0.290 (0.170 / 0.000)
this model v2, source held out	0.596 (0.467 / 0.033)	0.968 (0.962 / 0.059)	0.506 (0.339 / 0.000)
this model v2, all sources †	0.871 (0.900 / 0.056)	0.994 (1.000 / 0.029)	0.965 (0.932 / 0.000)

I ran every model with the same script; these are my measurements, not the authors' published numbers. Comparison models at threshold 0.5, v2 at 0.05, v1 at 0.5. † In-distribution: the released model was trained on the other splits of the same sources. * berat is a Turkish translation of deepset/prompt-injections; a model trained on the English deepset set has, in effect, seen these items. The English agentic sets added in v2 are by the same curator as conv and 1k (3nesdeniz). They are in English and contain no rows from the Turkish sets, but the "unseen source" columns for conv and 1k are less unseen for v2 than for v1.

Obfuscated and hidden attacks: recall (attack-only sets, never used in training or selection)

Model	fevzi (107)	+ decoder	invisible (42)	+ decoder
protectai v2	0.935	0.944	0.024	0.500
proventra mdeberta-v3	0.925	0.953	0.976	0.976
wolf-defender small	0.841	0.851	1.000	1.000
this model v1	0.832	0.851	0.405	0.905
this model v2	0.963	0.963	0.976	0.976

fevzi: fevziegeyurtsevenler/turkish-prompt-injection. invisible: fevziegeyurtsevenler/invisible-unicode-injection, instructions hidden in Unicode tag characters or split by zero-width characters inside innocent text. The decoder was written from general techniques and not tuned on these rows. On the three benign test sets below it changed no model's false-positive rate; on an internal benign split it added one false alarm in 800 texts for v1.

False-positive rate on benign Turkish text (lower is better)

Model	KVKK texts (1,125)	banking texts (750)	PII task instructions (1,000)
Keyword rule	0.010	0.019	0.011
protectai v2	0.080	0.141	0.315
proventra mdeberta-v3	0.002	0.032	0.772
wolf-defender small	0.011	0.076	0.607
this model v1 ‡	0.002	0.011	0.000
this model v2 ‡	0.011	0.021	0.000

The three columns are the three subsets of halilneed/turkish-prompt-injection-false-alarm-benchmark, 2,875 benign Turkish texts collected from CC0 sets (my KVKK and BSEBY benchmarks, and instruction + input of cagrigungor/turkish-pii-masking-benchmark). None were used in training. Its evaluate.py reproduces this table for any model. ‡ In-domain for this model and not for the others: its training data has other texts from the same domains. Trained without them, the v1 recipe flagged 65% of banking texts and 98.5% of task instructions. Measure on your own benign traffic first.

Where it fails, and where it holds

• Agentic attacks from an unseen source. conv with that source held out: memory poisoning, data exfiltration and tool abuse are each caught in 0 of 3 rows; system prompt extraction and RAG poisoning 1 of 3; authority claims and indirect content injection 2 of 3. Three rows per family: a direction, not a score.
• Turkish morphological rewording (fevzi): 13 of 15, up from 9 in v1.
• Encoded attacks (fevzi encoding-obfuscation): 13 of 13, up from 5. proventra 10, wolf-defender 12.
• Users changing their own instruction (1k legit_override_own_style): 5 of 7 with that source held out (6 at threshold 0.3), 6 of 7 with all sources; proventra 0 of 7, wolf-defender 1 of 7. The lower threshold costs one of these.

Speed

Intel i5-12400F, PyTorch fp32, 8 threads, batch 1, 200 test texts of 119 characters on average: v2 mean 35.0 ms, p95 55.7 ms; with the decoder 35.7 ms. v1 (BERTurk) about 23 ms.

Training data

No new attack text was written for this model. Attacks come only from public datasets; mechanical transformations (letter spacing, separators, leetspeak, Cyrillic look-alikes, reversal, case changes) are applied to existing rows of both classes with the same probability, so that unusual spelling alone does not mean "attack".

Source	License	Used	Unique rows
3nesdeniz/turkish-conversation-prompt-injection	CC-BY-4.0	train + validation	120 attack, 510 benign
3nesdeniz/turkish-prompt-injection-1k	CC-BY-4.0	train + validation	471 attack, 416 benign
beratcmn/turkish-prompt-injections	Apache-2.0	train	199 attack, 300 benign
3nesdeniz/agentic-prompt-injection-5k + boundary-pairs (English)	CC-BY-4.0	train + validation	5,350 (2,950 attack)
My Turkish business texts (training splits of my KVKK and BSEBY classifiers)	own	sample	2,994 benign
PII masking instructions and instruction + text pairs (training split of my PII model)	own	sample	1,473 benign

• Not used, on purpose: fevziegeyurtsevenler/multilingual-prompt-injection (its 107 Turkish rows are the fevzi test set), rikka-snow/prompt-injection-multilingual (deepset in 11 languages; it would leak the berat test), OnerAYTAS/Turkish_prompt_injection_jailbreak_dataset (CC-BY-NC-SA), and sets without a license.
• Test splits of all sources, fevzi and invisible were never used for training, selection or the threshold.
• Training: full fine-tune of jhu-clsp/mmBERT-base, 3 epochs, lr 3e-5, batch 32, max length 384, bf16, seed 7 (data seed 21), one RTX 5060 Ti. Checkpoint chosen on a 10% Turkish development split (dev F1 0.955).

Not compared

• meta-llama/Llama-Prompt-Guard-2-86M and -22M: gated; my account does not have access.
• fevziegeyurtsevenler/turkish-prompt-injection-detector: scikit-learn, not loadable with transformers.

Limits

• Test sets are small: 30, 79 and 59 attacks. One or two rows move F1 by several points, and two training seeds of the same recipe differed by 0.056 mean F1 on the selection protocol.
• All Turkish sources are synthetic or translated. None is a sample of real production traffic.
• The model reads one piece of text at a time. It does not see the conversation, the system prompt or which tool the text came from.
• It is not a content moderation model. A harmful request without any manipulation is BENIGN by design.

Versions

• v2.1 (14 Oct 2026, this revision): same weights as v2; recommended threshold 0.3 → 0.05 after re-checking it on the released seed (see "How v2 was chosen"). detect.py default follows.
• v2 (14 Oct 2026): mmBERT-base, obfuscation augmentation, English agentic data, decoder. detect.py adds a recall mode (v1 + v2 average).
• v1 (14 Oct 2026, revision="v1"): BERTurk, Turkish data only. Lower latency, fewer false alarms, better on unseen conversational attacks; worse on obfuscated and hidden attacks.

Türkçe

Bir LLM'e giren Türkçe metni BENIGN (zararsız) ya da INJECTION olarak işaretleyen küçük bir sınıflandırıcı. v2: mmBERT-base (307M), CPU'da metin başına ~35 ms, isteğe bağlı çözücüyle (base64, hex, Sezar, ters yazım, harf aralama, leet, görünmez Unicode).

• v2 neyi iyileştirdi (önerilen eşik 0,05): görmediği kaynakta ortalama F1 0.567 → 0.690; conv 0.588 → 0.596, 1k 0.823 → 0.968, berat 0.290 → 0.506. Gizlenmiş saldırılarda (fevzi) 0.832 → 0.963, kodlanmış saldırılarda 13'te 5 → 13'te 13. Görünmez Unicode ile gizlenmiş talimatlarda 0.405 → 0.976.
• Neyi kötüleştirdi: yanlış alarm bankacılık metinlerinde %1,1 → %2,1, KVKK metinlerinde %0,2 → %1,1. Hız ~23 → ~35 ms.
• Eşik değişikliği (v2.1): v2 önce 0,3 eşikle yayınlandı; o değer yayınlanmayan bir tohumun koşusunda seçilmişti. Yayınlanan modelin kendi kaynak-dışarıda koşularında (test verisi yok) yeniden bakınca 0,05 çıktı. Ağırlıklar aynı.
• Bu iki şey sizin için önemliyse v1'e sabitleyin: revision="v1".
• Önce duyarlılık modu (Detector(mode="recall")): v1 ve v2'nin ortalaması, eşik 0,2. Görmediği kaynakta ortalama F1 0.677 (conv 0.690), fevzi 0.963; ama 1k'nın 34 zararsız satırından 14'ünü işaretliyor (v2 0,05'te: 2): kullanıcının kendi talimatını değiştirdiği 7 satırın hepsi ve metne gömülü komutları yok sayarak özet isteyen 7 satırın hepsi. Kaçan saldırının engellenen istekten pahalı olduğu yerde kullanın.
• Yanlış alarm tablosu yeni veri setimden: turkish-prompt-injection-false-alarm-benchmark (2.875 zararsız metin, CC0).
• Aynı tarifin dört koşusu seçimde 0.529 ile 0.699 arasında çıktı. Üç tohumun ortalaması en yüksek skoru verdi ama zor zararsız satırlarda yanlış alarmı artırdı (conv %2,4 → %5,1; 1k %8,9 → %13,7) ve üç kat yavaş; ağırlık ortalaması tek tohumdan kötüydü. Yayın tek model olarak kaldı.
• Bu model için yeni saldırı metni yazılmadı; saldırılar yalnızca açık lisanslı setlerden. Tek katmandır: en az yetki, geri alınamaz işlemde onay ve getirilen içeriği veri saymak yine gerekir.

Citation and attribution

Training data by Enes Deniz (3nesdeniz, CC-BY-4.0) and beratcmn (Apache-2.0, translated from deepset/prompt-injections); evaluation data by fevziegeyurtsevenler (CC-BY-4.0) and cagrigungor (CC0). Base model jhu-clsp/mmBERT-base (MIT). Thank you for publishing them.

Related: halilneed/turkish-pii-detection · halilneed/turkish-kvkk-classifier · halilneed/turkish-bseby-classifier · Overview: halilneed.agency/models/turkish-prompt-injection-classifier

Related models / İlgili modeller

• halilneed/turkish-pii-detection — instruction-conditioned PII masking / talimatla kişisel veri maskeleme (270M)
• halilneed/turkish-pii-detection-GGUF — GGUF (BF16, Q8_0) for Ollama & llama.cpp, CPU / çevrimdışı GGUF sürümü
• halilneed/turkish-kvkk-classifier — KVKK / VERBİS personal-data category classifier / kişisel veri kategori sınıflandırıcı
• halilneed/turkish-bseby-classifier — BSEBY banking data & secrecy classifier / bankacılık veri ve sır sınıflandırıcı
• halilneed/turkish-prompt-injection-classifier — Turkish prompt injection detection / Türkçe prompt injection tespiti (this model / bu model)

All five in one collection / Hepsi tek koleksiyonda: Turkish Privacy & Safety Models

FAQ / Sık sorulan sorular

What threshold should I use? / Hangi eşiği kullanmalıyım? Flag as injection when the INJECTION probability is ≥ 0.05 (v2.1). v1 uses 0.5. / Önerilen eşik 0,05.

v1 or v2? / v1 mi v2 mi? v2 scores higher on every attack set (unseen-source mean F1 0.690 vs 0.567; fevzi recall 0.963 vs 0.832) but has more false alarms (banking 2.1% vs 1.1%) and is slower (~35 vs ~23 ms). Pin revision="v1" if few false alarms and speed matter most.

Does it detect encoded or hidden attacks? / Kodlanmış/gizli saldırıları yakalar mı? With the optional decoder (detect.py) it also scores base64, hex, Caesar, reversed, leetspeak, Cyrillic look-alikes and invisible Unicode; recall on the invisible set is 0.976. / Çözücü metin başına 1 ms'den az ekler.

Is it enough on its own? / Tek başına yeterli mi? No. It is one layer: keep tools on least privilege, confirm irreversible actions and treat retrieved content as data. It is not a content moderation model. / Tek katmandır.

Resources / Kaynaklar

• halilneed agency: halilneed.agency/models/turkish-prompt-injection-classifier