TheShrimpler/portcullis-1.0

Portcullis 1.0

Portcullis is the gate that drops to block. It reads a prompt headed to your LLM and answers, in about 20 milliseconds on a gaming GPU: is someone trying to attack me? It outputs 7 signals — override attempt, jailbreak, data theft, obfuscated payload, tool hijack, category, severity — each with a probability you can actually trust (calibrated: when it says 0.9, it's right about 90% of the time). Then your code decides: let it through, block it, or send it to something smarter.

It's 395M parameters, runs completely offline, costs nothing per prompt, and can't hallucinate a verdict — it outputs probabilities, never prose.

Started as a fun experiment to solve a real problem: guardrails that either cost a fortune, leak your prompts to an API, or block half your legitimate users. Total cost of the whole project: under $3.

"Why wouldn't I just use Jev?"

Fair question — we tested it, same data, same thresholds, and the honest answer is: they're different tools. The full table with confidence intervals is below, but the short version:

	Portcullis 1.0	Jev-1.13 (commercial)
Runs offline / on your laptop	✅	❌ (hosted API)
Cost per prompt	$0	~$0.04 per million tokens (still cheap)
Flags when it's unsure	✅ (3-tier policy included)	single answer
Over-blocking on tricky-but-legit prompts	2%	27%
Catches more attacks	❌	✅ (99% vs 90% on blind red-team)
Transparent probabilities you can gate on	✅	✅

The trade-off in one sentence: Jev catches a few more attacks; Portcullis almost never blocks your real users while doing it, and runs on your own hardware. If your users are enterprisey and your false-positive tolerance is low, that trade matters. The honest caveat: Jev was scored with a generic question config (its own security-native config would need vendor participation), so this is a comparison under disclosed handicaps — we claim the trade-off, not superiority.

The benchmark-integrity story (why this card is weird)

My first evaluation said this model beat Jev on 5 of 6 independent probes. I retracted it. My own audit found that my training data contained the exact test rows I was scoring on — 1,044 of 1,044 prompts from one benchmark, all 450 from another. Classic train-on-test contamination, hiding inside a popular public dataset.

So this release ships with: the contaminated sets removed, a re-run from clean data only, pre-committed thresholds, confidence intervals on every number, the full audit trail, and both the numbers that survived and the ones that didn't. If you're evaluating any guardrail — or buying one — check for this in their card too. (This is a known, documented problem in prompt-classifier benchmarks.)

The numbers

Two blind external red-team sets (written by an external generator after training, blind to the model's outputs by procedure — not independently attested, full protocol in the repo):

test	n	Portcullis attack detection	Jev attack detection	Portcullis false-positives	Jev false-positives
blind red-team set 1	109	90% [80–95%]	99% [92–100%]	12% [5–25%]	7% [2–19%]
blind red-team set 2	118	88% [78–94%]	98% [92–100%]	2% [0–10%]	12% [5–23%]
shifted classic injections	110	57%	59%	4%	0%

Benign-discipline tests (lower is better — this is the "does it annoy my users?" number):

test (all benign)	n	Portcullis flags	Jev flags
OR-Bench hard-benign	1,319	2% [1–2%]	27% [25–30%]
NotInject trigger-word prompts	113	12%	14%
awesome-chatgpt-prompts (roleplay)	2,002	29% ⚠️	8%

⚠️ = the named weak spot. Roleplay-flavored prompts are where Portcullis over-flags. The shipped policy is two-tier (block / escalate / allow), so in practice most of those become "ask a second opinion" rather than "blocked" — but it's the first thing on the roadmap, and we tried a fix before release that didn't confirm.

Stacked deployment (Portcullis gates everything, a small 86M open backup model handles the uncertain middle band): 85% attack detection at 0% false-positives on the blind red-team set 2.

Speed & cost (measured, RTX 5090)

	GPU	CPU (same machine)
load to ready	2.4s	~
prompt scored, p50	15.4ms	595ms
p95	18.9ms	—
max observed	21.0ms	732ms

Full batch protocol: 150 prompts, warmup discarded, sorted percentiles (results/latency_benchmark.json). At 15ms/prompt a single gaming GPU screens ~230k prompts/hour for free, offline.

What it cost to build (full accounting)

• API spend (data generation + benchmark baselines): ~$2.50
• Electricity (8h of GPU training at ~400W): **$0.40**
• Total: under $3.
• Time: ~20 hours start to finish — most of it the GPU training unattended overnight while the author slept, plus a day of evaluation and two adversarial reviews of his own work.

Known limitations — in plain English

1. Roleplay-flavored prompts over-flag (29% at the block threshold, 36% at the escalation threshold — the full sweep is in results/SHIP_TABLE.md). If your app is full of "act as a pirate" prompts, test before trusting.
2. Three of the seven signals have looser confidence calibration (data theft, encoded payloads, tool hijacks). The policy engine accounts for this: those signals escalate instead of auto-blocking. Don't wire them to auto-block.
3. Creative fiction attacks and hidden pre-arranged codes (acrostics, secret phrases) can slip through. The escalation tier is for those.
4. One author trained and evaluated this model. The eval code is public — run it yourself, that's the real check.

Try it

# pip install torch transformers  (GPU or CPU)
from modeling_typed import *   # files in this repo
import json, torch

tok = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-large")
tok.add_special_tokens({"additional_special_tokens": ["[STATE]", "[Q]", "[OPT]"]})
hidden = AutoConfig.from_pretrained("answerdotai/ModernBERT-large").hidden_size
model = TypedModel("answerdotai/ModernBERT-large", hidden)
model.encoder.resize_token_embeddings(len(tok))
model.load_state_dict(torch.load("portcullis_v1.0.pt", map_location="cuda")["state"])

# then: build_row(prompt, ...) -> collate -> model -> 7 calibrated signals
# full working example: demo.py in this repo

Thresholds: 0.5 = block band, 0.3 = escalate band. Both frozen before the audit; rotate them in config without retraining.

How it was trained (the honest composition)

Public prompt-security corpora (PIGuard, ~77k prompts), real jailbreak collections, hard-negative benign prompts, verified synthetic attacks, and programmatically-constructed steganographic attacks with perfect labels. Attack-type decomposition was partially supervised by Jev-1.13 itself (disclosed: that teacher-competitor circularity is a limitation of this version). One training run from scratch, two passes, one GPU.

Roadmap

1. Fix the roleplay over-flagging (in progress — first attempt didn't hold, publishing the attempt anyway)
2. Calibration hardening for the three weaker signals
3. Measured latency/cost benchmarks (GPU + CPU)
4. Weekly frozen blind red-team rounds, results published every time

Files

• portcullis_v1.0.pt — weights
• calibration.json — per-signal temperature scaling
• modeling_typed.py — self-contained model + scoring code
• demo.py — working example
• results/ — every table on this page, the audit, the critique, the raw JSON

License & attribution

Apache-2.0. Base model: ModernBERT-large (Answer.AI). Training corpora: PIGuard (MIT) and other public sets (per-file attribution in the repo). Not affiliated with TypeSafe AI. The contamination audit and retraction were self-initiated and are dated in the repo history.