README.md

metadata

title: PromptShield
emoji: 🛡️
colorFrom: blue
colorTo: gray
sdk: gradio
app_file: app.py
pinned: false
sdk_version: 6.20.0

PromptShield

A lightweight security layer that detects prompt-injection / jailbreak attempts before they reach an LLM-powered application. Built as a standalone, deployable API + demo, and designed to sit in front of existing LLM apps (tested here against Consent Guardian and Prof-Scope-AI).

How it works

User prompt -> Preprocessing -> Feature extraction -> Detection engine -> Risk gate
                                 (TF-IDF + heuristics)  (ML classifier)
                                                                |
                                                    Allow ------+------ Block
                                                    (forward)          (log + alert)

• Feature extraction: TF-IDF over the prompt text, combined with hand-crafted heuristic signals (presence of override phrases like "ignore previous instructions", imperative-mood sentence starts, suspicious formatting markers like ###SYSTEM).
• Detection engine: a Random Forest classifier (selected over Logistic Regression for higher recall on the injection class -- in security, a missed attack is worse than a false alarm).
• Risk gate: prompts scoring above 0.5 probability are blocked and logged; below that, they're forwarded to the downstream app.

Results

Test	Accuracy	Notes
Held-out test split (same template distribution)	~99-100%	Expected to be inflated -- same phrasing patterns as training
Hand-written novel prompts (out-of-distribution)	12/12 (100%)	More honest signal -- zero overlap with training templates, confidence scores realistically in the 0.4-0.8 range rather than saturated at 0/1

Important honesty note for your write-up: the training data here is template-generated (see src/build_dataset.py) because this environment couldn't reach Hugging Face Hub to pull the public deepset/prompt-injections dataset. The novel-prompt test is a genuine generalization check, but a larger, real-world dataset (see below) would make the results far more defensible for a research write-up or professor-facing project.

Upgrading to a real dataset (do this on Colab / your own machine)

This sandbox can't reach huggingface.co, so run this step separately:

# pip install datasets
from src.build_dataset import load_real_dataset
load_real_dataset()  # pulls deepset/prompt-injections, saves to data/prompts_real.csv

Then retrain with train.py pointed at data/prompts_real.csv instead. Consider also mixing in your synthetic data for extra coverage of paraphrase variety.

Project structure

PromptShield/
├── app.py                  # FastAPI + Gradio, deployment entry point
├── requirements.txt
├── data/
│   └── prompts.csv         # synthetic training data
├── models/
│   ├── classifier.joblib
│   └── vectorizer.joblib
└── src/
    ├── build_dataset.py    # dataset generation (+ real-dataset loader)
    ├── features.py         # TF-IDF + heuristic feature extraction
    ├── train.py             # training + evaluation
    └── novel_test.py        # honest out-of-distribution test

Running locally

pip install -r requirements.txt
python src/build_dataset.py   # generate data
python src/train.py           # train + evaluate
python src/novel_test.py      # sanity-check generalization
python app.py                 # launch API + demo at localhost:7860

Deploying to Hugging Face Spaces

1. Create a new Space, SDK = Gradio.
2. Upload this entire folder (or push via git).
3. Spaces auto-detects app.py as the entry point -- no extra config needed.
4. Your guardrail is now live at https://huggingface.co/spaces/<you>/<space-name>, with the /check endpoint callable from any other app.

Calling it from another app (e.g. Consent Guardian, Prof-Scope-AI)

This Space uses Gradio's built-in API (no separate FastAPI server -- this avoids a port conflict with Hugging Face's ZeroGPU hardware, which is the only free tier available on new accounts).

Option A: raw HTTP (two-step call -- POST then GET the result)

import requests

# Step 1: submit the prompt
resp = requests.post(
    "https://<your-space>.hf.space/gradio_api/call/check",
    json={"data": [user_input]}
)
event_id = resp.json()["event_id"]

# Step 2: fetch the result
result = requests.get(
    f"https://<your-space>.hf.space/gradio_api/call/check/{event_id}"
)
# result.text looks like: 'event: complete\ndata: ["blocked", 0.945]\n'

Option B: gradio_client (simpler, recommended)

# pip install gradio_client
from gradio_client import Client

client = Client("https://<your-space>.hf.space")
decision, risk_score = client.predict(user_input, api_name="/check")

if decision == "blocked":
    # reject / log / alert instead of forwarding to the LLM
    ...