README.md

---
title: PromptShield
emoji: 🛡️
colorFrom: blue
colorTo: gray
sdk: gradio
app_file: app.py
pinned: false
sdk_version: 6.20.0
---

# PromptShield

A lightweight security layer that detects prompt-injection / jailbreak attempts
before they reach an LLM-powered application. Built as a standalone,
deployable API + demo, and designed to sit in front of existing LLM apps
(tested here against Consent Guardian and Prof-Scope-AI).

## How it works

```
User prompt -> Preprocessing -> Feature extraction -> Detection engine -> Risk gate
                                 (TF-IDF + heuristics)  (ML classifier)
                                                                |
                                                    Allow ------+------ Block
                                                    (forward)          (log + alert)
```

- **Feature extraction**: TF-IDF over the prompt text, combined with
  hand-crafted heuristic signals (presence of override phrases like
  "ignore previous instructions", imperative-mood sentence starts,
  suspicious formatting markers like `###SYSTEM`).
- **Detection engine**: a Random Forest classifier (selected over Logistic
  Regression for higher recall on the injection class -- in security, a
  missed attack is worse than a false alarm).
- **Risk gate**: prompts scoring above 0.5 probability are blocked and
  logged; below that, they're forwarded to the downstream app.

## Results

| Test | Accuracy | Notes |
|---|---|---|
| Held-out test split (same template distribution) | ~99-100% | Expected to be inflated -- same phrasing patterns as training |
| **Hand-written novel prompts (out-of-distribution)** | **12/12 (100%)** | More honest signal -- zero overlap with training templates, confidence scores realistically in the 0.4-0.8 range rather than saturated at 0/1 |

**Important honesty note for your write-up**: the training data here is
template-generated (see `src/build_dataset.py`) because this environment
couldn't reach Hugging Face Hub to pull the public `deepset/prompt-injections`
dataset. The novel-prompt test is a genuine generalization check, but a
larger, real-world dataset (see below) would make the results far more
defensible for a research write-up or professor-facing project.

## Upgrading to a real dataset (do this on Colab / your own machine)

This sandbox can't reach huggingface.co, so run this step separately:

```python
# pip install datasets
from src.build_dataset import load_real_dataset
load_real_dataset()  # pulls deepset/prompt-injections, saves to data/prompts_real.csv
```

Then retrain with `train.py` pointed at `data/prompts_real.csv` instead.
Consider also mixing in your synthetic data for extra coverage of
paraphrase variety.

## Project structure

```
PromptShield/
├── app.py                  # FastAPI + Gradio, deployment entry point
├── requirements.txt
├── data/
│   └── prompts.csv         # synthetic training data
├── models/
│   ├── classifier.joblib
│   └── vectorizer.joblib
└── src/
    ├── build_dataset.py    # dataset generation (+ real-dataset loader)
    ├── features.py         # TF-IDF + heuristic feature extraction
    ├── train.py             # training + evaluation
    └── novel_test.py        # honest out-of-distribution test
```

## Running locally

```bash
pip install -r requirements.txt
python src/build_dataset.py   # generate data
python src/train.py           # train + evaluate
python src/novel_test.py      # sanity-check generalization
python app.py                 # launch API + demo at localhost:7860
```

## Deploying to Hugging Face Spaces

1. Create a new Space, SDK = Gradio.
2. Upload this entire folder (or push via git).
3. Spaces auto-detects `app.py` as the entry point -- no extra config needed.
4. Your guardrail is now live at `https://huggingface.co/spaces/<you>/<space-name>`,
   with the `/check` endpoint callable from any other app.

## Calling it from another app (e.g. Consent Guardian, Prof-Scope-AI)

This Space uses Gradio's built-in API (no separate FastAPI server -- this
avoids a port conflict with Hugging Face's ZeroGPU hardware, which is the
only free tier available on new accounts).

**Option A: raw HTTP (two-step call -- POST then GET the result)**

```python
import requests

# Step 1: submit the prompt
resp = requests.post(
    "https://<your-space>.hf.space/gradio_api/call/check",
    json={"data": [user_input]}
)
event_id = resp.json()["event_id"]

# Step 2: fetch the result
result = requests.get(
    f"https://<your-space>.hf.space/gradio_api/call/check/{event_id}"
)
# result.text looks like: 'event: complete\ndata: ["blocked", 0.945]\n'
```

**Option B: gradio_client (simpler, recommended)**

```python
# pip install gradio_client
from gradio_client import Client

client = Client("https://<your-space>.hf.space")
decision, risk_score = client.predict(user_input, api_name="/check")

if decision == "blocked":
    # reject / log / alert instead of forwarding to the LLM
    ...
```