README.md

---
library_name: peft
base_model: Qwen/Qwen2.5-7B-Instruct
tags:
  - cybersecurity
  - sql-injection
  - phishing
  - qlora
  - lora
  - instruction-tuning
  - qwen2
license: apache-2.0
datasets:
  - syedsaqlainhussain/sql-injection-dataset
  - pirocheto/phishing-url
language:
  - en
pipeline_tag: text-generation
metrics:
  - accuracy
  - f1
  - precision
  - recall
model-index:
  - name: qlora-cyber-security-classifier
    results:
      - task:
          type: text-classification
          name: Cybersecurity threat classification
        dataset:
          name: SQL Injection + Phishing (held-out test split)
          type: custom
        metrics:
          - type: accuracy
            value: 0.99
            name: Accuracy
          - type: f1
            value: 0.99
            name: Weighted F1
          - type: precision
            value: 0.99
            name: Weighted Precision
          - type: recall
            value: 0.99
            name: Weighted Recall
---

## Model Information

**QLoRA Cyber Security Classifier** is a LoRA fine-tuned adapter on top of
[Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct), trained
to detect **SQL injection attempts** and **phishing URLs** and explain the
reasoning behind each classification. It was trained as an instruction-tuned
security triage assistant: given a SQL query or a URL, it returns a
`Classification:` label plus a short `Reason:` for that call.

**Model developer:** [jayesh20](https://huggingface.co/jayesh20)

**Model Architecture:** Qwen2.5-7B-Instruct (decoder-only transformer) with
LoRA adapters injected into attention and MLP projection layers, fine-tuned
under 4-bit NF4 quantization (QLoRA).

| | Training Data | Params (base) | LoRA rank / alpha | Context length | Token count | Base model release |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| QLoRA Cyber Security Classifier | SQL injection (Kaggle) + Phishing URLs (HF) | 7B | 16 / 32 | 256 | ~3K training examples (subset) | Qwen2.5, Sep 2024 |

**Supported tasks:** binary security classification with explanation, for two domains:
- SQL query → `SQL Injection` / `Benign`
- URL → `Phishing` / `Legitimate`

**Model Release Date:** July 2026

**Status:** This is a research/prototype model trained on a limited subset of
data under a tight compute budget (single T4 GPU). See [Limitations](#limitations) below.

**License:** Apache 2.0 for the adapter weights. The base model
(Qwen2.5-7B-Instruct) carries its own license — check
[Qwen's license terms](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) before
redistribution or commercial use.

## Intended Use

**Intended use cases:** Assistive triage in a security pipeline — flagging
suspicious SQL queries or URLs for human review, or as one signal among
several in an automated detection tool. Useful for research and prototyping
LLM-based security classifiers.

**Out of scope:**
- **Not a standalone production security gate.** This does not replace
  parameterized queries / prepared statements (the actual defense against SQL
  injection), a WAF, or established phishing-detection services.
- Not evaluated against adversarial/obfuscated inputs (encoded payloads,
  homoglyph domains, case-mixing evasion).
- Not intended for classification tasks outside SQL queries and URLs.

## How to use

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER_REPO = "jayesh20/qlora-cyber-security-classifier"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_REPO)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, quantization_config=bnb_config, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO)
model.eval()

PROMPT = """### Instruction:
{instruction}

### Input:
{input}

### Response:
"""

def predict(text, task="sql"):
    instruction = (
        "Analyze the following input and determine if it is a SQL injection attempt."
        if task == "sql" else
        "Analyze this URL and classify whether it is phishing or legitimate."
    )
    prompt = PROMPT.format(instruction=instruction, input=text)
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=100, do_sample=False,
                              pad_token_id=tokenizer.eos_token_id)
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()

print(predict("SELECT * FROM users WHERE id = 1 OR 1=1 --", task="sql"))
print(predict("http://paypa1-secure-login.com/verify", task="phishing"))
```

## Training Data

| Dataset | Source | Role |
| :--- | :--- | :--- |
| SQL Injection Dataset | [syedsaqlainhussain/sql-injection-dataset](https://www.kaggle.com/datasets/syedsaqlainhussain/sql-injection-dataset) (Kaggle) | Labeled SQL queries (benign / injection) |
| Phishing URL Dataset | [pirocheto/phishing-url](https://huggingface.co/datasets/pirocheto/phishing-url) (HuggingFace) | Labeled URLs (phishing / legitimate) |

Both sources were cleaned (leaked header rows and non-numeric label values
removed, deduplicated), converted to `instruction` / `input` / `output`
format, class-balanced to a max 3:1 ratio, and split 85/10/5 into
train/val/test. Training used a **3,000-example subset** of the train split
(and 300 of val) to fit a constrained compute budget — see
[Limitations](#limitations).

## Training Procedure

**Method:** QLoRA — base model loaded in 4-bit NF4, LoRA adapters trained on
top via plain `transformers.Trainer` (no `trl` dependency).

**LoRA target modules:** `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`

| Hyperparameter | Value |
| :--- | :--- |
| LoRA rank (r) | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Max sequence length | 256 |
| Per-device batch size | 8 |
| Gradient accumulation | 2 |
| Effective batch size | 16 |
| Learning rate | 2e-4 (cosine schedule) |
| Max steps | 300 |
| Precision | bf16 compute, 4-bit NF4 base weights |
| Hardware | 1x Kaggle Tesla T4 |

### Training Loss

| Step | Training Loss | Validation Loss |
| :--- | :--- | :--- |
| 100 | 0.1922 | 0.2044 |
| 200 | 0.1901 | 0.1922 |
| 300 | 0.1493 | 0.1896 |

**Final training run summary:**

| Metric | Value |
| :--- | :--- |
| Global steps | 300 |
| Epochs completed | ~1.6 |
| Average training loss | 0.2993 |
| Training runtime | 33,671s (~9.35 hours) |
| Samples/sec | 0.143 |
| Steps/sec | 0.009 |

Both training and validation loss decreased steadily with no signs of
divergence, but note that **loss going down does not by itself confirm
classification accuracy** — see Evaluation below.

## Evaluation

Evaluated on the held-out test split (1,532 examples) using exact-match
comparison between the model's generated `Classification:` label and ground
truth.

| Class | Precision | Recall | F1-score | Support |
| :--- | :---: | :---: | :---: | :---: |
| benign | 1.00 | 1.00 | 1.00 | 585 |
| legitimate | 0.99 | 0.96 | 0.97 | 203 |
| phishing | 0.96 | 0.99 | 0.97 | 182 |
| sql injection | 1.00 | 1.00 | 1.00 | 562 |
| **accuracy** | | | **0.99** | 1532 |
| macro avg | 0.99 | 0.99 | 0.99 | 1532 |
| weighted avg | 0.99 | 0.99 | 0.99 | 1532 |

**Overall test accuracy: 99%.** The SQL injection task (benign / sql
injection) is essentially perfect on this test split. The phishing task
(legitimate / phishing) is slightly softer, with legitimate URLs occasionally
misclassified as phishing (96% recall) and phishing URLs very reliably
caught (99% recall) — i.e., the model is a little more likely to over-flag a
legitimate URL than to miss an actual phishing one.

Note this reflects performance on a **held-out split of the same cleaned
dataset** used for training — it does not measure generalization to
attack patterns or URL structures outside that distribution (see
[Limitations](#limitations)).

## Limitations

- **Small training subset:** trained on 3,000 of the available examples (not
  the full cleaned dataset), and for only ~1.6 epochs, in order to fit a
  ~1-hour-scale compute budget on a single T4. This trades off ceiling
  accuracy for turnaround time — expect headroom for improvement with more
  data/epochs.
- **Templated explanations:** the `Reason:` text is class-templated rather
  than generated per-example, so explanations are somewhat generic rather
  than deeply input-specific.
- **No adversarial evaluation:** the 99% accuracy above is on a clean
  held-out split from the same source datasets. Obfuscated SQL payloads
  (encoding, comment tricks, case-mixing) and homoglyph/lookalike phishing
  domains were not specifically tested, and performance on those is unknown.
- **Long training time relative to budget:** the run took ~9.35 hours rather
  than the intended ~1 hour, most likely due to 7B-parameter 4-bit inference
  overhead plus gradient checkpointing on a single T4 — worth profiling
  further if iterating on this model.

## Citation

```bibtex
@misc{qwen2.5,
  title={Qwen2.5 Technical Report},
  author={Qwen Team},
  year={2024}
}
```

## Model Card Contact

[jayesh20](https://huggingface.co/jayesh20)