Qwen2.5-Coder-1.5B XSS Analyst (LoRA)

A QLoRA adapter for Qwen2.5-Coder-1.5B-Instruct that reads source code and reports cross-site scripting (XSS) vulnerabilities as a grounded JSON report: sources, sinks, the data-flow path, output context, severity, a fix, and a proof-of-concept for authorized testing. It also answers XSS questions in prose.

It covers Vue 3, Django and Spring Boot (Thymeleaf/JSP) code.

โš ๏ธ For education and authorized security testing only. Use the generated payloads only on systems you own or are explicitly authorized to test.

Benchmarks

Metrics: detection accuracy (vulnerable/safe decision), recall (vulnerabilities found), false-positive rate (safe code wrongly flagged; lower is better), grounded (every cited file/line/code exists in the input), valid JSON, and type/context accuracy on true positives.

Out-of-distribution set (36 hand-written examples)

Patterns the training generator never produces, such as document.write, insertAdjacentHTML, setTimeout strings, FreeMarker ?no_esc, unquoted attributes and sanitiser-then-raw output. This is the honest test of generalization. This adapter is the "chat/mix" column.

Metric Base model Scan-only adapter This adapter (chat/mix)
Valid JSON 66.7% 97.2% 97.2%
Grounded 22.2% 66.7% 83.3%
Detection accuracy 41.7% 55.6% 58.3%
Recall 100% 64.7% 70.6%
False-positive rate โ†“ 90.0% 50.0% 50.0%
Sink line (on true positives) 64.3% 100% 100%
XSS type accuracy 42.9% 54.5% 75.0%
Output context accuracy 35.7% 81.8% 83.3%

With only 36 examples each one moves a metric by about 3 points, so read small differences as noise.

Held-out synthetic test set (593 examples)

Metric Base model Scan-only adapter This adapter (chat/mix)
Valid JSON 65.3% 100% 100%
Grounded 13.7% 99.3% 100%
Detection accuracy 43.0% 100% 86.0%
Recall 97.5% 100% 87.0%
False-positive rate โ†“ 87.5% 0% 15.8%
Sink line / type / context (true positives) 75.1 / 61.2 / 38.8% 100 / 100 / 100% 100 / 100 / 100%

The test set uses the same 20 code templates as training, so the scan-only adapter's near-perfect scores show it learned those templates. They do not show general XSS detection ability; use the OOD table for that.

Adding the chat data cost this adapter three templates. All 83 of its errors come from patterns the scan-only adapter got right:

Template What goes wrong Wrong
Vue :href bound to route.query missed 25 / 25
Vue v-html bound to route.query missed 24 / 24
Django {{ var }} with autoescaping on (safe) wrongly flagged 34 / 41

The other 17 templates stay at 100%. These templates appear in the training mix as often as in stage-1 training, so this looks like the chat Q&A interfering with learned patterns, not missing data. If you only need scanning, the scan-only adapter is stronger on known patterns; this adapter is slightly better on unseen ones (OOD table above) and can also chat.

Limitations

  • About half of safe code gets flagged on unseen patterns. The model often reacts to a dangerous-looking API (e.g. v-html, mark_safe) without checking whether the input is attacker-controlled or already sanitised.
  • It misses javascript: URL sinks it hasn't seen (e.g. iframe src, format_html building an href).
  • It misses Vue :href/v-html fed from route.query and flags safe autoescaped Django output, three patterns the scan-only adapter handles (see the held-out table).
  • Prose answers follow templated phrasing. The chat data was generated from templates, so answers are accurate but scripted.
  • It over-refuses some legitimate requests for exploit code in free-form chat. Scanning the code is the reliable way to get a PoC.
  • Treat output as a first pass for a human to verify, not as a security audit.

How to use

The adapter was trained on a specific prompt format: a system prompt plus code where every line is prefixed with its line number. The simplest way to get that format is the repo's CLI, which loads this adapter straight from the Hub:

git clone https://github.com/hareramray/qwen-cyber && cd qwen-cyber
pip install -r requirements-train.txt
python -m xssft.chat --adapter hareramray325/qwen2.5-coder-1.5b-xss path/to/views.py path/to/template.html
python -m xssft.chat --adapter hareramray325/qwen2.5-coder-1.5b-xss   # interactive: scan, paste, or chat

To call it directly with transformers + peft:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "hareramray325/qwen2.5-coder-1.5b-xss")

# For scanning, build messages with xssft.schema.build_messages(files, language, framework)
# from the GitHub repo; it adds the system prompt and line numbers the model expects.
messages = [{"role": "user", "content": "What is DOM-based XSS?"}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Training

Two stages, both QLoRA on a single 8 GB GPU:

Stage 1: scan Stage 2: chat mix (warm-start)
Data 4,814 synthetic scan examples 6,432 = 4,814 scan + 1,618 chat Q&A
Epochs 2 1
Started from Fresh LoRA Stage-1 adapter weights
Time 2 h 35 min 2 h 9 min
  • Base model in 4-bit NF4 with double quantization and bf16 compute.
  • LoRA: r = 16, alpha = 32, dropout 0.05, on q/k/v/o and gate/up/down projections.
  • Optimizer and schedule: learning rate 2e-4 with cosine decay, 8-bit AdamW.
  • Batch: 1 ร— gradient accumulation 16; max length 2,048 tokens.
  • Loss: computed only on the answer tokens, never the prompt.
  • Peak GPU memory: 2.34 GiB.

The scan data is synthetic: generators write Vue/Django/Spring code whose vulnerability is known by construction, so every label and cited line is correct. About a third of examples are safe lookalikes, there to teach the model not to over-report.

License

MIT. The base model, Qwen2.5-Coder-1.5B-Instruct, is released under Apache 2.0.

Downloads last month
41
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for hareramray325/qwen2.5-coder-1.5b-xss

Adapter
(239)
this model

Dataset used to train hareramray325/qwen2.5-coder-1.5b-xss

Evaluation results

  • Detection accuracy on xss-detection-bench (ood split)
    self-reported
    58.300
  • Recall (vulnerable found) on xss-detection-bench (ood split)
    self-reported
    70.600
  • False-positive rate (lower is better) on xss-detection-bench (ood split)
    self-reported
    50.000
  • Grounded citations on xss-detection-bench (ood split)
    self-reported
    83.300
  • Valid JSON report on xss-detection-bench (ood split)
    self-reported
    97.200
  • XSS type accuracy on xss-detection-bench (ood split)
    self-reported
    75.000
  • Detection accuracy on xss-detection-bench (test split)
    test set self-reported
    86.000
  • Recall (vulnerable found) on xss-detection-bench (test split)
    test set self-reported
    87.000