sijjeel/js-vuln-unixcoder

UniXcoder fine-tuned for JavaScript vulnerability detection

A fine-tuned UniXcoder classifier that estimates whether a single JavaScript function contains a security vulnerability. It outputs two labels: 0 = safe, 1 = vulnerable.

Read this first. The model performs well on code that resembles its training distribution but does not generalise to unfamiliar codebases (cross-dataset AUC ≈ 0.50, i.e. near random). It is a first-pass screening aid for human review, not a security guarantee. Do not use it to certify code as safe.

This model was produced for an MSc Artificial Intelligence project at the University of Hertfordshire, as part of a comparative study of code language models (CodeBERT, GraphCodeBERT, UniXcoder) for JavaScript vulnerability detection. UniXcoder was the strongest of the three on in-distribution data.

Model Details

Model Description

• Developed by: Muhammad Sijjeel (MSc Artificial Intelligence, University of Hertfordshire)
• Model type: Encoder-based sequence classifier (AutoModelForSequenceClassification, num_labels=2)
• Language(s): JavaScript source code
• License: MIT
• Finetuned from model: microsoft/unixcoder-base

Model Sources

• Base model: https://huggingface.co/microsoft/unixcoder-base
• Base model paper: Guo et al. (2022), UniXcoder: Unified Cross-Modal Pre-training for Code Representation, ACL 2022. doi:10.18653/v1/2022.acl-long.499

Uses

Direct Use

Pass a single JavaScript function as text; the model returns a probability of the function being vulnerable. Intended as a screening/triage aid that flags candidate functions for a human reviewer.

Out-of-Scope Use

• Not a security guarantee or a substitute for human code review, static analysis, or penetration testing.
• Not reliable on code drawn from a different distribution than the training data (see Evaluation). An external cross-dataset test scored AUC ≈ 0.50.
• Not suitable as an automated gate that certifies code as safe. A confident "safe" output on unfamiliar code is not trustworthy.
• Trained on function-level inputs only; not intended for whole files, repos, or languages other than JavaScript.

Bias, Risks, and Limitations

• Cross-dataset generalisation fails. On the external CrossVul set the model scores near random. This was stress-tested and confirmed to be genuine distribution shift, not a label, truncation, or evaluation artefact.
• Vulnerability-type bias. The training data was skewed toward cross-site scripting (CWE-79); even after capping, the model is likely stronger on web-injection patterns than on rarer vulnerability types.
• Sequence-length limit. Inputs are truncated to 256 tokens. Functions longer than this are only partially read, and the vulnerable line may fall outside the window (training median ≈ 227 tokens; some evaluation functions far exceed 256).
• False-assurance risk. As a security tool, an over-trusted "safe" label could let a real vulnerability through. Always keep a human in the loop.

Recommendations

Use the probability, not just the label, and treat scores near the threshold as "uncertain". Restrict use to human-in-the-loop screening. Do not deploy as an automated pass/fail check on arbitrary repositories.

How to Get Started with the Model

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForSequenceClassification

REPO = "sijjeel/js-vuln-unixcoder"  # replace with your repo id if different
tokenizer = AutoTokenizer.from_pretrained("microsoft/unixcoder-base")
model = AutoModelForSequenceClassification.from_pretrained(REPO).eval()

code = """function renderComment(req, res) {
  document.getElementById("c").innerHTML += "<p>" + req.query.text + "</p>";
}"""

enc = tokenizer(code, truncation=True, max_length=256, return_tensors="pt")
with torch.no_grad():
    prob_vulnerable = F.softmax(model(**enc).logits, dim=-1)[0, 1].item()

print(f"P(vulnerable) = {prob_vulnerable:.3f}")

Training Details

Training Data

• Derived from CVEfixes (Bhandari, Naseer & Moonen, 2021) plus an additional real JavaScript vulnerability dataset (Guo & Khan, 2024, MIT-licensed).
• Function-level examples: before-fix = vulnerable, after-fix = safe.
• Leakage-controlled: label conflicts, cross-split leakage, trivial snippets, duplicates, minified one-liners, and extreme-length functions removed via whitespace-normalised MD5 hashing.
• The dominant XSS (CWE-79) class was capped to preserve vulnerability-type diversity.
• Balanced training set: ~10,000 functions (50% vulnerable), split 70 / 15 / 15 into train (6,990) / validation (1,498) / test (1,499). Splits verified disjoint (exact and whitespace-normalised).

Training Procedure

Training Hyperparameters

• Training regime: FP16 mixed precision
• Learning rate 1e-5, batch size 4, 3 epochs, max sequence length 256
• Weight decay 0.1, warmup ratio 0.1, seed 42
• Optimiser: AdamW (Loshchilov & Hutter, 2019)
• Class-weighted cross-entropy loss (custom WeightedTrainer); early stopping patience 2; best checkpoint selected by validation F1
• On the balanced set the class weights self-adjust to ≈ 0.999 / 1.001 (effectively inactive; no oversampling applied)

Compute

• Kaggle Notebooks, NVIDIA Tesla T4 GPU; PyTorch + HuggingFace Transformers

Evaluation

Testing Data, Factors & Metrics

• In-distribution: balanced held-out test set (1,499 functions, 50% vulnerable).
• Cross-dataset (generalisation): external CrossVul set (432 functions, balanced), never seen in training.
• Metrics: AUC-ROC (threshold-independent; primary on balanced data), F1, precision, recall.

Results

In-distribution (balanced test set, threshold 0.5):

Model	Accuracy	Precision	Recall	F1	AUC-ROC
UniXcoder (this model)	0.843	0.823	0.875	0.848	0.923
GraphCodeBERT	0.841	0.831	0.857	0.844	0.914
CodeBERT	0.825	0.807	0.853	0.829	0.904
TF-IDF + LogReg (baseline)	0.760	0.748	0.784	0.765	0.831

Cross-dataset (external CrossVul set): AUC-ROC ≈ 0.49 — near random. Verified as genuine distribution shift (label orientation, truncation at max-length 256 vs 512, and evaluation method all ruled out).

Summary

UniXcoder is the strongest of the three encoders on in-distribution JavaScript (F1 ≈ 0.85, AUC ≈ 0.92) and is statistically indistinguishable from GraphCodeBERT, while both significantly outperform CodeBERT. None of the models generalise to an independently collected dataset — the honest, central finding of the study.

Environmental Impact

• Hardware Type: NVIDIA Tesla T4 (Kaggle)
• Cloud Provider: Kaggle
• Emissions not formally measured; training was short (3 epochs, ~125M-parameter model on a single T4).

Technical Specifications

Model Architecture and Objective

UniXcoder encoder with a sequence-classification head (2 labels), fine-tuned with class-weighted cross-entropy for binary vulnerable/safe classification of JavaScript functions.

Citation

Base model:

BibTeX:

@inproceedings{guo2022unixcoder,
  title     = {UniXcoder: Unified Cross-Modal Pre-training for Code Representation},
  author    = {Guo, Daya and Lu, Shuai and Duan, Nan and Wang, Yanlin and Zhou, Ming and Yin, Jian},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages     = {7212--7225},
  year      = {2022},
  doi       = {10.18653/v1/2022.acl-long.499}
}

Model Card Authors

Muhammad Sijjeel — MSc Artificial Intelligence, University of Hertfordshire.

Model Card Contact

Via the HuggingFace repository discussions.