VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification

Severity classification

This model is a fine-tuned version of roberta-base on the dataset CIRCL/vulnerability-scores.

The model was presented in the paper VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification [arXiv].

Abstract: VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.

Note: since VulnTrain 3.4.0, the dataset is split so that records of the same vulnerability (for example a CVE and its GitHub advisory) never appear in both train and test. Accuracy measured this way (74%) reflects performance on vulnerabilities the model has never seen. Earlier figures, including the one in the abstract above, were measured on a split where many test descriptions also appeared in training. Details in the technical report Improving vulnerability-scores: Leakage-Free Splits and Consistent CVSS 4.0 Scoring.

You can read this page for more information.

Model description

It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.

Labels:

Label CVSS score
Low < 4.0
Medium 4.0 – 6.9
High 7.0 – 8.9
Critical β‰₯ 9.0

How to get started with the model

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
revision = "b868a7d"
tokenizer = AutoTokenizer.from_pretrained(model_name, revision=revision)
model = AutoModelForSequenceClassification.from_pretrained(model_name, revision=revision)
model.eval()

test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)

# Run inference
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)

# Print results
print("Predictions:", predictions)
predicted_class = torch.argmax(predictions, dim=-1).item()
print("Predicted severity:", model.config.id2label[predicted_class])

Training data

Revision 58b4833 of CIRCL/vulnerability-scores, built with VulnTrain from CVE records, GitHub, PySec and CSAF (Red Hat, Cisco, CISA) advisories:

Split Rows Labelled rows
train 686,695 584,245
validation 85,405 73,419
test 85,167 72,240
  • Group-aware split (80/10/10). Rows that share a description, an identifier or an alias are kept in the same split, so no vulnerability appears in both train and test.
  • Labels come from the first available CVSS score, in the order v4.0, v3.1, v3.0, v2.0. CVSS 4.0 scores are read with base metrics only (CVSS-B), whatever the source.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 3e-05
  • train_batch_size: 32
  • eval_batch_size: 32
  • seed: 42
  • optimizer: adamw_torch_fused
  • lr_scheduler_type: linear
  • warmup_steps: 0
  • weight_decay: 0.01
  • num_epochs: 5

The checkpoint with the best validation accuracy (epoch 3) is the one published.

Training took 4 h 32 min on 4 x NVIDIA L40S and used 4.74 kWh, for 0.50 kg COβ‚‚eq (measured with CodeCarbon, see emissions.csv).

Results

Evaluated once on the leakage-free test split (72,240 labelled rows):

  • Accuracy: 0.7419 (95% CI 0.7387 – 0.7451)
  • F1 Macro: 0.6496
Class Precision Recall F1 Support
Low 0.5947 0.3563 0.4456 2,599
Medium 0.7790 0.8162 0.7971 32,602
High 0.7262 0.7409 0.7335 28,209
Critical 0.6659 0.5839 0.6222 8,830

Training results

Validation metrics at each epoch:

Epoch Step Training Loss Validation Loss Accuracy F1 Macro
1.0 18258 2.3938 2.6531 0.7289 0.6162
2.0 36516 2.1821 2.5989 0.7411 0.6482
3.0 54774 1.8636 2.6522 0.7488 0.6568
4.0 73032 1.6322 2.9344 0.7481 0.6566
5.0 91290 1.3337 3.1119 0.7473 0.6622

Previous versions

The previous model, trained on the earlier random split, is revision 10d9861 (dataset revision 34e6a14, test accuracy 81.17%, 73.02% on the test rows without a twin in train).

Framework versions

  • Transformers 5.22.2
  • Pytorch 2.15.0
  • Datasets 4.8.5
  • Tokenizers 0.23.3

Provenance

This model card is generated by VulnTrain 3.5.0 at the end of the training run, from the values the run measured.

  • bom.cdx.json: machine-readable record of the run (CycloneDX ML-BOM): base model, dataset revision, metrics, training options and energy use.
  • emissions.csv: energy use and COβ‚‚ emissions of the training, measured with CodeCarbon.

Citation

@article{bonhomme2025vlai,
  title   = {{VLAI}: A {RoBERTa}-Based Model for Automated Vulnerability
             Severity Classification},
  author  = {Bonhomme, C{\'e}dric and Dulaunoy, Alexandre},
  journal = {arXiv preprint arXiv:2507.03607},
  year    = {2025}
}

@techreport{bonhomme2026vulnscores,
  title       = {Improving vulnerability-scores: Leakage-Free Splits and
                 Consistent {CVSS} 4.0 Scoring},
  author      = {Bonhomme, C{\'e}dric},
  institution = {CIRCL - Computer Incident Response Center Luxembourg},
  year        = {2026},
  month       = oct,
  doi         = {10.5281/zenodo.23760802},
  url         = {https://doi.org/10.5281/zenodo.23760802}
}
Downloads last month
14,028
Safetensors
Model size
0.1B params
Tensor type
F32
Β·
Inference Providers NEW

Model tree for CIRCL/vulnerability-severity-classification-roberta-base

Finetuned
(2,423)
this model

Dataset used to train CIRCL/vulnerability-severity-classification-roberta-base

Spaces using CIRCL/vulnerability-severity-classification-roberta-base 2

Collection including CIRCL/vulnerability-severity-classification-roberta-base

Paper for CIRCL/vulnerability-severity-classification-roberta-base

Evaluation results