Text Classification
Transformers
Safetensors
English
roberta
Generated from Trainer
classification
nlp
vulnerability
Eval Results (legacy)
text-embeddings-inference
Instructions to use CIRCL/vulnerability-severity-classification-roberta-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use CIRCL/vulnerability-severity-classification-roberta-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="CIRCL/vulnerability-severity-classification-roberta-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("CIRCL/vulnerability-severity-classification-roberta-base") model = AutoModelForSequenceClassification.from_pretrained("CIRCL/vulnerability-severity-classification-roberta-base", device_map="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -4,23 +4,41 @@ license: cc-by-4.0
|
|
| 4 |
base_model: roberta-base
|
| 5 |
metrics:
|
| 6 |
- accuracy
|
|
|
|
| 7 |
tags:
|
| 8 |
- generated_from_trainer
|
| 9 |
- text-classification
|
| 10 |
- classification
|
| 11 |
- nlp
|
| 12 |
- vulnerability
|
| 13 |
-
model-index:
|
| 14 |
-
- name: vulnerability-severity-classification-roberta-base
|
| 15 |
-
results: []
|
| 16 |
datasets:
|
| 17 |
- CIRCL/vulnerability-scores
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
---
|
| 19 |
|
| 20 |
|
| 21 |
# VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
|
| 22 |
|
| 23 |
-
# Severity classification
|
| 24 |
|
| 25 |
This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
|
| 26 |
|
|
@@ -28,6 +46,14 @@ The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated
|
|
| 28 |
|
| 29 |
**Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
|
| 30 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
|
| 32 |
|
| 33 |
|
|
@@ -35,14 +61,21 @@ You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) f
|
|
| 35 |
|
| 36 |
It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
|
| 37 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
## How to get started with the model
|
| 39 |
|
| 40 |
```python
|
| 41 |
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 42 |
import torch
|
| 43 |
|
| 44 |
-
labels = ["low", "medium", "high", "critical"]
|
| 45 |
-
|
| 46 |
model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
|
| 47 |
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 48 |
model = AutoModelForSequenceClassification.from_pretrained(model_name)
|
|
@@ -62,9 +95,26 @@ with torch.no_grad():
|
|
| 62 |
# Print results
|
| 63 |
print("Predictions:", predictions)
|
| 64 |
predicted_class = torch.argmax(predictions, dim=-1).item()
|
| 65 |
-
print("Predicted severity:",
|
| 66 |
```
|
| 67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
## Training procedure
|
| 69 |
|
| 70 |
### Training hyperparameters
|
|
@@ -78,25 +128,31 @@ The following hyperparameters were used during training:
|
|
| 78 |
- lr_scheduler_type: linear
|
| 79 |
- num_epochs: 5
|
| 80 |
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
-
|
| 89 |
-
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
-
|
| 93 |
-
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
|
| 98 |
### Training results
|
| 99 |
|
|
|
|
|
|
|
| 100 |
| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
|
| 101 |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
|
| 102 |
| 2.5082 | 1.0 | 17749 | 2.7143 | 0.7229 | 0.6180 | 0.6764 | 0.2567 | 0.3722 | 0.8047 | 0.7647 | 0.7842 | 0.6872 | 0.7424 | 0.7137 | 0.5700 | 0.6374 | 0.6018 |
|
|
@@ -105,6 +161,11 @@ It achieves the following results on the evaluation set:
|
|
| 105 |
| 1.8782 | 4.0 | 70996 | 2.7383 | 0.7500 | 0.6608 | 0.6555 | 0.3622 | 0.4665 | 0.7892 | 0.8279 | 0.8081 | 0.7315 | 0.7352 | 0.7333 | 0.6601 | 0.6120 | 0.6351 |
|
| 106 |
| 1.4131 | 5.0 | 88745 | 3.0105 | 0.7457 | 0.6639 | 0.5635 | 0.4309 | 0.4884 | 0.7916 | 0.8126 | 0.8019 | 0.7271 | 0.7365 | 0.7317 | 0.6589 | 0.6101 | 0.6336 |
|
| 107 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
|
| 109 |
### Framework versions
|
| 110 |
|
|
@@ -112,3 +173,26 @@ It achieves the following results on the evaluation set:
|
|
| 112 |
- Pytorch 2.15.0+cu132
|
| 113 |
- Datasets 4.8.5
|
| 114 |
- Tokenizers 0.23.3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
base_model: roberta-base
|
| 5 |
metrics:
|
| 6 |
- accuracy
|
| 7 |
+
- f1
|
| 8 |
tags:
|
| 9 |
- generated_from_trainer
|
| 10 |
- text-classification
|
| 11 |
- classification
|
| 12 |
- nlp
|
| 13 |
- vulnerability
|
|
|
|
|
|
|
|
|
|
| 14 |
datasets:
|
| 15 |
- CIRCL/vulnerability-scores
|
| 16 |
+
model-index:
|
| 17 |
+
- name: vulnerability-severity-classification-roberta-base
|
| 18 |
+
results:
|
| 19 |
+
- task:
|
| 20 |
+
type: text-classification
|
| 21 |
+
name: Vulnerability severity classification
|
| 22 |
+
dataset:
|
| 23 |
+
type: CIRCL/vulnerability-scores
|
| 24 |
+
name: CIRCL/vulnerability-scores
|
| 25 |
+
split: test
|
| 26 |
+
revision: 55a886a483bb27aabb6461edc64e8058cc136e24
|
| 27 |
+
metrics:
|
| 28 |
+
- type: accuracy
|
| 29 |
+
value: 0.7415
|
| 30 |
+
name: Accuracy
|
| 31 |
+
- type: f1
|
| 32 |
+
value: 0.6460
|
| 33 |
+
name: F1 Macro
|
| 34 |
+
args:
|
| 35 |
+
average: macro
|
| 36 |
---
|
| 37 |
|
| 38 |
|
| 39 |
# VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
|
| 40 |
|
| 41 |
+
## Severity classification
|
| 42 |
|
| 43 |
This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
|
| 44 |
|
|
|
|
| 46 |
|
| 47 |
**Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
|
| 48 |
|
| 49 |
+
> **Note (October 2026):** since [VulnTrain 3.4.0](https://github.com/vulnerability-lookup/VulnTrain/releases/tag/v3.4.0),
|
| 50 |
+
> the dataset is split so that records of the same vulnerability (for example a CVE and its
|
| 51 |
+
> GitHub advisory) never appear in both train and test. Accuracy measured this way (**74%**)
|
| 52 |
+
> reflects performance on vulnerabilities the model has never seen. Earlier figures, including
|
| 53 |
+
> the one in the abstract above, were measured on a split where many test descriptions also
|
| 54 |
+
> appeared in training. Details in the technical report
|
| 55 |
+
> [Improving vulnerability-scores: Leakage-Free Splits and Consistent CVSS 4.0 Scoring](https://doi.org/10.5281/zenodo.23760802).
|
| 56 |
+
|
| 57 |
You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
|
| 58 |
|
| 59 |
|
|
|
|
| 61 |
|
| 62 |
It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
|
| 63 |
|
| 64 |
+
Labels:
|
| 65 |
+
|
| 66 |
+
| Label | CVSS score |
|
| 67 |
+
|-------|------------|
|
| 68 |
+
| Low | < 4.0 |
|
| 69 |
+
| Medium | 4.0 – 6.9 |
|
| 70 |
+
| High | 7.0 – 8.9 |
|
| 71 |
+
| Critical | ≥ 9.0 |
|
| 72 |
+
|
| 73 |
## How to get started with the model
|
| 74 |
|
| 75 |
```python
|
| 76 |
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 77 |
import torch
|
| 78 |
|
|
|
|
|
|
|
| 79 |
model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
|
| 80 |
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
| 81 |
model = AutoModelForSequenceClassification.from_pretrained(model_name)
|
|
|
|
| 95 |
# Print results
|
| 96 |
print("Predictions:", predictions)
|
| 97 |
predicted_class = torch.argmax(predictions, dim=-1).item()
|
| 98 |
+
print("Predicted severity:", model.config.id2label[predicted_class])
|
| 99 |
```
|
| 100 |
|
| 101 |
+
## Training data
|
| 102 |
+
|
| 103 |
+
Revision [`55a886a`](https://huggingface.co/datasets/CIRCL/vulnerability-scores/tree/55a886a483bb27aabb6461edc64e8058cc136e24)
|
| 104 |
+
of [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores),
|
| 105 |
+
built with VulnTrain 3.4.0 from CVE records, GitHub, PySec and CSAF (Red Hat, Cisco, CISA) advisories:
|
| 106 |
+
|
| 107 |
+
| Split | Rows | Labelled rows |
|
| 108 |
+
|-------|-----:|--------------:|
|
| 109 |
+
| train | 670,283 | 567,964 |
|
| 110 |
+
| validation | 83,550 | 71,470 |
|
| 111 |
+
| test | 83,376 | 70,472 |
|
| 112 |
+
|
| 113 |
+
- **Group-aware split (80/10/10).** Rows that share a description, an identifier or an alias are
|
| 114 |
+
kept in the same split, so no vulnerability appears in both train and test.
|
| 115 |
+
- **Labels** come from the first available CVSS score, in the order v4.0, v3.1, v3.0, v2.0.
|
| 116 |
+
CVSS 4.0 scores are read with base metrics only (CVSS-B), whatever the source.
|
| 117 |
+
|
| 118 |
## Training procedure
|
| 119 |
|
| 120 |
### Training hyperparameters
|
|
|
|
| 128 |
- lr_scheduler_type: linear
|
| 129 |
- num_epochs: 5
|
| 130 |
|
| 131 |
+
The checkpoint with the best validation accuracy (epoch 4) is the one published.
|
| 132 |
+
|
| 133 |
+
Training took 4 h 25 min on 4 × NVIDIA L40S and used 4.61 kWh, for 0.48 kg CO₂eq
|
| 134 |
+
(measured with CodeCarbon, see [`emissions.csv`](emissions.csv)).
|
| 135 |
+
|
| 136 |
+
### Results
|
| 137 |
+
|
| 138 |
+
Evaluated once on the leakage-free test split of
|
| 139 |
+
[CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores/tree/55a886a483bb27aabb6461edc64e8058cc136e24)
|
| 140 |
+
(revision `55a886a`, 70,472 labelled rows):
|
| 141 |
+
|
| 142 |
+
- Accuracy: **0.7415** (95% CI 0.7383 – 0.7447)
|
| 143 |
+
- F1 Macro: 0.6460
|
| 144 |
+
|
| 145 |
+
| Class | Precision | Recall | F1 |
|
| 146 |
+
|-------|----------:|-------:|---:|
|
| 147 |
+
| Low | 0.6146 | 0.3280 | 0.4278 |
|
| 148 |
+
| Medium | 0.7767 | 0.8251 | 0.8002 |
|
| 149 |
+
| High | 0.7328 | 0.7237 | 0.7282 |
|
| 150 |
+
| Critical | 0.6460 | 0.6107 | 0.6279 |
|
| 151 |
|
| 152 |
### Training results
|
| 153 |
|
| 154 |
+
Validation metrics at each epoch:
|
| 155 |
+
|
| 156 |
| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
|
| 157 |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
|
| 158 |
| 2.5082 | 1.0 | 17749 | 2.7143 | 0.7229 | 0.6180 | 0.6764 | 0.2567 | 0.3722 | 0.8047 | 0.7647 | 0.7842 | 0.6872 | 0.7424 | 0.7137 | 0.5700 | 0.6374 | 0.6018 |
|
|
|
|
| 161 |
| 1.8782 | 4.0 | 70996 | 2.7383 | 0.7500 | 0.6608 | 0.6555 | 0.3622 | 0.4665 | 0.7892 | 0.8279 | 0.8081 | 0.7315 | 0.7352 | 0.7333 | 0.6601 | 0.6120 | 0.6351 |
|
| 162 |
| 1.4131 | 5.0 | 88745 | 3.0105 | 0.7457 | 0.6639 | 0.5635 | 0.4309 | 0.4884 | 0.7916 | 0.8126 | 0.8019 | 0.7271 | 0.7365 | 0.7317 | 0.6589 | 0.6101 | 0.6336 |
|
| 163 |
|
| 164 |
+
### Previous versions
|
| 165 |
+
|
| 166 |
+
The previous model, trained on the earlier random split, is revision
|
| 167 |
+
[`10d9861`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/tree/10d9861)
|
| 168 |
+
(dataset revision `34e6a14`, test accuracy 81.17%, 73.02% on the test rows without a twin in train).
|
| 169 |
|
| 170 |
### Framework versions
|
| 171 |
|
|
|
|
| 173 |
- Pytorch 2.15.0+cu132
|
| 174 |
- Datasets 4.8.5
|
| 175 |
- Tokenizers 0.23.3
|
| 176 |
+
|
| 177 |
+
## Citation
|
| 178 |
+
|
| 179 |
+
```bibtex
|
| 180 |
+
@article{bonhomme2025vlai,
|
| 181 |
+
title = {{VLAI}: A {RoBERTa}-Based Model for Automated Vulnerability
|
| 182 |
+
Severity Classification},
|
| 183 |
+
author = {Bonhomme, C{\'e}dric and Dulaunoy, Alexandre},
|
| 184 |
+
journal = {arXiv preprint arXiv:2507.03607},
|
| 185 |
+
year = {2025}
|
| 186 |
+
}
|
| 187 |
+
|
| 188 |
+
@techreport{bonhomme2026vulnscores,
|
| 189 |
+
title = {Improving vulnerability-scores: Leakage-Free Splits and
|
| 190 |
+
Consistent {CVSS} 4.0 Scoring},
|
| 191 |
+
author = {Bonhomme, C{\'e}dric},
|
| 192 |
+
institution = {CIRCL - Computer Incident Response Center Luxembourg},
|
| 193 |
+
year = {2026},
|
| 194 |
+
month = oct,
|
| 195 |
+
doi = {10.5281/zenodo.23760802},
|
| 196 |
+
url = {https://doi.org/10.5281/zenodo.23760802}
|
| 197 |
+
}
|
| 198 |
+
```
|