Text Classification
Transformers
Safetensors
English
roberta
Generated from Trainer
classification
nlp
vulnerability
Eval Results (legacy)
text-embeddings-inference
Instructions to use CIRCL/vulnerability-severity-classification-roberta-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use CIRCL/vulnerability-severity-classification-roberta-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="CIRCL/vulnerability-severity-classification-roberta-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("CIRCL/vulnerability-severity-classification-roberta-base") model = AutoModelForSequenceClassification.from_pretrained("CIRCL/vulnerability-severity-classification-roberta-base", device_map="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
Update model card with evaluation metrics
Browse files
README.md
CHANGED
|
@@ -1,50 +1,120 @@
|
|
| 1 |
---
|
|
|
|
|
|
|
| 2 |
library_name: transformers
|
| 3 |
-
license:
|
| 4 |
base_model: roberta-base
|
| 5 |
-
|
| 6 |
-
- generated_from_trainer
|
| 7 |
metrics:
|
| 8 |
- accuracy
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
model-index:
|
| 10 |
- name: vulnerability-severity-classification-roberta-base
|
| 11 |
-
results:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
---
|
| 13 |
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
- Critical Recall: 0.5999
|
| 35 |
-
- Critical F1: 0.6263
|
| 36 |
|
| 37 |
## Model description
|
| 38 |
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
-
|
|
|
|
|
|
|
| 42 |
|
| 43 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
|
| 45 |
-
|
|
|
|
|
|
|
| 46 |
|
| 47 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
## Training procedure
|
| 50 |
|
|
@@ -55,24 +125,82 @@ The following hyperparameters were used during training:
|
|
| 55 |
- train_batch_size: 32
|
| 56 |
- eval_batch_size: 32
|
| 57 |
- seed: 42
|
| 58 |
-
- optimizer:
|
| 59 |
- lr_scheduler_type: linear
|
|
|
|
|
|
|
| 60 |
- num_epochs: 5
|
| 61 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
### Training results
|
| 63 |
|
| 64 |
-
|
| 65 |
-
|:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
|
| 66 |
-
| 2.3938 | 1.0 | 18258 | 2.6531 | 0.7289 | 0.6162 | 0.6395 | 0.2866 | 0.3958 | 0.7740 | 0.8113 | 0.7922 | 0.6914 | 0.7502 | 0.7196 | 0.6735 | 0.4752 | 0.5572 |
|
| 67 |
-
| 2.1821 | 2.0 | 36516 | 2.5989 | 0.7411 | 0.6482 | 0.6092 | 0.3464 | 0.4417 | 0.7798 | 0.8252 | 0.8019 | 0.7384 | 0.7106 | 0.7243 | 0.6171 | 0.6333 | 0.6251 |
|
| 68 |
-
| 1.8636 | 3.0 | 54774 | 2.6522 | 0.7488 | 0.6568 | 0.6077 | 0.3742 | 0.4632 | 0.7927 | 0.8197 | 0.8059 | 0.7243 | 0.7522 | 0.7380 | 0.6710 | 0.5767 | 0.6202 |
|
| 69 |
-
| 1.6322 | 4.0 | 73032 | 2.9344 | 0.7481 | 0.6566 | 0.6726 | 0.3430 | 0.4543 | 0.8058 | 0.8012 | 0.8035 | 0.7143 | 0.7647 | 0.7387 | 0.6495 | 0.6114 | 0.6299 |
|
| 70 |
-
| 1.3337 | 5.0 | 91290 | 3.1119 | 0.7473 | 0.6622 | 0.5590 | 0.4226 | 0.4813 | 0.7981 | 0.8080 | 0.8030 | 0.7260 | 0.7507 | 0.7382 | 0.6551 | 0.5999 | 0.6263 |
|
| 71 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
|
| 73 |
### Framework versions
|
| 74 |
|
| 75 |
- Transformers 5.22.2
|
| 76 |
-
- Pytorch 2.15.0
|
| 77 |
- Datasets 4.8.5
|
| 78 |
- Tokenizers 0.23.3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
library_name: transformers
|
| 5 |
+
license: cc-by-4.0
|
| 6 |
base_model: roberta-base
|
| 7 |
+
pipeline_tag: text-classification
|
|
|
|
| 8 |
metrics:
|
| 9 |
- accuracy
|
| 10 |
+
- f1
|
| 11 |
+
tags:
|
| 12 |
+
- generated_from_trainer
|
| 13 |
+
- text-classification
|
| 14 |
+
- classification
|
| 15 |
+
- nlp
|
| 16 |
+
- vulnerability
|
| 17 |
+
datasets:
|
| 18 |
+
- CIRCL/vulnerability-scores
|
| 19 |
model-index:
|
| 20 |
- name: vulnerability-severity-classification-roberta-base
|
| 21 |
+
results:
|
| 22 |
+
- task:
|
| 23 |
+
type: text-classification
|
| 24 |
+
name: Vulnerability severity classification
|
| 25 |
+
dataset:
|
| 26 |
+
type: CIRCL/vulnerability-scores
|
| 27 |
+
name: CIRCL/vulnerability-scores
|
| 28 |
+
split: test
|
| 29 |
+
revision: 58b483351be74f81ece78ff255a329027a6a8863
|
| 30 |
+
metrics:
|
| 31 |
+
- type: accuracy
|
| 32 |
+
value: 0.7419
|
| 33 |
+
name: Accuracy
|
| 34 |
+
- type: f1
|
| 35 |
+
value: 0.6496
|
| 36 |
+
name: F1 Macro
|
| 37 |
+
args:
|
| 38 |
+
average: macro
|
| 39 |
---
|
| 40 |
|
| 41 |
+
# VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
|
| 42 |
+
|
| 43 |
+
## Severity classification
|
| 44 |
+
|
| 45 |
+
This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
|
| 46 |
+
|
| 47 |
+
The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification](https://huggingface.co/papers/2507.03607) [[arXiv](https://arxiv.org/abs/2507.03607)].
|
| 48 |
+
|
| 49 |
+
**Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
|
| 50 |
+
|
| 51 |
+
> **Note:** since [VulnTrain 3.4.0](https://github.com/vulnerability-lookup/VulnTrain/releases/tag/v3.4.0),
|
| 52 |
+
> the dataset is split so that records of the same vulnerability (for example a CVE and its
|
| 53 |
+
> GitHub advisory) never appear in both train and test. Accuracy measured this way (**74%**)
|
| 54 |
+
> reflects performance on vulnerabilities the model has never seen. Earlier figures, including
|
| 55 |
+
> the one in the abstract above, were measured on a split where many test descriptions also
|
| 56 |
+
> appeared in training. Details in the technical report
|
| 57 |
+
> [Improving vulnerability-scores: Leakage-Free Splits and Consistent CVSS 4.0 Scoring](https://doi.org/10.5281/zenodo.23760802).
|
| 58 |
+
|
| 59 |
+
You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
|
| 60 |
+
|
|
|
|
|
|
|
| 61 |
|
| 62 |
## Model description
|
| 63 |
|
| 64 |
+
It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
|
| 65 |
+
|
| 66 |
+
Labels:
|
| 67 |
+
|
| 68 |
+
| Label | CVSS score |
|
| 69 |
+
|-------|------------|
|
| 70 |
+
| Low | < 4.0 |
|
| 71 |
+
| Medium | 4.0 – 6.9 |
|
| 72 |
+
| High | 7.0 – 8.9 |
|
| 73 |
+
| Critical | ≥ 9.0 |
|
| 74 |
+
|
| 75 |
+
## How to get started with the model
|
| 76 |
|
| 77 |
+
```python
|
| 78 |
+
from transformers import AutoModelForSequenceClassification, AutoTokenizer
|
| 79 |
+
import torch
|
| 80 |
|
| 81 |
+
model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
|
| 82 |
+
revision = "b868a7d"
|
| 83 |
+
tokenizer = AutoTokenizer.from_pretrained(model_name, revision=revision)
|
| 84 |
+
model = AutoModelForSequenceClassification.from_pretrained(model_name, revision=revision)
|
| 85 |
+
model.eval()
|
| 86 |
|
| 87 |
+
test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
|
| 88 |
+
that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
|
| 89 |
+
inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)
|
| 90 |
|
| 91 |
+
# Run inference
|
| 92 |
+
with torch.no_grad():
|
| 93 |
+
outputs = model(**inputs)
|
| 94 |
+
predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
|
| 95 |
+
|
| 96 |
+
# Print results
|
| 97 |
+
print("Predictions:", predictions)
|
| 98 |
+
predicted_class = torch.argmax(predictions, dim=-1).item()
|
| 99 |
+
print("Predicted severity:", model.config.id2label[predicted_class])
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
## Training data
|
| 103 |
+
|
| 104 |
+
Revision [`58b4833`](https://huggingface.co/datasets/CIRCL/vulnerability-scores/tree/58b483351be74f81ece78ff255a329027a6a8863)
|
| 105 |
+
of [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores),
|
| 106 |
+
built with VulnTrain from CVE records, GitHub, PySec and CSAF (Red Hat, Cisco, CISA) advisories:
|
| 107 |
+
|
| 108 |
+
| Split | Rows | Labelled rows |
|
| 109 |
+
|-------|-----:|--------------:|
|
| 110 |
+
| train | 686,695 | 584,245 |
|
| 111 |
+
| validation | 85,405 | 73,419 |
|
| 112 |
+
| test | 85,167 | 72,240 |
|
| 113 |
+
|
| 114 |
+
- **Group-aware split (80/10/10).** Rows that share a description, an identifier or an alias are
|
| 115 |
+
kept in the same split, so no vulnerability appears in both train and test.
|
| 116 |
+
- **Labels** come from the first available CVSS score, in the order v4.0, v3.1, v3.0, v2.0.
|
| 117 |
+
CVSS 4.0 scores are read with base metrics only (CVSS-B), whatever the source.
|
| 118 |
|
| 119 |
## Training procedure
|
| 120 |
|
|
|
|
| 125 |
- train_batch_size: 32
|
| 126 |
- eval_batch_size: 32
|
| 127 |
- seed: 42
|
| 128 |
+
- optimizer: adamw_torch_fused
|
| 129 |
- lr_scheduler_type: linear
|
| 130 |
+
- warmup_steps: 0
|
| 131 |
+
- weight_decay: 0.01
|
| 132 |
- num_epochs: 5
|
| 133 |
|
| 134 |
+
The checkpoint with the best validation accuracy (epoch 3) is the one published.
|
| 135 |
+
|
| 136 |
+
Training took 4 h 32 min on 4 x NVIDIA L40S and used 4.74 kWh, for 0.50 kg CO₂eq
|
| 137 |
+
(measured with CodeCarbon, see [`emissions.csv`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/blob/main/emissions.csv)).
|
| 138 |
+
|
| 139 |
+
### Results
|
| 140 |
+
|
| 141 |
+
Evaluated once on the leakage-free test split (72,240 labelled rows):
|
| 142 |
+
|
| 143 |
+
- Accuracy: **0.7419** (95% CI 0.7387 – 0.7451)
|
| 144 |
+
- F1 Macro: 0.6496
|
| 145 |
+
|
| 146 |
+
| Class | Precision | Recall | F1 | Support |
|
| 147 |
+
|-------|----------:|-------:|---:|--------:|
|
| 148 |
+
| Low | 0.5947 | 0.3563 | 0.4456 | 2,599 |
|
| 149 |
+
| Medium | 0.7790 | 0.8162 | 0.7971 | 32,602 |
|
| 150 |
+
| High | 0.7262 | 0.7409 | 0.7335 | 28,209 |
|
| 151 |
+
| Critical | 0.6659 | 0.5839 | 0.6222 | 8,830 |
|
| 152 |
+
|
| 153 |
### Training results
|
| 154 |
|
| 155 |
+
Validation metrics at each epoch:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 156 |
|
| 157 |
+
| Epoch | Step | Training Loss | Validation Loss | Accuracy | F1 Macro |
|
| 158 |
+
|:-----:|:----:|:-------------:|:---------------:|:--------:|:--------:|
|
| 159 |
+
| 1.0 | 18258 | 2.3938 | 2.6531 | 0.7289 | 0.6162 |
|
| 160 |
+
| 2.0 | 36516 | 2.1821 | 2.5989 | 0.7411 | 0.6482 |
|
| 161 |
+
| 3.0 | 54774 | 1.8636 | 2.6522 | 0.7488 | 0.6568 |
|
| 162 |
+
| 4.0 | 73032 | 1.6322 | 2.9344 | 0.7481 | 0.6566 |
|
| 163 |
+
| 5.0 | 91290 | 1.3337 | 3.1119 | 0.7473 | 0.6622 |
|
| 164 |
+
|
| 165 |
+
### Previous versions
|
| 166 |
+
|
| 167 |
+
The previous model, trained on the earlier random split, is revision
|
| 168 |
+
[`10d9861`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/tree/10d9861)
|
| 169 |
+
(dataset revision `34e6a14`, test accuracy 81.17%, 73.02% on the test rows without a twin in train).
|
| 170 |
|
| 171 |
### Framework versions
|
| 172 |
|
| 173 |
- Transformers 5.22.2
|
| 174 |
+
- Pytorch 2.15.0
|
| 175 |
- Datasets 4.8.5
|
| 176 |
- Tokenizers 0.23.3
|
| 177 |
+
|
| 178 |
+
## Provenance
|
| 179 |
+
|
| 180 |
+
This model card is generated by [VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) 3.5.0 at the end of the training run, from the values the run measured.
|
| 181 |
+
|
| 182 |
+
- [`bom.cdx.json`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/blob/main/bom.cdx.json): machine-readable record of the run ([CycloneDX](https://cyclonedx.org/) ML-BOM): base model, dataset revision, metrics, training options and energy use.
|
| 183 |
+
- [`emissions.csv`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/blob/main/emissions.csv): energy use and CO₂ emissions of the training, measured with [CodeCarbon](https://codecarbon.io/).
|
| 184 |
+
|
| 185 |
+
## Citation
|
| 186 |
+
|
| 187 |
+
```bibtex
|
| 188 |
+
@article{bonhomme2025vlai,
|
| 189 |
+
title = {{VLAI}: A {RoBERTa}-Based Model for Automated Vulnerability
|
| 190 |
+
Severity Classification},
|
| 191 |
+
author = {Bonhomme, C{\'e}dric and Dulaunoy, Alexandre},
|
| 192 |
+
journal = {arXiv preprint arXiv:2507.03607},
|
| 193 |
+
year = {2025}
|
| 194 |
+
}
|
| 195 |
+
|
| 196 |
+
@techreport{bonhomme2026vulnscores,
|
| 197 |
+
title = {Improving vulnerability-scores: Leakage-Free Splits and
|
| 198 |
+
Consistent {CVSS} 4.0 Scoring},
|
| 199 |
+
author = {Bonhomme, C{\'e}dric},
|
| 200 |
+
institution = {CIRCL - Computer Incident Response Center Luxembourg},
|
| 201 |
+
year = {2026},
|
| 202 |
+
month = oct,
|
| 203 |
+
doi = {10.5281/zenodo.23760802},
|
| 204 |
+
url = {https://doi.org/10.5281/zenodo.23760802}
|
| 205 |
+
}
|
| 206 |
+
```
|