cedricbonhomme commited on
Commit
d5cd435
·
verified ·
1 Parent(s): 5a0f0ed

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +107 -23
README.md CHANGED
@@ -4,23 +4,41 @@ license: cc-by-4.0
4
  base_model: roberta-base
5
  metrics:
6
  - accuracy
 
7
  tags:
8
  - generated_from_trainer
9
  - text-classification
10
  - classification
11
  - nlp
12
  - vulnerability
13
- model-index:
14
- - name: vulnerability-severity-classification-roberta-base
15
- results: []
16
  datasets:
17
  - CIRCL/vulnerability-scores
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
  ---
19
 
20
 
21
  # VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
22
 
23
- # Severity classification
24
 
25
  This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
26
 
@@ -28,6 +46,14 @@ The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated
28
 
29
  **Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
30
 
 
 
 
 
 
 
 
 
31
  You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
32
 
33
 
@@ -35,14 +61,21 @@ You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) f
35
 
36
  It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
37
 
 
 
 
 
 
 
 
 
 
38
  ## How to get started with the model
39
 
40
  ```python
41
  from transformers import AutoModelForSequenceClassification, AutoTokenizer
42
  import torch
43
 
44
- labels = ["low", "medium", "high", "critical"]
45
-
46
  model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
47
  tokenizer = AutoTokenizer.from_pretrained(model_name)
48
  model = AutoModelForSequenceClassification.from_pretrained(model_name)
@@ -62,9 +95,26 @@ with torch.no_grad():
62
  # Print results
63
  print("Predictions:", predictions)
64
  predicted_class = torch.argmax(predictions, dim=-1).item()
65
- print("Predicted severity:", labels[predicted_class])
66
  ```
67
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
  ## Training procedure
69
 
70
  ### Training hyperparameters
@@ -78,25 +128,31 @@ The following hyperparameters were used during training:
78
  - lr_scheduler_type: linear
79
  - num_epochs: 5
80
 
81
- It achieves the following results on the evaluation set:
82
- - Loss: 3.0105
83
- - Accuracy: 0.7457
84
- - F1 Macro: 0.6639
85
- - Low Precision: 0.5635
86
- - Low Recall: 0.4309
87
- - Low F1: 0.4884
88
- - Medium Precision: 0.7916
89
- - Medium Recall: 0.8126
90
- - Medium F1: 0.8019
91
- - High Precision: 0.7271
92
- - High Recall: 0.7365
93
- - High F1: 0.7317
94
- - Critical Precision: 0.6589
95
- - Critical Recall: 0.6101
96
- - Critical F1: 0.6336
 
 
 
 
97
 
98
  ### Training results
99
 
 
 
100
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
101
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
102
  | 2.5082 | 1.0 | 17749 | 2.7143 | 0.7229 | 0.6180 | 0.6764 | 0.2567 | 0.3722 | 0.8047 | 0.7647 | 0.7842 | 0.6872 | 0.7424 | 0.7137 | 0.5700 | 0.6374 | 0.6018 |
@@ -105,6 +161,11 @@ It achieves the following results on the evaluation set:
105
  | 1.8782 | 4.0 | 70996 | 2.7383 | 0.7500 | 0.6608 | 0.6555 | 0.3622 | 0.4665 | 0.7892 | 0.8279 | 0.8081 | 0.7315 | 0.7352 | 0.7333 | 0.6601 | 0.6120 | 0.6351 |
106
  | 1.4131 | 5.0 | 88745 | 3.0105 | 0.7457 | 0.6639 | 0.5635 | 0.4309 | 0.4884 | 0.7916 | 0.8126 | 0.8019 | 0.7271 | 0.7365 | 0.7317 | 0.6589 | 0.6101 | 0.6336 |
107
 
 
 
 
 
 
108
 
109
  ### Framework versions
110
 
@@ -112,3 +173,26 @@ It achieves the following results on the evaluation set:
112
  - Pytorch 2.15.0+cu132
113
  - Datasets 4.8.5
114
  - Tokenizers 0.23.3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  base_model: roberta-base
5
  metrics:
6
  - accuracy
7
+ - f1
8
  tags:
9
  - generated_from_trainer
10
  - text-classification
11
  - classification
12
  - nlp
13
  - vulnerability
 
 
 
14
  datasets:
15
  - CIRCL/vulnerability-scores
16
+ model-index:
17
+ - name: vulnerability-severity-classification-roberta-base
18
+ results:
19
+ - task:
20
+ type: text-classification
21
+ name: Vulnerability severity classification
22
+ dataset:
23
+ type: CIRCL/vulnerability-scores
24
+ name: CIRCL/vulnerability-scores
25
+ split: test
26
+ revision: 55a886a483bb27aabb6461edc64e8058cc136e24
27
+ metrics:
28
+ - type: accuracy
29
+ value: 0.7415
30
+ name: Accuracy
31
+ - type: f1
32
+ value: 0.6460
33
+ name: F1 Macro
34
+ args:
35
+ average: macro
36
  ---
37
 
38
 
39
  # VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
40
 
41
+ ## Severity classification
42
 
43
  This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
44
 
 
46
 
47
  **Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
48
 
49
+ > **Note (October 2026):** since [VulnTrain 3.4.0](https://github.com/vulnerability-lookup/VulnTrain/releases/tag/v3.4.0),
50
+ > the dataset is split so that records of the same vulnerability (for example a CVE and its
51
+ > GitHub advisory) never appear in both train and test. Accuracy measured this way (**74%**)
52
+ > reflects performance on vulnerabilities the model has never seen. Earlier figures, including
53
+ > the one in the abstract above, were measured on a split where many test descriptions also
54
+ > appeared in training. Details in the technical report
55
+ > [Improving vulnerability-scores: Leakage-Free Splits and Consistent CVSS 4.0 Scoring](https://doi.org/10.5281/zenodo.23760802).
56
+
57
  You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
58
 
59
 
 
61
 
62
  It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
63
 
64
+ Labels:
65
+
66
+ | Label | CVSS score |
67
+ |-------|------------|
68
+ | Low | < 4.0 |
69
+ | Medium | 4.0 – 6.9 |
70
+ | High | 7.0 – 8.9 |
71
+ | Critical | ≥ 9.0 |
72
+
73
  ## How to get started with the model
74
 
75
  ```python
76
  from transformers import AutoModelForSequenceClassification, AutoTokenizer
77
  import torch
78
 
 
 
79
  model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
80
  tokenizer = AutoTokenizer.from_pretrained(model_name)
81
  model = AutoModelForSequenceClassification.from_pretrained(model_name)
 
95
  # Print results
96
  print("Predictions:", predictions)
97
  predicted_class = torch.argmax(predictions, dim=-1).item()
98
+ print("Predicted severity:", model.config.id2label[predicted_class])
99
  ```
100
 
101
+ ## Training data
102
+
103
+ Revision [`55a886a`](https://huggingface.co/datasets/CIRCL/vulnerability-scores/tree/55a886a483bb27aabb6461edc64e8058cc136e24)
104
+ of [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores),
105
+ built with VulnTrain 3.4.0 from CVE records, GitHub, PySec and CSAF (Red Hat, Cisco, CISA) advisories:
106
+
107
+ | Split | Rows | Labelled rows |
108
+ |-------|-----:|--------------:|
109
+ | train | 670,283 | 567,964 |
110
+ | validation | 83,550 | 71,470 |
111
+ | test | 83,376 | 70,472 |
112
+
113
+ - **Group-aware split (80/10/10).** Rows that share a description, an identifier or an alias are
114
+ kept in the same split, so no vulnerability appears in both train and test.
115
+ - **Labels** come from the first available CVSS score, in the order v4.0, v3.1, v3.0, v2.0.
116
+ CVSS 4.0 scores are read with base metrics only (CVSS-B), whatever the source.
117
+
118
  ## Training procedure
119
 
120
  ### Training hyperparameters
 
128
  - lr_scheduler_type: linear
129
  - num_epochs: 5
130
 
131
+ The checkpoint with the best validation accuracy (epoch 4) is the one published.
132
+
133
+ Training took 4 h 25 min on 4 × NVIDIA L40S and used 4.61 kWh, for 0.48 kg CO₂eq
134
+ (measured with CodeCarbon, see [`emissions.csv`](emissions.csv)).
135
+
136
+ ### Results
137
+
138
+ Evaluated once on the leakage-free test split of
139
+ [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores/tree/55a886a483bb27aabb6461edc64e8058cc136e24)
140
+ (revision `55a886a`, 70,472 labelled rows):
141
+
142
+ - Accuracy: **0.7415** (95% CI 0.7383 – 0.7447)
143
+ - F1 Macro: 0.6460
144
+
145
+ | Class | Precision | Recall | F1 |
146
+ |-------|----------:|-------:|---:|
147
+ | Low | 0.6146 | 0.3280 | 0.4278 |
148
+ | Medium | 0.7767 | 0.8251 | 0.8002 |
149
+ | High | 0.7328 | 0.7237 | 0.7282 |
150
+ | Critical | 0.6460 | 0.6107 | 0.6279 |
151
 
152
  ### Training results
153
 
154
+ Validation metrics at each epoch:
155
+
156
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
157
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
158
  | 2.5082 | 1.0 | 17749 | 2.7143 | 0.7229 | 0.6180 | 0.6764 | 0.2567 | 0.3722 | 0.8047 | 0.7647 | 0.7842 | 0.6872 | 0.7424 | 0.7137 | 0.5700 | 0.6374 | 0.6018 |
 
161
  | 1.8782 | 4.0 | 70996 | 2.7383 | 0.7500 | 0.6608 | 0.6555 | 0.3622 | 0.4665 | 0.7892 | 0.8279 | 0.8081 | 0.7315 | 0.7352 | 0.7333 | 0.6601 | 0.6120 | 0.6351 |
162
  | 1.4131 | 5.0 | 88745 | 3.0105 | 0.7457 | 0.6639 | 0.5635 | 0.4309 | 0.4884 | 0.7916 | 0.8126 | 0.8019 | 0.7271 | 0.7365 | 0.7317 | 0.6589 | 0.6101 | 0.6336 |
163
 
164
+ ### Previous versions
165
+
166
+ The previous model, trained on the earlier random split, is revision
167
+ [`10d9861`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/tree/10d9861)
168
+ (dataset revision `34e6a14`, test accuracy 81.17%, 73.02% on the test rows without a twin in train).
169
 
170
  ### Framework versions
171
 
 
173
  - Pytorch 2.15.0+cu132
174
  - Datasets 4.8.5
175
  - Tokenizers 0.23.3
176
+
177
+ ## Citation
178
+
179
+ ```bibtex
180
+ @article{bonhomme2025vlai,
181
+ title = {{VLAI}: A {RoBERTa}-Based Model for Automated Vulnerability
182
+ Severity Classification},
183
+ author = {Bonhomme, C{\'e}dric and Dulaunoy, Alexandre},
184
+ journal = {arXiv preprint arXiv:2507.03607},
185
+ year = {2025}
186
+ }
187
+
188
+ @techreport{bonhomme2026vulnscores,
189
+ title = {Improving vulnerability-scores: Leakage-Free Splits and
190
+ Consistent {CVSS} 4.0 Scoring},
191
+ author = {Bonhomme, C{\'e}dric},
192
+ institution = {CIRCL - Computer Incident Response Center Luxembourg},
193
+ year = {2026},
194
+ month = oct,
195
+ doi = {10.5281/zenodo.23760802},
196
+ url = {https://doi.org/10.5281/zenodo.23760802}
197
+ }
198
+ ```