cedricbonhomme commited on
Commit
a8daeba
·
verified ·
1 Parent(s): b868a7d

Update model card with evaluation metrics

Browse files
Files changed (1) hide show
  1. README.md +168 -40
README.md CHANGED
@@ -1,50 +1,120 @@
1
  ---
 
 
2
  library_name: transformers
3
- license: mit
4
  base_model: roberta-base
5
- tags:
6
- - generated_from_trainer
7
  metrics:
8
  - accuracy
 
 
 
 
 
 
 
 
 
9
  model-index:
10
  - name: vulnerability-severity-classification-roberta-base
11
- results: []
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  ---
13
 
14
- <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
- should probably proofread and complete it, then remove this comment. -->
16
-
17
- # vulnerability-severity-classification-roberta-base
18
-
19
- This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
20
- It achieves the following results on the evaluation set:
21
- - Loss: 3.1119
22
- - Accuracy: 0.7473
23
- - F1 Macro: 0.6622
24
- - Low Precision: 0.5590
25
- - Low Recall: 0.4226
26
- - Low F1: 0.4813
27
- - Medium Precision: 0.7981
28
- - Medium Recall: 0.8080
29
- - Medium F1: 0.8030
30
- - High Precision: 0.7260
31
- - High Recall: 0.7507
32
- - High F1: 0.7382
33
- - Critical Precision: 0.6551
34
- - Critical Recall: 0.5999
35
- - Critical F1: 0.6263
36
 
37
  ## Model description
38
 
39
- More information needed
 
 
 
 
 
 
 
 
 
 
 
40
 
41
- ## Intended uses & limitations
 
 
42
 
43
- More information needed
 
 
 
 
44
 
45
- ## Training and evaluation data
 
 
46
 
47
- More information needed
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
48
 
49
  ## Training procedure
50
 
@@ -55,24 +125,82 @@ The following hyperparameters were used during training:
55
  - train_batch_size: 32
56
  - eval_batch_size: 32
57
  - seed: 42
58
- - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
59
  - lr_scheduler_type: linear
 
 
60
  - num_epochs: 5
61
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
  ### Training results
63
 
64
- | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
65
- |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
66
- | 2.3938 | 1.0 | 18258 | 2.6531 | 0.7289 | 0.6162 | 0.6395 | 0.2866 | 0.3958 | 0.7740 | 0.8113 | 0.7922 | 0.6914 | 0.7502 | 0.7196 | 0.6735 | 0.4752 | 0.5572 |
67
- | 2.1821 | 2.0 | 36516 | 2.5989 | 0.7411 | 0.6482 | 0.6092 | 0.3464 | 0.4417 | 0.7798 | 0.8252 | 0.8019 | 0.7384 | 0.7106 | 0.7243 | 0.6171 | 0.6333 | 0.6251 |
68
- | 1.8636 | 3.0 | 54774 | 2.6522 | 0.7488 | 0.6568 | 0.6077 | 0.3742 | 0.4632 | 0.7927 | 0.8197 | 0.8059 | 0.7243 | 0.7522 | 0.7380 | 0.6710 | 0.5767 | 0.6202 |
69
- | 1.6322 | 4.0 | 73032 | 2.9344 | 0.7481 | 0.6566 | 0.6726 | 0.3430 | 0.4543 | 0.8058 | 0.8012 | 0.8035 | 0.7143 | 0.7647 | 0.7387 | 0.6495 | 0.6114 | 0.6299 |
70
- | 1.3337 | 5.0 | 91290 | 3.1119 | 0.7473 | 0.6622 | 0.5590 | 0.4226 | 0.4813 | 0.7981 | 0.8080 | 0.8030 | 0.7260 | 0.7507 | 0.7382 | 0.6551 | 0.5999 | 0.6263 |
71
 
 
 
 
 
 
 
 
 
 
 
 
 
 
72
 
73
  ### Framework versions
74
 
75
  - Transformers 5.22.2
76
- - Pytorch 2.15.0+cu132
77
  - Datasets 4.8.5
78
  - Tokenizers 0.23.3
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
  library_name: transformers
5
+ license: cc-by-4.0
6
  base_model: roberta-base
7
+ pipeline_tag: text-classification
 
8
  metrics:
9
  - accuracy
10
+ - f1
11
+ tags:
12
+ - generated_from_trainer
13
+ - text-classification
14
+ - classification
15
+ - nlp
16
+ - vulnerability
17
+ datasets:
18
+ - CIRCL/vulnerability-scores
19
  model-index:
20
  - name: vulnerability-severity-classification-roberta-base
21
+ results:
22
+ - task:
23
+ type: text-classification
24
+ name: Vulnerability severity classification
25
+ dataset:
26
+ type: CIRCL/vulnerability-scores
27
+ name: CIRCL/vulnerability-scores
28
+ split: test
29
+ revision: 58b483351be74f81ece78ff255a329027a6a8863
30
+ metrics:
31
+ - type: accuracy
32
+ value: 0.7419
33
+ name: Accuracy
34
+ - type: f1
35
+ value: 0.6496
36
+ name: F1 Macro
37
+ args:
38
+ average: macro
39
  ---
40
 
41
+ # VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
42
+
43
+ ## Severity classification
44
+
45
+ This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
46
+
47
+ The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification](https://huggingface.co/papers/2507.03607) [[arXiv](https://arxiv.org/abs/2507.03607)].
48
+
49
+ **Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
50
+
51
+ > **Note:** since [VulnTrain 3.4.0](https://github.com/vulnerability-lookup/VulnTrain/releases/tag/v3.4.0),
52
+ > the dataset is split so that records of the same vulnerability (for example a CVE and its
53
+ > GitHub advisory) never appear in both train and test. Accuracy measured this way (**74%**)
54
+ > reflects performance on vulnerabilities the model has never seen. Earlier figures, including
55
+ > the one in the abstract above, were measured on a split where many test descriptions also
56
+ > appeared in training. Details in the technical report
57
+ > [Improving vulnerability-scores: Leakage-Free Splits and Consistent CVSS 4.0 Scoring](https://doi.org/10.5281/zenodo.23760802).
58
+
59
+ You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
60
+
 
 
61
 
62
  ## Model description
63
 
64
+ It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
65
+
66
+ Labels:
67
+
68
+ | Label | CVSS score |
69
+ |-------|------------|
70
+ | Low | < 4.0 |
71
+ | Medium | 4.0 – 6.9 |
72
+ | High | 7.0 – 8.9 |
73
+ | Critical | ≥ 9.0 |
74
+
75
+ ## How to get started with the model
76
 
77
+ ```python
78
+ from transformers import AutoModelForSequenceClassification, AutoTokenizer
79
+ import torch
80
 
81
+ model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
82
+ revision = "b868a7d"
83
+ tokenizer = AutoTokenizer.from_pretrained(model_name, revision=revision)
84
+ model = AutoModelForSequenceClassification.from_pretrained(model_name, revision=revision)
85
+ model.eval()
86
 
87
+ test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
88
+ that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
89
+ inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)
90
 
91
+ # Run inference
92
+ with torch.no_grad():
93
+ outputs = model(**inputs)
94
+ predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
95
+
96
+ # Print results
97
+ print("Predictions:", predictions)
98
+ predicted_class = torch.argmax(predictions, dim=-1).item()
99
+ print("Predicted severity:", model.config.id2label[predicted_class])
100
+ ```
101
+
102
+ ## Training data
103
+
104
+ Revision [`58b4833`](https://huggingface.co/datasets/CIRCL/vulnerability-scores/tree/58b483351be74f81ece78ff255a329027a6a8863)
105
+ of [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores),
106
+ built with VulnTrain from CVE records, GitHub, PySec and CSAF (Red Hat, Cisco, CISA) advisories:
107
+
108
+ | Split | Rows | Labelled rows |
109
+ |-------|-----:|--------------:|
110
+ | train | 686,695 | 584,245 |
111
+ | validation | 85,405 | 73,419 |
112
+ | test | 85,167 | 72,240 |
113
+
114
+ - **Group-aware split (80/10/10).** Rows that share a description, an identifier or an alias are
115
+ kept in the same split, so no vulnerability appears in both train and test.
116
+ - **Labels** come from the first available CVSS score, in the order v4.0, v3.1, v3.0, v2.0.
117
+ CVSS 4.0 scores are read with base metrics only (CVSS-B), whatever the source.
118
 
119
  ## Training procedure
120
 
 
125
  - train_batch_size: 32
126
  - eval_batch_size: 32
127
  - seed: 42
128
+ - optimizer: adamw_torch_fused
129
  - lr_scheduler_type: linear
130
+ - warmup_steps: 0
131
+ - weight_decay: 0.01
132
  - num_epochs: 5
133
 
134
+ The checkpoint with the best validation accuracy (epoch 3) is the one published.
135
+
136
+ Training took 4 h 32 min on 4 x NVIDIA L40S and used 4.74 kWh, for 0.50 kg CO₂eq
137
+ (measured with CodeCarbon, see [`emissions.csv`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/blob/main/emissions.csv)).
138
+
139
+ ### Results
140
+
141
+ Evaluated once on the leakage-free test split (72,240 labelled rows):
142
+
143
+ - Accuracy: **0.7419** (95% CI 0.7387 – 0.7451)
144
+ - F1 Macro: 0.6496
145
+
146
+ | Class | Precision | Recall | F1 | Support |
147
+ |-------|----------:|-------:|---:|--------:|
148
+ | Low | 0.5947 | 0.3563 | 0.4456 | 2,599 |
149
+ | Medium | 0.7790 | 0.8162 | 0.7971 | 32,602 |
150
+ | High | 0.7262 | 0.7409 | 0.7335 | 28,209 |
151
+ | Critical | 0.6659 | 0.5839 | 0.6222 | 8,830 |
152
+
153
  ### Training results
154
 
155
+ Validation metrics at each epoch:
 
 
 
 
 
 
156
 
157
+ | Epoch | Step | Training Loss | Validation Loss | Accuracy | F1 Macro |
158
+ |:-----:|:----:|:-------------:|:---------------:|:--------:|:--------:|
159
+ | 1.0 | 18258 | 2.3938 | 2.6531 | 0.7289 | 0.6162 |
160
+ | 2.0 | 36516 | 2.1821 | 2.5989 | 0.7411 | 0.6482 |
161
+ | 3.0 | 54774 | 1.8636 | 2.6522 | 0.7488 | 0.6568 |
162
+ | 4.0 | 73032 | 1.6322 | 2.9344 | 0.7481 | 0.6566 |
163
+ | 5.0 | 91290 | 1.3337 | 3.1119 | 0.7473 | 0.6622 |
164
+
165
+ ### Previous versions
166
+
167
+ The previous model, trained on the earlier random split, is revision
168
+ [`10d9861`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/tree/10d9861)
169
+ (dataset revision `34e6a14`, test accuracy 81.17%, 73.02% on the test rows without a twin in train).
170
 
171
  ### Framework versions
172
 
173
  - Transformers 5.22.2
174
+ - Pytorch 2.15.0
175
  - Datasets 4.8.5
176
  - Tokenizers 0.23.3
177
+
178
+ ## Provenance
179
+
180
+ This model card is generated by [VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) 3.5.0 at the end of the training run, from the values the run measured.
181
+
182
+ - [`bom.cdx.json`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/blob/main/bom.cdx.json): machine-readable record of the run ([CycloneDX](https://cyclonedx.org/) ML-BOM): base model, dataset revision, metrics, training options and energy use.
183
+ - [`emissions.csv`](https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base/blob/main/emissions.csv): energy use and CO₂ emissions of the training, measured with [CodeCarbon](https://codecarbon.io/).
184
+
185
+ ## Citation
186
+
187
+ ```bibtex
188
+ @article{bonhomme2025vlai,
189
+ title = {{VLAI}: A {RoBERTa}-Based Model for Automated Vulnerability
190
+ Severity Classification},
191
+ author = {Bonhomme, C{\'e}dric and Dulaunoy, Alexandre},
192
+ journal = {arXiv preprint arXiv:2507.03607},
193
+ year = {2025}
194
+ }
195
+
196
+ @techreport{bonhomme2026vulnscores,
197
+ title = {Improving vulnerability-scores: Leakage-Free Splits and
198
+ Consistent {CVSS} 4.0 Scoring},
199
+ author = {Bonhomme, C{\'e}dric},
200
+ institution = {CIRCL - Computer Incident Response Center Luxembourg},
201
+ year = {2026},
202
+ month = oct,
203
+ doi = {10.5281/zenodo.23760802},
204
+ url = {https://doi.org/10.5281/zenodo.23760802}
205
+ }
206
+ ```