cedricbonhomme commited on
Commit
b868a7d
·
verified ·
1 Parent(s): 8716332

End of training

Browse files
Files changed (5) hide show
  1. README.md +33 -69
  2. bom.cdx.json +220 -0
  3. emissions.csv +1 -1
  4. model.safetensors +1 -1
  5. tokenizer.json +2 -16
README.md CHANGED
@@ -1,69 +1,50 @@
1
  ---
2
  library_name: transformers
3
- license: cc-by-4.0
4
  base_model: roberta-base
5
- metrics:
6
- - accuracy
7
  tags:
8
  - generated_from_trainer
9
- - text-classification
10
- - classification
11
- - nlp
12
- - vulnerability
13
  model-index:
14
  - name: vulnerability-severity-classification-roberta-base
15
  results: []
16
- datasets:
17
- - CIRCL/vulnerability-scores
18
  ---
19
 
 
 
20
 
21
- # VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
22
-
23
- # Severity classification
24
-
25
- This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
26
-
27
- The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification](https://huggingface.co/papers/2507.03607) [[arXiv](https://arxiv.org/abs/2507.03607)].
28
-
29
- **Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
30
-
31
- You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
32
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
 
34
  ## Model description
35
 
36
- It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
37
-
38
- ## How to get started with the model
39
-
40
- ```python
41
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
42
- import torch
43
 
44
- labels = ["low", "medium", "high", "critical"]
45
 
46
- model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
47
- tokenizer = AutoTokenizer.from_pretrained(model_name)
48
- model = AutoModelForSequenceClassification.from_pretrained(model_name)
49
- model.eval()
50
 
51
- print("Model revision:", model.config._commit_hash)
52
 
53
- test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
54
- that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
55
- inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)
56
-
57
- # Run inference
58
- with torch.no_grad():
59
- outputs = model(**inputs)
60
- predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
61
-
62
- # Print results
63
- print("Predictions:", predictions)
64
- predicted_class = torch.argmax(predictions, dim=-1).item()
65
- print("Predicted severity:", labels[predicted_class])
66
- ```
67
 
68
  ## Training procedure
69
 
@@ -78,32 +59,15 @@ The following hyperparameters were used during training:
78
  - lr_scheduler_type: linear
79
  - num_epochs: 5
80
 
81
- It achieves the following results on the evaluation set:
82
- - Loss: 3.0881
83
- - Accuracy: 0.7477
84
- - F1 Macro: 0.6619
85
- - Low Precision: 0.5404
86
- - Low Recall: 0.4261
87
- - Low F1: 0.4765
88
- - Medium Precision: 0.7953
89
- - Medium Recall: 0.8114
90
- - Medium F1: 0.8033
91
- - High Precision: 0.7300
92
- - High Recall: 0.7469
93
- - High F1: 0.7384
94
- - Critical Precision: 0.6604
95
- - Critical Recall: 0.6015
96
- - Critical F1: 0.6296
97
-
98
  ### Training results
99
 
100
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
101
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
102
- | 2.5865 | 1.0 | 18258 | 2.6268 | 0.7302 | 0.6219 | 0.6499 | 0.2702 | 0.3817 | 0.7774 | 0.8076 | 0.7922 | 0.7061 | 0.7295 | 0.7176 | 0.6226 | 0.5720 | 0.5962 |
103
- | 2.3063 | 2.0 | 36516 | 2.5395 | 0.7447 | 0.6513 | 0.6407 | 0.3392 | 0.4436 | 0.7705 | 0.8451 | 0.8061 | 0.7491 | 0.6990 | 0.7231 | 0.6375 | 0.6272 | 0.6323 |
104
- | 1.6765 | 3.0 | 54774 | 2.6399 | 0.7478 | 0.6605 | 0.6133 | 0.3784 | 0.4681 | 0.7952 | 0.8150 | 0.8050 | 0.7342 | 0.7365 | 0.7354 | 0.6307 | 0.6362 | 0.6334 |
105
- | 1.6561 | 4.0 | 73032 | 2.8246 | 0.7483 | 0.6634 | 0.6211 | 0.3880 | 0.4776 | 0.8001 | 0.8076 | 0.8038 | 0.7282 | 0.7466 | 0.7373 | 0.6364 | 0.6334 | 0.6349 |
106
- | 1.2166 | 5.0 | 91290 | 3.0881 | 0.7477 | 0.6619 | 0.5404 | 0.4261 | 0.4765 | 0.7953 | 0.8114 | 0.8033 | 0.7300 | 0.7469 | 0.7384 | 0.6604 | 0.6015 | 0.6296 |
107
 
108
 
109
  ### Framework versions
 
1
  ---
2
  library_name: transformers
3
+ license: mit
4
  base_model: roberta-base
 
 
5
  tags:
6
  - generated_from_trainer
7
+ metrics:
8
+ - accuracy
 
 
9
  model-index:
10
  - name: vulnerability-severity-classification-roberta-base
11
  results: []
 
 
12
  ---
13
 
14
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
+ should probably proofread and complete it, then remove this comment. -->
16
 
17
+ # vulnerability-severity-classification-roberta-base
 
 
 
 
 
 
 
 
 
 
18
 
19
+ This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
20
+ It achieves the following results on the evaluation set:
21
+ - Loss: 3.1119
22
+ - Accuracy: 0.7473
23
+ - F1 Macro: 0.6622
24
+ - Low Precision: 0.5590
25
+ - Low Recall: 0.4226
26
+ - Low F1: 0.4813
27
+ - Medium Precision: 0.7981
28
+ - Medium Recall: 0.8080
29
+ - Medium F1: 0.8030
30
+ - High Precision: 0.7260
31
+ - High Recall: 0.7507
32
+ - High F1: 0.7382
33
+ - Critical Precision: 0.6551
34
+ - Critical Recall: 0.5999
35
+ - Critical F1: 0.6263
36
 
37
  ## Model description
38
 
39
+ More information needed
 
 
 
 
 
 
40
 
41
+ ## Intended uses & limitations
42
 
43
+ More information needed
 
 
 
44
 
45
+ ## Training and evaluation data
46
 
47
+ More information needed
 
 
 
 
 
 
 
 
 
 
 
 
 
48
 
49
  ## Training procedure
50
 
 
59
  - lr_scheduler_type: linear
60
  - num_epochs: 5
61
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
  ### Training results
63
 
64
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
65
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
66
+ | 2.3938 | 1.0 | 18258 | 2.6531 | 0.7289 | 0.6162 | 0.6395 | 0.2866 | 0.3958 | 0.7740 | 0.8113 | 0.7922 | 0.6914 | 0.7502 | 0.7196 | 0.6735 | 0.4752 | 0.5572 |
67
+ | 2.1821 | 2.0 | 36516 | 2.5989 | 0.7411 | 0.6482 | 0.6092 | 0.3464 | 0.4417 | 0.7798 | 0.8252 | 0.8019 | 0.7384 | 0.7106 | 0.7243 | 0.6171 | 0.6333 | 0.6251 |
68
+ | 1.8636 | 3.0 | 54774 | 2.6522 | 0.7488 | 0.6568 | 0.6077 | 0.3742 | 0.4632 | 0.7927 | 0.8197 | 0.8059 | 0.7243 | 0.7522 | 0.7380 | 0.6710 | 0.5767 | 0.6202 |
69
+ | 1.6322 | 4.0 | 73032 | 2.9344 | 0.7481 | 0.6566 | 0.6726 | 0.3430 | 0.4543 | 0.8058 | 0.8012 | 0.8035 | 0.7143 | 0.7647 | 0.7387 | 0.6495 | 0.6114 | 0.6299 |
70
+ | 1.3337 | 5.0 | 91290 | 3.1119 | 0.7473 | 0.6622 | 0.5590 | 0.4226 | 0.4813 | 0.7981 | 0.8080 | 0.8030 | 0.7260 | 0.7507 | 0.7382 | 0.6551 | 0.5999 | 0.6263 |
71
 
72
 
73
  ### Framework versions
bom.cdx.json ADDED
@@ -0,0 +1,220 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "$schema": "http://cyclonedx.org/schema/bom-1.7.schema.json",
3
+ "bomFormat": "CycloneDX",
4
+ "specVersion": "1.7",
5
+ "serialNumber": "urn:uuid:d8f4ff2a-040b-40c4-9486-8cd6b4d783e8",
6
+ "version": 1,
7
+ "metadata": {
8
+ "timestamp": "2026-11-19T11:07:44+00:00",
9
+ "tools": {
10
+ "components": [
11
+ {
12
+ "type": "application",
13
+ "name": "VulnTrain",
14
+ "version": "3.5.0"
15
+ }
16
+ ]
17
+ },
18
+ "component": {
19
+ "type": "machine-learning-model",
20
+ "name": "CIRCL/vulnerability-severity-classification-roberta-base",
21
+ "licenses": [
22
+ {
23
+ "license": {
24
+ "id": "CC-BY-4.0"
25
+ }
26
+ }
27
+ ],
28
+ "pedigree": {
29
+ "ancestors": [
30
+ {
31
+ "type": "machine-learning-model",
32
+ "name": "roberta-base",
33
+ "purl": "pkg:huggingface/roberta-base",
34
+ "externalReferences": [
35
+ {
36
+ "type": "model-card",
37
+ "url": "https://huggingface.co/roberta-base"
38
+ }
39
+ ]
40
+ }
41
+ ]
42
+ },
43
+ "bom-ref": "pkg:huggingface/CIRCL/vulnerability-severity-classification-roberta-base",
44
+ "purl": "pkg:huggingface/CIRCL/vulnerability-severity-classification-roberta-base",
45
+ "externalReferences": [
46
+ {
47
+ "type": "model-card",
48
+ "url": "https://huggingface.co/CIRCL/vulnerability-severity-classification-roberta-base"
49
+ }
50
+ ],
51
+ "modelCard": {
52
+ "modelParameters": {
53
+ "approach": {
54
+ "type": "supervised"
55
+ },
56
+ "task": "text-classification",
57
+ "datasets": [
58
+ {
59
+ "ref": "pkg:huggingface/CIRCL/vulnerability-scores@58b483351be74f81ece78ff255a329027a6a8863"
60
+ }
61
+ ]
62
+ },
63
+ "quantitativeAnalysis": {
64
+ "performanceMetrics": [
65
+ {
66
+ "type": "accuracy",
67
+ "value": "0.7418604651162791"
68
+ },
69
+ {
70
+ "type": "f1 (macro)",
71
+ "value": "0.6496194318818087"
72
+ },
73
+ {
74
+ "type": "precision",
75
+ "value": "0.5947334617854849",
76
+ "slice": "Low"
77
+ },
78
+ {
79
+ "type": "recall",
80
+ "value": "0.3562908811081185",
81
+ "slice": "Low"
82
+ },
83
+ {
84
+ "type": "f1",
85
+ "value": "0.4456207892204042",
86
+ "slice": "Low"
87
+ },
88
+ {
89
+ "type": "precision",
90
+ "value": "0.7789519906323185",
91
+ "slice": "Medium"
92
+ },
93
+ {
94
+ "type": "recall",
95
+ "value": "0.8161769216612478",
96
+ "slice": "Medium"
97
+ },
98
+ {
99
+ "type": "f1",
100
+ "value": "0.7971301039513495",
101
+ "slice": "Medium"
102
+ },
103
+ {
104
+ "type": "precision",
105
+ "value": "0.7262334954829743",
106
+ "slice": "High"
107
+ },
108
+ {
109
+ "type": "recall",
110
+ "value": "0.7409337445496118",
111
+ "slice": "High"
112
+ },
113
+ {
114
+ "type": "f1",
115
+ "value": "0.7335099756093281",
116
+ "slice": "High"
117
+ },
118
+ {
119
+ "type": "precision",
120
+ "value": "0.6658917732145163",
121
+ "slice": "Critical"
122
+ },
123
+ {
124
+ "type": "recall",
125
+ "value": "0.5839184597961495",
126
+ "slice": "Critical"
127
+ },
128
+ {
129
+ "type": "f1",
130
+ "value": "0.6222168587461534",
131
+ "slice": "Critical"
132
+ }
133
+ ]
134
+ },
135
+ "properties": [
136
+ {
137
+ "name": "cdx:ai-ml:model:hyperparameter:learning_rate",
138
+ "value": "3e-05"
139
+ },
140
+ {
141
+ "name": "cdx:ai-ml:model:hyperparameter:warmup_steps",
142
+ "value": "0"
143
+ },
144
+ {
145
+ "name": "cdx:ai-ml:model:hyperparameter:batch_size",
146
+ "value": "32"
147
+ },
148
+ {
149
+ "name": "cdx:ai-ml:model:hyperparameter:num_epochs",
150
+ "value": "5"
151
+ },
152
+ {
153
+ "name": "cdx:ai-ml:model:hyperparameter:weight_decay",
154
+ "value": "0.01"
155
+ },
156
+ {
157
+ "name": "cdx:ai-ml:model:hyperparameter:max_length",
158
+ "value": "512"
159
+ },
160
+ {
161
+ "name": "vulntrain:training:score_strategy",
162
+ "value": "first"
163
+ },
164
+ {
165
+ "name": "vulntrain:evaluation:test_samples",
166
+ "value": "72240"
167
+ }
168
+ ],
169
+ "considerations": {
170
+ "environmentalConsiderations": {
171
+ "energyConsumptions": [
172
+ {
173
+ "activity": "fine-tuning",
174
+ "energyProviders": [
175
+ {
176
+ "description": "Electricity at the location of the training run, as reported by CodeCarbon.",
177
+ "organization": {
178
+ "address": {
179
+ "country": "Luxembourg",
180
+ "region": "luxembourg"
181
+ }
182
+ },
183
+ "energySource": "unknown",
184
+ "energyProvided": {
185
+ "value": 4.7389288966304415,
186
+ "unit": "kWh"
187
+ }
188
+ }
189
+ ],
190
+ "activityEnergyCost": {
191
+ "value": 4.7389288966304415,
192
+ "unit": "kWh"
193
+ },
194
+ "co2CostEquivalent": {
195
+ "value": 0.0004988338724460101,
196
+ "unit": "tCO2eq"
197
+ }
198
+ }
199
+ ]
200
+ }
201
+ }
202
+ }
203
+ }
204
+ },
205
+ "components": [
206
+ {
207
+ "type": "data",
208
+ "name": "CIRCL/vulnerability-scores",
209
+ "version": "58b483351be74f81ece78ff255a329027a6a8863",
210
+ "bom-ref": "pkg:huggingface/CIRCL/vulnerability-scores@58b483351be74f81ece78ff255a329027a6a8863",
211
+ "purl": "pkg:huggingface/CIRCL/vulnerability-scores@58b483351be74f81ece78ff255a329027a6a8863",
212
+ "externalReferences": [
213
+ {
214
+ "type": "distribution",
215
+ "url": "https://huggingface.co/datasets/CIRCL/vulnerability-scores/tree/58b483351be74f81ece78ff255a329027a6a8863"
216
+ }
217
+ ]
218
+ }
219
+ ]
220
+ }
emissions.csv CHANGED
@@ -1,2 +1,2 @@
1
  timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
2
- 2026-11-13T16:33:32,VulnTrain,fa63d08b-de15-4b04-9d1f-2657c1449f00,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,16333.030918859999,0.49822975988256707,3.050442764467271e-05,70.00019598784039,904.829317126947,70.0,0.31755006313886025,4.098892778556039,0.31674697697422527,4.733189818669112,0.0,Luxembourg,LUX,luxembourg,,,Linux-6.8.0-149-generic-x86_64-with-glibc2.39,3.12.3,3.3.2,224,Intel(R) Xeon(R) Platinum 8480+,4,4 x NVIDIA L40S,6.1327,49.6098,2015.3354415893555,machine,0.6936078672403196,77.62334050399508,1.0999877074370008,21.564408882952353,N,1.0,0.0
 
1
  timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
2
+ 2026-11-19T11:04:51,VulnTrain,b0aefb1d-3994-4d23-9466-42d0b0b4b6c1,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,16299.303989824002,0.4988338724460102,3.060461187529496e-05,70.00019670046285,908.1111394410165,70.0,0.3168908880069359,4.105943293363121,0.3160947152603901,4.7389288966304415,0.0,Luxembourg,LUX,luxembourg,,,Linux-6.8.0-149-generic-x86_64-with-glibc2.39,3.12.3,3.3.2,224,Intel(R) Xeon(R) Platinum 8480+,4,4 x NVIDIA L40S,6.1327,49.6098,2015.335433959961,machine,0.6944358863762401,78.05245240002465,0.9934191878735598,19.45858310570418,N,1.0,0.0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:264e93c8df795b1dc66459bae6bf178bad6b30b7c5b553ca25ea49b251cfcae4
3
  size 498618976
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a04ea44a1fcf85ae2f4682c1276afb305420216a4eb7294fa3128bd1b2b3eee1
3
  size 498618976
tokenizer.json CHANGED
@@ -1,21 +1,7 @@
1
  {
2
  "version": "1.0",
3
- "truncation": {
4
- "direction": "Right",
5
- "max_length": 512,
6
- "strategy": "LongestFirst",
7
- "stride": 0
8
- },
9
- "padding": {
10
- "strategy": {
11
- "Fixed": 512
12
- },
13
- "direction": "Right",
14
- "pad_to_multiple_of": null,
15
- "pad_id": 1,
16
- "pad_type_id": 0,
17
- "pad_token": "<pad>"
18
- },
19
  "added_tokens": [
20
  {
21
  "id": 0,
 
1
  {
2
  "version": "1.0",
3
+ "truncation": null,
4
+ "padding": null,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5
  "added_tokens": [
6
  {
7
  "id": 0,