cedricbonhomme commited on
Commit
064daa2
·
verified ·
1 Parent(s): f810cd0

End of training

Browse files
Files changed (4) hide show
  1. README.md +35 -71
  2. config.json +1 -1
  3. emissions.csv +1 -1
  4. model.safetensors +1 -1
README.md CHANGED
@@ -1,69 +1,50 @@
1
  ---
2
  library_name: transformers
3
- license: cc-by-4.0
4
  base_model: roberta-base
5
- metrics:
6
- - accuracy
7
  tags:
8
  - generated_from_trainer
9
- - text-classification
10
- - classification
11
- - nlp
12
- - vulnerability
13
  model-index:
14
  - name: vulnerability-severity-classification-roberta-base
15
  results: []
16
- datasets:
17
- - CIRCL/vulnerability-scores
18
  ---
19
 
 
 
20
 
21
- # VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
22
-
23
- # Severity classification
24
-
25
- This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
26
-
27
- The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification](https://huggingface.co/papers/2507.03607) [[arXiv](https://arxiv.org/abs/2507.03607)].
28
-
29
- **Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
30
-
31
- You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
32
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
 
34
  ## Model description
35
 
36
- It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
37
-
38
- ## How to get started with the model
39
-
40
- ```python
41
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
42
- import torch
43
 
44
- labels = ["low", "medium", "high", "critical"]
45
 
46
- model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
47
- tokenizer = AutoTokenizer.from_pretrained(model_name)
48
- model = AutoModelForSequenceClassification.from_pretrained(model_name)
49
- model.eval()
50
 
51
- print("Model revision:", model.config._commit_hash)
52
 
53
- test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
54
- that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
55
- inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)
56
-
57
- # Run inference
58
- with torch.no_grad():
59
- outputs = model(**inputs)
60
- predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
61
-
62
- # Print results
63
- print("Predictions:", predictions)
64
- predicted_class = torch.argmax(predictions, dim=-1).item()
65
- print("Predicted severity:", labels[predicted_class])
66
- ```
67
 
68
  ## Training procedure
69
 
@@ -78,37 +59,20 @@ The following hyperparameters were used during training:
78
  - lr_scheduler_type: linear
79
  - num_epochs: 5
80
 
81
- It achieves the following results on the evaluation set:
82
- - Loss: 2.0756
83
- - Accuracy: 0.8117
84
- - F1 Macro: 0.7419
85
- - Low Precision: 0.6336
86
- - Low Recall: 0.5010
87
- - Low F1: 0.5595
88
- - Medium Precision: 0.8334
89
- - Medium Recall: 0.8652
90
- - Medium F1: 0.8490
91
- - High Precision: 0.8159
92
- - High Recall: 0.8090
93
- - High F1: 0.8124
94
- - Critical Precision: 0.7625
95
- - Critical Recall: 0.7316
96
- - Critical F1: 0.7467
97
-
98
  ### Training results
99
 
100
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
101
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
102
- | 2.7127 | 1.0 | 19683 | 2.5755 | 0.7332 | 0.6185 | 0.6488 | 0.2249 | 0.3340 | 0.7815 | 0.8084 | 0.7947 | 0.7195 | 0.7348 | 0.7271 | 0.6075 | 0.6290 | 0.6181 |
103
- | 2.3041 | 2.0 | 39366 | 2.3262 | 0.7628 | 0.6734 | 0.6275 | 0.3659 | 0.4623 | 0.7933 | 0.8364 | 0.8143 | 0.7516 | 0.7660 | 0.7587 | 0.7009 | 0.6209 | 0.6585 |
104
- | 1.9757 | 3.0 | 59049 | 2.2180 | 0.7782 | 0.6838 | 0.7351 | 0.3240 | 0.4498 | 0.7692 | 0.8968 | 0.8281 | 0.8093 | 0.7290 | 0.7671 | 0.7300 | 0.6547 | 0.6903 |
105
- | 1.7456 | 4.0 | 78732 | 2.0480 | 0.8026 | 0.7267 | 0.6764 | 0.4395 | 0.5328 | 0.8121 | 0.8795 | 0.8445 | 0.8177 | 0.7801 | 0.7985 | 0.7456 | 0.7170 | 0.7310 |
106
- | 1.6966 | 5.0 | 98415 | 2.0756 | 0.8117 | 0.7419 | 0.6336 | 0.5010 | 0.5595 | 0.8334 | 0.8652 | 0.8490 | 0.8159 | 0.8090 | 0.8124 | 0.7625 | 0.7316 | 0.7467 |
107
 
108
 
109
  ### Framework versions
110
 
111
- - Transformers 5.19.0
112
- - Pytorch 2.14.1+cu130
113
  - Datasets 4.8.5
114
  - Tokenizers 0.23.3
 
1
  ---
2
  library_name: transformers
3
+ license: mit
4
  base_model: roberta-base
 
 
5
  tags:
6
  - generated_from_trainer
7
+ metrics:
8
+ - accuracy
 
 
9
  model-index:
10
  - name: vulnerability-severity-classification-roberta-base
11
  results: []
 
 
12
  ---
13
 
14
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
+ should probably proofread and complete it, then remove this comment. -->
16
 
17
+ # vulnerability-severity-classification-roberta-base
 
 
 
 
 
 
 
 
 
 
18
 
19
+ This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
20
+ It achieves the following results on the evaluation set:
21
+ - Loss: 3.0105
22
+ - Accuracy: 0.7457
23
+ - F1 Macro: 0.6639
24
+ - Low Precision: 0.5635
25
+ - Low Recall: 0.4309
26
+ - Low F1: 0.4884
27
+ - Medium Precision: 0.7916
28
+ - Medium Recall: 0.8126
29
+ - Medium F1: 0.8019
30
+ - High Precision: 0.7271
31
+ - High Recall: 0.7365
32
+ - High F1: 0.7317
33
+ - Critical Precision: 0.6589
34
+ - Critical Recall: 0.6101
35
+ - Critical F1: 0.6336
36
 
37
  ## Model description
38
 
39
+ More information needed
 
 
 
 
 
 
40
 
41
+ ## Intended uses & limitations
42
 
43
+ More information needed
 
 
 
44
 
45
+ ## Training and evaluation data
46
 
47
+ More information needed
 
 
 
 
 
 
 
 
 
 
 
 
 
48
 
49
  ## Training procedure
50
 
 
59
  - lr_scheduler_type: linear
60
  - num_epochs: 5
61
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
  ### Training results
63
 
64
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
65
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
66
+ | 2.5082 | 1.0 | 17749 | 2.7143 | 0.7229 | 0.6180 | 0.6764 | 0.2567 | 0.3722 | 0.8047 | 0.7647 | 0.7842 | 0.6872 | 0.7424 | 0.7137 | 0.5700 | 0.6374 | 0.6018 |
67
+ | 2.3903 | 2.0 | 35498 | 2.5364 | 0.7437 | 0.6474 | 0.6961 | 0.3085 | 0.4275 | 0.7932 | 0.8130 | 0.8030 | 0.7202 | 0.7355 | 0.7277 | 0.6314 | 0.6317 | 0.6316 |
68
+ | 1.8962 | 3.0 | 53247 | 2.6291 | 0.7461 | 0.6624 | 0.5688 | 0.4115 | 0.4775 | 0.7755 | 0.8330 | 0.8032 | 0.7455 | 0.7106 | 0.7276 | 0.6600 | 0.6234 | 0.6412 |
69
+ | 1.8782 | 4.0 | 70996 | 2.7383 | 0.7500 | 0.6608 | 0.6555 | 0.3622 | 0.4665 | 0.7892 | 0.8279 | 0.8081 | 0.7315 | 0.7352 | 0.7333 | 0.6601 | 0.6120 | 0.6351 |
70
+ | 1.4131 | 5.0 | 88745 | 3.0105 | 0.7457 | 0.6639 | 0.5635 | 0.4309 | 0.4884 | 0.7916 | 0.8126 | 0.8019 | 0.7271 | 0.7365 | 0.7317 | 0.6589 | 0.6101 | 0.6336 |
71
 
72
 
73
  ### Framework versions
74
 
75
+ - Transformers 5.21.0
76
+ - Pytorch 2.15.0+cu132
77
  - Datasets 4.8.5
78
  - Tokenizers 0.23.3
config.json CHANGED
@@ -34,7 +34,7 @@
34
  "pad_token_id": 1,
35
  "problem_type": "single_label_classification",
36
  "tie_word_embeddings": true,
37
- "transformers_version": "5.19.0",
38
  "type_vocab_size": 1,
39
  "use_cache": true,
40
  "vocab_size": 50265
 
34
  "pad_token_id": 1,
35
  "problem_type": "single_label_classification",
36
  "tie_word_embeddings": true,
37
+ "transformers_version": "5.21.0",
38
  "type_vocab_size": 1,
39
  "use_cache": true,
40
  "vocab_size": 50265
emissions.csv CHANGED
@@ -1,2 +1,2 @@
1
  timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
2
- 2026-10-14T17:12:39,VulnTrain,10373828-fb5c-4b3a-90f1-e9abb881ff34,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,17484.878678262,0.5339587645288342,3.053831681386935e-05,70.00023979830081,906.1006087600176,70.0,0.3399453059461466,4.393544974833166,0.3391255911684121,5.0726158719477334,0.0,Luxembourg,LUX,luxembourg,,,Linux-6.8.0-146-generic-x86_64-with-glibc2.39,3.12.3,3.3.1,224,Intel(R) Xeon(R) Platinum 8480+,4,4 x NVIDIA L40S,6.1327,49.6098,2015.3354225158691,machine,0.9599655765920827,78.07448364888124,1.1008146873207114,22.84886298581778,N,1.0,0.0
 
1
  timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
2
+ 2026-10-29T18:04:54,VulnTrain,a0de6b0a-1c64-47a0-ac0e-f7afb09977be,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,15907.370135360998,0.48498217400176347,3.04878914537659e-05,70.00019464846667,903.8459795071008,70.0,0.30927252797905813,3.989522156337392,0.30854287970663974,4.607337564023099,0.0,Luxembourg,LUX,luxembourg,,,Linux-6.8.0-147-generic-x86_64-with-glibc2.39,3.12.3,3.3.2,224,Intel(R) Xeon(R) Platinum 8480+,4,4 x NVIDIA L40S,6.1327,49.6098,2015.3354148864746,machine,0.693132613772644,77.6417187401376,1.0988322918639148,21.301271207569027,N,1.0,0.0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:7baff74e729bfec490e02777031b29182e15ca13040a3651ffe57a022cafad12
3
  size 498618976
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7cd9b113688bcc19f36735492ca2c59c798e5463246b405e3d7e2780f991ebb8
3
  size 498618976