cedricbonhomme commited on
Commit
10d9861
·
verified ·
1 Parent(s): f7e803b

End of training

Browse files
Files changed (4) hide show
  1. README.md +35 -71
  2. config.json +1 -1
  3. emissions.csv +1 -1
  4. model.safetensors +1 -1
README.md CHANGED
@@ -1,69 +1,50 @@
1
  ---
2
  library_name: transformers
3
- license: cc-by-4.0
4
  base_model: roberta-base
5
- metrics:
6
- - accuracy
7
  tags:
8
  - generated_from_trainer
9
- - text-classification
10
- - classification
11
- - nlp
12
- - vulnerability
13
  model-index:
14
  - name: vulnerability-severity-classification-roberta-base
15
  results: []
16
- datasets:
17
- - CIRCL/vulnerability-scores
18
  ---
19
 
 
 
20
 
21
- # VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
22
-
23
- # Severity classification
24
-
25
- This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
26
-
27
- The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification](https://huggingface.co/papers/2507.03607) [[arXiv](https://arxiv.org/abs/2507.03607)].
28
-
29
- **Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
30
-
31
- You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
32
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
 
34
  ## Model description
35
 
36
- It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
37
-
38
- ## How to get started with the model
39
-
40
- ```python
41
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
42
- import torch
43
 
44
- labels = ["low", "medium", "high", "critical"]
45
 
46
- model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
47
- tokenizer = AutoTokenizer.from_pretrained(model_name)
48
- model = AutoModelForSequenceClassification.from_pretrained(model_name)
49
- model.eval()
50
 
51
- print("Model revision:", model.config._commit_hash)
52
 
53
- test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
54
- that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
55
- inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)
56
-
57
- # Run inference
58
- with torch.no_grad():
59
- outputs = model(**inputs)
60
- predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
61
-
62
- # Print results
63
- print("Predictions:", predictions)
64
- predicted_class = torch.argmax(predictions, dim=-1).item()
65
- print("Predicted severity:", labels[predicted_class])
66
- ```
67
 
68
  ## Training procedure
69
 
@@ -78,37 +59,20 @@ The following hyperparameters were used during training:
78
  - lr_scheduler_type: linear
79
  - num_epochs: 5
80
 
81
- It achieves the following results on the evaluation set:
82
- - Loss: 2.0906
83
- - Accuracy: 0.8109
84
- - F1 Macro: 0.7383
85
- - Low Precision: 0.6926
86
- - Low Recall: 0.4428
87
- - Low F1: 0.5402
88
- - Medium Precision: 0.8274
89
- - Medium Recall: 0.8662
90
- - Medium F1: 0.8464
91
- - High Precision: 0.8130
92
- - High Recall: 0.8124
93
- - High F1: 0.8127
94
- - Critical Precision: 0.7677
95
- - Critical Recall: 0.7406
96
- - Critical F1: 0.7539
97
-
98
  ### Training results
99
 
100
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
101
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
102
- | 2.3598 | 1.0 | 19363 | 2.6454 | 0.7298 | 0.6067 | 0.6491 | 0.1836 | 0.2862 | 0.7618 | 0.8161 | 0.7880 | 0.7188 | 0.7317 | 0.7252 | 0.6449 | 0.6110 | 0.6275 |
103
- | 2.3058 | 2.0 | 38726 | 2.3857 | 0.7570 | 0.6550 | 0.6429 | 0.2886 | 0.3984 | 0.7937 | 0.8234 | 0.8083 | 0.7285 | 0.7860 | 0.7562 | 0.7298 | 0.5979 | 0.6573 |
104
- | 2.0088 | 3.0 | 58089 | 2.2418 | 0.7775 | 0.6871 | 0.6675 | 0.3581 | 0.4661 | 0.8082 | 0.8369 | 0.8223 | 0.7501 | 0.8133 | 0.7804 | 0.7771 | 0.6041 | 0.6798 |
105
- | 1.5341 | 4.0 | 77452 | 2.1168 | 0.8009 | 0.7309 | 0.6294 | 0.4803 | 0.5448 | 0.8208 | 0.8561 | 0.8381 | 0.7964 | 0.8080 | 0.8021 | 0.7860 | 0.6963 | 0.7384 |
106
- | 1.4963 | 5.0 | 96815 | 2.0906 | 0.8109 | 0.7383 | 0.6926 | 0.4428 | 0.5402 | 0.8274 | 0.8662 | 0.8464 | 0.8130 | 0.8124 | 0.8127 | 0.7677 | 0.7406 | 0.7539 |
107
 
108
 
109
  ### Framework versions
110
 
111
- - Transformers 5.18.0
112
  - Pytorch 2.14.1+cu130
113
  - Datasets 4.8.5
114
- - Tokenizers 0.23.2
 
1
  ---
2
  library_name: transformers
3
+ license: mit
4
  base_model: roberta-base
 
 
5
  tags:
6
  - generated_from_trainer
7
+ metrics:
8
+ - accuracy
 
 
9
  model-index:
10
  - name: vulnerability-severity-classification-roberta-base
11
  results: []
 
 
12
  ---
13
 
14
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
+ should probably proofread and complete it, then remove this comment. -->
16
 
17
+ # vulnerability-severity-classification-roberta-base
 
 
 
 
 
 
 
 
 
 
18
 
19
+ This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
20
+ It achieves the following results on the evaluation set:
21
+ - Loss: 2.0756
22
+ - Accuracy: 0.8117
23
+ - F1 Macro: 0.7419
24
+ - Low Precision: 0.6336
25
+ - Low Recall: 0.5010
26
+ - Low F1: 0.5595
27
+ - Medium Precision: 0.8334
28
+ - Medium Recall: 0.8652
29
+ - Medium F1: 0.8490
30
+ - High Precision: 0.8159
31
+ - High Recall: 0.8090
32
+ - High F1: 0.8124
33
+ - Critical Precision: 0.7625
34
+ - Critical Recall: 0.7316
35
+ - Critical F1: 0.7467
36
 
37
  ## Model description
38
 
39
+ More information needed
 
 
 
 
 
 
40
 
41
+ ## Intended uses & limitations
42
 
43
+ More information needed
 
 
 
44
 
45
+ ## Training and evaluation data
46
 
47
+ More information needed
 
 
 
 
 
 
 
 
 
 
 
 
 
48
 
49
  ## Training procedure
50
 
 
59
  - lr_scheduler_type: linear
60
  - num_epochs: 5
61
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
  ### Training results
63
 
64
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
65
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
66
+ | 2.7127 | 1.0 | 19683 | 2.5755 | 0.7332 | 0.6185 | 0.6488 | 0.2249 | 0.3340 | 0.7815 | 0.8084 | 0.7947 | 0.7195 | 0.7348 | 0.7271 | 0.6075 | 0.6290 | 0.6181 |
67
+ | 2.3041 | 2.0 | 39366 | 2.3262 | 0.7628 | 0.6734 | 0.6275 | 0.3659 | 0.4623 | 0.7933 | 0.8364 | 0.8143 | 0.7516 | 0.7660 | 0.7587 | 0.7009 | 0.6209 | 0.6585 |
68
+ | 1.9757 | 3.0 | 59049 | 2.2180 | 0.7782 | 0.6838 | 0.7351 | 0.3240 | 0.4498 | 0.7692 | 0.8968 | 0.8281 | 0.8093 | 0.7290 | 0.7671 | 0.7300 | 0.6547 | 0.6903 |
69
+ | 1.7456 | 4.0 | 78732 | 2.0480 | 0.8026 | 0.7267 | 0.6764 | 0.4395 | 0.5328 | 0.8121 | 0.8795 | 0.8445 | 0.8177 | 0.7801 | 0.7985 | 0.7456 | 0.7170 | 0.7310 |
70
+ | 1.6966 | 5.0 | 98415 | 2.0756 | 0.8117 | 0.7419 | 0.6336 | 0.5010 | 0.5595 | 0.8334 | 0.8652 | 0.8490 | 0.8159 | 0.8090 | 0.8124 | 0.7625 | 0.7316 | 0.7467 |
71
 
72
 
73
  ### Framework versions
74
 
75
+ - Transformers 5.19.0
76
  - Pytorch 2.14.1+cu130
77
  - Datasets 4.8.5
78
+ - Tokenizers 0.23.3
config.json CHANGED
@@ -34,7 +34,7 @@
34
  "pad_token_id": 1,
35
  "problem_type": "single_label_classification",
36
  "tie_word_embeddings": true,
37
- "transformers_version": "5.18.0",
38
  "type_vocab_size": 1,
39
  "use_cache": true,
40
  "vocab_size": 50265
 
34
  "pad_token_id": 1,
35
  "problem_type": "single_label_classification",
36
  "tie_word_embeddings": true,
37
+ "transformers_version": "5.19.0",
38
  "type_vocab_size": 1,
39
  "use_cache": true,
40
  "vocab_size": 50265
emissions.csv CHANGED
@@ -1,2 +1,2 @@
1
  timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
2
- 2026-10-06T17:03:10,VulnTrain,f97f5062-11d3-48e3-baf8-1137ac1c569f,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,17323.065519523,0.527089473218873,3.042703224927748e-05,70.0002182038376,902.0948631070827,70.0,0.3367934261991617,4.334508493715132,0.33605558670125557,5.007357506615553,0.0,Luxembourg,LUX,luxembourg,,,Linux-6.8.0-146-generic-x86_64-with-glibc2.39,3.12.3,3.3.1,224,Intel(R) Xeon(R) Platinum 8480+,4,4 x NVIDIA L40S,6.1327,49.6098,2015.3354225158691,machine,1.0509953434225843,75.42930733410942,1.0833061699650757,21.249004599823248,N,1.0,0.0
 
1
  timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
2
+ 2026-10-14T17:12:39,VulnTrain,10373828-fb5c-4b3a-90f1-e9abb881ff34,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,17484.878678262,0.5339587645288342,3.053831681386935e-05,70.00023979830081,906.1006087600176,70.0,0.3399453059461466,4.393544974833166,0.3391255911684121,5.0726158719477334,0.0,Luxembourg,LUX,luxembourg,,,Linux-6.8.0-146-generic-x86_64-with-glibc2.39,3.12.3,3.3.1,224,Intel(R) Xeon(R) Platinum 8480+,4,4 x NVIDIA L40S,6.1327,49.6098,2015.3354225158691,machine,0.9599655765920827,78.07448364888124,1.1008146873207114,22.84886298581778,N,1.0,0.0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f41377fab50baa54c54fa7fe391c7485ae1e7c2bdaa2bcbbb942a6e1f58af8f3
3
  size 498618976
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7baff74e729bfec490e02777031b29182e15ca13040a3651ffe57a022cafad12
3
  size 498618976