cedricbonhomme commited on
Commit
1efd3ac
·
verified ·
1 Parent(s): 9e5c95b

End of training

Browse files
Files changed (4) hide show
  1. README.md +35 -70
  2. config.json +1 -1
  3. emissions.csv +1 -1
  4. model.safetensors +1 -1
README.md CHANGED
@@ -1,69 +1,50 @@
1
  ---
2
  library_name: transformers
3
- license: cc-by-4.0
4
  base_model: roberta-base
5
- metrics:
6
- - accuracy
7
  tags:
8
  - generated_from_trainer
9
- - text-classification
10
- - classification
11
- - nlp
12
- - vulnerability
13
  model-index:
14
  - name: vulnerability-severity-classification-roberta-base
15
  results: []
16
- datasets:
17
- - CIRCL/vulnerability-scores
18
  ---
19
 
 
 
20
 
21
- # VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification
22
-
23
- # Severity classification
24
-
25
- This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores).
26
-
27
- The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification](https://huggingface.co/papers/2507.03607) [[arXiv](https://arxiv.org/abs/2507.03607)].
28
-
29
- **Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service.
30
-
31
- You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information.
32
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
 
34
  ## Model description
35
 
36
- It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions.
37
-
38
- ## How to get started with the model
39
-
40
- ```python
41
- from transformers import AutoModelForSequenceClassification, AutoTokenizer
42
- import torch
43
 
44
- labels = ["low", "medium", "high", "critical"]
45
 
46
- model_name = "CIRCL/vulnerability-severity-classification-roberta-base"
47
- tokenizer = AutoTokenizer.from_pretrained(model_name)
48
- model = AutoModelForSequenceClassification.from_pretrained(model_name)
49
- model.eval()
50
 
51
- print("Model revision:", model.config._commit_hash)
52
 
53
- test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \
54
- that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system."
55
- inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True)
56
-
57
- # Run inference
58
- with torch.no_grad():
59
- outputs = model(**inputs)
60
- predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
61
-
62
- # Print results
63
- print("Predictions:", predictions)
64
- predicted_class = torch.argmax(predictions, dim=-1).item()
65
- print("Predicted severity:", labels[predicted_class])
66
- ```
67
 
68
  ## Training procedure
69
 
@@ -82,32 +63,16 @@ The following hyperparameters were used during training:
82
 
83
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
84
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
85
- | 2.6673 | 1.0 | 19209 | 2.6270 | 0.7275 | 0.6229 | 0.5891 | 0.2786 | 0.3783 | 0.7994 | 0.7680 | 0.7833 | 0.6809 | 0.7859 | 0.7296 | 0.6556 | 0.5536 | 0.6003 |
86
- | 2.2508 | 2.0 | 38418 | 2.3707 | 0.7599 | 0.6789 | 0.5790 | 0.4020 | 0.4745 | 0.7871 | 0.8371 | 0.8113 | 0.7704 | 0.7325 | 0.7510 | 0.6695 | 0.6882 | 0.6787 |
87
- | 1.5071 | 3.0 | 57627 | 2.2419 | 0.7817 | 0.6942 | 0.6690 | 0.3588 | 0.4671 | 0.8150 | 0.8384 | 0.8265 | 0.7653 | 0.7944 | 0.7796 | 0.7271 | 0.6819 | 0.7038 |
88
- | 1.7185 | 4.0 | 76836 | 2.0814 | 0.8013 | 0.7238 | 0.6724 | 0.4270 | 0.5223 | 0.8264 | 0.8579 | 0.8418 | 0.7933 | 0.8072 | 0.8002 | 0.7571 | 0.7067 | 0.7310 |
89
- | 1.4517 | 5.0 | 96045 | 2.0802 | 0.8101 | 0.7378 | 0.6473 | 0.4729 | 0.5465 | 0.8264 | 0.8726 | 0.8489 | 0.8190 | 0.7991 | 0.8089 | 0.7599 | 0.7339 | 0.7467 |
90
 
91
- It achieves the following results on the evaluation set:
92
- - Loss: 2.0802
93
- - Accuracy: 0.8101
94
- - F1 Macro: 0.7378
95
- - Low Precision: 0.6473
96
- - Low Recall: 0.4729
97
- - Low F1: 0.5465
98
- - Medium Precision: 0.8264
99
- - Medium Recall: 0.8726
100
- - Medium F1: 0.8489
101
- - High Precision: 0.8190
102
- - High Recall: 0.7991
103
- - High F1: 0.8089
104
- - Critical Precision: 0.7599
105
- - Critical Recall: 0.7339
106
- - Critical F1: 0.7467
107
 
108
  ### Framework versions
109
 
110
- - Transformers 5.17.0
111
- - Pytorch 2.14.0+cu130
112
  - Datasets 4.8.5
113
  - Tokenizers 0.23.2
 
1
  ---
2
  library_name: transformers
3
+ license: mit
4
  base_model: roberta-base
 
 
5
  tags:
6
  - generated_from_trainer
7
+ metrics:
8
+ - accuracy
 
 
9
  model-index:
10
  - name: vulnerability-severity-classification-roberta-base
11
  results: []
 
 
12
  ---
13
 
14
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
+ should probably proofread and complete it, then remove this comment. -->
16
 
17
+ # vulnerability-severity-classification-roberta-base
 
 
 
 
 
 
 
 
 
 
18
 
19
+ This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
20
+ It achieves the following results on the evaluation set:
21
+ - Loss: 2.0906
22
+ - Accuracy: 0.8109
23
+ - F1 Macro: 0.7383
24
+ - Low Precision: 0.6926
25
+ - Low Recall: 0.4428
26
+ - Low F1: 0.5402
27
+ - Medium Precision: 0.8274
28
+ - Medium Recall: 0.8662
29
+ - Medium F1: 0.8464
30
+ - High Precision: 0.8130
31
+ - High Recall: 0.8124
32
+ - High F1: 0.8127
33
+ - Critical Precision: 0.7677
34
+ - Critical Recall: 0.7406
35
+ - Critical F1: 0.7539
36
 
37
  ## Model description
38
 
39
+ More information needed
 
 
 
 
 
 
40
 
41
+ ## Intended uses & limitations
42
 
43
+ More information needed
 
 
 
44
 
45
+ ## Training and evaluation data
46
 
47
+ More information needed
 
 
 
 
 
 
 
 
 
 
 
 
 
48
 
49
  ## Training procedure
50
 
 
63
 
64
  | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 |
65
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:|
66
+ | 2.3598 | 1.0 | 19363 | 2.6454 | 0.7298 | 0.6067 | 0.6491 | 0.1836 | 0.2862 | 0.7618 | 0.8161 | 0.7880 | 0.7188 | 0.7317 | 0.7252 | 0.6449 | 0.6110 | 0.6275 |
67
+ | 2.3058 | 2.0 | 38726 | 2.3857 | 0.7570 | 0.6550 | 0.6429 | 0.2886 | 0.3984 | 0.7937 | 0.8234 | 0.8083 | 0.7285 | 0.7860 | 0.7562 | 0.7298 | 0.5979 | 0.6573 |
68
+ | 2.0088 | 3.0 | 58089 | 2.2418 | 0.7775 | 0.6871 | 0.6675 | 0.3581 | 0.4661 | 0.8082 | 0.8369 | 0.8223 | 0.7501 | 0.8133 | 0.7804 | 0.7771 | 0.6041 | 0.6798 |
69
+ | 1.5341 | 4.0 | 77452 | 2.1168 | 0.8009 | 0.7309 | 0.6294 | 0.4803 | 0.5448 | 0.8208 | 0.8561 | 0.8381 | 0.7964 | 0.8080 | 0.8021 | 0.7860 | 0.6963 | 0.7384 |
70
+ | 1.4963 | 5.0 | 96815 | 2.0906 | 0.8109 | 0.7383 | 0.6926 | 0.4428 | 0.5402 | 0.8274 | 0.8662 | 0.8464 | 0.8130 | 0.8124 | 0.8127 | 0.7677 | 0.7406 | 0.7539 |
71
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
72
 
73
  ### Framework versions
74
 
75
+ - Transformers 5.18.0
76
+ - Pytorch 2.14.1+cu130
77
  - Datasets 4.8.5
78
  - Tokenizers 0.23.2
config.json CHANGED
@@ -34,7 +34,7 @@
34
  "pad_token_id": 1,
35
  "problem_type": "single_label_classification",
36
  "tie_word_embeddings": true,
37
- "transformers_version": "5.17.0",
38
  "type_vocab_size": 1,
39
  "use_cache": true,
40
  "vocab_size": 50265
 
34
  "pad_token_id": 1,
35
  "problem_type": "single_label_classification",
36
  "tie_word_embeddings": true,
37
+ "transformers_version": "5.18.0",
38
  "type_vocab_size": 1,
39
  "use_cache": true,
40
  "vocab_size": 50265
emissions.csv CHANGED
@@ -1,2 +1,2 @@
1
  timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
2
- 2026-09-29T12:11:39,VulnTrain,6f754b7c-7147-4fa0-a032-65d6ce7b7a72,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,17152.291897212,0.5226085257974022,3.046872854830258e-05,70.00021089819239,903.3599994824584,70.0,0.3334758593117621,4.298650257806345,0.332662325139881,4.9647884422579835,0.0,Luxembourg,LUX,luxembourg,,,Linux-6.8.0-142-generic-x86_64-with-glibc2.39,3.12.3,3.3.1,224,Intel(R) Xeon(R) Platinum 8480+,4,4 x NVIDIA L40S,6.1327,49.6098,2015.3354301452637,machine,1.0355438287892114,76.39231896804456,1.0096100850190561,21.07601278165819,N,1.0,0.0
 
1
  timestamp,project_name,run_id,experiment_id,duration,emissions,emissions_rate,cpu_power,gpu_power,ram_power,cpu_energy,gpu_energy,ram_energy,energy_consumed,water_consumed,country_name,country_iso_code,region,cloud_provider,cloud_region,os,python_version,codecarbon_version,cpu_count,cpu_model,gpu_count,gpu_model,longitude,latitude,ram_total_size,tracking_mode,cpu_utilization_percent,gpu_utilization_percent,ram_utilization_percent,ram_used_gb,on_cloud,pue,wue
2
+ 2026-10-06T17:03:10,VulnTrain,f97f5062-11d3-48e3-baf8-1137ac1c569f,5b0fa12a-3dd7-45bb-9766-cc326314d9f1,17323.065519523,0.527089473218873,3.042703224927748e-05,70.0002182038376,902.0948631070827,70.0,0.3367934261991617,4.334508493715132,0.33605558670125557,5.007357506615553,0.0,Luxembourg,LUX,luxembourg,,,Linux-6.8.0-146-generic-x86_64-with-glibc2.39,3.12.3,3.3.1,224,Intel(R) Xeon(R) Platinum 8480+,4,4 x NVIDIA L40S,6.1327,49.6098,2015.3354225158691,machine,1.0509953434225843,75.42930733410942,1.0833061699650757,21.249004599823248,N,1.0,0.0
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e9a3c415f5df1d793f5862da7cfb2bf245c1f6515a2ce414bb7a1fea90c22f64
3
  size 498618976
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f41377fab50baa54c54fa7fe391c7485ae1e7c2bdaa2bcbbb942a6e1f58af8f3
3
  size 498618976