| --- |
| library_name: transformers |
| license: cc-by-4.0 |
| base_model: roberta-base |
| metrics: |
| - accuracy |
| tags: |
| - generated_from_trainer |
| - text-classification |
| - classification |
| - nlp |
| - vulnerability |
| model-index: |
| - name: vulnerability-severity-classification-roberta-base |
| results: [] |
| datasets: |
| - CIRCL/vulnerability-scores |
| --- |
| |
|
|
| # VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification |
|
|
| # Severity classification |
|
|
| This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the dataset [CIRCL/vulnerability-scores](https://huggingface.co/datasets/CIRCL/vulnerability-scores). |
|
|
| The model was presented in the paper [VLAI: A RoBERTa-Based Model for Automated Vulnerability Severity Classification](https://huggingface.co/papers/2507.03607) [[arXiv](https://arxiv.org/abs/2507.03607)]. |
|
|
| **Abstract:** VLAI is a transformer-based model that predicts software vulnerability severity levels directly from text descriptions. Built on RoBERTa, VLAI is fine-tuned on over 600,000 real-world vulnerabilities and achieves over 82% accuracy in predicting severity categories, enabling faster and more consistent triage ahead of manual CVSS scoring. The model and dataset are open-source and integrated into the Vulnerability-Lookup service. |
|
|
| You can read [this page](https://www.vulnerability-lookup.org/user-manual/ai/) for more information. |
|
|
|
|
| ## Model description |
|
|
| It is a classification model and is aimed to assist in classifying vulnerabilities by severity based on their descriptions. |
|
|
| ## How to get started with the model |
|
|
| ```python |
| from transformers import AutoModelForSequenceClassification, AutoTokenizer |
| import torch |
| |
| labels = ["low", "medium", "high", "critical"] |
| |
| model_name = "CIRCL/vulnerability-severity-classification-roberta-base" |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForSequenceClassification.from_pretrained(model_name) |
| model.eval() |
| |
| print("Model revision:", model.config._commit_hash) |
| |
| test_description = "SAP NetWeaver Visual Composer Metadata Uploader is not protected with a proper authorization, allowing unauthenticated agent to upload potentially malicious executable binaries \ |
| that could severely harm the host system. This could significantly affect the confidentiality, integrity, and availability of the targeted system." |
| inputs = tokenizer(test_description, return_tensors="pt", truncation=True, padding=True) |
| |
| # Run inference |
| with torch.no_grad(): |
| outputs = model(**inputs) |
| |
| predictions = outputs.logits |
| # Print results |
| print("Predictions:", predictions) |
| predicted_class = torch.argmax(predictions, dim=-1).item() |
| print("Predicted severity:", labels[predicted_class]) |
| ``` |
|
|
| ## Training procedure |
|
|
| ### Training hyperparameters |
|
|
| The following hyperparameters were used during training: |
| - learning_rate: 3e-05 |
| - train_batch_size: 32 |
| - eval_batch_size: 32 |
| - seed: 42 |
| - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments |
| - lr_scheduler_type: linear |
| - num_epochs: 5 |
| |
| It achieves the following results on the evaluation set: |
| - Loss: 2.0079 |
| - Accuracy: 0.8187 |
| - F1 Macro: 0.7495 |
| - Low Precision: 0.6490 |
| - Low Recall: 0.5059 |
| - Low F1: 0.5686 |
| - Medium Precision: 0.8468 |
| - Medium Recall: 0.8712 |
| - Medium F1: 0.8588 |
| - High Precision: 0.8140 |
| - High Recall: 0.8114 |
| - High F1: 0.8127 |
| - Critical Precision: 0.7671 |
| - Critical Recall: 0.7488 |
| - Critical F1: 0.7579 |
| |
| ### Training results |
| |
| | Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 | Critical Precision | Critical Recall | Critical F1 | |
| |:-------------:|:-----:|:-----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|:------------------:|:---------------:|:-----------:| |
| | 2.8382 | 1.0 | 16475 | 2.5695 | 0.7351 | 0.6552 | 0.4964 | 0.4244 | 0.4576 | 0.7986 | 0.7994 | 0.7990 | 0.7393 | 0.6933 | 0.7156 | 0.5868 | 0.7245 | 0.6484 | |
| | 2.3037 | 2.0 | 32950 | 2.3201 | 0.7709 | 0.6774 | 0.6294 | 0.3490 | 0.4490 | 0.8025 | 0.8541 | 0.8275 | 0.7629 | 0.7502 | 0.7565 | 0.6908 | 0.6630 | 0.6766 | |
| | 2.1765 | 3.0 | 49425 | 2.1006 | 0.7905 | 0.7077 | 0.6790 | 0.3867 | 0.4928 | 0.8246 | 0.8568 | 0.8404 | 0.7894 | 0.7659 | 0.7775 | 0.6903 | 0.7524 | 0.7201 | |
| | 1.7249 | 4.0 | 65900 | 2.0247 | 0.8091 | 0.7329 | 0.6677 | 0.4528 | 0.5396 | 0.8236 | 0.8874 | 0.8543 | 0.8136 | 0.7828 | 0.7979 | 0.7669 | 0.7144 | 0.7397 | |
| | 1.3227 | 5.0 | 82375 | 2.0079 | 0.8187 | 0.7495 | 0.6490 | 0.5059 | 0.5686 | 0.8468 | 0.8712 | 0.8588 | 0.8140 | 0.8114 | 0.8127 | 0.7671 | 0.7488 | 0.7579 | |
| |
| |
| ### Framework versions |
| |
| - Transformers 5.8.1 |
| - Pytorch 2.12.0+cu130 |
| - Datasets 4.8.5 |
| - Tokenizers 0.22.2 |
| |
