Update model card with evaluation metrics

README.md

@@ -1,75 +1,84 @@
 ---
+language:
+- zh
+license: cc-by-4.0
 library_name: transformers
-license: apache-2.0
-base_model: hfl/chinese-macbert-base
 tags:
-- generated_from_trainer
-metrics:
-- accuracy
-model-index:
-- name: vulnerability-severity-classification-chinese-macbert-base
-  results: []
+- text-classification
+- vulnerability
+- severity
+- cybersecurity
+- cnvd
+datasets:
+- CIRCL/Vulnerability-CNVD
+base_model: hfl/chinese-macbert-base
+pipeline_tag: text-classification
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# VLAI: Automated Vulnerability Severity Classification (Chinese Text)
+
+A fine-tuned [hfl/chinese-macbert-base](https://huggingface.co/hfl/chinese-macbert-base) model for classifying Chinese vulnerability descriptions from the [China National Vulnerability Database (CNVD)](https://www.cnvd.org.cn/) into three severity levels: **Low**, **Medium**, and **High**.
+
+Trained on the [CIRCL/Vulnerability-CNVD](https://huggingface.co/datasets/CIRCL/Vulnerability-CNVD) dataset as part of the [VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) project.
+
+## Evaluation results
+
+Evaluated on a **deduplicated test set** (25,664 samples) where no description text appears in both train and test splits, preventing data leakage from CNVD's reuse of boilerplate descriptions across different vulnerability IDs.
+
+| Class  | Precision | Recall | F1-score | Support |
+|--------|-----------|--------|----------|---------|
+| Low    | 0.6091    | 0.3966 | 0.4804   | 2,267   |
+| Medium | 0.7743 | 0.8387 | 0.8052 | 14,353 |
+| High   | 0.7808   | 0.7461 | 0.7631   | 9,044   |
+
+- **Overall accuracy**: 76.70%
+- **Macro F1**: 0.6829
 
-# vulnerability-severity-classification-chinese-macbert-base
+### Class distribution
 
-This model is a fine-tuned version of [hfl/chinese-macbert-base](https://huggingface.co/hfl/chinese-macbert-base) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 2.5112
-- Accuracy: 0.7649
-- F1 Macro: 0.6854
-- Low Precision: 0.5950
-- Low Recall: 0.4213
-- Low F1: 0.4933
-- Medium Precision: 0.7735
-- Medium Recall: 0.8355
-- Medium F1: 0.8033
-- High Precision: 0.7812
-- High Recall: 0.7391
-- High F1: 0.7595
+The dataset is imbalanced: Low (8.8%), Medium (55.9%), High (35.2%).
 
-## Model description
+## Usage
 
-More information needed
+```python
+from transformers import pipeline
 
-## Intended uses & limitations
+classifier = pipeline(
+    "text-classification",
+    model="CIRCL/vulnerability-severity-classification-chinese-macbert-base"
+)
 
-More information needed
+description = "TOTOLINK A3600R存在缓冲区溢出漏洞，攻击者可利用该漏洞在系统上执行任意代码或者导致拒绝服务。"
+result = classifier(description)
+print(result)
+```
 
-## Training and evaluation data
+## Known limitations
 
-More information needed
+- **Low severity recall**: the Low class has the lowest recall. Approximately 60% of Low-severity entries are misclassified, mostly as Medium. This reflects the vocabulary overlap between Low and Medium descriptions in CNVD data. Class-weighted loss and focal loss were tested but all degraded Medium recall disproportionately without a net benefit.
 
-## Training procedure
+- **Keyword dependency**: the model biases toward a vulnerability type's typical severity. For example, buffer overflow descriptions are predicted as High regardless of the actual assigned severity. On entries where the actual severity deviates from the type's typical severity, accuracy drops from ~89% to ~55%.
 
-### Training hyperparameters
+- **Negation blindness**: the model does not understand negation. Descriptions like "does NOT allow remote code execution" can still produce high-confidence High severity predictions.
 
-The following hyperparameters were used during training:
-- learning_rate: 3e-05
-- train_batch_size: 64
-- eval_batch_size: 64
-- seed: 42
-- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- num_epochs: 5
+- **CVE overlap**: 81% of CNVD entries have a corresponding CVE. The model primarily adds value for the ~19% of CNVD-only entries (concentrated in Chinese domestic software) where no CVE severity assessment exists.
 
-### Training results
+These limitations were identified through independent analysis in [VulnTrain#19](https://github.com/vulnerability-lookup/VulnTrain/issues/19).
 
-| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1 Macro | Low Precision | Low Recall | Low F1 | Medium Precision | Medium Recall | Medium F1 | High Precision | High Recall | High F1 |
-|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:-------------:|:----------:|:------:|:----------------:|:-------------:|:---------:|:--------------:|:-----------:|:-------:|
-| 2.5562        | 1.0   | 1608 | 2.4217          | 0.7411   | 0.6552   | 0.5078        | 0.3882     | 0.44   | 0.7645           | 0.7937        | 0.7788    | 0.7474         | 0.7461      | 0.7467  |
-| 1.9715        | 2.0   | 3216 | 2.3503          | 0.7604   | 0.6720   | 0.6029        | 0.3710     | 0.4593 | 0.7701           | 0.8306        | 0.7992    | 0.7684         | 0.7467      | 0.7574  |
-| 1.9176        | 3.0   | 4824 | 2.2952          | 0.7594   | 0.6822   | 0.5748        | 0.4389     | 0.4977 | 0.7632           | 0.8405        | 0.8000    | 0.7914         | 0.7110      | 0.7490  |
-| 1.6097        | 4.0   | 6432 | 2.3841          | 0.7670   | 0.6829   | 0.6091        | 0.3966     | 0.4804 | 0.7743           | 0.8387        | 0.8052    | 0.7808         | 0.7461      | 0.7631  |
-| 1.3958        | 5.0   | 8040 | 2.5112          | 0.7649   | 0.6854   | 0.5950        | 0.4213     | 0.4933 | 0.7735           | 0.8355        | 0.8033    | 0.7812         | 0.7391      | 0.7595  |
+## Training details
 
+- **Base model**: [hfl/chinese-macbert-base](https://huggingface.co/hfl/chinese-macbert-base)
+- **Dataset**: [CIRCL/Vulnerability-CNVD](https://huggingface.co/datasets/CIRCL/Vulnerability-CNVD)
+- **Train/test split**: deduplicated on description text (no leakage), 80/20 split
+- **Loss**: uniform cross-entropy (no class weighting)
+- **Learning rate**: 3e-05
+- **Batch size**: 16
+- **Epochs**: 5
+- **Best model selection**: by accuracy
 
-### Framework versions
+## References
 
-- Transformers 5.8.1
-- Pytorch 2.12.0+cu130
-- Datasets 4.8.5
-- Tokenizers 0.22.2
+- [Vulnerability-Lookup](https://vulnerability.circl.lu) — the vulnerability data source
+- [VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) — training pipeline
+- [ML-Gateway](https://github.com/vulnerability-lookup/ML-Gateway) — inference API
+- [VLAI paper](https://arxiv.org/abs/2507.03607) — Bonhomme, C., Dulaunoy, A. (2025)