Upload CVEscore model

README.md

@@ -11,20 +11,34 @@ base_model: microsoft/deberta-v3-base
 Fine-tuned `microsoft/deberta-v3-base` with 8 classification heads (AV, AC, PR, UI, S, C, I, A).
 The base score and severity are computed from the predicted vector with the official CVSS v3.1 formula.
 
+## Input format
+Prefix the description with who will score the CVE, as in training:
+
+- `[NVD] <description>` predicts the vector NVD would give.
+- `[CNA: <org>] <description>` predicts the vector that CNA would give. `<org>` is the domain of the CNA's
+  email without the TLD (`security@wordfence.com` -> `wordfence`), or `cisa-adp` for CISA's ADP.
+
+Remove CVSS vectors and scores quoted in the description first (`strip_cvss` in `src/cvss_text.py`):
+the model was trained without them.
+
+With every CVE prompted as `[NVD]`, exact match is 0.393 overall (0.552 on NVD-scored CVEs).
+Tagging each CVE with its real scorer raises CNA-scored exact match from 0.339 to 0.438.
+Ablation: https://github.com/atomic-crafter/CVEscore/blob/master/results/ablation.md
+
 ## Training data and split
 NVD CVE JSON 2.0 feeds, CVEs with a CVSS v3.1 vector (NVD score preferred, CNA score otherwise).
 **Time-based split**: train on CVEs published before 2024-10-01, validate up to the end of 2024,
 test on CVEs published from 2025 onwards. No random split, so no leakage from the future.
 
-## Test results (123,460 CVEs published in 2025+)
+## Test results (123,459 CVEs published in 2025+, each tagged with its scorer)
 | Metric | Value |
 |---|---|
-| Exact vector match | 0.401 |
-| Severity accuracy | 0.629 |
-| Base score MAE | 1.036 |
-| Inference speed | 0.99 ms/CVE (RTX 5080, batched) |
+| Exact vector match | 0.467 |
+| Severity accuracy | 0.691 |
+| Base score MAE | 0.860 |
+| Inference speed | 1.01 ms/CVE (RTX 5080, batched) |
 
-Per-metric macro F1: AV 0.761, AC 0.672, PR 0.741, UI 0.880, S 0.843, C 0.760, I 0.764, A 0.625
+Per-metric macro F1: AV 0.782, AC 0.707, PR 0.746, UI 0.903, S 0.851, C 0.819, I 0.819, A 0.796
 
 ## Intended use
 Triage aid: pre-score new CVEs before official scoring is available, and route low-confidence
@@ -33,7 +47,8 @@ Not a replacement for analyst scoring.
 
 ## Limitations
 - English descriptions only; vague descriptions ("unspecified vulnerability") are scored poorly.
-- Labels mix NVD and vendor (CNA) scores, which sometimes disagree.
+- NVD and vendor (CNA) scores disagree on most CVEs scored by both (18% exact agreement in 2025+),
+  which caps achievable accuracy.
 - CVSS v3.1 only (no v4.0).
 
 ## Usage

cvss_config.json

@@ -43,17 +43,17 @@
   "max_length": 256,
   "val_start": "2024-10-01",
   "best_val": {
-    "mean_macro_f1": 0.7967916131945477,
-    "exact": 0.4899665551839465,
-    "f1_AV": 0.727,
-    "f1_AC": 0.6857,
-    "f1_PR": 0.7472,
-    "f1_UI": 0.9159,
-    "f1_S": 0.9198,
-    "f1_C": 0.8651,
-    "f1_I": 0.8507,
-    "f1_A": 0.663,
-    "step": 10360,
+    "mean_macro_f1": 0.836034779434055,
+    "exact": 0.5682831661092531,
+    "f1_AV": 0.7348,
+    "f1_AC": 0.7293,
+    "f1_PR": 0.7529,
+    "f1_UI": 0.9366,
+    "f1_S": 0.9351,
+    "f1_C": 0.878,
+    "f1_I": 0.8784,
+    "f1_A": 0.8432,
+    "step": 17448,
     "epoch": 3
   }
 }

encoder/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9fe5687f3a53d137afd6d90ba6376fbf5b3822201f8d3933ea962bb0470c77df
+oid sha256:b009823848a5c5833226ddd386c081955551d3cfb44b2cefbf937f5873c37cb8
 size 735348840

heads.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:53e6f7f799ff589a01f8b87552653559c51c373d84441dc79be94eba323a58df
+oid sha256:b4e87e246fade5a7699ffe5571cc235a17774521a1512c78e31c590d198acacd
 size 72839