Rewrite model card: negative-result framing, usage, data, artifacts

README.md

@@ -1,43 +1,161 @@
 ---
 library_name: transformers
-license: mit
+license: cc-by-4.0
 base_model: roberta-base
+pipeline_tag: text-classification
+language:
+- en
+datasets:
+- CIRCL/vulnerability-attack-techniques
+- CIRCL/vulnerability-attack-techniques-llm-scaling
 tags:
+- security
+- vulnerability
+- cve
+- mitre-attack
+- cti
+- multi-label-classification
+- negative-result
 - generated_from_trainer
 model-index:
 - name: vulnerability-attack-technique-classification-roberta-base-llm-expanded
-  results: []
+  results:
+  - task:
+      type: text-classification
+      name: Multi-label MITRE ATT&CK technique classification
+    dataset:
+      name: CIRCL/vulnerability-attack-techniques
+      type: CIRCL/vulnerability-attack-techniques
+      split: test
+    metrics:
+    - type: recall
+      name: Recall@5
+      value: 0.6156
+    - type: recall
+      name: Recall@3
+      value: 0.5337
+    - type: f1
+      name: F1 micro
+      value: 0.3790
+    - type: f1
+      name: F1 macro
+      value: 0.1480
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 # vulnerability-attack-technique-classification-roberta-base-llm-expanded
 
-This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.5859
-- F1 Micro: 0.3790
-- F1 Macro: 0.1480
-- Precision Micro: 0.2690
-- Recall Micro: 0.6415
-- Recall At 3: 0.5337
-- Recall At 5: 0.6156
-
-## Model description
-
-More information needed
+**This is a negative-result comparison checkpoint, published for
+reproducibility. For applications, use
+[CIRCL/vulnerability-attack-technique-classification-roberta-base](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base).**
+
+A multi-label classifier that suggests [MITRE ATT&CK](https://attack.mitre.org/)
+(Enterprise) techniques from a free-text vulnerability description. It is
+identical to the released gold-only model — same base model
+([roberta-base](https://huggingface.co/roberta-base)), same 53-technique
+label vocabulary, same seed, same evaluation protocol — except for one
+thing: its training set folds 984 additional LLM-labeled CVEs
+([CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling),
+labeled by qwen3.5:122b at ≈0.39 agreement with the expert gold labels)
+into the 972 expert-labeled training rows.
+
+The paper
+[*Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and
+the Limits of LLM-Assisted Label Expansion*](https://github.com/vulnerability-lookup/cve-attack-mapping-paper)
+uses this pair of checkpoints to answer the question "can LLM-assisted
+labeling extend a small expert gold set?" — and the answer is **no, not at
+this agreement level**: no reliable ranking improvement at any expansion
+size from 100 to 984 CVEs, and measurable degradation of rare-technique
+coverage at scale.
+
+DOI: [10.57967/hf/9624](https://doi.org/10.57967/hf/9624)
+
+## What this checkpoint shows
+
+Five seeds, corrected protocol (checkpoint selection on the validation
+split), identical test split — gold-only vs. this configuration
+(gold + 984 LLM rows):
+
+| Metric | Gold-only | Gold + 984 LLM |
+|--------|-----------|----------------|
+| Recall@5 | **0.673 ± 0.019** | 0.651 ± 0.022 |
+| Recall@3 | 0.536 ± 0.032 | 0.534 ± 0.012 |
+| F1 micro | 0.410 ± 0.006 | 0.427 ± 0.028 |
+| F1 macro | **0.177 ± 0.014** | 0.151 ± 0.014 |
+
+The pattern: the noisy labels concentrate mass on frequent, "obvious"
+techniques (micro-F1 up a little) while deflating exactly the
+rare-technique coverage the expert labels paid for (macro-F1 down ≈3 SEM,
+no recall@5 gain). On CVE-2021-44077, for example, this checkpoint is more
+confident than the gold model about T1190 (*Exploit Public-Facing
+Application*) but drops the analyst-credited T1505 (*Server Software
+Component*) below the prediction threshold and pushes tail techniques such
+as T1136 (*Create Account*) from rank 18 to 32. Section 6 of the paper
+gives the full account, including why an earlier apparent gain turned out
+to be evaluation noise.
+
+## How to use
+
+Same interface as the gold-only model:
+
+```python
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+model_id = "CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSequenceClassification.from_pretrained(model_id)
+model.eval()
+
+description = "..."  # free-text vulnerability description
+inputs = tokenizer(description, truncation=True, max_length=512, return_tensors="pt")
+with torch.no_grad():
+    probs = torch.sigmoid(model(**inputs).logits)[0]
+
+for i in probs.argsort(descending=True)[:5]:
+    print(f"{model.config.id2label[int(i)]}  {probs[i]:.4f}")
+```
+
+Or side by side with the released model on a live CVE:
+
+```bash
+vulntrain-infer-attack-classification --cve CVE-2021-44077 \
+    --model CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded
+```
 
 ## Intended uses & limitations
 
-More information needed
+**Intended**: reproducing and extending the paper's expansion experiments —
+e.g. contrasting its per-technique behaviour with the gold-only checkpoint,
+or as a baseline for better silver-labeling strategies (higher-agreement
+labelers, agreement-weighted losses, human-in-the-loop curation).
+
+**Not intended**: production use. It is strictly dominated by the gold-only
+model on ranking and rare-technique metrics, which is why Vulnerability-Lookup
+deploys the gold-only checkpoint. All limitations of the gold-only model
+(53-technique vocabulary, KEV-skewed data, English only, 512-token
+truncation, uncalibrated scores, unverified suggestions) apply here too.
 
 ## Training and evaluation data
 
-More information needed
+- **Expert rows**: the 972-row train split of
+  [CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques)
+  (MITRE CTID gold mappings).
+- **LLM rows (train only)**: 984 CVEs from
+  [CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling),
+  labeled by qwen3.5:122b (Ollama, assertive single-call prompt following
+  the CTID methodology) — the best configuration of the paper's labeler
+  benchmark, at ≈0.39 F1 agreement with held-out expert labels.
+- The label vocabulary stays frozen to the gold train split, and the
+  validation (106) and test (118) splits contain **only** expert-labeled
+  rows; checkpoint selection uses the validation split.
 
 ## Training procedure
 
+Binary cross-entropy over 53 sigmoid outputs with balanced per-label
+`pos_weight`, trained with `vulntrain-train-attack-classification`
+(VulnTrain), like the gold-only model — only the training set differs
+(1,956 rows instead of 972).
+
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
@@ -48,6 +166,9 @@ The following hyperparameters were used during training:
 - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: linear
 - num_epochs: 40
+- max_length: 512
+- loss: BCEWithLogitsLoss, balanced pos_weight
+- checkpoint selection: best macro-F1 on the validation split
 
 ### Training results
 
@@ -94,10 +215,37 @@ The following hyperparameters were used during training:
 | 0.2794        | 39.0  | 2418 | 0.6142          | 0.3975   | 0.1468   | 0.3176          | 0.5311       | 0.4984      | 0.6801      |
 | 0.2661        | 40.0  | 2480 | 0.6123          | 0.4025   | 0.1516   | 0.3195          | 0.5436       | 0.4972      | 0.6825      |
 
-
 ### Framework versions
 
 - Transformers 5.13.0
 - Pytorch 2.12.1+cu130
 - Datasets 4.8.5
 - Tokenizers 0.22.2
+
+## Related artifacts
+
+| Artifact | Location | DOI |
+|----------|----------|-----|
+| **Released model (use this one)** | [CIRCL/vulnerability-attack-technique-classification-roberta-base](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base) | [10.57967/hf/9623](https://doi.org/10.57967/hf/9623) |
+| Gold dataset (1,207 CVEs, CTID-curated labels) | [CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques) | [10.57967/hf/9621](https://doi.org/10.57967/hf/9621) |
+| LLM expansion dataset (984 LLM-labeled CVEs) | [CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling) | [10.57967/hf/9622](https://doi.org/10.57967/hf/9622) |
+| Code | [vulnerability-lookup/VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) | — |
+| Paper + trainer logs | [vulnerability-lookup/cve-attack-mapping-paper](https://github.com/vulnerability-lookup/cve-attack-mapping-paper) | — |
+
+## Citation
+
+```bibtex
+@misc{bonhomme2026cveattack,
+  title  = {Mapping CVEs to MITRE ATT\&CK Techniques: A Curated Gold-Set
+            Classifier and the Limits of LLM-Assisted Label Expansion},
+  author = {Bonhomme, C{\'e}dric},
+  year   = {2026},
+  note   = {Preprint},
+}
+```
+
+## Acknowledgements
+
+Developed at [CIRCL](https://www.circl.lu) in the context of the
+[AIPITCH](https://www.science.nask.pl/en/research-areas/projects/12456)
+project, co-funded by the European Union.