README.md

---
library_name: transformers
license: cc-by-4.0
base_model: roberta-base
pipeline_tag: text-classification
language:
- en
datasets:
- CIRCL/vulnerability-attack-techniques
- CIRCL/vulnerability-attack-techniques-llm-scaling
tags:
- security
- vulnerability
- cve
- mitre-attack
- cti
- multi-label-classification
- negative-result
- generated_from_trainer
model-index:
- name: vulnerability-attack-technique-classification-roberta-base-llm-expanded
  results:
  - task:
      type: text-classification
      name: Multi-label MITRE ATT&CK technique classification
    dataset:
      name: CIRCL/vulnerability-attack-techniques
      type: CIRCL/vulnerability-attack-techniques
      split: test
    metrics:
    - type: recall
      name: Recall@5
      value: 0.6156
    - type: recall
      name: Recall@3
      value: 0.5337
    - type: f1
      name: F1 micro
      value: 0.3790
    - type: f1
      name: F1 macro
      value: 0.1480
---

# vulnerability-attack-technique-classification-roberta-base-llm-expanded

**This is a negative-result comparison checkpoint, published for
reproducibility. For applications, use
[CIRCL/vulnerability-attack-technique-classification-roberta-base](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base).**

A multi-label classifier that suggests [MITRE ATT&CK](https://attack.mitre.org/)
(Enterprise) techniques from a free-text vulnerability description. It is
identical to the released gold-only model — same base model
([roberta-base](https://huggingface.co/roberta-base)), same 53-technique
label vocabulary, same seed, same evaluation protocol — except for one
thing: its training set folds 984 additional LLM-labeled CVEs
([CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling),
labeled by qwen3.5:122b at ≈0.39 agreement with the expert gold labels)
into the 972 expert-labeled training rows.

The paper
[*Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and
the Limits of LLM-Assisted Label Expansion*](https://arxiv.org/abs/2607.25572)
(arXiv:2607.25572) uses this pair of checkpoints to answer the question "can LLM-assisted
labeling extend a small expert gold set?" — and the answer is **no, not at
this agreement level**: no reliable ranking improvement at any expansion
size from 100 to 984 CVEs, and measurable degradation of rare-technique
coverage at scale.

DOI: [10.57967/hf/9624](https://doi.org/10.57967/hf/9624)

## What this checkpoint shows

Five seeds, corrected protocol (checkpoint selection on the validation
split), identical test split — gold-only vs. this configuration
(gold + 984 LLM rows):

| Metric | Gold-only | Gold + 984 LLM |
|--------|-----------|----------------|
| Recall@5 | **0.673 ± 0.019** | 0.651 ± 0.022 |
| Recall@3 | 0.536 ± 0.032 | 0.534 ± 0.012 |
| F1 micro | 0.410 ± 0.006 | 0.427 ± 0.028 |
| F1 macro | **0.177 ± 0.014** | 0.151 ± 0.014 |

The pattern: the noisy labels concentrate mass on frequent, "obvious"
techniques (micro-F1 up a little) while deflating exactly the
rare-technique coverage the expert labels paid for (macro-F1 down ≈3 SEM,
no recall@5 gain). On CVE-2021-44077, for example, this checkpoint is more
confident than the gold model about T1190 (*Exploit Public-Facing
Application*) but drops the analyst-credited T1505 (*Server Software
Component*) below the prediction threshold and pushes tail techniques such
as T1136 (*Create Account*) from rank 18 to 32. Section 6 of the paper
gives the full account, including why an earlier apparent gain turned out
to be evaluation noise.

## How to use

Same interface as the gold-only model:

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

description = "..."  # free-text vulnerability description
inputs = tokenizer(description, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    probs = torch.sigmoid(model(**inputs).logits)[0]

for i in probs.argsort(descending=True)[:5]:
    print(f"{model.config.id2label[int(i)]}  {probs[i]:.4f}")
```

Or side by side with the released model on a live CVE:

```bash
vulntrain-infer-attack-classification --cve CVE-2021-44077 \
    --model CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded
```

## Intended uses & limitations

**Intended**: reproducing and extending the paper's expansion experiments —
e.g. contrasting its per-technique behaviour with the gold-only checkpoint,
or as a baseline for better silver-labeling strategies (higher-agreement
labelers, agreement-weighted losses, human-in-the-loop curation).

**Not intended**: production use. It is strictly dominated by the gold-only
model on ranking and rare-technique metrics, which is why Vulnerability-Lookup
deploys the gold-only checkpoint. All limitations of the gold-only model
(53-technique vocabulary, KEV-skewed data, English only, 512-token
truncation, uncalibrated scores, unverified suggestions) apply here too.

## Training and evaluation data

- **Expert rows**: the 972-row train split of
  [CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques)
  (MITRE CTID gold mappings).
- **LLM rows (train only)**: 984 CVEs from
  [CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling),
  labeled by qwen3.5:122b (Ollama, assertive single-call prompt following
  the CTID methodology) — the best configuration of the paper's labeler
  benchmark, at ≈0.39 F1 agreement with held-out expert labels.
- The label vocabulary stays frozen to the gold train split, and the
  validation (106) and test (118) splits contain **only** expert-labeled
  rows; checkpoint selection uses the validation split.

## Training procedure

Binary cross-entropy over 53 sigmoid outputs with balanced per-label
`pos_weight`, trained with `vulntrain-train-attack-classification`
(VulnTrain), like the gold-only model — only the training set differs
(1,956 rows instead of 972).

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- num_epochs: 40
- max_length: 512
- loss: BCEWithLogitsLoss, balanced pos_weight
- checkpoint selection: best macro-F1 on the validation split

### Training results

| Training Loss | Epoch | Step | Validation Loss | F1 Micro | F1 Macro | Precision Micro | Recall Micro | Recall At 3 | Recall At 5 |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:---------------:|:------------:|:-----------:|:-----------:|
| 0.7195        | 1.0   | 62   | 0.7652          | 0.2547   | 0.0278   | 0.1821          | 0.4232       | 0.2862      | 0.3870      |
| 0.6487        | 2.0   | 124  | 0.7303          | 0.2628   | 0.0431   | 0.1779          | 0.5021       | 0.2717      | 0.3564      |
| 0.6442        | 3.0   | 186  | 0.7133          | 0.2931   | 0.0528   | 0.2042          | 0.5187       | 0.3741      | 0.4821      |
| 0.6053        | 4.0   | 248  | 0.6903          | 0.3353   | 0.0752   | 0.2597          | 0.4730       | 0.3730      | 0.5167      |
| 0.6037        | 5.0   | 310  | 0.6715          | 0.3392   | 0.0886   | 0.2619          | 0.4813       | 0.4193      | 0.5561      |
| 0.5560        | 6.0   | 372  | 0.6557          | 0.3356   | 0.0970   | 0.2521          | 0.5021       | 0.4002      | 0.5483      |
| 0.5204        | 7.0   | 434  | 0.6496          | 0.3205   | 0.0969   | 0.2319          | 0.5187       | 0.3782      | 0.5023      |
| 0.5067        | 8.0   | 496  | 0.6369          | 0.3470   | 0.1098   | 0.2628          | 0.5104       | 0.4092      | 0.5781      |
| 0.5060        | 9.0   | 558  | 0.6256          | 0.3626   | 0.1195   | 0.2731          | 0.5394       | 0.4548      | 0.6115      |
| 0.4426        | 10.0  | 620  | 0.6207          | 0.3212   | 0.1076   | 0.238           | 0.4938       | 0.4304      | 0.5597      |
| 0.4414        | 11.0  | 682  | 0.6174          | 0.3840   | 0.1213   | 0.3049          | 0.5187       | 0.4700      | 0.6059      |
| 0.4453        | 12.0  | 744  | 0.6184          | 0.3163   | 0.1432   | 0.2217          | 0.5519       | 0.4156      | 0.5636      |
| 0.4328        | 13.0  | 806  | 0.6122          | 0.3351   | 0.1403   | 0.2447          | 0.5311       | 0.4441      | 0.5816      |
| 0.4219        | 14.0  | 868  | 0.6133          | 0.3773   | 0.1543   | 0.2866          | 0.5519       | 0.4642      | 0.6327      |
| 0.4109        | 15.0  | 930  | 0.6075          | 0.3722   | 0.1607   | 0.2720          | 0.5892       | 0.4682      | 0.6197      |
| 0.3905        | 16.0  | 992  | 0.6038          | 0.3778   | 0.1665   | 0.2771          | 0.5934       | 0.4642      | 0.6307      |
| 0.3948        | 17.0  | 1054 | 0.6025          | 0.3781   | 0.1448   | 0.2822          | 0.5726       | 0.4645      | 0.5977      |
| 0.3785        | 18.0  | 1116 | 0.6034          | 0.3845   | 0.1521   | 0.2939          | 0.5560       | 0.4529      | 0.6422      |
| 0.3723        | 19.0  | 1178 | 0.6038          | 0.3810   | 0.1467   | 0.2875          | 0.5643       | 0.4914      | 0.6543      |
| 0.3580        | 20.0  | 1240 | 0.6036          | 0.3790   | 0.1504   | 0.2842          | 0.5685       | 0.4524      | 0.6257      |
| 0.3343        | 21.0  | 1302 | 0.5993          | 0.3878   | 0.1522   | 0.2978          | 0.5560       | 0.5228      | 0.6688      |
| 0.3386        | 22.0  | 1364 | 0.6000          | 0.3994   | 0.1501   | 0.3103          | 0.5602       | 0.4819      | 0.6740      |
| 0.3444        | 23.0  | 1426 | 0.5977          | 0.3977   | 0.1586   | 0.3013          | 0.5851       | 0.4945      | 0.6787      |
| 0.3303        | 24.0  | 1488 | 0.6003          | 0.3988   | 0.1571   | 0.3072          | 0.5685       | 0.4862      | 0.6594      |
| 0.3206        | 25.0  | 1550 | 0.6044          | 0.4      | 0.1574   | 0.3124          | 0.5560       | 0.4792      | 0.6825      |
| 0.3180        | 26.0  | 1612 | 0.6080          | 0.4031   | 0.1515   | 0.3188          | 0.5477       | 0.5008      | 0.6744      |
| 0.3100        | 27.0  | 1674 | 0.6085          | 0.4037   | 0.1500   | 0.3211          | 0.5436       | 0.5197      | 0.6289      |
| 0.3075        | 28.0  | 1736 | 0.6061          | 0.4071   | 0.1622   | 0.3158          | 0.5726       | 0.4953      | 0.6656      |
| 0.3042        | 29.0  | 1798 | 0.6135          | 0.3982   | 0.1550   | 0.3155          | 0.5394       | 0.4961      | 0.6722      |
| 0.3044        | 30.0  | 1860 | 0.6097          | 0.4      | 0.1579   | 0.3178          | 0.5394       | 0.4874      | 0.6751      |
| 0.3032        | 31.0  | 1922 | 0.6028          | 0.3875   | 0.1588   | 0.2950          | 0.5643       | 0.4796      | 0.6509      |
| 0.2868        | 32.0  | 1984 | 0.6083          | 0.3994   | 0.1522   | 0.3157          | 0.5436       | 0.5063      | 0.6869      |
| 0.2821        | 33.0  | 2046 | 0.6093          | 0.4018   | 0.1535   | 0.3187          | 0.5436       | 0.4800      | 0.6727      |
| 0.2915        | 34.0  | 2108 | 0.6044          | 0.3982   | 0.1528   | 0.3115          | 0.5519       | 0.4972      | 0.6609      |
| 0.2829        | 35.0  | 2170 | 0.6112          | 0.3988   | 0.1545   | 0.3122          | 0.5519       | 0.4952      | 0.6869      |
| 0.2915        | 36.0  | 2232 | 0.6111          | 0.4062   | 0.1510   | 0.3227          | 0.5477       | 0.5079      | 0.6853      |
| 0.2781        | 37.0  | 2294 | 0.6153          | 0.4      | 0.1507   | 0.3208          | 0.5311       | 0.5020      | 0.6778      |
| 0.2724        | 38.0  | 2356 | 0.6115          | 0.4031   | 0.1509   | 0.3203          | 0.5436       | 0.4972      | 0.6778      |
| 0.2794        | 39.0  | 2418 | 0.6142          | 0.3975   | 0.1468   | 0.3176          | 0.5311       | 0.4984      | 0.6801      |
| 0.2661        | 40.0  | 2480 | 0.6123          | 0.4025   | 0.1516   | 0.3195          | 0.5436       | 0.4972      | 0.6825      |

### Framework versions

- Transformers 5.13.0
- Pytorch 2.12.1+cu130
- Datasets 4.8.5
- Tokenizers 0.22.2

## Related artifacts

| Artifact | Location | DOI |
|----------|----------|-----|
| **Released model (use this one)** | [CIRCL/vulnerability-attack-technique-classification-roberta-base](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base) | [10.57967/hf/9623](https://doi.org/10.57967/hf/9623) |
| Gold dataset (1,207 CVEs, CTID-curated labels) | [CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques) | [10.57967/hf/9621](https://doi.org/10.57967/hf/9621) |
| LLM expansion dataset (984 LLM-labeled CVEs) | [CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling) | [10.57967/hf/9622](https://doi.org/10.57967/hf/9622) |
| Code | [vulnerability-lookup/VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) | — |
| Paper | [arXiv:2607.25572](https://arxiv.org/abs/2607.25572) | — |
| Paper LaTeX source + trainer logs | [vulnerability-lookup/cve-attack-mapping-paper](https://github.com/vulnerability-lookup/cve-attack-mapping-paper) | — |

## Citation

```bibtex
@misc{bonhomme2026mappingcvesmitreattck,
      title={Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion},
      author={Cédric Bonhomme and Alexandre Dulaunoy},
      year={2026},
      eprint={2607.25572},
      archivePrefix={arXiv},
      primaryClass={cs.CR},
      url={https://arxiv.org/abs/2607.25572},
}
```

## Acknowledgements

Developed at [CIRCL](https://www.circl.lu) in the context of the
[AIPITCH](https://www.science.nask.pl/en/research-areas/projects/12456)
project, co-funded by the European Union.