--- library_name: transformers license: cc-by-4.0 base_model: roberta-base pipeline_tag: text-classification language: - en datasets: - CIRCL/vulnerability-attack-techniques - CIRCL/vulnerability-attack-techniques-llm-scaling tags: - security - vulnerability - cve - mitre-attack - cti - multi-label-classification - negative-result - generated_from_trainer model-index: - name: vulnerability-attack-technique-classification-roberta-base-llm-expanded results: - task: type: text-classification name: Multi-label MITRE ATT&CK technique classification dataset: name: CIRCL/vulnerability-attack-techniques type: CIRCL/vulnerability-attack-techniques split: test metrics: - type: recall name: Recall@5 value: 0.6156 - type: recall name: Recall@3 value: 0.5337 - type: f1 name: F1 micro value: 0.3790 - type: f1 name: F1 macro value: 0.1480 --- # vulnerability-attack-technique-classification-roberta-base-llm-expanded **This is a negative-result comparison checkpoint, published for reproducibility. For applications, use [CIRCL/vulnerability-attack-technique-classification-roberta-base](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base).** A multi-label classifier that suggests [MITRE ATT&CK](https://attack.mitre.org/) (Enterprise) techniques from a free-text vulnerability description. It is identical to the released gold-only model — same base model ([roberta-base](https://huggingface.co/roberta-base)), same 53-technique label vocabulary, same seed, same evaluation protocol — except for one thing: its training set folds 984 additional LLM-labeled CVEs ([CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling), labeled by qwen3.5:122b at ≈0.39 agreement with the expert gold labels) into the 972 expert-labeled training rows. The paper [*Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion*](https://arxiv.org/abs/2607.25572) (arXiv:2607.25572) uses this pair of checkpoints to answer the question "can LLM-assisted labeling extend a small expert gold set?" — and the answer is **no, not at this agreement level**: no reliable ranking improvement at any expansion size from 100 to 984 CVEs, and measurable degradation of rare-technique coverage at scale. DOI: [10.57967/hf/9624](https://doi.org/10.57967/hf/9624) ## What this checkpoint shows Five seeds, corrected protocol (checkpoint selection on the validation split), identical test split — gold-only vs. this configuration (gold + 984 LLM rows): | Metric | Gold-only | Gold + 984 LLM | |--------|-----------|----------------| | Recall@5 | **0.673 ± 0.019** | 0.651 ± 0.022 | | Recall@3 | 0.536 ± 0.032 | 0.534 ± 0.012 | | F1 micro | 0.410 ± 0.006 | 0.427 ± 0.028 | | F1 macro | **0.177 ± 0.014** | 0.151 ± 0.014 | The pattern: the noisy labels concentrate mass on frequent, "obvious" techniques (micro-F1 up a little) while deflating exactly the rare-technique coverage the expert labels paid for (macro-F1 down ≈3 SEM, no recall@5 gain). On CVE-2021-44077, for example, this checkpoint is more confident than the gold model about T1190 (*Exploit Public-Facing Application*) but drops the analyst-credited T1505 (*Server Software Component*) below the prediction threshold and pushes tail techniques such as T1136 (*Create Account*) from rank 18 to 32. Section 6 of the paper gives the full account, including why an earlier apparent gain turned out to be evaluation noise. ## How to use Same interface as the gold-only model: ```python import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer model_id = "CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForSequenceClassification.from_pretrained(model_id) model.eval() description = "..." # free-text vulnerability description inputs = tokenizer(description, truncation=True, max_length=512, return_tensors="pt") with torch.no_grad(): probs = torch.sigmoid(model(**inputs).logits)[0] for i in probs.argsort(descending=True)[:5]: print(f"{model.config.id2label[int(i)]} {probs[i]:.4f}") ``` Or side by side with the released model on a live CVE: ```bash vulntrain-infer-attack-classification --cve CVE-2021-44077 \ --model CIRCL/vulnerability-attack-technique-classification-roberta-base-llm-expanded ``` ## Intended uses & limitations **Intended**: reproducing and extending the paper's expansion experiments — e.g. contrasting its per-technique behaviour with the gold-only checkpoint, or as a baseline for better silver-labeling strategies (higher-agreement labelers, agreement-weighted losses, human-in-the-loop curation). **Not intended**: production use. It is strictly dominated by the gold-only model on ranking and rare-technique metrics, which is why Vulnerability-Lookup deploys the gold-only checkpoint. All limitations of the gold-only model (53-technique vocabulary, KEV-skewed data, English only, 512-token truncation, uncalibrated scores, unverified suggestions) apply here too. ## Training and evaluation data - **Expert rows**: the 972-row train split of [CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques) (MITRE CTID gold mappings). - **LLM rows (train only)**: 984 CVEs from [CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling), labeled by qwen3.5:122b (Ollama, assertive single-call prompt following the CTID methodology) — the best configuration of the paper's labeler benchmark, at ≈0.39 F1 agreement with held-out expert labels. - The label vocabulary stays frozen to the gold train split, and the validation (106) and test (118) splits contain **only** expert-labeled rows; checkpoint selection uses the validation split. ## Training procedure Binary cross-entropy over 53 sigmoid outputs with balanced per-label `pos_weight`, trained with `vulntrain-train-attack-classification` (VulnTrain), like the gold-only model — only the training set differs (1,956 rows instead of 972). ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 1e-05 - train_batch_size: 32 - eval_batch_size: 32 - seed: 42 - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - num_epochs: 40 - max_length: 512 - loss: BCEWithLogitsLoss, balanced pos_weight - checkpoint selection: best macro-F1 on the validation split ### Training results | Training Loss | Epoch | Step | Validation Loss | F1 Micro | F1 Macro | Precision Micro | Recall Micro | Recall At 3 | Recall At 5 | |:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:---------------:|:------------:|:-----------:|:-----------:| | 0.7195 | 1.0 | 62 | 0.7652 | 0.2547 | 0.0278 | 0.1821 | 0.4232 | 0.2862 | 0.3870 | | 0.6487 | 2.0 | 124 | 0.7303 | 0.2628 | 0.0431 | 0.1779 | 0.5021 | 0.2717 | 0.3564 | | 0.6442 | 3.0 | 186 | 0.7133 | 0.2931 | 0.0528 | 0.2042 | 0.5187 | 0.3741 | 0.4821 | | 0.6053 | 4.0 | 248 | 0.6903 | 0.3353 | 0.0752 | 0.2597 | 0.4730 | 0.3730 | 0.5167 | | 0.6037 | 5.0 | 310 | 0.6715 | 0.3392 | 0.0886 | 0.2619 | 0.4813 | 0.4193 | 0.5561 | | 0.5560 | 6.0 | 372 | 0.6557 | 0.3356 | 0.0970 | 0.2521 | 0.5021 | 0.4002 | 0.5483 | | 0.5204 | 7.0 | 434 | 0.6496 | 0.3205 | 0.0969 | 0.2319 | 0.5187 | 0.3782 | 0.5023 | | 0.5067 | 8.0 | 496 | 0.6369 | 0.3470 | 0.1098 | 0.2628 | 0.5104 | 0.4092 | 0.5781 | | 0.5060 | 9.0 | 558 | 0.6256 | 0.3626 | 0.1195 | 0.2731 | 0.5394 | 0.4548 | 0.6115 | | 0.4426 | 10.0 | 620 | 0.6207 | 0.3212 | 0.1076 | 0.238 | 0.4938 | 0.4304 | 0.5597 | | 0.4414 | 11.0 | 682 | 0.6174 | 0.3840 | 0.1213 | 0.3049 | 0.5187 | 0.4700 | 0.6059 | | 0.4453 | 12.0 | 744 | 0.6184 | 0.3163 | 0.1432 | 0.2217 | 0.5519 | 0.4156 | 0.5636 | | 0.4328 | 13.0 | 806 | 0.6122 | 0.3351 | 0.1403 | 0.2447 | 0.5311 | 0.4441 | 0.5816 | | 0.4219 | 14.0 | 868 | 0.6133 | 0.3773 | 0.1543 | 0.2866 | 0.5519 | 0.4642 | 0.6327 | | 0.4109 | 15.0 | 930 | 0.6075 | 0.3722 | 0.1607 | 0.2720 | 0.5892 | 0.4682 | 0.6197 | | 0.3905 | 16.0 | 992 | 0.6038 | 0.3778 | 0.1665 | 0.2771 | 0.5934 | 0.4642 | 0.6307 | | 0.3948 | 17.0 | 1054 | 0.6025 | 0.3781 | 0.1448 | 0.2822 | 0.5726 | 0.4645 | 0.5977 | | 0.3785 | 18.0 | 1116 | 0.6034 | 0.3845 | 0.1521 | 0.2939 | 0.5560 | 0.4529 | 0.6422 | | 0.3723 | 19.0 | 1178 | 0.6038 | 0.3810 | 0.1467 | 0.2875 | 0.5643 | 0.4914 | 0.6543 | | 0.3580 | 20.0 | 1240 | 0.6036 | 0.3790 | 0.1504 | 0.2842 | 0.5685 | 0.4524 | 0.6257 | | 0.3343 | 21.0 | 1302 | 0.5993 | 0.3878 | 0.1522 | 0.2978 | 0.5560 | 0.5228 | 0.6688 | | 0.3386 | 22.0 | 1364 | 0.6000 | 0.3994 | 0.1501 | 0.3103 | 0.5602 | 0.4819 | 0.6740 | | 0.3444 | 23.0 | 1426 | 0.5977 | 0.3977 | 0.1586 | 0.3013 | 0.5851 | 0.4945 | 0.6787 | | 0.3303 | 24.0 | 1488 | 0.6003 | 0.3988 | 0.1571 | 0.3072 | 0.5685 | 0.4862 | 0.6594 | | 0.3206 | 25.0 | 1550 | 0.6044 | 0.4 | 0.1574 | 0.3124 | 0.5560 | 0.4792 | 0.6825 | | 0.3180 | 26.0 | 1612 | 0.6080 | 0.4031 | 0.1515 | 0.3188 | 0.5477 | 0.5008 | 0.6744 | | 0.3100 | 27.0 | 1674 | 0.6085 | 0.4037 | 0.1500 | 0.3211 | 0.5436 | 0.5197 | 0.6289 | | 0.3075 | 28.0 | 1736 | 0.6061 | 0.4071 | 0.1622 | 0.3158 | 0.5726 | 0.4953 | 0.6656 | | 0.3042 | 29.0 | 1798 | 0.6135 | 0.3982 | 0.1550 | 0.3155 | 0.5394 | 0.4961 | 0.6722 | | 0.3044 | 30.0 | 1860 | 0.6097 | 0.4 | 0.1579 | 0.3178 | 0.5394 | 0.4874 | 0.6751 | | 0.3032 | 31.0 | 1922 | 0.6028 | 0.3875 | 0.1588 | 0.2950 | 0.5643 | 0.4796 | 0.6509 | | 0.2868 | 32.0 | 1984 | 0.6083 | 0.3994 | 0.1522 | 0.3157 | 0.5436 | 0.5063 | 0.6869 | | 0.2821 | 33.0 | 2046 | 0.6093 | 0.4018 | 0.1535 | 0.3187 | 0.5436 | 0.4800 | 0.6727 | | 0.2915 | 34.0 | 2108 | 0.6044 | 0.3982 | 0.1528 | 0.3115 | 0.5519 | 0.4972 | 0.6609 | | 0.2829 | 35.0 | 2170 | 0.6112 | 0.3988 | 0.1545 | 0.3122 | 0.5519 | 0.4952 | 0.6869 | | 0.2915 | 36.0 | 2232 | 0.6111 | 0.4062 | 0.1510 | 0.3227 | 0.5477 | 0.5079 | 0.6853 | | 0.2781 | 37.0 | 2294 | 0.6153 | 0.4 | 0.1507 | 0.3208 | 0.5311 | 0.5020 | 0.6778 | | 0.2724 | 38.0 | 2356 | 0.6115 | 0.4031 | 0.1509 | 0.3203 | 0.5436 | 0.4972 | 0.6778 | | 0.2794 | 39.0 | 2418 | 0.6142 | 0.3975 | 0.1468 | 0.3176 | 0.5311 | 0.4984 | 0.6801 | | 0.2661 | 40.0 | 2480 | 0.6123 | 0.4025 | 0.1516 | 0.3195 | 0.5436 | 0.4972 | 0.6825 | ### Framework versions - Transformers 5.13.0 - Pytorch 2.12.1+cu130 - Datasets 4.8.5 - Tokenizers 0.22.2 ## Related artifacts | Artifact | Location | DOI | |----------|----------|-----| | **Released model (use this one)** | [CIRCL/vulnerability-attack-technique-classification-roberta-base](https://huggingface.co/CIRCL/vulnerability-attack-technique-classification-roberta-base) | [10.57967/hf/9623](https://doi.org/10.57967/hf/9623) | | Gold dataset (1,207 CVEs, CTID-curated labels) | [CIRCL/vulnerability-attack-techniques](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques) | [10.57967/hf/9621](https://doi.org/10.57967/hf/9621) | | LLM expansion dataset (984 LLM-labeled CVEs) | [CIRCL/vulnerability-attack-techniques-llm-scaling](https://huggingface.co/datasets/CIRCL/vulnerability-attack-techniques-llm-scaling) | [10.57967/hf/9622](https://doi.org/10.57967/hf/9622) | | Code | [vulnerability-lookup/VulnTrain](https://github.com/vulnerability-lookup/VulnTrain) | — | | Paper | [arXiv:2607.25572](https://arxiv.org/abs/2607.25572) | — | | Paper LaTeX source + trainer logs | [vulnerability-lookup/cve-attack-mapping-paper](https://github.com/vulnerability-lookup/cve-attack-mapping-paper) | — | ## Citation ```bibtex @misc{bonhomme2026mappingcvesmitreattck, title={Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion}, author={Cédric Bonhomme and Alexandre Dulaunoy}, year={2026}, eprint={2607.25572}, archivePrefix={arXiv}, primaryClass={cs.CR}, url={https://arxiv.org/abs/2607.25572}, } ``` ## Acknowledgements Developed at [CIRCL](https://www.circl.lu) in the context of the [AIPITCH](https://www.science.nask.pl/en/research-areas/projects/12456) project, co-funded by the European Union.