README.md

---

license: apache-2.0
language:
  - en
base_model: answerdotai/ModernBERT-base
pipeline_tag: text-classification
tags:
  - cyber-threat-intelligence
  - mitre-attack
  - multi-label-classification
  - defensive-security
  - blue-team
  - threat-intelligence
datasets:
  - center-for-threat-informed-defense/tram
metrics:
  - f1
library_name: transformers
---


# cti-attack-mapper

Maps sentences from cyber threat intelligence reports to MITRE ATT&CK technique
IDs. 49 techniques, multi-label, sentence level.

> **Read this before you trust the numbers.** On the leak-free split, a
> fine-tuned ModernBERT scores 0.445 macro-F1. A plain TF-IDF baseline scores
> 0.454. The transformer does not beat it; they are tied inside the noise of a
> 151-report corpus. What does beat both is a blend of the two: 0.474 macro-F1
> (5-fold mean, standard deviation 0.022), about 0.04 ahead of each base model
> at the same threshold setting, and ahead in all five folds. One caveat:
> against TF-IDF at its own best threshold setting the blend's lead shrinks to
> 0.02 and is no longer outside the noise. Every number here is written by a
> script in the repo. See [Results](#results).

**Defensive use only.** It labels adversary behavior that has already been
written up in public threat reports. It produces no offensive capability.

---

## What this repo is for

There are already several ATT&CK classifiers on the Hub. This one exists for
three reasons those usually leave out.

1. **The baselines are published.** Most cyber-ML papers never say what a TF-IDF
   plus logistic regression floor scores. This one does. On the honest split
   that floor ties the fine-tuned transformer, and only a blend of the two gets
   past it. The repo shows by how much and with what spread.
2. **The test split is leak-free, and the naive alternative is measured.** The
   corpus is 19k sentences drawn from only 151 reports. The sentence-level
   random split that most work uses scatters near-identical prose from one
   report across train and test. That inflates macro-F1 by about 12 percent.
   Both splits ship here.
3. **Everything regenerates.** Every number below is written by
   `scripts/04_report.py` from JSON that the training and eval scripts produce.
   None of it is typed by hand.

## Quick start

```python

from transformers import AutoModelForSequenceClassification, AutoTokenizer

import torch



model_id = "ctokx/cti-attack-mapper-modernbert"

tok = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()



text = "The implant establishes persistence by creating a scheduled task that runs at logon."

with torch.no_grad():

    probs = torch.sigmoid(model(**tok(text, return_tensors="pt", truncation=True)).logits)[0]



for i, p in enumerate(probs):

    if p > 0.5:

        print(model.config.id2label[i], round(float(p), 3))

# T1053.005 0.94

```

Or use the repo's wrapper, which loads the tuned per-class thresholds and
resolves technique names:

```python

from cti_attack.predict import AttackMapper



mapper = AttackMapper("models/modernbert__document")

mapper.predict("The dropper base64-encodes its configuration before writing it to disk.")

# [Prediction(T1027, 'Obfuscated Files or Information', 0.912)]

```

## Results

<!-- BEGIN GENERATED RESULTS - scripts/04_report.py -->

### Test-set results



Per-class thresholds tuned on dev. Macro-F1 is the headline metric. It weights all 49 techniques equally, so a handful of common ones cannot cover for a weak long tail.



| Model | doc macro-F1 | doc micro-F1 | random macro-F1 | random micro-F1 | inflation |

|---|---|---|---|---|---|

| Frequency prior | 0.000 | 0.000 | 0.000 | 0.000 | n/a |

| ATT&CK keyword match | 0.167 | 0.170 | 0.171 | 0.153 | +2.4% |

| TF-IDF + one-vs-rest LR | 0.454 | 0.500 | 0.506 | 0.545 | +11.5% |

| **ModernBERT-base (fine-tuned)** | 0.445 | 0.478 | 0.482 | 0.540 | +8.3% |

| SecureBERT (fine-tuned) | 0.369 | 0.252 | 0.415 | 0.373 | +12.5% |



### Same table, single global threshold



| Model | doc macro-F1 | doc micro-F1 | random macro-F1 | random micro-F1 | inflation |

|---|---|---|---|---|---|

| Frequency prior | 0.000 | 0.000 | 0.000 | 0.000 | n/a |

| ATT&CK keyword match | 0.167 | 0.170 | 0.171 | 0.153 | +2.4% |

| TF-IDF + one-vs-rest LR | 0.449 | 0.472 | 0.515 | 0.552 | +14.7% |

| **ModernBERT-base (fine-tuned)** | 0.425 | 0.494 | 0.504 | 0.557 | +18.6% |

| SecureBERT (fine-tuned) | 0.352 | 0.425 | 0.388 | 0.470 | +10.2% |



### Ensemble + cross-validated results



One 70/15/15 split of a 151-document corpus is one draw, so the gap between models can be noise. To check it, here is 5-fold cross-validation grouped by document. Each report lands in exactly one test fold. The blend weight and thresholds are tuned on each fold's own dev set, never on its test set.



| Model | doc macro-F1 (5-fold CV, mean +/- std) |

|---|---|

| TF-IDF + LR (per-class) | 0.4326 +/- 0.0349 |

| TF-IDF + LR (global) | 0.4521 +/- 0.0420 |

| ModernBERT (per-class) | 0.4263 +/- 0.0256 |

| **Ensemble TF-IDF+ModernBERT (per-class)** | **0.4738 +/- 0.0223** |



Paired per-fold test, per-class thresholds on both sides:

- ensemble vs TF-IDF, same per-class regime: **+0.0412**, 5/5 folds positive, p = 0.005

- ensemble vs ModernBERT, same per-class regime: **+0.0475**, 5/5 folds positive, p = 0.010

- ensemble vs TF-IDF at its stronger global regime: **+0.0217**, 4/5 folds positive, p = 0.100



On the original single 70/15/15 document split, the same ensemble (70% TF-IDF, 30% ModernBERT) scores **0.4965** macro-F1, against 0.4536 for TF-IDF and 0.4454 for ModernBERT.



How to read this: the ensemble has the highest mean and the lowest variance of everything tested. Compared against each base model at the same threshold setting, it wins in all five folds. The one comparison it does not clearly win is against TF-IDF at its own best threshold setting, where the lead is small enough to be noise over five folds. That row is in the table on purpose.



### ModernBERT per-technique, document split (12 most frequent)



| Technique | Name | Support | P | R | F1 |

|---|---|---|---|---|---|

| `T1027` | Obfuscated Files or Information | 102 | 0.57 | 0.54 | 0.55 |

| `T1140` | Deobfuscate/Decode Files or Information | 68 | 0.73 | 0.84 | 0.78 |

| `T1105` | Ingress Tool Transfer | 57 | 0.51 | 0.44 | 0.47 |

| `T1059.003` | Command and Scripting Interpreter: Windows … | 51 | 0.39 | 0.59 | 0.47 |

| `T1055` | Process Injection | 40 | 0.68 | 0.53 | 0.59 |

| `T1106` | Native API | 34 | 0.64 | 0.47 | 0.54 |

| `T1047` | Windows Management Instrumentation | 28 | 0.69 | 0.64 | 0.67 |

| `T1053.005` | Scheduled Task/Job: Scheduled Task | 26 | 0.71 | 0.92 | 0.80 |

| `T1562.001` | Impair Defenses: Disable or Modify Tools ⚠️*revoked* | 26 | 0.92 | 0.46 | 0.62 |

| `T1574.002` | Hijack Execution Flow: DLL Side-Loading ⚠️*revoked* | 26 | 0.83 | 0.38 | 0.53 |

| `T1082` | System Information Discovery | 24 | 0.83 | 0.21 | 0.33 |

| `T1078` | Valid Accounts | 23 | 0.23 | 0.48 | 0.31 |



### Head vs tail, document split



| Bucket | Techniques | Mean F1 |

|---|---|---|

| head (>=20 test examples) | 15 | 0.570 |

| tail (<20 test examples) | 34 | 0.391 |



### Corpus



- 19,178 raw sentences, 18,437 after cleaning and dedup

- 4,024 carry at least one technique (21.8%)

- 49 techniques across 151 source documents

- 714 duplicate sentences removed (370 of them appeared in more than one document)

- dropped for having too few documents to split: T1557.001

<!-- END GENERATED RESULTS -->



### How to read this



- **No single model beats the linear baseline; a blend of two does.** On the

  leak-free split TF-IDF scores 0.454 and ModernBERT 0.445. The 0.009 gap is

  inside the noise you get from a 151-document corpus, so they are tied, not

  that TF-IDF wins. Blending their probabilities (about 70 percent TF-IDF)

  reaches 0.474 in 5-fold cross-validation and beats each base model at the same

  threshold setting in all five folds. The one place it does not clearly win is

  against TF-IDF at its own best threshold setting, where the 0.02 lead is

  within noise. The blend is also the steadiest model tested, with the smallest

  fold-to-fold spread.

- **The `random` column is the number you would have reported by accident.** It

  runs about 12 percent higher for the same models on the same data. If you are

  comparing against work that split at sentence level, that column is the

  comparable one, and it is not the true one.

- **The keyword row is the control.** It never trains, so leaked phrasing cannot

  help it. It moves 2.4 percent between splits, which is the size of plain

  test-set-composition noise. The trained models move 8 to 19 percent. That

  difference is what makes the leakage claim a measurement, not an assertion.

- **One claim this data does not support:** that leakage inflates high-capacity

  models more than linear ones. Under per-class thresholds TF-IDF inflates more

  (11.5 percent versus 8.3 percent); under a global threshold the order flips

  (14.7 percent versus 18.6 percent). A result that changes sign with the

  threshold setting is not a result, and it is not claimed here.

- **The tail is where this model is weak.** 0.570 mean F1 on the 15 techniques

  with at least 20 test examples, 0.391 on the 34 rarer ones. Micro-F1 hides

  this, which is why macro-F1 leads.



## Intended use



**In scope.** A triage aid for threat-intelligence and detection-engineering

work:



- First-pass ATT&CK tagging of a report, for an analyst to correct

- Deciding which reports to read first when clearing a backlog

- Rough coverage analysis: which techniques a body of reporting talks about

- A baseline to beat, with a published harness for beating it



**Out of scope.**



- Unreviewed labeling. Output is candidates, not conclusions. At macro-F1 around

  0.45, roughly half the predictions on rarer techniques are wrong.

- Compliance, audit, or attestation evidence.

- Detecting techniques outside the 49 covered.

- Anything but English prose describing adversary behavior. It is not a malware

  classifier; it does not read binaries, logs, or code.



## Limitations



- **49 techniques, not the full ATT&CK matrix.** Anything outside the label set

  is invisible, including techniques the text plainly describes. No prediction is

  not evidence of absence.

- **2 of the 49 labels are revoked in current ATT&CK.** `T1562.001` and

  `T1574.002` were valid when the TRAM corpus was annotated and MITRE has since

  revoked them. Map them forward before comparing output to current ATT&CK.

- **Sentence-level context only.** Techniques you can only infer from

  surrounding paragraphs are under-represented.

- **Long-tailed.** `T1027` has 678 training instances; the rarest retained

  techniques have about 20.

- **151 source documents.** Even the leak-free split is one draw from a small

  pool. Treat a point or two between models as noise, including the

  ModernBERT/TF-IDF gap above.

- **Annotation is not exhaustive.** Some unlabelled sentences do describe

  techniques, so measured recall is lower than the truth.

- **Domain shift is untested.** The training text is vendor threat-report prose.

  Behavior on incident tickets, chat logs, or non-native-English reporting is

  unknown and probably worse.

- **Per-class thresholds are fitted.** 49 thresholds tuned on a dev set with few

  positives per class can overfit, so the single-threshold numbers are reported

  next to them and you can see the size of the effect.



## Training details



| | |

|---|---|

| Base model | `answerdotai/ModernBERT-base` (149M) |

| Objective | Multi-label BCE, per-class `pos_weight`, capped at 50 |
| Max sequence length | 256 tokens |
| Batch size | 16, gradient accumulation 2 (effective 32) |
| Learning rate | 3e-5, linear schedule, 10% warmup |
| Weight decay | 0.01 (excluding bias and norm parameters) |
| Epochs | 6, best checkpoint by dev macro-F1 (epoch 5) |
| Precision | bf16 autocast |
| Hardware | 1x RTX 4060 Laptop, 8 GB |
| Peak VRAM | ~5.5 GB |
| Wall clock | ~30 min per split scheme |
| Seed | 20260802 |

The `pos_weight` is doing real work. 78 percent of sentences carry no label, and
without it the model learns to predict nothing.

Training loss reached 0.010 while dev macro-F1 peaked at 0.414. The model
memorised the training set instead of generalising from it. With 2,684 labelled
training sentences across 49 techniques, that is what you would expect, and it is
the likely reason a single transformer does not pull ahead of the linear
baseline. A blend of the two does, which points at data, not architecture, as
the limit.

## Dataset

See [DATASET_CARD.md](DATASET_CARD.md). In short:

- 19,178 raw sentences, 18,437 after cleaning and dedup
- 4,024 carry at least one technique (21.8%)
- 49 techniques across 151 source documents
- 714 duplicate sentences removed. 370 of them appeared in more than one
  document, which is the leakage the `random` split exposes.
- `T1557.001` dropped: it appears in exactly one document and cannot be split
  leak-free

## Reproducing

```bash

pip install -r requirements.txt

python scripts/reproduce_all.py              # ModernBERT only, ~40 min on an RTX 4060

python scripts/reproduce_all.py --all-models # adds DeBERTa-v3 and SecureBERT

```

Individual steps:

```bash

python scripts/01_build_dataset.py                              # build both splits

python scripts/02_run_baselines.py                              # frequency, keyword, TF-IDF

python scripts/03_train.py --model modernbert --scheme document # fine-tune

python scripts/05_ensemble.py --scheme document                 # blend TF-IDF + ModernBERT

python scripts/06_cv.py --folds 5                               # 5-fold document CV + error bars

python scripts/04_report.py                                     # regenerate the tables above

pytest tests/ -q                                                # invariants + smoke test

```

The published weights are the ModernBERT model. The blend is those weights plus
a TF-IDF and logistic regression model, which `scripts/05_ensemble.py` refits
from the included dataset in a few seconds, so the best-scoring setup reproduces
without shipping a second binary. The blend weight and every threshold are tuned
on dev, never on test.

The test suite checks the claims this card makes: that no document spans two
splits, that all 49 techniques reach every split, that no duplicate sentences
survive, and that the model loads and returns well-formed output.

## Licence and attribution

Apache-2.0. Derived from
[MITRE CTID TRAM](https://github.com/center-for-threat-informed-defense/tram)
(Apache-2.0). Technique names come from
[MITRE ATT&CK STIX data](https://github.com/mitre-attack/attack-stix-data)
under the ATT&CK Terms of Use.

ATT&CK® is a registered trademark of The MITRE Corporation. This project is not
affiliated with, endorsed by, or sponsored by The MITRE Corporation.

See [NOTICE](NOTICE) for full attribution.

## Citation

```bibtex

@misc{cti_attack_mapper,

  title  = {cti-attack-mapper: sentence-level MITRE ATT&CK classification

            with leak-free evaluation},

  author = {Varol Cagdas Tok},

  year   = {2026},

  url    = {https://huggingface.co/ctokx/cti-attack-mapper-modernbert},

  note   = {Derived from MITRE CTID TRAM, Apache-2.0}

}

```