DATASET_CARD.md

---
license: apache-2.0
task_categories:
  - text-classification
language:
  - en
tags:
  - cyber-threat-intelligence
  - mitre-attack
  - multi-label-classification
  - defensive-security
  - blue-team
size_categories:
  - 10K<n<100K
annotations_creators:
  - expert-generated
source_datasets:
  - center-for-threat-informed-defense/tram
pretty_name: TRAM ATT&CK Multi-Label (cleaned, leak-free splits)
configs:
  - config_name: document
    data_files:
      - split: train
        path: data/build/document_train.jsonl
      - split: dev
        path: data/build/document_dev.jsonl
      - split: test
        path: data/build/document_test.jsonl
  - config_name: random
    data_files:
      - split: train
        path: data/build/random_train.jsonl
      - split: dev
        path: data/build/random_dev.jsonl
      - split: test
        path: data/build/random_test.jsonl
---

# TRAM ATT&CK Multi-Label (cleaned, with leak-free splits)

Sentence-level multi-label mapping of cyber threat intelligence prose to MITRE
ATT&CK technique IDs. Derived from MITRE CTID's
[TRAM](https://github.com/center-for-threat-informed-defense/tram) corpus,
deduplicated and republished with **two split schemes so that leakage can be
measured rather than assumed**.

Everything here is regenerated by `python scripts/01_build_dataset.py`. No row
was edited by hand.

## Why this exists

The upstream corpus contains 19,178 sentences drawn from only **151 source
reports**. Threat reports repeat phrasing heavily, both within a report and
across reports covering the same campaign, so the sentence-level random split
that most work uses puts near-identical prose on both sides of the evaluation.

This release ships both splits and expects you to report both:

| Config | Construction | Use it for |
|---|---|---|
| `document` | No source report appears in more than one split | **Reporting.** The honest number. |
| `random` | Naive sentence-level shuffle | Comparison against prior work that split this way |

## Construction

| Step | Effect |
|---|---|
| Raw sentences | 19,178 |
| Strip scraped `title: … url: …` headers | 7 rows became empty and were dropped |
| Deduplicate (case/punctuation-insensitive) | 415 duplicate groups, **714 rows removed** |
| of which appeared in **more than one document** | **370** |
| Merge labels across duplicates | 12 labels recovered that a naive drop would have lost |
| Drop techniques in `<3` documents | `T1557.001` only (1 document), 7 label instances |
| **Final** | **18,437 sentences · 4,024 labelled (21.8%) · 49 techniques · 151 documents** |

Deduplication takes the **union** of labels across duplicates. Two copies of one
sentence annotated `[T1027]` and `[T1027, T1140]` are one sentence annotated
inconsistently; keeping the union recovers the label instead of discarding it
with the duplicate row. The surviving row keeps the first document it appeared
in, so no sentence can span two documents.

`T1557.001` is dropped because it appears in exactly one document. It cannot be
placed in train *and* test, so it is either untrainable or unevaluable. Dropping
it and saying so beats reporting F1 = 0.00 for it.

## Splits

| Config | Split | Sentences | Labelled |
|---|---|---|---|
| `document` | train | 11,668 | 2,684 |
| `document` | dev | 3,134 | 631 |
| `document` | test | 3,635 | 709 |
| `random` | train | 12,905 | 2,837 |
| `random` | dev | 2,765 | 610 |
| `random` | test | 2,767 | 577 |

The `document` split is built by group-aware multi-label stratification in two
phases. A plain greedy pass fails here: with a 70/15/15 target the train split
always shows the largest absolute label deficit, so it absorbs every document
containing a rare technique and the test set ends up with none. Phase 1
therefore reserves one document per technique for each split, rarest technique
first; phase 2 packs the remainder using deficit normalised by split size.

Consequences, both worth knowing:

- **All 49 techniques appear in all three `document` splits.** The naive
  `random` split does not manage this. `T1072` has zero dev examples, so its
  threshold cannot be tuned.
- Coverage seeding costs some size balance. The `document` split lands at
  roughly 63/17/20 rather than 70/15/15. Larger dev and test sets make the
  estimates *more* stable, so this was accepted rather than corrected.

## Fields

```json
{
  "sentence":  "The dropper base64-encodes its configuration before writing it to disk.",
  "labels":    ["T1027"],
  "doc_title": "NotPetya Technical Analysis  A Triple Threat ..."
}
```

`labels` is empty for 78.2% of rows. These are **real negatives** and should be
kept. A detector that fires on every sentence of a threat report is useless to
an analyst.

## Known limitations

- **50 techniques, not the full ATT&CK matrix.** The upstream corpus covers the
  49 retained here. A technique outside that set will never be predicted.
- **Heavy long tail.** `T1027` has 678 instances; the rarest retained techniques
  have roughly 20. Macro-averaged metrics are the meaningful ones.
- **Single-sentence context.** Labels were assigned per sentence, so techniques
  only inferable from surrounding paragraphs are under-represented.
- **151 documents is not many.** Even the document split is one draw from a
  small pool; treat differences of a point or two as noise.
- **Annotation is not exhaustive.** Some unlabelled sentences do describe
  techniques. Absolute recall is therefore pessimistic.

## Licence and provenance

Derived from the TRAM corpus, released by the MITRE Center for Threat-Informed
Defense under **Apache-2.0**; this derivative keeps Apache-2.0. The underlying
sentences are fragments of publicly-published vendor threat reports as
redistributed by MITRE CTID.

Technique names come from
[MITRE ATT&CK STIX data](https://github.com/mitre-attack/attack-stix-data),
used under the ATT&CK Terms of Use. ATT&CK® is a registered trademark of The
MITRE Corporation.

[AnnoCTR](https://github.com/boschresearch/anno-ctr-lrec-coling-2024) was
deliberately **not** merged in. It is CC-BY-SA 4.0, and share-alike would
relicense this entire dataset and restrict downstream reuse. It remains a good
candidate for a separately-licensed evaluation split.

## Citation

```bibtex
@misc{tram_attack_multilabel_clean,
  title  = {TRAM ATT&CK Multi-Label (cleaned, leak-free splits)},
  author = {Varol Cagdas Tok},
  year   = {2026},
  url    = {https://huggingface.co/datasets/ctokx/tram-attack-multilabel-clean},
  note   = {Derived from MITRE CTID TRAM, Apache-2.0}
}
```

Please also cite the upstream corpus:

```bibtex
@misc{tram,
  title        = {TRAM: Threat Report ATT&CK Mapper},
  author       = {{MITRE Center for Threat-Informed Defense}},
  howpublished = {\url{https://github.com/center-for-threat-informed-defense/tram}}
}
```