DATASET_CARD.md

metadata

license: apache-2.0
task_categories:
  - text-classification
language:
  - en
tags:
  - cyber-threat-intelligence
  - mitre-attack
  - multi-label-classification
  - defensive-security
  - blue-team
size_categories:
  - 10K<n<100K
annotations_creators:
  - expert-generated
source_datasets:
  - center-for-threat-informed-defense/tram
pretty_name: TRAM ATT&CK Multi-Label (cleaned, leak-free splits)
configs:
  - config_name: document
    data_files:
      - split: train
        path: data/build/document_train.jsonl
      - split: dev
        path: data/build/document_dev.jsonl
      - split: test
        path: data/build/document_test.jsonl
  - config_name: random
    data_files:
      - split: train
        path: data/build/random_train.jsonl
      - split: dev
        path: data/build/random_dev.jsonl
      - split: test
        path: data/build/random_test.jsonl

TRAM ATT&CK Multi-Label (cleaned, with leak-free splits)

Sentence-level multi-label mapping of cyber threat intelligence prose to MITRE ATT&CK technique IDs. Derived from MITRE CTID's TRAM corpus, deduplicated and republished with two split schemes so that leakage can be measured rather than assumed.

Everything here is regenerated by python scripts/01_build_dataset.py. No row was edited by hand.

Why this exists

The upstream corpus contains 19,178 sentences drawn from only 151 source reports. Threat reports repeat phrasing heavily, both within a report and across reports covering the same campaign, so the sentence-level random split that most work uses puts near-identical prose on both sides of the evaluation.

This release ships both splits and expects you to report both:

Config	Construction	Use it for
document	No source report appears in more than one split	Reporting. The honest number.
random	Naive sentence-level shuffle	Comparison against prior work that split this way

Construction

Step	Effect
Raw sentences	19,178
Strip scraped title: … url: … headers	7 rows became empty and were dropped
Deduplicate (case/punctuation-insensitive)	415 duplicate groups, 714 rows removed
of which appeared in more than one document	370
Merge labels across duplicates	12 labels recovered that a naive drop would have lost
Drop techniques in <3 documents	T1557.001 only (1 document), 7 label instances
Final	18,437 sentences · 4,024 labelled (21.8%) · 49 techniques · 151 documents

Deduplication takes the union of labels across duplicates. Two copies of one sentence annotated [T1027] and [T1027, T1140] are one sentence annotated inconsistently; keeping the union recovers the label instead of discarding it with the duplicate row. The surviving row keeps the first document it appeared in, so no sentence can span two documents.

T1557.001 is dropped because it appears in exactly one document. It cannot be placed in train and test, so it is either untrainable or unevaluable. Dropping it and saying so beats reporting F1 = 0.00 for it.

Splits

Config	Split	Sentences	Labelled
document	train	11,668	2,684
document	dev	3,134	631
document	test	3,635	709
random	train	12,905	2,837
random	dev	2,765	610
random	test	2,767	577

The document split is built by group-aware multi-label stratification in two phases. A plain greedy pass fails here: with a 70/15/15 target the train split always shows the largest absolute label deficit, so it absorbs every document containing a rare technique and the test set ends up with none. Phase 1 therefore reserves one document per technique for each split, rarest technique first; phase 2 packs the remainder using deficit normalised by split size.

Consequences, both worth knowing:

• All 49 techniques appear in all three document splits. The naive random split does not manage this. T1072 has zero dev examples, so its threshold cannot be tuned.
• Coverage seeding costs some size balance. The document split lands at roughly 63/17/20 rather than 70/15/15. Larger dev and test sets make the estimates more stable, so this was accepted rather than corrected.

Fields

{
  "sentence":  "The dropper base64-encodes its configuration before writing it to disk.",
  "labels":    ["T1027"],
  "doc_title": "NotPetya Technical Analysis  A Triple Threat ..."
}

labels is empty for 78.2% of rows. These are real negatives and should be kept. A detector that fires on every sentence of a threat report is useless to an analyst.

Known limitations

• 50 techniques, not the full ATT&CK matrix. The upstream corpus covers the 49 retained here. A technique outside that set will never be predicted.
• Heavy long tail. T1027 has 678 instances; the rarest retained techniques have roughly 20. Macro-averaged metrics are the meaningful ones.
• Single-sentence context. Labels were assigned per sentence, so techniques only inferable from surrounding paragraphs are under-represented.
• 151 documents is not many. Even the document split is one draw from a small pool; treat differences of a point or two as noise.
• Annotation is not exhaustive. Some unlabelled sentences do describe techniques. Absolute recall is therefore pessimistic.

Licence and provenance

Derived from the TRAM corpus, released by the MITRE Center for Threat-Informed Defense under Apache-2.0; this derivative keeps Apache-2.0. The underlying sentences are fragments of publicly-published vendor threat reports as redistributed by MITRE CTID.

Technique names come from MITRE ATT&CK STIX data, used under the ATT&CK Terms of Use. ATT&CK® is a registered trademark of The MITRE Corporation.

AnnoCTR was deliberately not merged in. It is CC-BY-SA 4.0, and share-alike would relicense this entire dataset and restrict downstream reuse. It remains a good candidate for a separately-licensed evaluation split.

Citation

@misc{tram_attack_multilabel_clean,
  title  = {TRAM ATT&CK Multi-Label (cleaned, leak-free splits)},
  author = {Varol Cagdas Tok},
  year   = {2026},
  url    = {https://huggingface.co/datasets/ctokx/tram-attack-multilabel-clean},
  note   = {Derived from MITRE CTID TRAM, Apache-2.0}
}

Please also cite the upstream corpus:

@misc{tram,
  title        = {TRAM: Threat Report ATT&CK Mapper},
  author       = {{MITRE Center for Threat-Informed Defense}},
  howpublished = {\url{https://github.com/center-for-threat-informed-defense/tram}}
}