File size: 6,328 Bytes
a8c31ba
 
 
 
 
 
 
868f41e
 
 
 
 
 
 
 
 
9b497dd
a8c31ba
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3a87004
a8c31ba
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
---
license: apache-2.0
base_model: cisco-ai/SecureBERT2.0-base
pipeline_tag: text-classification
library_name: transformers
language: [en]
tags: [mitre-attack, cyber-threat-intelligence, cti, multi-label, security, ttp, onnx, transformers.js]
model-index:
- name: attack-technique-tagger
  results:
  - task: {type: text-classification, name: ATT&CK technique tagging (multi-label, 50 techniques)}
    dataset: {type: tram2-report-split-sealed-test, name: TRAM2 sealed test (26 held-out reports, 2920 sentences)}
    metrics:
    - {type: f1, value: 0.593, name: micro F1}
    - {type: f1, value: 0.537, name: macro F1}
    - {type: f1, value: 0.730, name: any-technique detection F1}
inference: false
---

# attack-technique-tagger (experimental v0.1)

Tags a sentence from a threat report with the MITRE ATT&CK techniques it describes (50 techniques, multi-label).
Meant as an assistant for CTI analysts and detection engineers who triage reports; **it is only moderately accurate (sealed test micro F1 0.59)**, so use its output as suggestions to review, not as ground truth.

**[Try it in your browser](https://huggingface.co/spaces/Project-Bastion/attack-technique-tagger-demo)**: a static page that runs the model locally (transformers.js, ONNX). The text you paste never leaves your device.

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
repo = "Project-Bastion/attack-technique-tagger"
tok = AutoTokenizer.from_pretrained(repo); model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

sentences = ["The malware dumps credentials from LSASS memory using a Mimikatz-like routine.",
             "It then creates a scheduled task to maintain persistence on the host.",
             "Command and control traffic is sent over HTTPS to a hard-coded domain."]  # sentences of ONE report, in order

def tag(i, threshold=0.2):
    prev = sentences[i - 1] if i > 0 else ""; nxt = sentences[i + 1] if i + 1 < len(sentences) else ""
    ctx = prev[-300:] + " | " + nxt[:300]          # neighbouring sentences are part of the input (the model was trained this way)
    enc = tok(sentences[i], ctx, truncation="longest_first", max_length=256, return_tensors="pt")
    with torch.no_grad(): p = torch.sigmoid(model(**enc).logits[0])
    return [(model.config.id2label[j], round(float(p[j]), 3)) for j in torch.nonzero(p >= threshold).flatten().tolist()]

for i, s in enumerate(sentences): print(s[:60], "->", tag(i))
```

Details that matter: input is the sentence plus its previous and next sentence from the same report (adding them improved validation F1 slightly); outputs are independent sigmoids over 50 technique ids
(see `config.json`; look any id up at `attack.mitre.org/techniques/<ID>`); the threshold 0.2 was chosen on validation data and trades precision for recall (raise it for fewer false positives).

## Evaluation (sealed test)

Data: the TRAM2 annotated threat-report sentences (Center for Threat-Informed Defense, Apache-2.0): 19,178 sentences from 151 public reports, 4,070 of them labelled with 1 or more of 50 techniques.
Split **by report** (so no report is in two splits): 105 reports train, 20 validation, 26 test (2,920 sentences, 683 with a technique). Two techniques that occur in only 1 and 5 reports were pinned to train, so test covers 48 of the 50.
All model and threshold choices (about 15 training runs over 4 base models, learning rates, class weights, context and seeds) used the validation split only. The test split was scored once, on the final model.

| System | Micro F1 | Macro F1 | "has any technique" F1 |
|---|---|---|---|
| **attack-technique-tagger (this model)** | **0.593** | **0.537** | **0.730** |
| TF-IDF + logistic regression (same split) | 0.536 | 0.496 | 0.672 |
| nanda-rani/TTPXHunter (MIT; best existing open model I found; its confidence threshold tuned on validation) | 0.296 | 0.243 | 0.499 |

Micro precision / recall of this model on test: 0.595 / 0.590. Validation micro F1: 0.605 (three seeds: 0.597, 0.606, 0.596). TTPXHunter was trained on a different corpus (ATT&CK procedure examples), covers all 50 of these techniques among its 193 classes, and was run off the shelf; its low score reflects that domain shift as much as model quality.
Averaging three seeds scored 0.612 on validation (not shipped).

## Training

`cisco-ai/SecureBERT2.0-base` (Apache-2.0, ModernBERT architecture) fine-tuned with binary cross-entropy (positive-class weight exponent 0.5), lr 6e-5, 12 epochs, max length 256, bf16, one A10G; the epoch with the best validation micro F1 is kept.
Code: GitHub `Project-Bastion/models`, `scripts/ttp/`.

## ONNX / browser

`onnx/model.onnx` (fp32, **599 MB**) is published for transformers.js / onnxruntime. Checked against PyTorch on 200 validation sentences: onnxruntime fp32 differs by <1e-5 in probability; the in-browser code path (run in Node with transformers.js) gives identical decisions at the 0.2 threshold on all 200 sentences, with probabilities identical for 197 of them and within 0.014 on 3 long inputs (the demo works around a transformers.js quirk that drops the closing `[SEP]` when it truncates; a small residual on very long inputs remains, cause not fully isolated).
**int8 quantization does not work for this model**: four dynamic-quantization recipes kept only 91-94% of decisions, so no int8 file is published. The demo therefore downloads the full 599 MB once.

## Known limits

* 50 techniques only (the TRAM2 label set); sub-technique distinctions are limited to the ones in that set; anything else is invisible to the model.
* Sentence-level and context-limited; technique labels in the training reports are themselves noisy and sometimes depend on the whole report.
* Test set is small (26 reports); numbers will move on other report styles, languages (English only) or newer malware reports. Not evaluated on non-report text such as logs or alerts.
* Moderate accuracy: expect many missed and extra techniques. Do not use it to automate decisions.

## License and credits

Apache-2.0. Training data: TRAM2 data from the Center for Threat-Informed Defense TRAM project (Apache-2.0), derived from public threat reports labelled with MITRE ATT&CK techniques (ATT&CK is a trademark of The MITRE Corporation; use per the ATT&CK Terms of Use).