Text Classification
Transformers
ONNX
Safetensors
Transformers.js
English
modernbert
mitre-attack
cyber-threat-intelligence
cti
multi-label
security
ttp
Eval Results (legacy)
text-embeddings-inference
Instructions to use Project-Bastion/attack-technique-tagger with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Project-Bastion/attack-technique-tagger with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="Project-Bastion/attack-technique-tagger")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("Project-Bastion/attack-technique-tagger") model = AutoModelForSequenceClassification.from_pretrained("Project-Bastion/attack-technique-tagger", device_map="auto") - Transformers.js
How to use Project-Bastion/attack-technique-tagger with Transformers.js:
// npm i @huggingface/transformers import { pipeline } from '@huggingface/transformers'; // Allocate pipeline const pipe = await pipeline('text-classification', 'Project-Bastion/attack-technique-tagger'); - Notebooks
- Google Colab
- Kaggle
File size: 6,328 Bytes
a8c31ba 868f41e 9b497dd a8c31ba 3a87004 a8c31ba | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 | ---
license: apache-2.0
base_model: cisco-ai/SecureBERT2.0-base
pipeline_tag: text-classification
library_name: transformers
language: [en]
tags: [mitre-attack, cyber-threat-intelligence, cti, multi-label, security, ttp, onnx, transformers.js]
model-index:
- name: attack-technique-tagger
results:
- task: {type: text-classification, name: ATT&CK technique tagging (multi-label, 50 techniques)}
dataset: {type: tram2-report-split-sealed-test, name: TRAM2 sealed test (26 held-out reports, 2920 sentences)}
metrics:
- {type: f1, value: 0.593, name: micro F1}
- {type: f1, value: 0.537, name: macro F1}
- {type: f1, value: 0.730, name: any-technique detection F1}
inference: false
---
# attack-technique-tagger (experimental v0.1)
Tags a sentence from a threat report with the MITRE ATT&CK techniques it describes (50 techniques, multi-label).
Meant as an assistant for CTI analysts and detection engineers who triage reports; **it is only moderately accurate (sealed test micro F1 0.59)**, so use its output as suggestions to review, not as ground truth.
**[Try it in your browser](https://huggingface.co/spaces/Project-Bastion/attack-technique-tagger-demo)**: a static page that runs the model locally (transformers.js, ONNX). The text you paste never leaves your device.
## Usage
```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
repo = "Project-Bastion/attack-technique-tagger"
tok = AutoTokenizer.from_pretrained(repo); model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
sentences = ["The malware dumps credentials from LSASS memory using a Mimikatz-like routine.",
"It then creates a scheduled task to maintain persistence on the host.",
"Command and control traffic is sent over HTTPS to a hard-coded domain."] # sentences of ONE report, in order
def tag(i, threshold=0.2):
prev = sentences[i - 1] if i > 0 else ""; nxt = sentences[i + 1] if i + 1 < len(sentences) else ""
ctx = prev[-300:] + " | " + nxt[:300] # neighbouring sentences are part of the input (the model was trained this way)
enc = tok(sentences[i], ctx, truncation="longest_first", max_length=256, return_tensors="pt")
with torch.no_grad(): p = torch.sigmoid(model(**enc).logits[0])
return [(model.config.id2label[j], round(float(p[j]), 3)) for j in torch.nonzero(p >= threshold).flatten().tolist()]
for i, s in enumerate(sentences): print(s[:60], "->", tag(i))
```
Details that matter: input is the sentence plus its previous and next sentence from the same report (adding them improved validation F1 slightly); outputs are independent sigmoids over 50 technique ids
(see `config.json`; look any id up at `attack.mitre.org/techniques/<ID>`); the threshold 0.2 was chosen on validation data and trades precision for recall (raise it for fewer false positives).
## Evaluation (sealed test)
Data: the TRAM2 annotated threat-report sentences (Center for Threat-Informed Defense, Apache-2.0): 19,178 sentences from 151 public reports, 4,070 of them labelled with 1 or more of 50 techniques.
Split **by report** (so no report is in two splits): 105 reports train, 20 validation, 26 test (2,920 sentences, 683 with a technique). Two techniques that occur in only 1 and 5 reports were pinned to train, so test covers 48 of the 50.
All model and threshold choices (about 15 training runs over 4 base models, learning rates, class weights, context and seeds) used the validation split only. The test split was scored once, on the final model.
| System | Micro F1 | Macro F1 | "has any technique" F1 |
|---|---|---|---|
| **attack-technique-tagger (this model)** | **0.593** | **0.537** | **0.730** |
| TF-IDF + logistic regression (same split) | 0.536 | 0.496 | 0.672 |
| nanda-rani/TTPXHunter (MIT; best existing open model I found; its confidence threshold tuned on validation) | 0.296 | 0.243 | 0.499 |
Micro precision / recall of this model on test: 0.595 / 0.590. Validation micro F1: 0.605 (three seeds: 0.597, 0.606, 0.596). TTPXHunter was trained on a different corpus (ATT&CK procedure examples), covers all 50 of these techniques among its 193 classes, and was run off the shelf; its low score reflects that domain shift as much as model quality.
Averaging three seeds scored 0.612 on validation (not shipped).
## Training
`cisco-ai/SecureBERT2.0-base` (Apache-2.0, ModernBERT architecture) fine-tuned with binary cross-entropy (positive-class weight exponent 0.5), lr 6e-5, 12 epochs, max length 256, bf16, one A10G; the epoch with the best validation micro F1 is kept.
Code: GitHub `Project-Bastion/models`, `scripts/ttp/`.
## ONNX / browser
`onnx/model.onnx` (fp32, **599 MB**) is published for transformers.js / onnxruntime. Checked against PyTorch on 200 validation sentences: onnxruntime fp32 differs by <1e-5 in probability; the in-browser code path (run in Node with transformers.js) gives identical decisions at the 0.2 threshold on all 200 sentences, with probabilities identical for 197 of them and within 0.014 on 3 long inputs (the demo works around a transformers.js quirk that drops the closing `[SEP]` when it truncates; a small residual on very long inputs remains, cause not fully isolated).
**int8 quantization does not work for this model**: four dynamic-quantization recipes kept only 91-94% of decisions, so no int8 file is published. The demo therefore downloads the full 599 MB once.
## Known limits
* 50 techniques only (the TRAM2 label set); sub-technique distinctions are limited to the ones in that set; anything else is invisible to the model.
* Sentence-level and context-limited; technique labels in the training reports are themselves noisy and sometimes depend on the whole report.
* Test set is small (26 reports); numbers will move on other report styles, languages (English only) or newer malware reports. Not evaluated on non-report text such as logs or alerts.
* Moderate accuracy: expect many missed and extra techniques. Do not use it to automate decisions.
## License and credits
Apache-2.0. Training data: TRAM2 data from the Center for Threat-Informed Defense TRAM project (Apache-2.0), derived from public threat reports labelled with MITRE ATT&CK techniques (ATT&CK is a trademark of The MITRE Corporation; use per the ATT&CK Terms of Use).
|