Text Classification
Transformers
ONNX
Safetensors
Transformers.js
English
modernbert
mitre-attack
cyber-threat-intelligence
cti
multi-label
security
ttp
Eval Results (legacy)
text-embeddings-inference
Instructions to use Project-Bastion/attack-technique-tagger with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Project-Bastion/attack-technique-tagger with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="Project-Bastion/attack-technique-tagger")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("Project-Bastion/attack-technique-tagger") model = AutoModelForSequenceClassification.from_pretrained("Project-Bastion/attack-technique-tagger", device_map="auto") - Transformers.js
How to use Project-Bastion/attack-technique-tagger with Transformers.js:
// npm i @huggingface/transformers import { pipeline } from '@huggingface/transformers'; // Allocate pipeline const pipe = await pipeline('text-classification', 'Project-Bastion/attack-technique-tagger'); - Notebooks
- Google Colab
- Kaggle
card: disable the hosted inference widget (custom outputs; use the demo Space or the snippet)
9b497dd verified |
Download README.md from Project-Bastion/attack-technique-tagger: direct link, hf CLI and curl.
- Browser
- Download file 6.33 kB
-
https://huggingface.co/Project-Bastion/attack-technique-tagger/resolve/main/README.md
- Command line
-
hf download hf://Project-Bastion/attack-technique-tagger/README.md
-
curl -L -o README.md https://huggingface.co/Project-Bastion/attack-technique-tagger/resolve/main/README.md
6.33 kB
| license: apache-2.0 | |
| base_model: cisco-ai/SecureBERT2.0-base | |
| pipeline_tag: text-classification | |
| library_name: transformers | |
| language: [en] | |
| tags: [mitre-attack, cyber-threat-intelligence, cti, multi-label, security, ttp, onnx, transformers.js] | |
| model-index: | |
| - name: attack-technique-tagger | |
| results: | |
| - task: {type: text-classification, name: ATT&CK technique tagging (multi-label, 50 techniques)} | |
| dataset: {type: tram2-report-split-sealed-test, name: TRAM2 sealed test (26 held-out reports, 2920 sentences)} | |
| metrics: | |
| - {type: f1, value: 0.593, name: micro F1} | |
| - {type: f1, value: 0.537, name: macro F1} | |
| - {type: f1, value: 0.730, name: any-technique detection F1} | |
| inference: false | |
| # attack-technique-tagger (experimental v0.1) | |
| Tags a sentence from a threat report with the MITRE ATT&CK techniques it describes (50 techniques, multi-label). | |
| Meant as an assistant for CTI analysts and detection engineers who triage reports; **it is only moderately accurate (sealed test micro F1 0.59)**, so use its output as suggestions to review, not as ground truth. | |
| **[Try it in your browser](https://huggingface.co/spaces/Project-Bastion/attack-technique-tagger-demo)**: a static page that runs the model locally (transformers.js, ONNX). The text you paste never leaves your device. | |
| ## Usage | |
| ```python | |
| import torch | |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification | |
| repo = "Project-Bastion/attack-technique-tagger" | |
| tok = AutoTokenizer.from_pretrained(repo); model = AutoModelForSequenceClassification.from_pretrained(repo).eval() | |
| sentences = ["The malware dumps credentials from LSASS memory using a Mimikatz-like routine.", | |
| "It then creates a scheduled task to maintain persistence on the host.", | |
| "Command and control traffic is sent over HTTPS to a hard-coded domain."] # sentences of ONE report, in order | |
| def tag(i, threshold=0.2): | |
| prev = sentences[i - 1] if i > 0 else ""; nxt = sentences[i + 1] if i + 1 < len(sentences) else "" | |
| ctx = prev[-300:] + " | " + nxt[:300] # neighbouring sentences are part of the input (the model was trained this way) | |
| enc = tok(sentences[i], ctx, truncation="longest_first", max_length=256, return_tensors="pt") | |
| with torch.no_grad(): p = torch.sigmoid(model(**enc).logits[0]) | |
| return [(model.config.id2label[j], round(float(p[j]), 3)) for j in torch.nonzero(p >= threshold).flatten().tolist()] | |
| for i, s in enumerate(sentences): print(s[:60], "->", tag(i)) | |
| ``` | |
| Details that matter: input is the sentence plus its previous and next sentence from the same report (adding them improved validation F1 slightly); outputs are independent sigmoids over 50 technique ids | |
| (see `config.json`; look any id up at `attack.mitre.org/techniques/<ID>`); the threshold 0.2 was chosen on validation data and trades precision for recall (raise it for fewer false positives). | |
| ## Evaluation (sealed test) | |
| Data: the TRAM2 annotated threat-report sentences (Center for Threat-Informed Defense, Apache-2.0): 19,178 sentences from 151 public reports, 4,070 of them labelled with 1 or more of 50 techniques. | |
| Split **by report** (so no report is in two splits): 105 reports train, 20 validation, 26 test (2,920 sentences, 683 with a technique). Two techniques that occur in only 1 and 5 reports were pinned to train, so test covers 48 of the 50. | |
| All model and threshold choices (about 15 training runs over 4 base models, learning rates, class weights, context and seeds) used the validation split only. The test split was scored once, on the final model. | |
| | System | Micro F1 | Macro F1 | "has any technique" F1 | | |
| |---|---|---|---| | |
| | **attack-technique-tagger (this model)** | **0.593** | **0.537** | **0.730** | | |
| | TF-IDF + logistic regression (same split) | 0.536 | 0.496 | 0.672 | | |
| | nanda-rani/TTPXHunter (MIT; best existing open model I found; its confidence threshold tuned on validation) | 0.296 | 0.243 | 0.499 | | |
| Micro precision / recall of this model on test: 0.595 / 0.590. Validation micro F1: 0.605 (three seeds: 0.597, 0.606, 0.596). TTPXHunter was trained on a different corpus (ATT&CK procedure examples), covers all 50 of these techniques among its 193 classes, and was run off the shelf; its low score reflects that domain shift as much as model quality. | |
| Averaging three seeds scored 0.612 on validation (not shipped). | |
| ## Training | |
| `cisco-ai/SecureBERT2.0-base` (Apache-2.0, ModernBERT architecture) fine-tuned with binary cross-entropy (positive-class weight exponent 0.5), lr 6e-5, 12 epochs, max length 256, bf16, one A10G; the epoch with the best validation micro F1 is kept. | |
| Code: GitHub `Project-Bastion/models`, `scripts/ttp/`. | |
| ## ONNX / browser | |
| `onnx/model.onnx` (fp32, **599 MB**) is published for transformers.js / onnxruntime. Checked against PyTorch on 200 validation sentences: onnxruntime fp32 differs by <1e-5 in probability; the in-browser code path (run in Node with transformers.js) gives identical decisions at the 0.2 threshold on all 200 sentences, with probabilities identical for 197 of them and within 0.014 on 3 long inputs (the demo works around a transformers.js quirk that drops the closing `[SEP]` when it truncates; a small residual on very long inputs remains, cause not fully isolated). | |
| **int8 quantization does not work for this model**: four dynamic-quantization recipes kept only 91-94% of decisions, so no int8 file is published. The demo therefore downloads the full 599 MB once. | |
| ## Known limits | |
| * 50 techniques only (the TRAM2 label set); sub-technique distinctions are limited to the ones in that set; anything else is invisible to the model. | |
| * Sentence-level and context-limited; technique labels in the training reports are themselves noisy and sometimes depend on the whole report. | |
| * Test set is small (26 reports); numbers will move on other report styles, languages (English only) or newer malware reports. Not evaluated on non-report text such as logs or alerts. | |
| * Moderate accuracy: expect many missed and extra techniques. Do not use it to automate decisions. | |
| ## License and credits | |
| Apache-2.0. Training data: TRAM2 data from the Center for Threat-Informed Defense TRAM project (Apache-2.0), derived from public threat reports labelled with MITRE ATT&CK techniques (ATT&CK is a trademark of The MITRE Corporation; use per the ATT&CK Terms of Use). | |