--- license: apache-2.0 base_model: cisco-ai/SecureBERT2.0-base pipeline_tag: text-classification library_name: transformers language: [en] tags: [mitre-attack, cyber-threat-intelligence, cti, multi-label, security, ttp, onnx, transformers.js] model-index: - name: attack-technique-tagger results: - task: {type: text-classification, name: ATT&CK technique tagging (multi-label, 50 techniques)} dataset: {type: tram2-report-split-sealed-test, name: TRAM2 sealed test (26 held-out reports, 2920 sentences)} metrics: - {type: f1, value: 0.593, name: micro F1} - {type: f1, value: 0.537, name: macro F1} - {type: f1, value: 0.730, name: any-technique detection F1} inference: false --- # attack-technique-tagger (experimental v0.1) Tags a sentence from a threat report with the MITRE ATT&CK techniques it describes (50 techniques, multi-label). Meant as an assistant for CTI analysts and detection engineers who triage reports; **it is only moderately accurate (sealed test micro F1 0.59)**, so use its output as suggestions to review, not as ground truth. **[Try it in your browser](https://huggingface.co/spaces/Project-Bastion/attack-technique-tagger-demo)**: a static page that runs the model locally (transformers.js, ONNX). The text you paste never leaves your device. ## Usage ```python import torch from transformers import AutoTokenizer, AutoModelForSequenceClassification repo = "Project-Bastion/attack-technique-tagger" tok = AutoTokenizer.from_pretrained(repo); model = AutoModelForSequenceClassification.from_pretrained(repo).eval() sentences = ["The malware dumps credentials from LSASS memory using a Mimikatz-like routine.", "It then creates a scheduled task to maintain persistence on the host.", "Command and control traffic is sent over HTTPS to a hard-coded domain."] # sentences of ONE report, in order def tag(i, threshold=0.2): prev = sentences[i - 1] if i > 0 else ""; nxt = sentences[i + 1] if i + 1 < len(sentences) else "" ctx = prev[-300:] + " | " + nxt[:300] # neighbouring sentences are part of the input (the model was trained this way) enc = tok(sentences[i], ctx, truncation="longest_first", max_length=256, return_tensors="pt") with torch.no_grad(): p = torch.sigmoid(model(**enc).logits[0]) return [(model.config.id2label[j], round(float(p[j]), 3)) for j in torch.nonzero(p >= threshold).flatten().tolist()] for i, s in enumerate(sentences): print(s[:60], "->", tag(i)) ``` Details that matter: input is the sentence plus its previous and next sentence from the same report (adding them improved validation F1 slightly); outputs are independent sigmoids over 50 technique ids (see `config.json`; look any id up at `attack.mitre.org/techniques/`); the threshold 0.2 was chosen on validation data and trades precision for recall (raise it for fewer false positives). ## Evaluation (sealed test) Data: the TRAM2 annotated threat-report sentences (Center for Threat-Informed Defense, Apache-2.0): 19,178 sentences from 151 public reports, 4,070 of them labelled with 1 or more of 50 techniques. Split **by report** (so no report is in two splits): 105 reports train, 20 validation, 26 test (2,920 sentences, 683 with a technique). Two techniques that occur in only 1 and 5 reports were pinned to train, so test covers 48 of the 50. All model and threshold choices (about 15 training runs over 4 base models, learning rates, class weights, context and seeds) used the validation split only. The test split was scored once, on the final model. | System | Micro F1 | Macro F1 | "has any technique" F1 | |---|---|---|---| | **attack-technique-tagger (this model)** | **0.593** | **0.537** | **0.730** | | TF-IDF + logistic regression (same split) | 0.536 | 0.496 | 0.672 | | nanda-rani/TTPXHunter (MIT; best existing open model I found; its confidence threshold tuned on validation) | 0.296 | 0.243 | 0.499 | Micro precision / recall of this model on test: 0.595 / 0.590. Validation micro F1: 0.605 (three seeds: 0.597, 0.606, 0.596). TTPXHunter was trained on a different corpus (ATT&CK procedure examples), covers all 50 of these techniques among its 193 classes, and was run off the shelf; its low score reflects that domain shift as much as model quality. Averaging three seeds scored 0.612 on validation (not shipped). ## Training `cisco-ai/SecureBERT2.0-base` (Apache-2.0, ModernBERT architecture) fine-tuned with binary cross-entropy (positive-class weight exponent 0.5), lr 6e-5, 12 epochs, max length 256, bf16, one A10G; the epoch with the best validation micro F1 is kept. Code: GitHub `Project-Bastion/models`, `scripts/ttp/`. ## ONNX / browser `onnx/model.onnx` (fp32, **599 MB**) is published for transformers.js / onnxruntime. Checked against PyTorch on 200 validation sentences: onnxruntime fp32 differs by <1e-5 in probability; the in-browser code path (run in Node with transformers.js) gives identical decisions at the 0.2 threshold on all 200 sentences, with probabilities identical for 197 of them and within 0.014 on 3 long inputs (the demo works around a transformers.js quirk that drops the closing `[SEP]` when it truncates; a small residual on very long inputs remains, cause not fully isolated). **int8 quantization does not work for this model**: four dynamic-quantization recipes kept only 91-94% of decisions, so no int8 file is published. The demo therefore downloads the full 599 MB once. ## Known limits * 50 techniques only (the TRAM2 label set); sub-technique distinctions are limited to the ones in that set; anything else is invisible to the model. * Sentence-level and context-limited; technique labels in the training reports are themselves noisy and sometimes depend on the whole report. * Test set is small (26 reports); numbers will move on other report styles, languages (English only) or newer malware reports. Not evaluated on non-report text such as logs or alerts. * Moderate accuracy: expect many missed and extra techniques. Do not use it to automate decisions. ## License and credits Apache-2.0. Training data: TRAM2 data from the Center for Threat-Informed Defense TRAM project (Apache-2.0), derived from public threat reports labelled with MITRE ATT&CK techniques (ATT&CK is a trademark of The MITRE Corporation; use per the ATT&CK Terms of Use).