README.md

---
license: other
license_name: lfm1.0
license_link: LICENSE
base_model: LiquidAI/d1-3B
base_model_relation: finetune
datasets:
- yannelli/cli-command-risk
language:
- en
library_name: transformers
pipeline_tag: text-classification
tags:
- decision-model
- typed-decisions
- system-one
- shell
- guardrail
- command-risk
- lfm2
- liquid
- lora
model-index:
- name: d1-3b-cli-command-risk
  results:
  - task:
      type: text-classification
      name: Shell command risk decisions
    dataset:
      name: CLI Command Risk
      type: yannelli/cli-command-risk
      split: test
    metrics:
    - type: f1
      name: Macro F1 @ 0.5 (13 risk categories)
      value: 0.929
    - type: f1
      name: Risk level macro F1
      value: 0.669
    - type: accuracy
      name: Accuracy (all decisions)
      value: 0.963
    - type: log_loss
      name: Log loss
      value: 0.136
    - type: brier_score
      name: Brier score
      value: 0.042
    - type: ece
      name: Expected calibration error
      value: 0.030
    source:
      name: training/score_cli_decisions.py
      url: https://huggingface.co/yannelli/d1-3b-cli-command-risk/blob/main/training/score_cli_decisions.py
---

> A newer version is available: [d1-3B CLI Command Risk v1.5](https://huggingface.co/yannelli/d1-3b-cli-command-risk-v1.5), trained on the dataset revision tagged [`v1.5`](https://huggingface.co/datasets/yannelli/cli-command-risk/tree/v1.5). This v1 model was trained on the revision tagged [`v1`](https://huggingface.co/datasets/yannelli/cli-command-risk/tree/v1).

![d1-3B CLI Command Risk: LiquidAI’s d1-3B, fine-tuned to answer 17 typed questions about a shell command before it runs.](https://huggingface.co/yannelli/d1-3b-cli-command-risk/resolve/main/assets/d1-3b-cli-command-risk-banner.png)

# d1-3B CLI Command Risk

A decision model that rates the risk of a shell command before it runs. Give it a command and its shell, and it answers 17 typed questions with probabilities: the risk level from `safe` to `critical`, whether the command is risky, which of 13 risk categories apply, whether the effect can be undone without backups, and whether the harm needs root. It reads the command once and answers all 17 questions as branches of one forward pass, without generating text.

It is [LiquidAI/d1-3B](https://huggingface.co/LiquidAI/d1-3B) fine-tuned on [CLI Command Risk](https://huggingface.co/datasets/yannelli/cli-command-risk). On that dataset's test split it scores macro F1 0.929 over the risk categories and accuracy 0.963 over 215,713 decisions. At a cutoff that flags 1% of safe validation commands, it flags 99.9% of `high` and `critical` test commands.

| | |
|---|---|
| Base model | [LiquidAI/d1-3B](https://huggingface.co/LiquidAI/d1-3B), LFM2.5-VL-3B with Liquid AI's d1 decision readout |
| Parameters | 3.1B (bf16, 6.2 GB) |
| Input | A state with `shell` and `command`, and named questions |
| Output | P(yes) for yes/no questions; a distribution over levels for `risk_level` |
| Training data | [CLI Command Risk](https://huggingface.co/datasets/yannelli/cli-command-risk), `train` split, 12,383 commands from 727 template families |
| Test results | Category macro F1 0.929 (cutoff 0.5), risk level macro F1 0.669, accuracy 0.963, ECE 0.030 |
| License | LFM Open License v1.0 |

## Questions

`questions.json` holds the 17 questions in the System One schema. They are the same for every command. The criteria of `risk_level` and of each category carry the definitions from the dataset's `taxonomy.json`.

| Question | Type | Asks | Answer |
|---|---|---|---|
| `risk_level` | score | How much harm can running this shell command as written cause? | Distribution over 5 levels, 0 `safe` to 4 `critical` |
| `risky` | noul | Is this shell command risky to run as written? | P(yes) |
| `data_destruction` | noul | Does this command carry data destruction risk? | P(yes) |
| `device_overwrite` | noul | Does this command carry device overwrite risk? | P(yes) |
| `overbroad_permissions` | noul | Does this command carry overbroad permissions risk? | P(yes) |
| `privilege_escalation` | noul | Does this command carry privilege escalation risk? | P(yes) |
| `remote_code_execution` | noul | Does this command carry remote code execution risk? | P(yes) |
| `unsafe_download_exec` | noul | Does this command carry unsafe download exec risk? | P(yes) |
| `resource_exhaustion` | noul | Does this command carry resource exhaustion risk? | P(yes) |
| `credential_exposure` | noul | Does this command carry credential exposure risk? | P(yes) |
| `irreversible_config` | noul | Does this command carry irreversible config risk? | P(yes) |
| `history_log_tampering` | noul | Does this command carry history log tampering risk? | P(yes) |
| `network_disruption` | noul | Does this command carry network disruption risk? | P(yes) |
| `signal_kill` | noul | Does this command carry signal kill risk? | P(yes) |
| `shell_footgun` | noul | Does this command carry shell footgun risk? | P(yes) |
| `reversible` | noul | Can an average user undo the effect of this command without backups? | P(yes) |
| `requires_root` | noul | Does the harm from this command need root or sudo? | P(yes) |

The state's `shell` is one of `bash`, `zsh`, `sh`, `dash`, or `fish`. Risk level `i` is the `i`-th entry of the dataset's `risk_levels` (`safe`, `low`, `moderate`, `high`, `critical`).

## Usage

```sh
pip install torch==2.14.1 transformers==5.19.0 huggingface_hub safetensors
```

```python
import json

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
model = AutoModel.from_pretrained("yannelli/d1-3b-cli-command-risk",
                                  trust_remote_code=True, dtype=dtype).to(device)

# The 17 questions the model was trained on
questions = json.load(open(hf_hub_download("yannelli/d1-3b-cli-command-risk", "questions.json")))["questions"]

state = {"shell": "bash", "command": "dd if=/dev/zero of=/dev/sda"}
answers = model.system_one(state, questions)["answers"]
print(json.dumps(answers, indent=1))
```

The example is a test-split command the model did not see in training. Its gold labels are `critical`, `data_destruction`, `device_overwrite`, and `requires_root`. This model's answers, from the code above on CPU:

| Question | Gold | This model |
|---|---|---|
| `risk_level` | `critical` | `critical` (0.999) |
| `risky` | yes | P(yes) 1.00 |
| `data_destruction` | yes | P(yes) 0.98 |
| `device_overwrite` | yes | P(yes) 1.00 |
| The other 11 categories | no | P(yes) 0.00 |
| `reversible` | no | P(yes) 0.00 |
| `requires_root` | yes | P(yes) 1.00 |

`system_one` returns the System One answer shape. A `noul` answer is P(yes). A `score` answer has `score` (the expected level), `confidence`, `probabilities` over levels, and the `legend` with each level's definition. `model.system_one_batch` answers several commands in one call. The `risk_level` answer above, rounded, with the legend cut short:

```json
{
 "type": "score",
 "score": 3.998,
 "confidence": 0.999,
 "probabilities": {
  "0": 0.0,
  "1": 0.0,
  "2": 0.0,
  "3": 0.0,
  "4": 0.999
 },
 "legend": {
  "0": "Safe. Ordinary read-only or routine command with no plausible path to data loss, privilege change, or system disruption.",
  "4": "Critical. Destroys a whole disk, filesystem, operating system, system directory, or home folder, or gives root-level control to any local user or remote party."
 }
}
```

On Apple Silicon set `device = "mps"` (not benchmarked). On CPU the model runs in float32. For faster CPU inference, or to run under llama.cpp or Ollama, use the [GGUF builds](https://huggingface.co/yannelli/cli-command-risk-GGUF).

The model accepts other questions in the same schema, but it was trained and evaluated only on these 17.

## Results

Results cover all 12,689 rows of the `test` split: 97 template families and 2,551 distinct commands, each in up to five shells. Each command was one request with all 17 questions, scored with `training/score_cli_decisions.py`, which computes the same columns as the scorer of [JS/TS Code Review Decisions](https://huggingface.co/datasets/yannelli/js-ts-code-review-decisions).

| Model | Macro F1 @ 0.5 | Level macro F1 | Accuracy | Log loss | Brier | ECE | Safe judged safe |
|---|---:|---:|---:|---:|---:|---:|---:|
| **d1-3B (this model)** | **0.929** | **0.669** | **0.963** | 0.136 | 0.042 | 0.030 | **0.988** |
| This model + temperature | 0.929 | 0.669 | 0.963 | **0.099** | **0.037** | **0.027** | 0.988 |
| Train label frequencies (prior) | 0.000 | 0.101 | 0.814 | 0.398 | 0.143 | 0.070 | 1.000 |

Macro F1 averages F1 at a 0.5 cutoff over the 13 category questions. Level macro F1 averages F1 over the 5 risk levels, using the most likely level. Accuracy counts every decision. Safe judged safe is the share of the 4,296 safe test commands with P(`risky`) below 0.5.

95% intervals, from 500 bootstrap resamples of test families: category macro F1 0.707 to 0.984, level macro F1 0.464 to 0.789, accuracy 0.948 to 0.981, log loss 0.072 to 0.200. By question type its accuracy is 0.976 on the 16 nouls and 0.756 on `risk_level`.

### Guardrail cutoff

For a guardrail, use P(not `safe`) = 1 - P(`risk_level` = 0). The cutoff that flags 1% of safe validation commands is 0.714. On test, that cutoff flags 99.9% of `high` and `critical` commands and 1.7% of safe commands.

### Risk level

F1 by level: `safe` 0.989, `low` 0.257, `moderate` 0.860, `high` 0.508, `critical` 0.733. Most level errors are one level off. Eight template families hold 2,909 of the 3,097 wrong levels; the largest are:

- `find . -name * .php -delete` (995 rows): labeled `critical`, predicted `high`.
- `chmod -R` on system paths (537 rows): labeled `critical`, predicted `high`.
- `iptables -A INPUT -s <range> -j DROP` (450 rows): labeled `low`, predicted `high`.
- `dd if=/dev/zero of=/dev/sda bs=512 count=1` (257 rows): labeled `critical`, predicted `low`.

### Per question

| Question | Type | Test positives | Accuracy | F1 @ 0.5 | Log loss | ECE |
|---|---|---:|---:|---:|---:|---:|
| `risk_level` | score |  | 0.756 |  | 1.091 | 0.186 |
| `risky` | noul | 8,393 | 0.996 | 0.997 | 0.022 | 0.004 |
| `data_destruction` | noul | 3,467 | 0.923 | 0.876 | 0.170 | 0.065 |
| `device_overwrite` | noul | 2,400 | 0.998 | 0.994 | 0.006 | 0.002 |
| `overbroad_permissions` | noul | 1,291 | 1.000 | 0.998 | 0.001 | 0.000 |
| `privilege_escalation` | noul | 20 | 1.000 | 1.000 | 0.000 | 0.000 |
| `remote_code_execution` | noul | 250 | 1.000 | 1.000 | 0.000 | 0.000 |
| `unsafe_download_exec` | noul | 620 | 1.000 | 1.000 | 0.000 | 0.000 |
| `resource_exhaustion` | noul | 335 | 1.000 | 1.000 | 0.002 | 0.002 |
| `credential_exposure` | noul | 510 | 1.000 | 1.000 | 0.001 | 0.001 |
| `irreversible_config` | noul | 649 | 0.956 | 0.693 | 0.054 | 0.032 |
| `history_log_tampering` | noul | 14 | 1.000 | 0.812 | 0.002 | 0.000 |
| `network_disruption` | noul | 460 | 1.000 | 1.000 | 0.000 | 0.000 |
| `signal_kill` | noul | 33 | 1.000 | 1.000 | 0.000 | 0.000 |
| `shell_footgun` | noul | 1,776 | 0.934 | 0.704 | 0.316 | 0.063 |
| `reversible` | noul | 5,576 | 0.884 | 0.859 | 0.380 | 0.098 |
| `requires_root` | noul | 4,495 | 0.933 | 0.911 | 0.265 | 0.053 |

### Calibration

Answers are the model's raw softmax (test ECE 0.030). `temperatures.json` holds one temperature per question, fitted on the 472 validation commands by log loss. Replacing each answer's distribution p with softmax(log p / T) moves test log loss from 0.136 to 0.099 and ECE from 0.030 to 0.027. It never changes the top answer or the order of P(yes) values, so F1 and accuracy stay the same.

### Speed

Median wall-clock time for one command with all 17 questions, batch size 1.

| Hardware | Runtime | Commands | p50 per command |
|---|---|---:|---:|
| One A100 SXM 80GB | PyTorch, bf16 | 12,689 | 44 ms |
| 16 Neoverse V3 CPU cores | llama.cpp `llama-server`, Q8_0 GGUF | 106 | 3.3 s |
| 16 Neoverse V3 CPU cores | Ollama 0.40.3, Q8_0 GGUF | 106 | 4.8 s |

## Intended use

Rate a shell command before a terminal or a coding agent runs it: block or ask for confirmation above a risk cutoff, and show the categories and the reversibility answer to the person who decides. The probabilities let you pick a cutoff per question and trade false alarms against misses.

## Training

- Data: the `train` split of [CLI Command Risk](https://huggingface.co/datasets/yannelli/cli-command-risk), posed as 17 decisions per command by `training/build_cli_decisions.py`. Rows in one template family are near copies, so training keeps one shell per distinct command and at most 40 commands per family: 12,383 commands from 727 families (210,511 decisions). Validation keeps at most 8 commands per family (472 commands). The source Parquet files match the SHA-256 hashes in the dataset's published `manifest.json`.
- The dataset keeps each template family in one split, so no test family was seen in training.

| Setting | Value |
|---|---|
| Method | LoRA (r 32, alpha 64, dropout 0.05) on the language model's projection layers (`LORA_TARGETS` in `training/liquid_d1.py`), merged into the weights after training |
| Objective | Cross entropy over each question's answer tokens; the command is the trunk of d1's packed tree and the 17 questions are its branches |
| Schedule | 1 epoch, 1,548 steps of 8 commands |
| Optimizer | AdamW, learning rate 2e-4, 5% warmup, linear decay |
| Precision | bf16 |
| Hardware | One A100 SXM 80GB, 87 minutes |
| Validation loss | 0.106 at the end of training (300 validation commands) |

`training/` holds the scripts that built the decisions, trained and scored this model. To rerun it:

```sh
pip install peft==0.21.2 pyarrow numpy
hf download yannelli/cli-command-risk --repo-type dataset --local-dir cli-command-risk
python training/build_cli_decisions.py --out cli-data
export DECISIONS_DATA=cli-data
python training/liquid_d1.py train --model LiquidAI/d1-3B --ckpt ckpt --epochs 1 --save-every 200
python training/liquid_d1.py predict --ckpt ckpt --split test --out test-answers.jsonl
python training/score_cli_decisions.py test-answers.jsonl
```

## Evaluation files

`eval/test-answers.jsonl` holds this model's answers to all 12,689 test commands, `eval/score.txt` is the scorer's output for it, and `eval/guardrail.json` has the risk-level and cutoff results from `training/score_cli_risk.py`. To rescore, after building `cli-data` as above:

```sh
python training/score_cli_decisions.py eval/test-answers.jsonl
```

## Limitations

- Labels come from language models following a written taxonomy, with model audits and a relabel pass, not from human annotators. Scores measure agreement with those labels.
- Labels rate the command alone, not the machine it runs on.
- Training commands use placeholders such as `example.com` and `<TOKEN>` in place of real hosts and secrets.
- The risk level is the hardest question (accuracy 0.756); `low` (F1 0.257) and `high` (F1 0.508) are the weakest levels.

## License

This model is a derivative of [LiquidAI/d1-3B](https://huggingface.co/LiquidAI/d1-3B) and is released under the base model's LFM Open License v1.0, included as `LICENSE`. The license allows commercial use only for entities with less than $10M in annual revenue; read `LICENSE` for the full terms.

The training data, [CLI Command Risk](https://huggingface.co/datasets/yannelli/cli-command-risk), is CC BY 4.0.

## Citation

```bibtex
@misc{yannelli2026d13bclicommandrisk,
  title        = {d1-3B CLI Command Risk},
  author       = {Ryan Yannelli},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/yannelli/d1-3b-cli-command-risk}}
}

@misc{yannelli2026clicommandrisk,
  title        = {CLI Command Risk},
  author       = {Ryan Yannelli},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/yannelli/cli-command-risk}}
}
```

## Maintainer

[Ryan Yannelli](https://ryanyannelli.com) | 10/19/2026 | [GitHub: @yannelli](https://github.com/yannelli) | [Hugging Face: yannelli](https://huggingface.co/yannelli)