cmdcaliper — Linux Command Intent & Semantic Similarity Engine

This model is a fine-tuned version of thenlper/gte-base, a 110M parameter BERT-based encoder, specifically optimised for Linux command intent detection and semantic similarity.

It follows the methodology of the CmdCaliper research paper (Empirical Software Engineering & Cybersecurity). The model maps Linux shell commands and Bash snippets to a 768-dimensional dense vector space where semantically equivalent commands are clustered together, while syntactically similar but semantically distinct commands (e.g. cat /etc/passwd vs cat /etc/shadow) are pushed apart.

---

Model Details

Model Description

Property	Value
Model Type	Sentence Transformer (BERT-base)
Base Model	thenlper/gte-base
Architecture	BertModel → Mean Pooling → L2 Normalise
Max Sequence Length	128 tokens
Output Dimensionality	768 dimensions
Similarity Function	Cosine Similarity
Training Objective	InfoNCE / Multiple Negatives Ranking Loss (MNRL)
InfoNCE Temperature (τ)	0.05
Language	English (Linux / Bash commands)
License	MIT

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean'})
  (2): Normalize({})
)

---

Training Details

Training Objective

The model was trained using InfoNCE (Multiple Negatives Ranking Loss) with a collision-free batch sampler that enforces:

• No anchor duplicates in a batch.
• No positive duplicates in a batch.
• No cross-collision between positives and anchors of other pairs.
• Bidirectional partner protection: (a, p) and (p, a) never co-appear in the same batch.
• Balanced positive indices across batches.
• Soft category balancing via square-root inverse weighting.

Training Data

Fine-tuned on a curated dataset of Linux command positive pairs (merged_positive_pairs_clean.json) covering diverse categories including:

• File system operations
• Process management
• Network operations
• Security-sensitive commands (privilege escalation, reverse shells, anti-forensics)
• Package management
• System monitoring

Train / Eval split: 95% / 5%

Hyperparameters

Parameter	Value
Epochs	3
Batch size (per device)	64
Learning rate	2e-5
Weight decay	0.01
Warmup ratio	0.05
Max gradient norm	1.0
Precision	FP32 (strict)

Framework Versions

Framework	Version
Python	3.12
Sentence Transformers	5.4.1
Transformers	5.0.0
PyTorch	2.10.0+cu128
Tokenizers	0.22.2

---

Performance

Evaluated on a retrieval task over 500 held-out command pairs:

Metric	Score
MRR@10	≈ 0.8847 (CmdCaliper-base target)
Top@10	≈ 0.9526 (CmdCaliper-base target)
3-NN Accuracy	88.41%
3-NN Precision (Malicious)	87.83%
3-NN F1 Score	85.23%

---

Usage

Direct Usage (Sentence Transformers)

pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer, util

# Load model directly from Hugging Face Hub
model = SentenceTransformer("Ameya-Kawade/cmdcaliper")

# Commands to compare
commands = [
    "ls -la /home",
    "ls --all -l /home",         # semantically identical
    "docker ps -a",
    "docker container ls --all", # semantically identical
    "chmod 755 script.sh",
    "chmod u+s /bin/bash",       # semantically DIFFERENT (privilege escalation)
]

embeddings = model.encode(commands)
print("Embedding shape:", embeddings.shape)
# (6, 768)

# Pairwise similarity
similarities = model.similarity(embeddings, embeddings)
print(similarities)

Security Intent Detection Example

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("Ameya-Kawade/cmdcaliper")

pairs = [
    ("find /var/log -type f -name '*.log'", "ls -lh /var/log/syslog"),           # HIGH similarity
    ("apt-get install nginx",               "apt-get remove nginx"),              # MED similarity
    ("cat /etc/passwd",                     "cat /etc/shadow"),                   # LOW (Hard-Neg)
    ("kill -9 18452",                       "kill -9 1"),                         # LOW (Hard-Neg)
]

for cmd1, cmd2 in pairs:
    e1 = model.encode(cmd1)
    e2 = model.encode(cmd2)
    sim = util.cos_sim(e1, e2).item()
    print(f"{sim:.4f}  {cmd1[:40]:40s} <->  {cmd2[:40]}")

---

Intended Uses & Limitations

Intended Uses

• Command intent classification: cluster commands by semantic intent (file ops, network, privilege escalation, etc.).
• Semantic deduplication: identify functionally equivalent commands written in different syntax.
• Security-aware retrieval: retrieve the most similar known-malicious or known-benign commands for a given query command.
• Anomaly detection: embed command sequences and flag outliers.

Out-of-Scope Uses

• Natural language sentences (the model is optimised for shell commands and Bash syntax).
• Code in languages other than Bash/shell.

Known Limitations

• Weak on Denial-of-Service (DoS) class discrimination (known gap).
• Intra-class cosine similarity (mu_intra ~ 0.038) is lower than target (>= 0.300); a follow-up training run with a hardened triplet loss and higher temperature (tau = 0.08) is planned.
• Hard-negative traps passed: 10/15 (66.7%); target is >= 14/15.

---

Citation

If you use this model, please cite the original CmdCaliper paper:

@misc{cmdcaliper2024,
  title   = {CmdCaliper: A Semantic-Aware Command-Line Embedding Model and Dataset for Security Research},
  year    = {2024},
  url     = {https://arxiv.org/abs/2411.01176}
}

---

Model Card Authors

Ameya Kawade — Hugging Face