PentaCMD-47M

⚠️ Working name — rename PentaCMD throughout if you choose a different one.

A 47M-parameter, from-scratch (nanoGPT-style) small language model that turns a plain-English instruction + a family hint into a single terminal command, across five tools: bash, git, npm, python (pip/venv), and PowerShell.

It is "Version 0" — a baseline trained end-to-end as a learning project, to be iterated on.

What it does

Given a prompt in this exact format:

### Task (git): undo my last commit but keep my changes
### Command:

it generates:

git reset --soft HEAD~1

The family tag (bash | git | npm | python | powershell) is provided as an input hint so the same English can map to the right tool.

Results (held-out test set, exact-match)

Family	Exact-match	First-word
git	100.0%	100%
powershell	98.7%	100%
npm	97.3%	100%
python	69.3%	~100%
bash	68.0%	~75%
Blended	~86.7%	—

"Exact-match" = the generated command is byte-for-byte identical to the reference, including the argument. "First-word" = correct command/verb. Blended test number matches the validation number (~86.7%) — i.e. no overfitting.

Baseline before scale-up (20M model, ~36K examples): 58.7% blended.

Model details

• Architecture: decoder-only transformer (nanoGPT-style), 8 layers, 10 heads, 640 embedding dim, 256-token context, weight-tied embeddings, ~47.2M params.
• Tokenizer: byte-level BPE, vocab 12,000, special tokens <|endoftext|>, <|pad|>.
• Precision: trained in fp16 (mixed precision).

Training data

• 299,329 instruction→command pairs across 5 families, de-duplicated on (family, instruction).
• Per family: git 120,000 · python 90,000 · bash 41,329 · powershell 28,000 · npm 20,000.
• bash = the real-world NL2Bash corpus (~11K) + synthetic beginner commands (30K). The other four families are synthetic, generated from templates with large entity pools (thousands of distinct package/branch/file names) so the model learns to copy arbitrary arguments rather than memorize a fixed set.
• Leak-free 90/5/5 split: grouped by command so paraphrases never straddle splits → train 269,396 / val 14,967 / test 14,966 (~6.54M training tokens).

Training procedure

• Hardware: single NVIDIA T4 (Kaggle), ~53 minutes.
• Optimizer: AdamW (betas 0.9/0.95, weight decay 0.1).
• LR: 6e-4 peak, 300-step warmup, cosine decay to 6e-5.
• Batch size 32, 14,000 steps, gradient clip 1.0.
• Checkpointed by exact-match accuracy on a fixed val subset (not validation loss — val loss is misleading here because the leak-free split holds out unseen command groups). Best checkpoint was step 11,000.

Intended use & limitations

• Intended: an assistant/autocomplete for common CLI tasks for beginners; an educational example of an end-to-end small-LM project.
• Limitations:
  • python (69%) and bash (68%) are weaker. python has five installers (pip/uv/poetry/conda/pipx) so "install X" is ambiguous, and package names are long/multi-token; bash includes research-hard NL2Bash pipelines.
  • The model can produce a plausible-but-wrong argument (e.g. install the wrong package). Always review generated commands before running them, especially destructive ones (rm -rf, git reset --hard, Remove-Item).
  • Outputs are not validated by execution.

How to use

This is a custom architecture, so load the weights with the provided modeling code (modeling_pentacmd.py). Minimal inference:

import torch
from tokenizers import Tokenizer
from modeling_pentacmd import GPT   # the model class

device = "cuda" if torch.cuda.is_available() else "cpu"
tok = Tokenizer.from_file("tokenizer.json")
ckpt = torch.load("best_model.pt", map_location=device)
model = GPT(**ckpt["config"]).to(device)
model.load_state_dict(ckpt["model"]); model.eval()
eot = tok.token_to_id("<|endoftext|>")

@torch.no_grad()
def command_for(family, instruction, block_size=256, max_new=64):
    ids = tok.encode(f"### Task ({family}): {instruction}\n### Command:").ids
    x = torch.tensor([ids], device=device)
    for _ in range(max_new):
        nid = model(x[:, -block_size:])[:, -1, :].argmax(-1, keepdim=True)
        x = torch.cat([x, nid], 1)
        if nid.item() == eot: break
    return tok.decode(x[0].tolist()).split("### Command:")[-1].split("###")[0].strip()

print(command_for("npm", "install the express package"))  # -> npm install express

License / attribution

• Model weights & synthetic data: CC BY-NC 4.0 (attribution, non-commercial).
• bash portion: derives from TellinaTool/nl2bash and is subject to that project's own license (verify in their repo — likely GPL-3.0). Comply with it before redistributing the bash data or models trained on it.
• Code (architecture, inference, data pipeline) is MIT — see the GitHub repo.

Citation

If you use this, please credit the author and link the GitHub/Hugging Face repos.