Instructions to use micrictor/bonsai-shellai with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use micrictor/bonsai-shellai with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf micrictor/bonsai-shellai:Q2_K
# Run inference directly in the terminal:
llama cli -hf micrictor/bonsai-shellai:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf micrictor/bonsai-shellai:Q2_K
# Run inference directly in the terminal:
llama cli -hf micrictor/bonsai-shellai:Q2_K

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf micrictor/bonsai-shellai:Q2_K
# Run inference directly in the terminal:
./build/bin/llama cli -hf micrictor/bonsai-shellai:Q2_K

Use Docker

docker model run hf.co/micrictor/bonsai-shellai:Q2_K

Llama new
LM Studio
Jan

vLLM

How to use micrictor/bonsai-shellai with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "micrictor/bonsai-shellai"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "micrictor/bonsai-shellai",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/micrictor/bonsai-shellai:Q2_K

Ollama
How to use micrictor/bonsai-shellai with Ollama:
```
ollama run hf.co/micrictor/bonsai-shellai:Q2_K
```
Unsloth Desktop

How to use micrictor/bonsai-shellai with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf micrictor/bonsai-shellai:Q2_K

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "micrictor/bonsai-shellai:Q2_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use micrictor/bonsai-shellai with Docker Model Runner:
```
docker model run hf.co/micrictor/bonsai-shellai:Q2_K
```

Lemonade

How to use micrictor/bonsai-shellai with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull micrictor/bonsai-shellai:Q2_K

Run and chat with the model

lemonade run user.bonsai-shellai-Q2_K

List all available models

lemonade list

Hermes Agent

How to use micrictor/bonsai-shellai with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf micrictor/bonsai-shellai:Q2_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default micrictor/bonsai-shellai:Q2_K

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use micrictor/bonsai-shellai with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf micrictor/bonsai-shellai:Q2_K

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "micrictor/bonsai-shellai:Q2_K" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Bonsai ShellAI

A compressed, text-only GGUF derivative of Google DeepMind's Gemma 4 E2B instruction-tuned model, prepared for ShellAI command suggestions. The model generates text containing shell-command suggestions; the GGUF itself does not execute commands.

Artifact

bonsai-shellai-Q4_K_S-MLP10-PLE_Q2_K-VOCAB_Q2_K.gguf

Size: 1,926,811,424 bytes (1.927 GB; 1.794 GiB).
SHA256: d10317a1da0965d365a232500603863761f7ff76e1e04924f691ee080203049b.
Parent revision: 3e22461f65e89153144f8adb70e3b8c2cc9845a7.

Download

hf download micrictor/bonsai-shellai bonsai-shellai-Q4_K_S-MLP10-PLE_Q2_K-VOCAB_Q2_K.gguf --local-dir ./bonsai-shellai

The GGUF contains the tokenizer and model weights. Use a Gemma 4-compatible llama.cpp runtime; the tested runtime revision is recorded below. Configure the application to accept a single JSON Markdown fence as described in the evaluation.

Modifications and calibration

The nominal 10% MLP-channel cut is 8.333% after alignment: base FFN width 6144 becomes 5632, while the final 20 shared-KV layers retain twice that width. Depth, residual width, attention/KV sharing and vocabulary are preserved. The separate per-layer embedding (PLE) table and the tied main input/output vocabulary table are Q2_K. The body uses a calibrated Q4_K_S mixed-precision recipe, including eight Q5_K matrices and floating-point exceptions; this is not an all-Q2 model or a uniform four-bit model.

Activation-guided selection uses 95 verified training contexts. A fresh importance matrix was collected on the pruned BF16 model using those training contexts, processing 16,384 tokens in eight chunks. All 275 quantized body matrices passed coverage and dimension checks. Main and PLE lookup tables have no importance-matrix entries and use unweighted quantization; no lookup statistics were fabricated. Export starts from BF16, without requantizing a quantized model. No recovery training, SFT, RL or distillation was performed.

Conversion/quantization used llama.cpp revision 5f55650a78f92aff4d48d671423e888fac0469ff. Use a compatible Gemma 4 GGUF runtime and the native Gemma chat template. Runtime versions differ in architecture support; the artifact has been tested with that pinned implementation.

Measured behavior and limitations

On a reused, inspected 24-case development panel of executable shell-command fixtures, this model passed 14/24 when an unchanged normalization rule trims outer whitespace and unwraps one complete JSON Markdown fence. Those are exactly the same 14 passing cases as the full calibrated Q4 control. This is a narrow matched comparison, not a held-out benchmark or proof of general equivalence.

Under the strict JSON interface this model passed 5/24, versus 14/24 for the control. Fence-tolerant results do not represent strict-parser performance; the normalizer does not repair command content or extract commands from prose. Generated commands were checked in isolated fixtures during evaluation. That evaluation does not make arbitrary suggested commands safe to execute.

CPU evaluation used two threads, context 2048, at most 64 generated tokens, no repacking and swap allowed. Five timing prompts had median response wall time 6.579 s, and median prompt plus decode 5.216 s. Sampled peak benchmark RSS was 1.971 GiB (2.117 GB) with zero sampled process swap. RSS samples are lower bounds affected by paging and background load; they are not a physical-fit guarantee or GPU VRAM measurements. The model does not demonstrate 1.5 GB/GiB inference memory. No Raspberry Pi performance or GPU VRAM threshold has been validated. Review suggestions before execution and validate the actual parser and fresh acceptance cases before use.

License and attribution

The pinned upstream model card identifies Gemma 4 as Apache-2.0 and links to Google's official Apache 2.0 license page. This distribution includes LICENSE, NOTICE and a public artifact manifest. This is a modified derivative, not an official Google DeepMind release. Google/Google DeepMind attribution identifies the upstream source and does not imply endorsement.

Downloads last month: -

GGUF

Model size

4B params

Architecture

gemma4

Hardware compatibility

2-bit

Model tree for micrictor/bonsai-shellai

Base model

google/gemma-4-E2B

Finetuned

google/gemma-4-E2B-it

Quantized

(445)

this model