Instructions to use vinpix/Bonsai-8B-llama.cpp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use vinpix/Bonsai-8B-llama.cpp with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="vinpix/Bonsai-8B-llama.cpp",
	filename="Bonsai-8B-Q2_KT.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use vinpix/Bonsai-8B-llama.cpp with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vinpix/Bonsai-8B-llama.cpp:Q2_K
# Run inference directly in the terminal:
llama cli -hf vinpix/Bonsai-8B-llama.cpp:Q2_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf vinpix/Bonsai-8B-llama.cpp:Q2_K
# Run inference directly in the terminal:
llama cli -hf vinpix/Bonsai-8B-llama.cpp:Q2_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf vinpix/Bonsai-8B-llama.cpp:Q2_K
# Run inference directly in the terminal:
./llama-cli -hf vinpix/Bonsai-8B-llama.cpp:Q2_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf vinpix/Bonsai-8B-llama.cpp:Q2_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf vinpix/Bonsai-8B-llama.cpp:Q2_K

Use Docker

docker model run hf.co/vinpix/Bonsai-8B-llama.cpp:Q2_K

LM Studio
Jan

vLLM

How to use vinpix/Bonsai-8B-llama.cpp with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vinpix/Bonsai-8B-llama.cpp"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vinpix/Bonsai-8B-llama.cpp",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vinpix/Bonsai-8B-llama.cpp:Q2_K

Ollama
How to use vinpix/Bonsai-8B-llama.cpp with Ollama:
```
ollama run hf.co/vinpix/Bonsai-8B-llama.cpp:Q2_K
```

Unsloth Studio

How to use vinpix/Bonsai-8B-llama.cpp with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for vinpix/Bonsai-8B-llama.cpp to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for vinpix/Bonsai-8B-llama.cpp to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for vinpix/Bonsai-8B-llama.cpp to start chatting

How to use vinpix/Bonsai-8B-llama.cpp with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vinpix/Bonsai-8B-llama.cpp:Q2_K

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "vinpix/Bonsai-8B-llama.cpp:Q2_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use vinpix/Bonsai-8B-llama.cpp with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vinpix/Bonsai-8B-llama.cpp:Q2_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default vinpix/Bonsai-8B-llama.cpp:Q2_K

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use vinpix/Bonsai-8B-llama.cpp with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf vinpix/Bonsai-8B-llama.cpp:Q2_K

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "vinpix/Bonsai-8B-llama.cpp:Q2_K" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use vinpix/Bonsai-8B-llama.cpp with Docker Model Runner:
```
docker model run hf.co/vinpix/Bonsai-8B-llama.cpp:Q2_K
```

Lemonade

How to use vinpix/Bonsai-8B-llama.cpp with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull vinpix/Bonsai-8B-llama.cpp:Q2_K

Run and chat with the model

lemonade run user.Bonsai-8B-llama.cpp-Q2_K

List all available models

lemonade list

Bonsai-8B-llama.cpp / README.md

vinpix

Update model card: TQ2_0 (CPU) + Q2_KT (GPU) unified card

8d289c9 verified 1 day ago

preview code

Raw

History Blame Contribute Delete

8.88 kB

	---
	license: apache-2.0
	base_model: prism-ml/Ternary-Bonsai-8B
	tags:
	- gguf
	- llama.cpp
	- ternary
	- 1.58-bit
	- quantization
	- q2_k
	- tq2_0
	- qwen3
	- edge
	- cpu
	- vulkan
	language:
	- en
	pipeline_tag: text-generation
	library_name: gguf
	---

	# Ternary-Bonsai-8B for llama.cpp

	GGUF builds of [PrismML's Ternary-Bonsai-8B](https://huggingface.co/prism-ml/Ternary-Bonsai-8B) for `llama.cpp`, covering both deployment targets: a native `TQ2_0` build for CPU, and a ternary-aware zero-exact `Q2_K` build for GPU and every k-quant runtime — at matching perplexity.

	Ternary-Bonsai is an end-to-end 1.58-bit model (weights in `{-1, 0, +1}`, trained with ternary QAT, no higher-precision escape hatches). It punches far above its size class, but on `llama.cpp` it runs into a practical wall: the tight ternary format has no GPU kernel. This repo provides both halves of the solution.

	## Files

	\| File \| Format \| BPW \| Size \| Backends \| PPL (wikitext-2, 20x512) \|
	\|---\|---\|---\|---\|---\|---\|
	\| `Bonsai-8B-TQ2_0.gguf` \| native ternary \| 2.06 \| 2.50 GB \| CPU only \| 12.311 +/- 0.517 \|
	\| `Bonsai-8B-Q2_KT.gguf` \| ternary-aware `Q2_K` \| 2.911 \| 2.99 GB \| CPU - CUDA - Metal - Vulkan - ROCm - SYCL \| 12.299 +/- 0.516 \|

	Both decode to the same learned weights. They differ only in storage container, and therefore in which backends can run them and how fast.

	> TL;DR — which one?
	> CPU box -> `TQ2_0` (fewer bytes streamed, ~12-14 tok/s, smallest RAM).
	> GPU offload, or any runtime that only speaks k-quants -> `Q2_KT`.

	---

	## Why two files

	### `TQ2_0` — the native CPU format
	`TQ2_0` is the tightest faithful ternary layout in `ggml` (~2.06 BPW). It is the right choice on CPU and it is what PrismML ships. Its one hard limit:

	> `TQ2_0` has no GPU kernel. In current `llama.cpp` it implements a `vec_dot` for CPU only — no CUDA, Metal, or Vulkan path. `-ngl` silently keeps the ternary tensors on the host.

	### `Q2_KT` — the GPU-portable format
	`Q2_K` is one of the most broadly supported quant types in the ecosystem — hand-written kernels for CUDA, Metal, Vulkan, ROCm, SYCL. Re-hosting the ternary weights inside a `Q2_K` container makes Bonsai portable everywhere `Q2_K` runs.

	The catch: you cannot just run `llama-quantize ... Q2_K` on a ternary model. Doing so is actively harmful — for a reason worth explaining.

	---

	## The trap: naively quantizing a ternary model to `Q2_K`

	`Q2_K` dequantizes each weight as

	```
	w = d * (scale & 0xF) * q - dmin * (scale >> 4), q in {0, 1, 2, 3}
	```

	The stock `Q2_K` quantizer (`make_qkx2_quants`) is an L2-optimal grid fitter for continuous FP16 distributions. Handed a ternary tensor whose values are exactly `{-s, 0, +s}`, it fits a 4-level grid over 3-level data and places the levels so the ternary zero lands between grid points — at `q ~ 1.5`, rounding to an offset of `+/-s/3`.

	Measured directly on Bonsai-8B:

	\| Encoding \| Sign accuracy vs. original ternary \| Zero reconstruction \|
	\|---\|---\|---\|
	\| Stock `Q2_K` (`--pure`) \| 33 % \| `-0.0095` (should be `0`) \|
	\| `Q2_KT` (this repo) \| 100 % \| `0.000000` (exact) \|

	A `--pure` `Q2_K` of a ternary model gets two thirds of the signs wrong and shifts every zero — in practice a degenerate model that loops single tokens. The stock pipeline masks this by promoting sensitive tensors (`attn_output`, `ffn_down`, `attn_v`) to `Q3_K`/`Q4_K`, inflating the file to ~3.18 BPW and only partially repairing the damage. That promotion is a workaround, not a fix.

	`Q2_KT` fixes the encoding at the source.

	---

	## How `Q2_KT` is built (ternary-aware, zero-exact)

	The ternary structure admits an exact `Q2_K` encoding with no residual. For a group of magnitude `s`:

	```
	scale nibble sc = 15
	min nibble m = 15 (min-nibble == scale-nibble)
	d = dmin = s / 15
	q = t + 1 t in {-1, 0, +1} -> q in {0, 1, 2}
	```

	Substituting into the dequant:

	```
	w = d15q - dmin15 = d15(q - 1) = s(q - 1) = s*t <- exact ternary
	```

	The `-1` bias falls out of the min term for free; the ternary zero (`t = 0 -> q = 1`) reconstructs to exactly 0; every non-zero to exactly `+/-s`. The only error is the FP16 storage of the single per-group scale — which `TQ2_0` also carries. Hence sign accuracy 100 % and PPL parity.

	The `qs` nibbles are packed with the exact `ggml` `Q2_K` interleave (32-byte groups, stride-32 crumbs across 128-element spans), so the file is bit-compatible with the reference `dequantize_row_q2_K` and the SIMD `vec_dot_q2_K_q8_K` in the forward pass. Output projection and token embedding are kept at `Q4_0` (not ternary in the source).

	> Layout note. `TQ2_0` and `Q2_K` `qs` streams share the same stride-32 / 128-span interleave. Matching `ggml`'s canonical order is the whole game: an encoding that is self-consistent with a hand-rolled unpacker but disagrees with the canonical order will pass every fidelity check against itself and still emit garbage in the forward pass, because `dequantize_row` and `vec_dot` both assume the canonical order. This build matches it.

	---

	## Usage

	### CPU (use `TQ2_0`)

	```bash
	llama-cli -m Bonsai-8B-TQ2_0.gguf -p "The capital of France is" -n 64 -t 6
	```

	Fastest on CPU and smallest RAM footprint. ~12-14 tok/s on a low-end Zen2 desktop (Ryzen 3 PRO 4350GE, dual-channel DDR4).

	### GPU offload (use `Q2_KT`)

	```bash
	llama-cli -m Bonsai-8B-Q2_KT.gguf -ngl 99 -p "The capital of France is" -n 64
	```

	Verified running fully offloaded on an AMD Radeon Vega iGPU via the Vulkan `RADV` backend. Because `Q2_K` has real GPU kernels, `-ngl 99` actually places the ternary body on the device — unlike `TQ2_0`, which would stay on the host.

	### Any GGUF loader / llama.cpp server

	`Q2_KT` loads as an ordinary `Q2_K` model — no custom code, no patched runtime. Every existing loader already understands the container.

	---

	## Quality

	Perplexity on `wikitext-2-raw` (test split, 20 chunks of 512 tokens, `-t 4`, identical harness):

	\| Model \| Format \| BPW \| PPL (lower is better) \|
	\|---\|---\|---\|---\|
	\| `Bonsai-8B-TQ2_0` \| native ternary \| 2.06 \| 12.311 +/- 0.517 \|
	\| `Bonsai-8B-Q2_KT` \| ternary-aware `Q2_K` \| 2.911 \| 12.299 +/- 0.516 \|

	`Q2_KT` is perplexity-neutral vs. the native ternary build (delta within the +/-0.5 statistical error). You pay ~0.85 BPW over `TQ2_0` for universal backend support; you do not pay for it in quality — expected, since the encoding is lossless on the ternary levels.

	For task benchmarks (MMLU-Redux, GSM8K, HumanEval+, IFEval, ...), see [PrismML's numbers](https://prismml.com/news/ternary-bonsai) for the base model — these files change the container, not the learned weights.

	---

	## Technical notes & reproducibility

	- `Q2_KT` container: 252 ternary body tensors -> pure `Q2_K` (zero-exact); `output` + `token_embd` -> `Q4_0`; 145 norms -> `F32`.
	- No tensor promotion. Every ternary tensor is pure `Q2_K`; the zero-exact encoding makes the `Q3_K`/`Q4_K` promotion unnecessary — that is why `Q2_KT` is 2.91 BPW, not 3.18.
	- Bit-exactness: validated against `ggml`'s reference `dequantize_row_q2_K` (100 % sign accuracy, max abs error 1e-5 from the FP16 scale) and end-to-end against the forward `vec_dot` (coherent generation, PPL parity).
	- Architecture: Qwen3, 36 layers, `hidden 4096`, `FFN 12288`, GQA 32/8, full 65536-token context preserved in metadata.
	- Reproduce PPL: `llama-perplexity -m <file> -f wiki.test.raw --chunks 20`.

	### A note for the llama.cpp community

	The `+/-s/3` zero-error affects any end-to-end ternary model (Bonsai, BitNet, TriLM, ...) run through the stock `Q2_K` quantizer. A ternary-aware branch in `quantize_row_q2_K` — detect a block with <=3 symmetric levels, emit `d = dmin = s/15`, `sc = m`, `q = t+1` — would let the official pipeline produce zero-exact `Q2_K` for the whole class, without the promotion workaround. `Q2_KT` is a concrete demonstration that the encoding is exact and backend-portable. Contributions upstream welcome.

	---

	## Attribution & license

	- Base model: [Ternary-Bonsai-8B](https://huggingface.co/prism-ml/Ternary-Bonsai-8B) by PrismML (Babak Hassibi et al., Caltech). Ternary QAT, Qwen3 architecture, end-to-end 1.58-bit.
	- These files: GGUF re-encodes for `llama.cpp`. `TQ2_0` is the native ternary layout; `Q2_KT` is a lossless-on-ternary `Q2_K` re-encode for backend portability. Learned weights unchanged.
	- License: Apache 2.0, inherited from the base model.

	```bibtex
	@techreport{ternarybonsai,
	title = {Ternary Bonsai: 1.58-bit Language Models at 8B, 4B, and 1.7B Scale},
	author = {Prism ML},
	year = {2026},
	month = {April},
	url = {https://prismml.com}
	}
	```

	---

	Community re-encodes, not affiliated with or endorsed by PrismML. Quality figures are measured on the specific files in this repository.