--- license: apache-2.0 base_model: prism-ml/Ternary-Bonsai-8B tags: - gguf - llama.cpp - ternary - 1.58-bit - quantization - q2_k - tq2_0 - qwen3 - edge - cpu - vulkan language: - en pipeline_tag: text-generation library_name: gguf --- # Ternary-Bonsai-8B for llama.cpp **GGUF builds of [PrismML's Ternary-Bonsai-8B](https://huggingface.co/prism-ml/Ternary-Bonsai-8B) for `llama.cpp`, covering both deployment targets: a native `TQ2_0` build for CPU, and a ternary-aware zero-exact `Q2_K` build for GPU and every k-quant runtime — at matching perplexity.** Ternary-Bonsai is an end-to-end 1.58-bit model (weights in `{-1, 0, +1}`, trained with ternary QAT, no higher-precision escape hatches). It punches far above its size class, but on `llama.cpp` it runs into a practical wall: **the tight ternary format has no GPU kernel.** This repo provides both halves of the solution. ## Files | File | Format | BPW | Size | Backends | PPL (wikitext-2, 20x512) | |---|---|---|---|---|---| | `Bonsai-8B-TQ2_0.gguf` | native ternary | 2.06 | 2.50 GB | **CPU only** | 12.311 +/- 0.517 | | `Bonsai-8B-Q2_KT.gguf` | ternary-aware `Q2_K` | 2.911 | 2.99 GB | **CPU - CUDA - Metal - Vulkan - ROCm - SYCL** | 12.299 +/- 0.516 | Both decode to the same learned weights. They differ only in storage container, and therefore in which backends can run them and how fast. > **TL;DR — which one?** > **CPU box** -> `TQ2_0` (fewer bytes streamed, ~12-14 tok/s, smallest RAM). > **GPU offload, or any runtime that only speaks k-quants** -> `Q2_KT`. --- ## Why two files ### `TQ2_0` — the native CPU format `TQ2_0` is the tightest faithful ternary layout in `ggml` (~2.06 BPW). It is the right choice on CPU and it is what PrismML ships. Its one hard limit: > **`TQ2_0` has no GPU kernel.** In current `llama.cpp` it implements a `vec_dot` for CPU only — no CUDA, Metal, or Vulkan path. `-ngl` silently keeps the ternary tensors on the host. ### `Q2_KT` — the GPU-portable format `Q2_K` is one of the most broadly supported quant types in the ecosystem — hand-written kernels for CUDA, Metal, Vulkan, ROCm, SYCL. Re-hosting the ternary weights inside a `Q2_K` container makes Bonsai portable everywhere `Q2_K` runs. The catch: you **cannot** just run `llama-quantize ... Q2_K` on a ternary model. Doing so is actively harmful — for a reason worth explaining. --- ## The trap: naively quantizing a ternary model to `Q2_K` `Q2_K` dequantizes each weight as ``` w = d * (scale & 0xF) * q - dmin * (scale >> 4), q in {0, 1, 2, 3} ``` The stock `Q2_K` quantizer (`make_qkx2_quants`) is an L2-optimal grid fitter for **continuous** FP16 distributions. Handed a ternary tensor whose values are exactly `{-s, 0, +s}`, it fits a 4-level grid over 3-level data and places the levels so the ternary **zero lands between grid points** — at `q ~ 1.5`, rounding to an offset of `+/-s/3`. Measured directly on Bonsai-8B: | Encoding | Sign accuracy vs. original ternary | Zero reconstruction | |---|---|---| | Stock `Q2_K` (`--pure`) | **33 %** | `-0.0095` (should be `0`) | | **`Q2_KT` (this repo)** | **100 %** | `0.000000` (exact) | A `--pure` `Q2_K` of a ternary model gets **two thirds of the signs wrong** and shifts every zero — in practice a degenerate model that loops single tokens. The stock pipeline masks this by *promoting* sensitive tensors (`attn_output`, `ffn_down`, `attn_v`) to `Q3_K`/`Q4_K`, inflating the file to ~3.18 BPW and only partially repairing the damage. That promotion is a workaround, not a fix. `Q2_KT` fixes the encoding at the source. --- ## How `Q2_KT` is built (ternary-aware, zero-exact) The ternary structure admits an **exact** `Q2_K` encoding with no residual. For a group of magnitude `s`: ``` scale nibble sc = 15 min nibble m = 15 (min-nibble == scale-nibble) d = dmin = s / 15 q = t + 1 t in {-1, 0, +1} -> q in {0, 1, 2} ``` Substituting into the dequant: ``` w = d*15*q - dmin*15 = d*15*(q - 1) = s*(q - 1) = s*t <- exact ternary ``` The `-1` bias falls out of the min term for free; the ternary zero (`t = 0 -> q = 1`) reconstructs to **exactly 0**; every non-zero to **exactly `+/-s`**. The only error is the FP16 storage of the single per-group scale — which `TQ2_0` also carries. Hence sign accuracy 100 % and PPL parity. The `qs` nibbles are packed with the exact `ggml` `Q2_K` interleave (32-byte groups, stride-32 crumbs across 128-element spans), so the file is bit-compatible with the reference `dequantize_row_q2_K` **and** the SIMD `vec_dot_q2_K_q8_K` in the forward pass. Output projection and token embedding are kept at `Q4_0` (not ternary in the source). > **Layout note.** `TQ2_0` and `Q2_K` `qs` streams share the *same* stride-32 / 128-span interleave. Matching `ggml`'s canonical order is the whole game: an encoding that is self-consistent with a hand-rolled unpacker but disagrees with the canonical order will pass every fidelity check against itself and still emit garbage in the forward pass, because `dequantize_row` and `vec_dot` both assume the canonical order. This build matches it. --- ## Usage ### CPU (use `TQ2_0`) ```bash llama-cli -m Bonsai-8B-TQ2_0.gguf -p "The capital of France is" -n 64 -t 6 ``` Fastest on CPU and smallest RAM footprint. ~12-14 tok/s on a low-end Zen2 desktop (Ryzen 3 PRO 4350GE, dual-channel DDR4). ### GPU offload (use `Q2_KT`) ```bash llama-cli -m Bonsai-8B-Q2_KT.gguf -ngl 99 -p "The capital of France is" -n 64 ``` Verified running fully offloaded on an AMD Radeon Vega iGPU via the Vulkan `RADV` backend. Because `Q2_K` has real GPU kernels, `-ngl 99` actually places the ternary body on the device — unlike `TQ2_0`, which would stay on the host. ### Any GGUF loader / llama.cpp server `Q2_KT` loads as an ordinary `Q2_K` model — no custom code, no patched runtime. Every existing loader already understands the container. --- ## Quality Perplexity on `wikitext-2-raw` (test split, 20 chunks of 512 tokens, `-t 4`, identical harness): | Model | Format | BPW | PPL (lower is better) | |---|---|---|---| | `Bonsai-8B-TQ2_0` | native ternary | 2.06 | 12.311 +/- 0.517 | | `Bonsai-8B-Q2_KT` | ternary-aware `Q2_K` | 2.911 | **12.299 +/- 0.516** | `Q2_KT` is **perplexity-neutral vs. the native ternary build** (delta within the +/-0.5 statistical error). You pay ~0.85 BPW over `TQ2_0` for universal backend support; you do **not** pay for it in quality — expected, since the encoding is lossless on the ternary levels. For task benchmarks (MMLU-Redux, GSM8K, HumanEval+, IFEval, ...), see [PrismML's numbers](https://prismml.com/news/ternary-bonsai) for the base model — these files change the container, not the learned weights. --- ## Technical notes & reproducibility - **`Q2_KT` container:** 252 ternary body tensors -> pure `Q2_K` (zero-exact); `output` + `token_embd` -> `Q4_0`; 145 norms -> `F32`. - **No tensor promotion.** Every ternary tensor is pure `Q2_K`; the zero-exact encoding makes the `Q3_K`/`Q4_K` promotion unnecessary — that is why `Q2_KT` is 2.91 BPW, not 3.18. - **Bit-exactness:** validated against `ggml`'s reference `dequantize_row_q2_K` (100 % sign accuracy, max abs error 1e-5 from the FP16 scale) and end-to-end against the forward `vec_dot` (coherent generation, PPL parity). - **Architecture:** Qwen3, 36 layers, `hidden 4096`, `FFN 12288`, GQA 32/8, full 65536-token context preserved in metadata. - **Reproduce PPL:** `llama-perplexity -m -f wiki.test.raw --chunks 20`. ### A note for the llama.cpp community The `+/-s/3` zero-error affects **any** end-to-end ternary model (Bonsai, BitNet, TriLM, ...) run through the stock `Q2_K` quantizer. A ternary-aware branch in `quantize_row_q2_K` — detect a block with <=3 symmetric levels, emit `d = dmin = s/15`, `sc = m`, `q = t+1` — would let the official pipeline produce zero-exact `Q2_K` for the whole class, without the promotion workaround. `Q2_KT` is a concrete demonstration that the encoding is exact and backend-portable. Contributions upstream welcome. --- ## Attribution & license - **Base model:** [Ternary-Bonsai-8B](https://huggingface.co/prism-ml/Ternary-Bonsai-8B) by **PrismML** (Babak Hassibi et al., Caltech). Ternary QAT, Qwen3 architecture, end-to-end 1.58-bit. - **These files:** GGUF re-encodes for `llama.cpp`. `TQ2_0` is the native ternary layout; `Q2_KT` is a lossless-on-ternary `Q2_K` re-encode for backend portability. Learned weights unchanged. - **License:** Apache 2.0, inherited from the base model. ```bibtex @techreport{ternarybonsai, title = {Ternary Bonsai: 1.58-bit Language Models at 8B, 4B, and 1.7B Scale}, author = {Prism ML}, year = {2026}, month = {April}, url = {https://prismml.com} } ``` --- *Community re-encodes, not affiliated with or endorsed by PrismML. Quality figures are measured on the specific files in this repository.*