infosave/Bonsai-1.7Bcmf

Bonsai-1.7B — CMF q1 (334 MB, made for phones)

Bonsai-1.7B — a 1-bit-trained 1.7B qwen3 (full attention) — packed into a single CMF file with q1 encoding: 1.5 bits per weight, 6 bytes per 32 weights.

Because the model was trained at 1 bit, q1 is its native representation, not a lossy afterthought: generation is token-for-token identical to the same model stored at q8 — in a file 334 MB small. That size is the point: it fits comfortably in the RAM of a 4 GB phone or tablet, memory-maps straight off storage (start-up is instant, resident memory ≈ file size), and decodes at 80–87 tok/s on an Apple M4 through a whole-token Metal graph.

If you want the big sibling: Bonsai-27B in the same format (4.75 GB — needs a 16–24 GB machine; it does not fit 4 GB devices).

What is CMF?

CMF (Cortiq Model Format) is a single-file LLM container with a small pure-Rust runtime — no Python, no torch, no C++ toolchain, no CUDA install:

• One file carries the weights, tokenizer and chat template, and checks its own integrity.
• mmap-first: pages load lazily on first touch; start-up is fast and RAM stays near the file size.
• Per-tensor quantization (q8 / q4 / vbit / q1 for 1-bit-trained models).
• O(1) attention option (--o1): convert attention to a constant-memory streaming operator — no retraining, weights byte-identical.
• Skills: one file can carry a swarm of specialists sharing a base model, with self-routing. See the skills guide.

Format spec and engine: https://github.com/infosave2007/cmf

Install

Prebuilt binaries (macOS arm64/x86_64, Linux, Windows, Android): https://github.com/infosave2007/cmf/releases/latest

Or with a Rust toolchain:

cargo install cortiq-cli   # needs >= 0.3.11

Download & run

# download the model file (~334 MB)
huggingface-cli download infosave/Bonsai-1.7Bcmf bonsai-1.7b-q1.cmf --local-dir .

# chat (the file carries its own chat template)
cortiq run bonsai-1.7b-q1.cmf --prompt "Explain mmap in one sentence."

# Apple Silicon: whole-token Metal graph (80-87 tok/s on an M4)
CMF_GPU=1 cortiq run bonsai-1.7b-q1.cmf --prompt "Explain mmap in one sentence."

# raw completion mode (no chat template)
cortiq run bonsai-1.7b-q1.cmf --prompt "The capital of France is" --raw --greedy

Run on Android

Three ways:

Ready-made app — cmfmobile, a Flutter app built on this runtime: install, point it at a .cmf file (this 334 MB one is its natural fit), chat with streaming tokens on-device.

Termux / adb shell — grab cortiq-aarch64-linux-android.tar.gz from the releases, put the .cmf on storage, run the same commands as above.

Inside your own app — link cortiq-ffi (prebuilt .so for arm64-v8a / armeabi-v7a / x86_64 is attached to every release, plus an iOS static library). The C ABI is eight functions: load, chat (single- and multi-turn with the file's own chat template), streaming token callback, sampler options as JSON. The engine picks its cores from the kernel's capacity table (big cores on big.LITTLE, all cores on clock-binned parts) and the arm64 build can opt into Vulkan (Mali/Adreno) with CMF_GPU=1 — a runtime probe keeps whichever side actually wins on your silicon.

Phone speed depends on the SoC class — the M4 numbers below do not transfer. Rough expectations: a Snapdragon 778G-class phone lands in the tens of tok/s; an all-A55 budget tablet in the single digits. Measure on the device.

Run as a server

cortiq serve speaks the OpenAI API, so existing clients and SDKs work unchanged:

CMF_GPU=1 cortiq serve bonsai-1.7b-q1.cmf --port 8080   # + web dashboard on /

curl localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "cmf",
  "messages": [{"role": "user", "content": "Explain mmap in one sentence."}]
}'

Streaming ("stream": true), /v1/models, /v1/completions and /healthz work too. Note the scope honestly: requests are serialized (one at a time per model) and there is no authentication — a local-first server, not a multi-tenant gateway.

Performance (Apple M4, 24 GB MacBook, cortiq 0.3.11)

Mode	Speed
decode, CMF_GPU=1 (whole-token Metal graph)	80–87 tok/s
decode, CPU only	~75–79 tok/s
resident memory	≈ 334 MB (mmap)

The GPU decode is distribution-equivalent to the CPU path (reduction order differs — the usual GPU tolerance class).

Provenance

Converted from prism-ml/Bonsai-1.7B-unpacked (Apache-2.0) with:

cortiq convert --model prism-ml/Bonsai-1.7B-unpacked --quant q1 --output bonsai-1.7b-q1.cmf

q1 is intended for 1-bit-trained checkpoints (Bonsai / BitNet class). As post-training quantization of an ordinary model it destroys quality — the converter exposes it only as an explicit opt-in.