Instructions to use TheCommonAI/common-flash-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use TheCommonAI/common-flash-v0.1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheCommonAI/common-flash-v0.1:Q4_0
# Run inference directly in the terminal:
llama cli -hf TheCommonAI/common-flash-v0.1:Q4_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheCommonAI/common-flash-v0.1:Q4_0
# Run inference directly in the terminal:
llama cli -hf TheCommonAI/common-flash-v0.1:Q4_0

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf TheCommonAI/common-flash-v0.1:Q4_0
# Run inference directly in the terminal:
./build/bin/llama cli -hf TheCommonAI/common-flash-v0.1:Q4_0

Use Docker

docker model run hf.co/TheCommonAI/common-flash-v0.1:Q4_0

Llama new
LM Studio
Jan

vLLM

How to use TheCommonAI/common-flash-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheCommonAI/common-flash-v0.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheCommonAI/common-flash-v0.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/TheCommonAI/common-flash-v0.1:Q4_0

Ollama
How to use TheCommonAI/common-flash-v0.1 with Ollama:
```
ollama run hf.co/TheCommonAI/common-flash-v0.1:Q4_0
```
Unsloth Desktop

How to use TheCommonAI/common-flash-v0.1 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheCommonAI/common-flash-v0.1:Q4_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "TheCommonAI/common-flash-v0.1:Q4_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use TheCommonAI/common-flash-v0.1 with Docker Model Runner:
```
docker model run hf.co/TheCommonAI/common-flash-v0.1:Q4_0
```

Lemonade

How to use TheCommonAI/common-flash-v0.1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheCommonAI/common-flash-v0.1:Q4_0

Run and chat with the model

lemonade run user.common-flash-v0.1-Q4_0

List all available models

lemonade list

Hermes Agent

How to use TheCommonAI/common-flash-v0.1 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheCommonAI/common-flash-v0.1:Q4_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default TheCommonAI/common-flash-v0.1:Q4_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use TheCommonAI/common-flash-v0.1 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheCommonAI/common-flash-v0.1:Q4_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "TheCommonAI/common-flash-v0.1:Q4_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

common-flash-v0.1 / README.md

techguy1423

Add one-command run, charts and cross-model speeds

4f49b3d verified 21 days ago

preview code

Raw History Blame Contribute Delete

8.49 kB

---
license: apache-2.0
base_model: Qwen/Qwen3-0.6B
tags:
  - gguf
  - speculative-decoding
  - draft-model
  - bonsai
  - llama.cpp
pipeline_tag: text-generation
---
# Common Flash v0.1

A drafter for **speculative decoding of Ternary Bonsai 8B (1st gen, Q2_0_g64)** in llama.cpp.
A small model proposes tokens and Bonsai checks them in one pass, so the output is unchanged and decoding is faster.

`Qwen3-0.6B-bonsai-lora-Q4_0.gguf` (358 MiB) is Qwen3-0.6B, LoRA-tuned on Bonsai 8B's own completions
(1,000 completions of at most 192 tokens; rank 32, all linear layers), vocabulary trimmed to Bonsai's 151,669 tokens
(llama.cpp needs the draft vocabulary within 128 tokens of the target's), then quantized to Q4_0.
## Run it in one command
Needs a recent llama.cpp (`llama-server`, build 1268eda or newer; see Quick start below). This downloads Bonsai 8B and this drafter, then starts a speculative server with a chat page at http://127.0.0.1:8080:
```bash
llama-server \
  -hf  prism-ml/Ternary-Bonsai-8B-gguf:Q2_0_g64 \
  -hfd TheCommonAI/common-flash-v0.1 \
  -ngl 99 -ngld 99 -fa on -np 1 -c 4096 \
  --spec-type draft-simple --spec-draft-n-max 3
```
`-hf` fetches the target model, `-hfd` fetches the drafter from this repo. Tested from an empty cache; the first start downloads about 2.7 GB.
> **About the "Use this model" button.** Hugging Face generates that snippet itself, and for this repo it would start the drafter alone, which is not useful. Use the command above instead.
![Decode speed by model](assets/speed_by_model.png)
## Target model (required pairing)
This drafter only works with **`Ternary-Bonsai-8B-Q2_0_g64.gguf`** from
[prism-ml/Ternary-Bonsai-8B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf)
(2,310,125,920 bytes, sha256 starting `e17b298d84ee`). That is first-generation Ternary Bonsai 8B.
It was tuned on that model's outputs and trimmed to its vocabulary; it is not tested with other Bonsai models
(Bonsai 2, the 27B, or the 1.7B), and mismatched vocabularies are rejected by llama.cpp.
## Quick start
You need three things: a recent llama.cpp, the Bonsai 8B model, and this drafter.
**1. Build or install llama.cpp (recent upstream).** Tested with upstream commit `1268eda`. An older Homebrew build (b11429) was *slower* with speculation than without, so use a current build.
```bash
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build            # Apple Silicon: Metal is on by default
# NVIDIA GPU:  cmake -B build -DGGML_CUDA=ON      (Vulkan: -DGGML_VULKAN=ON)
cmake --build build --config Release -j
# binaries land in build/bin/ (build/bin/Release/ on Windows)
```
**2. Download both models.**
```bash
pip install -U "huggingface_hub[cli]"
hf download prism-ml/Ternary-Bonsai-8B-gguf Ternary-Bonsai-8B-Q2_0_g64.gguf --local-dir models
hf download TheCommonAI/common-flash-v0.1 Qwen3-0.6B-bonsai-lora-Q4_0.gguf --local-dir models
```
Disk: about 2.3 GB for Bonsai plus 0.4 GB for the drafter. Memory: roughly 3 GB plus the context cache.
**3. Run the server.**
```bash
./build/bin/llama-server \
  -m  models/Ternary-Bonsai-8B-Q2_0_g64.gguf \
  -md models/Qwen3-0.6B-bonsai-lora-Q4_0.gguf \
  -ngl 99 -ngld 99 -fa on -t 6 -np 1 -c 4096 \
  --spec-type draft-simple --spec-draft-n-max 3 \
  --host 127.0.0.1 --port 8080
```
| Flag | Meaning |
| --- | --- |
| `-m` / `-md` | Target model (Bonsai) / draft model (this repo) |
| `-ngl 99` / `-ngld 99` | Put all layers of the target / drafter on the GPU. Use `0` for CPU-only (untested, likely slower) |
| `-fa on` | Flash attention |
| `-t 6` | CPU threads; set to your number of performance cores |
| `-np 1` | One request slot. Speculation helps single-stream speed; more slots share the GPU |
| `-c 4096` | Context size in tokens. Lower it (for example `-c 2048`) if you run out of memory |
| `--spec-type draft-simple` | Classic draft-model speculative decoding |
| `--spec-draft-n-max 3` | Draft 3 tokens per round. 3 was best in our tests; 8 was close; 2 and 5 were no better |
**4. Try it.** Open http://127.0.0.1:8080 for the built-in chat page, or:
```bash
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "Explain speculative decoding in two sentences."}],
  "max_tokens": 128, "temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5,
  "chat_template_kwargs": {"enable_thinking": false}
}'
```
**5. Check that speculation is working.** The server's own `/completion` endpoint reports it:
```bash
curl -s localhost:8080/completion -d '{"prompt":"<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n","n_predict":64}' \
  | python3 -c "import json,sys; t=json.load(sys.stdin)['timings']; print(t['predicted_per_second'],'tok/s', t.get('draft_n_accepted'),'/',t.get('draft_n'),'drafts accepted')"
```
Acceptance of roughly 60-70% on code and copy-heavy text, and 35-45% on open chat, is normal. If `draft_n` is missing or 0, the drafter did not load.
**To compare against plain decoding**, start a second server on another port without `-md`, `-ngld` and `--spec-type`, and send both the same prompts. Run them one after the other on a quiet machine: absolute speeds on a busy machine are noisy, and only ratios are meaningful. Speculation cannot be switched off per request in this build, so restart the server to disable it.
## Troubleshooting
- **"draft model vocab must match" / vocab size error**: you are not using the exact Bonsai 8B 1st-generation Q2_0_g64 file named above.
- **Slower than without the drafter**: use a current upstream llama.cpp build; try `--spec-draft-n-max 2`; make sure nothing else is using the GPU.
- **Out of memory**: lower `-c`, and run one server at a time.
- **Output differs from plain decoding**: with temperature 0 the text is identical. With sampling the random draws differ, which is expected.
## Platform status
| Platform | Status |
| --- | --- |
| Apple Silicon (Metal) | Measured, about 1.36x (one M3, 16 GB) |
| NVIDIA CUDA (Linux/Windows) | Bonsai Q2_0 runs on CUDA; speculation speedup not yet measured |
| Vulkan, CPU-only | Untested |
If you try another platform, please report your plain versus speculative tokens per second.
## Results
One Apple M3 (16 GB), Metal, 36 prompts the drafter never trained on, 128 tokens each, the network worker's sampling
(temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5), configurations interleaved prompt by prompt.
Speed relative to plain decoding:
| | speedup |
| --- | ---: |
| overall | ~1.36x |
| chat | ~1.14x |
| code | ~1.31x |
| copy-heavy edits | ~1.67x |
Draft acceptance is about 65% overall but only 33-45% on chat. Run-to-run ratios varied between 1.19x and 1.41x on a busy machine, so treat these as approximate.
With greedy decoding every speculative output matched plain decoding. With sampling the random draws differ, so that check does not apply.
### Charts
![Speedup by workload](assets/speedup_by_workload.png)
![Why the runtime matters](assets/verify_throughput.png)
![What helped](assets/what_helped.png)
### Decode speed on the same M3 (plain, llama.cpp 1268eda, `llama-bench` tg128)
| Model | File size | tokens/s |
| --- | ---: | ---: |
| Ternary Bonsai 1.7B | 0.46 GB | 127 |
| Ternary Bonsai 2 2B | 0.51 GB | 97 |
| **Common Flash** (Bonsai 8B + drafter) | 2.3 GB + 0.4 GB | **~49** (36.3 x 1.36) |
| Ternary Bonsai 2 4B | 1.1 GB | 48 |
| Ternary Bonsai 8B, plain | 2.1 GB | 36 |
| Ternary Bonsai 2 9B | 2.4 GB | 28 |
The Common Flash row is derived: plain Bonsai 8B speed times the measured 1.36x ratio, not a separate `llama-bench` run. Smaller models are faster but are different, smaller models; the comparison that matters for quality is Bonsai 8B plain against Bonsai 8B with the drafter, which produce the same text.
## Limits
- Tested on one machine. Older llama.cpp builds or other hardware may not see a speedup.
- The test prompts come from Alpaca-cleaned, CodeAlpaca and HumanEval-style prompts, not real traffic.
- More training data, soft-label distillation, tighter quantization, MLP pruning and tuned draft lengths did not improve it; acceptance appears limited by the 0.6B drafter's size.
## Licence
Apache-2.0, inherited from Qwen3-0.6B. Training inputs were prompts from Alpaca-cleaned and CodeAlpaca-20k with Bonsai 8B's own completions as targets; check those datasets' terms and Prism ML's Bonsai terms before relying on this commercially.

	---
	license: apache-2.0
	base_model: Qwen/Qwen3-0.6B
	tags:
	- gguf
	- speculative-decoding
	- draft-model
	- bonsai
	- llama.cpp
	pipeline_tag: text-generation
	---

	# Common Flash v0.1

	A drafter for speculative decoding of Ternary Bonsai 8B (1st gen, Q2_0_g64) in llama.cpp.
	A small model proposes tokens and Bonsai checks them in one pass, so the output is unchanged and decoding is faster.

	`Qwen3-0.6B-bonsai-lora-Q4_0.gguf` (358 MiB) is Qwen3-0.6B, LoRA-tuned on Bonsai 8B's own completions
	(1,000 completions of at most 192 tokens; rank 32, all linear layers), vocabulary trimmed to Bonsai's 151,669 tokens
	(llama.cpp needs the draft vocabulary within 128 tokens of the target's), then quantized to Q4_0.

	## Run it in one command

	Needs a recent llama.cpp (`llama-server`, build 1268eda or newer; see Quick start below). This downloads Bonsai 8B and this drafter, then starts a speculative server with a chat page at http://127.0.0.1:8080:

	```bash
	llama-server \
	-hf prism-ml/Ternary-Bonsai-8B-gguf:Q2_0_g64 \
	-hfd TheCommonAI/common-flash-v0.1 \
	-ngl 99 -ngld 99 -fa on -np 1 -c 4096 \
	--spec-type draft-simple --spec-draft-n-max 3
	```

	`-hf` fetches the target model, `-hfd` fetches the drafter from this repo. Tested from an empty cache; the first start downloads about 2.7 GB.

	> About the "Use this model" button. Hugging Face generates that snippet itself, and for this repo it would start the drafter alone, which is not useful. Use the command above instead.

	![Decode speed by model](assets/speed_by_model.png)

	## Target model (required pairing)

	This drafter only works with `Ternary-Bonsai-8B-Q2_0_g64.gguf` from
	[prism-ml/Ternary-Bonsai-8B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf)
	(2,310,125,920 bytes, sha256 starting `e17b298d84ee`). That is first-generation Ternary Bonsai 8B.
	It was tuned on that model's outputs and trimmed to its vocabulary; it is not tested with other Bonsai models
	(Bonsai 2, the 27B, or the 1.7B), and mismatched vocabularies are rejected by llama.cpp.

	## Quick start

	You need three things: a recent llama.cpp, the Bonsai 8B model, and this drafter.

	1. Build or install llama.cpp (recent upstream). Tested with upstream commit `1268eda`. An older Homebrew build (b11429) was slower with speculation than without, so use a current build.

	```bash
	git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
	cmake -B build # Apple Silicon: Metal is on by default
	# NVIDIA GPU: cmake -B build -DGGML_CUDA=ON (Vulkan: -DGGML_VULKAN=ON)
	cmake --build build --config Release -j
	# binaries land in build/bin/ (build/bin/Release/ on Windows)
	```

	2. Download both models.

	```bash
	pip install -U "huggingface_hub[cli]"
	hf download prism-ml/Ternary-Bonsai-8B-gguf Ternary-Bonsai-8B-Q2_0_g64.gguf --local-dir models
	hf download TheCommonAI/common-flash-v0.1 Qwen3-0.6B-bonsai-lora-Q4_0.gguf --local-dir models
	```

	Disk: about 2.3 GB for Bonsai plus 0.4 GB for the drafter. Memory: roughly 3 GB plus the context cache.

	3. Run the server.

	```bash
	./build/bin/llama-server \
	-m models/Ternary-Bonsai-8B-Q2_0_g64.gguf \
	-md models/Qwen3-0.6B-bonsai-lora-Q4_0.gguf \
	-ngl 99 -ngld 99 -fa on -t 6 -np 1 -c 4096 \
	--spec-type draft-simple --spec-draft-n-max 3 \
	--host 127.0.0.1 --port 8080
	```

	\| Flag \| Meaning \|
	\| --- \| --- \|
	\| `-m` / `-md` \| Target model (Bonsai) / draft model (this repo) \|
	\| `-ngl 99` / `-ngld 99` \| Put all layers of the target / drafter on the GPU. Use `0` for CPU-only (untested, likely slower) \|
	\| `-fa on` \| Flash attention \|
	\| `-t 6` \| CPU threads; set to your number of performance cores \|
	\| `-np 1` \| One request slot. Speculation helps single-stream speed; more slots share the GPU \|
	\| `-c 4096` \| Context size in tokens. Lower it (for example `-c 2048`) if you run out of memory \|
	\| `--spec-type draft-simple` \| Classic draft-model speculative decoding \|
	\| `--spec-draft-n-max 3` \| Draft 3 tokens per round. 3 was best in our tests; 8 was close; 2 and 5 were no better \|

	4. Try it. Open http://127.0.0.1:8080 for the built-in chat page, or:

	```bash
	curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
	"messages": [{"role": "user", "content": "Explain speculative decoding in two sentences."}],
	"max_tokens": 128, "temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5,
	"chat_template_kwargs": {"enable_thinking": false}
	}'
	```

	5. Check that speculation is working. The server's own `/completion` endpoint reports it:

	```bash
	curl -s localhost:8080/completion -d '{"prompt":"<\|im_start\|>user\nHello<\|im_end\|>\n<\|im_start\|>assistant\n<think>\n\n</think>\n\n","n_predict":64}' \
	\| python3 -c "import json,sys; t=json.load(sys.stdin)['timings']; print(t['predicted_per_second'],'tok/s', t.get('draft_n_accepted'),'/',t.get('draft_n'),'drafts accepted')"
	```

	Acceptance of roughly 60-70% on code and copy-heavy text, and 35-45% on open chat, is normal. If `draft_n` is missing or 0, the drafter did not load.

	To compare against plain decoding, start a second server on another port without `-md`, `-ngld` and `--spec-type`, and send both the same prompts. Run them one after the other on a quiet machine: absolute speeds on a busy machine are noisy, and only ratios are meaningful. Speculation cannot be switched off per request in this build, so restart the server to disable it.

	## Troubleshooting

	- "draft model vocab must match" / vocab size error: you are not using the exact Bonsai 8B 1st-generation Q2_0_g64 file named above.
	- Slower than without the drafter: use a current upstream llama.cpp build; try `--spec-draft-n-max 2`; make sure nothing else is using the GPU.
	- Out of memory: lower `-c`, and run one server at a time.
	- Output differs from plain decoding: with temperature 0 the text is identical. With sampling the random draws differ, which is expected.

	## Platform status

	\| Platform \| Status \|
	\| --- \| --- \|
	\| Apple Silicon (Metal) \| Measured, about 1.36x (one M3, 16 GB) \|
	\| NVIDIA CUDA (Linux/Windows) \| Bonsai Q2_0 runs on CUDA; speculation speedup not yet measured \|
	\| Vulkan, CPU-only \| Untested \|

	If you try another platform, please report your plain versus speculative tokens per second.

	## Results

	One Apple M3 (16 GB), Metal, 36 prompts the drafter never trained on, 128 tokens each, the network worker's sampling
	(temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5), configurations interleaved prompt by prompt.
	Speed relative to plain decoding:

	\| \| speedup \|
	\| --- \| ---: \|
	\| overall \| ~1.36x \|
	\| chat \| ~1.14x \|
	\| code \| ~1.31x \|
	\| copy-heavy edits \| ~1.67x \|

	Draft acceptance is about 65% overall but only 33-45% on chat. Run-to-run ratios varied between 1.19x and 1.41x on a busy machine, so treat these as approximate.
	With greedy decoding every speculative output matched plain decoding. With sampling the random draws differ, so that check does not apply.


	### Charts

	![Speedup by workload](assets/speedup_by_workload.png)

	![Why the runtime matters](assets/verify_throughput.png)

	![What helped](assets/what_helped.png)

	### Decode speed on the same M3 (plain, llama.cpp 1268eda, `llama-bench` tg128)

	\| Model \| File size \| tokens/s \|
	\| --- \| ---: \| ---: \|
	\| Ternary Bonsai 1.7B \| 0.46 GB \| 127 \|
	\| Ternary Bonsai 2 2B \| 0.51 GB \| 97 \|
	\| Common Flash (Bonsai 8B + drafter) \| 2.3 GB + 0.4 GB \| ~49 (36.3 x 1.36) \|
	\| Ternary Bonsai 2 4B \| 1.1 GB \| 48 \|
	\| Ternary Bonsai 8B, plain \| 2.1 GB \| 36 \|
	\| Ternary Bonsai 2 9B \| 2.4 GB \| 28 \|

	The Common Flash row is derived: plain Bonsai 8B speed times the measured 1.36x ratio, not a separate `llama-bench` run. Smaller models are faster but are different, smaller models; the comparison that matters for quality is Bonsai 8B plain against Bonsai 8B with the drafter, which produce the same text.

	## Limits

	- Tested on one machine. Older llama.cpp builds or other hardware may not see a speedup.
	- The test prompts come from Alpaca-cleaned, CodeAlpaca and HumanEval-style prompts, not real traffic.
	- More training data, soft-label distillation, tighter quantization, MLP pruning and tuned draft lengths did not improve it; acceptance appears limited by the 0.6B drafter's size.

	## Licence

	Apache-2.0, inherited from Qwen3-0.6B. Training inputs were prompts from Alpaca-cleaned and CodeAlpaca-20k with Bonsai 8B's own completions as targets; check those datasets' terms and Prism ML's Bonsai terms before relying on this commercially.