Add one-command run, charts and cross-model speeds

README.md

@@ -7,6 +7,7 @@ tags:
   - draft-model
   - bonsai
   - llama.cpp
+pipeline_tag: text-generation
 ---
 
 # Common Flash v0.1
@@ -18,6 +19,24 @@ A small model proposes tokens and Bonsai checks them in one pass, so the output
 (1,000 completions of at most 192 tokens; rank 32, all linear layers), vocabulary trimmed to Bonsai's 151,669 tokens
 (llama.cpp needs the draft vocabulary within 128 tokens of the target's), then quantized to Q4_0.
 
+## Run it in one command
+
+Needs a recent llama.cpp (`llama-server`, build 1268eda or newer; see Quick start below). This downloads Bonsai 8B and this drafter, then starts a speculative server with a chat page at http://127.0.0.1:8080:
+
+```bash
+llama-server \
+  -hf  prism-ml/Ternary-Bonsai-8B-gguf:Q2_0_g64 \
+  -hfd TheCommonAI/common-flash-v0.1 \
+  -ngl 99 -ngld 99 -fa on -np 1 -c 4096 \
+  --spec-type draft-simple --spec-draft-n-max 3
+```
+
+`-hf` fetches the target model, `-hfd` fetches the drafter from this repo. Tested from an empty cache; the first start downloads about 2.7 GB.
+
+> **About the "Use this model" button.** Hugging Face generates that snippet itself, and for this repo it would start the drafter alone, which is not useful. Use the command above instead.
+
+![Decode speed by model](assets/speed_by_model.png)
+
 ## Target model (required pairing)
 
 This drafter only works with **`Ternary-Bonsai-8B-Q2_0_g64.gguf`** from
@@ -126,6 +145,28 @@ Speed relative to plain decoding:
 Draft acceptance is about 65% overall but only 33-45% on chat. Run-to-run ratios varied between 1.19x and 1.41x on a busy machine, so treat these as approximate.
 With greedy decoding every speculative output matched plain decoding. With sampling the random draws differ, so that check does not apply.
 
+
+### Charts
+
+![Speedup by workload](assets/speedup_by_workload.png)
+
+![Why the runtime matters](assets/verify_throughput.png)
+
+![What helped](assets/what_helped.png)
+
+### Decode speed on the same M3 (plain, llama.cpp 1268eda, `llama-bench` tg128)
+
+| Model | File size | tokens/s |
+| --- | ---: | ---: |
+| Ternary Bonsai 1.7B | 0.46 GB | 127 |
+| Ternary Bonsai 2 2B | 0.51 GB | 97 |
+| **Common Flash** (Bonsai 8B + drafter) | 2.3 GB + 0.4 GB | **~49** (36.3 x 1.36) |
+| Ternary Bonsai 2 4B | 1.1 GB | 48 |
+| Ternary Bonsai 8B, plain | 2.1 GB | 36 |
+| Ternary Bonsai 2 9B | 2.4 GB | 28 |
+
+The Common Flash row is derived: plain Bonsai 8B speed times the measured 1.36x ratio, not a separate `llama-bench` run. Smaller models are faster but are different, smaller models; the comparison that matters for quality is Bonsai 8B plain against Bonsai 8B with the drafter, which produce the same text.
+
 ## Limits
 
 - Tested on one machine. Older llama.cpp builds or other hardware may not see a speedup.