Vveerrgg commited on
Commit
fc555b4
·
verified ·
1 Parent(s): 891b20f

Speed section (measured 2026-11-19), correction: Q4_K_M at 8k uses 5.84 GiB with the default cache, Ollama thinking note

Browse files
Files changed (1) hide show
  1. README.md +52 -1
README.md CHANGED
@@ -31,7 +31,7 @@ app look it up) and it will read it.
31
  | File | Size | Use |
32
  |---|---|---|
33
  | `qwen3-8b-cooking-assistant-v1-Q6_K.gguf` | 6.7 GB | 8 GB cards; matches full precision on every check below |
34
- | `qwen3-8b-cooking-assistant-v1-Q4_K_M.gguf` **(default)** | 5.0 GB | 6 GB cards (5.3 GB used at 8k context); within noise of full precision |
35
  | `qwen3-8b-cooking-assistant-v1-Q8_0.gguf` | 8.7 GB | near-lossless |
36
  | `qwen3-8b-cooking-assistant-v1-BF16.gguf` | 16.4 GB | full precision, for your own quantisation |
37
  | `qwen3-8b-cooking-assistant-v1.imatrix.gguf` | 11 MB | importance matrix used for Q4_K_M and Q6_K |
@@ -51,6 +51,13 @@ ollama run vveerrgg/qwen3-8b-cooking-assistant:q6_k
51
  # or: ollama run hf.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q6_K (template, system prompt and settings ship in this repo)
52
  ```
53
 
 
 
 
 
 
 
 
54
  ## Versions
55
 
56
  The model is one product with dated versions. Each version is a git tag on this repo, so you can pin one with
@@ -60,6 +67,11 @@ The model is one product with dated versions. Each version is a git tag on this
60
  |---|---|---|---|---|
61
  | **v1** (current) | 2026-11-12 | 2.7% (base 25.8%) | 94.8% (base 83.1%) | first public release |
62
 
 
 
 
 
 
63
  **Corrections (2026-11-14):** the evaluation set covers 21 languages (this card said 26; the model was trained on 26); training counts removed from this card; Transformers example updated for transformers 5. No score changed. GGUF files re-issued: their metadata carried an internal build name and local build paths; the weights are bit-identical and only the header changed, so the checksums are new. Re-download, or `ollama pull` again, for the clean files.
64
 
65
  ## Prompt format
@@ -73,6 +85,45 @@ You are a cooking assistant. You answer questions about dishes, ingredients, tec
73
 
74
  To ask about a specific recipe, put the recipe text in the user turn, then the question.
75
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
76
  ## What it does
77
 
78
  The fine-tune teaches **behaviour, not facts**:
 
31
  | File | Size | Use |
32
  |---|---|---|
33
  | `qwen3-8b-cooking-assistant-v1-Q6_K.gguf` | 6.7 GB | 8 GB cards; matches full precision on every check below |
34
+ | `qwen3-8b-cooking-assistant-v1-Q4_K_M.gguf` **(default)** | 5.0 GB | 6 GB cards (8k context uses 5.84 GiB: tight with a display, see the correction below); within noise of full precision |
35
  | `qwen3-8b-cooking-assistant-v1-Q8_0.gguf` | 8.7 GB | near-lossless |
36
  | `qwen3-8b-cooking-assistant-v1-BF16.gguf` | 16.4 GB | full precision, for your own quantisation |
37
  | `qwen3-8b-cooking-assistant-v1.imatrix.gguf` | 11 MB | importance matrix used for Q4_K_M and Q6_K |
 
51
  # or: ollama run hf.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q6_K (template, system prompt and settings ship in this repo)
52
  ```
53
 
54
+ **Thinking in Ollama.** Some newer Ollama versions (seen on 0.40.16) use the chat template inside the GGUF file
55
+ instead of the one shipped with the model, and that template turns thinking on: the answer then starts with a
56
+ paragraph of reasoning. Ollama tags updated 2026-11-19 with the fix: `latest`, `q4_k_m`, `q6_k` and `q8_0` of
57
+ `vveerrgg/qwen3-8b-cooking-assistant`; pull them again. The pinned `v1` tags stay as first published. With a pinned
58
+ tag, an older pull or the `hf.co/` path, turn thinking off per request: `"think": false` (Ollama API),
59
+ `"reasoning_effort": "none"` (OpenAI-compatible `/v1`), `ollama run ... --think=false`, or `/set nothink` in a chat.
60
+
61
  ## Versions
62
 
63
  The model is one product with dated versions. Each version is a git tag on this repo, so you can pin one with
 
67
  |---|---|---|---|---|
68
  | **v1** (current) | 2026-11-12 | 2.7% (base 25.8%) | 94.8% (base 83.1%) | first public release |
69
 
70
+ **Correction (2026-11-19):** this card said Q4_K_M uses 5.3 GB at 8k context. That figure was measured with an
71
+ 8-bit (q8_0) KV cache. With the default cache, as in the quick start, it uses 5.84 GiB of GPU memory in llama-server
72
+ and 5.96 GiB in Ollama. On a 6 GB card that also drives a display, use a 4,096 context (about 5.3 GiB), the q8_0
73
+ cache (lossy, 5.3 GiB at 8k), or let Ollama put a few layers on the CPU. No quality number changed.
74
+
75
  **Corrections (2026-11-14):** the evaluation set covers 21 languages (this card said 26; the model was trained on 26); training counts removed from this card; Transformers example updated for transformers 5. No score changed. GGUF files re-issued: their metadata carried an internal build name and local build paths; the weights are bit-identical and only the header changed, so the checksums are new. Re-download, or `ollama pull` again, for the clean files.
76
 
77
  ## Prompt format
 
85
 
86
  To ask about a specific recipe, put the recipe text in the user turn, then the question.
87
 
88
+ ## Speed
89
+
90
+ Measured 2026-11-19 on one NVIDIA RTX PRO 4000 Blackwell SFF (24 GB, 70 W), Q4_K_M, 8k context, one user, thinking
91
+ off, short chats. [llama.cpp](https://github.com/ggml-org/llama.cpp) commit `08c252e2`; Ollama 0.40.16.
92
+
93
+ | Setup | GPU memory | Answer (tokens/s) | Prompt, ~1k / ~8k tokens (tokens/s) | First token, short prompt | Start-up |
94
+ |---|---|---|---|---|---|
95
+ | Ollama | 5.96 GiB | 65.6 | 2,566 / 2,360 | 28.5 ms | 10.8 s |
96
+ | llama-server `-c 8192 -np 1` | 5.84 GiB | 65.1 | 2,556 / 2,317 | 20.3 ms | 3.1 s |
97
+ | llama-server + Qwen3-0.6B draft model (8 GB cards) | 7.13 GiB | 68.7 | 2,059 / 1,887 | 26.3 ms | 4.1 s |
98
+ | **Lossy:** llama-server, 32k context, q8_0 KV cache | 7.20 GiB | 63.2 (one run) | 31k tokens: 1,635 | 22.3 ms | 2.1 s |
99
+
100
+ - Ollama and llama-server answer at the same speed (they run the same kernels). llama-server starts faster and
101
+ shows the first token sooner. Ollama's first start on a new install also compiles GPU kernels (here 45 s to start
102
+ plus 39 s for the first answer).
103
+ - On an 8 GB card, a Qwen3-0.6B draft model (speculative decoding, one token drafted per step) gives about 69
104
+ tokens/s: about 4% to 9% faster than the same server without it (+3.8% to +9.3% over five paired runs; +5.6% in
105
+ the final one), while prompts are read about 20% slower. Speculative decoding is meant to keep the model's output;
106
+ in greedy tests 4 of 10 answers differed slightly, each where the top two words were close (probability gap 0.02 to 0.17).
107
+ - 32k context with the default cache needs about 9.3 GiB. The 32k row fits 8 GiB by storing the KV cache in 8 bits,
108
+ which is lossy; its answer quality was not evaluated.
109
+
110
+ **Other hardware (calculated, not measured).** Answer speed is limited by memory bandwidth: each token reads about
111
+ 4.76 GB (the Q4_K_M weights plus a short chat's KV cache). Ceiling = bandwidth ÷ 4.76 GB; the estimate is 0.75 of
112
+ that, the share reached on the card above.
113
+
114
+ | Hardware | Bandwidth (GB/s) | Ceiling (tokens/s) | Estimate (tokens/s) |
115
+ |---|---|---|---|
116
+ | RTX 3050 6GB | 168 | 35 | 26 |
117
+ | RTX 3060 12GB | 360 | 76 | 57 |
118
+ | RTX 4060 8GB | 272 | 57 | 43 |
119
+ | RTX 4060 Ti 8GB | 288 | 61 | 45 |
120
+ | RTX 5060 8GB | 448 | 94 | 71 |
121
+ | MacBook Air (M4) | 120 | 25 | 19 |
122
+ | MacBook Pro (M4 Pro) | 273 | 57 | 43 |
123
+
124
+ A 6 GB card that also drives a display will not hold the 8k setup fully (see the correction above). Apple numbers use
125
+ the same fraction; llama.cpp's Metal backend was not measured.
126
+
127
  ## What it does
128
 
129
  The fine-tune teaches **behaviour, not facts**: