Instructions to use HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
Use Docker
docker model run hf.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
- Ollama
How to use HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF with Ollama:
ollama run hf.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF with Docker Model Runner:
docker model run hf.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
- Lemonade
How to use HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Qwen3-8B-Cooking-Assistant-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Speed section (measured 2026-11-19), correction: Q4_K_M at 8k uses 5.84 GiB with the default cache, Ollama thinking note
Browse files
README.md
CHANGED
|
@@ -31,7 +31,7 @@ app look it up) and it will read it.
|
|
| 31 |
| File | Size | Use |
|
| 32 |
|---|---|---|
|
| 33 |
| `qwen3-8b-cooking-assistant-v1-Q6_K.gguf` | 6.7 GB | 8 GB cards; matches full precision on every check below |
|
| 34 |
-
| `qwen3-8b-cooking-assistant-v1-Q4_K_M.gguf` **(default)** | 5.0 GB | 6 GB cards (5.
|
| 35 |
| `qwen3-8b-cooking-assistant-v1-Q8_0.gguf` | 8.7 GB | near-lossless |
|
| 36 |
| `qwen3-8b-cooking-assistant-v1-BF16.gguf` | 16.4 GB | full precision, for your own quantisation |
|
| 37 |
| `qwen3-8b-cooking-assistant-v1.imatrix.gguf` | 11 MB | importance matrix used for Q4_K_M and Q6_K |
|
|
@@ -51,6 +51,13 @@ ollama run vveerrgg/qwen3-8b-cooking-assistant:q6_k
|
|
| 51 |
# or: ollama run hf.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q6_K (template, system prompt and settings ship in this repo)
|
| 52 |
```
|
| 53 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
## Versions
|
| 55 |
|
| 56 |
The model is one product with dated versions. Each version is a git tag on this repo, so you can pin one with
|
|
@@ -60,6 +67,11 @@ The model is one product with dated versions. Each version is a git tag on this
|
|
| 60 |
|---|---|---|---|---|
|
| 61 |
| **v1** (current) | 2026-11-12 | 2.7% (base 25.8%) | 94.8% (base 83.1%) | first public release |
|
| 62 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
**Corrections (2026-11-14):** the evaluation set covers 21 languages (this card said 26; the model was trained on 26); training counts removed from this card; Transformers example updated for transformers 5. No score changed. GGUF files re-issued: their metadata carried an internal build name and local build paths; the weights are bit-identical and only the header changed, so the checksums are new. Re-download, or `ollama pull` again, for the clean files.
|
| 64 |
|
| 65 |
## Prompt format
|
|
@@ -73,6 +85,45 @@ You are a cooking assistant. You answer questions about dishes, ingredients, tec
|
|
| 73 |
|
| 74 |
To ask about a specific recipe, put the recipe text in the user turn, then the question.
|
| 75 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
## What it does
|
| 77 |
|
| 78 |
The fine-tune teaches **behaviour, not facts**:
|
|
|
|
| 31 |
| File | Size | Use |
|
| 32 |
|---|---|---|
|
| 33 |
| `qwen3-8b-cooking-assistant-v1-Q6_K.gguf` | 6.7 GB | 8 GB cards; matches full precision on every check below |
|
| 34 |
+
| `qwen3-8b-cooking-assistant-v1-Q4_K_M.gguf` **(default)** | 5.0 GB | 6 GB cards (8k context uses 5.84 GiB: tight with a display, see the correction below); within noise of full precision |
|
| 35 |
| `qwen3-8b-cooking-assistant-v1-Q8_0.gguf` | 8.7 GB | near-lossless |
|
| 36 |
| `qwen3-8b-cooking-assistant-v1-BF16.gguf` | 16.4 GB | full precision, for your own quantisation |
|
| 37 |
| `qwen3-8b-cooking-assistant-v1.imatrix.gguf` | 11 MB | importance matrix used for Q4_K_M and Q6_K |
|
|
|
|
| 51 |
# or: ollama run hf.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant-GGUF:Q6_K (template, system prompt and settings ship in this repo)
|
| 52 |
```
|
| 53 |
|
| 54 |
+
**Thinking in Ollama.** Some newer Ollama versions (seen on 0.40.16) use the chat template inside the GGUF file
|
| 55 |
+
instead of the one shipped with the model, and that template turns thinking on: the answer then starts with a
|
| 56 |
+
paragraph of reasoning. Ollama tags updated 2026-11-19 with the fix: `latest`, `q4_k_m`, `q6_k` and `q8_0` of
|
| 57 |
+
`vveerrgg/qwen3-8b-cooking-assistant`; pull them again. The pinned `v1` tags stay as first published. With a pinned
|
| 58 |
+
tag, an older pull or the `hf.co/` path, turn thinking off per request: `"think": false` (Ollama API),
|
| 59 |
+
`"reasoning_effort": "none"` (OpenAI-compatible `/v1`), `ollama run ... --think=false`, or `/set nothink` in a chat.
|
| 60 |
+
|
| 61 |
## Versions
|
| 62 |
|
| 63 |
The model is one product with dated versions. Each version is a git tag on this repo, so you can pin one with
|
|
|
|
| 67 |
|---|---|---|---|---|
|
| 68 |
| **v1** (current) | 2026-11-12 | 2.7% (base 25.8%) | 94.8% (base 83.1%) | first public release |
|
| 69 |
|
| 70 |
+
**Correction (2026-11-19):** this card said Q4_K_M uses 5.3 GB at 8k context. That figure was measured with an
|
| 71 |
+
8-bit (q8_0) KV cache. With the default cache, as in the quick start, it uses 5.84 GiB of GPU memory in llama-server
|
| 72 |
+
and 5.96 GiB in Ollama. On a 6 GB card that also drives a display, use a 4,096 context (about 5.3 GiB), the q8_0
|
| 73 |
+
cache (lossy, 5.3 GiB at 8k), or let Ollama put a few layers on the CPU. No quality number changed.
|
| 74 |
+
|
| 75 |
**Corrections (2026-11-14):** the evaluation set covers 21 languages (this card said 26; the model was trained on 26); training counts removed from this card; Transformers example updated for transformers 5. No score changed. GGUF files re-issued: their metadata carried an internal build name and local build paths; the weights are bit-identical and only the header changed, so the checksums are new. Re-download, or `ollama pull` again, for the clean files.
|
| 76 |
|
| 77 |
## Prompt format
|
|
|
|
| 85 |
|
| 86 |
To ask about a specific recipe, put the recipe text in the user turn, then the question.
|
| 87 |
|
| 88 |
+
## Speed
|
| 89 |
+
|
| 90 |
+
Measured 2026-11-19 on one NVIDIA RTX PRO 4000 Blackwell SFF (24 GB, 70 W), Q4_K_M, 8k context, one user, thinking
|
| 91 |
+
off, short chats. [llama.cpp](https://github.com/ggml-org/llama.cpp) commit `08c252e2`; Ollama 0.40.16.
|
| 92 |
+
|
| 93 |
+
| Setup | GPU memory | Answer (tokens/s) | Prompt, ~1k / ~8k tokens (tokens/s) | First token, short prompt | Start-up |
|
| 94 |
+
|---|---|---|---|---|---|
|
| 95 |
+
| Ollama | 5.96 GiB | 65.6 | 2,566 / 2,360 | 28.5 ms | 10.8 s |
|
| 96 |
+
| llama-server `-c 8192 -np 1` | 5.84 GiB | 65.1 | 2,556 / 2,317 | 20.3 ms | 3.1 s |
|
| 97 |
+
| llama-server + Qwen3-0.6B draft model (8 GB cards) | 7.13 GiB | 68.7 | 2,059 / 1,887 | 26.3 ms | 4.1 s |
|
| 98 |
+
| **Lossy:** llama-server, 32k context, q8_0 KV cache | 7.20 GiB | 63.2 (one run) | 31k tokens: 1,635 | 22.3 ms | 2.1 s |
|
| 99 |
+
|
| 100 |
+
- Ollama and llama-server answer at the same speed (they run the same kernels). llama-server starts faster and
|
| 101 |
+
shows the first token sooner. Ollama's first start on a new install also compiles GPU kernels (here 45 s to start
|
| 102 |
+
plus 39 s for the first answer).
|
| 103 |
+
- On an 8 GB card, a Qwen3-0.6B draft model (speculative decoding, one token drafted per step) gives about 69
|
| 104 |
+
tokens/s: about 4% to 9% faster than the same server without it (+3.8% to +9.3% over five paired runs; +5.6% in
|
| 105 |
+
the final one), while prompts are read about 20% slower. Speculative decoding is meant to keep the model's output;
|
| 106 |
+
in greedy tests 4 of 10 answers differed slightly, each where the top two words were close (probability gap 0.02 to 0.17).
|
| 107 |
+
- 32k context with the default cache needs about 9.3 GiB. The 32k row fits 8 GiB by storing the KV cache in 8 bits,
|
| 108 |
+
which is lossy; its answer quality was not evaluated.
|
| 109 |
+
|
| 110 |
+
**Other hardware (calculated, not measured).** Answer speed is limited by memory bandwidth: each token reads about
|
| 111 |
+
4.76 GB (the Q4_K_M weights plus a short chat's KV cache). Ceiling = bandwidth ÷ 4.76 GB; the estimate is 0.75 of
|
| 112 |
+
that, the share reached on the card above.
|
| 113 |
+
|
| 114 |
+
| Hardware | Bandwidth (GB/s) | Ceiling (tokens/s) | Estimate (tokens/s) |
|
| 115 |
+
|---|---|---|---|
|
| 116 |
+
| RTX 3050 6GB | 168 | 35 | 26 |
|
| 117 |
+
| RTX 3060 12GB | 360 | 76 | 57 |
|
| 118 |
+
| RTX 4060 8GB | 272 | 57 | 43 |
|
| 119 |
+
| RTX 4060 Ti 8GB | 288 | 61 | 45 |
|
| 120 |
+
| RTX 5060 8GB | 448 | 94 | 71 |
|
| 121 |
+
| MacBook Air (M4) | 120 | 25 | 19 |
|
| 122 |
+
| MacBook Pro (M4 Pro) | 273 | 57 | 43 |
|
| 123 |
+
|
| 124 |
+
A 6 GB card that also drives a display will not hold the 8k setup fully (see the correction above). Apple numbers use
|
| 125 |
+
the same fraction; llama.cpp's Metal backend was not measured.
|
| 126 |
+
|
| 127 |
## What it does
|
| 128 |
|
| 129 |
The fine-tune teaches **behaviour, not facts**:
|