Card: GGUF Q3_K_M / Q4_K_M / Q6_K with idle-machine CPU speeds; front matter on the first line again

README.md

@@ -1,4 +1,3 @@
-
 ---
 license: apache-2.0
 library_name: pytorch
@@ -33,7 +32,7 @@ tags:
 | Vocabulary | 248,320 (Qwen 3.5 tokenizer and chat template), untied input / output embeddings |
 | Extra head | 1-layer MTP (multi-token prediction) head kept from the base model, not trained in this SFT |
 | Code | `JiRackDeltaNet_70b.py`: single-file PyTorch model; parameter names match the Hugging Face `Qwen3_5ForCausalLM` state dict |
-| Weights | bf16 (repository root), an FP8 block-quantized copy in `sft_final_fp8/` and a Q4\_K\_M GGUF in `gguf/` (see below) |
+| Weights | bf16 (repository root), an FP8 block-quantized copy in `sft_final_fp8/` and GGUF quantizations (Q3\_K\_M, Q4\_K\_M, Q6\_K) in `gguf/` (see below) |
 
 ## How it was built
 
@@ -95,17 +94,17 @@ Assistant: Tom buys 7 pens at $3 each, so he spends 7 x $3 = $21. He pays with a
 
 ## GGUF (llama.cpp)
 
-`gguf/JiRackDeltaNet_70b_with_prompt.Q4_K_M.gguf` is a 42.65 GB Q4\_K\_M GGUF with a built-in system prompt (direct answers, no reasoning block). If a request has no system message, the chat template in the GGUF adds the prompt; your own system message replaces it. The prompt text, commands and checks are in gguf/README.md.
+Three GGUF quantizations in `gguf/`, each with a built-in system prompt (direct answers, no reasoning block). If a request has no system message, the chat template in the GGUF adds the prompt; your own system message replaces it. The prompt text, commands and checks are in [gguf/README.md](gguf/README.md).
 
 ```bash
 llama-server -m JiRackDeltaNet_70b_with_prompt.Q4_K_M.gguf -c 8192 --jinja --reasoning off --port 8080
 ```
 
-|  | Q4\_K\_M GGUF |
-| --- | --- |
-| Size | 42.65 GB |
-| CPU speed (GH200 Grace, 64 threads, `llama-bench`) | prompt 41.4 tokens/s, generation 0.87 tokens/s |
-| Pens question above | $29 |
+|  | Q3\_K\_M | Q4\_K\_M | Q6\_K |
+| --- | --- | --- | --- |
+| Size | 33.89 GB | 42.65 GB | 57.40 GB (2 parts) |
+| CPU speed, prompt / generation (GH200 Grace, 64 threads, `llama-bench`) | 20.1 / 5.17 tokens/s | 50.2 / 5.04 tokens/s | 45.3 / 4.11 tokens/s |
+| Pens question above | $29 | $29 | $29 |
 
 ## Usage
 

gguf/README.md

@@ -1,13 +1,18 @@
 # JiRackDeltaNet 70B: GGUF for llama.cpp
 
-A Q4_K_M GGUF of **JiRackDeltaNet 70B** (the bf16 SFT release in the root of this repository), with a
-**built-in system prompt**.
+GGUF quantizations of **JiRackDeltaNet 70B** (the bf16 SFT release in the root of this repository). Each file has the
+same **built-in system prompt**.
+
+| File | Size | Notes |
+|---|---|---|
+| `JiRackDeltaNet_70b_with_prompt.Q3_K_M.gguf` | 33.89 GB | smallest; on the ARM CPU we tested, prompt processing is 2.5× slower than Q4_K_M |
+| `JiRackDeltaNet_70b_with_prompt.Q4_K_M.gguf` | 42.65 GB | recommended balance of size, speed and quality |
+| `JiRackDeltaNet_70b_with_prompt.Q6_K-00001-of-00002.gguf` + `-00002-of-00002.gguf` | 57.40 GB (45.01 + 12.39) | closest to bf16; split in two parts, load the first one |
 
 | | |
 |---|---|
-| File | `JiRackDeltaNet_70b_with_prompt.Q4_K_M.gguf` |
-| Size | 42.65 GB (39.7 GiB), 69.93B parameters, 1,714 tensors |
-| Quantization | Q4_K_M: `convert_hf_to_gguf.py` (bf16 GGUF) → `llama-quantize` |
+| Model | 69.93B parameters, 1,714 tensors |
+| Quantization | `convert_hf_to_gguf.py` (bf16 GGUF) → `llama-quantize`, no importance matrix |
 | llama.cpp architecture | `qwen35` (Qwen 3.5 layout: 96 Gated DeltaNet + 32 gated full-attention layers, 128 in total) |
 | Built and tested with | llama.cpp build `5d7793d` (October 28, 2026) |
 | Context | trained with sequences up to 8,192 tokens |
@@ -22,6 +27,8 @@ cd llama.cpp
 cmake -B build                      # add -DGGML_CUDA=ON for NVIDIA GPUs
 cmake --build build -j --config Release
 hf download CMSManhattan/JiRackDeltaNet_70b gguf/JiRackDeltaNet_70b_with_prompt.Q4_K_M.gguf --local-dir .
+# Q6_K: both parts
+# hf download CMSManhattan/JiRackDeltaNet_70b --include "gguf/*Q6_K*" --local-dir .
 ```
 
 OpenAI-compatible server:
@@ -29,6 +36,7 @@ OpenAI-compatible server:
 ```bash
 ./build/bin/llama-server -m gguf/JiRackDeltaNet_70b_with_prompt.Q4_K_M.gguf \
     -c 8192 --jinja --reasoning off --port 8080
+# Q6_K: -m gguf/JiRackDeltaNet_70b_with_prompt.Q6_K-00001-of-00002.gguf (the second part is found automatically)
 ```
 
 ```bash
@@ -40,8 +48,8 @@ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/jso
 
 Interactive chat in the terminal: `./build/bin/llama-cli` with the same `-m`, `-c`, `--jinja` and `--reasoning off` flags.
 
-With a GPU build, `-ngl 999` puts all layers on the GPU (the weights alone are 42.65 GB). On a smaller GPU,
-offload part of the layers with `-ngl N`. GPU speed is not measured yet.
+With a GPU build, `-ngl 999` puts all layers on the GPU, which needs the file size in GPU memory plus the context. On a
+smaller GPU, offload part of the layers with `-ngl N`. GPU speed is not measured yet.
 
 ## Built-in system prompt
 
@@ -70,32 +78,40 @@ system message together with `"chat_template_kwargs": {"enable_thinking": true}`
 
 ## Measured
 
-All numbers below were measured on the CPU of an NVIDIA GH200 node (Grace, 64 threads) with a CPU-only llama.cpp build,
-while other jobs were running on the node's GPU.
+CPU speed: `llama-bench` on the CPU of an NVIDIA GH200 node (Grace, 64 threads), CPU-only llama.cpp build, nothing else
+running. Answers: `llama-server --reasoning off`, greedy, built-in prompt.
 
-| Test | Result |
-|---|---|
-| `llama-bench` prompt processing (pp512) | 41.4 tokens/s |
-| `llama-bench` generation (tg128) | 0.87 tokens/s |
-| Pens question (greedy, `--reasoning off`, built-in prompt) | "Tom buys 7 pens at $3 each, so the total cost is 7 * 3 = $21. He pays with a $50 bill, so the change he gets is 50 - 21 = $29." (53 tokens, 0.88 tokens/s) |
-| Template, no system message | built-in prompt added; the reply starts after an empty `<think></think>` block |
-| Template, own system message | built-in prompt not added |
-| Template, with tools | built-in prompt added after the tools block |
-| Template, `enable_thinking: true` | the reply starts inside `<think>` |
-
-Against the source GGUF, the file with the built-in prompt was checked as follows:
+| | Q3_K_M | Q4_K_M | Q6_K |
+|---|---|---|---|
+| Prompt processing (pp512) | 20.1 tokens/s | 50.2 tokens/s | 45.3 tokens/s |
+| Generation (tg128) | 5.17 tokens/s | 5.04 tokens/s | 4.11 tokens/s |
+| Pens question | $29 | $29 | $29 |
+
+Q4_K_M answer: "Tom buys 7 pens at $3 each, so the total cost is 7 * 3 = $21. He pays with a $50 bill, so the change
+he gets is 50 - 21 = $29."
+
+Each file was checked against the GGUF it was made from:
 - the only changed metadata key is the chat template;
 - all 1,714 tensors have the same names, types and shapes;
 - 8 of the tensors were compared byte for byte.
 
-On CPU, generation is limited by memory bandwidth (42.65 GB of weights are read for every token), so plan for about
-one token per second on CPU, as measured above. For interactive use, offload layers to a GPU.
+Template checks through `llama-server` (on Q4_K_M; the template is byte-identical in all three files):
+
+| Request | Result |
+|---|---|
+| no system message | built-in prompt added; the reply starts after an empty `<think></think>` block |
+| own system message | built-in prompt not added |
+| with tools | built-in prompt added after the tools block |
+| `enable_thinking: true` | the reply starts inside `<think>` |
+
+Generation reads the whole file once per token, so on CPUs with less memory bandwidth than Grace it is proportionally
+slower.
 
 ## Limitations
 
 - The 64 layers added by depth expansion have been trained only by the SFT run (~45 M tokens). See the main model card.
-- Q4_K_M is a lossy 4-bit quantization of the bf16 weights. Benchmarks of this GGUF (GSM8K, HumanEval, tool calling)
-  are not measured yet.
+- These are lossy quantizations of the bf16 weights, made without an importance matrix. Benchmarks of the GGUF files
+  (GSM8K, HumanEval, tool calling) are not measured yet.
 - Not ternary yet: ternary QAT (and a TQ2_0 GGUF) is a planned later stage.
 - An Ollama build is not published yet.
 - Not for medical, legal or financial advice.