Card: GGUF variants now available (text-only)

README.md

@@ -106,9 +106,30 @@ When serving via vLLM, pass `--reasoning-parser qwen3`; to disable thinking per
 
 The model serves with **vLLM** for production text and multimodal inference (Transformers ≥ 5.5). Greedy decoding (temperature 0) is recommended for legal tasks where determinism matters.
 
-### Quantized / GGUF / Ollama
+### Quantized GGUF variants (text-only)
 
-Quantized GGUF variants (and therefore Ollama hosting) are **planned but not yet released**. flash-1-mini uses Qwen3.5's hybrid linear-attention (Gated DeltaNet / SSM) architecture; GGUF conversion of this architecture succeeds, but verified GGUF *inference* depends on `llama.cpp` runtime support for these layers, which is still maturing. When verified, GGUF variants will be published as **text-only** (the vision tower is not carried in GGUF). Until then, use the bf16 weights above for both text and multimodal inference. To run in lower precision today, load the bf16 weights with `bitsandbytes` 4-bit/8-bit via `BitsAndBytesConfig`.
+GGUF quantizations for CPU / edge inference via `llama.cpp` and Ollama are available in the [`gguf/`](./tree/main/gguf) folder of this repository:
+
+| File | Quant | Size | Notes |
+|---|---|---|---|
+| `gguf/flash-1-mini-20260602-Q6_K.gguf` | Q6_K | 3.3 GB | Highest fidelity; closest to bf16 |
+| `gguf/flash-1-mini-20260602-Q5_K_M.gguf` | Q5_K_M | 2.9 GB | Balanced quality / size |
+| `gguf/flash-1-mini-20260602-Q4_K_M.gguf` | Q4_K_M | 2.6 GB | Smallest; quality holds on common tasks |
+
+**Important — these GGUFs are text-only.** The vision tower is not carried in the GGUF format, so image input is **not** supported by the GGUF variants. For multimodal (image) inference, use the bf16 safetensors weights above. Quality scales with bit-depth: Q6_K tracks the bf16 model most closely; lower bit-depths trade some fidelity for size, and on the most demanding legal-citation tasks the higher-bit quants are recommended.
+
+```bash
+# llama.cpp
+./llama-completion -m flash-1-mini-20260602-Q5_K_M.gguf \
+  -p "What is the legal test under section 1 of the Canadian Charter?" -n 200 --temp 0
+
+# Ollama (create a Modelfile pointing at the GGUF, then run)
+printf 'FROM ./flash-1-mini-20260602-Q5_K_M.gguf\n' > Modelfile
+ollama create flash-1-mini -f Modelfile
+ollama run flash-1-mini
+```
+
+GGUF inference of this architecture (Qwen3.5 hybrid linear-attention / Gated DeltaNet) requires a recent `llama.cpp` build with support for these layers. The multi-token-prediction (MTP) head is excluded from the GGUF (not used at inference). To run the bf16 weights in lower precision instead, load them with `bitsandbytes` 4-bit/8-bit via `BitsAndBytesConfig`.
 
 ## Benchmarks