Soofi-S-Instruct-Preview-GGUF

⚠️ Preview / internal checkpoint. Weights and metadata may still change.

GGUF quantizations of Soofi-Project/Soofi-S-Instruct-Preview for use with llama.cpp and Ollama.

Converted from fp16 safetensors with convert_hf_to_gguf.py and quantized with llama-quantize (llama.cpp).

Architecture support: SOOFI-S is a custom hybrid Mamba-2/MoE model and ships with its own modeling code. GGUF conversion requires a build of llama.cpp that understands this architecture — verify against the actual checkpoint before relying on these quants.

Available quantizations

File	Quant	Bits/weight	Size (approx.)	Recommendation
soofi-s-instruct-preview-Q8_0.gguf	Q8_0	8.5	~32 GB	Practically lossless, maximum quality
soofi-s-instruct-preview-Q5_K_M.gguf	Q5_K_M	6.6	~25 GB	Good size/quality trade-off, recommended default
soofi-s-instruct-preview-Q4_K_M.gguf	Q4_K_M	~5.5	~21 GB	Smallest, for tighter memory budgets

Sizes scale with the total 30B parameters (not the 3.5B active). Q4_K_M is an estimate; the others are measured.

No Q6_K: this architecture's tensor columns (2688/1856/3712) are not divisible by 256, so every K-quant tensor falls back to a non-K type. For Q6_K that fallback is q8_0, making it ~as large as Q8_0 for no gain; Q5_K_M and Q4_K_M fall back to q5_1/q4_1 and still shrink.

Usage with Ollama

Directly from this repo (select the quant level via the tag):

ollama run hf.co/Soofi-Project/Soofi-S-Instruct-Preview-GGUF:Q5_K_M

For a private repo, Ollama needs to know your HF token; otherwise use the local Modelfile route.

Locally with a Modelfile:

Ollama uses Go templates and does not apply the Jinja chat_template embedded in the GGUF, so supply a matching Go template yourself; otherwise Ollama falls back to a generic one that may miss this model's ChatML format and its non-thinking behaviour. Unlike the Isar/Rhine reasoning variants, the Instruct chat template carries no identity in its default system prompt (only a date) and is non-thinking by default, so the template below closes the <think> block immediately:

FROM ./soofi-s-instruct-preview-Q5_K_M.gguf
PARAMETER temperature 0.6
PARAMETER top_p 0.95
PARAMETER num_ctx 8192
PARAMETER stop "<|im_end|>"

TEMPLATE """{{- if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}
{{- range .Messages }}
{{- if eq .Role "user" }}<|im_start|>user
{{ .Content }}<|im_end|>
{{ else if eq .Role "assistant" }}<|im_start|>assistant
{{ .Content }}<|im_end|>
{{ end }}
{{- end }}<|im_start|>assistant
<think></think>"""

ollama create soofi-s-instruct:q5 -f Modelfile
ollama run soofi-s-instruct:q5

Thinking is disabled at the template level. The prompt ends with a closed <think></think>, so the model answers directly instead of emitting a reasoning block — you do not need to turn thinking off in your client (e.g. Open WebUI). Without this (a generic template, or the hf.co route that leaves <think> open), the whole answer ends up inside the think block.

No identity system prompt here. The Instruct chat template's default system prompt contains only a knowledge cutoff and date — no "You are Soofi …" text. If this variant still identifies as Soofi in vLLM, that identity is baked into the weights (and therefore survives quantization), not injected by the template. To force a fixed identity, add a SYSTEM """…""" block (see the Isar/Rhine GGUF cards for the wording).

Tool calling is intentionally omitted from this Go template. The model's native tool format (<tool_call><function=…>) is not reproduced here, so Ollama's native function-calling path is unavailable. In Open WebUI, MCP tools still work via the prompt-based Default function-calling mode (Open WebUI handles tool calls in its own layer). For robust native tool calling with the original format, run the model under llama-server --jinja instead, which applies the GGUF's embedded Jinja template (identity and tools) verbatim — see the llama.cpp section below.

Usage with llama.cpp

llama-cli -m soofi-s-instruct-preview-Q5_K_M.gguf -p "Explain AI sovereignty in one sentence."
# or as an OpenAI-compatible server:
llama-server -m soofi-s-instruct-preview-Q5_K_M.gguf --jinja --port 8080

Use --jinja. It makes llama.cpp apply the GGUF's embedded chat template (the model's own Jinja), so the native tool-calling format works out of the box — no manual template needed. This is the recommended backend for native function calling (e.g. as an OpenAI endpoint in Open WebUI).

Architecture note

This is a hybrid Mixture-of-Experts model designed from scratch: 23 Mamba-2/MoE layers + 6 attention layers, 128 routing experts + 1 shared expert per MoE layer, 6 experts active per token (30B total / 3.5B active). During quantization, router/gate and certain attention tensors may intentionally be kept at higher precision; the same applies to the SSM/recurrent (Mamba-2) tensors. A recent version of llama.cpp is recommended.

Related models

• Base (bf16): Soofi-Project/Soofi-S-Instruct-Preview
• FP8 quantization (vLLM): Soofi-Project/Soofi-S-Instruct-Preview-FP8

License & provenance

Released under a custom license ("Other"), following the base model Soofi-Project/Soofi-S-Instruct-Preview. TODO: mirror the full license text once the base model card defines it.