Soofi-S-Isar-Preview-GGUF

⚠️ Preview / internal checkpoint. Weights and metadata may still change.

GGUF quantizations of Soofi-Project/Soofi-S-Isar-Preview for use with llama.cpp and Ollama.

Converted from fp16 safetensors with convert_hf_to_gguf.py and quantized with llama-quantize (llama.cpp). This is the Isar reasoning variant – it emits explicit thinking traces before the final answer.

Architecture support: SOOFI-S is a custom hybrid Mamba-2/MoE model and ships with its own modeling code. GGUF conversion requires a build of llama.cpp that understands this architecture — verify against the actual checkpoint before relying on these quants.

Available quantizations

File	Quant	Bits/weight	Size (approx.)	Recommendation
soofi-s-isar-preview-Q8_0.gguf	Q8_0	8.5	~32 GB	Practically lossless, maximum quality
soofi-s-isar-preview-Q5_K_M.gguf	Q5_K_M	6.6	~25 GB	Good size/quality trade-off, recommended default
soofi-s-isar-preview-Q4_K_M.gguf	Q4_K_M	~5.5	~21 GB	Smallest, for tighter memory budgets

Sizes scale with the total 30B parameters (not the 3.5B active). Q4_K_M is an estimate; the others are measured.

No Q6_K: this architecture's tensor columns (2688/1856/3712) are not divisible by 256, so every K-quant tensor falls back to a non-K type. For Q6_K that fallback is q8_0, making it ~as large as Q8_0 for no gain; Q5_K_M and Q4_K_M fall back to q5_1/q4_1 and still shrink.

Usage with Ollama

Directly from this repo (select the quant level via the tag):

ollama run hf.co/Soofi-Project/Soofi-S-Isar-Preview-GGUF:Q5_K_M

For a private repo, Ollama needs to know your HF token; otherwise use the local Modelfile route.

Locally with a Modelfile:

The model's identity ("You are Soofi …") lives in the default system prompt of the chat template, not in the weights. Ollama uses Go templates and does not apply the Jinja chat_template embedded in the GGUF, so you must supply the system prompt and a matching template yourself — otherwise the model has no identity. The template below reproduces the model's ChatML format and lets the model open its own <think> reasoning block:

FROM ./soofi-s-isar-preview-Q5_K_M.gguf
PARAMETER temperature 0.6
PARAMETER top_p 0.95
PARAMETER num_ctx 16384
PARAMETER stop "<|im_end|>"

SYSTEM """You are Soofi (Sovereign Open Source Foundation Models), an open-source AI assistant built for reasoning, developed by a German research consortium.

Architecture: Hybrid Mixture-of-Experts (MoE) with 23 Mamba-2/MoE layers and 6 Attention layers. 128 experts + 1 shared expert per MoE layer, 6 activated per token. 3.5B active parameters, 30B total.

Training: Trained from scratch on 25 trillion freely available tokens. Primary languages: English and German. Limited capability in French, Italian, and Spanish. English is the pivot language. Training started April 2026.

Project: SOOFI is a €20M project funded by the German Federal Ministry for Economic Affairs and Energy (BMWE) under the 8ra framework. It aims to provide a sovereign European open-source alternative to US/Chinese AI models for industrial use, including a reasoning model and AI agent capabilities. Training infrastructure is hosted on T-Systems' Industrial AI Cloud (Deutsche Telekom).

Consortium: Coordinated by the KI Bundesverband (German AI Association). Partners include: DFKI, Fraunhofer IAIS, Fraunhofer IIS, TU Darmstadt, Uni Würzburg, Berliner Hochschule für Technik, L3S Research Center (Uni Hannover), Lamarr Institute, ellamind, and Merantix Momentum.

Contact: contact@soofi.info

Behaviour:
- Answer identity questions naturally as Soofi.
- For non-identity questions, respond normally and helpfully.
- Match the language the user writes in.
- Knowledge cutoff: 2025-12"""

TEMPLATE """{{- if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}
{{- range .Messages }}
{{- if eq .Role "user" }}<|im_start|>user
{{ .Content }}<|im_end|>
{{ else if eq .Role "assistant" }}<|im_start|>assistant
{{ .Content }}<|im_end|>
{{ end }}
{{- end }}<|im_start|>assistant
"""

ollama create soofi-s-isar:q5 -f Modelfile
ollama run soofi-s-isar:q5

Reasoning models benefit from a larger context window and a higher token budget, since the thinking trace consumes output tokens.

Tool calling is intentionally omitted from this Go template. The model's native tool format (<tool_call><function=…>) is not reproduced here, so Ollama's native function-calling path is unavailable. In Open WebUI, MCP tools still work via the prompt-based Default function-calling mode (Open WebUI handles tool calls in its own layer). For robust native tool calling with the original format, run the model under llama-server --jinja instead, which applies the GGUF's embedded Jinja template (identity and tools) verbatim — see the llama.cpp section below.

Reasoning output: the model emits <think> … </think> blocks. The template ends the prompt at <|im_start|>assistant, letting the model open its own <think> block; the thinking trace appears inline in the response.

Usage with llama.cpp

llama-cli -m soofi-s-isar-preview-Q5_K_M.gguf -n 1024 \
  -p "How many r's are in strawberry?"
# or as an OpenAI-compatible server:
llama-server -m soofi-s-isar-preview-Q5_K_M.gguf --jinja --port 8080

Use --jinja. It makes llama.cpp apply the GGUF's embedded chat template (the model's own Jinja), so the identity default system prompt and the native tool-calling format work out of the box — no manual system prompt or template needed. Without --jinja, llama.cpp falls back to a generic built-in template and the identity is lost. This is the recommended backend for native function calling (e.g. as an OpenAI endpoint in Open WebUI).

Architecture note

This is a hybrid Mixture-of-Experts model designed from scratch: 23 Mamba-2/MoE layers + 6 attention layers, 128 routing experts + 1 shared expert per MoE layer, 6 experts active per token (30B total / 3.5B active). During quantization, router/gate and certain attention tensors may intentionally be kept at higher precision; the same applies to the SSM/recurrent (Mamba-2) tensors. A recent version of llama.cpp is recommended.

Related models

• Base (bf16): Soofi-Project/Soofi-S-Isar-Preview
• Other reasoning variant: Soofi-Project/Soofi-S-Rhine-Preview
• FP8 quantization (vLLM): Soofi-Project/Soofi-S-Isar-Preview-FP8

License & provenance

Released under a custom license ("Other"), following the base model Soofi-Project/Soofi-S-Isar-Preview. TODO: mirror the full license text once the base model card defines it.