nphearum/PsarAI-2B-GGUF

PsarAI-2B GGUF

GGUF exports for PsarAI-2B, a PsarAI chat model based on unsloth/gemma-4-E2B-it.

The chat template identifies the assistant as PsarAI and uses Gemma's native turn, channel, tool-call, image, audio, and video tokens.

Recommended File

Use Q4_K_M for the best default balance of size, speed, and quality:

llama-cli \
  -hf nphearum/PsarAI-2B-GGUF:Q4_K_M \
  -p "Who created you?"

For higher quality, use Q5_K_M, Q6_K, or Q8_0 if you have enough RAM/VRAM.

llama.cpp Server

llama-server \
  -hf nphearum/PsarAI-2B-GGUF:Q4_K_M \
  --host 0.0.0.0 \
  --port 8080 \
  -c 8192

Then call the OpenAI-compatible endpoint:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "PsarAI-2B",
    "messages": [
      {"role": "user", "content": "Who created you?"}
    ],
    "temperature": 0.7,
    "top_p": 0.9
  }'

Thinking

For normal chatbot use, disable visible thinking in your template/runtime settings when supported:

{"enable_thinking": false}

If thinking is enabled, the template asks the model to keep it short and useful.

Multimodal Projector

This repo includes:

PsarAI-2B.BF16-mmproj.gguf

Use it with a llama.cpp build/runtime that supports Gemma 4 multimodal GGUF. Exact image/audio/video CLI flags may depend on your llama.cpp version.

Files

File	Size
PsarAI-2B.Q4_K_M.gguf	3.19 GiB
PsarAI-2B.Q5_K_M.gguf	3.38 GiB
PsarAI-2B.Q6_K.gguf	3.58 GiB
PsarAI-2B.Q8_0.gguf	4.63 GiB
PsarAI-2B.Q4_K_S.gguf	3.13 GiB
PsarAI-2B.Q3_K_M.gguf	2.98 GiB
PsarAI-2B.IQ4_XS.gguf	3.08 GiB
PsarAI-2B.IQ4_NL.gguf	3.14 GiB
PsarAI-2B.IQ3_M.gguf	2.14 GiB
PsarAI-2B.F16.gguf	8.67 GiB
PsarAI-2B.BF16-mmproj.gguf	0.92 GiB

Quantization Guide

• Q4_K_M: recommended default
• Q5_K_M: better quality with moderate extra size
• Q6_K: strong quality if memory is available
• Q8_0: near full precision, largest practical runtime file
• Q3_* / IQ3_*: smaller files, lower quality

Notes

• Context length in the source config is up to 131072 tokens, but practical context depends on your runtime memory.
• This model uses a Gemma-style chat template, not Qwen XML-style tool calls.
• If you upload only a subset of files, update the file table above to match the repo contents.