Instructions to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF",
	filename="Grug-12B-ROCmFP4_FAST.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF
# Run inference directly in the terminal:
llama cli -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF
# Run inference directly in the terminal:
llama cli -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF
# Run inference directly in the terminal:
./llama-cli -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

Use Docker

docker model run hf.co/maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

LM Studio
Jan
Ollama
How to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with Ollama:
```
ollama run hf.co/maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF
```

Unsloth Studio

How to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF to start chatting

How to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with Docker Model Runner:
```
docker model run hf.co/maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF
```

Lemonade

How to use maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull maczzzzzz/Grug-12B-ROCmFP4_FAST-GGUF

Run and chat with the model

lemonade run user.Grug-12B-ROCmFP4_FAST-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Grug-12B ROCmFP4_FAST — GGUF

ROCmFP4_FAST quant of kai-os/Grug-12B — a compact-reasoning fine-tune of google/gemma-4-12B-it, produced via charlie12345/ROCmFPX. Benchmarked on an RDNA4 RX 9060 XT (16 GB). Context scaling is comfortable: 152k at q4_0 KV with ~30 t/s throughput and 8+ GB VRAM headroom.

Grug-12B is a specialized reasoning model that was trained to compress chain-of-thought while maintaining reasoning quality — shorter reasoning traces without losing correctness on math and code tasks. It was converted from the original safetensors (no pre-quant GGUF existed) using convert_hf_to_gguf.py and quantized in a single step.

The big caveat: this compact-reasoning fine-tune comes at a real cost on instruction-following benchmarks. The model over-reasons on every prompt — responses are 12-15k chars even for simple formatting tasks — which depresses IFEval scores significantly vs. the base Gemma-4-12B-it. Use this quant where reasoning quality and trace compression matter, not where you need clean, concise output on the first shot.

File

File	Size	Quant	BPW
`Grug-12B-ROCmFP4_FAST.gguf`	6.0 GB	Q4_0_ROCMFP4_FAST	~4.25 bpw

NOT a stock llama.cpp quant

ROCmFP4_FAST is a custom weight format from charlie12345/ROCmFPX (Q4_0_ROCMFP4 preset). Stock llama.cpp will exit with unknown quantization at load time. The system_fingerprint of a correctly-served ROCmFPX GGUF is b1-11d76c2.

Scope of these benchmarks — read this first

These numbers are a light baseline, not a thorough quant evaluation. The bench suite is built for production agent workload regression-detection, not for the kind of multi-axis sweep that upstream quant maintainers typically publish. Specifically:

Harness scope is bounded. The numbers come from llama-bench (KV context ladder). That's a regression suite, not a quality benchmark.
Sample sizes are small. Throughput is single-rep on a single GPU.
No perplexity / wikitext / MMLU / GSM8K / IFEval. The AMD IFEval run was killed at 22/50 samples after 89 min (timeout escalation at 25-30 t/s with 8k max tokens). The TQ3_4S companion has full IFEval data. AMD parity is expected within 2-8pp based on sanity probes showing identical math answers with similar reasoning structure.
Single GPU class (RDNA4 16 GB). All measurements on an RX 9060 XT (gfx1201), ROCm 7.2.3. No Strix unified-memory, no CDNA, no multi-GPU. Cross-hardware generalization is NOT implied. The companion TQ3_4S quant for Blackwell is in a separate repo.
No human eval. "30 t/s throughput with comfortable VRAM" is not a quality verdict.

What this IS good for: a quick signal that the quant (a) loads, (b) runs at good throughput on RDNA4, (c) has ample VRAM headroom, (d) scales to full 152k context. What this is NOT good for: claiming "this is the best quant of this model," reproducing academic benchmark results, or substituting for upstream's validation work.

For a rigorous view, see kai-os/Grug-12B (parent model), google/gemma-4-12B-it (base model), and the raw bench reports attached as BENCH-*.md in this repo.

What we measured

Context ceiling (q4_0 KV, AMD RDNA4 RX 9060 XT)

The 12B footprint leaves ample VRAM. Throughput is flat across the entire ladder. Blackwell is ~1.8x faster at every tier.

| KV type | Ctx | AMD TG t/s | AMD PP t/s | AMD VRAM | BW TG t/s (TQ3_4S) | |---|---|---|---:|---:|---:|---:| | f16 | 32k | 35.1 | 293.8 | 8.3 GB | 56.9 | | q8_0 | 64k | 31.2 | 263.1 | 8.2 GB | 54.7 | | q4_0 | 64k | 30.4 | 260.0 | 7.9 GB | 54.6 | | q4_0 | 98k | 30.4 | 261.2 | 8.0 GB | 54.7 | | q4_0 | 131k | 30.4 | 262.1 | 8.2 GB | 54.7 | | q4_0 | 152k | 30.4 | 260.7 | 8.3 GB | 54.7 |

VRAM headroom at 152k: >7 GB. No VRAM constraints on either card class.

Quick start

# Build charlie12345/ROCmFPX
git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX
mkdir build && cd build
cmake .. -DGGML_CUDA=ON
make -j$(nproc)

# Serve
./bin/llama-server \
  -m /path/to/Grug-12B-ROCmFP4_FAST.gguf \
  --port 8081 \
  -ngl 99 \
  -c 32768 \
  --reasoning-format none

Do NOT set GGML_HIP_ENABLE_UNIFIED_MEMORY=1 in the environment on discrete AMD cards — it moves the entire model to system RAM (30× regression).

Reproduce the quant

# From kai-os/Grug-12B safetensors, converted to BF16 GGUF first:
python3 convert_hf_to_gguf.py /path/to/kai-os/Grug-12B --outfile Grug-12B-BF16.gguf

# Then quantize:
/path/to/llama-quantize \
  --allow-requantize \
  Grug-12B-BF16.gguf \
  Grug-12B-ROCmFP4_FAST.gguf \
  Q4_0_ROCMFP4_FAST

AEON Bench Full Suite — 150 cases (120 scored)

Overall mean: 0.558 (ROCmFP4_FAST) — 0.133 on the TQ3_4S companion quant ⚠️

The Grug-12B ROCmFP4_FAST quant performs well, matching Qwen3.6-27B's overall mean (0.558). It excels at reasoning (0.683) and coding (0.775).

⚠️ CRITICAL: The companion Grug-12B TQ3_4S quant scores catastrophically low (0.133). This quant format appears broken for this model architecture. Do NOT use the TQ3_4S quant of Grug-12B in production.

Files in this repo

File	Purpose
`Grug-12B-ROCmFP4_FAST.gguf`	The quantized model (LFS-tracked, 6.0 GB)
`README.md`	This file
`BENCH-*.md`	Raw bench data and cross-format context-ceiling reports

What's NOT in this repo (caveats)

Stock llama.cpp will not load this file. ROCmFP4_FAST is a custom weight format unique to charlie12345/ROCmFPX.
Grug is NOT a general-purpose instruct model. It's a specialized compact-reasoning model. Use for math, code reasoning, and agentic pipelines where reasoning-token compression matters. Do NOT use for instruction-following tasks (IFEval, formatting) or direct chat where clean output is expected — the reasoning bloat will hurt quality.
No CUDA / non-AMD GPU bench. All measurements are RDNA4 (gfx1200).
No IFEval on this arm. The AMD IFEval run was killed at 22/50 after 89 min. The companion TQ3_4S Blackwell repo has full IFEval data. Expect AMD parity within 2-8pp.
No vision/multimodal test. This variant is text-only.
No MTP / speculative-decode bench. Not applicable on this architecture.
No quality benchmark (perplexity, MMLU, GSM8K). The fine-tune optimizes for reasoning-token compression; academic benchmarks may not reflect its strengths or weaknesses.

Provenance

Source model: kai-os/Grug-12B (license: other — verify commercial use terms)
Base model: google/gemma-4-12B-it (Gemma 4 license)
Conversion: convert_hf_to_gguf.py (BF16 GGUF from 7 sharded safetensors, 23.8 GB → ROCmFP4_FAST 6.0 GB)
Quantizer: charlie12345/ROCmFPX commit 5b39566, preset Q4_0_ROCMFP4_FAST
Build hardware: Node B — RDNA4 RX 9060 XT 16 GB (gfx1201), ROCm 7.2.3, NixOS
Bench harnesses: llama-bench (context ladder)

License

This quant is derived from kai-os/Grug-12B (license: other — verify commercial use is permitted before deployment), which is a fine-tune of google/gemma-4-12B-it (Gemma 4 license). The ROCmFP4 quant format is provided by charlie12345/ROCmFPX (MIT).