Instructions to use guideboardlabs/Voltron-1-Mini-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use guideboardlabs/Voltron-1-Mini-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
Use Docker
docker model run hf.co/guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use guideboardlabs/Voltron-1-Mini-GGUF with Ollama:
ollama run hf.co/guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use guideboardlabs/Voltron-1-Mini-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use guideboardlabs/Voltron-1-Mini-GGUF with Docker Model Runner:
docker model run hf.co/guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
- Lemonade
How to use guideboardlabs/Voltron-1-Mini-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Voltron-1-Mini-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use guideboardlabs/Voltron-1-Mini-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use guideboardlabs/Voltron-1-Mini-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "guideboardlabs/Voltron-1-Mini-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Voltron-1 Mini GGUF
GGUF quantizations of Akahsizrr/voltron-1-mini, a LoRA post-train of LiquidAI/LFM3-1.7B by @RoliumGens.
The model shipped as BF16 safetensors on 13 November 2026 with no GGUF anywhere, so these files were converted and quantized at Guideboard Labs to run it on consumer GPUs. Same architecture, tokenizer and chat template as the base model. Nothing about the weights was changed beyond quantization.
Files
| File | Quant | Size | BPW | Notes |
|---|---|---|---|---|
voltron-1-mini-Q4_K_M.gguf |
Q4_K_M | 1.10 GB | 5.05 | Benchmarked here. Tensor map identical to Liquid's own LFM3-1.7B Q4_K_M |
voltron-1-mini-Q8_0.gguf |
Q8_0 | 1.85 GB | 8.51 | Near-lossless, for comparison runs |
The Q4_K_M is a clean drop-in for Liquid's official quant. Both files carry 287 tensors and the
same type map (136 q4_K, 18 q6_K, 133 f32), and the two files differ by 352 bytes. That means a quality
or speed comparison against LiquidAI/LFM3-1.7B-GGUF at Q4_K_M measures the fine-tune, not the
quantization recipe.
How these were built
Conversion and quantization ran with llama.cpp at 0.7.0-dev, build 50, commit 338fce651, the tree
that first carried conversion/lfm3.py locally. That converter is upstream now: LFM3 support landed in
PR #31252, merged 2026-10-27 as 19cce69.
# 1. weights
hf download Akahsizrr/voltron-1-mini --local-dir voltron-1-mini --exclude "eval/*"
# 2. convert (BF16 safetensors -> F16 GGUF)
python3 convert_hf_to_gguf.py voltron-1-mini --outtype f16 --outfile voltron-1-mini-F16.gguf
# 3. quantize
llama-quantize voltron-1-mini-F16.gguf voltron-1-mini-Q4_K_M.gguf Q4_K_M
llama-quantize voltron-1-mini-F16.gguf voltron-1-mini-Q8_0.gguf Q8_0
Verified to load and generate on a plain upstream build (0.9.0-dev, build 1, commit 3ba1ce0), no
patch needed.
Run it
llama-server -m voltron-1-mini-Q4_K_M.gguf \
-ngl 999 -fa on -c 32768 --jinja \
--temp 0.1 --top-k 50 --repeat-penalty 1.1
Give it a wide window. Thinking is on by default in the chat template and the trace lands in
reasoning_content, separated from content by llama.cpp. On a trivial code prompt this model spent
4,860 characters of reasoning and 1,174 completion tokens before answering, so a small -c or
max_tokens gets spent inside the trace and returns an empty reply. 16K works, 32K is comfortable,
and the KV cost is small: 1.59 GiB of VRAM in total at 32K on a 1.7B model.
Sampling above is the model's own generation_config.json (temperature 0.1, top_k 50, repetition
penalty 1.1). The model card's own evals used temperature 0.6 and top_p 0.95, so try both.
Measured performance
One RX 5700 XT (RDNA1, 8 GB), Ryzen 5 1600, 32 GB RAM, llama.cpp Vulkan, Q4_K_M:
| decode | 135.3 t/s (509k tokens over 66 requests, median 137.3) |
| prefill | ~1,700 t/s |
| VRAM | 1.59 GiB at a 32K window, 0.28 GiB GTT, no spill |
| load | about 2 s |
| file | 1.10 GB |
Benchmark: Agon-Bench, 66 points
Full three-event pass, three runs per event, against the base model measured on the same harness:
| event | max | LFM3-1.7B | Voltron-1 Mini | delta |
|---|---|---|---|---|
| Agent | 24 | 18.00 | 18.67 | +0.67 |
| Code | 25 | 13.00 | 17.67 | +4.67 |
| Reason | 17 | 15.67 | 15.67 | 0.00 |
| Total | 66 | 46.67 | 52.00 | +5.33 |
Rank 21 of 36 visible entries on the Agon-Bench board (harness and board: GuideboardLabs/agon-arena), at the same 1.1 GB and the same speed as the base.
Three disclosures, because a table without them is a claim too strong:
- These are automated Agon scores, checked against the harness's known grading defects but not hand-graded task by task.
- The code event was re-run at a 32K window. At the base entry's 16K window this model scored 16.67 on code, with four responses cut off mid-answer at around 70,000 characters, which is a window limit and not a capability result. On the fully comparable 16K configuration the total is 51.01.
- The base model's 46.67 was measured at 16K, so the +4.67 on code compares a widened window to a narrow one. The like-for-like figure is +4.34 overall.
What it is good for, and what it is not
Best on code tasks that fit in one file: string handling, parsers, a multi-file CLI. Agent tasks are mostly fine. On the hardest code task in that suite (a regex engine written from scratch) all three runs produced scripts that crash on load, so the 17.67 is not evidence of that capability.
At 1.73B it is small. It will lose to any competent 7B and up on open-ended work. Its case is speed and footprint: 1.1 GB of weights at 135 t/s on an 8 GB card, with a real gain over the base model.
Attribution and license
- Fine-tune: Akahsizrr / @RoliumGens (model)
- Base model: Liquid AI (LFM3-1.7B)
- GGUF conversion: Guideboard Labs
Licensed under the LFM Open License v1.0, the same license as the base model. A copy ships as
LICENSE in this repo, and redistribution requires you to keep it. Section 5 limits
commercial use by entities above the license's revenue threshold, so read the license before using
this commercially. Model names and marks belong to their owners, used here only to describe origin.
Checksums
a4b5fd8249851528101c548e37f16e5911fcb1ea1378474817ca684ce3be2191 voltron-1-mini-Q4_K_M.gguf
1c7c3ce8f269f6dde72a2bf28133cbe00d2114f30426bc969ae331cfe9e38bdf voltron-1-mini-Q8_0.gguf
- Downloads last month
- 416
4-bit
8-bit