Instructions to use Bastion-AI/SmolLM3-3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Bastion-AI/SmolLM3-3B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Bastion-AI/SmolLM3-3B-GGUF",
	filename="SmolLM3-3B.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use Bastion-AI/SmolLM3-3B-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use Bastion-AI/SmolLM3-3B-GGUF with Ollama:
```
ollama run hf.co/Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M
```

Unsloth Studio new

How to use Bastion-AI/SmolLM3-3B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Bastion-AI/SmolLM3-3B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Bastion-AI/SmolLM3-3B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Bastion-AI/SmolLM3-3B-GGUF to start chatting

Pi new

How to use Bastion-AI/SmolLM3-3B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Bastion-AI/SmolLM3-3B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use Bastion-AI/SmolLM3-3B-GGUF with Docker Model Runner:
```
docker model run hf.co/Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M
```

Lemonade

How to use Bastion-AI/SmolLM3-3B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Bastion-AI/SmolLM3-3B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.SmolLM3-3B-GGUF-Q4_K_M

List all available models

lemonade list

SmolLM3-3B GGUF Quantized Models

📌 Original Model Reference

This repository provides quantized GGUF versions of the original model:

HuggingFaceTB/SmolLM3-3B

All quantizations are derived from this base model. Please refer to the original model card for architecture, training details, and upstream updates.

🧩 Model Description

This repository contains multiple quantized GGUF versions of the SmolLM3-3B model, converted for efficient inference on CPU and mobile devices.

Quantized by: Bastion-AI
BastionAI Website: https://bastionai.github.io/
Format: GGUF (for llama.cpp and compatible)

BastionAI empowers users with advanced AI capabilities that run entirely on your device. Learn more about our privacy-first, local AI solutions at bastionai.github.io.

🚀 How to Run Locally

You can use these GGUF files with:

Example command:

./main -m SmolLM3-3B.Q4_K_M.gguf -p "Explain gravity in simple terms."

📦 Quantized GGUF Downloads

Quant Type	Disk Size	Download Link	Details
Q2_K	1.2G	Download	Standard 2-bit quant
Q3_K	1.5G	Download	Standard 3-bit quant
Q3_K_S	1.3G	Download	Small 3-bit quant
Q3_K_M	1.5G	Download	Medium 3-bit quant
Q4_0	1.7G	Download	Standard 4-bit quant
Q4_K	1.8G	Download	Standard 4-bit quant
Q4_K_S	1.7G	Download	Small 4-bit quant
Q4_K_M	1.8G	Download	Medium 4-bit quant
Q5_0	2.0G	Download	Standard 5-bit quant
Q5_K	2.1G	Download	Standard 5-bit quant
Q5_K_S	2.0G	Download	Small 5-bit quant
Q5_K_M	2.1G	Download	Medium 5-bit quant
Q6_K	2.4G	Download	Standard 6-bit quant
Q8_0	3.0G	Download	Standard 8-bit quant

🛡️ About BastionAI

BastionAI empowers users with advanced AI capabilities that run entirely on your device. Experience the full potential of AI conversation, document processing, and voice interaction without compromising your privacy or security.

Website: https://bastionai.github.io/
Products: BastionChat, BastionSDK
Mission: Democratizing AI for everyone, privacy-first, open source, cross-platform.

📖 License

This model is distributed under the Apache 2.0 license, as per the original HuggingFaceTB/SmolLM3-3B.

🙏 Credits

Original model by HuggingFaceTB
GGUF conversion and quantization by Bastion-AI
Learn more about BastionAI: https://bastionai.github.io/

🔗 Links

📣 Community & Support

📜 Citation

If you use this model, please cite the original SmolLM3-3B and BastionAI as appropriate.

@misc{smollm3,
  title={SmolLM3-3B},
  author={HuggingFaceTB},
  year={2024},
  url={https://huggingface.co/HuggingFaceTB/SmolLM3-3B}
}

Downloads last month: 158

GGUF

Model size

3B params

Architecture

smollm3

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Bastion-AI/SmolLM3-3B-GGUF

Base model

HuggingFaceTB/SmolLM3-3B-Base

Finetuned

HuggingFaceTB/SmolLM3-3B

Quantized

(72)

this model