Instructions to use bartowski/Llama-3.2-1B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use bartowski/Llama-3.2-1B-Instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use bartowski/Llama-3.2-1B-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bartowski/Llama-3.2-1B-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bartowski/Llama-3.2-1B-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Ollama
How to use bartowski/Llama-3.2-1B-Instruct-GGUF with Ollama:
```
ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M
```

Unsloth Studio

How to use bartowski/Llama-3.2-1B-Instruct-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bartowski/Llama-3.2-1B-Instruct-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bartowski/Llama-3.2-1B-Instruct-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for bartowski/Llama-3.2-1B-Instruct-GGUF to start chatting

How to use bartowski/Llama-3.2-1B-Instruct-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use bartowski/Llama-3.2-1B-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M
```

Lemonade

How to use bartowski/Llama-3.2-1B-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Llama-3.2-1B-Instruct-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use bartowski/Llama-3.2-1B-Instruct-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use bartowski/Llama-3.2-1B-Instruct-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Hello, I have some issues with my CMD Ollama while installing this model

by NCGWRjason - opened Nov 21, 2024

Discussion

NCGWRjason

Nov 21, 2024

Hello,

I followed the official CMD installation instructions to install your Llama-3.2-1B-Instruct-GGUF model from Hugging Face using the following command:
https://huggingface.co/docs/hub/ollama

CMD query
ollama run hf.co/mlabonne/Meta-Llama-3.1-8B-Instruct-abliterated-GGUF

I noticed that the Llama-3.2-1B-Instruct-GGUF repository contains multiple GGUF files.

However, after downloading, I found only one model file, specifically:
hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:latest 807MB

Could you please clarify which specific GGUF file this corresponds to and why only one file was downloaded?

Thank you.

deleted

Nov 21, 2024

•

edited Nov 21, 2024

Not that it helps you directly but if i dont create it myself, i would normally just download the GGUF i want manually from HF to my machine. Then I import that into whatever tool i want to use, depending on its needs ( openwebui, oobas text gen, llamafile, my own code, whatever ). Doing it that way always works for me and i have a backup of the file in the process i can store.

I don't normally try using the 'tools utilities' to download for me

huggingkot

Nov 22, 2024

Could you please clarify which specific GGUF file this corresponds to and why only one file was downloaded?

RTFM?
From https://huggingface.co/docs/hub/ollama that you linked:

Custom Quantization

By default, the Q4_K_M quantization scheme is used, when it’s present inside the model repo. If not, we default to picking one reasonable quant type present inside the repo.

To select a different scheme, simply:

From Files and versions tab on a model page, open GGUF viewer on a particular GGUF file.

Choose ollama from Use this model dropdown.

The snippet would be in format (quantization tag added):

ollama run hf.co/{username}/{repository}:{quantization}

this is clear, which part you don't understand?

However, after downloading, I found only one model file, specifically:
hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:latest 807MB

you downloaded Q4_K_M.

NCGWRjason

Nov 22, 2024

Could you please clarify which specific GGUF file this corresponds to and why only one file was downloaded?

RTFM?
From https://huggingface.co/docs/hub/ollama that you linked:

Custom Quantization

By default, the Q4_K_M quantization scheme is used, when it’s present inside the model repo. If not, we default to picking one reasonable quant type present inside the repo.

To select a different scheme, simply:

From Files and versions tab on a model page, open GGUF viewer on a particular GGUF file.

Choose ollama from Use this model dropdown.

The snippet would be in format (quantization tag added):

ollama run hf.co/{username}/{repository}:{quantization}

this is clear, which part you don't understand?

However, after downloading, I found only one model file, specifically:
hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:latest 807MB

you downloaded Q4_K_M.

I means I use the CMD terminal query,
ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF

The /bartowski/Llama-3.2-1B-Instruct-GGUF repository contains multiple GGUF files. (https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/tree/main)

However, after downloading, I found only one model file in the ollama list, specifically:
hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF:latest 807MB

So, I specifically ran bartowski/Llama-3.2-1B-Instruct-GGUF, but it only downloaded Q4_K_M.gguf.
Is this the default model being downloaded? Or is it because this was the most recently uploaded version?
That’s why it appears as Llama-3.2-1B-Instruct-GGUF:latest in the ollama list.

Thank you

bartowski

Owner Nov 22, 2024

Is there a reason you want to download all the sizes..?

Q4_K_M is just the default that ollama uses

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment