Instructions to use QuantFactory/UwU-7B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantFactory/UwU-7B-Instruct-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="QuantFactory/UwU-7B-Instruct-GGUF",
	filename="UwU-7B-Instruct.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use QuantFactory/UwU-7B-Instruct-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

Use Docker

docker model run hf.co/QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use QuantFactory/UwU-7B-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantFactory/UwU-7B-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantFactory/UwU-7B-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

Ollama
How to use QuantFactory/UwU-7B-Instruct-GGUF with Ollama:
```
ollama run hf.co/QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M
```

Unsloth Studio new

How to use QuantFactory/UwU-7B-Instruct-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/UwU-7B-Instruct-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/UwU-7B-Instruct-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for QuantFactory/UwU-7B-Instruct-GGUF to start chatting

Pi new

How to use QuantFactory/UwU-7B-Instruct-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use QuantFactory/UwU-7B-Instruct-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use QuantFactory/UwU-7B-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M
```

Lemonade

How to use QuantFactory/UwU-7B-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QuantFactory/UwU-7B-Instruct-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.UwU-7B-Instruct-GGUF-Q4_K_M

List all available models

lemonade list

lbourdois commited on Apr 27, 2025

Commit

2829274

verified ·

1 Parent(s): 5040073

Improve language tag

Browse files

Hi! As the model is multilingual, this is a PR to add other languages than English to the language tag to improve the referencing. Note that 29 languages are announced in the README, but only 13 are explicitly listed. I was therefore only able to add these 13 languages.

Files changed (1) hide show

README.md +112 -103

README.md CHANGED Viewed

@@ -1,103 +1,112 @@
----
-license: apache-2.0
-datasets:
-- qingy2024/FineQwQ-142k
-language:
-- en
-base_model:
-- Qwen/Qwen2.5-7B
-pipeline_tag: text-generation
----
-[![QuantFactory Banner](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)](https://hf.co/QuantFactory)
-# QuantFactory/UwU-7B-Instruct-GGUF
-This is quantized version of [qingy2024/UwU-7B-Instruct](https://huggingface.co/qingy2024/UwU-7B-Instruct) created using llama.cpp
-# Original Model Card
-## UwU 7B Instruct
-Small QwQ, full-finetuned on FineQwQ-142K. Unlike my previous models, this one is a general-purpose reasoning machine!
-Oh, and it also passes the strawberry test:
-```
-> How many 'r's are in the word strawberry?
-Let's see. I have this word: "strawberry." I need to find out how many 'r's are in it. Okay, first, I should probably write down the word and look at each letter one by one.
-So, "s-t-r-a-w-b-e-r-r-y."
-Now, I need to count the 'r's. Let's go through each letter:
-- The first letter is 's', which is not an 'r'.
-- The second letter is 't', not an 'r'.
-- The third letter is 'r', that's the first 'r'.
-- The fourth letter is 'a', not an 'r'.
-- The fifth letter is 'w', not an 'r'.
-- The sixth letter is 'b', not an 'r'.
-- The seventh letter is 'e', not an 'r'.
-- The eighth letter is 'r', that's the second 'r'.
-- The ninth letter is 'r', that's the third 'r'.
-- The tenth letter is 'y', not an 'r'.
-So, I have three 'r's in the word "strawberry." But wait, let me double-check. Maybe I missed one.
-Let me count again:
-1. 's'
-2. 't'
-3. 'r' (first 'r')
-4. 'a'
-5. 'w'
-6. 'b'
-7. 'e'
-8. 'r' (second 'r')
-9. 'r' (third 'r')
-10. 'y'
-Yes, that's three 'r's. I think that's correct.
-Alternatively, maybe there's a smarter way to count them without listing out each letter. Let's see.
-I know that the word "strawberry" has 10 letters, and I can recall that it has three 'r's. But to be sure, maybe I can look for patterns or something.
-Another way is to use a tally mark or something similar. Let's try that.
-First 'r': | (that's one)
-Second 'r': | (that's two)
-Third 'r': | (that's three)
-Yes, that also gives me three 'r's.
-I think I'm confident with that answer now. There are three 'r's in the word "strawberry."
-**Final Answer**
-\[ \boxed{3} \]
-```

+---
+license: apache-2.0
+datasets:
+- qingy2024/FineQwQ-142k
+language:
+- zho
+- eng
+- fra
+- spa
+- por
+- deu
+- ita
+- rus
+- jpn
+- kor
+- vie
+- tha
+- ara
+base_model:
+- Qwen/Qwen2.5-7B
+pipeline_tag: text-generation
+---
+[![QuantFactory Banner](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)](https://hf.co/QuantFactory)
+# QuantFactory/UwU-7B-Instruct-GGUF
+This is quantized version of [qingy2024/UwU-7B-Instruct](https://huggingface.co/qingy2024/UwU-7B-Instruct) created using llama.cpp
+# Original Model Card
+## UwU 7B Instruct
+Small QwQ, full-finetuned on FineQwQ-142K. Unlike my previous models, this one is a general-purpose reasoning machine!
+Oh, and it also passes the strawberry test:
+```
+> How many 'r's are in the word strawberry?
+Let's see. I have this word: "strawberry." I need to find out how many 'r's are in it. Okay, first, I should probably write down the word and look at each letter one by one.
+So, "s-t-r-a-w-b-e-r-r-y."
+Now, I need to count the 'r's. Let's go through each letter:
+- The first letter is 's', which is not an 'r'.
+- The second letter is 't', not an 'r'.
+- The third letter is 'r', that's the first 'r'.
+- The fourth letter is 'a', not an 'r'.
+- The fifth letter is 'w', not an 'r'.
+- The sixth letter is 'b', not an 'r'.
+- The seventh letter is 'e', not an 'r'.
+- The eighth letter is 'r', that's the second 'r'.
+- The ninth letter is 'r', that's the third 'r'.
+- The tenth letter is 'y', not an 'r'.
+So, I have three 'r's in the word "strawberry." But wait, let me double-check. Maybe I missed one.
+Let me count again:
+1. 's'
+2. 't'
+3. 'r' (first 'r')
+4. 'a'
+5. 'w'
+6. 'b'
+7. 'e'
+8. 'r' (second 'r')
+9. 'r' (third 'r')
+10. 'y'
+Yes, that's three 'r's. I think that's correct.
+Alternatively, maybe there's a smarter way to count them without listing out each letter. Let's see.
+I know that the word "strawberry" has 10 letters, and I can recall that it has three 'r's. But to be sure, maybe I can look for patterns or something.
+Another way is to use a tally mark or something similar. Let's try that.
+First 'r': | (that's one)
+Second 'r': | (that's two)
+Third 'r': | (that's three)
+Yes, that also gives me three 'r's.
+I think I'm confident with that answer now. There are three 'r's in the word "strawberry."
+**Final Answer**
+\[ \boxed{3} \]
+```