Instructions to use BaunRobotics/Qwen-tinybaun-k12-test-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use BaunRobotics/Qwen-tinybaun-k12-test-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M

Use Docker

docker model run hf.co/BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use BaunRobotics/Qwen-tinybaun-k12-test-GGUF with Ollama:
```
ollama run hf.co/BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M
```

Unsloth Studio

How to use BaunRobotics/Qwen-tinybaun-k12-test-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for BaunRobotics/Qwen-tinybaun-k12-test-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for BaunRobotics/Qwen-tinybaun-k12-test-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for BaunRobotics/Qwen-tinybaun-k12-test-GGUF to start chatting

Docker Model Runner
How to use BaunRobotics/Qwen-tinybaun-k12-test-GGUF with Docker Model Runner:
```
docker model run hf.co/BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M
```

Lemonade

How to use BaunRobotics/Qwen-tinybaun-k12-test-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull BaunRobotics/Qwen-tinybaun-k12-test-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen-tinybaun-k12-test-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

BaunRobotics commited on Jul 8, 2024

Commit

f8b80ce

verified ·

1 Parent(s): 3f0b002

Upload folder using huggingface_hub

Browse files

Files changed (16) hide show

.gitattributes +15 -0
Q2_K.gguf +3 -0
Q3_K.gguf +3 -0
Q3_K_M.gguf +3 -0
Q3_K_S.gguf +3 -0
Q4_0.gguf +3 -0
Q4_1.gguf +3 -0
Q4_K_M.gguf +3 -0
Q4_K_S.gguf +3 -0
Q5_0.gguf +3 -0
Q5_1.gguf +3 -0
Q5_K.gguf +3 -0
Q5_K_M.gguf +3 -0
Q5_K_S.gguf +3 -0
Q6_K.gguf +3 -0
Q8_0.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -34,3 +34,18 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 FP16.gguf filter=lfs diff=lfs merge=lfs -text

 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 FP16.gguf filter=lfs diff=lfs merge=lfs -text
+Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Q3_K.gguf filter=lfs diff=lfs merge=lfs -text
+Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+Q5_1.gguf filter=lfs diff=lfs merge=lfs -text
+Q5_K.gguf filter=lfs diff=lfs merge=lfs -text
+Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c943834d5ed8f8977903177eb8c53852b7dbc973d195228a6818650c1b2bbc4b
+size 338321728

Q3_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bad494e171ceae19ea4388cb704f98010a87554168746c2bca7ee49eb4c5b39f
+size 355180864

Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bad494e171ceae19ea4388cb704f98010a87554168746c2bca7ee49eb4c5b39f
+size 355180864

Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:35af871404c6be5144d98e586a75dc5c32b663b801ba96f946907f3976b2ef29
+size 337977664

Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2153c9027f723ff7c0ca182405c95f8c5d18c9f620697b32bf3f4647e9416389
+size 351869248

Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c4851085990f3c994e1d3ab62ba69d253e8a506e3f74776f6f2286e187ae25f6
+size 374233408

Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:022146d579182f88f7c1faaaa55b546e0c96a04af4d415b7b1d7f826c1eeb3b6
+size 397522240

Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4d05d1dea82701e661cc71f6f6c657ba6c78990554421c1d78aad2b474b4b6ee
+size 385186112

Q5_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:342f06c16e2c7dcd862c94c3c28c47cf22e6da0c18d913dc12554c8458f7f969
+size 396597568

Q5_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8fb39208b435eb496b4ec3abf1b4610098de03fd31967f5f8f32f0beba878641
+size 418961728

Q5_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:eacaebd8a7420ca1dddc4d999ec98d8defde105a377ef55a8a038bfe8bc1af0a
+size 419800384

Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:eacaebd8a7420ca1dddc4d999ec98d8defde105a377ef55a8a038bfe8bc1af0a
+size 419800384

Q5_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:5f860a3910001e3899a3909414d3a15030485bfd9a5e717ca5d9ac69be75162a
+size 412424512

Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c3ab958e2423d28a080a122b51bb0cbddc67ef5f6d6d983073f6f6161cb0877c
+size 505450816

Q8_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1b9e46199d5d2872bec2d123b3b28af5d4a9baf6b1c4c615c5e2d4e3d1f6710d
+size 530782528