Instructions to use SupraLabs/Supra-Router-51M-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SupraLabs/Supra-Router-51M-gguf with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="SupraLabs/Supra-Router-51M-gguf")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("SupraLabs/Supra-Router-51M-gguf", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use SupraLabs/Supra-Router-51M-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SupraLabs/Supra-Router-51M-gguf:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SupraLabs/Supra-Router-51M-gguf:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SupraLabs/Supra-Router-51M-gguf:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf SupraLabs/Supra-Router-51M-gguf:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf SupraLabs/Supra-Router-51M-gguf:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf SupraLabs/Supra-Router-51M-gguf:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf SupraLabs/Supra-Router-51M-gguf:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf SupraLabs/Supra-Router-51M-gguf:Q4_K_M

Use Docker

docker model run hf.co/SupraLabs/Supra-Router-51M-gguf:Q4_K_M

LM Studio
Jan

vLLM

How to use SupraLabs/Supra-Router-51M-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SupraLabs/Supra-Router-51M-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SupraLabs/Supra-Router-51M-gguf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/SupraLabs/Supra-Router-51M-gguf:Q4_K_M

SGLang

How to use SupraLabs/Supra-Router-51M-gguf with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "SupraLabs/Supra-Router-51M-gguf" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SupraLabs/Supra-Router-51M-gguf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "SupraLabs/Supra-Router-51M-gguf" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SupraLabs/Supra-Router-51M-gguf",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use SupraLabs/Supra-Router-51M-gguf with Ollama:
```
ollama run hf.co/SupraLabs/Supra-Router-51M-gguf:Q4_K_M
```

Unsloth Studio

How to use SupraLabs/Supra-Router-51M-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SupraLabs/Supra-Router-51M-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SupraLabs/Supra-Router-51M-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for SupraLabs/Supra-Router-51M-gguf to start chatting

Docker Model Runner
How to use SupraLabs/Supra-Router-51M-gguf with Docker Model Runner:
```
docker model run hf.co/SupraLabs/Supra-Router-51M-gguf:Q4_K_M
```

Lemonade

How to use SupraLabs/Supra-Router-51M-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull SupraLabs/Supra-Router-51M-gguf:Q4_K_M

Run and chat with the model

lemonade run user.Supra-Router-51M-gguf-Q4_K_M

List all available models

lemonade list

Atomic Chat

QyrouNnet-AI commited on Jul 8

Commit

3d08e26

verified ·

1 Parent(s): 58f00ea

Upload 19 files

Browse files

Files changed (20) hide show

.gitattributes +19 -0
Supra-Router-51M-F16.gguf +3 -0
Supra-Router-51M-IQ3_M.gguf +3 -0
Supra-Router-51M-IQ3_S.gguf +3 -0
Supra-Router-51M-IQ4_NL.gguf +3 -0
Supra-Router-51M-IQ4_XS.gguf +3 -0
Supra-Router-51M-Q1_0.gguf +3 -0
Supra-Router-51M-Q2_K.gguf +3 -0
Supra-Router-51M-Q3_K_L.gguf +3 -0
Supra-Router-51M-Q3_K_M.gguf +3 -0
Supra-Router-51M-Q3_K_S.gguf +3 -0
Supra-Router-51M-Q4_0.gguf +3 -0
Supra-Router-51M-Q4_1.gguf +3 -0
Supra-Router-51M-Q4_K_M.gguf +3 -0
Supra-Router-51M-Q4_K_S.gguf +3 -0
Supra-Router-51M-Q5_K_M.gguf +3 -0
Supra-Router-51M-Q6_K.gguf +3 -0
Supra-Router-51M-Q8_0.gguf +3 -0
Supra-Router-51M-TQ1_0.gguf +3 -0
Supra-Router-51M-TQ2_0.gguf +3 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,22 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-F16.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q1_0.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q4_1.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-TQ1_0.gguf filter=lfs diff=lfs merge=lfs -text
+Supra-Router-51M-TQ2_0.gguf filter=lfs diff=lfs merge=lfs -text

Supra-Router-51M-F16.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fef7da6d9981d741aa77e168a6ecd0eee7430957652fb7e8a689f76019329a89
+size 104740416

Supra-Router-51M-IQ3_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:949df81f89163298accce33bd078749e5ae2cd5739ed935cd83535d1a590e717
+size 31705664

Supra-Router-51M-IQ3_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c55abca86789a56d3d3a38fc95b7da3896fcddc5844f30504d036d1e9775a4c3
+size 30988864

Supra-Router-51M-IQ4_NL.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0ec187179684ffa31aa9d3f51c60d8062f5e9303033ab7300db144d776b04f93
+size 34675264

Supra-Router-51M-IQ4_XS.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c5fd7192df16ce2228cf1e58ab9dea66d283c9537e09287166ae4ea75fd0feba
+size 33839680

Supra-Router-51M-Q1_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bacc6e575b4eee094f270c038e68a40b05bae98400a65a05ded48b251178f934
+size 19610176

Supra-Router-51M-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f4281d243bd3fa2405370ea73d752539908f9e665093cd03d7870be639137a8d
+size 28752448

Supra-Router-51M-Q3_K_L.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2426a5fdd1de088a82981e1792b17318cee62d6e404109bd467f81d1dbf46840
+size 33827392

Supra-Router-51M-Q3_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a756e7d266a9313288eb4b54a79a6a60c119d4942e2f61bc750210b02c8a6dce
+size 32729664

Supra-Router-51M-Q3_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cd85d491c0e41e7248e87aec483c3ca33b93d186fa259dd8482225808cbe0b1c
+size 30988864

Supra-Router-51M-Q4_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:22d0b582c9eee1c1d3514017d73d4ef0ec4417af1faab5aed90e8e1934a0a81e
+size 34540096

Supra-Router-51M-Q4_1.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2a8c877e49a62fe9424603297f1349fe1b663c017a7a799e53413b21b0814015
+size 36751936

Supra-Router-51M-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:f762404fd9e940b64c1f95b50e155724d4d90adc43f2652d42dd13f95943aebf
+size 37446208

Supra-Router-51M-Q4_K_S.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e7e00fb017eb1f92ef8a7585186f19eadfbf95852551114c413f09143f92011e
+size 35732032

Supra-Router-51M-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e52d1ad9c74f04bb08f6e71031ab7ab4fc8bab1149e7c65d73ab13e41247f60a
+size 40960576

Supra-Router-51M-Q6_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:20ad7f36feb62948685eeeb7e409731235e803c464e6c68c359b44ef91c55bc2
+size 45759040

Supra-Router-51M-Q8_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:853f6629561aa34883453ccd838f49bb73f3a6f77a7f8fb7a85addba2f46b9d8
+size 56202816

Supra-Router-51M-TQ1_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c1793ecbee3445c5995496f637a38fdad268961047b53a20011ba439985230d8
+size 25139776

Supra-Router-51M-TQ2_0.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:730c16bf284c8bd45c055eda930736aaa332fe87d3718bf9378d0754e54d5243
+size 26393152