Instructions to use texdata/Sokol-4B-SLO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use texdata/Sokol-4B-SLO with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="texdata/Sokol-4B-SLO")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("texdata/Sokol-4B-SLO")
model = AutoModelForMultimodalLM.from_pretrained("texdata/Sokol-4B-SLO", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use texdata/Sokol-4B-SLO with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf texdata/Sokol-4B-SLO:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf texdata/Sokol-4B-SLO:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf texdata/Sokol-4B-SLO:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf texdata/Sokol-4B-SLO:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf texdata/Sokol-4B-SLO:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf texdata/Sokol-4B-SLO:Q4_K_M

Use Docker

docker model run hf.co/texdata/Sokol-4B-SLO:Q4_K_M

LM Studio
Jan

vLLM

How to use texdata/Sokol-4B-SLO with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "texdata/Sokol-4B-SLO"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "texdata/Sokol-4B-SLO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/texdata/Sokol-4B-SLO:Q4_K_M

SGLang

How to use texdata/Sokol-4B-SLO with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "texdata/Sokol-4B-SLO" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "texdata/Sokol-4B-SLO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "texdata/Sokol-4B-SLO" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "texdata/Sokol-4B-SLO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use texdata/Sokol-4B-SLO with Ollama:
```
ollama run hf.co/texdata/Sokol-4B-SLO:Q4_K_M
```

Unsloth Studio

How to use texdata/Sokol-4B-SLO with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for texdata/Sokol-4B-SLO to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for texdata/Sokol-4B-SLO to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for texdata/Sokol-4B-SLO to start chatting

How to use texdata/Sokol-4B-SLO with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "texdata/Sokol-4B-SLO:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use texdata/Sokol-4B-SLO with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default texdata/Sokol-4B-SLO:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use texdata/Sokol-4B-SLO with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf texdata/Sokol-4B-SLO:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "texdata/Sokol-4B-SLO:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use texdata/Sokol-4B-SLO with Docker Model Runner:
```
docker model run hf.co/texdata/Sokol-4B-SLO:Q4_K_M
```

Lemonade

How to use texdata/Sokol-4B-SLO with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull texdata/Sokol-4B-SLO:Q4_K_M

Run and chat with the model

lemonade run user.Sokol-4B-SLO-Q4_K_M

List all available models

lemonade list

Sokol-4B-SLO / README.md

texdata

Sokol 4B (SLO) — initial release

41b5ce9 3 days ago

preview code

Raw

History Blame Contribute Delete

6.39 kB

	---
	language:
	- sl
	- en
	license: other
	license_name: mixed-see-provenance
	library_name: transformers
	base_model: SicariusSicariiStuff/Qwen3.5-4B_Abliterated
	pipeline_tag: text-generation
	tags:
	- slovenian
	- qwen3
	- assistant
	- on-device
	- tool-calling
	- reasoning
	- gguf
	---

	# Sokol — Slovenian Assistant LLM (4B, on-device)

	Sokol (falcon) is a small 4B Slovenian assistant built for fully-local, on-device use
	(desktop via GGUF / llama.cpp / LM Studio). It chats in fluent Slovenian, translates EN↔SL, reasons
	through tasks (`<think>`), and makes native tool / function calls (Qwen3.6 XML format).

	It is a text LLM, not a speech model — pair it with your own ASR + TTS to build a local voice
	assistant (it routes `<think>` to `reasoning_content`, so reasoning is never spoken).

	This single repo ships both the full-precision weights (safetensors) and quantized GGUFs
	(under [`gguf/`](./gguf)), so you can run it with transformers, or fully local with llama.cpp / LM Studio.

	## What it is

	- Base: `SicariusSicariiStuff/Qwen3.5-4B_Abliterated` (dense, `qwen3_5` arch, natively supported
	in transformers ≥ 5.6; multimodal-capable).
	- Phase 2 — CPT (full-parameter): continued pretraining on ~1.78 B tokens of Slovenian text
	for language adaptation (full-FT, DeepSpeed ZeRO-2, 1 epoch).
	- Phase 3 — SFT (LoRA): chat + translation + tool-calling + reasoning + identity (~250 k examples).
	- Deployment: GGUF (desktop, llama.cpp / LM Studio) and full safetensors (transformers).

	## Files in this repo

	\| path \| format \| use \|
	\|---\|---\|---\|
	\| `*.safetensors` (+ `config.json`, tokenizer, `chat_template.jinja`) \| full BF16 \| transformers, further fine-tuning, requantization \|
	\| `gguf/sokol-4b-slo-Q8_0.gguf` \| GGUF Q8_0 \| highest-quality local (largest) \|
	\| `gguf/sokol-4b-slo-Q6_K.gguf` \| GGUF Q6_K \| near-lossless, smaller \|
	\| `gguf/sokol-4b-slo-Q5_K_M.gguf` \| GGUF Q5_K_M \| balanced \|
	\| `gguf/sokol-4b-slo-Q4_K_M.gguf` \| GGUF Q4_K_M \| smallest, fastest \|

	> GGUFs are exported with `--no-mtp` (the multi-token-prediction head is dropped), which is required
	> for the `qwen3_5` GDN-hybrid architecture to load cleanly in current llama.cpp.

	## Run it (GGUF, llama.cpp)

	```bash
	# thinking + native tool calls; --reasoning-budget routes <think> to reasoning_content
	llama-server -m gguf/sokol-4b-slo-Q5_K_M.gguf --jinja --reasoning-budget 8192 \
	-c 8192 --temp 0.4 --top-p 0.9
	```

	Or open the `.gguf` directly in LM Studio. For clean, non-thinking output pass
	`--reasoning-budget 0` (useful when driving TTS in a voice stack, so `<think>` is never spoken).

	## Capabilities

	- Slovenian chat — fluent, formal Slovenian conversation.
	- Translation — bidirectional EN↔SL.
	- Reasoning — emits `<think>` traces (distilled from a large Qwen3.8-Max teacher); pass
	`enable_thinking=False` for direct answers.
	- Tool / function calling — native Qwen3.6 XML, e.g.
	`<tool_call><function=get_weather><parameter=city>Ljubljana</parameter></function></tool_call>`.
	- Identity — presents as Sokol, a local Slovenian assistant.

	## Training data (provenance)

	### CPT — Slovenian text (~1.68 M docs / ~1.78 B tokens)
	\| dataset \| docs \| source \| license \|
	\|---\|---\|---\|---\|
	\| `sl_wiki` \| 160,539 \| Slovenian Wikipedia \| CC BY-SA \|
	\| `sl_fineweb` \| 1,523,000 \| FineWeb2 (sl subset) \| ODC-BY \|

	### SFT — ~250 k examples
	\| dataset \| examples \| what \| source \|
	\|---\|---\|---\|---\|
	\| `sft_chat_sl` \| 79,324 \| Slovenian conversation \| `cjvt/GaMS-Nemotron-Chat` \|
	\| `sft_translate` \| 120,000 \| bidirectional EN↔SL translation \| OPUS-derived \|
	\| `sft_toolcall_hermes` \| 5,086 \| tool calling \| Hermes function-calling \|
	\| `sft_toolcall_apigen_sokol` \| 4,829 \| tool calling (length-filtered) \| APIGen-MT \|
	\| `sft_toolcall_toolace` \| 9,171 \| tool calling \| ToolACE \|
	\| `qwen38max_reasoning_sl` \| 31,255 \| reasoning `<think>` \| distilled from Qwen3.8-Max \|
	\| `identity_sokol_sl` \| 56 \| Sokol identity \| hand-written \|

	Nothing is truncated: all SFT examples are ≤ 7942 content tokens (cutoff 8192).

	## Evaluation

	### Slovenian-LLM-Eval — acc_norm (`cjvt/slovenian-llm-eval`, 200/task)

	Sokol (CPT + SFT) vs the abliterated base, same eval, via `flywheel-sl`:

	\| task \| base Qwen3.5-4B \| Sokol \| Δ \|
	\|---\|---\|---\|---\|
	\| arc_easy \| 0.565 \| 0.610 \| +0.045 \|
	\| arc_challenge \| 0.425 \| 0.450 \| +0.025 \|
	\| hellaswag \| 0.460 \| 0.550 \| +0.090 \|
	\| piqa \| 0.605 \| 0.665 \| +0.060 \|
	\| openbookqa \| 0.400 \| 0.445 \| +0.045 \|
	\| winogrande \| 0.525 \| 0.590 \| +0.065 \|
	\| boolq \| 0.825 \| 0.825 \| +0.000 \|
	\| AVG \| 0.544 \| 0.591 \| +0.047 \|

	Wins 6/7 tasks, ties 1, no regressions. CPT also halved Slovenian perplexity
	(6.996 → 3.438 on held-out SL text, −50.9%).

	### Judge track — chat quality (1–5)

	Open-ended answers to 32 general Slovenian questions (8 categories), judged by a stronger **27B
	Slovenian judge model** on correctness / helpfulness / language:

	\| metric \| base Qwen3.5-4B \| Sokol \| Δ \|
	\|---\|---\|---\|---\|
	\| jezik (fluency) \| 4.88 \| 4.97 \| +0.09 \|
	\| koristnost (helpfulness) \| 4.84 \| 4.81 \| −0.03 \|
	\| pravilnost (correctness) \| 4.78 \| 4.63 \| −0.16 \|

	Sokol clearly wins language fluency and tool-calling (per-category +0.67) and the objective
	acc_norm; the small aggregate correctness dip comes almost entirely from two n=4 categories
	(grammar, instruction-format), i.e. 1–2 questions each — noise at this eval size, not a systematic
	regression. (v1 is 32 questions; per-category deltas need a larger v2 to be reliable.)

	## ⚠️ Disclaimers & license

	- General assistant. Outputs may be incorrect or fabricated; not professional advice.
	Verify anything important.
	- Mixed data-license provenance (Wikipedia CC BY-SA, FineWeb2 ODC-BY, GaMS-Nemotron-Chat, OPUS,
	Hermes/APIGen/ToolACE, Qwen3.8-Max distillation). Marked `license: other`; **resolve data terms
	before public/commercial use.** Derivative of Qwen3.5-4B — the upstream Qwen license also applies.
	- Provided AS IS, no warranty. See `LICENSES.md`.

	## Citation

	```
	@misc{sokol_4b_slovenian_assistant,
	title = {Sokol: a small on-device Slovenian assistant LLM (4B)},
	author = {Tadej Fius},
	year = {2026},
	note = {Derivative of Qwen3.5-4B. Slovenian CPT+SFT. General Slovenian assistant LLM.}
	}
	```