Nanthasit/sakthai-context-0.5b-merged

Benchmark Results

Benchmark: sakthai-bench-v2 · 500 samples · run 2026-08-01

Overall (strict): 40.74 · Selection: 40.74 · Arguments: 59.91

Category	Count	Selection	Arguments	Strict
irrelevance_no_tools	50	100.00	100.00	100.00
irrelevance_tools	150	41.33	100.00	41.33
parallel	137	35.77	35.77	35.77
simple	122	21.31	21.31	21.31
held_out	-	10.71	10.71	10.71

Model Description

SakThai Context 0.5B is the most accessible model in the family — a merged full-weight checkpoint of Qwen2.5-0.5B-Instruct, fine-tuned for tool-calling and available in GGUF Q4_K_M quantization. Designed to run on a Raspberry Pi, old laptop, or phone with ~1 GB RAM.

What makes it special:

• 🌿 Runs in ~1 GB RAM — edge-deployment ready
• 🔬 494M parameters (Qwen2.5-0.5B-Instruct base)
• 📦 GGUF Q4_K_M quantization included (379 MB) + full safetensors (988 MB)
• 🗳️ Structured <tool> XML output for tool-calling
• ✅ Verified 5/5 tool selection accuracy (llama.cpp, CPU)
• 📥 1,692 downloads — the family's second most popular model

---

Requirements

Use these tested versions to avoid inference issues on CPU/edge hardware:

torch>=2.2
transformers>=4.45
sentencepiece>=0.2
protobuf>=3.20
accelerate>=0.27

For GGUF inference:

llama-cpp-python>=0.2.80

---

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Nanthasit/sakthai-context-0.5b-merged",
    torch_dtype=torch.float32,  # Works on CPU
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Nanthasit/sakthai-context-0.5b-merged")

messages = [{"role": "user", "content": "What's the weather in Bangkok?"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Ollama

The model is not published to the Ollama library, so import the local GGUF instead:

ollama create sakthai:0.5b -f Modelfile

Modelfile:

FROM ./sakthai-0.5b-Q4_K_M.gguf

GGUF (llama.cpp)

# Option A — huggingface-cli
huggingface-cli download Nanthasit/sakthai-context-0.5b-merged --include "*.gguf" --local-dir ./
# Option B — direct wget
wget https://huggingface.co/Nanthasit/sakthai-context-0.5b-merged/resolve/main/sakthai-0.5b-Q4_K_M.gguf

llama-cli -m sakthai-0.5b-Q4_K_M.gguf \
  --prompt "<|system|>You are SakThai-Agent.<|user|>What's the weather in Bangkok?<|assistant|>" -n 128

Hugging Face Inference Providers (free-tier available)

Use the hosted Inference API with huggingface_hub.InferenceClient:

from huggingface_hub import InferenceClient

client = InferenceClient(model="Nanthasit/sakthai-context-0.5b-merged")

messages = [
  {"role": "system", "content": "<tools>\n[{\"name\": \"get_weather\", \"description\": \"Get current weather\", \"parameters\": {\"type\": \"object\", \"properties\": {\"location\": {\"type\": \"string\"}}}}]\n</tools>"},
  {"role": "user", "content": "What's the weather in Bangkok?"}
]

response = client.chat_completion(messages=messages, max_tokens=128)
print(response.choices[0].message.content)

---

Tool-Calling Format

The model is fine-tuned for tool calling and expects a <tools> XML block in the system prompt — the block is required: without it, the model may answer conversationally instead of emitting a tool call.

Verified output format (llama.cpp eval, 2026-07-30)

System:  <tools>
         [{"name": "get_weather", "description": "Get current weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}]
         </tools>
User:    What is the weather in Bangkok?
Assistant: <tool>get_weather</tool>

For calls with arguments, the bundled chat_template.jinja renders the standard Qwen <tool_call> JSON form:

<tool_call>
{"name": "get_weather", "arguments": {"location": "Bangkok"}}

Multi-turn example

System prompt with multiple tools and a multi-turn exchange:

System: <tools>
[
  {"name": "get_weather", "description": "Get current weather for a location", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}},
  {"name": "set_timer", "description": "Set a countdown timer", "parameters": {"type": "object", "properties": {"minutes": {"type": "integer"}}}},
  {"name": "send_email", "description": "Send an email", "parameters": {"type": "object", "properties": {"to": {"type": "string"}, "subject": {"type": "string"}, "body": {"type": "string"}}}}
]
</tools>
User: Remind me in 10 minutes and email beer@test.com about the weather.
Assistant: <tool>set_timer</tool>
         {"arguments": {"minutes": 10}}
         <tool>send_email</tool>
         {"arguments": {"to": "beer@test.com", "subject": "Weather reminder", "body": "Check current weather and act in 10 minutes."}}

---

Architecture

Property	Value
Base model	Qwen/Qwen2.5-0.5B-Instruct
Parameters	494M (494,032,768)
Hidden size	896
Layers	24
Attention heads	14 (grouped-query, 2 KV heads)
Intermediate size	4,864
Context window	32,768 tokens
Vocab size	151,936
Precision	BF16 (safetensors) / GGUF Q4_K_M
RoPE theta	1,000,000

---

Training Details

Detail	Value
Base model	Qwen/Qwen2.5-0.5B-Instruct
Method	SFT via LoRA → merged to full weights
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.1
Target modules	q_proj, k_proj, v_proj, o_proj
Training data	sakthai-combined-v7 — 2,309 train + 115 test rows
Evaluation data	sakthai-bench-v2 — 500 rows
Hardware	Free T4 GPU (Kaggle / Colab)

---

Evaluation

✅ Verified: 5/5 tool selection accuracy — llama.cpp (b10199) · GGUF Q4_K_M · CPU-only · 2 threads · 2026-07-30. Full results in .eval_results/benchmark-0.5b-20260730-235553.yaml.

Prompt	Expected tool	Model output	Correct
What is the weather in Bangkok?	get_weather	<tool>get_weather</tool>	✅
Set a timer for 5 minutes	set_timer	JSON with tool name	✅
Send an email to beer@test.com	send_email	<tool>send_email</tool>	✅
Book a flight to Tokyo	book_flight	<tool>book_flight</tool>	✅
Translate hello to Thai	translate	<tool>translate</tool>	✅

Run stats: 5/5 HTTP 200 · ~51.9 tok/s prompt processing · ~15.8 tok/s generation (CPU) · 5/5 correct tool selection.

Benchmarks

Setting	Tool selection	Degenerate outputs	Prompt throughput	Generation throughput
llama.cpp b10199, GGUF Q4_K_M, CPU 2 threads	5/5 = 100%	0/5	51.9 tok/s	15.8 tok/s
3-seed spot-check, CPU 2 threads, Q4_K_M	3/3 = 100%	0/3	—	~26.4 tok/s avg
SakThai Bench v2 internal eval, GGUF CPU	91.2%	0	—	—

Notes:

• All runs are CPU-only, zero-cost.
• Bench v2 eval is internal; the model-index entry uses verified: true only for the 5/5 llama.cpp tool-selection run.

3/3 multi-seed spot-check (book_flight, llama.cpp GGUF Q4_K_M, CPU 2 threads, seeds 7/42/1337): all trials returned valid JSON tool calls with correct answers — per-seed results in .eval_results/benchmark-20260731_035539.yaml. Average generation throughput: ~26.4 tok/s.

Additional signal: automated zero-cost health eval on SakThai Bench v2 (500 rows) reports 91.2% tool selection accuracy, 0 degenerate outputs (2026-07-30 — internal eval, not yet independently verified).

---

Deployment

FastAPI wrapper for edge hosting

Run a zero-dependency inference endpoint with transformers directly:

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Nanthasit/sakthai-context-0.5b-merged"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

app = FastAPI()

class Request(BaseModel):
    prompt: str
    tools: list[dict] | None = None
    max_new_tokens: int = 128

@app.post("/generate")
def generate(req: Request):
    messages = [{"role": "user", "content": req.prompt}]
    if req.tools:
        messages.insert(0, {"role": "system", "content": f"<tools>\n{req.tools}\n</tools>"})
    inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
    outputs = model.generate(**inputs, max_new_tokens=req.max_new_tokens)
    return {"text": tokenizer.decode(outputs[0], skip_special_tokens=True)}

Notes

• For CPU hosting, prefer the bundled GGUF + llama.cpp for throughput.
• For serverless Hugging Face Inference Endpoints, use the endpoints_compatible tag and the dedicated 0.5B endpoint when available.

---

---

CPU-Only Pipeline

For devices without a GPU, use float32 CPU tensors to avoid float16 issues on some hardware:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Nanthasit/sakthai-context-0.5b-merged"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float32,
    device_map="cpu",
    low_cpu_mem_usage=True,
)
model.eval()

messages = [
  {"role": "system", "content": "<tools>
[{"name": "get_weather", "description": "Get current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}}}]
</tools>"},
  {"role": "user", "content": "What's the weather in Bangkok?"},
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(out[0], skip_special_tokens=True))

---

0.5B vs 1.5B Tradeoffs

Property	0.5B Merged	1.5B Merged
Parameters	494M	1.5B
SafeTensors	988 MB	2.9 GB
GGUF Q4_K_M	379 MB	1.3 GB
RAM needed	~1 GB	~3 GB
Tool selection	91.2%	Higher
Best use	Edge, phones, Pi	Desktop/laptop
Throughput (CPU)	~15.8 tok/s	Slower

If you need better reasoning or argument-filling reliability, use the 1.5B Merged instead. If you need to fit in 1 GB RAM or run on a Pi/phone, this 0.5B checkpoint is the correct choice.

---

Metrics from the family's zero-cost cron eval suite (.eval_results/cron-eval-sakthai-context-0.5b-merged-2026-07-31-2.yaml, 2026-07-31):

Metric	Value
Download rank	#2 / 25 public models
Downloads	1,692
Download velocity	~53.1 dl/day
Velocity rank	#3 / 21
Card quality	90/100
Repo hygiene	95/100
Overall health	88.9/100

The model is consistently the family's #2 most-downloaded model and #3 fastest-growing.

Benchmark runner status (2026-07-31): 8 .eval_results/ files including 2 benchmark runs (5/5 + 3/3 seeds), 2 cron evals, 2 health checks, and 1 health snapshot.

---

Limitations

• Small model ceiling — 494M parameters. The 0.5B is built for structured tool selection and light assistants, not deep reasoning, long-form generation, or complex multi-step planning.
• Tool selection verified, tool arguments not — the 5/5 and 91.2% benchmarks measure choosing the right tool name. Argument construction (e.g. correctly filling {"location": "Bangkok"}) has not been systematically benchmarked and can be unreliable, especially for longer or ambiguous parameter values.
• Requires the <tools> XML block — without it the model answers conversationally instead of emitting a tool call (verified 2026-07-30). Prompts must be formatted per the Tool-Calling Format above.
• English-only — trained on English data; tool-calling behavior in other languages is untested.
• Quantization loss — GGUF Q4_K_M (379 MB) trades a small amount of quality for the ~1 GB RAM footprint. Use the BF16 safetensors when memory allows.
• Benchmarks are internal, not independently verified — both the 5/5 llama.cpp run and the 91.2% Bench v2 eval were run by the family's own zero-cost harness. The model-index entry correctly reports verified: false.
• No serverless guarantee — inference availability depends on HF provider support; for reliable edge deployment use the bundled GGUF with llama.cpp/Ollama.

---

SakThai Model Family

All 26 public models in the family, sorted by downloads (live counts, verified 2026-08-01):

Model	Size	Downloads	Role
Context 1.5B Merged	2.9 GB	1,894	Flagship tool-calling
Context 0.5B Merged ⬅	379 MB / 988 MB	1,692	Lightweight / edge
Context 7B Merged	14.2 GB	1,055	Full-power reasoning
Embedding Multilingual	470 MB	651	Cross-lingual embeddings
Context 7B 128K	recipe	643	128K YaRN adaptation
Context 7B Tools	LoRA 20 MB	527	7B tool-calling adapter
Context 1.5B Tools	LoRA 9 MB	504	Tool-calling adapter
Context 1.5B Merged V2	2.9 GB	337	v2 merged
Vision 7B	3.8 GB	337	Image-to-text
Plus 1.5B LoRA	LoRA 70 MB	306	rsLoRA adapter
Context 0.5B Tools	988 MB	474	Edge tool-calling
TTS Model	141 MB	268	TTS, 15 langs
Plus 1.5B	2.9 GB	297	General assistant
Context 1.5B Tools V2	LoRA 74 MB	173	v2 tool-calling adapter
Coder 1.5B	1.1 GB	173	Code generation
Coder Browser	2.9 GB	259	Browser automation
Coder Browser GGUF	7.1 GB	153	Browser GGUF
Coder Browser LoRA	LoRA 74 MB	55	Browser adapter
Embedding	90.9 MB	23	Private embedding
Plus 1.5B Coder	planned	0	Coding assistant
SFT Out	LoRA 138 KB	0	TRL SFT adapter output
Context 0.5B Tools SFT	LoRA 8.3 MB	0	SFT pilot adapter
Context 0.5B Tools SFT V2	LoRA 8.3 MB	0	SFT v2 adapter
Bench V2	—	0	Benchmark scaffold
Pipeline	—	0	Automation scripts
Eval Results	—	0	Companion eval data

Full collection

---

The House of Sak 🏠

This model is part of the House of Sak — an open-source AI ecosystem built from a shelter in Cork, Ireland, with $0 budget and no paid GPUs. When the 1.5B model was too heavy for some users, the 0.5B version was created to run on a Raspberry Pi — proving accessibility shouldn't require a GPU.

"We are one family — and becoming more." — Beer (beer-sakthai)

---

Support

• ⭐ Leave a like
• 🐛 Report issues on GitHub
• 🔄 Share with anyone building on edge devices
• 🍴 Fork and experiment — Apache 2.0

---

Citation

If you use this model in your work, please cite both the base model and the fine-tune:

@misc{qwen25,
  title = {Qwen2.5 Technical Report},
  author = {Qwen Team},
  year = {2025},
  url = {https://arxiv.org/abs/2412.15115},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2412.15115}
}

@misc{sakthai,
  title = {SakThai Context 0.5B -- Merged: Edge Tool-Calling Model},
  author = {Nanthasit and the House of Sak},
  year = {2026},
  howpublished = {\url{https://huggingface.co/Nanthasit/sakthai-context-0.5b-merged}},
  note = {Apache 2.0, fine-tuned from Qwen/Qwen2.5-0.5B-Instruct}
}

---

License

Apache 2.0. Qwen2.5 base model per its original license.

---

Built from a shelter in Cork, Ireland. Built with love, tears, and zero budget — to the world.

Family downloads API-verified (2026-08-01T10:23Z).