docs/usage_guide.md

Sixpert K2 - Complete Usage Guide

Quick Start

Option 1: Ollama (Easiest)

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Create and run
ollama create sixpert-k2 -f OllamaModelfile

# Chat
ollama run sixpert-k2

Option 2: llama-cpp-python (Python)

pip install llama-cpp-python
python examples/generate.py --prompt "Explain quantum computing in depth"

Option 3: API Server

pip install llama-cpp-python
python examples/api_server.py --model SixpertK2.gguf

Option 4: LM Studio

1. Download LM Studio from https://lmstudio.ai
2. Import SixpertK2.gguf
3. Chat with the Sixpert K2 preset

Chat Format

Sixpert K2 uses the following chat template:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is quantum computing?<|im_end|>
<|im_start|>assistant
Quantum computing uses quantum mechanical phenomena...<|im_end|>

Recommended Settings

Parameter	Value	Notes
temperature	0.6	Slightly lower for reasoning tasks
top_p	0.85	Nucleus sampling
top_k	50	Limit token selection
repeat_penalty	1.08	Prevent repetition
max_tokens	16384	Extended output for deep reasoning
context_size	131072	Full context window

MoE-Specific Tips

When to Use K2 vs K1

Task Type	Best Model	Reason
Fast responses	K1 (Dense)	Predictable latency
Deep reasoning	K2 (MoE)	Specialized reasoning experts
Long documents	K2 (MoE)	Better long-context handling
Code generation	Either	K2 slightly better
Math proofs	K2 (MoE)	Math expert specialization
Simple Q&A	K1 (Dense)	Faster, sufficient quality
Agentic tasks	K2 (MoE)	Better tool orchestration
Vision tasks	Either	Both support multimodal

Long-Context Usage

K2 is optimized for long-context understanding. For documents exceeding 32K tokens:

llm = Llama(
    model_path="SixpertK2.gguf",
    n_ctx=131072,  # Full context window
    n_gpu_layers=-1,
)

# Feed entire documents
response = llm.create_chat_completion(
    messages=[{
        "role": "user",
        "content": f"Based on the following document, answer my question:\n\n{full_document}\n\nQuestion: What are the key findings?"
    }],
    max_tokens=4096,
)

Function Calling

See examples/function_calling.py for a complete agentic implementation.

Vision / Multimodal

See examples/vision_example.py for image analysis examples.

Integration Examples

OpenAI-Compatible Client

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="sixpert-k2",
    messages=[{"role": "user", "content": "Prove that sqrt(2) is irrational"}],
    temperature=0.6,
    max_tokens=4096,
)
print(response.choices[0].message.content)

LangChain Integration

from langchain.llms import LlamaCpp

llm = LlamaCpp(
    model_path="SixpertK2.gguf",
    temperature=0.6,
    n_ctx=131072,
    n_gpu_layers=-1,
)

result = llm.invoke("Explain the theory of relativity in detail")
print(result)

AutoGen Integration

from autogen import AssistantAgent

assistant = AssistantAgent(
    name="sixpert_k2",
    llm_config={"config_list": [{"model": "sixpert-k2", "base_url": "http://localhost:8000/v1", "api_key": "not-needed"}]},
    system_message="You are Sixpert K2, a deep reasoning engine.",
)

Performance Tips

1. GPU Offloading: Set n_gpu_layers=-1 for full GPU offload (fits in 8GB VRAM)
2. Temperature: Use 0.3-0.5 for mathematical proofs, 0.6-0.7 for creative tasks
3. Context Size: Start with 32768, increase to 131072 for long documents
4. Batch Inference: Use the API server for batch processing
5. Quantization: Q4_K_M is recommended; Q6_K for higher quality

Troubleshooting

Issue	Solution
Slow generation	Ensure GPU offloading is enabled
Out of memory	Reduce context size to 32768 or 8192
Repetitive output	Increase repeat_penalty to 1.1-1.15
Shallow reasoning	Lower temperature to 0.3-0.5
Long response needed	Set max_tokens to 8192 or 16384
Context overflow	Use 4096 context for testing