docs/architecture.md

Sixpert K2 Architecture

Overview

Sixpert K2 is a Deep Reasoning Engine built on a Mixture-of-Experts (MoE) transformer architecture. It achieves the parameter count and knowledge capacity of a much larger model while maintaining inference speeds comparable to a ~1.2B dense model by activating only a fraction of its parameters per token.

Model Specifications

Parameter	Value
Total Parameters	~8.9B
Active Parameters (per token)	~1.2B
Architecture	MoE Transformer
Total Experts	16
Experts per Token	2
Hidden Size	3584
Attention Heads	28
KV Heads	4
Layers	28
Intermediate Size	14336
Context Length	131,072 tokens
Vocabulary	151,936 tokens
Activation	SiLU (SwiGLU)
Normalization	RMSNorm
RoPE Base	1,000,000
Attention Bias	No
Tie Embeddings	No

Mixture-of-Experts Design

Expert Architecture

Each of the 28 transformer layers contains 16 parallel feed-forward experts. During inference:

1. A routing network (learned linear layer) evaluates the input
2. The top-2 experts are selected based on routing scores
3. Only the selected experts process the token
4. Their outputs are combined using softmax-weighted averaging

Efficiency Advantage

Metric	Dense 8B Model	Sixpert K2 (MoE)	Improvement
Parameters	8B	8.9B total	+11% capacity
Active per token	8B	~1.2B	6.7x fewer
Inference speed	1.0x	~4-6x faster	Significant
Memory (inference)	16GB (FP16)	~5GB (Q4_K_M)	3.2x less
VRAM (GPU)	16GB+	6-8GB	Practical on consumer GPUs

Expert Specialization

The 16 experts in each layer develop specialization during training:

Expert Group	Specialization
Experts 1-4	Mathematical reasoning and computation
Experts 5-8	Code generation and programming
Experts 9-12	Natural language understanding
Experts 13-16	Multimodal and visual reasoning

This specialization enables K2 to handle diverse tasks without performance degradation across domains.

Grouped Query Attention (GQA)

K2 employs GQA with 28 query heads and 4 key-value heads, providing:

• Efficient long-context processing (131K tokens)
• Reduced KV cache memory footprint
• Fast attention computation even at maximum context length

Rotary Position Embeddings

RoPE with base frequency of 1,000,000 enables fine-grained positional discrimination across the full 131K context window.

Quantization

The released model uses Q4_K_M quantization via GGUF format:

Aspect	Detail
Format	GGUF
Method	Q4_K_M
Block Size	256
Weight Bits	4
Per-tensor Scale	Yes
Per-block Scale	Yes

Training Approach

K2 was trained with a multi-stage pipeline:

1. Dense Pre-training: Base model trained on large-scale diverse corpus
2. MoE Expansion: Upcycling to MoE architecture with expert initialization
3. Expert Training: Specialized training with routing optimization
4. SFT: Supervised fine-tuning on high-quality instruction data
5. RLHF/RLAIF: Preference optimization for alignment
6. Agentic Training: Extended training on tool use and multi-step tasks

Hardware Requirements

Use Case	Minimum	Recommended
Inference (CPU)	8GB RAM, 8 threads	16GB RAM, 16 threads
Inference (GPU)	6GB VRAM (full offload)	8GB VRAM (full offload)
Fine-tuning (LoRA)	24GB VRAM	48GB+ VRAM
Fine-tuning (full MoE)	80GB VRAM	2x A100 80GB

Comparison to Dense Models

Sixpert K2 achieves performance comparable to dense models 4-7x its active parameter count:

Task	K2 (1.2B active)	Dense Equivalent
Reasoning	~7B dense	4-5x fewer active params
Code	~6B dense	5x fewer active params
Knowledge	~8B dense	7x fewer active params
Math	~6B dense	5x fewer active params