docs/architecture.md

Sixpert K1 Architecture

Overview

Sixpert K1 is a precision logic engine designed for complex reasoning, code generation, and agentic workflows. The model architecture is built on a transformer backbone with several key design decisions that distinguish it from conventional models.

Model Specifications

Parameter	Value
Active Parameters	~8.7B
Architecture	Transformer (Dense)
Hidden Size	3584
Attention Heads	28
KV Heads	4
Layers	28
Intermediate Size	18944
Context Length	131,072 tokens
Vocabulary	151,936 tokens
Activation	SiLU (SwiGLU)
Normalization	RMSNorm
RoPE Base	1,000,000
Attention Bias	Yes
Tie Embeddings	No

Architecture Details

Transformer Blocks

Each transformer block in Sixpert K1 consists of:

1. RMSNorm - Root Mean Square normalization applied before attention and FFN (pre-norm architecture)
2. Multi-Head Attention with Grouped Query Attention (GQA) for memory efficiency
3. SwiGLU Feed-Forward Network - Uses SiLU activation with gated linear units
4. Residual Connections - Standard residual pathways around both attention and FFN

Grouped Query Attention (GQA)

Sixpert K1 employs GQA with 28 query heads and 4 key-value heads. This design significantly reduces the KV cache size while maintaining strong attention quality, enabling efficient long-context inference.

Rotary Position Embeddings (RoPE)

The model uses Rotary Position Embeddings with a base frequency of 1,000,000. This high base frequency enables the model to maintain fine-grained positional discrimination even at very long context lengths (up to 131K tokens).

Vocabulary

The 151,936 token vocabulary is designed for multilingual coverage and efficient encoding. Special tokens include:

Token	ID	Purpose
<|im_start|>	151643	Message boundary / BOS
<|im_end|>	151643	Message boundary / EOS
<|pad|>	151644	Padding
<|think_start|>	-	Chain-of-thought start
<|think_end|>	-	Chain-of-thought end
<|tool_start|>	-	Tool call start
<|tool_end|>	-	Tool call end

Quantization

The released model uses Q4_K_M quantization via GGUF format. This provides an excellent balance of:

• Memory efficiency: ~5GB VRAM/RAM for full model loading
• Quality retention: Minimal degradation from FP16 baseline
• Inference speed: Fast token generation even on consumer hardware

Quantization Format Details

Aspect	Detail
Format	GGUF
Method	Q4_K_M (4-bit K-quant, medium)
Block Size	256
Weight Bits	4
Per-tensor Scale	Yes
Per-block Scale	Yes

Training Approach

Sixpert K1 was trained using a multi-stage approach:

1. Pre-training: Large-scale corpus with diverse domains
2. Supervised Fine-tuning (SFT): High-quality instruction following data
3. Reinforcement Learning: RLHF/RLAIF with preference optimization
4. Domain Specialization: Targeted training for reasoning, code, and agentic tasks

Design Philosophy

The model was designed with several principles:

• Reasoning-first: Chain-of-thought reasoning is a first-class capability
• Agentic-ready: Native function calling and tool use support
• Long-context: 131K token context window for document-level understanding
• Multimodal: Vision and multimodal input processing
• Efficient: GQA architecture for memory-efficient inference
• Unrestricted: No artificial limitations on output format or content

Hardware Requirements

Use Case	Minimum	Recommended
Inference (CPU)	8GB RAM, 8 threads	16GB RAM, 16 threads
Inference (GPU)	6GB VRAM (partial offload)	8GB VRAM (full offload)
Fine-tuning	24GB VRAM (LoRA)	48GB+ VRAM (full)