docs/architecture.md

# Sixpert K1 Architecture

## Overview

Sixpert K1 is a precision logic engine designed for complex reasoning, code generation, and agentic workflows. The model architecture is built on a transformer backbone with several key design decisions that distinguish it from conventional models.

## Model Specifications

| Parameter | Value |
|---|---|
| Active Parameters | ~8.7B |
| Architecture | Transformer (Dense) |
| Hidden Size | 3584 |
| Attention Heads | 28 |
| KV Heads | 4 |
| Layers | 28 |
| Intermediate Size | 18944 |
| Context Length | 131,072 tokens |
| Vocabulary | 151,936 tokens |
| Activation | SiLU (SwiGLU) |
| Normalization | RMSNorm |
| RoPE Base | 1,000,000 |
| Attention Bias | Yes |
| Tie Embeddings | No |

## Architecture Details

### Transformer Blocks

Each transformer block in Sixpert K1 consists of:

1. **RMSNorm** - Root Mean Square normalization applied before attention and FFN (pre-norm architecture)
2. **Multi-Head Attention** with Grouped Query Attention (GQA) for memory efficiency
3. **SwiGLU Feed-Forward Network** - Uses SiLU activation with gated linear units
4. **Residual Connections** - Standard residual pathways around both attention and FFN

### Grouped Query Attention (GQA)

Sixpert K1 employs GQA with 28 query heads and 4 key-value heads. This design significantly reduces the KV cache size while maintaining strong attention quality, enabling efficient long-context inference.

### Rotary Position Embeddings (RoPE)

The model uses Rotary Position Embeddings with a base frequency of 1,000,000. This high base frequency enables the model to maintain fine-grained positional discrimination even at very long context lengths (up to 131K tokens).

### Vocabulary

The 151,936 token vocabulary is designed for multilingual coverage and efficient encoding. Special tokens include:

| Token | ID | Purpose |
|---|---|---|
| `<\|im_start\|>` | 151643 | Message boundary / BOS |
| `<\|im_end\|>` | 151643 | Message boundary / EOS |
| `<\|pad\|>` | 151644 | Padding |
| `<\|think_start\|>` | - | Chain-of-thought start |
| `<\|think_end\|>` | - | Chain-of-thought end |
| `<\|tool_start\|>` | - | Tool call start |
| `<\|tool_end\|>` | - | Tool call end |

## Quantization

The released model uses Q4_K_M quantization via GGUF format. This provides an excellent balance of:

- **Memory efficiency**: ~5GB VRAM/RAM for full model loading
- **Quality retention**: Minimal degradation from FP16 baseline
- **Inference speed**: Fast token generation even on consumer hardware

### Quantization Format Details

| Aspect | Detail |
|---|---|
| Format | GGUF |
| Method | Q4_K_M (4-bit K-quant, medium) |
| Block Size | 256 |
| Weight Bits | 4 |
| Per-tensor Scale | Yes |
| Per-block Scale | Yes |

## Training Approach

Sixpert K1 was trained using a multi-stage approach:

1. **Pre-training**: Large-scale corpus with diverse domains
2. **Supervised Fine-tuning (SFT)**: High-quality instruction following data
3. **Reinforcement Learning**: RLHF/RLAIF with preference optimization
4. **Domain Specialization**: Targeted training for reasoning, code, and agentic tasks

## Design Philosophy

The model was designed with several principles:

- **Reasoning-first**: Chain-of-thought reasoning is a first-class capability
- **Agentic-ready**: Native function calling and tool use support
- **Long-context**: 131K token context window for document-level understanding
- **Multimodal**: Vision and multimodal input processing
- **Efficient**: GQA architecture for memory-efficient inference
- **Unrestricted**: No artificial limitations on output format or content

## Hardware Requirements

| Use Case | Minimum | Recommended |
|---|---|---|
| Inference (CPU) | 8GB RAM, 8 threads | 16GB RAM, 16 threads |
| Inference (GPU) | 6GB VRAM (partial offload) | 8GB VRAM (full offload) |
| Fine-tuning | 24GB VRAM (LoRA) | 48GB+ VRAM (full) |