Kimi-K3-0.40B-F16-GGUF

This repository contains an F16 GGUF conversion of inference-optimization/Kimi-K3-0.40B, a tiny test/development version of moonshotai/Kimi-K3.

Model Overview

Property	Value
Base Model	moonshotai/Kimi-K3
Architecture	KimiK3ForConditionalGeneration (kimi_linear)
Total Parameters	0.40B
Activated Parameters	~0.06B (2 of 8 experts active per token)
Quantization	F16 (16-bit floating point)
File Size	~748 MB
GGUF Tensors	218

Architecture Highlights

Kimi-K3 introduces a novel hybrid attention architecture that combines:

Kimi Delta Attention (KDA)

• Type: Linear/delta attention
• Used on layers: 0–2, 4–6 (6 of 8 layers)
• Benefits: Efficient sequence processing with linear complexity
• Key parameters: head_dim=32, num_heads=8, short_conv_kernel_size=4

Multi-Latent Attention (MLA)

• Type: Full multi-latent attention (similar to DeepSeek-V2)
• Used on layers: 3, 7 (2 of 8 layers)
• Benefits: High-quality attention with KV cache compression
• Key parameters: q_lora_rank=256, kv_lora_rank=128, qk_nope_head_dim=64

Attention Ratio

• 3:1 KDA:MLA pattern (preserved from the original model)
• Attention + MLP residuals: Enabled with block_size=4

Mixture of Experts (MoE)

• Total experts: 8
• Active experts per token: 2
• Shared experts: 1
• Expert hidden size: 512
• Router activation: Sigmoid
• MoE layers: 1-7 (layer 0 uses dense MLP)

Configuration Details

Parameter	Value
num_hidden_layers	8
hidden_size	1024
intermediate_size	2048
num_attention_heads	8
num_key_value_heads	8
vocab_size	163,840
max_position_embeddings	4096
rope_theta	10000.0
rms_norm_eps	1e-5
hidden_act	situ

GGUF Conversion

This GGUF was converted using a specialized llama.cpp fork that supports Kimi-K3's custom architecture.

Conversion Details

• Conversion script: convert_hf_to_gguf.py
• Output type: F16 (full 16-bit precision)
• Tokenizer: TikToken-based with custom vocabulary

Important Notes

• ⚠️ This model requires the Kimi-K3-compatible llama.cpp fork — stock llama.cpp may not support this architecture
• ⚠️ Standard Ollama and LM Studio may not load this model without the custom runtime

Usage

Installation

# Clone the Kimi-K3-compatible fork
git clone -b k3/out-of-core https://github.com/murillo128/llama.cpp.git
cd llama.cpp

# Build
cmake -B build
cmake --build build --config Release -j

Running Inference

# Linux/macOS
./build/bin/llama-cli \
  -m ./Kimi-K3-0.40B-F16.gguf \
  -p "According to all known laws" \
  -n 20 \
  --temp 0 \
  --seed 1

# Windows
build/bin/Release/llama-cli.exe \
  -m ./Kimi-K3-0.40B-F16.gguf \
  -p "According to all known laws" \
  -n 20 \
  --temp 0 \
  --seed 1

Expected Output

> According to all known laws
 the start.

Technical Specifications

Model Architecture

KimiK3ForConditionalGeneration
├── Vision Tower (2 layers, 256 hidden size)
├── Language Model (8 layers)
│   ├── Layer 0: KimiDeltaAttention + Dense MLP
│   ├── Layer 1: KimiDeltaAttention + MoE
│   ├── Layer 2: KimiDeltaAttention + MoE
│   ├── Layer 3: KimiMLAAttention + MoE
│   ├── Layer 4: KimiDeltaAttention + MoE
│   ├── Layer 5: KimiDeltaAttention + MoE
│   ├── Layer 6: KimiDeltaAttention + MoE
│   └── Layer 7: KimiMLAAttention + MoE
└── LM Head

Tokenizer

• Type: TikToken-based custom tokenizer
• Vocabulary size: 163,840 tokens
• Special tokens: BOS, EOS, EOT, media tokens, chat markers

Comparison with Original Model

Aspect	moonshotai/Kimi-K3	Kimi-K3-0.40B
Total Parameters	~32B	0.40B
Hidden Size	7168	1024
Num Layers	93	8
Num Experts	896	8
Attention Type	KDA + MLA	KDA + MLA
KDA:MLA Ratio	3:1	3:1
MoE Experts/Token	16	2

Limitations

• This is a tiny test model — not suitable for production use
• Vision tower is present but untrained for vision tasks
• Limited context length (4096 tokens)
• May not match the quality of larger Kimi-K3 variants

License

This model is released under the MIT License, consistent with the base model.

Acknowledgments

• Moonshot AI for the original Kimi-K3 architecture
• inference-optimization for creating the tiny test version
• murillo128 for the Kimi-K3-compatible llama.cpp fork
• llm-compressor for the tiny model creation tools

Citation

@article{kimi-k3,
  title={Kimi-K3: A Hybrid Attention Architecture},
  author={Moonshot AI},
  year={2025}
}

Related Models

• moonshotai/Kimi-K3 - Original full-size model
• inference-optimization/Kimi-K3-0.40B - Base HF model for this GGUF
• Kimi-K3-0.40B-GGUF - Reference GGUF conversions