README.md

metadata

license: mit
base_model:
  - moonshotai/Kimi-K3
library_name: transformers

Kimi-K3-0.40B

This is a tiny version of moonshotai/Kimi-K3 created for testing and development.

Model Details

• Base Model: moonshotai/Kimi-K3
• Architecture: kimi_linear (KimiK3ForConditionalGeneration)
• Total Parameters: 0.40B
• Activated Parameters: ~0.06B (2 of 8 experts active per token)

Configuration Changes

The following parameters were reduced from the original model:

Parameter	Original	Tiny
num_hidden_layers	93	8
hidden_size	7168	1024
intermediate_size	33792	2048
moe_intermediate_size	3072	256
num_attention_heads	96	8
num_key_value_heads	96	8
q_lora_rank	1536	256
kv_lora_rank	512	128
qk_nope_head_dim	128	64
qk_rope_head_dim	64	32
v_head_dim	128	64
num_experts	896	8
num_shared_experts	2	1
num_experts_per_token	16	2
routed_expert_hidden_size	3584	512
attn_res_block_size	12	4
linear_attn head_dim	74	32
linear_attn num_heads	96	8
vt_num_hidden_layers	27	2
vt_hidden_size	1024	256

Architecture Preserved

• Mixed attention: KDA (linear/delta attention) on layers 0–2, 4–6 and MLA (full multi-latent attention) on layers 3, 7 — same 3:1 KDA:MLA ratio as the original
• Attention + MLP residuals: attn_res_block_size=4 enabled on all layers
• MoE: layers 1–7 use sparse MoE with latent expert projection; layer 0 is dense MLP
• Vision tower: included but reduced

Checkpoint Structure

Single-shard checkpoint (model.safetensors). Key prefix: language_model.model.layers.{i}.*, matching the original sharded checkpoint structure.

Usage

import sys
sys.path.insert(0, "/path/to/llm-compressor/src")

from transformers import AutoTokenizer
from llmcompressor.modeling.kimi_k3 import KimiK3ForConditionalGeneration

model = KimiK3ForConditionalGeneration.from_pretrained(
    "inference-optimization/Kimi-K3-0.40B", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "inference-optimization/Kimi-K3-0.40B", trust_remote_code=True
)

inputs = tokenizer("According to all known laws", return_tensors="pt").to(model.device)
output = model.language_model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Validation Output

Total parameters: 0.396B
Number of layers: 8
Layer 0 attn type: KimiDeltaAttention (KDA/linear)
Layer 3 attn type: KimiMLAAttention (MLA/full)
Layer 7 attn type: KimiMLAAttention (MLA/full)
Layer 0 has MLP: True
Layer 1 has MoE: True
Attention residuals enabled: True
Attn res block size: 4

Forward pass loss: 0.0013
Generated: The FitnessGram Pacer Test is a multistage aerobic capacity test that progressively gets more difficult as it continues. The 20 meter pacer test will begin in 30 seconds

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

1. Inspected the original moonshotai/Kimi-K3 config to identify key architecture parameters
2. Reduced layer count, hidden dimensions, and expert counts to target ~0.4B total parameters
3. Preserved the 3:1 KDA:MLA attention pattern and attention/MLP residual connections
4. Initialized all weights from scratch (normal distribution, std=0.02; norms → 1.0; biases → 0.0)
5. Fine-tuned on a toy copypasta dataset until perplexity < 3.0 (achieved in ~57 epochs)

Notes

• The model uses custom modeling code from llmcompressor.modeling.kimi_k3 — it cannot be loaded with AutoModelForCausalLM without that module on the path
• KimiK3ForConditionalGeneration does not inherit from GenerationMixin; use model.language_model.generate(...) for text generation
• The vision tower is present but untrained for vision tasks