Kimi-K3-0.18B

This is a tiny version of moonshotai/Kimi-K3 created for testing and development.

Model Details

• Base Model: moonshotai/Kimi-K3
• Architecture: kimi_k3
• Total Parameters: 0.18B
• Activated Parameters: ~0.10B (MoE: 2 of 8 experts active per token)

Configuration Changes

The following parameters were reduced from the original model:

Parameter	Original	Tiny
num_hidden_layers	93	4
hidden_size	7168	512
num_attention_heads	96	4
num_key_value_heads	96	4
intermediate_size (dense)	33792	1024
moe_intermediate_size	3072	256
routed_expert_hidden_size	3584	256
num_experts	896	8
num_experts_per_token	16	2
num_shared_experts	2	1
kv_lora_rank	512	64
q_lora_rank	1536	128
qk_nope_head_dim	128	64
qk_rope_head_dim	64	32
v_head_dim	128	64
linear_attn head_dim	128	64

The 4-layer architecture preserves all original layer types:

• Layer 0: Dense FFN + KDA linear attention
• Layers 1–2: MoE FFN + KDA linear attention
• Layer 3: MoE FFN + MLA full attention

Checkpoint Structure

Single-file checkpoint (model.safetensors, ~0.69 GB). Keys follow the same language_model.model.layers.X.* hierarchy as the original, without quantization (the base model is MX4-quantized; this tiny model uses plain float weights).

Validation

Success: perplexity=2.12 (target <= 10.0)

Generation: "According to all known laws of aviation, there is no way a bee should
be able to fly. Its wings are too small to get its fat little"

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "inference-optimization/Kimi-K3-0.18B",
    trust_remote_code=True,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "inference-optimization/Kimi-K3-0.18B",
    trust_remote_code=True,
)

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model claude skill.

Several compatibility fixes were applied to the custom modeling code to work with newer transformers:

1. OutputRecorder import moved from transformers.utils.generic to transformers.utils.output_capturing
2. tie_weights() updated to accept **kwargs for newer transformers API
3. _tied_weights_keys converted from list to dict
4. create_causal_mask call updated to new signature (inputs_embeds, removed cache_position)
5. KimiDynamicCache gained get_query_offset and get_mask_sizes(int, ...) compatibility
6. Flash attention 2 gracefully falls back to eager when unavailable
7. KimiMoEGate training assertion removed; KimiSparseMoeBlock.moe_train added for gradient-compatible dispatch

Notes

• Fine-tuned on a small copypasta dataset to validate learning; not intended for production use.
• Quantization config from the base model is stripped; all weights are in bfloat16/float32.
• trust_remote_code=True is required to load the custom modeling files.