README.md

---
license: mit
base_model:
- moonshotai/Kimi-K3
library_name: transformers
---

# Kimi-K3-0.40B

This is a tiny version of [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) created for testing and development.

## Model Details

- **Base Model**: moonshotai/Kimi-K3
- **Architecture**: kimi_linear (KimiK3ForConditionalGeneration)
- **Total Parameters**: 0.40B
- **Activated Parameters**: ~0.06B (2 of 8 experts active per token)

## Configuration Changes

The following parameters were reduced from the original model:

| Parameter | Original | Tiny |
|-----------|----------|------|
| `num_hidden_layers` | 93 | 8 |
| `hidden_size` | 7168 | 1024 |
| `intermediate_size` | 33792 | 2048 |
| `moe_intermediate_size` | 3072 | 256 |
| `num_attention_heads` | 96 | 8 |
| `num_key_value_heads` | 96 | 8 |
| `q_lora_rank` | 1536 | 256 |
| `kv_lora_rank` | 512 | 128 |
| `qk_nope_head_dim` | 128 | 64 |
| `qk_rope_head_dim` | 64 | 32 |
| `v_head_dim` | 128 | 64 |
| `num_experts` | 896 | 8 |
| `num_shared_experts` | 2 | 1 |
| `num_experts_per_token` | 16 | 2 |
| `routed_expert_hidden_size` | 3584 | 512 |
| `attn_res_block_size` | 12 | 4 |
| `linear_attn head_dim` | 74 | 32 |
| `linear_attn num_heads` | 96 | 8 |
| `vt_num_hidden_layers` | 27 | 2 |
| `vt_hidden_size` | 1024 | 256 |

## Architecture Preserved

- **Mixed attention**: KDA (linear/delta attention) on layers 0–2, 4–6 and MLA (full multi-latent attention) on layers 3, 7 — same 3:1 KDA:MLA ratio as the original
- **Attention + MLP residuals**: `attn_res_block_size=4` enabled on all layers
- **MoE**: layers 1–7 use sparse MoE with latent expert projection; layer 0 is dense MLP
- **Vision tower**: included but reduced

## Checkpoint Structure

Single-shard checkpoint (`model.safetensors`). Key prefix: `language_model.model.layers.{i}.*`, matching the original sharded checkpoint structure.

## Usage

```python
import sys
sys.path.insert(0, "/path/to/llm-compressor/src")

from transformers import AutoTokenizer
from llmcompressor.modeling.kimi_k3 import KimiK3ForConditionalGeneration

model = KimiK3ForConditionalGeneration.from_pretrained(
    "inference-optimization/Kimi-K3-0.40B", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "inference-optimization/Kimi-K3-0.40B", trust_remote_code=True
)

inputs = tokenizer("According to all known laws", return_tensors="pt").to(model.device)
output = model.language_model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Validation Output

```
Total parameters: 0.396B
Number of layers: 8
Layer 0 attn type: KimiDeltaAttention (KDA/linear)
Layer 3 attn type: KimiMLAAttention (MLA/full)
Layer 7 attn type: KimiMLAAttention (MLA/full)
Layer 0 has MLP: True
Layer 1 has MoE: True
Attention residuals enabled: True
Attn res block size: 4

Forward pass loss: 0.0013
Generated: The FitnessGram Pacer Test is a multistage aerobic capacity test that progressively gets more difficult as it continues. The 20 meter pacer test will begin in 30 seconds
```

## Creation Process

This model was created using the llm-compressor `create-tiny-model` claude skill.

1. Inspected the original `moonshotai/Kimi-K3` config to identify key architecture parameters
2. Reduced layer count, hidden dimensions, and expert counts to target ~0.4B total parameters
3. Preserved the 3:1 KDA:MLA attention pattern and attention/MLP residual connections
4. Initialized all weights from scratch (normal distribution, std=0.02; norms → 1.0; biases → 0.0)
5. Fine-tuned on a toy copypasta dataset until perplexity < 3.0 (achieved in ~57 epochs)

## Notes

- The model uses custom modeling code from `llmcompressor.modeling.kimi_k3` — it cannot be loaded with `AutoModelForCausalLM` without that module on the path
- `KimiK3ForConditionalGeneration` does not inherit from `GenerationMixin`; use `model.language_model.generate(...)` for text generation
- The vision tower is present but untrained for vision tasks