README.md

---
license: cc-by-4.0
tags:
  - kimi-k3
  - mixture-of-experts
  - linear-attention
  - research-notes
  - survey
  - open-source-models
---

# Kimi K3 — Research Notes & Open-Source Capability Inventory

> Study notes on **Kimi K3: Open Frontier Intelligence** (Moonshot AI, July 2026) — the first open
> 3T-class model — plus a **verified inventory of everything the K3 release actually open-sourced**.
> Maintained by **[Diogenes](https://huggingface.co/DiogenesLab)** / **[Oratis](https://huggingface.co/Oratis)**.

Kimi K3 is a 2.78T-parameter / 104.2B-activated natively multimodal MoE with a 1M-token context
window. It is interesting well beyond its size: it solves three problems that normally appear
separately — **sequence-length scaling, extreme MoE width, and depth-wise information flow** — and
each solution detaches cleanly from the rest of the model.

These notes are written to be *learned from*, not just skimmed: the emphasis is on **why** each
design exists and what failure mode it removes.

## Contents

| File | What it is |
|------|-----------|
| [`k3_architecture_notes.md`](k3_architecture_notes.md) | Architecture deep dive — Kimi Delta Attention and the lower-bounded decay trick, Attention Residuals, Stable LatentMoE (Normalized LatentMoE + SiTU-GLU + Quantile Balancing), MoonViT-V2 trained from scratch, Per-Head Muon. Includes the K2 → K3 spec diff. |
| [`k3_training_and_infra_notes.md`](k3_training_and_infra_notes.md) | Pre-training (data, scaling law methodology, the four-stage 8K→1M context curriculum), post-training (SFT → 9 domain × effort RL experts → multi-teacher on-policy distillation, MXFP4 QAT, EAGLE-3 draft + LK loss), RL environments and task synthesis, and the infrastructure (FlashKDA, KDA Context Parallelism, MoonEP, AgentENV, KDA-aware prefix caching). |
| [`k3_open_source_inventory.md`](k3_open_source_inventory.md) | **The practical artifact.** Every open-sourced component of the K3 release, each independently verified against its HF/GitHub page: what it does, its license, its hardware requirements, and how it maps back to a section of the technical report. Includes a breakdown of the Kimi K3 License, which is *not* MIT. |
| [`k3_evaluation_summary.md`](k3_evaluation_summary.md) | Where K3 actually lands — public benchmarks, in-house suites, the cyber-capability evaluation, third-party results (Artificial Analysis, Vals AI, LMArena), and cost-efficiency. |

## Three things worth knowing if you read nothing else

1. **A parameterization change bought an order of magnitude of hardware utilization.** KDA's chunkwise
   form needs to rescale keys by the reciprocal cumulative decay `1/Γ`, which grows without bound and
   overflows in low precision — so the predecessor (Kimi Linear) had to compute diagonal tiles with an
   explicit position-pair path that cannot use Tensor Cores. K3 bounds the log-decay from below with a
   scaled sigmoid (`g_min = −5`), which puts the cumulative decay over a 16-token tile in `(−80, 0)` and
   the reciprocal below `e^80` — **inside BF16's dynamic range**. Diagonal and off-diagonal tiles now both
   run as dense Tensor Core matmuls, and the special-case path is deleted. The algorithm did not change;
   only the range of one quantity did.

2. **Contrastive vision pre-training turned out to be unnecessary as an initialization.** K3's vision
   tower, MoonViT-V2, is trained **entirely from scratch with next-token prediction** rather than
   initialized from SigLIP. The reported motivation is stability — the SigLIP-initialized baseline shows
   persistently higher vision-tower gradient norms with frequent spikes throughout joint optimization —
   and MoonViT-V2 **matches it on vision evaluations anyway**. This is a direct challenge to a default
   design choice in multimodal LLMs. (Caveat worth keeping: this is evidence *at 2.8T scale with a full
   multimodal corpus*; it does not automatically transfer to small-scale fine-tuning regimes.)

3. **The open-source surface is much larger than the weights.** Six engineering repositories ship under
   MIT or Apache-2.0 — including the attention kernels, the expert-parallelism library, and the microVM
   sandbox platform that powered the agentic RL — while the weights themselves carry a *different*,
   more restrictive license. Two of the repos (`minitriton`, `nano-kpu`) were written by K3 itself and
   are explicitly labeled demonstrations rather than products. See
   [`k3_open_source_inventory.md`](k3_open_source_inventory.md).

## Notes on method

- Primary source is the **47-page Kimi K3 technical report** (Kimi Team, Moonshot AI). Every
  open-source claim in the inventory was **independently re-verified** against the live HF or GitHub
  page rather than transcribed from the report — a few details (exact hardware requirements, merge
  status of the upstream FLA context-parallel PR, license thresholds) are only available there.
- Claims are marked **[report]** (stated in the technical report), **[verified]** (independently
  checked against a live page), or **[analysis]** (our inference, not a claim of the original authors).
- This is a **curated public subset** of a larger internal research effort. Organization-specific
  roadmap judgments are intentionally not included; the focus here is the general method, the
  mechanisms, and the reusable artifacts.
- Last refreshed **2026-07-29**.

## Citation

The underlying work is Moonshot AI's:

```bibtex
@techreport{kimi2026k3,
  title  = {Kimi K3: Open Frontier Intelligence},
  author = {Kimi Team},
  year   = {2026},
  institution = {Moonshot AI},
  url    = {https://www.kimi.com/blog/kimi-k3}
}
```

## License

These notes released under **CC BY 4.0**. The Kimi K3 model, its technical report, and all cited
repositories belong to their respective authors under their own licenses — see
[`k3_open_source_inventory.md`](k3_open_source_inventory.md) for the per-artifact breakdown.