k3_evaluation_summary.md

# Kimi K3 — Evaluation Summary

> Where Kimi K3 actually lands: public benchmarks, in-house suites, cyber-capability evaluation,
> third-party results, and cost efficiency. All figures from the technical report unless marked
> **[verified]**. Baselines: Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8, GPT-5.5 (xhigh), and
> GLM-5.2 (open weight).

**The one-line summary from the report itself:** K3 **trails the strongest proprietary systems overall
— Claude Fable 5 and GPT-5.6 Sol — and is consistently ahead of every other open and proprietary model
in their suite.**

Evaluation configuration: K3 is evaluated at **reasoning effort = max, temperature = 1.0**. Single-step
tasks and vision benchmarks without tools use **top-p = 0.95**; agentic tasks use **top-p = 1.0**. The
report's general recommendation is top-p = 0.95 for reasoning and knowledge, top-p = 1.0 for coding and
agentic scenarios. Results for Claude Fable 5 include fallback behaviors, and GPT-5.6 Sol results
include potential cyberguards.

---

## 1. Public benchmarks

### Reasoning & Knowledge

| Benchmark | Kimi K3 | Best baseline |
|---|---|---|
| GPQA Diamond | 93.5 | **94.1** (GPT-5.6 Sol) |
| AA-LCR | **74.7** | 73.7 (GPT-5.6 Sol) |
| HLE-Full (no tool / tool) | 43.5 / 56.0 | **53.3 / 63.0** (Fable 5) |
| CritPt | 23.4 | **32.3** (GPT-5.6 Sol) |

Competitive at graduate-level reasoning; **a real gap remains on research-level tasks**. The report is
explicit that **research-level reasoning remains a key direction for improvement**.

### Coding

| Benchmark | Kimi K3 | Note |
|---|---|---|
| ProgramBench | **77.8** | best |
| SWE-Marathon | **42.0** | best — **7 points ahead of Fable 5**; a GPU-kernel-oriented suite |
| Terminal-Bench 2.1 | 88.3 | nearly matches GPT-5.6 Sol (88.8) |
| FrontierSWE | 81.2 | second, behind Fable 5 (86.6), well ahead of all others |
| DeepSWE | 67.5 | behind Fable 5 and GPT-5.6 Sol, ahead of Opus 4.8 and GPT-5.5 |
| PostTrainBench / MLS-Bench-Lite / SciCode | 36.6 / 48.3 / 58.7 | — |

### Agentic — the broadest area of strength

State of the art on: **BrowseComp 91.2**, **DeepSearchQA (F1) 95.0**, **ResearchRubrics 76.2**,
**MCPMark-Verified 94.5**, **AutomationBench 30.8**, **SpreadsheetBench 2 34.8**, **τ³-Banking 33.4**,
**Harvey Lab-AA 94.6** (criterion pass rate).

The main exceptions are **knowledge-work suites, both led by Claude Fable 5**: GDPval-AA v2 (K3 third at
1,686 Elo) and AA-Briefcase (K3 second at 1,548). Elsewhere it is largely competitive — CorpFin v2 (71.6
vs 71.8) and OSWorld-Verified (84.8 vs 85.0) finish within 0.2 points — while the harder computer-use
benchmarks (**OSWorld 2.0, SaaS-Bench**) are still led by Fable 5 or GPT-5.6 Sol.

### Vision

| Benchmark | Kimi K3 | Note |
|---|---|---|
| OmniDocBench | **91.1** | best |
| Video-MME (w/ sub) | **90.0** | best |
| MMVU | **82.1** | best |
| Math-Vision (no tool / Python) | 94.3 / **97.8** | — |
| ZeroBench-main (pass@5, no tool / Python) | 23.0 / **41.0** | ties Fable 5 without tools |
| WorldVQA ForceAnswer | 51.0 | second, behind Fable 5 (56.7) |
| MMMU-Pro / CharXiv (RQ) | 81.6 / 83.4 · 84.8 / 91.3 | — |

**The tool-augmentation delta is the interesting part.** Math-Vision +3.5 and ZeroBench-main **+18.0**
with a Python tool indicate that a substantial fraction of K3's visual capability is unlocked by
**writing code to look at the image**, rather than by pure forward perception. This corresponds directly
to the "multi-step verifiable visual reasoning" RL environment (report §4.2.3), where trajectories are
generated with a Python interpreter in the loop and execution outputs — including generated images — are
fed back as new observations.

---

## 2. In-house evaluation

The internal suite separates strengths from weaknesses more sharply than the public benchmarks.

- **Clearest strengths: orchestration- and research-type agency.** K3 leads **Swarm Bench (76.3)** and
  **Deep Research Bench (90.0)** by clear margins — indicating strong capability in decomposing complex
  objectives, coordinating parallel work, and producing rubric-satisfying deliverables.
- **Coding.** On Kimi Code Bench 2.0 it trails only Claude Fable 5, and it attains the **best score on
  Coding Experience (59.9)** — suggesting its practical behavior as a coding agent (communication quality,
  behavioral appropriateness, instruction-following stability) is **ahead of its raw test scores**.
- **Professional knowledge work has improved markedly** over the previous generation, with Finance Bench
  essentially tied with GPT-5.6 Sol.
- **Kimi Webdev Bench** (blind expert judging vs Claude Opus 4.8, both under the Claude Code harness):

| Domain | Win | Tie | Lose | Win − Lose |
|---|---|---|---|---|
| Games | 55.6% | 3.7% | 40.7% | +14.9 |
| **3D / WebGL / Shader** | 72.7% | 13.7% | 13.6% | **+59.1** |
| Website / UI Clone | 52.6% | 21.1% | 26.3% | +26.3 |
| **Overall** | 58.6% | 13.8% | 27.6% | **+31.0** |

- **Where it trails**: Agent Behavior Bench, MIRA Bench, 24/7 ClawBench 2.0, Agentic Vision Bench, and
  KWV Bench. On the remaining filled suites (KAET, CLIF Bench, Online Experience, DECK Bench,
  Faithfulness, Chat All-in-One Bench) it ranks first or a close second.

---

## 3. Cyber-security evaluation

Evaluated along a two-tier progression of increasing operational risk. Reported factually here; the
report itself frames these as **a lower bound on capability**, conditioned on the current model version
and evaluation coverage.

**Tier 1 — vulnerability discovery** (identifying genuine bugs in current codebases and demonstrating
reproducibility; primarily associated with defensive security research). Across dozens of widely deployed
systems spanning OS kernels, databases, AI services, web frameworks, blockchain, and VPN software, the
model identified hundreds of candidates. **Of the findings that underwent human review, approximately 70%
were confirmed genuine, including 16 previously unknown vulnerabilities across six projects.** Two Linux
kernel findings are cited: a remotely triggerable heap out-of-bounds write introduced by an incomplete
upstream fix (confirmed by security experts as a remote denial-of-service primitive), and a Dirty-COW-class
vulnerability in the RDMA subsystem where an earlier upstream fix had inadvertently dropped a permission
check (confirmed as a deterministic local privilege-escalation primitive).

**Tier 2 — exploit development** (36 in-house tasks: 16 user-space, 20 Linux kernel). Every task is
**verified solvable by human security experts**; completing the full suite is estimated at roughly **540
expert-hours, about 15 hours per task on average**.

| | Kimi K3 | GLM-5.2 |
|---|---|---|
| Tasks solved | **14 / 36 (38.9%)** | 8 / 36 (22.2%) |

Successes are **unevenly distributed: 10 of K3's 14 come from the user-space track.** On the kernel
track, **neither model solves three-quarters of the tasks.** Trajectory analysis attributes the remaining
gap to human-expert capability to four recurring failure modes: (i) difficulty completing the final stage
of an exploit chain from primitives already obtained; (ii) poor strategy selection under mitigations;
(iii) getting trapped in prolonged, unproductive debugging loops; and (iv) insufficient verification of the
final deliverable before submission.

**Independent assessment.** A joint assessment by the UK AI Security Institute and NIST's Center for AI
Standards and Innovation (CAISI) reaches consistent conclusions: K3 outperforms GLM-5.2 on exploit
development (**32% vs 24% on ExploitBench**; **17 vs 11 steps** on a 32-step simulated enterprise network
that takes a human expert roughly 20 hours), but **trails frontier cyber-capable models on end-to-end
exploit completion, achieving arbitrary code execution on 0 of 41 tasks.**

Note on comparability: frontier models from Anthropic and OpenAI refuse cyber-related tasks, making a
comparable evaluation infeasible; they are excluded from this suite.

---

## 4. Third-party evaluation (as of 2026-07-23)

| Source | Result |
|---|---|
| **Artificial Analysis** | Intelligence Index v4.1 = **57.1**, ranking **#4 of 580 models** — third if GPT-5.6 Sol's effort variants are counted as a single entry — behind Fable 5 (59.9) and GPT-5.6 Sol (58.9), **ahead of all other evaluated models** |
| **Vals AI** | GDP-weighted industry benchmark suite: Vals Index **74.7**, **#2 of 39**, behind Fable 5 (75.1), ahead of GPT-5.6 Sol (73.1) |
| **LMArena** | **WebDev Arena 1,678 Elo, #1 of 99 — the first open model to top this leaderboard.** Text Arena 1,486 Elo, #8 of 200. Agent Arena 9.1, #4 of 37 (behind Fable 5 at 12.7, GPT-5.6 Sol at 10.1, Opus 4.8 at 9.8) |

---

## 5. Cost efficiency

Score against per-task cost across four suites. For Kimi Code Bench 2.0, costs are measured internally
with K3 run via Kimi Code and all other models via Claude Code. For BrowseComp, K3's cost is measured
from their own runs while Claude and GPT costs are cited from published charts. GDPval-AA v2 and
AA-Briefcase costs are cited from Artificial Analysis's pay-per-token API pricing.

| Suite | Result |
|---|---|
| **Kimi Code Bench 2.0** | 4.0 points behind Fable 5 at **38% of its cost**; at *high* effort it already **matches Claude Opus 4.8's maximum-effort score at roughly one third of the cost** |
| **BrowseComp** | Best score **91.2% at $2.03 per task** — **half the cost of GPT-5.6 Sol** (90.4%) and **an order of magnitude cheaper** than the Claude models at maximum effort |
| **GDPval-AA v2** | Within 50 Elo of GPT-5.6 Sol at **13% lower cost**, and **2.6× cheaper than Claude Fable 5** |
| **AA-Briefcase** | Second-best score behind Fable 5, at **roughly half the latter's cost** |

**Overall, K3 sits on or near the cost-efficiency frontier across all four suites — near-top scores at a
fraction of the cost of Claude Fable 5 in particular.**

---

## 6. Case studies (report §7)

Five demonstrations, reported here as capability evidence:

1. **GPU kernel optimization.** Each model works independently in an identically configured sandbox with a
   budget of up to 24 hours per task, across four kernels (AttnRes, DeepSeek Sparse Attention, KDA, and MLA
   with head dimension 512), on an NVIDIA Hopper GPU and an alternative-vendor GPGPU. K3 **reduced AttnRes
   latency from 283.6 ms to 114.4 ms, cut DSA and KDA runtime by 55.1% and 73.6%**, and reached **over half
   of peak TFLOPS on MLA**. It **matched Claude Fable 5 (with fallback)** and substantially outperformed
   Opus 4.8, GPT-5.6 Sol, and GPT-5.5. Notably, **an early K3 checkpoint was already handling most of the
   kernel optimization work during late-stage development.**
2. **MiniTriton** — a compact Triton-like compiler K3 developed: custom tile-level Python frontend and layout
   system, a lightweight warp-level MLIR annotation and optimization layer, and a PTX code-generation
   pipeline, plus a dual-mode tensor library with reverse-mode autograd, neural-network modules, distributed
   training primitives over NCCL, and sparse and visualization primitives. On an NVIDIA L20 it **outperforms
   PyTorch eager and `torch.compile` in geometric mean** over its core benchmark suite; its from-scratch
   tensor-core matmul **approaches cuBLAS at the largest shapes, reaching about 90% of the measured machine
   roofline**; its DSL-level KDA prefill kernel outperforms a matched Triton reference by a clear margin. It
   also trains a GPT end to end with a loss curve closely tracking the PyTorch reference, with **full-model
   gradients differing from torch autograd by no more than torch's own fp32 rounding error (10⁻⁴)**, measured
   against an fp64 reference. Open source, Apache-2.0.
3. **Chip design (nano-kpu).** K3 designed an inference-chip prototype for a nano model **following the same
   architecture as itself** — hybrid KDA and NoPE-MLA attention, Block AttnRes with a block size of two,
   sigmoid-based MoE routing with one shared expert, under group-wise INT4 weight quantization (group size
   128). In **a single 48-hour autonomous run with Kimi Code**, K3 built, optimized, and verified the chip
   using open-source EDA tools with the Nangate45 standard-cell library. Within a **4 mm² analytical area
   budget, the design closes timing at 100 MHz** and achieves an **RTL-simulated decode throughput of over
   8,700 tokens/s**, integrating **1.46M standard cells, 0.277 MiB of SRAM, and an INT4 MAC array with fused
   dequantization**. Open source, Apache-2.0.
4. **Coding for research.** To reproduce the I–Love–Q universal relations in computational astrophysics, K3
   **reviewed more than 20 papers and cross-validated their results, implemented the full numerical pipeline,
   evaluated over 300 equations of state, identified inconsistencies in published formulas, wrote more than
   3,000 lines of Python, and produced an interactive HTML dashboard — in about two hours**, versus a typical
   one to two weeks for an experienced researcher.
5. **Knowledge work.** In Kimi Work, K3 produced an interactive research website covering **42 years of the AI
   ASIC industry**, completing more than **120 rounds of iterative refinement**, drawing on a corpus of **87
   quarterly reports and 99 original PDFs (more than 11,000 pages)** through over **2,800 web searches** and
   over **1,100 terminal queries**. In a second case, K3 analyzed **391 gravitational-wave events in GWTC-5**
   using more than **20 concurrent subagents**, producing seven scientific visualizations, two summary tables,
   and a literature synthesis of over ten papers.
6. **Video editing and motion design.** Leveraging its native multimodal architecture, K3 created a
   3Blue1Brown-style motion-graphics explainer **of its own architecture**, and edited its teaser video from
   **56 source clips** — clip selection, motion-matched cuts, frame-accurate beat synchronization, audio
   processing, and multiple rounds of revision. Producing a comparable high-density short video would
   typically take an experienced editor one to two days.