Kimi K3: Open Frontier Intelligence

URL

Notes

ABSTRACT

native vision capabilitie (1)

2.8T parameter (1)

Mixture-of-Experts model (1)

1-million-token context window (1)

104 billion activated parameters (1)

Activated parameters is a feature of MoE models. 

This reduces cost of inference drastically compared to using all 2.8T parameters.

Kimi Delta Attention [63] and Attention Residuals [57], which improve information flow across sequence length and model depth (1)

Stable LatentMoE, which effectively activates 16 of 896 routed experts per token, and refined training and data recipes (1)

2.5× improvement in overall scaling efficiency over Kimi K2 (1)

Scaling Efficiency = Capability Gained / Extra Params

infrastructure advances in multiple areas: algorithm–system co-design for KDA, perfectly balanced expert-parallel training with efficient memory management, million-token agentic RL with persistent rollout and sandbox states, and deployment innovations (1)

1 Introduction

rise of reasoning models has established test-time computation as a second axis of scaling (2)

test-time scaling a central focus of frontier research (2)

axis, it has progressed slowly on the first: many recent models remain within or slightly above the 1T-class (2)

Kimi K3, we pursue both scaling axes together to the frontier: scaling the pre-trained foundation to unprecedented 3T-class parameters while scaling reinforcement learning, reasoning effort, and long-horizon interaction (2)

Kimi Delta Attention (KDA) [63] provides efficient longsequence mixing (2)

Gated MLA layers preserving global interaction (2)

MLA: Multi-head Latent Attention

Attention Residuals (AttnRes) [57] allows each layer to selectively attend to representations from all preceding layers (2)

Stable LatentMoE expands the routed expert space to 896 experts, with 16 activated per token (2)

normalization, SiTU-GLU, and Quantile Balancing stabilize optimization at extreme sparsity (2)

SiLU: Sigmoid Linear Unit

GLU: Gated Linear Unit

Training environments (2)

verifiable search (2)

professional knowledge work (2)

software engineering (2)

kernel optimization (2)

vision-in-the-loop tool use (2)

persistent assistant workflows (2)

web development (2)

autonomous execution tasks (2)

Domain- and effort-specialized policies are consolidated into a unified model through multi-teacher on-policy distillation (2)

For systems co-design for KDA, we develop fused kernels, KDA Context Parallelism, and state-aware prefix caching to make KDA efficient within devices, across devices, and across requests (2)

MoonEP provides perfectly balanced expert execution with static computation shapes and zero-copy communication (2)

memory efficient training and multimodal encoder optimizations sustain utilization within bounded memory (2)

For million-token agentic RL, our co-located system combines partial rollouts, external KV-cache retention, adaptive throttling and resumable microVM sandboxes to preserve long-lived model and environment state (2)

3 KDA followed by 1 Gated MLA.

w: pseudo-queries alpha: attention weights

Stable Latent MoE Module.

Native Vision Pathway.

KDA Module.

2 Model Architecture

sequence dimension, Hybrid Attention combines three Kimi Delta Attention (KDA) [63] layers with one Gated MLA layer in each block (3)

three complementary dimensions: sequence length, network depth, and model width (3)

depth dimension, Attention Residuals (AttnRes) [57] enable each module to selectively retrieve representations from the embedding, the current block, and preceding blocks (3)

width dimension, each attention layer is followed by a Stable LatentMoE layer that performs sparse channel mixing (3)