Kimi Linear (48B-A3B)

2025-10-30 · Publisher : Moonshot AI · Domain : Artificial Intelligence

Moonshot AI releases Kimi Linear, a 48-billion-parameter MoE (3 billion active) with hybrid linear attention (Kimi Delta Attention), which reduces the KV cache by up to 75% and speeds up decoding by up to 6 times.

### Context Full attention is expensive in memory (KV cache) and in decoding time as context grows. Linear attention architectures promise better, but until now lost in quality against full attention. At the end of October 2025, Moonshot AI publishes a hybrid architecture that claims to surpass it under equal training conditions. ### What the model brings - Kimi Delta Attention (KDA): a refinement of Gated DeltaNet with a finer gating mechanism to make better use of the finite state memory. - Hybrid architecture with 3 KDA layers for 1 global MLA attention layer. - According to the vendor, KV cache reduction of up to 75% and decoding throughput up to 6 times higher for contexts of up to 1 million tokens. - KDA kernel released as open source in the FLA library. - This building block becomes the basis of the Kimi K3 architecture. ### Architecture and specifications 48B total parameters, 3B active, 1M-token context, two checkpoints (Base and Instruct) trained on 5.7 T tokens. Text only. License: MIT. Weights: huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct. Report: arXiv 2510.26692. Recommended environment: Python 3.10 or higher, PyTorch 2.6 or higher, fla-core 0.4.0 or higher. ### Cost and access | Item | Detail | |---|---| | Official API | None in the sources consulted | | Self-hosting | 48 B total model; required hardware not disclosed | | License | MIT | ### Benchmarks The model card and the report compare Kimi Linear to a full-attention (MLA) model of the same size, not to closed models. Figures cited by the vendor: 51.0 on MMLU-Pro (4K context) at a speed comparable to full attention; 84.3 on RULER (128K) with a 3.98x speed gain; 6.3 times faster on time per output token (TPOT) at 1 M tokens. | Measure | Reported result (vendor) | |---|---| | MMLU-Pro (4K) | 51.0 | | RULER (128K) | 84.3 (3.98x speedup) | | TPOT at 1M tokens | 6.3x faster than MLA | | KV cache | up to 75% less | Source: Hugging Face card of Kimi-Linear-48B-A3B-Instruct (vendor measurements). No comparison table with a closed model is published. ### Versus closed models This sheet cannot place the model against closed models: the source publishes no such comparison, and at 3B active parameters it does not target the same category. Its interest lies elsewhere, in the cost of long-context inference. The KV cache and speed gains are vendor claims measured against full attention of the same size. For an open or closed choice, the contribution is indirect: the architecture reappears in Kimi K3, a much larger model, where the vendor cites it as the foundation of the API's competitive price. ### Key takeaways Kimi Linear is an architecture milestone: it validates hybrid linear attention at scale and paves the way for K3. It is aimed at teams working on very long contexts and inference efficiency.

Tags : LLM, Open Source, Moonshot AI, Kimi, MIT, MoE, Attention linéaire