Moonlight (Muon at scale)

2025-02-22 · Publisher : Moonshot AI · Domain : Artificial Intelligence

Moonshot AI releases Moonlight, a 16 billion parameter MoE (3 billion active) trained with the Muon optimizer, with code, checkpoints and a technical report under the MIT license.

### Context In February 2025, the AdamW optimizer dominates the training of large models, and Muon has only been validated on small models. Moonshot AI seeks to prove that it scales. Moonlight applies this idea to an open MoE, and Kimi K2 is described by the vendor as a scale-up of Moonlight. ### What the model brings - Two adjustments to scale Muon: a weight decay and an update magnitude (RMS) aligned between matrices and other parameters. - According to the vendor's scaling laws, Muon reaches AdamW's performance with about 52% of the training FLOPs (about 2 times more sample-efficient). - Distributed implementation of Muon, memory- and communication-efficient (ZeRO-1 style), published as open source. - Release of the pretrained, instruction-tuned and intermediate checkpoints. ### Architecture and specifications MoE with 15.29 billion parameters in total and 2.24 billion active (excluding embeddings, figures from the model card), trained on 5.7 T tokens, 8K context. Two checkpoints: Moonlight-16B-A3B and Moonlight-16B-A3B-Instruct. License: MIT (free commercial use). Weights: huggingface.co/moonshotai/Moonlight-16B-A3B. Report: arXiv 2502.16982 (submitted on 2025-02-24). ### Cost and access | Item | Detail | |---|---| | Official API | None | | Self-hosting | 16 B total model; required hardware not disclosed | | License | MIT | ### Benchmarks The model card compares Moonlight only with open models of similar size (scores published by the vendor). | Benchmark | Llama3.2-3B | Qwen2.5-3B | DeepSeek-V2-Lite | Moonlight | |---|---|---|---|---| | MMLU | 54.75 | 65.6 | 58.3 | 70.0 | | MMLU-Pro | 25.0 | 34.6 | 25.5 | 42.4 | | HumanEval | 28.0 | 42.1 | 29.9 | 48.1 | | MATH | 8.5 | 42.6 | 17.1 | 45.3 | | Training tokens | 9T | 18T | 5.7T | 5.7T | Source: Moonlight Hugging Face card. No comparison with a closed model is published: table limited to open models. ### Versus closed models Moonlight is not positioned against closed models: its size (3 B active) places it in the small model category, and no table in the source compares it with GPT-4o or Claude. Its value is methodological. At an equal number of tokens (5.7T) with DeepSeek-V2-Lite, it obtains better scores according to the vendor, which supports the idea of a gain of Muon over AdamW. The open-versus-closed choice does not arise here: this model serves as a research base and a validation of an optimizer. ### Key takeaways Moonlight is the link that validates Muon for the rest of the family (MuonClip in Kimi K2). It is mainly of interest for research and training, not for a production deployment. The figures on the card concern open models of the same size, and the optimizer code is published, which makes it possible to reproduce the experiment.

Tags : LLM, Open Source, Moonshot AI, Kimi, MIT, MoE, Muon