MiniMax-Text-01 (and MiniMax-VL-01)

2025-01-15 · Publisher : MiniMax · Domain : Artificial Intelligence

MiniMax releases MiniMax-Text-01 with open weights, a 456-billion-parameter MoE that combines Lightning linear attention and softmax attention, with a 4-million-token context at inference.

### Context In January 2025, the most widely used open models (Llama 3.1 405B, DeepSeek-V3, Qwen2.5-72B) topped out at roughly 128,000 tokens of context, while Gemini 1.5 Pro already offered a very long context on the closed side. MiniMax, a Chinese lab previously known mostly for its consumer applications, opens its first family of foundation models here, with a vision-language variant, MiniMax-VL-01. The goal is to make long context affordable without relying on quadratic attention on every layer. ### What the model brings - Hybrid architecture: according to the official post, in each block of 8 layers, 7 use Lightning Attention (linear attention) and 1 uses classic softmax attention. The compute cost then grows much more slowly with text length. - Context: 1 million tokens at training, extrapolated up to 4 million at inference (technical report), i.e. 20 to 32 times more than the competing models cited by the vendor. - Long-context retrieval: the vendor claims a 100% success rate on a needle-in-a-haystack test at 4 million tokens. - MoE with 32 experts and routing to 2 experts per token, and parallelism and compute/communication overlap strategies designed for training and inference at this scale. - A vision-language variant, MiniMax-VL-01, released the same day. ### Architecture and specifications - Parameters: 456 billion in total, 45.9 billion active per token, 80 layers, 32 experts (configuration published on Hugging Face). - Context: 1 million training tokens, up to 4 million at inference. - Modalities: text for Text-01, image and text for VL-01. - Training data and knowledge cutoff: not disclosed in the sources consulted. - License: MiniMax Model License (the vendor's own license, not an OSI license). According to the text reviewed, it grants a worldwide, non-exclusive, royalty-free right to use, copy and modify, including for commercial purposes, with no user threshold, with an acceptable use policy (military use, illegal content, harm to minors, fully automated decisions with legal effect) and an obligation to mention the license. Singapore law, SIAC arbitration. - Weights: https://huggingface.co/MiniMaxAI/MiniMax-Text-01 ### Cost and access | Item | Value | |---|---| | Official API, input | $0.2 per million tokens | | Official API, output | $1.1 per million tokens | | Self-hosting | hardware not disclosed in the sources consulted (456 billion parameters in total) | Pricing recorded from the January 15, 2025 post. ### Benchmarks Scores published by the vendor in the model card (same evaluation conditions for all models in the table). | Test | GPT-4o (11-20) | Claude 3.5 Sonnet (10-22) | Gemini 1.5 Pro (002) | MiniMax-Text-01 | |---|---|---|---|---| | MMLU | 85.7 | 88.3 | 86.8 | 88.5 | | MMLU-Pro | 74.4 | 78.0 | 75.8 | 75.7 | | GPQA Diamond | 46.0 | 65.0 | 59.1 | 54.4 | | HumanEval | 90.2 | 93.7 | 86.6 | 86.9 | | MATH | 76.6 | 74.1 | 84.6 | 77.4 | Source: MiniMax-Text-01 model card on Hugging Face, figures measured by the vendor. On the needle-in-a-haystack task at 1 million tokens, the same page gives 0.910 for MiniMax-Text-01 and 0.850 for Gemini 1.5 Pro, the other two models being absent at this length. ### Versus closed models At the time of its release, the closed references were GPT-4o and Claude 3.5 Sonnet, both evaluated in the vendor's table. The model is close to them on general knowledge (MMLU 88.5 versus 88.3 for Claude 3.5 Sonnet) and behind on scientific reasoning (GPQA Diamond 54.4 versus 65.0) and coding (HumanEval 86.9 versus 93.7). Its advantage lies in long context: 1 million tokens at a retrieval score of 0.910 versus 0.850 for Gemini 1.5 Pro, while GPT-4o and Claude 3.5 Sonnet are not measured at this length. The API at $0.2 input and $1.1 output is claimed by the vendor to be far below competing offers, but the post provides no quantified price comparison. For an open or closed choice, the license allows self-hosting and commercial use, provided you have hardware suited to a 456-billion-parameter model. ### Key takeaways MiniMax-Text-01 is the first open model of this size to bet on hybrid linear attention for long context. It does not match the best closed models on reasoning and coding, but it opens up uses on documents of several hundred thousand tokens with downloadable weights.

Tags : LLM, Open Source, MiniMax, MoE, Lightning Attention, Contexte long, Licence MiniMax Model License