Qwen2.5-1M

2025-01-27 · Publisher : Alibaba · Domain : Artificial Intelligence

Alibaba Qwen releases Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, two open-weights models accepting up to 1 million tokens, with an adapted vLLM inference framework and sparse attention.

### Context At the end of 2024, Qwen's open models topped out at 128,000 tokens (Qwen2.5-128K). The official blog compares the new model with GPT-4o-mini, whose window is 128,000 tokens, and with Qwen2.5-Turbo, the version available through the API. The goal is to make very long context deployable by yourself, not only through a service. ### What the model brings - Two sizes, 7 billion and 14 billion parameters, in Instruct version, with a context of up to 1 million tokens. - Progressive training: the length goes from 4,000 to 256,000 tokens, then extrapolation up to 1 million relies on Dual Chunk Attention (DCA), a technique that reuses the positions seen during training without additional retraining. - Sparse attention based on MInference, integrated into the authors' vLLM inference framework: the vendor claims prefill that is 3.2 to 6.7 times faster on sequences of 1 million tokens. - Reliable information retrieval (passkey) up to 1 million tokens, with rare errors on the 7B model according to the blog. - Performance on short tasks close to GPT-4o-mini for the 14B, according to the vendor, with a context eight times longer. ### Architecture and specifications Dense Qwen2.5-type models (RoPE, SwiGLU, RMSNorm, QKV bias). The 14B has 14.7 billion parameters (13.1 billion excluding embeddings), 48 layers, GQA with 40 query heads for 8 key-value heads. Full context: 1,010,000 tokens, generation up to 8,192 tokens. Apache 2.0 license (declared on the Hugging Face page of the 7B and 14B models). A technical report is published (arXiv 2501.15383). ### Cost and access The weights are downloadable on Hugging Face and ModelScope. The blog gives the hardware requirements for 1 million tokens: at least 120 GB of VRAM for the 7B and 320 GB for the 14B (Ampere or Hopper GPUs recommended, CUDA 12.1 or 12.3). API pricing not verified for these models (Qwen2.5-Turbo, a separate model, is cited by the vendor). | Item | Value (source: official blog) | |---|---| | Minimum VRAM, 7B, 1M tokens | 120 GB | | Minimum VRAM, 14B, 1M tokens | 320 GB | | Prefill speedup | 3.2x to 6.7x | ### Benchmarks The detailed results (RULER, LV-Eval, LongBench-Chat, passkey) are provided as images in the blog, with no usable text table. Only vendor claims are therefore reported: the 14B-Instruct-1M outperforms Qwen2.5-Turbo and “consistently outperforms” GPT-4o-mini on several long datasets (scores published by the vendor, not transcribed here for lack of a text source). ### Versus closed models The only closed model cited by the vendor is GPT-4o-mini, limited to 128,000 tokens according to the blog, versus 1 million for Qwen2.5-14B-Instruct-1M, i.e. eight times more. The vendor claims parity on short tasks and an advantage on long tasks, without the figures being available in text form. The hardware cost is however high: 320 GB of VRAM to use the full context of the 14B, which puts full self-hosting out of reach of a single workstation. The price gap with the GPT-4o-mini API is not documented in the sources consulted. For a very-long-context use with a sovereignty requirement, the open option thus becomes technically possible but requires multi-GPU infrastructure. ### Key takeaways Qwen2.5-1M is Qwen's first milestone on very long context with open weights, with an inference framework delivered alongside the models. The memory requirements (120 to 320 GB) remain the main barrier to adoption.

Tags : LLM, Open Source, Alibaba, Qwen, Apache 2.0, Long contexte