OLMo 2 32B

2025-03-13 · Publisher : Ai2 · Domain : Artificial Intelligence

Ai2 releases OLMo 2 32B, a fully open 32-billion-parameter model (weights, data, code) that, according to Ai2, outperforms GPT-3.5 Turbo and GPT-4o mini on academic benchmarks.

### Context OLMo 2 had existed in 7B and 13B since November 2024. The 32B is its largest model and the first fully open model that Ai2 says is superior to small closed OpenAI models. It also serves as a testbed for the Tülu 3.1 post-training recipe. ### What the model brings - Training on 6 trillion tokens (1.5 epochs of OLMo-Mix-1124) followed by Tülu 3.1 post-training: SFT, DPO and RLVR. - A pretraining cost announced at one third of that of Qwen 2.5 32B for comparable performance. - Ai2 states that the model matches or exceeds GPT-3.5 Turbo, GPT-4o mini, Qwen 2.5 32B and Mistral 24B, and comes close to Qwen 2.5 72B and Llama 3.1 and 3.3 70B. - A cluster of 160 nodes of 8 H100 GPUs on Google Cloud, with more than 1,800 tokens per second per GPU (about 38% MFU). - Release of the pretraining data (OLMo-Mix-1124), mid-training data (Dolmino-Mix-1124), post-training data and the OLMo-core code. ### Architecture and specifications 32 billion parameters. License: Apache 2.0 (HF model card of the Instruct). Full openness: weights, pretraining, mid-training and post-training data, training code, logs and details. Context: not specified in the pages consulted (the card mentions a maximum length of 2,048 tokens for the post-training training configuration, without this being the model's limit). Weights: OLMo-2-0325-32B (base), SFT, DPO and Instruct on Hugging Face, compatible with Transformers and vLLM. ### Cost and access | Item | Detail | |---|---| | Official API | None; demo in the Ai2 Playground | | Hardware | Not disclosed; about 64 GB in BF16 (32 billion parameters, calculation) | ### Benchmarks | Benchmark (HF card, Instruct) | OLMo 2 32B Instruct | Llama-3.3-70B | Qwen2.5-32B | |---|---|---|---| | Average | 68.8 | 73.0 | 66.5 | | MATH | 49.7 | 71.8 | 77.9 | | GSM8k | 87.6 | 93.6 | 87.5 | | MMLU | 77.3 | 85.9 | 84.7 | Source: Hugging Face card allenai/OLMo-2-0325-32B-Instruct, published by Ai2; open comparators. The Ai2 post cites GPT-3.5 Turbo and GPT-4o mini without figures that can be extracted from the text. ### Versus closed models Ai2 states that the model matches or exceeds GPT-3.5 Turbo and GPT-4o mini on academic benchmarks; the detailed figures could not be recorded, so this claim remains the vendor's. At the time of its release, these two closed OpenAI models were small or already old, not frontier models. Against open models, the card's table places the 32B ahead of Qwen2.5-32B on average (68.8 versus 66.5) but behind Llama-3.3-70B (73.0) and clearly behind on MATH (49.7 versus 77.9 for Qwen). The value lies in full reproducibility, useful for research and auditing, more than in raw performance. ### Key takeaways OLMo 2 32B is a transparency milestone: a 32-billion-parameter model whose entire training is published. Its performance remains behind the best open models of comparable size on math.

Tags : LLM, Open Source, Entièrement ouvert, Ai2, OLMo 2, Apache 2.0, Poids, données et code ouverts