Olmo Hybrid (7B)

2026-03-05 · Publisher : Ai2 · Domain : Artificial Intelligence

Ai2 releases Olmo Hybrid, a fully open 7-billion-parameter model that alternates attention and Gated DeltaNet layers, and matches Olmo 3's accuracy on MMLU with 49% fewer tokens.

### Context After Olmo 3, Ai2 is testing a hybrid architecture, in line with the Mamba-Transformer trend at other vendors. The question is whether hybrids scale better than pure Transformers. ### What the model brings - A 3-to-1 architecture: three Gated DeltaNet sublayers for one multi-head attention sublayer. - On MMLU, the same accuracy as Olmo 3 with 49% fewer tokens, or about twice the data efficiency in controlled studies. - On long context (64K), a RULER score of 85.0 versus 70.9 for Olmo 3 7B. - A token efficiency estimated at about 1.3 times (1 billion parameters) up to 1.9 times (70 billion) at scale, according to Ai2. - A technical report on theoretical expressivity, scaling laws and ablations on the hybrid ratio. ### Architecture and specifications 7 billion parameters pretrained on 6 trillion tokens with improved data from Olmo 3 32B. Context extended to 64K tokens with DRoPE. License: Apache 2.0 (Olmo-Hybrid-7B HF model card). Openness: models, technical report and data announced by Ai2 as fully open; the detail of the published artifacts (code, checkpoints) was not listed on the page consulted. Limitations: slightly behind Olmo 3 on code and general questions early on, a gap that narrows after mid-training. ### Cost and access | Item | Detail | |---|---| | Official API | None | | Hardware | Not disclosed; about 15 GB in BF16 (7.43 billion parameters, calculation) | | Weights | Olmo-Hybrid-7B (January 28, 2026 on HF), Instruct SFT/DPO and Think SFT variants (February 2026) | ### Benchmarks | Benchmark (published by Ai2) | Olmo Hybrid 7B | Olmo 3 7B | |---|---|---| | MMLU, tokens needed for the same accuracy | 51% of tokens | 100% | | RULER 64K | 85.0 | 70.9 | Source: Ai2 post of March 5, 2026, Ai2 measurements; comparison with Olmo 3, a model from the same vendor. ### Versus closed models No comparison with a closed model is presented. The milestone is architectural: Ai2 compares Olmo Hybrid to its own previous generation and not to competing models. The RULER 64K score of 85.0 versus 70.9 shows a gain on long context, without this being linked to a closed model. The token efficiency gains at large scale are extrapolation estimates by Ai2 derived from scaling laws, not direct measurements at 70 billion parameters. The model remains small (7 billion parameters), which limits the comparison with large closed models. The interest for enterprises is indirect: it signals that hybrid architectures are gaining ground in fully open models as well. ### Key takeaways Olmo Hybrid brings a more data-efficient hybrid architecture to the fully open family. It is a research signal more than a production model at 7 billion parameters.

Tags : LLM, Open Source, Entièrement ouvert, Ai2, Olmo Hybrid, Apache 2.0, Architecture hybride