Nemotron-H

2025-03-21 · Publisher : NVIDIA · Domain : Artificial Intelligence

NVIDIA releases Nemotron-H, a family of hybrid Mamba-2 and Transformer models (8B, 47B and 56B) that replaces most of the attention with Mamba layers to speed up inference.

### Context The inference cost of standard Transformers grows with context length. NVIDIA here releases a hybrid family trained at scale, whose architecture is later reused in Nemotron Nano 2 and then Nemotron 3. Only base models are available at launch; the Instruct and reasoning variants are announced for later. ### What the model brings - Replacement of most attention layers with Mamba-2 layers, which require constant memory per generated token. - A throughput announced at up to 2.9 times higher for the 47B against Qwen-2.5-72B and Llama-3.1-70B on long context (65,536 tokens), and 1.8 and 3 times for the 8B against Qwen-2.5-7B and Llama-3.1-8B. - FP8 pretraining with per-tensor scaling, presented as one of the largest public demonstrations of this format: 56B on 20 trillion tokens with 6,144 H100 GPUs. - A 47B model obtained by compressing the 56B, for a trade-off between accuracy and speed. - Weights published in Hugging Face and NeMo formats, with FP8 support in the NeMo framework. ### Architecture and specifications Models: 8B (15 trillion tokens), 47B and 56B (20 trillion). The 56B mainly combines Mamba-2 and MLP layers with ten attention layers (HF card), 8K context for the base models, data cutoff September 2024, languages: English, German, Spanish, French, Italian, Korean, Portuguese, Russian, Japanese, Chinese. License: NVIDIA Internal Scientific Research and Development Model License, limited to research and development, so no free commercial use. Training data not published with the announcement in the pages consulted. The card mentions the use of synthetic data generated with Qwen. ### Cost and access | Item | Detail | |---|---| | Official API | Not disclosed | | Hardware | Optimized for H100 80 GB and A100 (56B HF card); memory requirements not detailed | | Weights | Base 8B, 47B and 56B on Hugging Face and NGC; 56B upload date on HF: April 14, 2025 | ### Benchmarks | Benchmark (published by NVIDIA) | Nemotron-H 47B | Qwen-2.5 (72B) | Llama-3.1 (70B) | |---|---|---|---| | MMLU-Pro (project page) | 61.8 | 58.8 | 51.3 | Additional, 56B card: MMLU 84.21 (5-shot), GSM8K 93.71 (8-shot CoT), HumanEval 60.37, ARC Challenge 94.97 (25-shot). Source: NVIDIA ADLR project page (research.nvidia.com/labs/adlr/nemotronh/) and Hugging Face card nvidia/Nemotron-H-56B-Base-8K, NVIDIA measurements. Report: arXiv 2504.03624. ### Versus closed models Nemotron-H is compared by NVIDIA with open models (Qwen-2.5, Llama-3.1), not with closed models, and no comparison with a closed model was found in the sources consulted. Its contribution is inference efficiency: the announced throughput on long context is 2.9 times that of larger open models. The research license is more restrictive than that of closed models accessible through an API for commercial use, which limits its practical value in enterprises for this generation. The milestone is primarily architectural. ### Key takeaways Nemotron-H validates the Mamba-Transformer hybrid and FP8 pretraining at scale, two choices later reused in Nano 2 and Nemotron 3. In enterprises, the research license rules out productive use.

Tags : LLM, Open Source, NVIDIA, Nemotron-H, Mamba, Licence recherche