Llama Nemotron (Nano, Super, Ultra)

2025-03-18 · Publisher : NVIDIA · Domain : Artificial Intelligence

NVIDIA launches the Llama Nemotron family at GTC, reasoning models derived from Llama in three sizes (8B, 49B, 253B), released with their weights, their post-training dataset and their code.

### Context At the start of 2025, open reasoning models are multiplying after DeepSeek-R1, against which NVIDIA's technical report compares itself. NVIDIA chooses to start from existing Llama models and rework them through architecture search, distillation and post-training, rather than pretraining from scratch. The Nano and Super models are available at launch, Ultra follows on April 7, 2025. ### What the model brings - Three tiers: Nano (PC and edge), Super (best trade-off between accuracy and throughput on a single data center GPU) and Ultra (maximum accuracy on multiple GPUs), according to the NVIDIA press release. - A reasoning toggle at inference: reasoning mode can be turned on or off on demand, which the technical report presents as a first for open models. - A lighter architecture through neural architecture search: attention blocks replaced by linear layers, variable feed-forward widths, feed-forward layer fusion (FFN Fusion) on Ultra. - A three-stage training on Ultra: knowledge distillation (65 billion tokens), continued pretraining (88 billion tokens), then supervised fine-tuning and reinforcement learning (GRPO). - The Llama-Nemotron-Post-Training-Dataset and the NeMo, NeMo-Aligner and Megatron-LM frameworks are released with the weights. ### Architecture and specifications Nano 8B and Super 49B (8.03 and 49.87 billion parameters respectively according to the Hugging Face metadata) are dense models. Ultra-253B is a dense decoder derived from Llama-3.1-405B-Instruct, 128K-token context. Ultra languages: English, code, German, French, Italian, Portuguese, Hindi, Spanish, Thai. License: NVIDIA Open Model License (commercial use allowed according to the cards). As the models derive from Llama, the terms of the Llama license probably apply as well (point not verified in the cards consulted). Openness: weights, post-training dataset and training code are open; the Llama pretraining corpus is not open, so this is an open-weights family with open post-training, not fully open. ### Cost and access | Item | Detail | |---|---| | Official API | Trial on build.nvidia.com; free use for development, testing and research for NVIDIA Developer Program members; production through NVIDIA AI Enterprise (NVIDIA press release). Pricing per million tokens not disclosed. | | Ultra self-hosting | One 8 x H100 node (Hugging Face card); FP8 variant published on April 30, 2025 | | Nano/Super self-hosting | Nano for PC and edge, Super for one data center GPU (NVIDIA positioning, exact hardware not included here) | ### Benchmarks | Benchmark (Ultra-253B, published by NVIDIA) | Reasoning off | Reasoning on | |---|---|---| | GPQA | 56.60 | 76.01 | | AIME25 | 16.67 | 72.50 | | MATH500 | 80.40 | 97.00 | | LiveCodeBench | 29.03 | 66.31 | Source: Hugging Face card nvidia/Llama-3_1-Nemotron-Ultra-253B-v1, scores measured by NVIDIA. The technical report (arXiv 2505.00949) reports performance comparable to DeepSeek-R1, an open model. ### Versus closed models The sources consulted contain no numerical comparison with a closed model, and no time-matched comparison with the best closed model of March-April 2025 was established from a source. NVIDIA announces up to 20% more accuracy compared with the base model and up to five times higher throughput than other open reasoning models, without figures in the press release. The gap with closed models therefore cannot be quantified here. The value lies in self-hosting on a single 8 x H100 node (Ultra) and in the ability to turn reasoning off to save tokens. The NVIDIA license allows commercial use, subject to the terms of the underlying Llama license. ### Key takeaways Llama Nemotron opens the way for NVIDIA reasoning models with a published dataset and training code. The reasoning on/off toggle becomes a selection criterion for controlling token cost.

Tags : LLM, Open Source, NVIDIA, Llama Nemotron, Raisonnement, NVIDIA Open Model License