Llama 3.1 405B: an open model on par with the best closed models

2024-07-23 · Publisher : Meta · Domain : Artificial Intelligence

Meta releases Llama 3.1, whose 405B version is the first open-weight model presented as on par with GPT-4o and Claude 3.5 Sonnet. The gap between open and closed models narrows sharply.

On July 23, 2024, Meta released the Llama 3.1 family, including a 405-billion-parameter model. For the first time, a model with downloadable weights was presented as rivaling the best proprietary models of the moment. The same day, Mark Zuckerberg published a letter titled “Open Source AI Is the Path Forward.” ### What happened - Three models: the flagship 405B, plus updated 8B and 70B versions. - Context extended to 128,000 tokens and support for eight languages, including French. - A standard decoder-only transformer architecture, without mixture of experts, with iterative post-training (supervised fine-tuning, rejection sampling, direct preference optimization). - Meta also released Llama Guard 3 (a multilingual safety model) and Prompt Guard (a prompt injection filter). ### Key figures | Item | Value | |---|---| | Flagship model parameters | 405 billion | | Training data | over 15 trillion tokens | | GPUs used | over 16,000 H100 | | Context window | 128,000 tokens | | Launch partners | more than 25 | ### Why it is a turning point Meta called the 405B the “first frontier-level open source AI model.” A company can now host a leading model in-house, or with the provider of its choice, adapt it to its data and use it to generate synthetic data or train smaller models. The license explicitly allows this use, previously prohibited. ### Caveats The weights are open, but the Llama license is not an open source license as defined by the Open Source Initiative: it imposes usage conditions and a specific agreement above 700 million monthly active users. The training data is not published. Running the 405B requires substantial GPU infrastructure, beyond the reach of most workstations. ### Key takeaways Llama 3.1 405B showed that an open-weight model can approach the state of the art. For businesses, it is a credible option for sovereignty and cost control compared with closed APIs.

Tags : IA, Open Source, LLM, Meta, Llama