Claude 3.5 Sonnet

2024-06-20 · Publisher : Anthropic · Domain : Artificial Intelligence

Anthropic launches Claude 3.5 Sonnet, the first model of the 3.5 family: it outperforms Claude 3 Opus on most evaluations, at 2x the speed and a fifth of the price, and introduces Artifacts in Claude.ai.

What changes

  • A mid-tier model that outperforms Claude 3 Opus, then the top of the range, on reasoning, coding and vision evaluations.
  • Twice as fast as Claude 3 Opus and a fifth of the price: USD 3 input and USD 15 output per million tokens, versus 15 and 75.
  • Internal agentic coding evaluation: 64% of problems solved, versus 38% for Claude 3 Opus.
  • New records announced on GPQA (59.4%), MMLU (88.7% 5-shot) and HumanEval (92.0%), according to the scores published by the vendor.
  • Artifacts display generated code, documents and designs in a dedicated window next to the conversation.

Anthropic launches Claude 3.5 Sonnet, the first model of the Claude 3.5 generation. Positioned as a mid-tier model, it nevertheless outperforms Claude 3 Opus, the vendor's most powerful model until then, while being faster and much cheaper. ### What changes - Better than Claude 3 Opus across all published reasoning, math, coding and reading comprehension evaluations. - Twice as fast as Claude 3 Opus, at mid-tier pricing: a fifth of the price. - Agentic coding: 64% of problems solved on an internal evaluation (fixing a bug or adding a feature in an open source codebase), versus 38% for Claude 3 Opus. - Vision: the best Claude model on charts, diagrams and transcribing text from imperfect images. - Artifacts in Claude.ai: generated code, documents and designs appear in a dedicated window that can be edited as the conversation goes on. ### Specifications - 200,000-token context. - Available in Claude.ai (including the free tier), the iOS app, the Anthropic API, Amazon Bedrock and Google Cloud Vertex AI. - Rated ASL-2; evaluated before launch by the UK AI Safety Institute. ### Launch pricing | Model (USD per million tokens) | Input | Output | | --- | --- | --- | | Claude 3.5 Sonnet | 3 | 15 | | Claude 3 Opus (for comparison) | 15 | 75 | ### Benchmarks published by the vendor Scores published by the vendor (addendum to the Claude 3 model card). | Benchmark | Claude 3.5 Sonnet | Claude 3 Opus | GPT-4o | | --- | --- | --- | --- | | GPQA Diamond (0-shot CoT) | 59.4% | 50.4% | 53.6% | | MMLU (5-shot) | 88.7% | 86.8% | not disclosed | | HumanEval | 92.0% | 84.9% | 90.2% | | MATH (0-shot CoT) | 71.1% | 60.1% | 76.6% | | MMMU (vision) | 68.3% | 59.4% | 69.1% | | Agentic coding (internal) | 64% | 38% | not disclosed | ### Why it is a turning point The middle model of the new generation beats the top of the range of the previous one, at a fifth of its price. Claude 3.5 Sonnet becomes a reference for assisted software development, and Artifacts turn the conversation into a workspace. ### Key takeaways A model that is more capable, faster and cheaper than the previous flagship, which firmly establishes Anthropic in the coding space.

Links

Tags : LLM, Anthropic, Claude, Artifacts, Claude 3.5, Code, Multimodal